How I Accidentally Discovered Crypto-Miner Malware (perfctl) on an Ubuntu Server

It all started with an Upwork job described as follows:

Help Installing MongoDB CE 8 on Ubuntu 24.04 LTS

I followed the instructions here:
https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-ubuntu/
But obviously, I’m missing something. Any help would be greatly appreciated.

I submitted my proposal, thinking it would be a straightforward task—just follow the MongoDB installation guide, and it’d be done in minutes.

Turns out, it was far more frustrating than I had anticipated.

The installation succeeded, but when I attempted to connect to the database using mongosh, I encountered this error:

$ mongosh
Current Mongosh Log ID:	677b0ed656630de8e9d4b0c1
Connecting to:		mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+2.3.7
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017

This error suggests that the mongod service was likely not running. To confirm, I checked the mongod service status:

$ sudo systemctl status mongod
● mongod.service - MongoDB Database Server
     Loaded: loaded (/usr/lib/systemd/system/mongod.service; disabled; preset: enabled)
     Active: active (running) since Mon 2025-01-06 06:04:44 WIB; 7s ago
       Docs: https://docs.mongodb.org/manual
   Main PID: 56087 (mongod)
     Memory: 183.7M (peak: 270.9M)
        CPU: 920ms
     CGroup: /system.slice/mongod.service
             └─56087 /usr/bin/mongod --config /etc/mongod.conf

The status indicated that the mongod service was running. However, there were inconsistencies that suggested otherwise:

1. The /tmp/mongodb-27017.sock file, which should exist, was missing.
2. The log files in /var/log/mongodb were absent.
3. The data files in /var/lib/mongodb were also missing.

These signs indicated that mongod wasn’t actually running despite the service status. To investigate further, I stopped the service and tried to run the mongod binary manually:

$ sudo systemctl stop mongod
$ ./mongod --help
# It should display help information instantly,
# but it hangs instead.

Surprisingly, mongod still refused to run, hanging indefinitely instead of displaying the expected help information.

I tried purging and reinstalling MongoDB, checking the ownership and permissions of /var/lib/mongodb and /var/log/mongodb, and investigating other usual suspects.
But mongod still wouldn’t run.

To narrow down the issue, I created a fresh Ubuntu 24.04 VM on DigitalOcean and installed MongoDB there. It worked perfectly, proving that the installation instructions and MongoDB binaries were not the problem.
The culprit had to be something specific to my client’s VM.

After further searching and troubleshooting, I still couldn’t pinpoint the issue. Frustrated, I offered my client a Docker container solution as a last resort.
They agreed, so I wrote a Docker Compose file, and MongoDB started successfully on their VM.

That seemed like the end of the job.

However, my client preferred a bare-metal installation for certain reasons. Driven by curiosity, I decided to investigate further.

Enter strace: A Diagnostic, Instructional, and Debugging Tool for the Linux Kernel

I asked ChatGPT how to debug a Linux binary that refuses to run.
It suggested several options, and I chose strace for its simplicity.

I installed and ran strace on both my client’s VM and my DigitalOcean VM:

$ sudo apt install strace
$ strace -o mongod_strace.log -f mongod --help

I retrieved the log files from both VMs using scp and compared them.
You can check the logs yourself: bad mongod strace log and good mongod strace log.

At first, the logs made little sense to me.
However, I noticed patterns suggesting library loading operations, such as:

1942061 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcurl.so.4", O_RDONLY|O_CLOEXEC) = 3

Using grep, I filtered the libraries being accessed and compared the results. Initially, nothing seemed amiss—until I found this peculiar line in the bad log:

94917 openat(AT_FDCWD, "/lib/libgcwrap.so", O_RDONLY|O_CLOEXEC) = 4

While most libraries were being loaded from /lib/x86_64-linux-gnu/, this one came from /lib/libgcwrap.so.

What could it be?

Uncovering the Malware

I searched online for information about /lib/libgcwrap.so and stumbled upon a Ubuntu bug report.

A comment from cemarrio caught my attention:

I shut down the VM and used a live ISO to take a look at the filesystem. You were correct - there is an /etc/ld.so.preload which points to /lib/libgcwrap.so. There is also a /usr/lib/libgcwrap.so copy.

These files were marked as immutable and were not visible from the booted system. I took a look at the cron configurations and found the root user had an entry for “perfcc”. Looking this up, perfcc/perfctl is a crypto-miner malware.

Thanks for all your help in troubleshooting this. I hope it clarifies for someone what may be afoot if they find libgcwrap present on their system; there was nothing available on the popular search engines about it.

If you examine the strace log closely, you’ll notice that the mongod process first attempts to open /etc/ld.so.preload. However, the malware had modified /etc/ld.so.preload to point to /lib/libgcwrap.so, causing mongod to fail to run.

Following this lead, I also inspected the cron configuration on my client’s VM and discovered an entry for perfclean, which executed perfcc every hour!

$ cat /etc/cron.daily/perfclean
#!/bin/sh
perfcc

I informed my client about the suspected crypto-miner malware, providing references.

When they learned that mongod’s refusal to run stemmed from malware, we couldn’t help but laugh at the absurdity of it all.

The Solution

Fortunately, my client’s VM data was not corrupted.
They decided to wipe the VM and reinstall a fresh Ubuntu 24.04.

We followed the MongoDB installation guide again, and everything worked perfectly—no more malware or mongod issues!

The client closed the contract and left me a 5-star review on Upwork!

Conclusion

I hope this case helps anyone facing a similar issue.
Stay patient, keep troubleshooting, and don’t give up!

Thanks for reading, and see you in my next post!

References

• perfctl: A Stealthy Malware Targeting Millions of Linux Servers
• Bug #1641230 “/usr/share/initramfs-tools/hooks/fsck failed with …” : Bugs : initramfs-tools package : Ubuntu