Joomla 3 forensics: 47 hidden admins and a cron in /tmp

The handover

It was a Friday in October. A logistics company in Rotterdam had fired their previous agency and asked us to take over their Joomla site. The credentials arrived in a single PDF: SSH, FTP, database, and a phpMyAdmin link with no port restriction and no IP allowlist. We opened the link to verify it worked. It loaded. We were not logged in. We could log in with the root MySQL user from the credentials sheet.

That was minute one. We spent the next six hours making the bleeding stop.

Mapping the damage before touching it

Rule one when you inherit a site you suspect is compromised: do not start cleaning. You take a snapshot first, then you map. We pulled a full filesystem tarball over SFTP, a mysqldump of the database, and a copy of /var/log and /etc/cron.d. Everything went to an isolated VPS we keep for forensics. Only then did we open phpMyAdmin again, this time through an SSH tunnel after killing the public listener.

The #__users table had 47 rows. The client had told us there were "maybe three or four" admins. We sorted by registerDate and the picture sharpened: nine accounts created in the first year of the site, 38 created at 03:14, 03:15, 03:16 UTC on a single night in 2022. All in the super user group. All with lastvisitDate of NULL. Accounts that were never used. Just there, waiting.

SELECT id, username, email, registerDate, lastvisitDate
FROM j_users
ORDER BY registerDate DESC;

-- 38 rows clustered within three minutes, all NULL lastvisitDate.
-- 9 rows from the legitimate site history.
-- The remaining gap was the moment the site changed hands.

The phpMyAdmin nobody owned

Public phpMyAdmin is the kind of finding that ages you. The install at /pma was version 4.7.0, released in 2017. The Joomla front-end was version 3.10.x, which the project end-of-lifed on 17 August 2023 (see the official Joomla 3.10 end-of-life note). The site had been past EOL for over two years and the patch budget had been zero.

phpMyAdmin in that version had three years of unpatched advisories on top of whatever bespoke trouble was sitting in config.inc.php. We did not open the file with curiosity. We moved the directory out of the webroot and set a firewall rule to drop traffic to port 3306 from anywhere except localhost.

/tmp wasn't temporary

The cron caught my eye on the second pass through /etc/cron.d. A user-mode crontab for www-data ran /tmp/.x every five minutes. /tmp/.x was a 17 KB ELF binary. strings showed callbacks to a domain registered three weeks earlier. The file mtime claimed 2019. The inode change time said two months ago.

This is the standard shape of webshell persistence on a PHP site. The attacker drops a payload through an upload form or an unpatched component, registers an @reboot or */5 cron, and stops needing the original entry point. The web server is the toehold. Cron is the lease.

We killed the cron, removed the binary, and audited every other cron entry against the client's documentation. There were two more we did not expect: a daily job in /var/spool/cron/crontabs/root that pulled and ran a shell script from a paste URL, and an @hourly line that re-wrote /tmp/.x if it ever went missing. The cron had a guardian cron. The guardian had a guardian. We dug until we found nothing else.

# What we actually ran, in order, after isolating the box.
sudo crontab -l -u www-data
sudo crontab -l -u root
ls -lat /tmp /var/tmp /dev/shm
find / -newer /etc/hostname -type f 2>/dev/null | head -200
ss -tlnp

The order we drained the swamp in

You triage by blast radius. We worked in this order, and we recommend it to anyone walking into a compromised PHP site.

1. Snapshot everything offline. Filesystem, database, all logs, all crontabs, /etc/passwd, the lot. Put it on a host the attacker has never seen.
2. Cut public access to admin surfaces: phpMyAdmin, the Joomla admin URL, anything on the same host that does not need to be on the open internet. A Cloudflare Access rule or an Apache Require ip block will both do the job in under a minute.
3. Rotate every credential the previous agency had. In this order: MySQL, SSH, FTP, Joomla super user, email, DNS registrar. The DNS registrar matters more than people think. If they own DNS, they own your email recovery flow.
4. Audit #__users and #__user_usergroup_map. Look at registerDate, lastvisitDate, requireReset. Anomalies cluster.
5. Audit cron, at, systemd timers, and the user crontab for www-data. PHP sites get exploited through the web user, so that user's automation is where the persistence lives.
6. Audit /tmp, /var/tmp, /dev/shm for ELF binaries and shell scripts that should not be there.
7. Only then, start migrating.

Migration was the cure, not patching

We did not try to patch Joomla 3.10 back to clean. The component graveyard alone made it pointless: the client had four commercial extensions whose vendors had folded, and the bridge tooling for Joomla 3 to 4 was already deprecated for two of them. A site EOL'd two years ago does not get cured. It gets replaced.

We rebuilt the site on a current stack over four weeks, ported the content tables across with a small Python script, and decommissioned the old VPS after a clean snapshot. The new admin surface sits behind Cloudflare Access. phpMyAdmin is not installed; the few times we need it, it runs locally over an SSH tunnel. Database connections only accept localhost. Cron is version-controlled in /etc/cron.d with file integrity monitoring via aide. We do not run binaries from /tmp.

What the client did not know they were paying for

The client had been billed monthly by the previous agency for "security updates." There had not been a Joomla core update applied in 19 months. No patches had been applied to any third-party components. The phpMyAdmin install was unmanaged. The compromised cron had been live for at least two months before we arrived, based on the binary's inode timestamps and the registrar date on the command-and-control domain.

This is the standard pattern when a legacy site is "maintained" without anyone looking at the actual server. The invoice keeps going out. The audit trail is empty. Then one day the brand starts getting reported to Google Safe Browsing and the calls land in your inbox.

A useful baseline: OWASP's current Top Ten still puts broken access control at number one and outdated components at number six. Both were in play on this site. Neither finding would have surprised any auditor who actually logged in.

Two checks any operator can run this afternoon

If you run an older PHP-based site and you cannot afford a full audit this quarter, run these two checks before the week is out. They take ten minutes.

Check one: log into the admin and open the users table. Count the super users. If the number does not match the names of the people in your company who should have super user access, your problem is now urgent, not theoretical. Pay attention to lastvisitDate values that are NULL: a super user who has never logged in is almost never a real user.

Check two: SSH to the box and run crontab -l -u www-data, then ls -lat /tmp /var/tmp /dev/shm. If anything in those directories has a recent mtime you cannot explain, you have homework. Anything that looks like a dotfile binary in /tmp is homework with a deadline.

When we built the migration off this Joomla 3 site, the thing we ran into was that two of the client's custom components stored content as serialized PHP, which does not survive a clean copy out of MySQL into a new schema. We ended up writing a small unserializer that emits flat JSON, then imported into the new CMS through its content API. That kind of one-off bridge is most of what a legacy migration looks like in practice. The clean-up is the easy half; the data shape is the hard half.

If the only thing you do this week is run crontab -l -u www-data on every PHP server you operate, you will be ahead of where most agencies leave their clients. That is the bar.