What the hell is that title and how the hell did we get here?
Docker is one of humanity’s greatest technologies and frankly one of my favorite headaches. As someone who’s extremely hellbent on organization, Docker is definitely one of my favorite sysadmin tools, and I almost always organize big projects into Docker containers (or Docker Compose files) to aid in getting a project from zero to running within minutes. Docker is, however, not without its flaws, and a particularly problematic one happened to strike the Hourglass server this past month.
On 21 May 2023, a threat actor began mass-sending emails to Chinese email provider QQ through Hourglass’ SMTP service. Alerting for this had not been set up; this was the first time that a threat like this had appeared on Hourglass servers. Postfix logs were promptly dumped and the server was shut down for a few minutes for investigation.
A bit of background on the mail delivery setup for Hourglass: Postfix, Dovecot, and other mail-related services are handled through docker-mailserver, a single-container setup that contains the entire mail stack. This setup is particularly much more advantageous compared to manually setting up Postfix, Dovecot, and the rest on the actual server, which in turn decentralizes logs and configuration data across the server’s filesystem. As the name implies, the container runs through Docker, and the relevant ports are exposed to allow service communication.
Two things stood out from the findings initial shutdown: (1) The origin IP of the malicious sender appeared to be a private address, particularly the Docker host’s, and (2) the threat actor only targeted QQ email addresses, avoiding other providers entirely. Point (2) is of significant note, as it means the server did not suffer any “reputational damage” which would have resulted in it being listed on a mailserver blacklist. Point (1), however, was a matter of significant horror.
Initially, I had assumed that this was only another nftables issue, as Docker does not set NAT rules after nftables is restarted. This would mean that any public access reflects as private access, with the host IP being shown as the source. However, some IP addresses in the logs were public IP addresses, so an nftables issue was unlikely. Out of choices, I restarted the server and introduced alerting in case message sending went past a realistic limit (as the server only ever sends mail once at a time — whenever I tell it to).
This same threat actor reappeared today, 15 June 2023. After identifying it, the newly-introduced relevant emergency scripts were run and the container was locked down, logs were preserved, and running processes and connections were recorded. This showed the same issue as it did in the first attack: the mail originated from a private IP address alongside public IP addresses. Looking at the dump of active connections, there were no active programs on the server connected to port 25, so a malicious program running on the system was ruled out. This, however, still does not explain the IP issue. After some pondering, I remembered this passage from docker-mailserver’s documentation:
WARNING: Adding the docker network’s gateway to the list of trusted hosts, e.g. using the
Environment Variables – Docker Mailservernetworkorconnected-networksoption, can create an open relay, for instance if IPv6 is enabled on the host machine but not in Docker.
Looking back at the netstat for the system while the attack was ongoing, this does seem to be the case.
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 280481/docker-proxy
tcp6 0 0 :::25 :::* LISTEN 280487/docker-proxyDocker had bound port 25 not just on 0.0.0.0, but also on ::. Though it accommodated IPv4 NAT and reflected the correct addresses there, it did not accommodate IPv6 NAT, and instead pushed all connections down the IPv4 network of the container. The result? All IPv6 connections appeared to originate from the host container’s internal IPv4 address.
Whoops!
While debugging all this in the Mapúa University library, I had no access to IPv6. Therefore, there was no way for me to test if Hourglass had been susceptible to this, especially as any attempt at trying to get a machine with IPv6 address failed. Thus, an experiment was performed.
I re-started the server and intentionally waited for the malicious emails to be sent again. As soon as this happened, I removed the IPv6 addresses from the Hourglass machine, stopping all IPv6 access to the server. This reflected as a few minutes of Hourglass 01 downtime on status.chlod.net, but the machine (and its services) were actually up. Lost connections were also gracefully restored on Hourglass following the sudden disconnection of IPv6. Most notably, the attack had stopped in its tracks, and the issue had been determined.
After changing the Docker configuration a bit, namely changing the port binding from 25:25 to 0.0.0.0:25:25 (exclusive IPv4 bind), the mail container was restarted and IPv6 access to the server was restored. After checking netstat post-changes, port 25 was no longer open through IPv6, and thus the server was no longer susceptible to this attack.
A proper long-term solution to this would be to properly configure IPv6 forwarding for Docker and to add an IPv6 network to the container, however this is already beyond the scope of resolving this incident, and would take some time to perform. An estimated 100+ messages were sent to various QQ addresses during the two attacks; most of these being incomprehensible Chinese gibberish (quite literal spam mail, if you will).
This postmortem is written as a cautionary tale for aspiring mailserver-havers. When dealing with NAT and forwarding with Docker, watch out for IPv6. You do NOT want people using your server as an open relay. Oh, and read documentation properly.
