My VPS recently got banned for spam which surprised me since none of my soft there sending email. So my first thoughts were that this is a mistake (e.g. due to IP address spoofing) or some vulnerability in that VPS is exploited. Since I could not verify or affect first and second one is more dangerous I started investigation.
So now I assume that someone broke into VPS. It could be 'traditional' breach with gaining login or transient one without leaving traces on disk.
Inspected my services: top; netstat; lsof -i; df; docker ps -a
See logs: /var/log/syslog*, /var/log/auth*
And there are continuous attempts to brute-force ssh passwords - someone who already got in would not to do that. Miss.
Checked for extra users or installed email servers - none.
Syslog contained multiple messages like
TCP: TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters.
nf_conntrack: table full, dropping packet
net_ratelimit: 20 callbacks suppressed
then
ziproxy invoked oom-killer
So now I get a clue. Now I get something that looks like DOS but no 8080 port should be visible outside... It turns out that syslog shows port of docker container not host's.
One of slightly surprising things of docker is that it uses LXC wchich are "chroot on steroids". So any process running "in docker" is actually normal process running on host's kernel and there's no "container" process. That said, ziproxy that runs in Docker was accessed from outside what caused it to send emails. Now ziproxy is a web proxy. Was there some overflow that injected outside code or proxy was used as is?
I found no known vulnerabilities for ziproxy, so it's probably normal operation. Since complaints were for SMTP it should have been CONNECT method. Looking at ziproxy sources I found out that CONNECT is supported by ziproxy. To check this I tried:
$ telnet localhost 8888
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CONNECT my-vps-domain.net:80 HTTP/1.1
HTTP/1.0 200 Connection established
GET /favicon.ico
301 Moved Permanently
nginx/1.2.1Connection closed by foreign host.
So now I assume that someone broke into VPS. It could be 'traditional' breach with gaining login or transient one without leaving traces on disk.
Inspected my services: top; netstat; lsof -i; df; docker ps -a
See logs: /var/log/syslog*, /var/log/auth*
And there are continuous attempts to brute-force ssh passwords - someone who already got in would not to do that. Miss.
Checked for extra users or installed email servers - none.
Syslog contained multiple messages like
TCP: TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters.
nf_conntrack: table full, dropping packet
net_ratelimit: 20 callbacks suppressed
then
ziproxy invoked oom-killer
So now I get a clue. Now I get something that looks like DOS but no 8080 port should be visible outside... It turns out that syslog shows port of docker container not host's.
One of slightly surprising things of docker is that it uses LXC wchich are "chroot on steroids". So any process running "in docker" is actually normal process running on host's kernel and there's no "container" process. That said, ziproxy that runs in Docker was accessed from outside what caused it to send emails. Now ziproxy is a web proxy. Was there some overflow that injected outside code or proxy was used as is?
I found no known vulnerabilities for ziproxy, so it's probably normal operation. Since complaints were for SMTP it should have been CONNECT method. Looking at ziproxy sources I found out that CONNECT is supported by ziproxy. To check this I tried:
$ telnet localhost 8888
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CONNECT my-vps-domain.net:80 HTTP/1.1
HTTP/1.0 200 Connection established
GET /favicon.ico
301 Moved Permanently
nginx/1.2.1Connection closed by foreign host.
(lines in bold is what I typed)
Inspecting syslog I found that first "Possible SYN flooding" was exactly at the moment of spam message sent. And there are no similar events before that.
Now most likely situation was that I accidentally exposed ziproxy (it was intended to be visible from localhost only) and someone used CONNECT method to relay email sending. Far simpler than I was afraid of.
To verify this I started ziproxy again and there's "Possible SYN flooding" again.
The actual problem with my setup was that I misunderstood defaults of "-P" and "-p" Docker switches. "-P" exposes all ports even without parameters. "-p", given ports only exposes ports on all interfaces while I needed only "loopback".
I used "nmap" to find opened ports onl VPS and that helped to fix Docker settings.
And I also installed fail2boot. It's default settings are reasonable to throttle DOS and brute-forces.
