LKM in my server. Rootkitted.

 From: root@example.com (Cron Daemon)
 Subject: Cron test -x /usr/sbin/anacron || ( cd / && run-parts
 --report /etc/cron.daily )
 To: root@example.com
 Date: Sun, 27 Jan 2008 09:25:10 -0500

 You have 1 process hidden for readdir command
 You have 1 process hidden for ps command
 chkproc: Warning: Possible LKM Trojan installed

When I received that email from one of my public servers, two
questions popped up: did I just get rootkited again?

I had been rootkitted about three years ago. At that time the only
network-accessible services on the computer was a stock apache 1.3
serving static pages and an unpatched qmail. I didn't have tripwire
setup and ended up reinstalling the system from scratch because I
didn't know which files had been compromised.

This time, I had tripwire (AIDE, actually) setup and was able to
identify the infected files. Yet I still ended up reinstalling the
system from scratch partly because things were too easy for me and
mostly because I just found the excuse I needed.

Firstly, the compromiser didn't take out the regular chkrootkit
report. Secondly, subsequent chkrootkit report didn't report any
warning. Did the chkrootkit itself was compromised? Later verification
with an uncompromised AIDE data file showed that it was not, but
still, you wouldn't know that then. Thirdly, the same question for the
AIDE binary itself. Fourthly, it was really painful trying to do an
off-line verification on a remote system. Fifthly, was it apache 1.3
again?

One thing I discovered was that Linux allows you to disable loadable
kernel module (LKM) which supposedly makes it impervious to LKM
Trojan, like the one I had. No use crying over spilt milk.

The server was a Xen virtual machine hosted by quantact.com. It was
setup two years ago. This break-in gave me an excuse to reorganize my
hosting choices. Around last year, they started offering a cheaper
hosting option on OpenVZ. So I tried it and found that I liked it. I
liked it so much I installed one at home.

OpenVZ does not allow virtual machines to load kernel modules. I hope
that is enough to circumvent the same attack vector.