I’m sure everyone who has a server that has any sort of remote user accessing it has heard of this 0-day exploit (not so zero-day now, is it?) — Sadly, I belonged to those who heard of it after a few servers kernel panicked, I had no clue on what was the cause but I was going through the logs and noticed that LFD (props to ConfigServer for that useful software) had reported a suspicious process, decided to look what exactly it was, there was a x.c file, a simple cat show’s that it was copied from milw0rm, visited the related milw0rm page and I had discovered that I’m screwed. Thankfully, all of our servers ran CentOS 5 so all that happened was a kernel panic instead of being rooted.
After reading a bit about it, I prepared for a kernel compile, I had read about grsecurity and how it’s very commonly used so I decided to add it to my kernel plus the patch for the vmsplice exploit, I have setup a link to it here, sorry 64-bit folks, I only did it for i386/i686, even if our hardware is all 64-bit hardware, but I won’t start ranting about the software we use and how terrible the 64-bit support is. I have uploaded the kernel with grsecurity added and the patch for vmsplice here: linux-2.6.22.9-grsec-i686.tar.gz
I must admit, all of our servers are pretty fast and powerful, I mean when your lowest spec. server has the following specifications: 2x Intel Dual-Core 2.33Ghz Woodcrest CPUs, 4GB of RAM, 2TB total storage. Even with these specifications, kernel compiles took around 15-20 minutes, that may be fast compared to other servers however thankfully we have fast servers as I had to compile, twice, because when Linux means 4GB memory limit, it really means 2.5GB, I had to re-set the limit to 64GB for memory so the server can see all the memory.
Now, the instructions are pretty complicated at the beginning but at the end they are very straight forward, we’ll download the kernel first, you’ll either use the .config file that I had used or use your current kernel’s .config file, then a few make commands, I’ll also show how you can setup grub to fallback to your stable kernel if it actually crashes or something doesn’t work out.
mkdir /usr/src/linux
cd /usr/src/linux
wget http://files.momotonic.com/linux-2.6.22.9-grsec-i686.tar.gz
tar -xvzf linux-2.6.22.9-grsec-i686.tar.gz
The extraction should take a good few minutes as there are a lot of files to make all of these 1’s and 0’s. After that we have extracted the files, those who would want to use my configuration file can just stick with it but if you have more than 2-3GB of RAM, I’d suggest typing make menuconfig (when you are in the linux-2.6.22.9 directory), use your arrow keys to go down to “Processor type and features —>” and press Enter, then go to “High Memory Support (4GB) —>” and press Enter, select “( ) 64GB ” and press Enter, then use your right arrow to move in the bottom menu to exit, do that twice till it asks you to save the new kernel configuration, pick Yes (duh!) — If you just want to keep using your same configuration, then just copy it with this command: cp /boot/config-`uname -r` . (ofcourse, run that command in the linux-2.6.22.9 directory. You can modify a few other options and then we go to the longest process, compiling!
make
make modules_install
make install
Now, I will only cover instructions on how to do the fault-free reboot so if it does not work out, it will automatically be switched to the old kernel and everything will be like before. However, I have never worked with lilo and I’m not sure if you can do this so I will only cover instructions for doing it on grub. First, we’ll modify our /boot/grub/grub.conf file and change any of your options to the following ones
default=saved
fallback=1
Then we’ll need to modify the title for the CentOS boot and add the savedefault fallback so it should be something like the following:
title CentOS (2.6.22.9-grsec)
root (hd0,0)
kernel /boot/vmlinuz-2.6.22.9-grsec ro root=LABEL=/
initrd /boot/initrd-2.6.22.9-grsec.img
savedefault fallback
Now, we’ll first set grub to manually see that the main or default OS to run as the main operating system, in this setup, if the system does not boot up in a reasonable time, you can call your data center or use whatever method to reboot the server and it’ll boot to your old previous kernel that works fine, then you can take a look at your logs (mostly /var/log/messages) for the reason why it didn’t boot. We will need to run this command before rebooting:
echo "savedefault --default=0 --once" | grub --batch
Now you can safely reboot your server, I won’t even mention how to do that because if you do not know how to, I don’t think you should really be messing with kernels! Once you reboot, you can try re-running the exploit from milw0rm but I won’t mention on how to do that as evil minds will go and start using it.
Until then, I’m back to recompiling the kernels with 64GB memory limit because 4GB limit is actually 2.5GB!
0 Responses to “vmsplice, belated.”