LAMP Security Through Virtualization

Consider the typical LAMP stack. You've got your trusty PHP web application running on Apache, connecting to your MySQL database all centralized on one easy to manage machine that you can SSH to. This architecture is compact, it's convenient, and it's stable. Now, for a moment, consider this architecture from a security standpoint and your perception becomes quite different. You've got all your eggs in one basket. A compromise at almost any level of the LAMP stack can lead to compromises on any other level. If an intruder compromises the web application, they might be able to get file system access, read files, including the plain text MySQL database files, or modify logs. If one of your SSH accounts is compromised via brute force your entire server is in jeopardy - your web sites, your databases, even your logs could be tampered with.

The obvious solution to this problem is to split up the various layers of the LAMP stack to provide more insulation and defense in depth. If you have your database running on a separate machine, a compromised apache web server process can't access the MySQL database flat files from the filesystem. A compromised SSH account might still get access to the PHP files, but they can't tamper with the database any longer. Proper compartmentalization of MySQL accounts and permissions (http://LAMPsecurity.org/defensive-MySQL) means that even if an attacker can expose connection credentials, they can only carry out authorized, predefined tasks as the web application, but can only leverage a predefined set of procedures (and can't do something like drop all the tables).

In order to be thorough it would be best to split our simple LAMP stack server into four separate servers. You would have a web server, that would host the PHP files for the web application, a database server from which MySQL could be hosted, a syslog server that you could use for the collection of log files, and a bastion server that would allow SSH access. With this architecture it becomes much more difficult to completely compromise the web application. What's more, it becomes extremely difficult to compromise the servers and escape notice. Because the SSH server, the web server, and the MySQL server are all logging to a fourth server, evidence of attacks and intruder activity is likely to be stored safely away from the attacker. The attackers task has ballooned from simply compromising one host to needing to compromise four in order to take over the web application and hide evidence of the attack.

This four server model seems ideal, unless you only have one piece of hardware, which is often the case in small LAMP based applications. The solution to this dilemma is to utilize virtualization. With a free product, like VMware Server (http://www.vmware.com/products/server/), you can run four virtual machines on top of one actual piece of hardware. This setup requires more processing power and memory than running a standalone LAMP stack server, but provides enormous security advantages.

To start we need to select an appropriate hardware host. This should be a machine with CPU and RAM sufficient for running the multiple guest operating systems. The one major limitation of virtualization is that you need sufficient hardware for the combined totals of all your virtual machine demands. In other words, if you're planning on running four virtual machines with 256 MB of RAM each then you should have at least 1 GB of RAM available on the host system. Most virtualization platforms are capable of 'bursting' or dynamic allocation of RAM so that in the above example each machine wouldn't necessarily allocate all 256 MB of RAM at once. Remember that you'll need RAM for the host operating system as well.

Once you've selected a machine with sufficient CPU power and RAM available of all the virtual machines you wish to run you're ready to install the host operating system. The security of the host is critical, as any compromise of the host means a compromise of all the guest virtual machines. In many circumstances you'll want to install the host operating system with the minimum services necessary to support the virtualization engine (such as VMware Server). You may even elect to prohibit all remote network connections, requiring any interaction with the host to take place via an actual console. This enhances the security of the host by making it inaccessible to remote attackers.

Another factor worth considering is the networking of virtual machines. Having more than one network adapter available helps to segregate network traffic for each virtual machines. In many cases it is impractical to have a unique network adapter for each virtual machine, but it's worth considering having at least two network adapters on the hardware host and using one for the host, and reserving one adapter to be shared for all the virtual machines. This enables you to have network services offered from the host, but segregated from the guests. In some circumstances virtual machines who are able to put their network adapters into promiscuous mode are able to sniff the traffic of all other machines using the same physical adapter.

Disk I/O is also a limitation of a virtualized infrastructure. Consider that if each virtual machine is housed on the same hard disk then reads and writes from the file system may be slowed as the host OS struggles to handle the multiple read and write requests from various areas of the disk. At the very least having the virtual machines on a separate hard disk from the host OS could provide performance gains. Having a unique hard disk for each virtual machine would probably be ideal but in all likelihood is impractical. At a minimum employ hard disks that can provide large amounts of data throughput and maximum responsiveness.

Once you've decided on your hardware configuration and the virtual machine topology you want to deploy it's time to build your virtual machine host. To do this you'll need to download VMware Server from the VMware site. You can download the rpms, but regardless of the installation package you're likely to have to build the kernel modules during installation. This means having your kernel-header or kernel-devel packages installed along with GCC and other compilation supporting packages. Once VMware Server is installed you can configure it with /usr/bin/VMware-config.pl.

After VMware Server is installed and configured you can begin to manage your virtual machines through the Server web interface. This runs on port 8222 or over SSL on port 8333. One easy way to manage this configuration without having to expose either of these ports to remote connection is to tunnel a connection to the VMware Server host over SSH.

For a VMware Server host you can build the guest machines in almost any VMware product, including VMware Workstation, Player, or even Server itself. I find that it's easiest to develop the virtual machines in VMware Workstation, then rsync them to the host over SSH. VMware Workstation is much faster and more responsive than building virtual machines via the Server console. One caveat I have noticed is that when building virtual machines it's best to create one monolithic hard drive binary file rather than splitting it up into 2GB parts. Once you've build the machine simply transfer the entire virtual machine folder to the server host in /var/lib/vmware/Virtual Machines.

Once you've build your virtual machines you can add them to your Server deployment by clicking on the 'Virtual Machines' tab in the Server console, then selecting 'Add Virtual Machine to Inventory' and browsing to the VMX file for the machine you wish to add. Once the machine is added to the inventory you can start it up by selecting the machine in the 'Inventory' pane on the left and clicking the green 'Play' button at the top of the console.

Once all of your virtual machines are started they'll operate just like real machines. One really nice feature of VMware Server is that you can snapshot running machines. This allows you to take images of virtual machine states so that if anything adverse happens to the virtual machine you can roll back to a previous snapshot with the 'Revert to Snapshot' option in the 'Commands' window when you're viewing a specific virtual machine summary in the console.

Once we've built our web server, our database server, our bastion (SSH server) and our syslog server you need to configure each. Each virtual machine should have unique root and user account passwords (using the same username/password combination on multiple machines weakens the overall security stance of the virtualized environment). Each machine should be firewalled to limit connections from authorized hosts.

The only incoming connection that should be allowed to pass through the firewall from the internet are SSH connections that are forwarded to the bastion host and web requests which are forwarded to the web server. You should be able to SSH around the internal machines from one another, but one must first access the bastion host. The web server should be allowed to issue MySQL requests over port 3306 to the MySQL server. The syslog server should accept syslog connections from all the servers. This allows for effective defence in depth and effectively compartmentalizes each component of our application.

It's illustrative to consider a classic attack vector in the context of a single host LAMP stack and compare it with an analysis of the same attack in our new virtualized environment. For this exercise we'll take a case where an attacker manages to find and exploit a SQL injection vulnerability in our web application that happens to be using the MySQL root account.

In the case of a classic LAMP stack environment the attacker can utilize the 'INTO OUTFILE' function built into MySQL to write to any MySQL writable directories. Assuming the attacker can write to a web accessible directory they can simply write a PHP based command shell backdoor. Once this malicious PHP command interpreter script is written they can visit that page, and use their web interface to take over the Apache web server account. Once the web server account is exposed the attacker can read configuration information stored in web accessible files, crack passwords in .htpasswd files, and take other steps to crack user accounts and escalate privilege. The user might be able to recover usernames and passwords that allow them SSH access or use the Apache web account to exploit local privilege escalation vulnerabilities and gain root privileges.

In our virtualized environment the attacker can compromise the MySQL server and write files using the 'INTO OUTFILE' function, but can't write to any web accessible directories as the web server exists on a separate machine. What's more, even if the attacker manages to reveal files using the 'INFILE' MySQL function the accounts are local to the database server only and wouldn't allow the attacker SSH access to the virtualized environment as there is no SSH access allowed to the database server except from within the virtualized environment. The attacker is stymied because even if they can gain access to one level of the LAMP stack they can't leverage it into access to other layers. Additionally all the evidence of their attacks is being aggregated on the remote syslog server, so they can't cover their tracks.

By deploying a virtualized network of machines responsible for each layer of the LAMP stack we have exponentially increased the overall security stance of our web application. This strategy may be a bit costly in terms of setup and maintenance but the security gains are huge. Additionally, by virtualizing servers they become easier to backup, restore, and even migrate to other hardware platforms. Because virtual machines are simply binary files on the host filesystem they can easily be ported to other machines or even burned to CD if they are small enough (or tape if they are larger). The ability to back up and restore entire machines in minutes helps to recoup the cost of having to maintain multiple hosts. Additionally, the centralized logging provided by the syslog server helps to ease the task of monitoring the various virtual machines in the network.