Investigating Rogue Ports
Often times a system administrator will find open ports on a machine and wonder what they are for. It is easy enough to check to see the default use of a well known port (for instance, SMTP uses the default port 25), but this is no guarantee that the actual port observed is using the default service. It is very easy to utilize non standard ports to run services. SSH uses the default port 22, but it is a simple matter of changing the configuration file to allow SSH to listen on an alternative port. Given this situation, how can a sysadmin track down the use of a port?
Several methods exist for discovering the use of an open port. One of the best tools in a sysadmin's arsenal is NMAP port scanner. NMAP usually requires elevated privileges to run, but can be installed so that regular system users can utilize it. A Windows port exists as well and even includes a graphical front end. I find the command line version easiest to use on both Windows and Linux/Unix hosts. To start with we'll explore our target (in this case our local machine) to find what ports are open. Note that when NMAP is run on the local host it operates behind any firewall filters, so it may show ports that are in fact blocked from external access. To scan the local host use:
# nmap localhost Starting Nmap 4.20 ( http://insecure.org ) at 2007-08-02 11:42 EDT Interesting ports on reims (127.0.0.1): Not shown: 1684 closed ports PORT STATE SERVICE 22/tcp open ssh 24/tcp open priv-mail 25/tcp open smtp 5901/tcp open vnc-1 6000/tcp open X11 6001/tcp open X11:1 10000/tcp open snet-sensor-mgmt Nmap finished: 1 IP address (1 host up) scanned in 0.203 seconds
You can use the netstat command in a similar way. Using the '-l' to show listening services, '-p' to show the process identification number (PID), and '-n' flag to suppress lookups. The initial output will display relevant information:
# netstat -lnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:2208 0.0.0.0:* LISTEN 2115/hpiod tcp 0 0 0.0.0.0:5801 0.0.0.0:* LISTEN 5445/Xvnc tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 20948/sendmail: acc tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 5205/smbd tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN 5445/Xvnc tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN 5167/perl tcp 0 0 0.0.0.0:6000 0.0.0.0:* LISTEN 3557/X tcp 0 0 0.0.0.0:6001 0.0.0.0:* LISTEN 5445/Xvnc tcp 0 0 192.168.0.12:53 0.0.0.0:* LISTEN 20898/named8 tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 20898/named8 tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN 3553/cupsd tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 20948/sendmail: acc tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 23157/1 tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 5205/smbd tcp 0 0 127.0.0.1:2207 0.0.0.0:* LISTEN 2222/python tcp 0 0 :::6000 :::* LISTEN 3557/X tcp 0 0 :::6001 :::* LISTEN 5445/Xvnc tcp 0 0 :::22 :::* LISTEN 30860/sshd tcp 0 0 :::631 :::* LISTEN 3553/cupsd tcp 0 0 :::24 :::* LISTEN 30860/sshd tcp 0 0 ::1:6010 :::* LISTEN 23157/1 udp 0 0 0.0.0.0:32768 0.0.0.0:* 3492/avahi-daemon: udp 0 0 192.168.0.12:137 0.0.0.0:* 5217/nmbd udp 0 0 0.0.0.0:137 0.0.0.0:* 5217/nmbd udp 0 0 192.168.0.12:138 0.0.0.0:* 5217/nmbd udp 0 0 0.0.0.0:138 0.0.0.0:* 5217/nmbd udp 0 0 0.0.0.0:10000 0.0.0.0:* 5167/perl udp 0 0 0.0.0.0:32919 0.0.0.0:* 20898/named8 udp 0 0 192.168.0.12:53 0.0.0.0:* 20898/named8 udp 0 0 127.0.0.1:53 0.0.0.0:* 20898/named8 udp 0 0 0.0.0.0:5353 0.0.0.0:* 3492/avahi-daemon: udp 0 0 :::32918 :::* 20898/named8
You can omit the '-l' flag to inspect active connections. This output can be trusted more than the original NMAP scan because here we're looking at the information the machine is reporting from within. NMAP can be used to much more intelligently scan a host, however. Using the '-sV' flag NMAP will carry on a dialogue with the open ports and attempt to identify what service is actually running on the port (rather than simply reporting the default services that are usually running on the port). Remember that the scan above reported port 24 as follows:
24/tcp open priv-mail
NMAP shows that port 24 is open and that that port is usually used for priv-mail service. Lets try the more interactive scanning and see if we can determine the service that is actually running. Using the '-sV' flag NMAP will engage in a full protocol communication, examining sequences, not just banners, to attempt to determine the service:
# nmap -sV localhost Starting Nmap 4.20 ( http://insecure.org ) at 2007-08-02 11:43 EDT Interesting ports on reims (127.0.0.1): Not shown: 1684 closed ports PORT STATE SERVICE VERSION 22/tcp open ssh OpenSSH 4.6 (protocol 2.0) 24/tcp open ssh OpenSSH 4.6 (protocol 2.0) 25/tcp open smtp Sendmail 8.14.0/8.14.0 53/tcp open domain ISC Bind 4.X 587/tcp open smtp Sendmail 8.14.0/8.14.0 5901/tcp open vnc VNC (protocol 3.7) 6000/tcp open X11 (access denied) 6001/tcp open X11 (access denied) 10000/tcp open http Webmin httpd Service Info: Host: madirish.net; OS: Unix Service detection performed. Please report any incorrect results at http://insecure.org/nmap/submit/ . Nmap finished: 1 IP address (1 host up) scanned in 88.775 seconds
You'll notice that port 24 has now been identified as running an SSH session. Our next step in determining the actual process that might be using the port is the lsof tool. Lsof stands for 'list open files'. We're going to use the '-i' flag, which lists files opened by a specific internet address. In this case we're going to use an abbreviation of the full form utilized for the address, which is 'protocol@[hostname|hostaddress]:[service|port]'. Using the following command:
# lsof -i :22
We can list the files utilized by the current process running on the local port 22. This gives results similar to:
# lsof -i :22 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 30860 root 4u IPv6 430493 TCP *:ssh (LISTEN)
You'll notice we can find the executable being used, the PID and even the user account that is running the process. The PID is especially useful because we can actually drop into the PID directory in /proc and examine what resources the process is using:
# cd /proc/30860 # ls -lah total 0 dr-xr-xr-x 4 root root 0 Aug 2 11:46 ./ dr-xr-xr-x 126 root root 0 Jul 8 16:16 ../ -r-------- 1 root root 0 Aug 2 11:46 auxv -r--r--r-- 1 root root 0 Aug 2 11:46 cmdline lrwxrwxrwx 1 root root 0 Aug 2 11:46 cwd -> // -r-------- 1 root root 0 Aug 2 11:46 environ lrwxrwxrwx 1 root root 0 Aug 2 11:46 exe -> /usr/sbin/sshd* dr-x------ 2 root root 0 Aug 2 11:46 fd/ -rw-r--r-- 1 root root 0 Aug 2 11:46 loginuid -r--r--r-- 1 root root 0 Aug 2 11:46 maps -rw------- 1 root root 0 Aug 2 11:46 mem -r--r--r-- 1 root root 0 Aug 2 11:46 mounts -r-------- 1 root root 0 Aug 2 11:46 mountstats -rw-r--r-- 1 root root 0 Aug 2 11:46 oom_adj -r--r--r-- 1 root root 0 Aug 2 11:46 oom_score lrwxrwxrwx 1 root root 0 Aug 2 11:46 root -> // -rw------- 1 root root 0 Aug 2 11:46 seccomp -r--r--r-- 1 root root 0 Aug 2 11:46 smaps -r--r--r-- 1 root root 0 Aug 2 11:46 stat -r--r--r-- 1 root root 0 Aug 2 11:46 statm -r--r--r-- 1 root root 0 Aug 2 11:46 status dr-xr-xr-x 3 root root 0 Aug 2 11:47 task/ -r--r--r-- 1 root root 0 Aug 2 11:46 wchan
As you can see it is easy to spot the executable being run as well as several supporting components of the SSH process. We can also look at environmental variables set up by the process using the following:
# cat environ LC_PAPER=en_US.UTF-8LC_ADDRESS=en_US.UTF-8LC_MONETARY=en_US.UTF-8SHELL=/bin/bashTERM=xtermLC_SOURCED=1TMPDIR=/tmpLC_NUMERIC=en_US.UTF-8LOCPATH=/etc/localeUSER=rootLC_TELEPHONE=en_US.UTF-8SUDO_USER=justinSUDO_UID=500PATH=/sbin:/usr/sbin:/bin:/usr/binLC_MESSAGES=en_US.UTF-8LC_COLLATE=en_US.UTF-8LC_IDENTIFICATION=en_US.UTF-8PWD=/home/justinLANG=en_US.UTF-8LC_MEASUREMENT=en_US.UTF-8SHLVL=1SUDO_COMMAND=/etc/rc.d/init.d/sshd restartHOME=/home/justinLANGUAGE=en_US.UTF-8:en_US:enTMP=/tmpLOGNAME=rootLC_CTYPE=en_US.UTF-8SUDO_GID=500DISPLAY=localhost:11.0LC_TIME=en_US.UTF-8TEXTDOMAINDIR=/etc/localeLC_NAME=en_US.UTF-8_=/usr/sbin/sshd
And we can check out the sockets that the process is using by listing the fd directory:
# ls -lah fd/ total 5.0K dr-x------ 2 root root 0 Aug 2 11:46 ./ dr-xr-xr-x 4 root root 0 Aug 2 11:46 ../ lrwx------ 1 root root 64 Aug 2 11:46 0 -> /dev/null lrwx------ 1 root root 64 Aug 2 11:46 1 -> /dev/null lrwx------ 1 root root 64 Aug 2 11:46 2 -> /dev/null lrwx------ 1 root root 64 Aug 2 11:46 3 -> socket:[430489] lrwx------ 1 root root 64 Aug 2 11:46 4 -> socket:[430493]
We can also look at the open files utilized by the process using the lsof command again, this time with the '-p' flag and providing the PID:
# lsof -p 30860 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME sshd 30860 root cwd DIR 3,1 4096 2 / sshd 30860 root rtd DIR 3,1 4096 2 / sshd 30860 root txt REG 3,1 356816 558088 /usr/sbin/sshd sshd 30860 root mem REG 0,0 0 [heap] (stat: No such file or directory) sshd 30860 root mem REG 3,1 34324 132543 /lib/libnss_files-2.4.so sshd 30860 root mem REG 3,1 34792 491727 /usr/lib/libkrb5support.so.0.1 sshd 30860 root mem REG 3,1 1232576 132408 /lib/i686/libc-2.4.so sshd 30860 root mem REG 3,1 6064 132507 /lib/libcom_err.so.2.1 sshd 30860 root mem REG 3,1 167752 491717 /usr/lib/libk5crypto.so.3.0 sshd 30860 root mem REG 3,1 643584 491725 /usr/lib/libkrb5.so.3.2 sshd 30860 root mem REG 3,1 208828 491564 /usr/lib/libgssapi_krb5.so.2.2 sshd 30860 root mem REG 3,1 21912 132511 /lib/libcrypt-2.4.so sshd 30860 root mem REG 3,1 71456 132537 /lib/libnsl-2.4.so sshd 30860 root mem REG 3,1 73576 132591 /lib/libz.so.1.2.3 sshd 30860 root mem REG 3,1 9704 132584 /lib/libutil-2.4.so sshd 30860 root mem REG 3,1 1439348 491566 /usr/lib/libcrypto.so.0.9.8 sshd 30860 root mem REG 3,1 63316 132569 /lib/libresolv-2.4.so sshd 30860 root mem REG 3,1 9688 132518 /lib/libdl-2.4.so sshd 30860 root mem REG 3,1 39108 132552 /lib/libpam.so.0.81.6 sshd 30860 root mem REG 3,1 31288 493192 /usr/lib/libwrap.so.0.7.6 sshd 30860 root mem REG 3,1 545447 132486 /lib/ld-2.4.so sshd 30860 root 0u CHR 1,3 971 /dev/null sshd 30860 root 1u CHR 1,3 971 /dev/null sshd 30860 root 2u CHR 1,3 971 /dev/null sshd 30860 root 3u IPv6 430489 TCP *:lmtp (LISTEN) sshd 30860 root 4u IPv6 430493 TCP *:ssh (LISTEN)
This shows the executable and all the supporting libraries that it is using. These could be important if you want to verify the validity of the process (for instance if you're hunting for a trojan). Of course, if you suspect a trojan is on the machine you may also likely suspect a rootkit that could be subverting the operations of lsof and other programs on the machine. In the case of a legitimate service though you should have most of what you need. For one further step you can use RPM to figure out what package the executable belongs to. Using the query:
# rpm -qf /usr/sbin/sshd openssh-server-4.6p1-1mdv2007.1
You can find out what package the SSH daemon belongs to, in this case openssh-server-4.6p1.
Conclusions
Tracking down a the actual usage of a port on a machine is a little more complex than simply consulting the /etc/services file. Although you can certainly check on ports using:
$ cat /etc/services | grep 22/tcp ssh 22/tcp # SSH Remote Login Protocol
This only gives you the name of the commonly used protocol on the specific port. To be sure of what processes are using an open port you need to check a little more. Using tools like NMAP you can carry out this discovery remotely. If you have local shell access using lsof and the PID will give you more concrete information about the processes using the open port. Be careful though, as malicious rootkits could be subverting your investigative programs. If you're wary of hostile activity on your machine it is much safer and wiser to boot from trusted media to investigate a filesystem. Using a live CD or bootable linux distribution is often a good option in these circumstances.