Investigating Rogue Ports

30 November -0001

Often times a system administrator will find open ports on a machine and wonder what they are for. It is easy enough to check to see the default use of a well known port (for instance, SMTP uses the default port 25), but this is no guarantee that the actual port observed is using the default service. It is very easy to utilize non standard ports to run services. SSH uses the default port 22, but it is a simple matter of changing the configuration file to allow SSH to listen on an alternative port. Given this situation, how can a sysadmin track down the use of a port?

Several methods exist for discovering the use of an open port. One of the best tools in a sysadmin's arsenal is NMAP port scanner. NMAP usually requires elevated privileges to run, but can be installed so that regular system users can utilize it. A Windows port exists as well and even includes a graphical front end. I find the command line version easiest to use on both Windows and Linux/Unix hosts. To start with we'll explore our target (in this case our local machine) to find what ports are open. Note that when NMAP is run on the local host it operates behind any firewall filters, so it may show ports that are in fact blocked from external access. To scan the local host use:

# nmap localhost

Starting Nmap 4.20 ( http://insecure.org ) at 2007-08-02 11:42 EDT
Interesting ports on reims (127.0.0.1):
Not shown: 1684 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
24/tcp    open  priv-mail
25/tcp    open  smtp
5901/tcp  open  vnc-1
6000/tcp  open  X11
6001/tcp  open  X11:1
10000/tcp open  snet-sensor-mgmt

Nmap finished: 1 IP address (1 host up) scanned in 0.203 seconds

You can use the netstat command in a similar way. Using the '-l' to show listening services, '-p' to show the process identification number (PID), and '-n' flag to suppress lookups. The initial output will display relevant information:

# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 127.0.0.1:2208              0.0.0.0:*                   LISTEN      2115/hpiod
tcp        0      0 0.0.0.0:5801                0.0.0.0:*                   LISTEN      5445/Xvnc
tcp        0      0 0.0.0.0:587                 0.0.0.0:*                   LISTEN      20948/sendmail: acc
tcp        0      0 0.0.0.0:139                 0.0.0.0:*                   LISTEN      5205/smbd
tcp        0      0 0.0.0.0:5901                0.0.0.0:*                   LISTEN      5445/Xvnc
tcp        0      0 0.0.0.0:10000               0.0.0.0:*                   LISTEN      5167/perl
tcp        0      0 0.0.0.0:6000                0.0.0.0:*                   LISTEN      3557/X
tcp        0      0 0.0.0.0:6001                0.0.0.0:*                   LISTEN      5445/Xvnc
tcp        0      0 192.168.0.12:53             0.0.0.0:*                   LISTEN      20898/named8
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      20898/named8
tcp        0      0 0.0.0.0:631                 0.0.0.0:*                   LISTEN      3553/cupsd
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      20948/sendmail: acc
tcp        0      0 127.0.0.1:6010              0.0.0.0:*                   LISTEN      23157/1
tcp        0      0 0.0.0.0:445                 0.0.0.0:*                   LISTEN      5205/smbd
tcp        0      0 127.0.0.1:2207              0.0.0.0:*                   LISTEN      2222/python
tcp        0      0 :::6000                     :::*                        LISTEN      3557/X
tcp        0      0 :::6001                     :::*                        LISTEN      5445/Xvnc
tcp        0      0 :::22                       :::*                        LISTEN      30860/sshd
tcp        0      0 :::631                      :::*                        LISTEN      3553/cupsd
tcp        0      0 :::24                       :::*                        LISTEN      30860/sshd
tcp        0      0 ::1:6010                    :::*                        LISTEN      23157/1
udp        0      0 0.0.0.0:32768               0.0.0.0:*                               3492/avahi-daemon:
udp        0      0 192.168.0.12:137            0.0.0.0:*                               5217/nmbd
udp        0      0 0.0.0.0:137                 0.0.0.0:*                               5217/nmbd
udp        0      0 192.168.0.12:138            0.0.0.0:*                               5217/nmbd
udp        0      0 0.0.0.0:138                 0.0.0.0:*                               5217/nmbd
udp        0      0 0.0.0.0:10000               0.0.0.0:*                               5167/perl
udp        0      0 0.0.0.0:32919               0.0.0.0:*                               20898/named8
udp        0      0 192.168.0.12:53             0.0.0.0:*                               20898/named8
udp        0      0 127.0.0.1:53                0.0.0.0:*                               20898/named8
udp        0      0 0.0.0.0:5353                0.0.0.0:*                               3492/avahi-daemon:
udp        0      0 :::32918                    :::*                                    20898/named8

You can omit the '-l' flag to inspect active connections. This output can be trusted more than the original NMAP scan because here we're looking at the information the machine is reporting from within. NMAP can be used to much more intelligently scan a host, however. Using the '-sV' flag NMAP will carry on a dialogue with the open ports and attempt to identify what service is actually running on the port (rather than simply reporting the default services that are usually running on the port). Remember that the scan above reported port 24 as follows:

24/tcp    open  priv-mail

NMAP shows that port 24 is open and that that port is usually used for priv-mail service. Lets try the more interactive scanning and see if we can determine the service that is actually running. Using the '-sV' flag NMAP will engage in a full protocol communication, examining sequences, not just banners, to attempt to determine the service:

# nmap -sV localhost

Starting Nmap 4.20 ( http://insecure.org ) at 2007-08-02 11:43 EDT
Interesting ports on reims (127.0.0.1):
Not shown: 1684 closed ports
PORT      STATE SERVICE     VERSION
22/tcp    open  ssh         OpenSSH 4.6 (protocol 2.0)
24/tcp    open  ssh         OpenSSH 4.6 (protocol 2.0)
25/tcp    open  smtp        Sendmail 8.14.0/8.14.0
53/tcp    open  domain      ISC Bind 4.X
587/tcp   open  smtp        Sendmail 8.14.0/8.14.0
5901/tcp  open  vnc         VNC (protocol 3.7)
6000/tcp  open  X11          (access denied)
6001/tcp  open  X11          (access denied)
10000/tcp open  http        Webmin httpd

Service Info: Host: madirish.net; OS: Unix

Service detection performed. Please report any incorrect results at http://insecure.org/nmap/submit/ .
Nmap finished: 1 IP address (1 host up) scanned in 88.775 seconds

You'll notice that port 24 has now been identified as running an SSH session. Our next step in determining the actual process that might be using the port is the lsof tool. Lsof stands for 'list open files'. We're going to use the '-i' flag, which lists files opened by a specific internet address. In this case we're going to use an abbreviation of the full form utilized for the address, which is 'protocol@[hostname|hostaddress]:[service|port]'. Using the following command:

# lsof -i :22

We can list the files utilized by the current process running on the local port 22. This gives results similar to:

# lsof -i :22
COMMAND   PID USER   FD   TYPE DEVICE SIZE NODE NAME
sshd    30860 root    4u  IPv6 430493       TCP *:ssh (LISTEN)

You'll notice we can find the executable being used, the PID and even the user account that is running the process. The PID is especially useful because we can actually drop into the PID directory in /proc and examine what resources the process is using:

# cd /proc/30860	
# ls -lah
total 0
dr-xr-xr-x   4 root root 0 Aug  2 11:46 ./
dr-xr-xr-x 126 root root 0 Jul  8 16:16 ../
-r--------   1 root root 0 Aug  2 11:46 auxv
-r--r--r--   1 root root 0 Aug  2 11:46 cmdline
lrwxrwxrwx   1 root root 0 Aug  2 11:46 cwd -> //
-r--------   1 root root 0 Aug  2 11:46 environ
lrwxrwxrwx   1 root root 0 Aug  2 11:46 exe -> /usr/sbin/sshd*
dr-x------   2 root root 0 Aug  2 11:46 fd/
-rw-r--r--   1 root root 0 Aug  2 11:46 loginuid
-r--r--r--   1 root root 0 Aug  2 11:46 maps
-rw-------   1 root root 0 Aug  2 11:46 mem
-r--r--r--   1 root root 0 Aug  2 11:46 mounts
-r--------   1 root root 0 Aug  2 11:46 mountstats
-rw-r--r--   1 root root 0 Aug  2 11:46 oom_adj
-r--r--r--   1 root root 0 Aug  2 11:46 oom_score
lrwxrwxrwx   1 root root 0 Aug  2 11:46 root -> //
-rw-------   1 root root 0 Aug  2 11:46 seccomp
-r--r--r--   1 root root 0 Aug  2 11:46 smaps
-r--r--r--   1 root root 0 Aug  2 11:46 stat
-r--r--r--   1 root root 0 Aug  2 11:46 statm
-r--r--r--   1 root root 0 Aug  2 11:46 status
dr-xr-xr-x   3 root root 0 Aug  2 11:47 task/
-r--r--r--   1 root root 0 Aug  2 11:46 wchan

As you can see it is easy to spot the executable being run as well as several supporting components of the SSH process. We can also look at environmental variables set up by the process using the following:

# cat environ
LC_PAPER=en_US.UTF-8LC_ADDRESS=en_US.UTF-8LC_MONETARY=en_US.UTF-8SHELL=/bin/bashTERM=xtermLC_SOURCED=1TMPDIR=/tmpLC_NUMERIC=en_US.UTF-8LOCPATH=/etc/localeUSER=rootLC_TELEPHONE=en_US.UTF-8SUDO_USER=justinSUDO_UID=500PATH=/sbin:/usr/sbin:/bin:/usr/binLC_MESSAGES=en_US.UTF-8LC_COLLATE=en_US.UTF-8LC_IDENTIFICATION=en_US.UTF-8PWD=/home/justinLANG=en_US.UTF-8LC_MEASUREMENT=en_US.UTF-8SHLVL=1SUDO_COMMAND=/etc/rc.d/init.d/sshd restartHOME=/home/justinLANGUAGE=en_US.UTF-8:en_US:enTMP=/tmpLOGNAME=rootLC_CTYPE=en_US.UTF-8SUDO_GID=500DISPLAY=localhost:11.0LC_TIME=en_US.UTF-8TEXTDOMAINDIR=/etc/localeLC_NAME=en_US.UTF-8_=/usr/sbin/sshd

And we can check out the sockets that the process is using by listing the fd directory:

# ls -lah fd/
total 5.0K
dr-x------ 2 root root  0 Aug  2 11:46 ./
dr-xr-xr-x 4 root root  0 Aug  2 11:46 ../
lrwx------ 1 root root 64 Aug  2 11:46 0 -> /dev/null
lrwx------ 1 root root 64 Aug  2 11:46 1 -> /dev/null
lrwx------ 1 root root 64 Aug  2 11:46 2 -> /dev/null
lrwx------ 1 root root 64 Aug  2 11:46 3 -> socket:[430489]
lrwx------ 1 root root 64 Aug  2 11:46 4 -> socket:[430493]

We can also look at the open files utilized by the process using the lsof command again, this time with the '-p' flag and providing the PID:

# lsof -p 30860
COMMAND   PID USER   FD   TYPE DEVICE    SIZE   NODE NAME
sshd    30860 root  cwd    DIR    3,1    4096      2 /
sshd    30860 root  rtd    DIR    3,1    4096      2 /
sshd    30860 root  txt    REG    3,1  356816 558088 /usr/sbin/sshd
sshd    30860 root  mem    REG    0,0              0 [heap] (stat: No such file or directory)
sshd    30860 root  mem    REG    3,1   34324 132543 /lib/libnss_files-2.4.so
sshd    30860 root  mem    REG    3,1   34792 491727 /usr/lib/libkrb5support.so.0.1
sshd    30860 root  mem    REG    3,1 1232576 132408 /lib/i686/libc-2.4.so
sshd    30860 root  mem    REG    3,1    6064 132507 /lib/libcom_err.so.2.1
sshd    30860 root  mem    REG    3,1  167752 491717 /usr/lib/libk5crypto.so.3.0
sshd    30860 root  mem    REG    3,1  643584 491725 /usr/lib/libkrb5.so.3.2
sshd    30860 root  mem    REG    3,1  208828 491564 /usr/lib/libgssapi_krb5.so.2.2
sshd    30860 root  mem    REG    3,1   21912 132511 /lib/libcrypt-2.4.so
sshd    30860 root  mem    REG    3,1   71456 132537 /lib/libnsl-2.4.so
sshd    30860 root  mem    REG    3,1   73576 132591 /lib/libz.so.1.2.3
sshd    30860 root  mem    REG    3,1    9704 132584 /lib/libutil-2.4.so
sshd    30860 root  mem    REG    3,1 1439348 491566 /usr/lib/libcrypto.so.0.9.8
sshd    30860 root  mem    REG    3,1   63316 132569 /lib/libresolv-2.4.so
sshd    30860 root  mem    REG    3,1    9688 132518 /lib/libdl-2.4.so
sshd    30860 root  mem    REG    3,1   39108 132552 /lib/libpam.so.0.81.6
sshd    30860 root  mem    REG    3,1   31288 493192 /usr/lib/libwrap.so.0.7.6
sshd    30860 root  mem    REG    3,1  545447 132486 /lib/ld-2.4.so
sshd    30860 root    0u   CHR    1,3            971 /dev/null
sshd    30860 root    1u   CHR    1,3            971 /dev/null
sshd    30860 root    2u   CHR    1,3            971 /dev/null
sshd    30860 root    3u  IPv6 430489            TCP *:lmtp (LISTEN)
sshd    30860 root    4u  IPv6 430493            TCP *:ssh (LISTEN)

This shows the executable and all the supporting libraries that it is using. These could be important if you want to verify the validity of the process (for instance if you're hunting for a trojan). Of course, if you suspect a trojan is on the machine you may also likely suspect a rootkit that could be subverting the operations of lsof and other programs on the machine. In the case of a legitimate service though you should have most of what you need. For one further step you can use RPM to figure out what package the executable belongs to. Using the query:

# rpm -qf /usr/sbin/sshd
openssh-server-4.6p1-1mdv2007.1

You can find out what package the SSH daemon belongs to, in this case openssh-server-4.6p1.

Conclusions

Tracking down a the actual usage of a port on a machine is a little more complex than simply consulting the /etc/services file. Although you can certainly check on ports using:

$ cat /etc/services | grep 22/tcp
ssh             22/tcp                          # SSH Remote Login Protocol

This only gives you the name of the commonly used protocol on the specific port. To be sure of what processes are using an open port you need to check a little more. Using tools like NMAP you can carry out this discovery remotely. If you have local shell access using lsof and the PID will give you more concrete information about the processes using the open port. Be careful though, as malicious rootkits could be subverting your investigative programs. If you're wary of hostile activity on your machine it is much safer and wiser to boot from trusted media to investigate a filesystem. Using a live CD or bootable linux distribution is often a good option in these circumstances.