I've been reading about botHunter, which is a recently announced free bot net detection utility. botHunter is a new system designed by researchers at the Georgia Institute of Technology and the Computer Science Laboratory of SRI International. It is an interesting approach to detecting bot infection in local networks. Designed to be deployed at the perimeter of a network, botHunter looks for patterns in dialogues between computers in search of well known sequences that indicate bot activity. Whereas typical bot detection is carried out by virus/worm detection tools like host based virus scanners and network intrusion detection tools using signature analysis, botHunter uses an analysis distributed over time rather than packets or files. The main advantage of botHunter, as I see it, is that it can extremely accurately identify hosts that conform to it's predefined behavior patterns.
botHunter looks at series of communications, specifically for initial infection payload delivery, download of 'egg' code (the malicious body of the infection), contact to an IRC command and control server for instructions, and the infected bot searching for other hosts to infect. Because this behavior is so strictly defined, it is unlikely to get many false positives. By the same token, there are probably a lot of bot infections that will escape notice due to non-conformity to this specific model.
botHunter also looks like it might be fairly resource intensive. Because the system searches for patterns in network activity over time it must store fairly voluminous amounts of information. It might also be possible to use encryption to evade botHunter in much the same way that encryption can be used to evade virus detection. Since encrypted payloads appear almost random, there isn't any way to distinguish signatures in packets.
While I feel that botHunter has a lot of potential in terms of developing new forms of intrusion tracking and detection, I also think it serves as another notch in the ever increasing arms race between virus writers and security professionals. By exposing 'expected' infection models researchers effectively encourage virus writers to modify their infection routines to evade detection. By subtly altering infection and propagation schemes, virus writers can remove their products from the models that allow detection.
I'm very interested to see what becomes of botHunter. The code is being release open source right now and professionals are encouraged to download, install, and comment on it. For more information about botHunter check out their website at http://www.cyber-ta.org/BotHunter/.