OWASP Releases DirBuster 0.11.1
Two days ago OWASP announced the release of a new version of their DirBuster tool. DirBuster is a Java based web application scanner. Basically you give it a host and it scans that host for directories on the host. DirBuster can utilize a list of directories and files or it can brute force them. DirBuster is nice because it can find files directories that might not be directly linked to. This can be used to expose information on the host that you might not find otherwise. DirBuster will also parse the HTML of files that it does discover, allowing it to follow links present in discoverable files as well. You can find more information about DirBuster at the OWASP site at https://www.owasp.org/index.php/Category:OWASP_DirBuster_Project.
I ran DirBuster on a test machine just to see what it was capable of. I set it up to brute force files and folders and let it run over night. When I came back to DirBuster the next morning it still hadn't gotten through the two letter directory names. DirBuster is multi-threaded and scary fast, but this gives you a sense of how long a blind brute force discovery would take. Also, as soon as I logged into the target I noticed something very wrong. The entire hard drive was filled to capacity. Not only was there an entry in the Apache log for each DirBuster file or directory scan, but I had OSSEC running and there were two entries in the OSSEC logs for each 404 that DirBuster was producing (OSSEC, by default, produces alerts for multiple 404 errors from the same source in a short period of time). The combination of the logs added up to gigabytes of storage! I've never seen log files get so out of control so fast. This behavior should serve as a warning to others utilizing DirBuster. While it can be great for discovery, keep in mind that it could be used as a denial of service tool if it fills up logs and crashes services.
Also interesting to note is that you can run DirBuster through a proxy. This allows you to run DirBuster through TOR's (The Onion Router) privoxy. This allows you to do anonymous scanning of a target (your scan will appear to come from different TOR host). To run DirBuster in this way first install TOR and DirBuster. Once TOR is installed you should be able to run Privoxy. Privoxy will run on port 8118 by default. Next start DirBuster, select Options -> Advanced Options, then click on the 'Http Options' tab. Check the 'Run Through a Proxy' checkbox and set the 'Host' to localhost and the 'Port' to 8118. Click on OK, then run DirBuster as normal (through TOR).
The one problem with running DirBuster through TOR is that it utilizes only one TOR host for the scan. This means that your hundreds of requests all come from the same host. This can be problematic if the target has any rules that limit connections from a single requester. For instance, an astute admin could see the brute force discovery attempts and block the TOR host that you were using as a relay. You would have to restart the scan in order to utilize a new TOR host, which could slow things down.
So far I'm unaware of any published lists that you could use to pre-populate your directory/file list. Something that included default directories for well known web applications would be nice. You can always start your own. Utilizing the brute force option seems attractive at first, but given the fact that it is so slow and noisy I would advise against it. That said, I'm very impressed with the functionality of DirBuster and would recommend it for others doing penetration tests.