Kali Linux comes with a number of web path brute force utilities and when using these tools, you will find that one will work better over another when pointing at Server A versus Server B.  That could be any number of reasons including defense mechanisms which is why I’d suggest changing the user agent -- something I wrote about for Nikto.

These tools are pretty simple as long as you have the correct syntax.  That is -- until they don’t work which happens.  In those moments, you start bouncing around between this tool, that tool, and another tool expecting a better outcome.  In pentesting, there are a lot of tools and techniques to learn and the web brute force utilities are simple enough that we don’t spend time figuring out what they do behind the scenes.  That said, if you take a moment and look at it from the server side, you might see why the scan is failing. 

Dirbuster, for example, is a great tool because it really dumbs down further something that is already simple.  I don’t even need to remember syntax because of it’s GUI, I just point and shoot.  It’s easy but there are two problems from the default setup.  First, the pesky user agent: 

DirBuster-1.0-RC1 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)

Which we’re going to change to:

Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36

For the second problem, we need to catch traffic from an actual web browser and then from DirBuster to see the difference. 

This is a connection from Firefox trying to access a page which does not exist:

Connection from: ('192.168.100.100', 36500)

GET /login.php HTTP/1.1
Host: 192.168.100.101:443
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1

This is a connection from DirBuster trying to access a page which does not exist:

Connection from: ('192.168.100.100', 37579)

HEAD /$6482.php HTTP/1.1
User-Agent: DirBuster-1.0-RC1 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)
Host: 192.168.100.101:443

Aside from the user agent, which we’ll change and won’t make a difference in this instance, the default setting for DirBuster is a “HEAD” request.  The difference between a HEAD request and a GET request is that in the former, we are not requesting the message body.  In terms of speed, the HEAD request is probably faster but what browser is making a request but does not want to see the message body?  I would venture a guess and say none.  In terms of detection, if you’re seeing a bunch of HEAD requests, you might filter those out.  This is a true story. 

If we go back to the main page of DirBuster when you start it, at the top, just under the Target URL, you’ll see “Work Method”.  By default, it’s set for “Auto Switch (HEAD and GET).  Change this setting to “use GET requests only”.  And since we’re trying to look like a browser, under the Options menu, select “Advanced Options”, select the “Http Options” tab, and change the user agent:

Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36

This is not foolproof but these two changes have pushed the ball further down the road while enumerating web servers.