Saturday, 9 October 2010

A Wireshark capture filter for HTTP 503 errors

Using Wireshark capture filters to track down specific HTTP error codes on a web server.

Wireshark is an excellent tool, and capture filters can be quite daunting, but they can also be very powerful and useful.

Recently I was helping to troubleshoot some HTTP 503 error codes which were happening on an couple of IIS servers. These systems were handling over a million requests per day between the two systems, and intermittently responding with 503 errors for a small percentage of requests.

It was difficult for us to find any reason for these errors, and unfortunately, IIS does not log 503 errors (as far as we could see) so we could not tell how many were occurring, when they were occurring, or if there were any other associated patterns. We really needed to log the 503 errors, so that we could continue to troubleshoot and perform some configuration optimization.

Just leaving Wireshark running and capturing all requests and responses for 24 hours+ was not a good option, because of the additional load, and data logged. We wanted to come up with a special Wireshark filter, to capture and record only the 503 errors over a longer period of time, so that we could do more analysis and fix the problem.

I remembered a recent course I studied which pointed to some capture filtering that could be done to find HTTP GET requests. More detail here http://wiki.wireshark.org/CaptureFilters

"Capture HTTP GET requests. This looks for the bytes 'G', 'E', 'T', and ' ' (hex values 47, 45, 54, and 20) just after the TCP header. "tcp[12:1] & 0xf0) >> 2" figures out the TCP header length. port 80 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420"
 
This is pretty clever stuff, but I figured I could extend this filter to look for 503 responses.

Capture filters are more difficult than display filters in Wireshark.
  • With display filters you can easily just right-click on a relevant node you want to filter on, and "Apply as filter".
  • Capture filters are a lot more manual to configure, and they are more time-consuming to tweak and get working correctly.
So, to capture the errors we were investigating, I wanted to look for the sequence "HTTP/1.1 503" after the TCP header, and with a bit of hacking I came up with the following capture which works great!

port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450) and (tcp[((tcp[12:1] & 0xf0) >> 2)+4:4] = 0x2f312e31) and (tcp[((tcp[12:1] & 0xf0) >> 2)+8:4] = 0x20353033)

This filter can easily be adjusted to look for other error codes such as 500, 404 etc, simply by changing the last three hex characters ("353033")
 
So for example to look for 500 error codes:
 
port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450) and (tcp[((tcp[12:1] & 0xf0) >> 2)+4:4] = 0x2f312e31) and (tcp[((tcp[12:1] & 0xf0) >> 2)+8:4] = 0x20353030)

This filter was so useful in tracking down our problem, that I thought I would share it with the web.

The lesson to be learned here, is do not fear the apparent complexity of Wireshark capture filters. Take an example, and tweak it until you get the result that you are looking for.

Let me know if you find this useful.

4 comments:

  1. Thanks for the info.
    What was the actual cause to the 503 errors?

    ReplyDelete
  2. The issue was related to throttling of connections on the site.

    We increased the number of connections and it fixed the problem, but the only way to confirm that it had been fixed was to use this filter.

    ReplyDelete
  3. can you share how to calculate the last 3 digits for other error code?

    ReplyDelete