Friday, October 15, 2010

On Network Debugging

I've been working on a project that involves essentially screen scraping. Basically, I perform an http get, parse the response, form a request, and then perform an http post (launder, rinse, repeat). It's not pretty, but this particular site doesn't offer an API and this was their suggestion.

As I work through this, I spend a lot of time looking at the requests and responses going over the wire, trying to figure out what parameter I didn't set properly, etc. To do this debugging, I had my choice of network monitoring tools. The ones I use most are:
  • Wireshark
  • tcpmon

Wireshark is a network packet tracing tool. It runs as a wrapper around your network driver and picks up all the traffic. This is great when you are trying to figure out basic connectivity, any sort of network congestion, or the like.

tcpmon is a tool specifically for monitoring TCP traffic. It lets you see the requests and responses, and even lets you modify a request and resend it.

Choosing a network monitoring tool depends on where you think your problem is. The OSI Model for networks has seven layers, and you should aim your tool at the layer(s) where you think you have a problem. Think about the kinds of problems that you might see (bottom to top):
  • Physical Layer
  • Data Link Layer
  • Network Layer - Wireshark shows here up
  • Transport Layer - Tcpmon shows here up
  • Session Layer
  • Presentation Layer
  • Application Layer

For this problem, I only cared about the transport layer and up - what was in my tcp request and what the app did with it. So I was able to use tcpmon rather than Wireshark. Neither is better than the other, but tcpmon showed me what I was looking for without the extraneous information Wireshark offered.

I tend to choose tools that are as high level as possible while still showing me the error I'm seeking. It's not a perfect rule, but as a general rule of thumb, it works pretty well.


  1. I've been using Wireshark on and off for 6 or 7 years now, since back when it was called Ethereal. It's a handy tool.

    It's never been part of my job description, but I like hacking on TCP/IP networks, and I've learned a lot about the protocols and the security issues inherent in such networks.

    A while back I wrote a privately-published white paper called "Network Security: Analyze Your Hosts and Ports with nmap, Nessus, and netcat". I like it, I think it's pretty good. Let me know and I'll send you a copy.

  2. Reading the beginning of the article, I thought maybe Fiddler would also fit.

    It shows you the HTTP queries/responses, decrypts the HTTPS ones, even can generate queries.

    It came handy recently, debugging a web app deployed with HTTPS, where retors didn't happen in other environments.