Wireshark Reviews

The most daunting problem to troubleshoot is when the application spits out a generic error that could mean anything. Here’s the analogy; how helpful is the ‘Check Engine’ light on your car dashboard.

The worst part is when the customer tries to take the cryptic, generic application error message and tries to make sense of it in an attempt to assist the analyst. Don’t get me wrong, any information is helpful while troubleshooting, but you have to be selective in what you pursue.

In this example FTP works one moment and fails the next. Of course the customer immediately called the help desk, who pings the ftp server and comments that is up and no outages have been recorded by the network management system. Then the ticket goes to the server dept who ftp’s without an issue, unfortunately by now so can the customer. The server department says the connection error must be a ‘network thing’.

I captured some packets and have recreated what I found and how the application, Chrome in this example, failed to pass on the FTP server connection limit error. The only way I was able to get real meaningful data is from the wire.

This isn’t a Chrome ‘bash’ session since I have seen many applications not report what was on the wire or reinterpret what was reported by the server.
In summary, the ftp server ran out of connections or had a limit on the number of connections an IP address could have. The administrator was told about this and the FTP server configuration was adjusted to allow more connections.

http://www.youtube.com/watch?v=zJoeYugcvTA

It always gives me sense of satisfaction when I have a challenge and can leverage some knowledge to figure out.

Today I was in the lab and was powering on two Cisco switches when I noticed that they weren’t labeled with their IP addresses. I’m not sure why I did not label them, but now I have to pay for it.

For those of you who have not been in this situation before I will explain. My switches have a DB9 serial connection and of course good luck finding a computer with a serial port. So now I have to rummage through the box of wires to find the serial to USB adapter. I have had to buy a second one in 2 years since my original does not have a Windows 7 driver, but I digress. After I find the cable, I have to find the installation disk because last week I migrated to a new laptop…. I’m sure you get the picture.

On to plan B. I know the switches have IP addresses since I hard code IP addresses on all of my switches.

Now here’s where a bit of knowledge comes in. I know that when a device powers up and either obtains an IP addresses via DHCP/BOOTP or statically has an IP assigned it will send out a specific ARP called a gratuitous ARP.

Perfect, now all I have to do is make sure the switch port is connected to my subnet, start any protocol analyzer (I chose Wireshark) and power up the switches.

In this video I show you how to find the Gratuitous ARP quickly, create a display filter and lastly, locate the 2 switches’ IP addresses.

Enjoy
http://www.youtube.com/watch?v=EUmHdVeBBNc

A customer called me and wanted some help troubleshooting some wireless problems. Their users have been reporting intermittent wireless performance issues and getting ‘dropped’. To top it all off, their WLAN controller has also been reporting ‘containment’ error messages that weren’t to descriptive or helpful.

I showed up on site and did all the basic RF checks with my AirMagnet Spectrum XT to make sure there wasn’t an RF issue like an interferer or channel planning issues. Like I always say, “Start at Layer 1”.

Then I moved up a layer using my Fluke Networks AirCheck and AirMagnet WiFi Analyzer. Everything looked pretty quiet and nothing jumped up at me, so I saved some trace files to review later.

Then I thought I would take the trace file and open it with Wireshark since I have more experience with packet analysis than I do using the AirMagnet/AirCheck tools.

In this video I show you some of the filters I used, what they mean and what I found.

I always enjoy getting to the packet level since packets don’t lie, but would also like to spend more time with the other tools now that I know what issues are to see how, or what, they report.

In closing there are a few points I want to make sure aren’t lost throughout the video;

1. Just because I used Wireshark to find some clues does not mean that the other tools were less effective, I just have more experience with protocol analysis/Wireshark.

2. If you deploy any kind of wireless intrusion system, make sure you don’t just turn it on without proper network due diligence.

http://www.youtube.com/watch?v=rpL5irIj_Qo

Excellent packet analyzer tool. I have used this a lot and had very good luck with it, it is pretty easy to use and can provide a lot of information and insight when troubleshooting network issues.

NAT Packet Analysis Using Wireshark

One of the most popular questions I get when people get the hang of protocol analysis is the daunting exercise of multitrace analysis. As with anything else the best advice is to start with the basics before tackling anything complicated.

Multitrace analysis is only effective if you truly understand your vendors products, networking and how it relates to the OSI model or packet analysis. I always suggest that you start at layer 1 and work yourself up. The key is to know what fields in the frame or packet changes, or remains the same. Ideally when you figure this out you can use a better capture or display filter

A multitrace capture of a hub, switched, or bridged network is most straight forward since a hub or switch is transparent at layer 1 or 2 and doesn’t change anything in the packet.

When you move up to layer 3 or routing, several things change in the packet such as MAC address, IP TTL and TOS. Of course your mileage will vary, and any device could be configured to muck with more bits in the packet, but I figure I would give you a point of reference.

At layer 4 we get into application gateways, proxy, firewalls and NAT type devices where the following packet fields gets modified; MAC address, IP address, IP TOS, TCP/UDP port numbers, TCP ACK/SEQ values, etc.

Lastly at layer 7, we are dealing with multi-tiered applications and basically everything changes in the packet.

In this video example I do a multitrace analysis of a simple netgear router/NAT/firewall device where I take a trace from the WAN and LAN side to compare. Not to sound like a broken record, but please remember that your devices might behave totally differently and these notes and techniques should only be used as a reference in your environment.

http://www.youtube.com/watch?v=J9FzaFryQIw

Multitrace analysis can be the most interesting, rewarding and unfortunately, most frustrating exercise an analyst will face.

Before we get to the packet analysis, setting up your tools for simultaneous capturing can be a feat in itself.

The time issue is the most critical when using 2 devices since the time is used to calculate the delay, jitter or latency. Some people are fine with syncing both devices to a common ntp server.

Then there’s the “how the #!!$!@#!!” do I physically capture . This is where you have to be familiar with the problem, the network you are working on and what equipment is available to you. If you are lucky enough to be able to change the speed and duplex to 100 half duplex a good old hub fits the bill. Other than the mirror/span command, a tap is also very helpful. Trust me every one of these suggestions comes with their own caveats. You may have to try different tools for different scenarios.

For example, if I am doing a simple pc bootup/login baseline, I am interested in things like total data transferred, which IP’s I am talking to, protocols used, errors, etc. In this case speed and duplex is not important and I can go with a hub. But if I was troubleshooting why something is taking too long, like a backup or replication, changing the speed and duplex would not be a good idea.

If you are lucky enough and can capture from one device, the time accuracy issue goes away and life does get a bit easier. But now you have 2 different captures in the same trace, Yikes!!!! Not to mention that different network interfaces have different latency or behaviors. I remember trying a usb to 10/100 ethernet adapter to capture packets and quickly realized that this adapter added 30 ms to every packet. Again, if I was troubleshooting latency, this won’t do.

Lastly, if you’re fortunate enough, you might even have an application that takes multiple trace files and calculates all sorts of stuff out for you (hmm.. next article?).

In this example I use Wireshark, my laptops WiFi and Ethernet ports to capture my packet traversing a residential home router. I show some tips and tricks along the way and hope this will help you out.

http://www.youtube.com/watch?v=CAS_Kb4VYjo

Documenting a Problem With Wireshark

I remember talking to a group about the ‘superman syndrome’ where the analyst wants to swoop in and save the day. I explained that like most forensic tasks, protocol analysis can be tedius, confusing and downright boring at times. Alright who wants to capture some packets now!?

If you can’t see it, you can’t fix it. That is why I like to use protocol analysis to minimally document the problem that I’m experiencing. Even if the packets don’t show any anomalies, that worth knowing as well, isn’t it? If you do see an anomaly, you might not have the solution but at least you know what it looks like when its broken.

Ideally protocol analysis is most helpful when you have two traces to compare; the good and bad trace. In most realistic scenarios, the client will not have a good trace and just the current bad trace. I’m our classes I review how to make use of what you have.

In this example the customer had a DSL line with an issue and another DSL line what worked fine. The customer mentioned that whenever the DSL circuit ‘acted up’, they simply rebooted the modem. Both DSL circuits went to the same carrier, ordered at the same time, provisioned the same way and even use the same hardware. Perfect, example of something I can compare. I also noticed that these are not just modems, but they route, dhcp, firewall and NAT.

What I found, is that the problem circuit was having issues passing larger frames, while the other had no issues. After the reboot the problem circuit now behaves like the good one. Upon further investigsation I noticed the problem modem had older firmware and suggested they get that firmware updated.

So, even though I couldn’t ‘fix’ the problem, we know exactly what the problem is and what to look for if the problem returns.

http://www.youtube.com/watch?v=OBT5XGOA3EU