SSLDump – Hey where’d it go?

Today’s Issue:

How to monitor & dump HTTP and HTTPS traffic for troubleshooting a Sharepoint 2007 website with F5 Support, on an LTM 6400 running 10.0.1, configured for SSL Offloading

Where We Are:

As we’ve talked about in the previous post, I can still reliably reproduce the problem of trying to modify a Sharepoint Web Part using the Rich Text Editor while in IE.  I’ve been working with F5 support to get to the bottom of this, but it sounds like they’re as stumped as me.  So now we’re doing some real-time monitoring and TCPDumps, which they’re trying to decipher… good choice of words, as this is all SSL traffic, so how do you read it?  Here’s how to get from point gobbledygook to point B.

How We Got Here:

Once again, in a nutshell:

  • Sharepoint 2007 Application Template deployment on and LTM 6400 running 10.0.1
  • SSL Offloading configured by using OpenSSL to break PFX into SSL Cert & Key, then imported onto LTM
  • Browse to site as admin from outside (hitting F5 on port 443 first, then F5 passes you to web server over port 80 in the back), edit the web part, and you get errors
  • Browse to site as admin from inside (hitting web server directly over port 443 from an internal subnet) edit the web part and life is good

Recommendation from F5 is to run HTTPWatch on the sessions while concurrently running a TCPDump locally on the F5, and send them the goods on both a working and non-working set of transactions.  Here’s what we did, and what I messed up (still wiating to hear back from today’s uploads)

Step 1 – Load up HTTPWatch on your Client

Nothing real earth-shattering here.  It’s about a 9MB download, but installs quickly.  Took a few extra clicks in IE8 to get the button to show up nicely on the toolbar, but no real bother (Tools, Toolbars, Customize, Current Toolbar Buttons, and move it on up).

Step 2 – Login to F5 and initiate the TCPDump

The guys at F5 were kind enough to provide me with the entire command they wanted to use (unfortuantely I’m new at this, so I don’t know all of the switches they used).  Login to the console of your LTM and run:

tcpdump -ni 0.0 -s0 -w /var/tmp/capturefile.pcap host <Public IP Address> or <Back End IP Address>

You won’t really see anything happening at this point, except for “tcpdump: listening on 0.0

Another thing to remember here is, who you’re logged in as, as this will determine which /tmp folder you’re writing to.  I made this mistake the first time around, when I went back to get the .pcap file, and it wasn’t where I was looking… more on this later.

Step 3 – Initiate HTTPWatch

Now that you’ve provided yourself a nice button to click on your toolbar, you can crank up HTTPWatch, and be sure to click on the Record Button before you begin.  Once you get going, moving around your page, you;ll see all sorts of traffic whiz by in the HTTPWatch session window… this is a good thing.

This is the point at which we intentionally broke things, and hoped all of the recorders were doing their jobs.

Step 4 – End the HTTPWatch Session

So now that we’ve made it do what we want it not to do/broken it, it’s time to send in the results.  First things first.  Click on the STOP button on HTTPWatch, and then be sure to SAVE your session.  Another point of note, in our situation, when we log out of the Sharepoint site, it wants to close the window… don’t let it until you’ve stopped and saved your session!

Step 5 – Stop the TCPDump

Flip back to your Console session on the LTM, and press Control-C.  you should get something like:

<X,XXX>packets received by filter
0 packets dropped by kernel

Step 6 – Decrypt your TCPDump

So as you recall, in our case all of this traffic runs as an HTTPS session, so to see it to troubleshoot, you need to decrypt it.  So while you’re still on the LTM Console screen run what F5 support provides to do that:

ssldump -Aden -k /config/ssl/ssl.key/www.mySSLsite.com.key -p <password used to create key> -r /var/tmp/capturefile.pcap > ssldump.out

Here’s a little something else about to keep in mind about file locations.  The command as written above dumps the unencrypted session traffic to a file called “ssldump.out” that will reside in your current working directory.  Keep that in mind when you go looking for the file to copy somewhere else.  Workarounds include:

  • run the command pwd to make sure of where you are when you’re looking for the output file
  • direct the output file to a specific location (…. > /var/tmp/ssldump.out)
  • change your working directory to somewhere else before running the command (cd /var/tmp)

Step 7 – Go get yer stuff

Now it’s time to go get everything and beam it up to the mothership so they can find the real problem.  I use WinSCP to transfer files directly to/from the LTM:

  • HTTPWatch files – .HWL files that you saved from the client your web browser is running on
  • TCPDump files – .PCAP files that were created on the LTM, in our example in /var/tmp (make sure that you login to WinSCP as the same user you ran the TCPDump command as, or know where to look)
  • SSLDump files – .OUT files that were created on the LTM from the SSLDump command (if your traffic is all from an HTTP session, you don’t need to go through this step)

So off they go to the people who know.  Hopefully they find something nifty that we can all learn from.  Once I know, you will too.


1 Response to “SSLDump – Hey where’d it go?”

  1. 1 Wael
    August 15, 2009 at 9:08 pm

    What you really wanna do is run a capture on the external and internal interfaces of your LTM. The external one you’ll need to decode, whereas the internal one will be in the clear (because of the SSL termination).

    That way you can compare both streams and see whether the LTM is altering the traffic as it goes through it in any way that would be breaking your app from working as intended.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: