IBM Support

Capture network trace

Troubleshooting


Problem

How is network trace captured?

Resolving The Problem

Background

Encryption

If you are capturing non-encrypted traffic (for example, HTTP without TLS), it may include sensitive data and the capture files should be treated sensitively.

If you are capturing encrypted traffic (for example, HTTP with TLS), depending on the negotiated cipher, even if you have the private key, it's usually impossible to decrypt the traffic without advanced diagnostics.

What to capture

Ideally, capture both sides of a network conversation. For example, if packets are lost on a source host, they will not be visible if the network capture is only taken on the destination host.

Overhead

Gathering network traces has an impact on response times, throughput, and disk usage. These impacts must be carefully reviewed before enabling network traces in a production environment.

The main determinants of the impacts are how many bytes per packet are captured and whether any filtering is done (for example, by port). If impact is a concern, minimize the number of bytes per packet and/or filter to particular ports. More generally, run a performance test in a performance environment without network tracing as a baseline and then run another test with network tracing and compare relative values of key performance indicators.

There are downsides to reducing how much is captured. For example, if you use port filtering to capture HTTP traffic and there is a slow DNS response time related to handling that traffic, then that will not be immediately seen. In general, for encrypted traffic that you plan to decrypt, you should capture the entire packet to allow for the decryption.


Linux

Preparation

If tcpdump is not installed, install it using operating system tools. For example:

  • Modern Fedora, RHEL, CentOS, ubi, and ubi-init:
    • sudo dnf install -y tcpdump
  • Older Fedora, RHEL, and CentOS:
    • sudo yum install -y tcpdump
  • Debian and Ubuntu:
    • sudo apt-get update && DEBIAN_FRONTEND=noninteractive TZ=${TZ:-UTC} sudo apt-get -y install tcpdump
  • Alpine:
    • sudo apk update && sudo apk add tcpdump
Capture without port filtering
1. Start the capture; replace $INTERFACE with the target network interface (for example, eth0):
 

sudo sh -c "date >> nohup.out && (nohup tcpdump -nn -v -i $INTERFACE -B 4096 -s 0 -C 1024 -W 10 -Z root -w diag_capture_$(hostname)_$(date +%Y%m%d_%H%M%S).pcap &) && sleep 3 && cat nohup.out"

a. This example captures up to 10 files (-W) of 1024MB (-C) each, so up to 10GB. Ensure sufficient disk space is available and increase these values as much as possible to capture maximum data.
b. This example captures the entire packet (-s 0). To minimize bytes per packet, set -s to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture:
 

sudo pkill -INT tcpdump

5. Upload diag_*, a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.), and nohup.out
Capture with port filtering
1. Start the capture; replace $INTERFACE with the target network interface (for example, eth0) and replace $PORT with the target port (for example, 80, 443, and so on):
 

sudo sh -c "date >> nohup.out && (nohup tcpdump -nn -v -i $INTERFACE -B 4096 -s 0 -C 1024 -W 10 -Z root -w diag_capture_$(hostname)_$(date +%Y%m%d_%H%M%S).pcap 'port $PORT' &) && sleep 3 && cat nohup.out"

a. This example captures up to 10 files (-W) of 1024MB (-C) each, so up to 10GB. Ensure sufficient disk space is available and increase these values as much as possible to capture maximum data.
b. This example captures the entire packet (-s 0). To minimize bytes per packet, set -s to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture:
 

sudo pkill -INT tcpdump

5. Upload diag_*, a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.), and nohup.out


It might be possible to capture network traffic without root access with certain configuration but this option is not covered in this document.


Linux on OpenShift

Gather tcpdump for a particular pod: https://access.redhat.com/solutions/4569211


AIX

Capture without port filtering
1. As root, start the capture:
 

startsrc -s iptrace "-a -b -B -L 2147483648 -S 1500 diag_aixiptrace.bin"

a. This command captures up to 2 files of 2GB each. Change -L as needed.
b. This command captures up to 1500 bytes per packet. To minimize bytes per packet, set -S to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture:
 

stopsrc -s iptrace

5. Upload diag_* and a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.)
Capture with port filtering
1. As root, start the capture; replace $PORT with the target port (for example, 80, 443, and so on):
 

startsrc -s iptrace "-a -b -B -p $PORT -L 2147483648 -S 1500 diag_aixiptrace.bin"

a. This command captures up to 2 files of 2GB each. Change -L as needed.
b. This command captures up to 1500 bytes per packet. To minimize bytes per packet, set -S to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture:
 

stopsrc -s iptrace

5. Upload diag_* and a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.)


Windows

Capture without port filtering
1. Right-click Command Prompt } Run as Administrator
2. Start the capture:
 

netsh trace start provider=Microsoft-Windows-TCPIP persistent=yes capture=yes packettruncatebytes=1500 tracefile=C:\diag_networktrace.etl maxSize=10240 perf=no

a. This command captures up to 10GB each of total data. Change maxSize in MB as needed.
b. This command capture up to 1500 bytes per packet. To minimize bytes per packet, set packettruncatebytes to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

3. Check for any errors running the previous commands in your terminal.
4. Reproduce the problem.
5. Stop the capture:
 

netsh trace stop

6. Upload diag_* and a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.)
For analysts: use etl2pcapng to convert the ETL to pcapng for Wireshark.
Capture with port filtering

netsh does not have an option to filter by a TCP port. It can filter by source or destination IP address (run: netsh trace show capturefilterHelp). Alternatively, you may use the Windows (pktmon) instructions below taking into account the caveats in the introduction or install Wireshark and use a port capture filter; however, Microsoft suggests that Wireshark’s capture technology is less performant.


z/OS

See https://www.ibm.com/support/pages/how-collect-packet-traces-and-other-tcpip-related-traces-zos
 


IBM i

See https://www.ibm.com/docs/en/ssw_ibm_i_73/rzaku/rzakupdf.pdf#page=20

Solaris
Capture without port filtering
1. As root, start the capture; replace $INTERFACE with the target network interface (for example, eth0):
 

nohup snoop -r -o diag_$(hostname)_$(date +"%Y%m%d_%H%M").snoop -s 1500 -q -d $INTERFACE &

a. This command captures up to 1500 bytes per packet. To minimize bytes per packet, set -S to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture by finding the process ID of snoop and killing it:
 

kill ${PID}

5. Upload diag_*, a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.), and nohup.out
Capture with port filtering
1. As root, start the capture; replace $INTERFACE with the target network interface (for example, eth0) and replace $PORT with the target port (for example, 80, 443, and so on):
 

nohup snoop -r -o diag_$(hostname)_$(date +"%Y%m%d_%H%M").snoop -s 1500 -q -d $INTERFACE "port $PORT" &

a. This command captures up to 1500 bytes per packet. To minimize bytes per packet, set -S to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture by finding the process ID of snoop and killing it:
 

kill ${PID}

5. Upload diag_*, a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.), and nohup.out
HP-UX
Capture without port filtering
1. As root, start the capture:
 

nohup nettl -tn all -e all -f diag_networktrace &

2. Check for any errors running the previous commands in your terminal.
3. Reproduce the problem.
4. Stop the capture by finding the process ID of nettl and killing it:
 

kill ${PID}

5. Upload diag_*, a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.), and nohup.out
Windows (pktmon)
pktmon is generally not recommended except for advanced use cases because packets will be captured at each network driver. If there are multiple network drivers, Wireshark will get confused and consider them retransmissions.

Windows (pktmon)

Capture without port filtering
1. Right-click Command Prompt } Run as Administrator
2. Start the capture:
 

pktmon start --capture --pkt-size 1500 --file-size 10240 --log-mode circular

a. This command captures up to 10GB of total data. Change file-size in MB as needed.
b. This command captures up to 1500 bytes per packet. To minimize bytes per packet, set pkt-size to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.
c. If you receive the error, "Packet monitor is already started," then first run "pktmon stop" and then re-run the "start" command.

3. Check for any errors running the previous commands in your terminal.
4. Reproduce the problem.
5. Stop the capture:
 

pktmon stop

6. Convert the capture to pcapng format:
 

pktmon etl2pcap PktMon.etl

7. Upload PktMon.etl, PktMon.pcapng, and a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.)
Capture with port filtering
1. Right-click Command Prompt } Run as Administrator
2. Configure the filtered port; replace %PORT% with the target port (for example, 80, 443, and so on):
 

pktmon filter add -t tcp -p %PORT%

3. Start the capture:
 

pktmon start --capture --pkt-size 1500 --file-size 10240 --log-mode circular

a. This command captures up to 10GB of total data. Change file-size in MB as needed.
b. This command captures up to 1500 bytes per packet. To minimize bytes per packet, set pkt-size to 80 for IPv4 and 128 for IPv6 traffic. You can go lower but it’s sometimes useful to have at least some of the TCP packet.
c. If you receive the error, "Packet monitor is already started," then first run "pktmon stop" and then re-run the "start" command.

4. Check for any errors running the previous commands in your terminal.
5. Reproduce the problem.
6. Stop the capture:
 

pktmon stop

7. Convert the capture to pcapng format:
 

pktmon etl2pcap PktMon.etl

8. Upload PktMon.etl, PktMon.pcapng, and a list of IP addresses that are relevant to the captured conversations and what each IP address represents (e.g. web server, WebSphere, database, etc.)


Notes:

  1. Microsoft suggests that Wireshark’s capture technology is less performant than ETL.
  2. etl2pcapng does not work on ETL files; pktmon etl2pcap must be used to create Wireshark files.

Document Location

Worldwide

[{"Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"ARM Category":[{"code":"a8m50000000CdEtAAK","label":"Web Container\/Servlet issues"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions","Type":"MASTER"}]

Document Information

Modified date:
11 October 2024

UID

ibm16471353

Page Feedback