Using Wireshark to Monitor and Debug OpenDDS Systems

Introduction

Like all network-based communication, DDS systems can be monitored and debugged using Wireshark. Wireshark is an open source packet analyzer that can be used to monitor a wide range of network activity. More information about Wireshark, including how to install it, can be found at wireshark.org.

Wireshark uses modules called dissectors to decode packets on a protocol-by-protocol basis. This allows users to view and filter packets based on the behavior of a particular protocol.

There are two dissectors that can be used with the OpenDDS project:

1. The built-in Wireshark RTPS dissector
2. The OpenDDS DCPS dissector

The former is for any DDS system using the RTPS protocol, which can include OpenDDS systems when configured to use RTPS, while the latter is for OpenDDS systems that are not using RTPS, such as ones using the tcp transport.

The following examples and screenshots were made using OpenDDS project 3.17.0 and Wireshark 3.4.5. If you'd like to follow along, you can download the capture files here.

Wireshark RTPS Dissector

It is helpful to set the capture filter to udp when setting up to capture RTPS packets. As RTPS is carried over UDP, this cuts down on the number of packets that Wireshark has to save. The capture filter (not to be confused with the display filter) is below the capture interfaces on the welcome screen and in the "Capture Options" dialog, as shown below:

• On Windows, you will want to use the loopback interface to do a capture.
• On Linux you will want to use the any interface.
• If you’re on macOS or you’re not sure what to pick, you should select the loopback interface and your main network interface, which could be your Ethernet or WiFi interface.

Example

As an example, we will use the OpenDDS project’s Messenger test. The Messenger test can be found in tests/DCPS/Messenger in the OpenDDS source tree.

The following screenshot is a capture from running the Messenger test with run_test.pl rtps_disc, which uses RTPS:

The packets in the middle were edited out because they were all similar data samples. The capture file is attached to this article if you want to follow along. We won’t cover every single packet in detail, but we’ll at least cover the gist.

The capture file being used is called messenger_rtps.pcapng.

Packets 1 and 2: Participant Announcements

All RTPS messages have a header and one or more submessages.

• The header contains basic information, including the protocol version (2.4 in this case), the DDS implementation (OpenDDS), and the participant GUID prefix.
• In this case, there is only one submessage, which is a DATA submessage. DATA submessages contain the serialized data that an RTPS writer sends to any RTPS readers listening.

DDS DataWriters that send the user’s data using RTPS will also be RTPS writers, but in this case the RTPS writer is a special built-in writer indicated by the DATA(p) in the packet list and ENITITYID_BUILTIN_PARTICIPANT_WRITER in the packet details.

The participant writer writes about the participant it’s part of, so the first packets are what allow participants to know about each other.

One interesting thing to notice is that the IP destination isn’t the same as the source, even though this is a local test. This destination is a multicast address because RTPS discovery uses multicast to automatically discover other hosts on the network.

The RTPS transport also uses multicast by default, but it can be configured to use unicast.

If we open up the serializedData field and then the serializedData field inside of that, we can see most of the same information from the header, along with some more details. Unlike the header, the information found here can differ depending upon various settings and the DDS implementation in use. For example PID_VENDOR_ID will be different for each implementation, and DDS extensions such as XTypes and Security can add additional parameters to this list.

If you are trying to debug why two participants can’t communicate with each other, trying to find the participant announcements is a good place to start.

Packet 3: Writer Announcement

The next packet is from another built-in RTPS writer called the publication writer. A message from a publication writer is indicated by DATA(w) in the packets list and ENTITYID_BUILTIN_PUBLICATIONS_WRITER in the packet details.

The publication writer gives out information about DataWriters so DataReaders can get ready to receive messages if they match. This information includes the topic name (PID_TOPIC_NAME), the topic type name (PID_TYPE_NAME), and whether the writer is reliable (PID_RELIABILITY).

Packet 4: Instance Registration

This packet is an RTPS message with a DATA submessage like the previous ones, but this one is a DDS instance registration message from the DDS DataWriter.

Instance registration messages just contain a subset of what is in a normal message. This subset is the serialized key of the instance. They are indicated in older versions of Wireshark with DATA([__]) in the packet list. Newer versions of Wireshark will include the topic name if possible, like DATA -> Movie Discussion List([__]) below. If the discovery messages required for this are missing from the capture, then the topic name will be missing.

(The reason for the [__] is that the data that appears here varies; some kinds of instance messages have letters between the brackets. We will cover those messages shortly.)

Packets 5-44: Samples

These are the RTPS messages carrying data submessages with the normal serialized data DDS applications write to one another. Below is a screenshot of the first sample.

Inside, you can see the sample data, which includes the strings “Comic Book Guy”, “Review”, and “Worst. Movie Ever.”.

Packets 45 and 46: Participant Announcements (Again)

Participant announcements are repeated periodically to let new participants know about the existing participants. This period can be decreased in OpenDDS systems for potentially quicker discovery time at the expense of more traffic on the network.

Packets 47-61: Reliability in Action

Except for the DATA submessage in packet 56, these packets represent reliable RTPS writers and readers negotiating with each other to try to make sure the readers got all the data. This includes all the reliable built-in entities as well as the user-defined ones.

How it works is that each message has a sequence number that identifies a unique message from an RTPS writer. If you look back at the screenshot of packet 4, you'll see that it has writerSeqNumber: 1, and the screenshot of packet 5 has writerSeqNumber: 2. The sequence number increases for each unique message that is written.

If an RTPS writer is reliable, which is the default for DDS DataWriters, it will periodically send out a HEARTBEAT submessage. These submessages say what the sequence number of the latest message is and what the sequence number of the oldest message it can still resend is.

This is the HEARTBEAT submessage for the “Movie Discussion List” DataWriter:

Here, it’s saying it has sent sequence numbers 1 - 41, which comprise the instance registration message and the 40 sample messages.

If the RTPS reader is reliable, like the “Movie Discussion List” DataReader is in this capture, it will respond with a ACKNACK submessage listing what it has got so far.

Note that "reliable" is not the default QoS for DDS DataReaders like it is for DDS DataWriters.

If the reader thinks it got everything, it will set a final flag. Here is one of the ACKNACKs for the DDS DataReader:

Here the reader is saying it has received the instance registration message and all 40 sample messages, and it’s expecting a data message with the sequence number 42. Number 42 hasn’t been sent quite yet, so the writer doesn’t have to do anything.

If the reader is missing messages that the writer can still resend, it will do so. If the writer cannot send the messages (this can happen for various reasons due to QoS), the writer will send a GAP submessage, saying it can't send those messages anymore.

Packets 56: Unregister and Dispose Instance

This is a data submessage like the previous ones. However, it doesn't have a complete data payload, so it's an instance management message like packet 4.

Instead of registering an instance, though, this packet is unregistering and disposing it. This is indicated by the ([UD]) in the packet list and the Unregistered, Disposed in DATA → inlineQos → Flags field in the packet details.

The unregister flag tells readers that the writer isn't going to publish new data about the DDS instance.

The dispose flag tells readers that the DDS instance in general is "finished." What dispose actually means is up to the application using DDS, but these two are sent at the same time by default when a DDS DataWriter is deleted. That can be disabled by the application, and separate unregister and disposes can be sent manually.

Packets 62-66: Wrapping Up

• Packet 62 is the publication writer writing an unregister/dispose because the DDS DataWriter was deleted.
• Packets 64 and 65 are the DDS participant that the DDS Reader belongs to doing some more announcements.
• Packets 63 and 66 are the participant writers writing unregister/disposes for their respective participants because they were deleted.

After that communication is finished.

Filtering

Like other dissectors, you can use display filters to reduce the potentially thousands of packets or more down to ones that you're interested in seeing. See https://www.wireshark.org/docs/dfref/r/rtps.html for the display filters available for RTPS.

You can also use filter options like ip.src on RTPS packets, since they are also IP packets.

Security

Authentication

DDS Security enables many options to protect RTPS DDS communications. These include adding certificates to participant announcements to verify that a participant can be trusted, at least for the initial participant association.

This basic level of security can be enabled in the same Messenger test as above using run_test.pl rtps_disc_sec.

The capture file being used is called messenger_sec.pcapng.

Here the participant announcements have additional contents that allow participants to decide if they should try to associate with other participants. If they decide to, then they go through a process to authenticate each other. These are the following submessages with DATA(s).

After that, the way in which security changes things depends on how it's been configured. If it’s like Messenger, then it’s basically the same, but we will explore how different encrypt options change the captures.

Because the Messenger test has the simplest possible security configuration, the rest of the security examples will use the "attributes" test found in tests/security/attributes in the OpenDDS source tree. It is based on the Messenger test, so the data that is written is very similar.

Payload Encryption

The sample payload can be encrypted, which hides the serialized payload of user DATA submessages. This example uses run_test.pl SC3, and there is a capture attached named attributes_payload.pcapng.

This is sample data whose contents were similar to the Messenger test when written. We could read the strings in the samples previously, but that’s not the case here since it’s encrypted.

Full Message Encryption

Full message encryption, despite its name, doesn’t actually encrypt everything in the RTPS message. It wraps and encrypts most, but not all, submessages as shown below.

This example uses ./run_test.pl FullMsgEncrypt, and there is a capture attached named attributes_message.pcapng.

This hides all of the payloads as before, as well as most of the DDS data, except what’s required to deliver the message.

To use Wireshark with an OpenDDS system with any sort of encryption on the payload enabled, it might be helpful to temporarily disable encryption by using the DCPSSecurityFakeEncryption configuration option that’s documented in the OpenDDS Developer’s Guide. Wireshark will not be able to dissect things that would otherwise be encrypted, but the data will be sent unencrypted, so it can be observed in the binary dump output.

OpenDDS DCPS Dissector

The OpenDDS DCPS Dissector is built with OpenDDS technology and must be set up properly to use. See tools/dissector/README.md for the complete documentation.

This section uses the same Messenger test as before, run with just ./run_test.pl so that it uses the default InfoRepo discovery and the tcp transport. The capture file being used is called messenger_repo_tcp.pcapng.

Here is a small example of what it looks like:

InfoRepo discovery uses a central CORBA server to discover participants and set up DataWriters with their corresponding DataReaders. This happens over the TCP-based GIOP protocol and serves the same purpose as the UDP-based RTPS discovery.

After that, the DataWriters and DataReaders use the OpenDDS DCPS protocol to exchange the data. In this case it’s TCP-based, but there are also transports based on unicast and multicast UDP (udp and multicast).

An advantage to the OpenDDS DCPS dissector is that, once set up properly, it can view and filter data sample messages based on their contents, using the fields defined in the IDL file.

Conclusion

This was just a quick exploration of what’s possible, but it shows that Wireshark is a powerful tool when it comes to network applications, and DDS is no exception. It can be used to analyze OpenDDS-based systems in action, either to debug them or just to see if there’s room for improvement in the QoS settings.