Reading Data Dynamically with OpenDDS DynamicData

Middleware News Brief (MNB) features news and technical information about Open Source middleware technologies.

March 04, 2022 - By Son Dinh, Senior Software Engineer

Introduction

In order to accommodate changes to functional requirements, it may become necessary to alter the data types that a Data Distribution Service (DDS®) application's DataReaders and DataWriters use to communicate with each other. The DDS XTypes specification addresses this need by providing mechanisms that allow a data type to be extended, while maintaining compatibility with the type as currently defined.

Using DDS XTypes, applications can also read and write data samples for types that are unknown until runtime. This interesting capability is made possible via the DynamicData and DynamicType interfaces.

In this article, we discuss a use case where DynamicData proves to be useful and demonstrate how users can apply DynamicData, as implemented in the OpenDDS project, in their applications.

Overview of DDS XTypes

Suppose we are operating an HVAC system comprising a network of devices throughout the building. The devices are managed by a control center that uses commands to control how they operate. Each of the HVAC devices periodically sends its operating status to the control center via a DDS topic named “HVAC Status” with type HVACStatus.

@topic // "HVAC Status" topic
struct HVACStatus {
  @key long id; // id of the device
  double temperature;
  double air_speed;
};

The control center reads the samples for the “HVAC Status” topic from the devices and performs any processing needed before showing the data to the maintainers that are monitoring the system.

The control center can also write samples of another topic to control the devices. For the sake of simplicity, we will ignore this topic in this example.

Figure 1. HVAC System

Figure 1. HVAC System

Now, we want to upgrade the HVAC system so that a part of the system is deployed with newer devices, while the remaining still uses the existing ones. Both the new and the existing devices are connected to the existing control center; we want to reuse the same control center while minimizing any changes needed. Hence, we want the new devices to also send their statuses to the monitor using the “HVAC Status” topic.

However, the new devices also have a feature that controls the air humidity and another that detects the condition of the air filter, so maintainers know when it needs to be replaced. We want the samples sent from the new devices to convey that additional information, as well.

Before DDS XTypes, it would not have been possible to have the existing and new devices use the same topic to communicate with the control center while their data types were not exactly the same. Users would have had to use a different topic that contained the new devices' two new fields. This approach was not sustainable because a new topic had to be added every time the type changed.

With XTypes, we can keep the same topic while having the data type of the topic change over time. The control center will still be compatible with the updated types without any changes.

For our example, the existing devices can define their topic type as follows.

@topic // "HVAC Status" topic
@mutable
@autoid(HASH)
struct HVACStatus {
  @key long id; // id of the device
  double temperature;
  double air_speed;
};

The topic type of the new devices can be defined as follows.

enum AirFilterCondition { GOOD, MODERATE, BAD };
 
@topic // HVAC Status topic
@mutable
@autoid(HASH)
struct HVACStatus {
  @key long id; // id of the device
  double temperature;
  double air_speed;
  unsigned short humidity;
  AirFilterCondition af_condition;
};

Note the additions of the @mutable and @autoid(HASH) annotations.

The @mutable annotation, described by the DDS XTypes specification, tells the DDS middleware that a remote endpoint can have a different data type for the same topic but may still be compatible with the local data type. Thus, they can still communicate with each other.

The @autoid(HASH) annotation assigns each field in the type an identifier by hashing the field's name.

OpenDDS middleware implements the DDS XTypes specification and is capable of handling such type evolution of a DDS system while maintaining its compatibility [2].

Using DynamicData in an OpenDDS Application

In the preceding example, the data types are known at compile time. OpenDDS middleware processes the Interface Definition Language (IDL) defining the types and generates type support code. The type support code constructs local type information. This local type information is then propagated to the remote DataReaders and DataWriters in the same DDS domain for checking type compatibility. Type compatibility checking is one of the steps that determines whether two endpoints can be matched. The type support code also contains logic for serializing and deserializing samples (encoding and decoding in the binary format used for network messages).

However, there are use cases in which knowing the data types at compile time isn't necessary or desirable, and instead users may want to be able to handle types dynamically at runtime. For example, suppose we want to make the building in our example “smarter” by installing DDS-enabled lights and cameras that help operate the building more efficiently and securely. These new lights and cameras are also connected to the same control center as the HVAC devices.

Figure 2. Upgraded HVAC System with New DDS-enabled Devices

Figure 2. Upgraded HVAC System with New DDS-enabled Devices

One way to implement this using only the DDS XTypes features discussed so far is to have two new topics, one for light status and one for camera status. The new lights and cameras will use these topics, respectively, to update the statuses to the control center. The control center, however, also needs to know about these new topics in order to receive their samples. That means we have to define these topic types in the control center as well and recompile its DDS application code to make these changes take effect.

An issue with this implementation is that if we want to install any new device classes, such as smart locks, in the future, we will need to recompile the control center's application code again to add a new topic for the new devices. We want to minimize changes needed by the control center while making it possible to operate with any new classes of equipment that may be added to the building in the future.

In such cases, we can use the dynamic language binding feature of XTypes, in particular, DynamicData and DynamicType interfaces.

Overview of DynamicData and DynamicType

An instance of DynamicType is used to represent a DDS type. It contains complete information about the type and provides methods to query this information. Using DynamicType and its related interfaces, such as DynamicTypeBuilder (not covered in this article), users can construct a type at runtime. Users can also convert types defined at compile time to corresponding DynamicType objects. OpenDDS middleware provides methods to convert types defined in IDL to DynamicType.

DynamicData represents a data sample of a type. A DynamicData object is associated with a DynamicType object that describes the type of the corresponding sample. Users can construct a DynamicData object and use methods that populate the fields of the sample. The DynamicData object can then be passed to a DataWriter before being sent to the wire.

In the other direction, a DynamicData object has methods that allow a user to retrieve the corresponding sample's fields. In both cases, the associated DynamicType object is used to guide the construction, as well as the traversal of the sample. Note that all of this can be done at runtime, which means that an application doesn't need to be recompiled each time a new type is added.

At the time of writing, the OpenDDS project has implemented the reading methods of DynamicData. They allow reading the values of basic types, including signed and unsigned integers (8-bit, 16-bit, 32-bit, and 64-bit), floating point types (32-bit, 64-bit, 128-bit), chars, wide chars, strings, wide strings, bytes, and booleans. There is also a method to read members with complex types, such as structs, unions, arrays, and sequences of non-basic types. The read methods have the following form:

DDS::ReturnCode_t get_int32_value(ACE_CDR::Long& value, MemberId id);

As the name of the method suggests, it reads the value of a 32-bit integer member with the given id from the DynamicData object. If the member with the given id is actually a 32-bit integer (or an enum represented by a 32-bit integer), its value is written to an ACE_CDR::Long output argument. Otherwise, the method returns an error code.

The methods for reading other basic types are similar, except that the method name and the output parameter must match the corresponding type.

For each of the basic types, there is also a method to read a sequence of values of that type, which take the following form.

DDS::ReturnCode_t get_int32_values(LongSeq& value, MemberId id);

This method is similar to the previous method, except that it checks that the member with the provided id is actually a sequence of the requested basic type, and it returns a sequence of values of that type instead of a single value.

To read a member whose type is not a basic type or a sequence of a basic type, such as a nested struct or union member, we can use the following DynamicData method.

DDS::ReturnCode_t get_complex_value(DynamicData& value, MemberId id);

The contents of the complex member with the given id is read into a separate DynamicData object. Using this DynamicData object, users can recursively traverse the fields of the complex member.

The get_complex_value method is quite flexible and can even read a member of a basic type into a DynamicData object. More information about using the reading methods of DynamicData can be found in the OpenDDS Developer's Guide [2] and DDS XTypes specification [1].

Now, we have learned how to read values from a DynamicData object. Let's return to our example with the smart building.

Using OpenDDS DynamicData

OpenDDS applications can use DynamicData via the Recorder feature of the OpenDDS project. This allows an application to record samples of a particular topic [2]. The application creates a typeless topic object for the topic it wants to read.

DDS::DomainParticipantFactory_var dpf = TheParticipantFactoryWithArgs(argc, argv);
 
DDS::DomainId_t domain_id = 1;
DDS::DomainParticipant_var participant =
  dpf->create_participant(domain_id,
                          PARTICIPANT_QOS_DEFAULT,
                          0,
                          0);
 
const char* const topic_name = "HVAC Status";
const char* const type_name = "HVACStatus";
Service_Participant* const service = TheServiceParticipant;
 
DDS::Topic_var topic = 
  service->create_typeless_topic(participant,
                                 topic_name,
                                 type_name,
                                 true,
                                 TOPIC_QOS_DEFAULT,
                                 0,
                                 0);

We can use the created topic object to create a Recorder, which will listen to the samples published on that topic.

DDS::SubscriberQos sub_qos;
participant->get_default_subscriber_qos(sub_qos);
 
DDS::DataReaderQos dr_qos = service->initial_DataReaderQos();
dr_qos.representation.value.length(1);
dr_qos.representation.value[0] = DDS::XCDR2_DATA_REPRESENTATION;
dr_qos.reliability.kind = DDS::RELIABLE_RELIABILITY_QOS;
 
Recorder_var recorder = service->create_recorder(participant,
                                                 topic,
                                                 sub_qos,
                                                 dr_qos,
                                                 recorder_listener);

The last argument to the create_recorder call above is a RecorderListener object that implements the following interface:

class RecorderListener {
public:
  virtual void on_sample_data_received(Recorder* recorder,
                                       const RawDataSample& sample) = 0;
 
  virtual void on_recorder_matched(Recorder* recorder,
                                   const DDS::SubscriptionMatchedStatus& status) = 0;
};

on_recorder_matched is called when a DataWriter is matched with the Recorder.

on_sample_data_received is called when a sample of the topic associated with the Recorder is received.

This is where the application can acquire the DynamicData of the received sample.

class RecorderListenerImpl : public RecorderListener {
public:
  virtual void on_sample_data_received(Recorder* rec,
                                       const RawDataSample& sample)
  {
    OpenDDS::XTypes::DynamicData dd = rec->get_dynamic_data(sample);
 
    // Read and process the sample via DynamicData's methods...
  }
 
  virtual void on_recorder_matched(Recorder*,
                                   const DDS::SubscriptionMatchedStatus& status)
  {
    std::cout << "Associated with a writer..." << std::endl;
  }
};

After the Recorder is created, it will learn about the types of any matched DataWriters and store the corresponding DynamicTypes in an internal data structure. The get_dynamic_data call inside on_sample_data_received returns a DynamicData object constructed from the received sample and the stored DynamicType of the corresponding topic. Now, the application can read the fields of the received sample using the reading methods of DynamicData discussed in the previous section. The application is free to process the data as necessary; for example, it can log the interpreted sample in real time to a monitor or store it to a database.

Note that the application only needs to know the name of the topic and the name of its type; it doesn't need to know the definition of the type. When we add a new topic to our system, such as a topic for the status of the new lights or cameras in our smart building example, the application can create a new Recorder that will record the samples published on the new topic.

The application can learn about the existence of new topics via the DCPSPublication Built-In Topic [2]. It contains information about DataWriters, including the name and type name of the topic each DataWriter is writing.

The OpenDDS project provides a tool, located under tools/inspect in the OpenDDS repository, that works in a similar way to our example. It receives a topic name and the name of the topic type as two of its arguments and prints the contents of the matched samples to the screen. The tool can be helpful for developing or debugging a DDS application.

Conclusion

In this article, we have briefly discussed the capabilities of DDS XTypes and its implementation in OpenDDS applications. We demonstrated some of its capabilities through an example of a DDS system that requires support for type evolution. We showed how DynamicData can be useful for an extensible DDS application and how it can be used in an OpenDDS application.

The current version of the OpenDDS project implements only the reading direction of DynamicData. Once the writing direction is complete, it will allow applications to dynamically construct samples for a topic that the application only learns at runtime.

Acknowledgements

Thanks to the OpenDDS team at Object Computing for help with revising the article.

References

  • [1] DDS XTypes Specification, https://www.omg.org/spec/DDS-XTypes/1.3
  • [2] OpenDDS Developer's Guide, https://download.objectcomputing.com/OpenDDS/OpenDDS-latest.pdf