Code Generation with OpenDDS [Part II]

Middleware News Brief (MNB) features news and technical information about Open Source middleware technologies.

July 02, 2010 - By Charles Calkins, OCI Senior Software Engineer

INTRODUCTION

As stated in Part I [1] of this article, applications that use the Data Distribution Service (DDS) typically have two elements in common:

Boilerplate code. The sequence of steps to initialize the DDS framework, and to create and destroy domain participants, is the same from project to project. Simplifying the code to write for the application's skeleton reduces development time.
Knowledge of IDL and C++. DDS implementations, such as OpenDDS [2], are written in C++ and require structures to be used as data samples to be described in the Object Management Group's Interface Definition Language (IDL). Allowing a developer to write code which uses DDS in their language of choice, rather than in IDL and C++, can lead to a shorter learning curve and wider use of DDS as a technology.

Part I of this article addressed the boilerplate code issue — a wrapper around OpenDDS reduced the code written for publisher and subscriber applications to simple Publish() and Subscribe() methods. The code to provide the wrapper, however, still needed to be written by hand to yield the simplified interface.

This article will build on Part I and show how the wrapper code can be generated automatically, given the definition of a structure to be used for the OpenDDS data type.

CODE GENERATION

In an application that uses DDS, the structure used for the data sample is described in IDL. This requires a developer to be familiar with IDL syntax and to implement code that is different than the language used for the rest of the project. It is our desire that the developer should be able to describe the data sample in the same programming language as used for the rest of the project.

This is possible in the .NET world through the use of attributes [3], and reflection [4]. A data type for use as a data sample can be defined in a .NET language, and annotated with attributes to identify that it should be made visible to OpenDDS. Through reflection, these attributes can be programmatically examined, and appropriate IDL definitions automatically generated.

Creating attributes is straightforward. For our purposes, we need two. One attribute marks structures that should be made visible to OpenDDS, and the other identifies fields within the structures that should be used as DDS key fields. We create a project named Attrib, containing the file Attrib.cs which will contain our attribute definitions. The full implementation is available in the code archive that accompanies this article.

Our first attribute, DCPSDataTypeAttribute, will be used to mark data types for use with OpenDDS. AttributeUsage is an attribute of its own, giving the C# compiler information on how the attribute being defined should operate. AttributeTargets specifies where the attribute being defined can appear — Struct indicates that it only applies to structures (a value type). AllowMultiple is true if this attribute can be applied more than once to a type, while false indicates that it cannot. Inherited is true if the attribute can propagate to derived classes of the annotated type, while false indicates that it cannot. Finally, inheriting from theSystem.Attribute class is all that is needed to complete the attribute definition.

// Attrib/Attrib.cs
[AttributeUsage(AttributeTargets.Struct, AllowMultiple = false, Inherited = false) ]
public class DCPSDataTypeAttribute : Attribute
{
    public DCPSDataTypeAttribute()
    {
    }
}

We now create a second attribute, DCPSKeyAttribute, to identify fields that should be used as DDS keys. The definition is nearly the same as for DCPSDataTypeAttribute except that this attribute only applies to fields.

[AttributeUsage(AttributeTargets.Field, AllowMultiple = false, Inherited = false)]
public class DCPSKeyAttribute : Attribute
{
    public DCPSKeyAttribute()
    {
    }
}

Recall that the definition of the Message type in IDL is as follows:

// Messenger_IDL/Messenger.idl
module Messenger {
 
#pragma DCPS_DATA_TYPE "Messenger::Message"
#pragma DCPS_DATA_KEY "Messenger::Message subject_id"
 
    struct Message {
        string from;
        string subject;
        long subject_id;
        string text;
        long count;
    };
};

Use of the above attributes allows this structure, defined in C# in the MessengerGenCS_CS_Struct project, to have the same semantic meaning:

// MessengerGenCS_CS_Struct/Struct.cs
namespace Messenger 
{
    [DCPSDataType]
    public struct Message
    {
        public string from;
        public string subject;
        [DCPSKey]
        public int subject_id;
        public string text;
        public int count;
    }    
}

We can now write an application which will generate the IDL definition that corresponds to the annotated structure, as well as the C++ wrapper that was described in Part I of this article. We will call this application DDSGen.

Three arguments are presented to DDSGen: A .NET assembly containing annotated types (such as MessengerGenCS_CS_Struct), a base name to use for IDL creation, and an output directory for where the generated files will be placed. After validating the arguments, the Generate() function is called.

// DDSGen/DDSGen.cs
    public void Generate(string structAssembly, string baseName, 
        string outputDirectory)
    {
        Dictionary<string, List<Type>> ddsTypes = 
            GetDDSTypes(Assembly.LoadFrom(structAssembly));
        GenerateIDLFile(ddsTypes, baseName, outputDirectory);
        GenerateDDSImpl(ddsTypes, outputDirectory);
    }

Generate() performs three actions: obtains a list of OpenDDS-annotated types from the supplied assembly, generates the IDL that corresponds to the types, and then generates the C++ wrapper for the types. The first of these, GetDDSTypes(), is defined as follows.

First, a string is defined to be used when the annotated structure is not contained within a namespace. In the Message example, the structure is within the Messenger namespace.

 const string NO_NAMESPACE = "NO_NAMESPACE";

The GetDDSTypes() uses reflection to open and examine an assembly. The previous call to Assembly.LoadFrom() opened the supplied file name and created an Assembly object from it, which is passed to GetDDSTypes().

    Dictionary<string, List<Type>> GetDDSTypes(Assembly a)
    {
        Dictionary<string, List<Type>> ddsTypes = 
            new Dictionary<string, List<Type>>();

A call to GetTypes() returns all types in the assembly, and each type is examined in turn.

        foreach (Type type in a.GetTypes())
        {

As the DCPSDataTypeAttribute is only applicable to value types, only the custom attributes for value types are examined.

            if (type.IsValueType)
            {
                foreach (Attribute attr in type.GetCustomAttributes(true))
                {

If the custom data type is a DCPSDataTypeAttribute, then the type is stored in the ddsTypes dictionary, sorted by the namespace that it is contained within. This dictionary is returned to the caller.

                    DCPSDataTypeAttribute dataAttr = attr as DCPSDataTypeAttribute;
                    if (null != dataAttr)
                    {
                        // NO_NAMESPACE as can't use a null as a key
                        string ns = String.IsNullOrEmpty(type.Namespace) ? 
                            NO_NAMESPACE : type.Namespace;
                        if (!ddsTypes.ContainsKey(ns))
                            ddsTypes.Add(ns, new List<Type>());
                        ddsTypes[ns].Add(type);
                    }
                }
            }
        }
 
        return ddsTypes;
    }

With the types extracted, we can generate an IDL file containing the types. We start by opening a text file for writing with the supplied base name, in the specified output directory, if there are OpenDDS types to process.

    void GenerateIDLFile(Dictionary<string, List<Type>> ddsTypes, 
        string baseName, string outputDirectory)
    {
        if (ddsTypes.Count == 0)
            return;  // nothing to do
 
        TextWriter idlFile = new StreamWriter(outputDirectory + "\\" + 
            baseName + ".idl");

OpenDDS types are sorted by namespace, so, for each namespace (if not NO_NAMESPACE), emit an IDL module declaration.

        foreach (string ns in ddsTypes.Keys)
        {
            if (ns != NO_NAMESPACE)
                idlFile.WriteLine("module " + ns + " {");

Next, for each type, emit an appropriate DCPS_DATA_TYPE pragma. This code does not support nested namespaces as that is a feature not needed by the Messenger example, but it can be modified to if desired.

            foreach (Type ddsType in ddsTypes[ns])
            {
                string fullTypeName = "";
                if (ns != NO_NAMESPACE)
                    fullTypeName = ns + "::";
                fullTypeName += ddsType.Name;
 
                idlFile.WriteLine("#pragma DCPS_DATA_TYPE \"" + fullTypeName + "\"");

For each variable in the structure, we look to see if any are OpenDDS key fields, again by using reflection. If keys are present, appropriate DCPS_DATA_KEY pragmas are emitted.

                // build key list
                List<FieldInfo> keys = new List<FieldInfo>();
                foreach (FieldInfo fi in ddsType.GetFields())
                {
                    foreach (Attribute attr2 in fi.GetCustomAttributes(true))
                    {
                        DCPSKeyAttribute keyAttr = attr2 as DCPSKeyAttribute;
                        if (null != keyAttr)
                            keys.Add(fi);
                    }
                }
                foreach (FieldInfo fi in keys)
                    idlFile.WriteLine("#pragma DCPS_DATA_KEY \"" + 
                        fullTypeName + " " + fi.Name + "\"");

We can now write the structure itself. Only a few .NET types are referenced here, but are sufficient for the Messenger application. Additional types can be specified for a more complete .NET-to-IDL conversion.

                // structure
                idlFile.WriteLine("struct " + ddsType.Name + " {");
                foreach (FieldInfo fi in ddsType.GetFields())
                {
                    idlFile.Write("  ");
                    if (fi.FieldType == typeof(String))
                        idlFile.Write("string");
                    else if (fi.FieldType == typeof(int))
                        idlFile.Write("long");
                    else if (fi.FieldType == typeof(long))
                        idlFile.Write("long long");
                    else  // for now, emit only types needed for the Messenger example 
                        Console.WriteLine(fi.Name + " with a type of " + 
                        fi.FieldType + " is not supported");
 
                    idlFile.WriteLine(" " + fi.Name + ";");
                }
                idlFile.WriteLine("};");
                idlFile.WriteLine();
            }
 
            if (ns != NO_NAMESPACE)
                idlFile.WriteLine("};");
        }
        idlFile.Close();
    }

With the IDL file written, we can now move on to the C++ wrapper. Recall that in Part I of this article, a generic wrapper was developed, and then specialized for the specific structure being wrapped (the Message structure). Given the list of OpenDDS data types, we can automatically generate the specialized wrapper. The method GenerateDDSImpl generates the DDSImpl.[h,cpp] files that we constructed by hand in Part I. As with the generation of the IDL file, text files are opened in the output directory for the C++ wrapper. Various additional methods are used to write sections of the wrapper.

    void GenerateDDSImpl(Dictionary<string, List<Type>> ddsTypes, string outputDirectory)
    {
        TextWriter ddsImplHFile = new StreamWriter(outputDirectory + "\\DDSImpl.h");
        TextWriter ddsImplCPPFile = new StreamWriter(outputDirectory + "\\DDSImpl.cpp");
 
        ddsImplHFile.WriteLine("#ifndef __DDSIMPL_H__");
        ddsImplHFile.WriteLine("#define __DDSIMPL_H__");
        ddsImplHFile.WriteLine();
        WriteTypeSupportImplIncludes(ddsImplHFile, ddsTypes);
        WriteTypeTraits(ddsImplHFile, ddsTypes);
        ddsImplHFile.WriteLine("#include \"Common/Common.h\"");
        WriteTypeSupportImplIncludes(ddsImplHFile, ddsTypes);
        ddsImplHFile.WriteLine();
        WriteNetTypes(ddsImplHFile, ddsImplCPPFile, ddsTypes);
        GenerateDataReaderListener(ddsImplHFile, ddsTypes);
        GenerateReaderWriterMaps(ddsImplHFile, ddsTypes);
        ddsImplHFile.WriteLine();
        GenerateDDSImplClass(ddsImplHFile, ddsTypes);
        GenerateDDSNetClass(ddsImplHFile, ddsTypes);
        ddsImplHFile.WriteLine("#endif");
        ddsImplHFile.Close();
        ddsImplCPPFile.Close();
    }

For instance, the GenerateDataReaderListener() method is as follows. The methods GetFullTypeName() and GetFullNetTypeName() are used to generate correctly formatted forms of the type names, and the remainder of the method emits the appropriate C++ code. As the other code generation methods work similarly, they will not be discussed here — please refer to the code archive for details.

    void GenerateDataReaderListener(TextWriter ddsImplHFile, Dictionary<string, 
        List<Type>> ddsTypes) 
    {
        foreach (string ns in ddsTypes.Keys)
        {
            foreach (Type ddsType in ddsTypes[ns])
            {
                string fullTypeName, fullTypeNameSep, fullNetTypeName, fullNetTypeNameSep;
                GetFullTypeName(ns, ddsType.Name, out fullTypeName, 
                    out fullTypeNameSep);
                GetFullNetTypeName(ns, ddsType.Name, out fullNetTypeName, 
                    out fullNetTypeNameSep);
 
                ddsImplHFile.WriteLine("class " + fullTypeName + 
                    "DataReaderListenerImpl : public DataReaderListenerImplBase<" + 
                    fullTypeNameSep + "DataReader, " + fullTypeNameSep + "> {");
                ddsImplHFile.WriteLine("    gcroot<EventManager<" + fullNetTypeNameSep + 
                    ">^> eventManager_;");
                ddsImplHFile.WriteLine("public:");
                ddsImplHFile.WriteLine("    " + fullTypeName + 
                    "DataReaderListenerImpl(gcroot<EventManager<" + fullNetTypeNameSep + 
                    ">^> eventManager) : eventManager_(eventManager) {}");
                ddsImplHFile.WriteLine("    void Process(const " + fullTypeNameSep + 
                    " &sample) {");
                ddsImplHFile.WriteLine("        eventManager_->Process(eventManager_,");
                ddsImplHFile.WriteLine("            gcnew ProcessEventArgs<" + 
                    fullNetTypeNameSep + ">(Convert(sample)));");
                ddsImplHFile.WriteLine("    }");
                ddsImplHFile.WriteLine("    void AddHandler(gcroot<EventManager<" + 
                    fullNetTypeNameSep + ">::ProcessEventHandler^> handler) {");
                ddsImplHFile.WriteLine("        eventManager_->Process += handler;");
                ddsImplHFile.WriteLine("    }");
                ddsImplHFile.WriteLine("};");
                ddsImplHFile.WriteLine();
            }
        }
    }

The MessengerGenCS_CPP_DDSGen utility project executes DDSGen on MessengerGenCS_CS_Struct.dll to generate the corresponding IDL and C++/.NET wrapper. The IDL definition is compiled in the the MessengerGenCS_IDL project, and the C++/.NET wrapper in the MessengerGenCS_CPP_DDSImplLib project.

In Part I of this article, we created the wrapper, Messenger_CPP_DDSImplLib, by hand, where here we have created an identical wrapper, MessengerGenCS_CPP_DDSImplLib automatically, based only on the Message structure as defined in C# (in the MessengerGen_CS_CS_Structproject). To demonstrate that the behavior is identical to the hand-written version, the MessengerGenCS_CS_Publisher and MessengerGenCS_CS_Subscriber projects are identical to the Messenger_CS_Publisher and Messenger_CS_Subscriber projects we used in Part I, except that these use the automatically-generated IDL and wrapper instead of the hand-written ones. Executing the run_test_MessengerGenCS.pl test from the Test directory shows that the behavior is the same as before.

Thus, we have demonstrated that, with respect to the developer, C# alone can be used to implement the Messenger Developer's Guide example. The structure used for the OpenDDS data type is written in C#, as are the publisher and subscriber processes. Automatic code generation, coupled with a prewritten library, is sufficient for the developer to use only C# and not also to write code in C++ and IDL as well.

LANGUAGE OF CHOICE

We have shown that one need write only in C# to create an OpenDDS application, but other .NET languages work as well. Consider this structure written in Visual Basic.NET:

// MessengerGenVB_Struct/Struct.vb
Namespace Messenger
    <DCPSDataType()> _
    Structure Message
        Dim from As String
        Dim subject As String
        <DCPSKey()> Dim subject_id As Integer
        Dim text As String
        Dim count As Integer
    End Structure
End Namespace

We use the same DCPSDataType and DCPSKey attributes as before, but now in VB syntax. We now write the publisher in Visual Basic.NET:

// MessengerGenVB_Publisher/Publisher.vb
Module Publisher
 
    Sub Main()
        Dim dds As DDSNet = New DDSNet()
        dds.MessengerMessageWaitForSubscriber(42, "Movie Discussion List")
 
        For i As Integer = 0 To 9
            Dim messageNet As MessengerNet.MessageNet
            messageNet.subject_id = 99
            messageNet.from = "Comic Book Guy"
            messageNet.subject = "Review"
            messageNet.text = "Worst. Movie. Ever."
            messageNet.count = i
 
            dds.Publish(42, "Movie Discussion List", messageNet)
        Next
 
        dds.MessengerMessageWaitForAcknowledgements(42, "Movie Discussion List")
        dds.Dispose()
    End Sub
 
End Module

We also write the subscriber in Visual Basic.NET:

// MessengerGenVB_Subscriber/Subscriber.vb
Module Subscriber
 
    Public Class Print
        Public Sub MessengerNetMessageNetEventHandler(ByVal sender As Object, _
            ByVal args As ProcessEventArgs(Of MessengerNet.MessageNet))
            System.Console.WriteLine( _
                "MessageNetEventHandler:    subject = {0}", args.Sample.subject)
            System.Console.WriteLine( _
                "MessageNetEventHandler: subject_id = {0}", args.Sample.subject_id)
            System.Console.WriteLine( _
                "MessageNetEventHandler:       from = {0}", args.Sample.from)
            System.Console.WriteLine( _
                "MessageNetEventHandler:      count = {0}", args.Sample.count)
            System.Console.WriteLine( _
                 "MessageNetEventHandler:       text = {0}", args.Sample.text)
        End Sub
    End Class
 
 
    Sub Main()
        Dim dds As DDSNet = New DDSNet()
        dds.Subscribe(42, "Movie Discussion List", _
            New EventManager(Of MessengerNet.MessageNet).ProcessEventHandler( _
                AddressOf New Print().MessengerNetMessageNetEventHandler))
        dds.MessengerMessageWaitForPublisherToComplete(42, "Movie Discussion List")
        dds.Dispose()
    End Sub
 
End Module

Executing run_test_MessengerGenVB.pl from the Test directory shows that, once again, the output of the Visual Basic.NET version is the same as the others, and once again, the developer only needs to write in his or her language of choice, without needing knowledge of IDL or C++. Finally, demonstrating interoperability, the run_test_csvb.pl script executes the C# version of the publisher, but the Visual Basic.NET version of the subscriber, demonstrating that the output is as expected.

SUMMARY

As demonstrated in the article, one can remain in one's language of choice, and still make use of OpenDDS. Code written for the .NET Framework was used as an example, but the technique presented in this article can work in other situations. One needs a means to generate C++ wrapper code from an annotated structure description, and the means to use that wrapper from the language of choice (such as via a foreign language binding layer, or library linkage compatibility).

REFERENCES

[1] Code Generation with OpenDDS, Part I
http://mnb.ociweb.com/mnb/MiddlewareNewsBrief-201006.html

[2] OpenDDS
http://www.opendds.org/

[3] Attributes Tutorial
http://msdn.microsoft.com/en-us/library/aa288454%28VS.71%29.aspx

[4] Reflection Overview
http://msdn.microsoft.com/en-us/library/f7ykdhsy%28VS.80%29.aspx