Using Automated Tests to Document Software Architectures

Using Automated Tests to Document Software Architectures

By Jerry Overton, OCI Senior Software Engineer

February 2008

Introduction

Kent Beck's book, Extreme Programming Explained, suggests that the architecture of an application is better documented using automated tests rather than detailed specifications. The book briefly describes an example of documenting the required processing capacity of an application using tests. Improvements to the architecture of the application are made by updating the software to pass the tests. The example by Beck was straightforward, but it also lacked detail.

For more detailed software quality requirements, is it possible to write tests that can verify that an architecture satisfies these complex requirements? This article presents a case study designed to explore the viability of documenting more complex architectural requirements using automated tests.

The Case Study

The design of the case study for this article is taken from the book Essential Software Architecture by Ian Gorton. In the book, Gorton describes a design for the Information Capture and Dissemination Environment (ICDE). The ICDE is an application that automatically captures events as a result of various actions of workstation users. The information captured is made available to third-party tools interested in analyzing various aspects of user behavior.

This case study is based on an actual system and should provide the kind of complexity needed to better test the viability of automated tests as software architecture documentation. This article attempts to partially re-create the architectural solution given in the book, but to do so using automated tests rather than detailed specifications.

Augmenting Source Code Documentation

When comparing real examples of automated tests to the architectural solution described in Essential Software Architecture, it becomes clear that source code (written in Java) alone is not expressive enough to replace every aspect of an architecture specification. Even for an experienced software developer reading well-written code, it is often necessary to have additional background and context information to understand what the code does.

Java source code does not provide a good mechanism for documenting context or background for concerns that span classes located in several different source files. This documentation could be created as comments in the source files, but the information would have to be repeated for all applicable files or placed in a single location. Repeating the same documentation in multiple files makes the documentation much more difficult to maintain and placing the documentation in one place assumes that the reader will know where to find the information.

In this article, UML models are used to document context information that cannot be easily described using source code alone. In keeping with the agile style of automated testing, all models were created using Agile Modeling. Models were created using the simplest tools possible and contain only the elements necessary to communicate context. Once a model was created, it was updated only when absolutely necessary. The models are not meant to be exact representations of the code.

How to Read the Rest of this Article

The reaminder of this article is written as an architecture description of the ICDE application. The next section gives an overview of the entire architecture and each major section after that describes the solution for a specific architecturally significant scenario. All scenarios are introduced in the architecture overview, but for brevity, only one scenario is described in detail.

The fact that the Notify Event test class both extends JUnit TestCase and implements an interface from the architecture may look weird at first. The test was written according to the The Self-Shunting Unit Test Pattern. In this pattern, tests impersonate collaborators in order to find out things that only a collaborator could know. This trick was very useful for testing some of the more complex architectural requirements.

Start with the Architecture Overview to get an idea of the elements introduced and why. Read the Event Notification section for an explanation of how the elements are used to satisfy the quality requirements of that scenario. The Conclusion section summarizes the most important lessons learned by working through this exercise. A .zip file containing the full source code shown below is available for review.

Architecture Overview

The ICDE application records events generated by workstation users. Information from users is stored in the ICDE application and that information is available for query by third-party tools. Third-party tools can register to receive notifications of particular events. When those events occur, the system notifies registered listeners.

Overview

The overall design for the application uses a combination of the Presentation-Abstraction-Control Pattern (PAC) and the Observer Pattern. The user's state is modeled after the observer pattern and the third party tool is modeled after the PAC pattern. The listener class is the intersection between the two patterns and allows updates from the user to be communicated to third-party tools.

Overview3

Key to the architecture solution is the user state. The user state is responsible for maintaining a list of listeners and their associated interests. The user state updates listeners with new information as the state changes. Listeners are allowed to poll for all relevant updates or query for specific ones.

  1.         package userFramework;
  2.  
  3. 	public class AbstractUserState implements UserState{
  4. 		private HashMap<Integer, String> events = new HashMap<Integer, String>();
  5. 		private HashMap<StateListener, int[]> register = new HashMap<StateListener, int[]>();
  6.  
  7. 		public void update(int eventType, String event){
  8. 			events.put(eventType, event);
  9. 		}
  10. 		public String poll(StateListener listener){
  11. 			//return all events that the listener has registered for
  12. 			String message = "";
  13.  
  14. 			for (int i=0; i < register.get(listener).length; i++){
  15. 				message += events.get(register.get(listener)[i]);
  16. 			}
  17. 			return message;
  18. 		}
  19. 		public String query(int eventType){
  20. 			//return the events that match the given type
  21. 			return events.get(eventType);
  22. 		}
  23. 		public void register(StateListener listener, int[] events){
  24. 			register.put(listener, events);
  25. 		}
  26. 	}

For the third-party tool solution, the most architecturally significant class is the listener. The listener is registered with the user state and is responsible for keeping the user interface and local user view of the third-party tool up to date.

  1.         package thirdPartyToolFramework;
  2.  
  3. 	import userFramework.UserState;
  4.  
  5. 	public class AbstractStateListener implements StateListener{
  6.  
  7. 		private UserState userState;
  8. 		private UserView userView;
  9. 		private UI ui;
  10.  
  11. 		public AbstractStateListener
  12. 			(UserState state, UI userInterface, UserView view){
  13.  
  14. 			userState = state;
  15. 			ui = userInterface;
  16. 			userView = view;
  17. 		}
  18. 		public void update(){
  19. 			String state = userState.poll(this);
  20. 			userView.update(state);
  21. 			ui.setState(state);
  22. 		}
  23. 	}

Event Notification

When the user performs an action of interest on the workstation, the user state is updated accordingly. All interested listeners are responsible for polling for updates at regular intervals. If the user state has been updated with messages of interest to the listener, that information is returned to the listener when the listener polls. After the listener has received and update, the listener is responsible for updating the user interface of the third-party tool.

Event Notification

The quality requirements and satisfying architecture for the Event Notification scenario is documented by the NotifyEventTest class.

  1.         package architectureRules;
  2.  
  3. 	public class NotifyEventTest extends TestCase implements UserState {
  4. 		...
  5. 	}

Location Transparency

To encourage adoption by third-party developers, the ICDE API has to support location transparency for event notification. Third-party tools should not have to be coupled to a particular application distribution, nor should they have to rely on specific users for their updates. The system should allow the event generation mechanisms to be swapped out without disruption to the third-party application.

  1.         public class NotifyEventTest extends TestCase implements UserState {
  2. 		private NotifyEventTest servant;
  3. 		private StateListener listener;
  4. 		...
  5.  
  6. 		public void testLocationTransparency(){
  7. 			//the architecture must allow replacement of user
  8. 			//states to be transparent to the listeners
  9.  
  10. 			//create two versions of the same service
  11. 			NotifyEventTest servantA = new NotifyEventTest();
  12. 			NotifyEventTest servantB = new NotifyEventTest();
  13.  
  14. 			//use the test as a proxy and configure it with a servant. 
  15. 			//use the proxy to query data and poll for data
  16. 			servant = servantA;
  17. 			String firstPoll = this.poll(listener);
  18. 			String firstQuery = this.query(UserState.DEFAULT_EVENT);
  19.  
  20. 			//run the same test with a different servant
  21. 			//and compare the results
  22. 			servant = servantB;
  23. 			String secondPoll = this.poll(listener);
  24. 			String secondQuery = this.query(UserState.DEFAULT_EVENT);
  25.  
  26. 			assertTrue(firstPoll.equals(secondPoll));
  27. 			assertTrue(firstQuery.equals(secondQuery));
  28. 		}
  29.  
  30. 		...
  31.  
  32. 		public String poll(StateListener listener){
  33. 			return "Poll Successful";
  34. 		}
  35. 		public String query(int eventType){
  36. 			return "Query Successful";
  37. 		}
  38. 		public String proxyPoll(StateListener listener){
  39. 			return servant.poll(listener);
  40. 		}
  41. 		public String proxyQuery(int eventType){
  42. 			return servant.query(eventType);
  43. 		}
  44. 		public void register(StateListener listener, int[] events){}
  45. 		public void update(int eventType, String event){}
  46. 	}

Performance

Event notifications should be fast. Once an event notification is sent out, it should be received rapidly by interested third-party tools. Any mechanism responsible for disseminating information must do so without unnecessary delay. The event notification mechanism should provide sub-second message delivery performance.

  1.         public class NotifyEventTest extends TestCase implements UserState {
  2. 		private NotifyEventTest servant;
  3. 		private StateListener listener;
  4. 		private static long MAX_TIME = 100;
  5. 		...
  6.  
  7. 		public void testPerformance(){
  8. 			//event notifications have to be fast.
  9. 			//measure the time from when an event occurs until
  10. 			//the notification is received by a subscriber and
  11. 			//make sure that the time elapse is acceptable
  12. 			List<StateListener> listeners = new ArrayList
  13.     <StateListener>();
  14.  
  15.         UserState userState = new AbstractUserState();
  16.         listeners.add(newRegisteredListener(userState));
  17.  
  18.         long time = timeToUpdateAndNotifyEvent(userState, listeners);
  19.         assertTrue(time < MAX_TIME);
  20.         }
  21.  
  22.         ...
  23.  
  24.         private long timeToUpdateAndNotifyEvent
  25.         (UserState userState, List
  26.         <StateListener> listeners){
  27.  
  28.             //register all listeners with the given state
  29.             for (StateListener listener : listeners)
  30.             registerListener(userState, listener);
  31.  
  32.             //update the user state and measure how long it takes
  33.             //the listeners to get an update
  34.             return timeToUpdate(userState, listeners);
  35.             }
  36.             private StateListener newRegisteredListener(UserState userState){
  37.             UI userInterface = new AbstractUI();
  38.             UserView userView = new AbstractUserStateView();
  39.             AbstractStateListener listener =
  40.             new AbstractStateListener
  41.             (userState, userInterface, userView);
  42.             registerListener(userState, listener);
  43.  
  44.             return listener;
  45.             }
  46.             private void registerListener
  47.             (UserState userState, StateListener listener){
  48.  
  49.             int[] events = {UserState.DEFAULT_EVENT};
  50.             userState.register(listener, events);
  51.             }
  52.             private long timeToUpdate
  53.             (UserState userState, List
  54.             <StateListener> listeners){
  55.  
  56.                 long startTime = System.currentTimeMillis();
  57.                 userState.update(UserState.DEFAULT_EVENT, "Test Event");
  58.                 for (StateListener listener : listeners)
  59.                 userState.poll(listener);
  60.                 long endTime = System.currentTimeMillis();
  61.  
  62.                 return endTime - startTime;
  63.                 }
  64.  
  65.                 ...
  66.                 }

Scalability

The ICDE system must be capable of scaling to up to 150 concurrent users. A successful architecture will incur minimal performance degradation due to additional users. The performance of the event notification mechanism must be capable of keeping up with a growth in the number of users, up to the anticipated maximum.

  1.         public class NotifyEventTest extends TestCase implements UserState {
  2. 		private NotifyEventTest servant;
  3. 		private StateListener listener;
  4. 		private static long MAX_TIME = 100;
  5. 		private static int MAX_USERS = 150;
  6.  
  7. 		...
  8.  
  9. 		public void testScalability(){
  10. 			//event notification must scale to 100-150 users
  11. 			//the performance of the notification design should
  12. 			//scale linearly with the number of users up to the 
  13. 			//maximum expected capacity.
  14.  
  15. 			List<StateListener> listeners = new ArrayList
  16.     <StateListener>();
  17.         UserState userState = new AbstractUserState();
  18.  
  19.         for (int i=0; i < MAX_USERS; i++){
  20.         listeners.add(newRegisteredListener(userState));
  21.         }
  22.         long time = timeToUpdateAndNotifyEvent(userState, listeners);
  23.         assertTrue(time < MAX_TIME * MAX_USERS);
  24.         }
  25.  
  26.         ...
  27.  
  28.         }

Conclusions

With help from a few agile models and some prose, automated tests were capable of documenting some fairly detailed software architecture requirements like location transparency and scalability. Although it remains to be seen whether or not tests are sufficient for other software qualities, the approach seems viable so far.

Using tests to document architectural requirements helped reduce the tendency to over-engineer a solution. The best solution is the simplest solution that passes the test. Any complexity beyond that is unnecessary. The tests provide an objective measure for minimum simplicity required. Tests also make it obvious where the solution's risks are. Any tests failing indicate qualities that are at risk.

Tests make it much easier to estimate the cost of deferring a design decision. When architecture is documented using tests, the cost of changing a design later will be roughly proportional to the cost of the refactoring required to make the change. Without tests, the true cost of a design change is hidden by how easy it is to update models and descriptions.

The certainty that comes with documenting architecture using tests comes with a price. Generally, it takes longer to write a test (and code that could pass the test) for a software quality than it does to create UML depictions and explanatory text. However, it is not clear if time saved with less formal documentation would result in an increase in re-work due to missed requirements later.

Also, it is not clear to what limit this technique can be extended. The example presented in this article is more detailed than the example given in Extreme Programming Explained, but certainly does not represent the most complex and detailed example possible.

References