R. Weinreich, ObjectWire – A Framework-based Platform for Distributed Object-Oriented Applications Technical Report, CD-Laboratory for Software Engineering, Johannes Kepler University of Linz, 1996.
ObjectWire – Frameworks for Distributed Object-Oriented Applications
1995 – 1997
Contents
Motivation
Architecture Overview
Communication
Message Dispatching
Proxy-based Communication
Configuration
Monitoring
Event Generation
Event Pre-Processing and Distribution
Configuring the Monitoring Environment
Related Publications
Distributed computing introduces a vast amount of additional complexity in contrast to local computing. Distributed application engineers need support in dealing with this complexity in terms of high-level easy-to-use interfaces and services. Therefore standardization efforts for object-based distributed computing like CORBA concentrate on providing mechanisms for transparent remote object access (RPC) and on specifying high-level interfaces for required services. The specification of high-level interfaces enables mainly two things: (1) a certain amount of transparency, which shields programmers from low-level communication interfaces and (2) the standardization of some common interfaces to software services, which encourages vendors to code to these standardized interfaces instead of creating island solutions (which is important for creating a component market). It is important to note that the specification of high-level interfaces does not provide interoperability per se. It is also important to agree on low-level wire protocols, at least between different platforms (like IIOP for CORBA), and to provide bridges for protocol translations.
The concentration on high-level RPC interfaces also has some drawbacks. One is that the decisions made in implementing a particular communication infrastructure may have significant influence on its usability. The implementation of communication buffer management, of parameter marshaling, the kind of the underlying transport protocol and other “implementation details” may significantly affect performance and reliability and thus require changes in the architecture of a system. Thus it is important to give the client control over implementation strategies. Another problem is that a high-level RPC is not well suited for all kinds of applications. For example, transmitting large quantities of data or continous data streams are not well supported by an RPC primitive. Thus clients need communication interfaces on different levels of abstraction that allow either low-level fine-grained control or high-level procedure call transparency and ease of use.
An issue that is especially important (and often underestimated) in a distributed environment is configuration and monitoring support. The focus of monitoring and analyzing dynamic program behavior is not only on finding programming errors, which are more diverse and harder to detect than in non-distributed systems, but also on identifying performance problems due to incorrect structure, placement and configuration, and on detecting stability weaknesses that are the result of ignoring or neglecting error management. In addition, dynamic program analysis is a valuable aid in understanding the system in the maintenance phase.
Figure 1 shows an overview of the basic components of the ObjectWire platform. ObjectWire consists of frameworks that are organized in layers and offer communication services on different levels of abstraction. The individual layers are decoupled using object-oriented techniques to facilitate the independent usage of services as far as possible. The five basic frameworks that are parts of ObjectWire are: an IPC Services framework, a basic communication framework (which includes monitoring support), a configuration framework, a distributed architecture framework, and an object-sharing framework.
Figure 1: ObjectWire Overview
The lowest layer is the IPC Services framework , which offers IPC services like shared memory, message queues, sockets and TLI that can be used for transmitting byte streams. The framework includes object-oriented C++ wrappers for these services and provides one common abstract service interface. This interface can be used to substitute one service for another, although one has to be aware that the semantics of the underlying protocol (e.g., TCP or UDP) may influence communication behavior.
The basic communication framework is based on these low-level services and supports the transmission of self-describing typed message objects. Each message may contain arbitrary other objects that are passed along with the message. Messages (and object parameters) with unknown types are ignored by the receiver unless the code implementing the type can be loaded dynamically. The framework also supports an optional dynamic remote method invocation interface (DII) that is based on the typed message interface (TMI). A remote method call is a special message object of type RMCMsg that supports construction of remote method invocations at run time (including parameter marshaling). The basic communication framework supports synchronous, deferred synchronous and asynchronous communication, as well as time-outs, priorities and various message filters. The design of this framework allows exchanging the currently used communication service at run time. The framework also contains basic interfaces and services to support tools for dynamic program analysis (i.e., monitoring).
The configuration framework supports system configuration on the basis of coarse-grained distribution only (although the basic communication layer supports remote method calls to individual objects within a process). This means that this layer provides proxies for remote objects that usually represent application subsystems as the units of distribution. A central configuration manager maintains all configuration information and can be used to make queries about the current configuration and to change the configuration dynamically. The configuration manager works on an initial configuration supplied as a readable configuration file. Group communication (multicasting) is also supported at this layer by means of group proxies. The configuration framework provides location independence, supplies means for tuning the system, can be used to enhance flexibility (by allowing queries on the system structure) and provides information that is needed by system management tools.
The most specialized frameworks are the distributed architecture framework and the object sharing management framework. The former provides classes implementing specialized interaction protocols or client and server types. An example for a specialized protocol is the 2-phase-commit protocol that is used for the simple atomic multicast provided by our framework. The object sharing management framework was developed for the implementation of a cooperative software development environment. It provides a replication mechanism that allows clients to hold local copies of all server objects they need during execution of their work. Changes to a replicated object structure are propagated to the server and to other clients managing replicas. The problem of synchronizing replicas is not automatically supported and has to be handled by defining a clear cooperation model on the application level..
As described above, communication may be performed on different levels of abstraction. Communication at the basic communication layer is performed using typed message objects. The following example illustrates the usage of such objects.
IPCMsg *msg= new ConfigMsg(…);
msg->SetDestination(dest, objID);
msg->Execute(); //— or msg->ExecuteSync();
…
retVal= msg->GetReturnValue();
A message is an arbitrary subclass of the class IPCMsg. After creation of a message, the destination may be set using a destination specifier and an optional object identifier. At this level of abstraction the location has to be specified. The request may be directed to a specific object at the location using the optional parameter objID. If this parameter is not used, the message is sent to the destination and delivered to any object expressing its interest in messages of this type. We call such a message undirected. If the message cannot be handled because an appropriate object cannot be found, a message-not-understood error is returned. (For a more detailed description of message handling on the server side see below).
By telling the message request to execute itself, the message is packed and sent to the destination. The method is sent asynchronously or deferred synchronously if its method Execute is called. It is sent synchronously with the sender blocking until the request has completed if the method ExecuteSync is used. The method GetReturnValue can be used to retrieve a return value object. It is possible to specify additional parameters like ID, priority and time-outs (the latter in case of a synchronous send).
Remote method calls are implemented by a class RMCMsg, which is a subclass of the class IPCMsg. An RMCMsg is created and sent like any other typed message, but offers a generic interface to compose a request at run time. In addition to general methods like AddParameter, this class also offers a simplified interface that is based on overloading the C++ stream operators as illustrated in the following:
RMCMsg *getConfig= new RMCMsg(dest, objID);
getConfig << attributeName << value;
getConfig->ExecuteSync();
getConfig >> …It is optionally possible to check the parameter types at run time.
On the receiver side messages can be read directly from a message queue. Filters can be used to select incoming messages by type, ID, priority, etc. Since filters are themselves based on an abstract filter class, arbitrary filter criteria may be specified by defining new filter classes. In addition, filters can be composed using a composite filter.
A server may also install a dispatch manager for a message queue. This manager maintains plugable dispatchers for specific message types (or for object types in the case of RMC messages) and is responsible for finding an appropriate dispatcher for each incoming message. A dispatcher may handle the message directly (dispatchers of this kind are called handlers in ObjectWire) or dispatch the request to a registered object. The process of handling an incoming message using the dispatch manager is illustrated in Figure 2.
Figure 2: Message dispatching using plugable message handlers and RMC dispatchers
The server depicted in Figure 2 contains a number of objects of type Person. If such an object is to be accessed by a remote method call, it first has to register itself using an object ID at the dispatch manager (step 1). If the server receives a message (step 2), it is forwarded to the dispatch manager. The dispatch manager searches for a message handler that is able to handle messages that are of the received message’s type. In the case of the example depicted above, the message is of type RMCMsg and is forwarded to the RMC message handler, which is a generic handler for all remote calls to methods of specific objects in the server process. The RMC handler itself maintains RMC dispatchers for each object type (not message type!) and uses a type specific RMC dispatcher for dispatching a request to the registered object. In the example above the RMC handler would use a RMC dispatcher for objects of type Person.
RMC dispatchers of ObjectWire are similar to object skeletons in CORBA and to interface- or server-side stubs in DCOM. The architecture at the level of the RMC handler is in fact very similar (but not identical) to DCOM’s standard marshaling architecture.
The presented dispatching architecture also supports system evolution. As already mentioned above, message handlers and RMC dispatchers are provided as plug-in components and selected dynamically at run time. If the server is extended by a new subsystem, this subsystem may register a new message handler for remote messages that are specific to this new program part. If the subsystem itself is designed as plugable component, the existing server code neither has to be changed nor is recompilation necessary. We use this technique for attaching monitoring related components to applications, which can be controlled remotely (and completely transparent for the host application) using monitoring messages. Likewise new RMC dispatchers can be installed for new object types in the server application.
If a new server is installed on a system with old clients (i.e., with clients expecting an old server), the new server may provide special message handlers for old clients that act as adapters and perform necessary transformations. If such an adapter is not installed and a new server is thus not able to perform a request issued by an old client, the client will receive a message-not-understood-error. If an new client (expecting a new server) is installed on a system with an old server, it may be possible to install new dispatchers and handlers in the old server that are able to perform the necessary adaptation tasks.
Finally, generic message handlers for message forwarding can also be provided. This is similar to the dynamic skeleton interface of CORBA based environments.
Communication is usually performed using proxies that are provided at the configuration layer, even if only the dynamic invocation interface as illustrated above is used. Each proxy offers a method HandleMsg to send arbitrary messages to the object represented by the proxy. In this case setting of location and object ID is performed by the proxy and is transparent to the user.
To summarize, requests are typed messages that can be sent asynchronously, synchronously, or deferred synchronously. RMCs are just generic message-objects that can be mixed with any other messages. All message objects, including RMCs, can be sent directed or undirected, can have priorities, can be arbitrarily filtered, etc. We have also implemented a static invocation interface (see [24]) that is built on RMC objects.
We use a combination of static and dynamic system configuration. The configuration may be defined statically in a configuration file that contains descriptions of applications and associated attributes, placement and groups. Figure 2 shows an excerpt from a configuration file used by a cooperative software development environment that was implemented using the framework.
The configuration file is interpreted by a central configuration manager that creates and maintains proxies for applications and groups defined in the configuration file. Applications may use the configuration manager at run-time to request proxies to other system members or to groups. Group proxies may be used for issuing a multicast to the members of the group.
|
NODE hilo.swe.uni-linz.ac.at { NODE genf.swe.uni-linz.ac.at { …. GROUP coopmanagers { |
The configuration may be changed at run time by sending requests to the Configuration Manager. Additional proxies and groups can be added to the configuration (register), existing ones can be removed (unregister), and attributes specified for a proxy can be changed (update).
In addition, the configuration manager offers a simple query interface (get) that can be used to retrieve proxies based on specified attributes. The result of a query is a group proxy that can be used to issue a multicast to all applications matching the query. Group proxies that are registered with the configuration manager are updated at the clients end when a member enters or leaves the group.
The configuration layer has a number of important purposes. It provides location independence by offering the possibility to assign logical names to coarse-grained objects. It allows tuning of the system by providing options to change system parameters such as the IPC service and communication protocol. It can be used to enhance flexibility by allowing queries on the system structure. It offers basic services for group communication and management. Finally, the configuration layer provides information that is needed by system management tools that are also being developed as part of the ObjectWire framework.
The architecture of ObjectWire supports the generation of analysis information at various levels of abstraction and provides an adaptable monitoring front end for forwarding events to the appropriate analysis tools. We have developed a a few small independent tools for interpreting and visualizing the generated event information. A larger environment, called OrWell, is currently under construction and aims at combining the analysis of various application specific aspects in a well integrated environment.
The monitoring front end consists of a set of predefined event classes (see Figure 3) and an abstract interface to event-notifier components. Usually events are generated automatically at various places in the different layers of the framework. For example, events are generated when a message is sent or received or when a group multicast is issued, but also when a message is read from the mailbox and processed, or when the appropriate class-code for a parameter of a received message cannot be found, and so on.
Figure 3: Event Class Hierarchy
For some events, however, the appropriate event class is provided, but the user is responsible for event generation. User-defined event generation includes two steps: (1) creating and initializing an object of a specific event class and (2) triggering this event using the high level interface provided by the monitoring framework. The latter is simply a call to a method IPCMonitor::MonitorEvt(evt), which takes care of setting further event attributes and informing all attached monitors.
An example where user-generated events are needed, is the analysis of error management. Error-events and error-handling events can be used to determine that some failure occurred and to analyze whether or how it was handled. If some kind of error is encountered, an error event containing the description of the error is generated and forwarded to the analysis tools. Events for acknowledging the proper handling of an error are also provided, but the user is responsible for generating and parametrizing them correctly.
In addition to predefined events, a user may define arbitrary additional events by defining subclasses of the predefined event class OWEvent. In this case, the user is also responsible for issuing these events at the correct places.
Monitoring (Event Pre-Processing and Distribution)
Events that are generated have to be forwarded to one or more analysis tools for visualization and semantic interpretation. This is performed by event notifier plug-ins (ENPs) that may be attached to the components of a distributed application dynamically (see Figure 4). The observed components are completely decoupled from the attached ENPs and know nothing about the location, the kind and or number of the tools to which the generated events are sent.
Figure 4: Plugable Event Notifiers
Based on their built-in logic, we distinguish different kinds of ENPs. A simple ENP just forwards each event to an analysis tool. Another kind of ENP may act as a plug-in filter that only forwards events concerning a specific aspect to be monitored (e.g., error handling). But an ENP may also build high-level events from lower-level ones and just forward the high-level event. Preprocessing events with plug-ins reduces not only the number of events that have to be forwarded to the analysis tools but also allows configuring the monitoring environment on the side of the observed components by means of different ENPs.
ENPs can be added to or removed from event sources at run time and in arbitrary numbers. This makes it possible to install ENPs for different monitoring tools at the same time. In addition, it allows attaching multiple ENPs that report to one analysis tool but implement different preprocessing strategies.
In order to support event-ordering based on causal relationships it is necessary to attach monitoring-specific information (e.g., timestamp vectors) to messages sent between the components of the observed application. The message format of ObjectWire supports piggybacking an arbitrary number of objects representing monitoring-specific information. The information is attached transparently to all messages by event notifier plug-ins that are provided for a specific analysis tool.
Configuring the Monitoring Environment
As described above, ENPs can be attached to the components of a distributed application to be observed dynamically by the analysis tool. Depending on the aspect to be monitored, different ENPs may be used by the analysis tools to preprocess the generated events. Still, the major task of filtering and semantic interpretation has to be done by the analysis tool itself. The current implementation lacks a mechanism to transport ENPs over the network. Although this is basically possible with ObjectWire, we have not implemented this feature yet. It is conceivable that environments like Java that are designed for transmitting code across the network support such an architecture well.