EnkiDesign

From Eigenpedia

Jump to: navigation, search

Contents

Overview

Enki provides build- and run-time support for persistent storage of objects based on the Meta Object Facility (MOF), Java Metadata Interface (JMI), and XML Metadata Interchange (XMI) specifications.

Enki introduces the notion of a metadata repository provider ("provider"), which represents the build- and run-time components necessary to persist objects to a particular storage model. For example, Enki initially provides a Netbeans MDR provider, which wraps and slightly extends the functionality of the old Netbeans MDR libraries and a Hibernate provider, which uses the Hibernate project to store objects in an RDBMS. In both cases, the provider's build-time code generation is used to produce Java code which is then compiled and makes use of the provider's run-time infrastructure to persist objects.

Build-time support is provided through Enki-specific Ant tasks. Note that Enki's code generation framework is designed to be extended without having to introduce a new provider (see #Code Generation). A project could provide a code generation that emits model-based code (or other files). In fact, Enki actually contains code generators that produce information about a metamodel's classes and attributes or produce a .dot format graph file of a metamodel's inheritance hierarchy.

Run-time support is provider-specific and is accessed via a factory class.

Infrastructure

Enki's provides a generic infrastructure for MDR providers.

Code Generation

The top-level construct in Enki's code generation infrastructure is the org.eigenbase.enki.codegen.Generator. A Generator is responsible for loading a MOF metamodel, typically via XMI, and recursively visiting each model element. For each model element, the Generator invokes one or more org.eigenbase.enki.codegen.Handler instances which are responsible for generating the desired output. Generators should ignore values in the generic options map that they do not recognize.

NOTE: As it stands, the Generator interface contains a great deal of JMI-specific code, which should be refactored into a JMI code generation utility class.

The abstract base class org.eigenbase.enki.codegen.GeneratorBase is provided to implement the majority of the Generator interface. Concrete implementations instantiate one or more handlers and set the handler's output directory, generic options (via the options map). org.eigenbase.enki.codegen.MdrGenerator is an abstract base class that automatically instantiates a Netbeans MDR repository, imports an XMI model into it, and invokes its configured handlers.

Handlers are sub-classed for each type of metamodel object: Class proxy, class instance, association, package, enumeration class, enumeration interface, exception, and structure. If a particular Handler does not implement, say org.eigenbase.enki.codegen.AssociationHandler, GeneratorBase will simply ignore associations.

Handlers may specify that they required multiple passes over the metamodel. Each Handler is notified at the start of generation, the start of each pass, the end of each pass, and the end of generation. Handlers may produce many files (e.g., one per metamodel element), a single file (e.g., one for the entire metamodel), or something in between. The abstract base class org.eigenbase.enki.codegen.HandlerBase provides support for generating a text file with varying levels of indentation.


Ant Tasks

Enki provides an Ant task, org.eigenbase.enki.ant.EnkiTask, which accepts MDR provider configuration information and provides several subtasks to perform useful build-time functions.

Code Generation

The org.eigenbase.enki.ant.MapJavaSubTask takes the following parameters invokes a subclass of MdrGenerator, optionally passing a set of name/value pair options, which are specific to the generator.

--Jvs 15:13, 30 June 2008 (PDT): "takes the following parameters...?"

The MapJava subtask cannot be combined with other subtasks, since it always uses Netbeans MDR to load a metamodel and perform code generation. It ignores any provider configuration information provided at the EnkiTask level.

Create & Drop Extent

The org.eigenbase.enki.ant.CreateExtentSubTask and org.eigenbase.enki.ant.DropExtentSubTask create and drop metamodel or model extents. They take provider configuration from the enclosing EnkiTask.

Import & Export XMI & DTD

The org.eigenbase.enki.ant.ImportXmiSubTask and org.eigenbase.enki.ant.ExportXmiSubTask import and export metamodel or model extents to XMI files. They make use of the Netbeans MDR library's XMI reader and writer.

The org.eigenbase.enki.ant.WriteDtdSubTask generates a DTD that represents the format of the XMI documents produced and accepts by the import and export sub tasks for a specific metamodel.

These tasks take their provider configuration from the enclosing EnkiTask.

Print Extent Names

Prints the names of the extents currently instantiated in the configured repository. This task takes its provider configuration from the enclosing EnkiTask.

Common Enki Task Configuration

Provider Configuration

Most of the sub tasks obtain their provider configuration from the enclosing EnkiTask (with MapJavaSubTask being the exception).

The enclosing EnkiTask accepts provider configuration information in two ways:

  1. Primarily, properties are passed via a repository storage properties file. This file contains, at a minimum, a property called org.eigenbase.enki.implementationType which specifies the name of the Enki provider to use. All other properties in the file are passed to the provider.
  2. In addition to, or instead of, a properties file, configuration properties can be passed using nested StorageProperty elements. For example,
<enki.task>
  <storageProperty name="foo" value="bar"/>
</enki.task>

Storage properties are scanned for variable references, using the same format as Ant (e.g., "${var.name}"). Variable values are obtained from a nested PropertySet element. For example,

<enki.task>
  <propertyset>
    <propertyref name="CATALOG_DIR"/>
  <propertyset/>
<enki.task/>

Logging

The Enki task takes a java.util.logging configuration properties file to allow log output from Enki operations to be diverted to a file (by default it will appear on the standard output).

Model Code Classpath Reference

The Enki task takes a reference to a classpath containing generated metamodel code. If this option is not used, the Enki task must typically be invoked from a forked Ant instance in order to pick up JAR files and Java class files generated earlier in the script.

Backwards Compatibility with Netbeans MDR

The Netbeans MDR Ant task assumed that its storage properties were passed as Java system properties.

The Enki Ant task provides backwards compatibility by assuming that the properties are typically stored in a file and that Netbeans MDR tasks are typically executed in a forked Ant script. The storage properties file is may be passed as a system property called enki.compatibility.storagePropertiesFile. In addition, a single variable can be substituted in those properties via a pair of system properties called enki.compatibility.substPropertyName and enki.compatibility.substPropertyValue. These properties only come into play when no other storage properties are specified. Thus, Enki-aware build environments can specify properties explicitly via the PropertyFile attribute (or StorageProperty elements):

<enki.task propertiesFile="xyz.properties">
  ...
<enki.task/>

while old-style build environments needs only have their storage properties file specified as a system property in the Ant script that forks MDR task execution.

MOF Metamodel

Enki contains an internal implementation of the MOF metamodel, which can be used by Enki providers to implement the MOF extent (wherein MOF is described as a MOF metamodel) and to implement metamodel extents.

The built-in MOF metamodel was generated using Enki's code generation infrastructure (org.eigenbase.enki.codegen.MofGenerator and org.eigenbase.enki.codegen.MofHandler).

Enki/Hibernate

The Enki/Hibernate provider name, for the purposes of configuration, is ENKI_HIBERNATE.

Hibernate Mapping & Code Generation

Enki/Hibernate code generation controlled by org.eigenbase.enki.hibernate.code.HibernateGenerator, which instantiates several Handler implementations:

  1. a JmiTemplateHandler to produce the JMI template interfaces
  2. a HibernateJavaHandler to produce implementations of the JMI interfaces and metamodel-specific subclasses of some Enki/Hibernate run-time classes
  3. a HibernateMappingHandler to produce a Hibernate mapping configuration file and an Enki model properties file
  4. a HibernateMofInitHandler to produce an initializer for the built-in MOF metamodel implementation

Polymorphism

Hibernate provides several ways to model polymorphism in a database schema. After experimenting with a prototype, we settled on table-per-concrete-subclass. Each non-abstract metamodel class is represented as a table which contains a column for each of its attributes, references, and associations, including those defined by super classes.

Simple Attributes

Simple attributes are attributes with types from the PrimitiveTypes MOF package with an upper multiplicity bound of 1.

For each simple, mutable attribute, two pairs of methods are generated in the class instance implementation. The first pair (interface methods) implement the getter and setter methods defined in the class instance interface. The interface methods generate MDR attribute events as necessary, and then invoke the second pair of methods (Hibernate methods). The Hibernate methods are referenced in the Hibernate mapping file and always operate on object references. That is, the interface methods may specify int, but the Hibernate methods use java.lang.Integer. This allows Enki/Hibernate to detect unset attributes even when the class instance interface calls for a primitive type. It also allows Hibernate to invoke its getter and setter methods to persist and restore objects without generating spurious MDR attribute events.

Collection Attributes

For attributes with an upper multiplicity bound greater than 1, there is no setter interface method. The getter method wraps the underlying attribute in org.eigenbase.enki.hibernate.storage.AttributeCollectionWrapper or org.eigenbase.enki.hibernate.storage.AttributeListWrapper, as appropriate.

Entity Attributes

Enki/Hibernate treats entity attributes (attributes whose type is a metamodel class) as associations with special deletion and event semantics. See #Composition.

Associations and References

Associations are modeled in Enki/Hibernate as one of three basic types:

  1. one-to-one - org.eigenbase.enki.hibernate.storage.HibernateOneToOneAssociation
  2. one-to-many - org.eigenbase.enki.hibernate.storage.HibernateOneToManyAssociation
  3. many-to-many - org.eigenbase.enki.hibernate.storage.HibernateManyToManyAssociation

Each association end that a metamodel class participates in is represented as a persistent instance of one of those types. The instance is stored as a property of the class instance implementation. A getter and setter method is generated for Hibernate to use in persisting and loading the class instance implementation. In addition, generic methods are generated for each class instance implementation that allow access to the association objects via the association end's name.

Enki/Hibernate provides abstract implementations of the RefAssociation interface for each association type that use the generic class instance methods to obtain and manipulate the association object corresponding to a particular association end for a particular RefObject instance. A concrete RefAssociation implementation is generated that configures the abstract base class with the correct end names.

References are a special case of associations. In addition to the code generated for associations, getter methods are generated to allow access to the referenced end of an association as a collection or list when the upper multiplicity bound of the referenced end is not 1. These methods instantiate a org.eigenbase.enki.hibernate.storage.ListProxy instance configured with the class instance and association end information. ListProxy uses the generic methods described above to manipulate the association. Single-value getter and setter methods are generated to manipulate the referenced end of an association when its upper multiplicity bound is 1. Note that this means that even un-ordered associations are stored as ordered associations. (--Jvs 13:31, 30 April 2008 (PDT): I think this is no longer true, right?)

The RefAssociation base classes and ListProxy are responsible for firing MDR association events.

Generic Storage

Because association ends may refer to types with many sub-types, and because plugin models may introduce new sub-types of a type already participating in an association, Enki/Hibernate provides a custom type mapping class that maps a string to the concrete metamodel type it represents. The mapping is configured at run-time as metamodels and their plugins are loaded. Therefore, in the tables that store associations, each object is referenced using two columns: one to store the type name and one to store the object's MOF identifier.

One-to-one associations are stored in a single table with a "parent" and "child" types and identifiers. The parent refers to the first end of the association and the child the second end.

One-to-many associations are stored in two tables. The first table contains the "parent" type and identifier. In this case, the parent always refers to the end of the association with multiplicity 1, regardless of whether it is the first or second end of the association. The second table refers to the first table and contains the types and identifiers of the "many" end of the association. The first table contains a flag to indicate whether the association has been reversed. If true, it means that the "parent" object is actually the second end of the association.

Similar to one-to-many associations, many-to-many associations are also stored in two tables. "Parent" and "child" are referred to as "source" and "target." Also, each class instance refers to a separate association instance. So if A1 is in a many-to-many association with B1 and B2, the following many-to-many association instances exist:

  1. A1 as source with B1 and B2 as targets
  2. B1 as source with A1 as a target, with the reversed flag set
  3. B2 as source with A1 as a target, with the reversed flag set

Enki/Hibernate automatically updates the association instances bi-directionally when modifications are made.

Lazy Associations

Enki/Hibernate provides for a special type of lazy association whereby Hibernate itself is not aware of the actual associations between objects. This allows very precise control of when and how objects are loaded as the association is traversed. In particular is allows association information to be loaded in a single SQL statement via a join. Subsequent access to the RefObject instances referenced by the association is then batched by the concrete type of the referenced objects.

Lazy associations are useful for large associations and particularly for associations that are used infrequently. Because Hibernate is unaware of the relationship between the association instances and the represented objects, it will not prematurely traverse an association to load an object. This can significantly reduce the number of objects loaded during a repository transaction.

Composition

Composition is modeled as a special case of association in Enki/Hibernate. When one metamodel class is composed of an attribute whose type is another metamodel class, an implicit association is generated. The implicit association operates identically to a normal association except that

  1. collections and lists are proxied by org.eigenbase.enki.hibernate.storage.AttributeListProxy, which fires MDR attribute events instead of association events, and
  2. when a composed class instance is deleted, it automatically deletes its composite class instances.

Note that RefAssociation instances are not generated for the implicit association used to model composite attributes.

Enumerations

Enumerations are stored as strings via a custom type mapping using Hibernate type definitions. The custom type mapping is used to convert instances of the enumeration into strings for storage in the database and to convert those strings back into enumeration class instances when loaded from the database.

Clustered Package Imports

Enki supports metamodels which make use of MOF's clustered package import feature for allowing the same package to appear as the child of multiple parent packages. Enki implements this by arbitrarily designating one such parent as the primary, and the rest as aliases. The implementation for the containment from the primary is the same as that used for composite package containment; the aliased containment is implementing by looking up a reference to the imported package via the unique containment path through primaries.

MOF ID Generation

Enki/Hibernate uses a table to store the next available block of MOF identifiers. By default the identifiers are grouped in blocks of 100. When a particular repository operating against a database schema (perhaps at the same time as another repository) requires a new MOF ID, it grabs the currently block and increments the next available block entry to the next valid value. This technique was adopted from Hibernate's built-in key generators, but provides keys that are unique across the schema.

Enki/Hiberate stores MOF identifiers as 64-bit integers. MOF identifiers with the bit 0x4000 0000 0000 0000 set are used internally.

Metamodel Storage

Enki/Hibernate does not store the metamodel itself in the database. Instead, Enki's builtin MOF metamodel is used to store the metamodel. The metamodel is initialized by a metamodel initializer class generated by org.eigenbase.enki.hibernate.codegen.HibernateMofInitHandler. When an Enki/Hibernate repository is instantiated, it determines which, if any, models and/or metamodels are already stored in the repository. Stored metamodels are automatically initialized by invoking the generated initializer.

Queries

Enki/Hibernate uses Hibernate queries for some operations.

refAllOfClass and refAllOfType

The Hibernate mapping for each concrete metamodel class contains a query named allOfClass which returns all instances of that exact type by querying the table which stores instances of the exact type.

For each metamodel class (concrete or otherwise), there is a query named "allOfType" and prefixed with the class interface name. These queries return via Hibernate's polymorphic query all instances of the class interface. This requires querying each table which stores sub types of the queried type.

getByMofId

To avoid systematically querying every concrete metamodel class table in the implementation of the MDR API method getByMofId, Enki/Hibernate stores a map of each instance's MOF identifier to its concrete type. When getByMofId is invoked, the type associated with the MOF id is looked up via a Hibernate query, and then the correct table is queried for the instance object.

In addition, Enki/Hibernate is capable of skipping the type lookup via the MDR API extension that looks up a particular type of RefObject by MOF ID.

findAllOfClass and findAllOfType

Enki/Hibernate uses a criteria query to implement these MDR API extension methods.

Schema Generation and Validation

Whenever a model extent is created, Enki/Hibernate automatically validates the database schema to determine whether it matches the schema required by the model. If validation succeeds, the schema is used without modification. If validation fails, Enki/Hiberate attempts to update the schema to the version required by the metamodel.

Multiple Metamodels Per Schema

The HibernateGenerator provides an option for specifying a prefix to use for all tables in the metamodel. This allows multiple models, perhaps with similarly named classes, to reside concurrently in the same schema. The models may be loaded by a single Enki/Hibernate repository or by multiple repositories.

TODO: Resolve issues related to duplicate metamodel initialization.

Configuration

Enki/Hibernate provides several metamodel tags which can be used to control the behavior of its code generation.

Enki/Hibernate Tags
Tag Identifier Description
org.eigenbase.enki.maxLength Controls the maximum length for a string attribute. It may be applied to a simple string attribute, a string collection attribute, or to an entire class. Values of the tag applied to an attribute take precedence over the class's value even if the attribute is inherited from a supertype. Values of the tag applied to a class apply to all to all untagged string attributes in the class. Subclasses inherit the value. The value must be an integer greater than 0 or the special value "unlimited." Lengths greater than 32768 use the Hibernate "text" type.
org.eigenbase.enki.transientPackage Controls whether a package is considered transient (not persisted to database). If this tag appears on a package, with any value, the package and its contents (including other packages) are considered transient.
org.eigenbase.enki.lazyAssociation Controls whether an association is loaded via Enki's lazy association mechanism. This differs from normal Hibernate laziness and is useful for especially large collections.

Extents

Extents are tracked by Enki/Hibernate via a table which contains the extent name and the metamodel that describes it. Extents that contain metamodels are stored in the same table and refer to the MOF metamodel as their descriptor.

Sessions and Transactions

Enki/Hibernate requires explicit session management to allow Hibernate to perform optimally and to allow callers to re-use repository objects across transaction boundaries. In addition, Enki/Hibernate provides a mechanism to detach a session from the current thread and reattach it to another thread without disrupting the session or any ongoing transactions.

Enki/Hibernate does not support implicit write transactions.

Enki/Hibernate can be configured to implicitly begin a session if one does not already exist when a transaction is started.

Packaging

Enki/Hibernate produces Java interface code for the metamodel, implementations of the interfaces, a Hibernate mapping configuration file to map the the implementation classes to a database schema, a MOF metamodel initializer and a properties file to describe the metamodel contents to Enki/Hibernate at run-time.

Enki/Hibernate searches the available model properties file (those that are on the classpath when the repository is instantiated) and automatically loads the model descriptions.

Plugin Models

Plugin models extend another model by providing additional interfaces and classes (which may extend the base model's interfaces), a Hibernate mapping for the plugin classes, a MOF metamodel initializer that adds to the initialization performed by the base model, and a properties file to describe the plugin model's contents to Enki/Hibernate at run-time.


Run-time Configuration

The following properties may be specified in the repository storage properties for an Enki/Hibernate repository:

Enki/Hibernate Repository Storage Properties
Property Name Description
org.eigenbase.enki.hibernate.allowImplicitSessions Controls whether or not implicit sessions are allowed. Defaults to false.
org.eigenbase.enki.hibernate.trackSessions Controls whether session begin/end pairs are tracked with a unique identifier. Useful for determining where a particular session begins and ends when sessions are unexpectedly nested. Note that the generated identifiers are only unique within a repository instance. Defaults to false.
org.eigenbase.enki.hibernate.typeLookupFlushSize Controls whether or not and how frequently insertions into the MOF ID/type lookup table are flushed. Defaults to the value of the hibernate.jdbc.batch_size property.
hibernate.* All properties are passed to Hibernate's Configuration without modification. Note that the property hibernate.default.batch_fetch_size controls batch fetch size for lazy associations.

Enki/Netbeans

The Enki/Netbeans provider name, for the purposes of configuration, is NETBEANS_MDR.

Code Generation

Enki/Netbeans (org.eigenbase.enki.netbeans) provides a code generator that produces a subset of the interfaces and classes described by the JMI specification. Interfaces are produced for all packages, classes, associations, and enumerations. Class interfaces include both the class proxy interface (a sub-interface of RefClass) and the class instance interface (a sub-interface of RefObject). Implementation classes are generated for enumerations.

Note that Enki presently does not support the generation or structure of exception interfaces.

Netbeans Repository Wrapper

Enki/Netbeans provides a wrapper around the Netbeans MDR implementation of org.netbeans.mdr.api.MDRepository. The wrapper primarily passes calls through to the underlying MDRepository implementation. In addition, it implements some common functionality such as dropping repository storage and provides a standardized interface for configuring a custom class loader for metamodel code.

Sessions and Transactions

Enki/Netbeans does not require explicit session management, but the wrapper interface provides implementations of the methods for completeness. Enki/Netbeans does require that a session surround any explicitly created MDR transaction, but does emit warnings to the MDR logger whenever it detects a transaction without a session to help maintain compatibility between Enki/Netbeans and Enki/Hibernate.

MDR API Extensions

Enki provides several MDR API extensions to facilitate operations fall into several categories:

  • Operations that were formerly performed in Netbeans MDR using non-public API methods (e.g. dropExtentStorage and ClassLoader Management).
  • Operations that are required for correct operation of Enki/Hibernate (e.g., Session Management)
  • Common operations which can be implemented with improved performance when combined

Enki's MDR API extensions are primarily accessed via the org.eigenbase.enki.mdr.EnkiMDRepository interface.

Session Management

The org.eigenbase.enki.mdr.EnkiMDRepository interface provides methods to begin and end sessions, as well as a method that can be used to detach and reattach sessions from the current thread.

Session management methods are provided in Enki's Netbeans wrapper, but perform no significant actions.

ClassLoader Management

ClassLoader management allows repository users to provide a java.lang.ClassLoader at runtime which is aware of generated repository code that was not present when the JVM was started.

An org.eigenbase.enki.mdr.ClassLoaderProvider can be passed to Enki via the setClassLoaderProvider method in org.eigenbase.enki.mdr.MDRepositoryFactory. The default ClassLoader obtained from the provider can be accessed via the getDefaultClassLoader method of the same class or via the similarly named method on the EnkiMDRepository interface.

For Netbeans, these operations delegate to the org.netbeans.mdr.handlers.BaseObjectHandler class's ClassLoader methods.

Typed getByMofId

Enki provides a method on EnkiMDRepository that allows a RefObject instance to be loaded more quickly if the object's type is known a priori. For Enki/Netbeans, this method looks up the object using the traditional getByMofId method and then validates that the type is correct. For Enki/Hibernate, this method bypasses the MOF ID-to-type lookup step.

findAllOfType

Finds a RefObject of the given type (including its subtypes) with a given value for a given feature. The feature may be specified by name or by passing the RefObject that represents the feature. If multiple matching RefObject instances are found, only one is returned. The feature's type must be a simple Java type (e.g., not a subclass of RefObject).

In Enki/Netbeans this method is implemented by invoking refAllOfType on the given type and then searching the results for the first object where refValue(feature).equals(givenValue). Enki/Hibernate implements this method via a criteria query.

findAllOfClass

Finds a RefObject of the given type (not including its subtypes) with a given value for a given feature. Other than the scope of the search, these methods behave identically to findAllOfType.

dropExtentStorage

Closes a repository extent and deletes the contents of the repository. For Enki/Netbeans, this method obtains the org.netbeans.mdr.Storage object associated with the repository and then invokes the close() and delete() methods on it. For Enki/Hibernate, this method drops all tables associated with the repository from the underlying database.

Future

See also EnkiToDo.

Migration of Farrago JMI Utilities

The Farrago project contains a number of JMI-related utility classes that may eventually migrate to Enki.

Complete Implementation of Transient Packages

Enki/Hibernate currently contains an incomplete implementation of transient packages. This implementation should eventually be completed and used to implement transient packages in projects such as Farrago. In addition, the transient packages implementation should be refactored to allow use from Enki/Netbeans as well.

Personal tools