OpenWP Project

• HomeLatest news
• AboutOpenWP quick presentation
• DirectivesDirectives available to describe a workflow with OpenWP
• How it worksHow OpenWP turns your code into a grid enabled workflow
• StateDevelopment status, limits and next steps
• TeamWho is involved in OpenWP
• DownloadGet OpenWP!
• contact us

How openWP works

OpenWP main tasks

As a grid enabled, directive based, workflow system, OpenWP needs to handle multiple aspects of distributed programming, among which:

• Refactoring the initial annoted code in order to define "independant" tasks
• Memory management
• Workflow definition and enactment

OpenWP main components

OpenWP is composed of a code refactoring component and a runtime component.

The code refactoring component parses the code, finds the OpenWP sections defined by the programmer, extracts methods from each section, defines workflow (and task description files) from dependencies, then compiles the modified java file using the standard "javac" program.

The runtime component is responsible for launching the task, providing and storing the necessary data, and handling some of the "standard" operations (i.e. not directly related to the task) such as spliting arrays or agregating results.

Code refactoring

The OpenWP parser finds openWP sections as defined by the user and turns it into methods (for now only static methods can be defined). Special instructions are added at the begining and at the end of the method to get and store data. For each variable needed a call to OpenWP.getData(String name) is added, the returned value cast into the correct type and stored into the corresponding local variable. In the same way a call to OpenWP.storeData(String name,Object value) is added at the end of the method.

Therefore the newly constructed method only takes String parameters, as the actual value is obtained through the call of getData(String name), and doesn't return anything, as it stores results thereafter through the call of storeData(String name, Object value).

Memory management

The OpenWP annotation language is designed after OpenMP and, as OpenMP, OpenWP assumes a shared memory system is used. OpenWP uses a peer to peer Distributed HashTable (DHT) to resolve references into data location. There are multiple projects of DHT systems, as such or as part of bigger projects, that could be used. For this matter we used a component based approach so it's easy to develop a new connector to use another DHT.

OpenWP currently uses a virtual DHT (local filesystem is used, for testing purpose or to use with grid file systems), JGroups DHT, or DKS DHT.

Workflow definition and enactment

You probably noticed that none of the components described in "OpenWP main components" section actually deals with the workflow itself. That's because there is none. OpenWP uses for that purpose an external workflow enactor. As for the DHT, the structure of OpenWP allows the use of any Workflow engine, but as of now only DAGman, from the condor project (see condor web site) can be used. Other tools should be available as the necessary "workflow writers" are developped.

OpenWP workflow

Here is a global view of a compile and launch process:

• The user annotates his code, defining workflow tasks
• He launches the "CompileAndLaunch" program
• The tool parses the code, seeking for openWP directives
• It isolates openWP sections and turns it into method
• It generates an internal workflow representation
• It uses a workflow writer class to write the actual workflow (dagman for now)
• It calls javac to compile the modified file
• It stores in the "distributed memory" the used jars and the generated class
• It calls the workflow enactor to launch the workflow exectution

Please note that the OpenWP parser doesn't check the code for correctness, it's just a basic tool for cutting a correct java code into smaller parts and relies on "javac" afterwards to compile the code. As a consequence the programmer should test its code with javac before any use of OpenWP "CompileAndLaunch" because OpenWP will rewrite the code and any error indicated by javac won't be relevant for the initial source code (the location of the error will be meaningless for the initial file).