Medical Bioinformatics and e-Bioscience

MOTEUR

What is MOTEUR

MOTEUR is a workflow management system developed by the Modalis Team, in particular Tristan Glatard and Johan Montagnat.

For more information check:

For advanced details: Development pages: New MOTEUR Web Service:

What are Workflows

In simple words, workflows are pipelines where several components (programs) are combined to perform a given analysis task. Workflows can be very simple, composed of only one program, or very large and complex (for example, such as a workflow for analysis of functional MRI data).

Workflow technology offers several facilities that are attractive to implement applications on grids. For example, with MOTEUR independent tasks are automatically performed in parallel and the complexity of grid job submission is hidden from the programmer. The Generic Application Service Wrapper (GASW) is used to automatically wrap non-interactive command line linux applications (=programs) to run them as jobs on the grid.

For more info please read/follow this tutorial.

Executing workflows on the e-BioInfra

Overview

Using workflows to implement and execute applications on the e-BioInfra involves the following steps (see Figure 1):

  • describe a workflow in the Simple Conceptual Unified Flow Language (SCUFL). This description is stored in a file using XML. It specifies all workflow inputs (parameters and files); workflow outputs (results), the workflow components (programs to be executed), and how they are connected to each other.
  • specify the workflow inputs using the GUI of a special plugin developed for the VBrowser ( MOTEUR plugin).
  • start the workflow from the plugin, which then contacts MOTEUR.This requires a valid grid proxy.
  • MOTEUR executes the workflow on the grid using the gLite middleware.
  • The grid jobs run on some grid node and store results on grid storage (LFC).
  • The user monitors the progress of workflows via HTML pages and of jobs using the Job Monitoring plugin
  • The workflow is complete when all boxes show in blue or in red.

Figure 1 - Sequence diagram for workflow execution on e-BioInfra with MOTEUR

For more detailed information about workflow execution, generated files and monitoring of individual jobs, read the MOTEUR advanced page.

MOTEUR Plugin for the VBrowser

The VBrowser MOTEUR plugin provides a GUI to specify inputs for a workflow and run it on the grid with MOTEUR. If the VBrowser is correcty configured, this plugin is automatically started when the user left-clicks on a SCUFL document (suffix .scufl).

The MOTEUR plug-in is distributed with the VBrowser and needs to be configured by the user after installation as explained in the VBrowser page.

Workflow and job monitoring pages

Figure 2 - Workflow (left) and job (right) monitoring pages generated by MOTEUR

MOTEUR maintains monitoring information about all executed workflows available the web server of the e-BioInfra server.

Figure 2-left illustrates the monitoring page for the complete workflow. The boxes indicate workflow component corresponding to programs that are executed as grid jobs. The color of the boxes indicate status:

  • grey: component not active yet
  • green: jobs are running (or have been at least submitted)
  • red: all jobs completed, some with error
  • blue: all jobs completed ok
NOTE: If the same box needs to be executed several times for different input values, MOTEUR will start individual jobs for each one of them and show the box in blue only when all the jobs are completed.

Clicking on the boxes will open the job monitoring page (figure 2-right).

The job monitoring page is composed of three sections:

  1. Summary: simplified view of all jobs
    • Waiting: Jobs that have yet to start running.
    • Running: Jobs that are currently executing.
    • Successfully completed: Jobs that have completed without error.
    • Failed: Jobs that have either failed to run, or have run and failed completing.
  2. Job graph: all possible states of a job. Each bar shows how many jobs are in that particular state.
  3. Job details for each job that has been submitted by MOTEUR for this workflow: the current state, the time it reached this state, and the id of the job. NOTE: job ids can be used with gLite commands to directly obtain more information about the job.
Topic revision: r31 - 2012-05-16 - MahdiJaghouri
 
This site is powered by the TWiki collaboration platformCopyright © 2008-2014 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback