shesmu

Shesmu is part of a bioinformatics workflow launching system. It does not do workflow launching in the usual sense.

Shesmu acts as an intermediary between two systems: a database of analysis provenance and a workflow scheduler. It scans the provenance system for a list of which files have been produced and then uses decision-action blocks called olives to decide what “actions” should be run. Actions can be launching workflows or filing tickets.

It can filter and aggregate the provenance information to decide what actions to perform. Shesmu is designed to operate in a stateless way. When the server starts up, it creates a list of all actions that need to be performed, then it issues commands to the workflow launcher, which should politely recognise which commands have been previously executed and only launch the new tasks.

Plugins are used to communicate with other systems to perform the real work–Shesmu is an empty vessel that requires plugins to find data and take action. The olives are supplied by a script that is compiled and run over all the provenance data.

Documentation Quick Reference

For olive developers:

• Olive Tutorial
• Olive Language Reference
• Complex Olive Cookbook
• The Mandatory Guide to Optional Values
• Algebraic Values without Algebra

For operators:

• Training Guide for Operators
• ShAWK - Command-line Data Extraction
• Static Actions
• Writing Guided Meditations

For system administrators:

• Running the Demo
• Installation Guide
• Ask your doctor if Shesmu is right for you
• Configuring Input Formats
• JSON-Defined Input Formats

For developers:

• Plugin Implementation Guide
• Compiler Hacking
• Java API Documentation

Miscellanea

• Shesmu Glossary
• Shesmu FAQ

Plugins

Shesmu is meant to be a pluggable system. The base system provides a few plugins that might be useful, but it is likely that custom plugins are needed. Please read the plugin implementation guide for information about how to extend the system. The plugins available are:

• Cerberus
• Git and GitHub
• Guanyin reporting
• JIRA ticket management
• Loki logging
• MongoDB
• Nabu file QC
• Pinery
• Prometheus Alert Manager
• Token Bucket Throttling
• Run Scanner
• SFTP servers
• Standard Output Logger
• Tab-separated files
• Víðarr

Once a plugin is configured, it can provide:

• actions: the launchable elements described in Run olives
• constants: fixed parameters external to the olive
• dumpers: when debugging Shesmu olives, it is possible to log all output passing through an olive to a dumper. Dumping to a non-existent dumper simply discards the debugging output.
• functions: data manipulation functions that can be used in the olives
• input formats: the data that olives can draw on using the Input declaration at the start of a file, or in Join or LeftJoin clauses
• signers: special variables that are computed based on the current row being processed in the olive
• source linker: once a Shesmu server is deployed, it can be useful to have links from the Shesmu dashboard to the original .shesmu sources, especially when they are stored in git or the like. A source linker knows how to convert a path on the local file system into a URL.
• throttlers: when Shesmu has actions to perform, it will perform them as quickly as possible. It may be useful to throttle Shesmu based on external criteria.

To view what olives may use, from the main Shesmu status page, use the Definition menu to view the available resources including documentation and type information. The plugins may not be able to provide static documentation since they may dynamically provide actions, constants, functions, or signers.

For many plugins, the filename will determine the name of things available to the olives.

This site is open source. Improve this page.