AWE
AWE 2.0
A light weight workflow manager for scientific computing at scale that:
-
executes CWL workflows
-
is a cloud native workflow platform designed from the ground up to be fast, scalable and fault tolerant
-
supports CWLProv
-
is RESTful. The API is accessible from desktops, HPC systems, exotic hardware, the cloud and your smartphone
-
is designed for complex scientific computing and supports computing on multiple platforms with zero setup
-
supports containerized environments with Docker and Singularity
-
is part of our reproducible science platform Skyport combined to create Researchobjects when combined with CWL and CWLProv
AWE is actively being developed at github.com/MG-RAST/AWE.
You can use AWE simultaenously on Clouds, Clusters and HPC systems with dozens, hundreds or thousands of nodes to run tens of thousands to hundreds of thousands of individuals workflows.
AWE Quickstart
This assumes that you have docker and docker-compose installed and curl is available locally.
Start AWE services
source ./init.sh
docker-compose up
This will start AWE server, one AWE worker, Shock object store and corresponding MongoDB containers.
Don’t forget to later docker-compose down and do not forget, by default this demo configuration does not store data persistently.
Submit a job for simple workflow
This example consists of CWL workflow that takes a PDF file as input, extracts all words and generates a visual wordcloud. Execute the followin command in a new terminal from within the AWE folder:
./awe_submit.sh -w test/tests/pdf2wordcloud.cwl -j test/tests/rules-of-acquisition.job.cwl -d tmp
View result
open image with wordle/ preview here
Documentation
Related software repositories and documentation
| repository | description | link |
|---|---|---|
| AWE monitor | UI for the AWE server | github.com/MG-RAST/awe-monitor |
| Shock | object store | github.com/MG-RAST/Shock |
| Skyport2 | demo environment using docker-compose | github.com/MG-RAST/Skyport2 |
Papers to cite
-
W. Tang, J. Wilkening, N. Desai, W. Gerlach, A. Wilke, F. Meyer, “A scalable data analysis platform for metagenomics,” in Proc. of IEEE International Conference on Big Data, 2013.[ieeexplore] [pdf]
-
W. Gerlach, W. Tang, K. Keegan, T. Harrison, A. Wilke, J. Bischof, M. D’Souza, S. Devoid, D. Murphy-Olson, N. Desai, F. Meyer, “Skyport – Container-Based Execution Environment Management for Multi-Cloud Scientific Workflows,” in Proc. of the 5th International Workshop on Data Intensive Computing in the Clouds, 2014. [pdf]