fireworks
https://github.com/materialsproject/fireworks
FireWorks stores, executes, and manages calculation workflows.
- Website (including documentation): https://materialsproject.github.io/fireworks/
- Help/Support: https://discuss.matsci.org/c/fireworks
- Source: https://github.com/materialsproject/fireworks/
https://materialsproject.github.io/fireworks/#citing-label
While FireWorks provides many features, its basic operation is simple. You can run FireWorks on a single laptop or at a supercomputing center.
Centralized Server and Worker Model
There are essentially just two components of a FireWorks installation:
-
A server (“LaunchPad”) that manages workflows. You can add workflows (a DAG of “FireWorks”) to the LaunchPad, query for the state of your workflows, orrerun workflows. The workflows can be a straightforward series of scripts or dynamically adapt depending on the results obtained.
-
One or more workers(“FireWorkers”) that run your jobs. The FireWorkers request workflows from the LaunchPad, execute them, and send back information. The FireWorker can be as simple as the same workstation used to host the LaunchPad, or complicated like a national supercomputing center with a queueing system.
The basic infrastructure looks like this:
The components are largely decoupled, which makes FireWorks easier touse. End users can add new workflows to the LaunchPad without worrying about the details of how and where the workflows will be run (unless they really want to tailor the details of job execution). This keeps theworkflow specifications lightweight, tidy, and easy to learn and use (if you’ve ever seen lengthy XML-based specifications in other workflow software, you’ll notice the difference in FireWorks right away).
On the opposite end, administrators can configure worker computers without worrying about where workflows are coming from or what they looklike (although you can assign jobs to certain resources if desired). Running on a heterogeneous set of worker computers is simple because essentially the same code is used internally by FireWorks for running onsimple workstations or a large supercomputing center, submitting to a traditional or web-based queue system, or packing together many jobs into a single queue submission.
Workflow Model
Workflows in FireWorks are made up of three main components:
-
A Firetask isan atomic computing job. It can call a single shell script or execute asingle Python function that you define (either within FireWorks, or in an external package).
-
A Firework contains the JSON specthat includes all the information needed to bootstrap your job. For example, the spec contains an array of Firetasks to execute in sequence.The spec also includes any input parameters to pass to your Firetasks. You can easily perform the same function over different input data by creating Fireworks with identical Firetasks but different input parameters in the spec. You can design your spec however you’d like, as long as it’s valid JSON. The JSON format used for Firework specs is extremely flexible, very easy to learn (Python users familiar with dicts and arraysessentially already know JSON), and immediately makes rich searches over the input data available to end users through MongoDB’s JSON document search capabilities.
-
A Workflow isa set of FireWorks with dependencies between them. For example, you might need a parent Firework to finish and generate some output files before running two child FireWorks.
Between FireWorks, you can return a FWActionthat can store data or modify the Workflow depending on the output (e.g., pass data to the next step, cancel the remaining parts of the Workflow, or even add new FireWorks that are defined within the object).
The FireWorks tutorials and FW design tips explain how to connect these components to achieve the desired behavior.
https://materialsproject.github.io/fireworks/quickstart.html