Portable by Design
Deploying Notebook-based Scientific Workflows Across HPC Clusters
Scientific workflows can be difficult to reproduce and move between high-performance computing (HPC) systems. Software stacks differ, data may live in site-specific locations, and each cluster exposes its own resources and execution environment.
This hands-on tutorial introduces three complementary open-source tools—TaskVine, Sciunit, and Floability—for addressing these challenges. You will build a distributed workflow with TaskVine, capture a reproducible execution with Sciunit, and deploy a portable notebook-based workflow as a Floability backpack.
Short presentations introduce each concept, followed immediately by guided exercises that put it into practice.
Tutorial at a glance
- Format: short presentations and guided hands-on exercises
- Duration: 3 hours total, including a 15-minute break
- Tools: TaskVine, Sciunit, and Floability
- Setup: prepared tutorial server or a self-managed x86-64 Linux system
Tutorial outline
Part I — TaskVine and Sciunit · 90 minutes
| Time | Topic | Materials |
|---|---|---|
| 10 min | Tutorial overview — tutorial goals, the portability problem, the three-tool ecosystem, and the TaskVine manager-worker model | Slides |
| 15 min | Access, setup, and TaskVine quickstart | Setup · Hands-on |
| 35 min | TaskVine application structure and execution — tasks, files, workers, and result collection | Slides · Documentation · Hands-on |
| 30 min | Reproducible execution with Sciunit — capture, inspect, and reproduce an execution | Slides · Hands-on |
Break · 15 minutes
Part II — Floability · 75 minutes
| Time | Topic | Materials |
|---|---|---|
| 10 min | Floability overview — portable workflow deployment and the backpack model | Slides · Overview |
| 10 min | Run your first backpack — execute a Python workflow and watch its results in the terminal | Hands-on |
| 15 min | Run your first interactive backpack — run the same workload from a notebook through JupyterLab | Hands-on |
| 10 min | Structure of a backpack — compare their workflow, software, data, and compute specifications | Slides · Guide |
| 15 min | Create and run a backpack — generate and customize a script or notebook template | Hands-on |
| 10 min | Generate a backpack automatically — use Audit to produce initial specifications from a working notebook environment | Hands-on |
| 5 min | Wrap-up and questions | Key takeaways and next steps |
How the tools fit together
TaskVine provides the distributed execution layer. A manager creates tasks, workers obtain those tasks dynamically, and TaskVine moves required files and returns results.
Sciunit captures an execution and its dependencies so that it can be inspected and reproduced outside the original environment.
Floability packages a workflow with its software, data, and compute specifications in a portable deployment unit called a backpack. It prepares the execution environment and connects the workflow to computing resources through TaskVine.
What you will learn
By the end of the tutorial, you will be able to:
- build and run a distributed workflow with TaskVine;
- capture and reproduce an application with Sciunit; and
- package and run script or notebook workflows as Floability backpacks.
Related resources
Ready to begin?
Start by connecting to the prepared tutorial environment or setting up the exercises on your own Linux system.