Generate a Backpack Automatically

Session time: 10 minutes

You already have a Jupyter notebook that works. But how do you turn it into a Floability Backpack without manually specifying all of its software and data dependencies?

👨‍💻 Notebook User

"I already have a working notebook.
How do I generate a Backpack from it?"

                ↓

📓 Notebook  +  🐍 Environment  +  📁 Input Data

                ↓

         floability audit

                ↓

       🎒 Floability Backpack

For floability audit to work, you should already have:

a working Jupyter notebook;
the Conda environment in which the notebook runs successfully; and
the notebook's required input data files available locally.

Then floability audit can observe the notebook while it runs and generate an initial backpack for you.

Instead of manually identifying every software dependency and input file, floability audit executes the notebook and observes what it actually uses.

Important: Audit does not create the notebook's environment or recover missing input data. The notebook must already run successfully with its required environment and input files available. Audit is currently experimental, so the generated backpack should be reviewed and tested before being shared or used for larger runs.

1. Before we begin

Check which Conda environment is active:

echo "${CONDA_PREFIX:-No Conda environment is active}"

The path should end in /tutorial-env. If it does not, activate the environment for your setup.

Live tutorial

source /opt/tutorial/activate.sh

Self-managed

conda activate tutorial-env

Continue with either setup

The environment supplied to floability audit must already contain the dependencies required to run the notebook successfully. The tutorial environment includes NumPy, pandas, Matplotlib, and TaskVine for this notebook.

2. Check the Notebook and Input Data availability

Navigate to the audit example directory:

cd ~/tutorial/examples/audit

Check its contents:

ls -lahtr

You should see the matrix multiplication notebook and a data directory.

Now inspect the input data:

ls -lahtr data/matrices

You should see some csv files.

If the above are available, then the example is already set up with everything floability audit needs:

audit/
├── matrix-multiplication.ipynb
└── data/
    └── matrices/
        └── input matrix files

At this point, we have the three things needed for an audit:

📓 Working notebook
        +
🐍 Working Conda environment
        +
📁 Available input data
        ↓
   floability audit

3. Run Floability Audit

Before running the audit, let's determine your manager port from your assigned username.

⚠️ Important: Use your own manager port

Each user must use a different manager port to avoid conflicts when running the tutorial concurrently.

Add the numeric part of your username to 9000. Tutorial accounts user01 through user25 therefore use ports 9001 through 9025.

Examples:

user03 → 9000 + 3 = 9003

user11 → 9000 + 11 = 9011

⚠️ Replace <MANAGER_PORT> in the command below with your calculated port number.

Now run:

floability audit \
    --notebook matrix-multiplication.ipynb \
    --conda-env "$CONDA_PREFIX" \
    --data-dirs ./data \
    --manager-port <MANAGER_PORT> \
    --backpack-name matrix-backpack

The important options are:

Option	What it does
`--notebook`	Specifies the notebook to execute and audit
`--conda-env`	Specifies the working Conda environment
`--data-dirs`	Tells Audit where possible input data files are located
`--manager-port`	Assigns this participant a unique local manager port
`--backpack-name`	Sets the name of the generated backpack

There are several additional options that you can check using floability audit -h.

During the audit, Floability executes the notebook and observes the dependencies used during that execution.

Instead of manually specifying things, Floability collects this information from the execution and uses it to create an initial backpack.

4. Inspect the Generated Backpack

After the audit finishes, check the generated backpack:

ls -lahtr matrix-backpack

The generated directory will have a structure similar to:

matrix-backpack/
├── compute/
│   └── compute.yml
├── data/
│   └── data.yml
├── software/
│   └── environment.yml
└── workflow/
    └── matrix-multiplication.ipynb

The main files are:

workflow/matrix-multiplication.ipynb — the notebook
software/environment.yml — discovered software dependencies
data/data.yml — detected input data
compute/compute.yml — initial compute configuration

For example, inspect the generated software specification:

cat matrix-backpack/software/environment.yml

Then inspect the detected data:

cat matrix-backpack/data/data.yml

5. Run the Backpack

Run the backpack interactively:

floability run --backpack matrix-backpack

Floability uses the generated specifications to prepare the workflow and launch JupyterLab.

Self-managed

Floability is running on your own computer. Open the complete URL printed by Floability in your browser.

Live tutorial

Floability is running on a remote server. The Jupyter server is intentionally not exposed to the public Internet. Open a new terminal on your own computer and create an SSH tunnel:

Floability prints a ready-to-copy command

When startup finishes, Floability prints the complete SSH tunnel command with the correct server address and port. You can copy and run that command directly. The pattern below is available if you need to enter it manually.

ssh -N -L 8888:localhost:<REMOTE_PORT> <USERNAME>@<SERVER>

Replace:

<REMOTE_PORT> with the port in Floability's JupyterLab URL;
<USERNAME> with your assigned remote username; and
<SERVER> with your assigned server address.

Keep the tunnel terminal open. In your browser, replace the remote port in the printed URL with local port 8888:

http://localhost:8888/lab?token=<TOKEN>

If port 8888 is already in use on your computer, choose another local port, such as 8889, in both the SSH command and browser URL.

The remote port belongs to this specific Floability run. Close any tunnel from an earlier run with Ctrl-C, open a new terminal on your computer, and use the new tunnel command printed by Floability.

Continue with either setup

Open workflow/matrix-multiplication.ipynb and run its cells. When you are finished, save the notebook and return to the terminal running Floability. Press Ctrl-C to stop JupyterLab and its worker processes. Live participants should also return to the SSH tunnel terminal and press Ctrl-C. For another interactive run, open a new local terminal and use the new tunnel command and port printed by Floability.

You can also execute the notebook without opening Jupyter:

floability execute --backpack matrix-backpack

For a Slurm cluster:

floability run \
    --backpack matrix-backpack \
    --batch-type slurm

Reset

To repeat the Audit exercise, remove only the generated backpack before returning to step 3:

if test -d "$HOME/tutorial/examples/audit/matrix-backpack"; then
  rm -rf -- "$HOME/tutorial/examples/audit/matrix-backpack"
fi

What Did We Do?

We started with an existing working notebook setup:

📓 Notebook  +  🐍 Environment  +  📁 Input Data
                ↓
         floability audit
                ↓
       🎒 Generated Backpack

Instead of manually specifying everything the notebook requires, we let Floability observe a successful execution and generate the initial backpack specifications.

Next: Optional backpack examples →