Jupyter Notebooks#

Jupyter notebooks are a great way to perform post-processing. They let you interactively explore your data and quickly iterate by writing small code blocks.

Launch locally#

TorX comes with a copy of Jupyter installed. If you’re running TorX on your local machine (i.e. laptop), you can launch a Jupyter session by running

source <path to torx home>/venv
jupyter lab

or alternatively

jupyter notebook

This should open a web-browser (generally localhost:8888/) where you can create and open notebooks.

To check which kernels are available, run

jupyter kernelspec list

Use port forwarding#

If you’re running on a remote machine (i.e. the TOK clusters), you can use port forwarding. In one terminal window run (where XXXX is some 4-digit number)

source <path to torx home>/venv
jupyter lab --no-browser --port=XXXX

and then in another terminal run (where YYYY is another four-digit number)

localuser@localhost: ssh -N -f -L localhost:YYYY:localhost:XXXX remoteuser@remotehost

then open localhost:YYYY/ in a web-browser on your local machine. Note that this doesn’t work for all machines.

Use RVS#

If you want to do your analysis on a MPCDF machine, you can use the remote visualization service. You’ll first need to click Initialise Remote Visualization for the machine that you want to run on (only needs to be done once), then SSH into the machine and install TorX (see the installation guide) to get a TorX kernel on the machine. Next, launch a RVS session using the web-interface, and once you’ve opened a notebook select torx (env) in the list of kernels, or torx (<name>) if you installed the environment with --name <name>.

RVS launches JupyterLab from the Anaconda/3 module and never activates a TorX environment, so the environment a notebook runs in is decided solely by the kernel you select.

  • The kernel records the full path of its interpreter, so the environment must live somewhere the compute node can read, such as your home directory rather than node-local scratch.

  • If you use FLARE, install with -fr. The install script then records LD_LIBRARY_PATH in the kernel, because RVS cannot module-load libraries that set it from inside a Jupyter session.

  • The MPCDF Jupyter does not have jupytext installed, so notebooks are not paired automatically. Editing a notebook in a RVS session changes only the .ipynb. To update the committed .py, run nbsync (see sync notebooks) in a terminal with the TorX environment activated.

Choose which environment a notebook uses#

Each TorX environment installs its own Jupyter kernel, named after the environment: an environment installed with --name v2 gives a kernel displayed as torx (v2). This is what lets you keep several environments side by side and run one notebook against one environment and another notebook against another. Run jupyter kernelspec list to see which kernels you have.

A notebook records only the name of its kernel. The path to the Python interpreter lives in the kernel itself, not in the notebook. Note that:

  • Moving or renaming an environment directory breaks its kernel, because the recorded interpreter path no longer exists. Reinstall the environment or manually update the path in the notebook metadata to fix it.

  • If you install a new environment under a different name, an existing notebook still asks for the old kernel. If that old environment is still present, the notebook keeps running against it, and nothing tells you that you are not using the new one. If the old environment has been deleted, the kernel fails to start instead.

Either way, you have to point the notebook at the new kernel yourself. There are two ways to do this.

From within Jupyter#

Open the notebook, choose Kernel -> Change Kernel..., select torx (v2), and save. This keeps the cell outputs, so it is the right choice for a notebook whose results you want to hold on to. It works for any notebook, including ones you created outside the TorX repository.

From the command line#

nbsync writes the kernel of the currently active environment whenever it generates an .ipynb, so converting a notebook to .py and back re-points it:

source <path to v2 env>/bin/activate
nbsync -t py -p mynotebook.ipynb -o
nbsync -t nb -p mynotebook.py -o

The -p flag takes any path, so the notebook does not need to live in the repository or be registered in notebooks/notebooks_m.py.

Warning

The .py representation does not store cell outputs, so this round trip clears them. Use Change Kernel in Jupyter instead if you want to keep your results.

Notebooks tracked in the repository need no manual step, since they are generated from .py files that deliberately carry no kernel. Regenerate them with the environment you want activated:

nbsync -t nb -k <key> -o

This is also the fix for a notebook written before the kernel was named after its environment, which may still refer to a kernel called torx that does not exist on a fresh installation.

Remove old kernels#

Installing a new environment never removes the kernel of an old one, so kernels of environments you have deleted linger in the list. Remove them yourself:

jupyter kernelspec list
jupyter kernelspec uninstall torx-v1