Sync notebooks / conversion of py and nb#

Jupyter notebooks have one significant drawback. They’re complicated JSON files with lots of information about when cells were run and what the output looks like. This is useful if you want to open a notebook in the same state, but it makes them a pain to review in Gitlab.

Because of this, before sending notebooks to Gitlab, we convert them into normal Python files using a Python package called jupytext. For convenience, TorX provides a helper tool called nbsync which applies jupytext to the notebooks defined in notebooks/notebooks_m.py. Note that to use this tool, you must first have the TorX virtual environment activated:

source <path to torx home/venv

.ipynb files are git-ignored, so only the .py files should be committed to Gitlab.

Usage#

nbsync requires a target format (-t) and a selection (-k, -p, or -a):

nbsync -t [nb|py] [-k KEY | -p PATH | -a] [-b | -o] [-r]

Flag	Long	Description
`-t`	`--to`	Target format: `nb` (notebook) or `py` (py:percent)
`-k`	`--key`	Notebook key
`-p`	`--path`	Path to a file or folder
`-a`	`--all`	Convert all registered notebooks
`-r`	`--recursive`	When `--path` is a folder, recurse into subdirectories
`-b`	`--backup`	If destination exists, back it up with a timestamp
`-o`	`--overwrite`	If destination exists, overwrite it
`-h`	`--help`	Show help

If the destination file already exists and neither -b nor -o is specified, nbsync will raise an error.

The notebook keys are automatically generated from path and name, for example the key for notebooks/genex/analyze_moments.py will be genex_analyze_moments.

Tip

Use nbsync -h to see all available keys and options.

Common workflows#

First checkout: convert all .py files to .ipynb

Warning

Be careful when you run this on a repository where you have uncommitted changes to notebooks, as it will overwrite your .ipynb files if overwrite is specified.

nbsync -t nb -a

Convert a single notebook by key

nbsync -t nb -k genex_analyze_moments

Write changes back to .py

nbsync -t py -k genex_analyze_moments

Convert all notebooks to .py before committing

nbsync -t py -a

Convert all .py files in a folder

nbsync -t nb -p notebooks/genex/

Convert all .py files in a folder and its subfolders

nbsync -t nb -p notebooks/genex/ -r

Overwrite or backup existing files

nbsync -t nb -k genex_analyze_moments -o
nbsync -t nb -k genex_analyze_moments -b

Point a notebook at a different TorX environment

When nbsync generates an .ipynb it records the Jupyter kernel of the environment that is currently activated, and when it writes a .py it leaves the kernel out. This keeps the choice of environment out of version control, and means a round trip moves a notebook to whichever environment you have active:

nbsync -t py -p mynotebook.ipynb -o
nbsync -t nb -p mynotebook.py -o

Note that the .py format does not store cell outputs, so this clears them. See choosing an environment for the alternative that keeps them.

Tip

If you accidentally delete work in a .ipynb file, check the hidden .ipynb_checkpoints folder in the same directory, as it may contain a recent backup.