(nbsync)=
# Sync notebooks / conversion of py and nb

Jupyter notebooks have one significant drawback. They're complicated JSON files with lots of information about when cells were run and what the output looks like. This is useful if you want to open a notebook in the same state, but it makes them a pain to review in Gitlab.

Because of this, before sending notebooks to Gitlab, we **convert them into normal Python files** using a Python package called [`jupytext`](https://jupytext.readthedocs.io/en/latest/index.html). For convenience, TorX provides a helper tool called `nbsync` which applies `jupytext` to the notebooks defined in `notebooks/notebooks_m.py`. Note that to use this tool, you must first have the TorX virtual environment activated:

```bash
source <path to torx home/venv
```

`.ipynb` files are git-ignored, so only the `.py` files should be committed to Gitlab.

## Usage

`nbsync` requires a target format (`-t`) and a selection (`-k`, `-p`, or `-a`):

```
nbsync -t [nb|py] [-k KEY | -p PATH | -a] [-b | -o] [-r]
```

| Flag | Long | Description |
|------|------|-------------|
| `-t` | `--to` | Target format: `nb` (notebook) or `py` (py:percent) |
| `-k` | `--key` | Notebook key |
| `-p` | `--path` | Path to a file or folder |
| `-a` | `--all` | Convert all registered notebooks |
| `-r` | `--recursive` | When `--path` is a folder, recurse into subdirectories |
| `-b` | `--backup` | If destination exists, back it up with a timestamp |
| `-o` | `--overwrite` | If destination exists, overwrite it |
| `-h` | `--help` | Show help |

If the destination file already exists and neither `-b` nor `-o` is specified, `nbsync` will raise an error.

The notebook keys are automatically generated from path and name, for example the key for
`notebooks/genex/analyze_moments.py` will be `genex_analyze_moments`.

```{tip}
Use `nbsync -h` to see all available keys and options.
```

## Common workflows

**First checkout: convert all `.py` files to `.ipynb`**

```{warning}
Be careful when you run this on a repository where you have uncommitted changes to notebooks, 
as it will overwrite your `.ipynb` files if overwrite is specified.
```

```bash
nbsync -t nb -a
```

**Convert a single notebook by key**

```bash
nbsync -t nb -k genex_analyze_moments
```

**Write changes back to `.py`**

```bash
nbsync -t py -k genex_analyze_moments
```

**Convert all notebooks to `.py` before committing**

```bash
nbsync -t py -a
```

**Convert all `.py` files in a folder**

```bash
nbsync -t nb -p notebooks/genex/
```

**Convert all `.py` files in a folder and its subfolders**

```bash
nbsync -t nb -p notebooks/genex/ -r
```

**Overwrite or backup existing files**

```bash
nbsync -t nb -k genex_analyze_moments -o
nbsync -t nb -k genex_analyze_moments -b
```

**Point a notebook at a different TorX environment**

When `nbsync` generates an `.ipynb` it records the Jupyter kernel of the environment that is currently activated, and when it writes a `.py` it leaves the kernel out. This keeps the choice of environment out of version control, and means a round trip moves a notebook to whichever environment you have active:

```bash
nbsync -t py -p mynotebook.ipynb -o
nbsync -t nb -p mynotebook.py -o
```

Note that the `.py` format does not store cell outputs, so this clears them. See [choosing an environment](#switching-kernels) for the alternative that keeps them.

```{tip}
If you accidentally delete work in a `.ipynb` file, check the hidden `.ipynb_checkpoints` folder in the same directory, as it may contain a recent backup.
```