Jupyter Notebooks

A collection of Python notebooks repositories to support the science planning activities of the SOC are available under the JUICE Git server. There is a number of SOC Internal notebooks and a number of Public notebooks.

Warning

The JUICE SOC Notebooks only work with UNIX systems: Mac and Linux. Support for Windows is not provided nor planned. The notebooks should work in Windows systems but the Environmental variables and paths can be problematic and will require some work from the user.

Overview

The purpose of these notebooks is to engage collaborative and interactive work especially for quick-look analysis to support rapid decision-making during the science operations and planning processes. The notebooks are also meant to provide a plethora of resources to conduct science operations.

Some of these notebooks are also part of the SOC internal User Cases, or procedures and also part of the SGS science operations and planning activities.

Accessibility

The notebooks are accessible via the JUICE SOC Git server In the following projects:

• Top Level Project

• Public Notebooks Main Repository

• Internal Project

• Internal Notebooks Main Repository

The Public Notebooks are accessible by all the public but in order to contribute to their development you need an account in the JUICE SOC Git server. Please contact the JUICE SOC if you need a user.

Notebook Organisation

The Python notebooks used by the JUICE SOC have a particular organization.

For both the Public and the Internal notebooks there is a top-level repository. This repository has a notebooks directory that includes a reference to individual repositories which are called sub-modules in the Git jargon.

These individual repositories contain a number of Python notebooks and are grouped by theme or subject. For example, all the 2022 Rings Workshop notebooks are in the JUICE Public Notebooks - 2022 Rings Workshop repository.

The top-level repositories have a list of Available Notebooks in their README file that includes:

• an Abstract with a brief description of the notebook to facilitate the access or usage of these notebooks,

• the Keywords which are meant to facilitate the access and search of particular notebooks,

• and Requirements that provides a list of the non-standard Python Packages (Packages), external data taken from Git repositories or the internet (Data), internal functions defined in the notebook (Internal), and external dependencies like software packages (External),

Please note that there is an umbrella project, JUICE Notebooks. This umbrella project is public and contains the Public Notebooks Main Repository, all the Public notebooks repositories and the umbrella project of the internal notebooks. The internal notebooks project replicates the top-level umbrella project structure, including the Internal Notebooks Main Repository and all the individual internal notebooks.

Working with the notebooks

The most up to date reference for the notebooks are the Main repositories of the Internal and Public Notebooks README files:

• Public Notebooks readme

• Internal Notebooks readme

These README files contain all the information you should need to work locally with the notebooks. This information is also extended hereunder.

How to use the notebooks

You can use these resources in at least 3 different ways:

1. By cloning the repos (top-level or sub-module), to get all these notebooks at once.

2. By downloading or cloning one or more notebooks that you want to try to reproduce on your own.

3. By browsing the notebooks online to cherry-pick the part that you want to copy/paste.

In the 3rd case, you don’t need to install anything on your computer, you only have to click on the link in a table above to see a static representation of the notebook.

In the 1st and 2nd to cases, you can execute the notebook locally. In the future you will be able to execute them remotely on the JUICE ESA DataLabs.

If you intend to use the notebooks locally, please follow the steps described below.

Setup the notebooks

You will first need to install and configure the notebooks. Also please keep in mind that you will have to update the notebooks from time to time or when instructed. Details are provided hereunder.

Get the notebooks locally

To obtain all the notebook at once you can use:

git clone --recursive https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks.git

or for the SOC Internal notebooks:

git clone --recursive https://juigitlab.esac.esa.int/notebooks/juice-soc-notebooks.git

A note about dependencies

In order to use most of the notebooks you need the JUICE SPICE Kernels and some NAIF utilities.

Most of the notebooks already contain commands aimed to download the kernels but it also a good idea to obtain the JUICE SPICE Kernels follow the instructions from the BitBucket repository. In a nutshell:

git clone --depth 1 https://s2e2.cosmos.esa.int/bitbucket/scm/spice_kernels/juice.git

The NAIF utilities can be obtained from the NAIF website.

Warning

Some notebooks might require other access to other JUICE SOC Git repositories or external data. Some of these data might require a user or granted access from the JUICE SOC.

Setup the Environmental Variables

You will need to define the notebook environmental variables with the location of these items (this allows notebooks to work in most computers).

To do so, copy the env.txt file provided in this repository (or in any of the sub-modules) in your home directory or in the top directory of the repository as .env and edit the paths accordingly:

cp <path_to_repo>/env.txt ~/.env
vi ~/.env

An example of an environmental variable is the location of the JUICE SPICE Kernels in your computer e.g.: KERNELS_JUICE. Here’s an example of the content of a .env file:

#
# JUICE SOC Paths for JUICE SOC Notebooks
#
KERNELS_JUICE="/Users/marc.costa/tmp/juice/kernels"
CONF_REPO="/Users/marc.costa/juice/repos/conf"
NAIF_UTILS="/Users/marc.costa/naif/spice_utilities/exe"
COSMOGRAPHIA="/Users/marc.costa/cosmographia-4.1/Cosmographia.app/Contents/MacOS/Cosmographia"
JUICE_PREOPS="/Users/marc.costa/JUICE_PREOPS"

Warning

Please note that the top-level repo env.txt file might not contain all the environmental variables for all the sub-modules. If you have an issue with an environmental variable in a sub-module, check the env.txt file of that sub-module.

Install Jupyter Locally

Python Notebooks can be executed in different multiple ways. It is highly recommended to use the JupyterLab Desktop application. You can follow the instructions in the link.

When installing JupyterLab it is highly recommended to choose the option to use a new virtual environment and not one already existing in your laptop. In case of doubts please read the JupyterLab Desktop documentation.

Once installed, launch Jupyter by double-clicking on JupyterLab Desktop icon.

Updating the notebooks

You can either update the individual repositories or the top-level repository.

To update the individual repos, go to their directory and use git pull, same applies to push updates.

To update all the repositories from the top-level directory do so with:

git pull --recurse-submodules

Updating the top-level repository is most likely the best option for you.

Run the notebooks in Jupyter

You can run the notebook cell by cell, with Shift + Enter (or with the FORWARD ARROW button in the top bar).

All the cells are editable by the user and the output will reflect any changes that you made. Feel free to play with them.

Contributing to notebooks

The Notebooks are implemented in different thematic Git repositories containing a number of notebooks. These repositories are sub-modules of a top-level repository (either the Public or the SOC internal one). Because of this structure the configuration control of the notebook is not trivial and requires a bit of attention.

From the development perspective sub-modules are to be considered as individual repositories and therefore any changes need to be added, commited, and pushed to the repository in the JUICE SOC Git server.

What is best is to open an issue branch the sub-module repository, make the updates and then do a Pull Request to the JUICE SOC Notebook maintainer:

Here is the steps to suggest a change:

1. Clone and install the notebook sub-module repository locally.

2. Create your own branch.

3. Make your edits.

4. Double-check that the tests are successful py.test --nbval-lax

5. Open a new pull request explaining what you changed.

Warning

An account on the JUICE Giltab is required.

SOC Internal Development

For SOC members, the Pull Request is an option but preferably follow the procedure described hereunder.

After the sub-module has been updated, the top-level repository also needs to be updated. Given that for that repo the sub-modules are a reference more concretely a reference to a given commit of that repository, first you need to ensure that you are on the main branch of the repository. You will need to add, commit, and push the updated sub-module in the top-level repository as well.

After doing so you will most likely also have to update the sub-module README.md file and the top-level repo README.md file. Do so by running in the top-level repository the following command:

cd docs & python readme.py

This command will most likely update the top-level and sub-module README files. Add and commit them to git as convenient.

For a more detailed procedure please follow the working example hereunder.

Working example

For example if we make a change in the majis-janus-12pj repository which is a sub-module of the JUICE Public Notebooks by either adding a new notebook or updating a notebook, say:

juice-public-notebooks/notebooks/majis-janus-12pj/juice-preops-12pj-crema-52.ipynb

First of all make sure that you are in the head of the main branch. To do so you can simply execute:

cd juice-public-notebooks/notebooks/majis-janus-12pj
git checkout main

Then we need to re-generate the README file using the Python Function in the top-level repo:

cd ../../docs
python readme.py

Now we go back to the sub-module repo and push the updated notebook and README file:

cd notebooks/majis-janus-12pj
git add README.md
git add juice-preops-12pj-crema-52.ipynb
git commit -m "Updated the Abstract of the Notebook."
git push

This commit and push is performed in the ``majis-janus-12pj`` sub-module repo.

Finally we need to push the updated top-level repo README and the updates of the sub-module we do so with:

cd ../..
git add README.md
git add notebooks/majis-janus-12pj
git commit -m "Updated the Abstract of the Notebook."
git push

Please note that this commit and push is performed in the ``juice-public-notebooks`` repo.

Adding new sub-modules

If you want to add a new sub-module repo in a top-level repo do so with:

git submodule add https://juigitlab.esac.esa.int/notebooks/<sub-module> notebooks/sub-module>

The list of sub-modules is available in the .gitmodules file of the top-level main directory.

For all new sub-modules please follow the obvious convention defined by the rest of sub-modules.

In addition please note that the sub-module must include a description.txt file on its main directory. This text file must include a single line description of the purpose of the sub-module. This description is used by the Python script that generates and updates the README files.

Warning

To add a sub-module repo in a top-level repo the sub-module MUST BE an already existing repository present either at https://juigitlab.esac.esa.int/notebooks/ if it is meant to be public or at https://juigitlab.esac.esa.int/notebooks/internal/ if it is meant to be internal.

Removing sub-modules

To remove a sub-module is not straightforward. You will need to follow the commands underneath:

git rm <sub-module>
rm -rf .git/modules/<sub-module>
git config -f .git/config --remove-section submodule.<sub-module> 2> /dev/null
git add .
git commit -m "Removed submodule <sub-module>"
git push origin

Notebooks conventions

There are is a number of basic conventions that must be followed when modifying or creating new notebooks.

Notebook Sections

1. The first cell must only contain the notebook title:

   # Notebook Title
   

2. The second cell must only contain an abstract as follows:

   ## Abstract
   This is an abstract for the Notebook.
   

This abstract must provide a concise summary of the purpose of the notebook.

3. The third cell must only contain keywords as follows:

   ## Keywords
   *keyword1*, *keyword1*, *keyword1*, *keyword1*.
   

These keywords must provide an easy way to search for the convenient notebook.

3. The fourth cell must only contain notebooks requisites as follows:

   ## Requisites
   - Packages: `package1`, `package2`, `package3`
   - Data: Data1, Data2
   - Internal: *Python Function 1*, *Python Function 2*
   - External: *External Dependency 1*, *External Dependency 2*
   

These keywords must provide an easy way to understand the requisites of the notebook. The three bullets must be present, but they can be set to None. Packages are required Python Packages, Data is data required not available in the data directory of the repository, and External are external dependencies which might be executables, non-Python code, etc.

Good Practises

When updating or writing new notebooks take into account that the notebooks should be able to work for any or most of the users regardless if the notebook is public or not. Make sure there are no local paths and that all the dependencies are configurable from the notebook.

It is also a good idea to run some tests in new or updated notebooks:

py.test --nbval-lax

Important

All the Python Notebooks in the Git repository should have all their cells executed. Contact the JUICE SOC if that is not the case.