*****************
Jupyter Notebooks
*****************

A collection of `Python notebooks <https://realpython.com/jupyter-notebook-introduction/>`_
repositories to support the science planning activities of the SOC are available under the JUICE Git server.
There is a number of SOC *Internal* notebooks and a number of *Public* notebooks.

.. warning::
   The JUICE SOC Notebooks only work with UNIX systems: Mac and Linux.
   Support for Windows is not provided nor planned. The notebooks should
   work in Windows systems but the Environmental variables and paths can
   be problematic and will require some work from the user.


Overview
========

The purpose of these notebooks is to engage collaborative and interactive work especially for
quick-look analysis to support rapid decision-making during the science operations and planning
processes. The notebooks are also meant to provide a plethora of resources to conduct
science operations.

Some of these notebooks are also part of the SOC internal User Cases, or procedures and also
part of the SGS science operations and planning activities.


Accessibility
=============

The notebooks are accessible via the JUICE SOC Git server In the following projects:

   - `Top Level Project <https://juigitlab.esac.esa.int/notebooks>`_
   - `Public Notebooks Main Repository <https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks>`_
   - `Internal Project <https://juigitlab.esac.esa.int/notebooks/internal>`_
   - `Internal Notebooks Main Repository <https://juigitlab.esac.esa.int/notebooks/internal/juice-soc-notebooks>`_

The Public Notebooks are accessible by all the public but in order to contribute to their
development you need an account in the JUICE SOC Git server. Please contact the
`JUICE SOC <juice_soc@sciops.esa.int>`_ if you need a user.


Notebook Organisation
=====================

The Python notebooks used by the JUICE SOC have a particular organization.

For both the Public and the Internal notebooks there is a *top-level* repository.
This repository has a ``notebooks`` directory that includes a reference to individual repositories
which are called *sub-modules* in the Git jargon.

These individual repositories contain a number of Python notebooks and are grouped by *theme* or
*subject*. For example, all the **2022 Rings Workshop** notebooks are in the
`JUICE Public Notebooks - 2022 Rings Workshop repository <https://juigitlab.esac.esa.int/notebooks/2022-rings-workshop>`_.

The top-level repositories have a list of **Available Notebooks** in their ``README`` file that includes:

   * an **Abstract** with a brief description of the notebook to facilitate the access or usage of these notebooks,
   * the **Keywords** which are meant to facilitate the access and search of particular notebooks,
   * and **Requirements** that provides a list of the non-standard Python Packages (**Packages**),
     external data taken from Git repositories or the internet (**Data**), internal functions defined
     in the notebook (**Internal**), and external dependencies like software packages (**External**),

Please note that there is an umbrella project, `JUICE Notebooks  <https://juigitlab.esac.esa.int/notebooks>`_. This
umbrella project is public and contains the `Public Notebooks Main Repository <https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks>`_,
all the Public notebooks repositories and the umbrella project of the `internal notebooks <https://juigitlab.esac.esa.int/notebooks/internal>`_.
The internal notebooks project replicates the top-level umbrella project structure, including the
`Internal Notebooks Main Repository <https://juigitlab.esac.esa.int/notebooks/internal/juice-soc-notebooks>`_ and all
the individual internal notebooks.


Working with the notebooks
==========================

The most up to date reference for the notebooks are the Main repositories of the Internal and Public Notebooks
``README`` files:

   - `Public Notebooks readme <https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks/-/blob/main/README.md>`_
   - `Internal Notebooks readme <https://juigitlab.esac.esa.int/notebooks/internal/juice-soc-notebooks/-/blob/main/README.md>`_

These ``README`` files contain all the information you should need to work locally with the notebooks. This information
is also extended hereunder.


How to use the notebooks
------------------------

You can use these resources in *at least* 3 different ways:

   1. By cloning the repos (top-level or sub-module), to get all these notebooks at once.
   2. By downloading or cloning one or more notebooks that you want to try to reproduce on your own.
   3. By browsing the notebooks online to cherry-pick the part that you want to copy/paste.

In the 3rd case, you don't need to install anything on your computer, you only have to click on the link in a table
above to see a static representation of the notebook.

In the 1st and 2nd to cases, you can execute the notebook locally. In the future you will be able to execute them
remotely on the JUICE ESA DataLabs.

If you intend to use the notebooks locally, please follow the steps described below.


Setup the notebooks
===================

You will first need to install and configure the notebooks. Also please keep in mind that you will have to **update**
the notebooks from time to time or when instructed. Details are provided hereunder.


Get the notebooks locally
-------------------------

To obtain all the notebook at once you can use::

   git clone --recursive https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks.git

or for the SOC Internal notebooks::

   git clone --recursive https://juigitlab.esac.esa.int/notebooks/juice-soc-notebooks.git


A note about dependencies
-------------------------

In order to use *most* of the notebooks you need the JUICE SPICE Kernels and some NAIF utilities.

Most of the notebooks already contain commands aimed to download the kernels but it also a good idea to obtain
the JUICE SPICE Kernels follow the instructions from the
`BitBucket repository <https://s2e2.cosmos.esa.int/bitbucket/projects/SPICE_KERNELS/repos/juice/browse>`_. In a nutshell::

   git clone --depth 1 https://s2e2.cosmos.esa.int/bitbucket/scm/spice_kernels/juice.git

The NAIF utilities can be obtained from the `NAIF website <https://naif.jpl.nasa.gov/naif/utilities.html>`_.

.. warning::
   Some notebooks might require other access to other JUICE SOC Git repositories or external data. Some of these data
   might require a user or granted access from the JUICE SOC.


Setup the Environmental Variables
---------------------------------

You will need to define the notebook environmental variables with the location of these items (this allows notebooks to
work in most computers).

To do so, copy the `env.txt` file provided in this repository (or in any of the sub-modules)
in your home directory or in the top directory of the repository as `.env` and edit the paths accordingly::

    cp <path_to_repo>/env.txt ~/.env
    vi ~/.env

An example of an environmental variable is the location of the JUICE SPICE Kernels in your computer e.g.: ``KERNELS_JUICE``.
Here's an example of the content of a ``.env`` file::

     #
     # JUICE SOC Paths for JUICE SOC Notebooks
     #
     KERNELS_JUICE="/Users/marc.costa/tmp/juice/kernels"
     CONF_REPO="/Users/marc.costa/juice/repos/conf"
     NAIF_UTILS="/Users/marc.costa/naif/spice_utilities/exe"
     COSMOGRAPHIA="/Users/marc.costa/cosmographia-4.1/Cosmographia.app/Contents/MacOS/Cosmographia"
     JUICE_PREOPS="/Users/marc.costa/JUICE_PREOPS"

.. warning::
   Please note that the top-level repo ``env.txt`` file **might not** contain all the environmental variables for
   all the sub-modules. If you have an issue with an environmental variable in a sub-module, check the ``env.txt``
   file of that sub-module.


Install Jupyter Locally
-----------------------

Python Notebooks can be executed in different multiple ways. It is highly recommended to use the
`JupyterLab Desktop <https://github.com/jupyterlab/jupyterlab-desktop/releases>`_ application. You can follow the
instructions in the link.

When installing JupyterLab it is **highly recommended** to choose the option to use a new virtual environment and not
one already existing in your laptop. In case of doubts please read the JupyterLab Desktop documentation.

Once installed, launch Jupyter by double-clicking on `JupyterLab Desktop` icon.

.. warning::
   If you are on a **Windows** system, we recommend to install `Anaconda <https://repo.anaconda.com/archive/Anaconda3-2021.11-Windows-x86_64.exe>`_.
   Then, launch Anaconda and click on `JupyterLab` to start a Jupyter environment locally. JupterLab Desktop is also
   available for Windows.


Updating the notebooks
----------------------

You can either update the individual repositories or the top-level repository.

To update the individual repos, go to their directory and use ``git pull``, same applies to push updates.

To update all the repositories from the top-level directory do so with::

   git pull --recurse-submodules

Updating the top-level repository is most likely the best option for you.


Run the notebooks in Jupyter
============================

You can run the notebook cell by cell, with  ``Shift + Enter`` (or with the ``FORWARD ARROW`` button in the top bar).

All the cells are editable by the user and the output will reflect any changes that you made.
Feel free to play with them.

.. warning::
      - if a computation takes too much time, you can stop it with the ``STOP`` button in the top bar.
      - you can restart and clear your variable with the ``COUNTER CLOCKWISE ARROW`` button.
      - you can run all the cells at once with the ``FAST FORWARD`` button.


Contributing to notebooks
=========================

The Notebooks are implemented in different thematic Git repositories containing
a number of notebooks. These repositories are *sub-modules* of a top-level
repository (either the Public or the SOC internal one). Because of this structure
the configuration control of the notebook is not trivial and requires a bit of
attention.

From the development perspective *sub-modules* are to be considered as individual
repositories and therefore any changes need to be *added*, *commited*, and *pushed*
to the repository in the JUICE SOC Git server.

What is best is to `open an issue <https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks/-/issues/new>`_
branch the sub-module repository, make the updates and then do a *Pull Request* to the JUICE SOC Notebook maintainer:

Here is the steps to suggest a change:

   1. Clone and install the notebook sub-module repository locally.

   2. Create your own branch.

   3. Make your edits.

   4. Double-check that the tests are successful ``py.test --nbval-lax``

   5. Open a new pull request explaining what you changed.

.. warning::
   An account on the `JUICE Giltab <https://juigitlab.esac.esa.int/notebook>`_ is required.


SOC Internal Development
------------------------

For SOC members, the *Pull Request* is an option but preferably follow the
procedure described hereunder.

After the sub-module has been updated, the top-level repository also needs to
be updated. Given that for that repo the sub-modules are a *reference*
more concretely a reference to a given *commit* of that repository, first you
need to ensure that you are on the ``main`` branch of the repository. You will
need to *add*, *commit*, and *push* the updated sub-module in the top-level
repository as well.

After doing so you will most likely also have to update the sub-module ``README.md``
file and the top-level repo ``README.md`` file. Do so by running in the top-level repository the
following command::

   cd docs & python readme.py

This command will most likely update the top-level and sub-module ``README`` files. Add and commit them to
git as convenient.

For a more detailed procedure please follow the working example hereunder.


Working example
~~~~~~~~~~~~~~~

For example if we make a change in the ``majis-janus-12pj`` repository which is a
sub-module of the `JUICE Public Notebooks <https://juigitlab.esac.esa.int/notebooks/juice-public-notebooks>`_
by either adding a new notebook or updating a notebook, say::

   juice-public-notebooks/notebooks/majis-janus-12pj/juice-preops-12pj-crema-52.ipynb

First of all make sure that you are in the head of the ``main`` branch. To do so you can simply
execute::

   cd juice-public-notebooks/notebooks/majis-janus-12pj
   git checkout main

Then we need to re-generate the ``README`` file using the Python Function in the top-level repo::

   cd ../../docs
   python readme.py

Now we go back to the sub-module repo and push the updated notebook and ``README`` file::

   cd notebooks/majis-janus-12pj
   git add README.md
   git add juice-preops-12pj-crema-52.ipynb
   git commit -m "Updated the Abstract of the Notebook."
   git push

**This commit and push is performed in the ``majis-janus-12pj`` sub-module repo.**

Finally we need to push the updated top-level repo ``README`` and the updates of the
sub-module we do so with::

   cd ../..
   git add README.md
   git add notebooks/majis-janus-12pj
   git commit -m "Updated the Abstract of the Notebook."
   git push

**Please note that this commit and push is performed in the ``juice-public-notebooks`` repo.**


Adding new sub-modules
~~~~~~~~~~~~~~~~~~~~~~

If you want to add a new sub-module repo in a top-level repo do so with::

   git submodule add https://juigitlab.esac.esa.int/notebooks/<sub-module> notebooks/sub-module>

The list of sub-modules is available in the ``.gitmodules`` file of the top-level main directory.

For all new sub-modules please follow the *obvious* convention defined by the rest of sub-modules.

In addition please note that the sub-module must include a ``description.txt`` file on its main directory. This text
file must include a **single line** description of the purpose of the sub-module. This description is used by the Python
script that generates and updates the ``README`` files.

.. warning::
   To add a sub-module repo in a top-level repo the sub-module **MUST BE** an already existing repository
   present either at ``https://juigitlab.esac.esa.int/notebooks/`` if it is meant to be public or at
   ``https://juigitlab.esac.esa.int/notebooks/internal/`` if it is meant to be internal.


Removing sub-modules
~~~~~~~~~~~~~~~~~~~~

To remove a sub-module is not straightforward. You will need to follow the commands underneath::

    git rm <sub-module>
    rm -rf .git/modules/<sub-module>
    git config -f .git/config --remove-section submodule.<sub-module> 2> /dev/null
    git add .
    git commit -m "Removed submodule <sub-module>"
    git push origin


Notebooks conventions
=====================

There are is a number of basic conventions that must be followed when modifying
or creating new notebooks.


Notebook Sections
-----------------

1. The **first cell** must only contain the notebook title::

     # Notebook Title

2. The **second cell** must only contain an abstract as follows::

     ## Abstract
     This is an abstract for the Notebook.

This abstract must provide a concise summary of the purpose of the notebook.

3. The **third cell** must only contain keywords as follows::

     ## Keywords
     *keyword1*, *keyword1*, *keyword1*, *keyword1*.

These keywords must provide an easy way to search for the convenient notebook.

3. The **fourth cell** must only contain notebooks requisites as follows::

     ## Requisites
     - Packages: `package1`, `package2`, `package3`
     - Data: Data1, Data2
     - Internal: *Python Function 1*, *Python Function 2*
     - External: *External Dependency 1*, *External Dependency 2*

These keywords must provide an easy way to understand the requisites of the notebook.
The three bullets must be present, but they can be set to *None*. **Packages** are
required Python Packages, **Data** is data required not available in the ``data`` directory
of the repository, and **External** are external dependencies which might be executables,
non-Python code, etc.


Good Practises
--------------

When updating or writing new notebooks take into account that the notebooks should be able to work for
any or most of the users regardless if the notebook is public or not. Make sure there are no local paths
and that all the dependencies are configurable from the notebook.

It is also a good idea to run some tests in new or updated notebooks::

    py.test --nbval-lax


.. important::
   All the Python Notebooks in the Git repository should have all their cells executed. Contact the
   `JUICE SOC <juice_soc@sciops.esa.int>`_ if that is not the case.