Contributing

How to use your own working-copy?

This section details how to use a working copy of MultiscaleRun. Typically, these methods are employed by developers that want to customize MultiscaleRun itself or one or more dependencies.

There are 2 ways to do that:

  • MultiscaleRun virtualenv: recommended when you just need to change MultiscaleRun code.

  • Spack environments: all the other cases.

MultiscaleRun virtualenv (python-venv)

Create a Python virtual environment .venv with this simple series of commands:

git clone git@bbpgitlab.epfl.ch:molsys/multiscale_run.git /path/to/multiscale_run
cd /path/to/multiscale_run
module load unstable py-multiscale-run
multiscale-run virtualenv

Then activate the virtualenv environment to work with your working-copy: source .venv/bin/activate

$ .venv/bin/multiscale-run --version
multiscale-run 0.7

Note

This may also work on your spack-powered machine!

Spack environments

This section is a specialization of this generic spack guide.

As a concrete example, let’s say that we want to make some modifications in Neurodamus and MultiscaleRun and see how the code performs in rat V6. Let’s also assume that we are on BB5 with a working spack. If it is not the case please check the spack documentation on BB5 on how to install spack on BB5.

It is imperative to allocate a node before starting as heavy usage of the login nodes is prohibited.

Let’s first clone the repositories:

git clone git@bbpgitlab.epfl.ch:molsys/multiscale_run.git
git clone git@github.com:BlueBrain/neurodamus.git

Our environment will be called spackenv. Let’s create and activate it:

spack env create -d spackenv
spack env activate -d spackenv

Now, we should have 3 folders:

.
├ multiscale_run
├ neurodamus
└ spackenv

Let’s add Neurodamus and tell spack to use the source code that we cloned before:

spack add py-neurodamus@develop
spack develop -p ${PWD}/neurodamus --no-clone py-neurodamus@develop

And let’s do the same for MultiscaleRun:

spack add py-multiscale-run@develop
spack develop -p ${PWD}/multiscale_run --no-clone py-multiscale-run@develop

Now we can finally install:

spack install

In order to be sure that all changes have been in effect and $PYTHONPATH is populated properly (note that this is only needed when you set up the Spack environment the first time):

spack env deactivate
spack env activate -d spackenv

Now you are ready to test your version of MultiscaleRun (follow the section How to use the ``multiscale-run` executable?). If you use SLURM you need to remove the py-multiscale-run and, instead, load the spackenv environment. In concrete terms, in simulation.sbatch you need to replace this line:

module load py-multiscale-run

with these lines (assuming that your spackenv is in ~. Change the location accordingly):

module load llvm
spack env activate -d ~/spackenv

If you want to run an interactive session instead you need the following modules too:

module load unstable llvm

Note

Remember that every time you add a modification to the code you need to call spack install before testing it.

Note

This may also work on your spack-powered machine!

How to check a contribution?

Before submitting a contribution, it is suggested to use tox utility to make preliminary checks:

  1. tox to run unit and integration tests.

  2. tox -e lint to perform static analysis of the code and ensure it is properly formatted.

  3. tox -e docs to ensure the documentation builds properly.

  4. tox -e fixlint to format the code and applies ruff recommended fixes.

Note

If tox utility is not installed already, use a Python virtual environment for isolation purpose:

python -mvenv venv
. venv/bin/activate
pip install -m tox
tox

How to release a new version?

MultiscaleRun relies on setuptools-scm utility to infer the Python package version from the SCM. It is not needed to increase the version manually. Anyway, there are a few things to perform before creating the git tag.

  1. If the structure of the JSON configuration changed during this release (key addition, removal, …), then increment the JSON config_format key in the files:

  • multiscale_run/data/config/rat_sscxS1HL_V6/simulation_config.json

  • multiscale_run/data/config/rat_sscxS1HL_V10/simulation_config.json

  1. Ensure the Sphinx documentation is up-to-date. The fastest way is to check the artifacts of the docs stage in the CI.

  2. Ensure the Releases Notes section is completed for this version.

  3. Ensure the list of authors is up-to-date.

  4. Ensure no spurious files were added to the repository by mistake (log files, process core dumps, …)

  5. Ensure the source distribution can be built: python -m build --sdist .

  6. Ensure the CI/CD pipeline passes on the main branch. Start one manually if necessary.

  7. Create a git tag named after the new version, for instance: git tag 0.7

  8. Push the tag: git push origin --tags. This operation triggers a CI/CD pipeline that builds and tests the package, and upload the new documentation to the BBP Software Catalog. The documentation will appear the next day.

  9. Create a new pull-request to the BlueBrain/spack GitHub repository mentioning the new version in the py-multiscale-run Spack package.

  10. Ensure that the bbp workflow still works with the new version.

How to rebuild the shared Julia environment on BB5?

A Julia environment providing all the packages required to execute the Metabolism model is available on BB5 at the following location /gpfs/bbp.cscs.ch/project/proj12/jenkins/subcellular/multiscale_run/julia-environment. By default, the command multiscale-run init uses this directory rather than creating a new Julia environment (which takes approximately 10min).

When this shared Julia environment becomes out of date (newer Julia version or newer packages), then it is required to recreate it.

Prerequisite: access to BBP project proj12

  1. go to the shared folder:

    cd /gpfs/bbp.cscs.ch/project/proj12/jenkins/subcellular/multiscale_run/julia-environment

  2. load the MultiscaleRun module:

    module load unstable py-multiscale-run

  3. Setup a new simulation with a fresh Julia environment. Usually we name the Julia environment based on the day. For example:

    multiscale-run init --julia=create 2024-04-22

where 2024-04-22 is the name of the folder.

  1. Remove everything in the folder that is not .julia or .julia_environment.

  2. Create 2 symbolic links in the folder:

cd 2024-04-22
ln -s julia .julia
ln -s julia_environment .julia_environment

  1. Finally, link latest to this new folder (in /gpfs/bbp.cscs.ch/project/proj12/jenkins/subcellular/multiscale_run/julia-environment):

cd ..
ln -s 2024-04-22 latest

How to build the Sphinx documentation locally?

  1. Ensure the tox utility is available (see note above for installation)

  2. Build the HTML documentation : tox -e docs

  3. Open the generated documentation created in: ./docs/build/html/index.html

Note

Troubleshooting if the build fails

By default, the creation of the documentation is canceled if at least one error occurs. In case of unsuccessful build, either fix the issues reported by Sphinx to the console or update tox.ini to ignore these errors.

diff --git a/tox.ini b/tox.ini
index 0796eba..4774331 100644
--- a/tox.ini
+++ b/tox.ini
@@ -12,7 +13,7 @@ deps =
     sphinx-mdinclude
     mistune<3 # there is a conflict with nbconvert
-commands = sphinx-build -W --keep-going docs docs/build/html
+commands = sphinx-build docs docs/build/html

Anyway, the continuous-integration process requires the build of the documentation to pass without error.

Source

contributing.rst