CI#

At first, you’ll want to write your tests locally, and test them against as many local browsers as possible. However, to really test out your features, you’ll want to:

  • run them against as many real browsers on other operating systems as possible

  • have easy access to human- and machine-readable test results and build assets

  • integration with development tools like GitHub

Enter Continuous Integration (CI).

Providers: Cloud#

Multi-Provider#

Historically, Jupyter projects have used a mix of free-as-in-beer-for-open source hosted services:

Each brings their own syntax, features, and constraints to building and maintaining robust CI workflows.

JupyterLibrary started on Travis-CI, but as soon as we wanted to support more platforms and browsers…

Azure Pipelines#

At the risk of putting all your eggs in one (proprietary) basket, Azure Pipelines provides a single-file approach to automating all of your tests against reasonably modern versions of browsers.

JupyterLibrary was formerly built on Azure, and looking through pipeline and various jobs and steps shows some evolving approaches…

Github Actions#

At the risk of putting all your eggs in one (proprietary) basket, if your code is on Github, Github Actions offers the tightest integration, requiring no aditional accounts.

JupyterLibrary is itself built on Github Actions, and looking at the workflows offers some of the best patterns we have found.

Providers: On-Premises#

Jenkins#

If you are working on in-house projects, and/or have the ability to support it, Jenkins is the gold standard for self-hosted continuous integration. It has almost limitless configurability, and commercial support is available.

  • warnings-ng can consume many outputs of robotframework

Approach: Environment management#

Acceptance tests need benefit from tightly-controlled, but flexibly-defined environments.

  • this repo uses (and recommends) conda-lock and mamba to manage multiple environments

  • simpler cases, such as pure-python projects, can use tox

Approach: It’s Just Scripts#

No matter how shiny or magical your continuous integration tools appear, the long-term well-being of your repo depends on techniques that are:

  • simple

  • cross-platform

  • as close to real browsers as possible

  • easily reproducible outside of CI

Practically, since this is Jupyter, this boils down to putting as much as possible into platform-independent python (and, when neccessary, nodejs) code.

JupyterLibrary uses doit to manage a relatively complex lifecycle across multiple environments with minimal CLI churn.

  • doit has very few runtime dependencies, and works well with caching, etc.

Environment variables are used for feature flags

  • aside from some inevitable path issues, environment variables are easy to migrate onto another CI provider

A small collection of development scripts, not shipped as part of the distribution, provide some custom behaviors around particularly complex tasks.

  • sometimes doit is too heavy of a hammer for delicate work

Approach: Single Test Script#

Having a single command that runs all unit, integration, and acceptance tests is a useful property of a project.

  • make (or the more pythonic doit, used in this repo) make this most robust

    • usually, all unit tests need to be re-run when any functional source, e.g. *.ts and *.py

    • acceptance tests often need to be run when almost anything changes, including .css, build configuration files, etc.

  • wrap robot execution in another tool

Approach: Log Centralization#

After a full test run, it can be useful to combine many test results into a single, navigable page

  • in CI, download all the test result archives and put them together

    • rebot can combine multiple runs, including retries, into a single HTML report

  • embed different kinds of results

  • create a single log aggregation HTML page

    • jupyterlab-deck generates and publishes a notebook/slideshow containing all of its logs

      • this is served as a JupyterLite site, so the underlying (semi-)machine-readable is also available to

Approach: Caching#

Most of the CI providers offer nuanced approaches to caching files. Things to try caching (it doesn’t always help):

  • packages/metadata for your package manager, e.g. conda, pip, yarn

  • built web assets

Approach: Pay technical debt forward#

A heavy CI pipeline can become necessary to manage many competing concerns. Each non-trivial, browser-based robot test can easily cost tens of seconds. Some approaches:

  • use an up-front dry-run robot test

    • this can help catch whitespace errors in robot syntax

    • this usually costs $\frac{\sim1}{100}$ the time of running the full test

  • run tests in subsets, in parallel, and in random order with pabot

Approach: Get More Value#

While the pass/fail results of a test are useful in their own right, acceptance tests can provide useful artifacts for other project goals.

  • gather additional coverage insrumentation

    • [x] client:

      • [x] jupyterlab-deck uses istanbul and nyc to collect browser code coverage

    • [x] kernel and widgets:

    • [ ] serverextension: TODO

    • [ ] .robot suites: TODO

  • use generated screenshots

    • [ ] reporting: TODO

    • [ ] accessibility: TODO

    • [ ] documentation: TODO

    • [ ] PDF generation: TODO