{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CI\n",
    "At first, you'll want to write your tests locally, and test them against as many local browsers as possible. However, to really test out your features, you'll want to:\n",
    "\n",
    "- run them against as many **real** browsers on other operating systems as possible\n",
    "- have easy access to human- and machine-readable test results and build assets\n",
    "- integration with development tools like GitHub\n",
    "\n",
    "Enter Continuous Integration (CI). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Providers: Cloud"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-Provider\n",
    "Historically, Jupyter projects have used a mix of free-as-in-beer-for-open source hosted services:\n",
    "- [Appveyor](https://www.appveyor.com) for Windows\n",
    "- [Circle-CI](https://circleci.com) for Linux\n",
    "- [TravisCI](https://travis-ci.org) for Linux and MacOS\n",
    "\n",
    "Each brings their own syntax, features, and constraints to building and maintaining robust CI workflows.\n",
    "\n",
    "> `JupyterLibrary` started on Travis-CI, but as soon as we wanted to support more platforms and browsers..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Azure Pipelines\n",
    "At the risk of putting all your eggs in one (proprietary) basket, [Azure Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/) provides a single-file approach to automating all of your tests against reasonably modern versions of browsers. \n",
    "\n",
    "> `JupyterLibrary` was formerly built on Azure, and looking through [pipeline][] and various [jobs and steps][] shows some evolving approaches...\n",
    "\n",
    "[pipeline]: https://github.com/robots-from-jupyter/robotframework-jupyterlibrary/blob/v0.2.0/azure-pipelines.yml\n",
    "[jobs and steps]: https://github.com/robots-from-jupyter/robotframework-jupyterlibrary/tree/v0.2.0/ci"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Github Actions\n",
    "At the risk of putting all your eggs in one (proprietary) basket, if your code is on Github, [Github Actions](https://github.com/features/actions) offers the tightest integration, requiring no aditional accounts.\n",
    "\n",
    "> `JupyterLibrary` is itself built on Github Actions, and looking at the [workflows][] offers some of the best patterns we have found.\n",
    "\n",
    "[workflows]: https://github.com/robots-from-jupyter/robotframework-jupyterlibrary/blob/main/.github/workflows/ci.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Providers: On-Premises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jenkins\n",
    "If you are working on in-house projects, and/or have the ability to support it, [Jenkins](https://jenkins.io) is the gold standard for self-hosted continuous integration. It has almost limitless configurability, and commercial support is available.\n",
    "\n",
    "- [`warnings-ng`](https://plugins.jenkins.io/warnings-ng/) can consume many outputs of `robotframework`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: Environment management\n",
    "\n",
    "Acceptance tests need benefit from tightly-controlled, but flexibly-defined environments. \n",
    "\n",
    "- this repo uses (and recommends) `conda-lock` and `mamba` to manage multiple environments\n",
    "- simpler cases, such as pure-python projects, can use [`tox`](https://github.com/tox-dev/tox)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: It's Just Scripts\n",
    "No matter how shiny or magical your continuous integration tools appear, the long-term well-being of your repo depends on techniques that are: \n",
    "- simple\n",
    "- cross-platform\n",
    "- as close to real browsers as possible\n",
    "- easily reproducible outside of CI\n",
    "\n",
    "Practically, since this is Jupyter, this boils down to putting as much as possible into platform-independent python (and, when neccessary, nodejs) code. \n",
    "\n",
    "> `JupyterLibrary` uses [doit][] to manage a relatively complex lifecycle across multiple environments with minimal CLI churn.\n",
    ">\n",
    "> - `doit` has very few runtime dependencies, and works well with caching, etc.\n",
    "> \n",
    "> Environment variables are used for feature flags\n",
    ">\n",
    ">  - aside from some inevitable path issues, environment variables are easy to migrate onto another CI provider\n",
    ">\n",
    "> A small collection of development [scripts][], not shipped as part of the distribution, provide some custom behaviors around particularly complex tasks.\n",
    "> \n",
    "> - sometimes `doit` is too heavy of a hammer for delicate work\n",
    "\n",
    "[scripts]: https://github.com/robots-from-jupyter/robotframework-jupyterlibrary/tree/main/scripts\n",
    "[doit]: https://github.com/robots-from-jupyter/robotframework-jupyterlibrary/blob/main/dodo.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: Single Test Script\n",
    "\n",
    "Having a single command that runs _all_ unit, integration, and acceptance tests is a useful property of a project.\n",
    "\n",
    "- `make` (or the more pythonic [`doit`](https://github.com/pydoit/doit), used in this repo) make this most robust\n",
    "  - usually, all _unit_ tests need to be re-run when any functional source, e.g. `*.ts` and `*.py`\n",
    "  - acceptance tests often need to be run when almost _anything_ changes, including `.css`, build configuration files, etc.\n",
    "- wrap `robot` execution in another tool\n",
    "  - for example, [jupyter-server-proxy] launches `robot` from within `pytest`\n",
    "  - use [`tox`](https://github.com/tox-dev/tox) for pure-python test management \n",
    "  \n",
    "[jupyter-server-proxy]: https://github.com/jupyterhub/jupyter-server-proxy/blob/v3.2.2/tests/acceptance/test_acceptance.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: Log Centralization\n",
    "\n",
    "After a full test run, it can be useful to combine many test results into a single, navigable page\n",
    "\n",
    "- in CI, download all the test result archives and put them together\n",
    "  - [`rebot`][rebot] can combine multiple runs, including retries, into a single HTML report\n",
    "- embed different kinds of results\n",
    "  - [`pytest-html`][pytest-html] can embed generated reports\n",
    "    - when embedding `robot` reports with screenshots, use [`Set Screenshot Directory   EMBED`][embed] to make this easier\n",
    "    - other files like logs can also be embedded\n",
    "- create a single log aggregation HTML page\n",
    "    - [jupyterlab-deck] generates and publishes a notebook/slideshow containing all of its logs\n",
    "        - this is served as a [JupyterLite] site, so the underlying (semi-)machine-readable is also available to \n",
    "\n",
    "[rebot]: https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#using-rebot\n",
    "[pytest-html]: https://github.com/pytest-dev/pytest-html\n",
    "[embed]: https://robotframework.org/SeleniumLibrary/SeleniumLibrary.html#Set%20Screenshot%20Directory\n",
    "[jupyterlab-deck]: https://deathbeds.github.io/jupyterlab-deck\n",
    "[jupyterlite]: https://github.com/jupyterlite/jupyterlite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: Caching\n",
    "Most of the CI providers offer nuanced approaches to caching files. Things to try caching (it doesn't always help):\n",
    "- packages/metadata for your package manager, e.g. `conda`, `pip`, `yarn`\n",
    "- built web assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: Pay technical debt forward\n",
    "\n",
    "A heavy CI pipeline can become necessary to manage many competing concerns. Each non-trivial, browser-based robot test can easily cost tens of seconds. Some approaches:\n",
    "- use an up-front dry-run `robot` test\n",
    "  - this can help catch whitespace errors in robot syntax\n",
    "  - this usually costs $\\frac{\\sim1}{100}$ the time of running the full test\n",
    "- run tests in subsets, in parallel, and in random order with [`pabot`](https://github.com/mkorpela/pabot)\n",
    "  - requires avoiding shared resources, e.g. network ports, databases, logfiles\n",
    "    - if neccessary, declare explicit dependencies with e.g. [`DependencyLibrary`](https://pypi.org/project/robotframework-dependencylibrary) or [`pabot`'s `#DEPENDS`](https://github.com/mkorpela/pabot#controlling-execution-order-and-level-of-parallelism)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Approach: Get More Value\n",
    "\n",
    "While the pass/fail results of a test are useful in their own right, acceptance tests can provide useful artifacts for other project goals.\n",
    "\n",
    "- gather additional coverage insrumentation\n",
    "  - [x] **client**:\n",
    "    - [x] [jupyterlab-deck][deck-cov] uses `istanbul` and `nyc` to collect browser code coverage\n",
    "  - [x] **kernel** and **widgets**:\n",
    "    - [x] this repo gathers kernel coverage from JupyterLab-based tests iof its custom `%%robot` [IPython magic][ipython-magic]\n",
    "    - [x] [ipyforcegraph](https://github.com/jupyrdf/ipyforcegraph/pull/89) tests custom [Jupyter widgets][widgets]\n",
    "  - [ ] _**serverextension**: TODO_\n",
    "  - [ ] _**`.robot` suites**: TODO_\n",
    "- use generated screenshots\n",
    "  - [ ] _**reporting**: TODO_\n",
    "  - [ ] _**accessibility**: TODO_ \n",
    "  - [ ] _**documentation**: TODO_\n",
    "  - [ ] _**PDF generation**: TODO_\n",
    "    - [ ] revisit when supported by [`geckodriver`](https://github.com/mozilla/geckodriver/issues/2001)  \n",
    "\n",
    "[deck-cov]: https://deathbeds.github.io/jupyterlab-deck/files/nyc/index.html\n",
    "[deck-screens]: https://deathbeds.github.io/jupyterlab-deck/lab/index.html?path=README.ipynb\n",
    "[ipython-magic]: https://ipython.readthedocs.io/en/stable/interactive/magics.html\n",
    "[widgets]: https://github.com/jupyter-widgets/ipywidgets"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}