.. title: Package and Distribute Your Test Suite
.. slug: package-and-distribute-your-test-suite
.. date: 2016-02-23 22:26:33 UTC
.. updated: 2016-02-23 22:26:33 UTC
.. tags: testing
.. category:
.. link:
.. description:
.. type: text

Automated test suites must be packaged for installation and distributed just
like your other build artifacts. They are not some special things that live in
a code repository (repo) separate from the rest of your code.

My team was writing tests and keeping them all in a git repo. The repo would be
cloned and then PyTest would run these tests. All users of tests - from
developers to Continuous Integration (CI) and everyone else in between - would
run steps akin to::

    git clone git@github.com:username/testrepo.git
    cd testrepo
    git checkout main
    pip install -r requirements.txt
    py.test tests/

This worked for a while but was not a good fit as time progressed
for these reasons:

.. TEASER_END: Read more

* All users had to muck around with git
* There was no way to release the test suite at release time
* Only the current state of tests could be used with no good option to rollback to a previous state

My team set out to create a process more like:

* Release automated test suite as an installable artifact from CI
* All users do a ``pip install --upgrade testsuite`` and execute script ``run_test_suite``

When this new process was implemented it gave us many benefits:

* Tests are released as installable Python packages multiple times a day
* Each project - production code or tests - is treated the same and goes through the same CI pipeline; no special snowflakes
* The test suite itself is run through CI more easily and accepted or rejected
* Teams can leverage CI to create custom test suites
* During product release tests get released as well
* It's easy to pick and choose the state of test suite to use based on release versioning
* Users can install and uninstall test suites just like any other software; less cognitive load

Based on my experience I highly recommend your teams to write automated test
suites to be packaged, released, distributed, and installed like any other
software component.

P.S. I'm planning to write a more technical post (over on Code Ghar) on how
we release and use our automated test suite. I'll link to it here when it's
written.