========================
Checkbox Whitelist Files
========================

When creating a test suite for a particular purpose, it will be necessary to
specify which tests to run and which order they should run in. For this purpose
Checkbox provides the concept of Whitelists.

Whitelist Format
================

A whitelist is a text file containing a line-seperated sequence of patterns, 
each representing one or more 'jobs'. These patterns are in the Python regular 
expression syntax. Comments may be included in the file by starting the line 
with '#'.

Minimal Whitelist File
======================

In order to be useful a whitelist file needs to include a particular subset of
jobs which provide Checkbox with all of the information it needs to run tests 
properly. These include jobs which attach hardware information and resource 
jobs which provide other jobs with information of the environment they a
re running in (available hardware, available packages etc). To make this easy 
to do a single job exists whose purpose is to execute all of these other jobs::

    miscellanea/submission-resources

This should be included as the first job in any whitelist.

Job Categories
==============

In order to allow Checkbox to display jobs by category in the UI it is 
necessary to include a particular local job which itself generates jobs which 
belong to that category. This job will normally look like ``__<category>__`` 
where <category> is the name of the job file which contains the job. This is 
indicated again by the prefix of the job (before the ``/`` in the job name). 
As a quick example, the job ``graphics/glxgears`` is contained in 
``graphics.txt``. Therefore we should include the ``__graphics__`` job so that 
the ``graphics/glxgears`` job shows correctly under the category. The 
``__graphics__`` job itself looks like::

    name: __graphics__
    plugin: local
    _description: Graphics tests
    command:
      shopt -s extglob
      cat $CHECKBOX_SHARE/jobs/graphics.txt?(.in)

Checkbox will interpret this job as a request to display any job in 
``graphics.txt`` (or its untranslated version ``graphics.txt.in``) under the 
heading shown in the description of this job (in this case 'Graphics tests').

Tutorial
========

To compound what we discussed before, below is a brief tutorial which walks 
through assembling a basic whitelist file.

1. First we need to create a file, let's name it tutorial.whitelist. 
Whitelists don't have to end with the .whitelist suffix but this is the 
convention used to help identify them.
2. We start by adding the one job that is required for all whitelists, as 
explained above in the section 'Minimal Whitelist File', so our whitelist file 
looks like::

    miscellanea/submission-resources

3. Next we should choose some jobs that we want to run. This all depends on 
your specific use-case of course, but I've selected a few jobs that will help 
clearly illustrate more of the concepts involved in whitelists. These jobs will 
give us a whitelist file that looks like::

    miscellanea/submission-resources
    cpu/clocktest
    ethernet/multi_nic
    ethernet/multi_nic_eth0
    graphics/glxgears

 If we run this whitelist now then all of these jobs will be executed and a 
 valid test submission will be created, but we can still improve it in a couple 
 of ways.

4. The first way is by adding the necessary jobs to allow the Checkbox UI to 
group the jobs into specific categories. To do this we need to add a job with 
a name like ``__<category>__`` for each category. We have three categories in 
our whitelist file - cpu, ethernet and graphics. The category of the job is 
the prefix of the job name prior to the ``/``. So now our whitelist file looks 
like::

    miscellanea/submission-resources
    __cpu__
    __ethernet__
    __graphics__
    cpu/clocktest
    ethernet/multi_nic
    ethernet/multi_nic_eth0
    graphics/glxgears

  Now the Checkbox UI will group the jobs into these categories.

5. Although it's not immediately apparent there is another problem with this 
whitelist. The ``ethernet/multi_nic`` tests are only able to include one job 
for the ethernet port 'eth0'. It would be better if we included all of the 
jobs generated by 'ethernet/multi_nic', no matter how many ethernet ports are 
present on the system under test. The best way to do this is to write the 
pattern so that it matches all of the possible job names. We can take advantage 
of the Python regular expression syntax and use the ``\d`` special character 
to match any decimal number. After doing this the whitelist file will look 
like this::

    miscellanea/submission-resources
    __cpu__
    __ethernet__
    __graphics__
    cpu/clocktest
    ethernet/multi_nic
    ethernet/multi_nic_eth\d
    graphics/glxgears
