<?xml version="1.0"?>
<page title="Building Cgreen test suites" here="Cgreen test suites">
    <long_title>
        Tutorial: C test suites using the Cgreen unit testing framework
    </long_title>
    <content>
        <p>
            Cgreen is a tool for building unit tests in the C language.
            These are usually written alongside the production code
            by the programmer to prevent bugs.
            Even though the test suites are created by software developers,
            they are intended to be human readable C code, as part of
            their function is an executable specification.
            Used in this way, the test harness delivers constant quality assurance.
        </p>
        <p>
            In other words you'll get less bugs.
        </p>
        <section name="writing" title="Writing basic tests">
            <p>
                Cgreen tests are simply C functions with no parameters
                and a <code>void</code> return value.
                An example might be...
<c><![CDATA[
static void <strong>strlen_of_hello_should_be_five</strong>() {
    <strong>assert_equal</strong>(strlen("Hello"), 5);
}
]]></c>
                The test function name can be anything you want.
                The <code>assert_equal()</code> call is an example
                of an assertion.
                Assertions send messages to Cgreen, which in turn
                outputs the results.
            </p>
            <p>
                Here are the standard assertions...
                <table>
                    <tr><th>Assertion</th><th>Description</th></tr>
                    <tr><td><code>assert_true(boolean)</code></td><td>Passes if boolean evaluates true</td></tr>
                    <tr><td><code>assert_false(boolean)</code></td><td>Fails if boolean evaluates true</td></tr>
                    <tr><td><code>assert_equal(first, second)</code></td><td>Passes if <code>first == second</code></td></tr>
                    <tr><td><code>assert_not_equal(first, second)</code></td><td>Passes if <code>first != second</code></td></tr>
                    <tr><td><code>assert_string_equal(char *, char *)</code></td><td>Uses <code>strcmp()</code> and passes if the strings are equal</td></tr>
                    <tr><td><code>assert_string_not_equal(char *, char *)</code></td><td>Uses <code>strcmp()</code> and fails if the strings are equal</td></tr>
                </table>
                The boolean assertion macros accept an <code>int</code> value.
                The equality assertions accept anything that can be cast to <code>intptr_t</code>
                and simply perform an <code>==</code> operation.
                The string comparisons are slightly different in that they use the
                <code><![CDATA[<string.h>]]></code> library function <code>strcmp()</code>.
                If <code>assert_equal()</code> is used on <code>char *</code> pointers
                then the pointers have to point at the same string.
            </p>
            <p>
                Each assertion has a default message comparing the two values.
                If you want to substitute your own failure messages, then you
                must use the <code>*_with_message()</code> counterparts...
                <table>
                    <tr><th>Assertion</th></tr>
                    <tr><td><code>assert_true_with_message(boolean, message, ...)</code></td></tr>
                    <tr><td><code>assert_false_with_message(boolean, message, ...)</code></td></tr>
                    <tr><td><code>assert_equal_with_message(first, second, message, ...)</code></td></tr>
                    <tr><td><code>assert_not_equal_with_message(first, second, message, ...)</code></td></tr>
                    <tr><td><code>assert_string_equal_with_message(char *, char *, message, ...)</code></td></tr>
                    <tr><td><code>assert_string_not_equal_with_message(char *, char *, message, ...)</code></td></tr>
                </table>
                All these assertions have an additional <code>char *</code> message parameter,
                which is the message you wished to display on failure.
                If this is set to <code>NULL</code>, then the default
                message is shown instead.
                The most useful assertion from this group is <code>assert_true_with_message()</code>
                as you can use that to create your own assertion functions with your
                own messages.
            </p>
            <p>
                Actually the assertion macros have variable argument lists.
                The failure message acts like the template in <code>printf()</code>.
                We could change the test above to be...
<c><![CDATA[
static void strlen_of_hello_should_be_five() {
    const char *greeting = "Hello";
    int length = strlen(greeting);
    assert_equal_with_message(length, 5,
            <strong>"[%s] should be 5, but was %d"</strong>, greeting, length);
}
]]></c>
                A slightly more user friendly message when things go wrong.
            </p>
            <p>
                For the test above to work there needs to be a running test suite.
                We can create one expecially for this test like so...
<c><![CDATA[
<strong>TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_should_be_five);
    return suite;
}</strong>
]]></c>
                In case you have spotted that <code>strlen_of_hello_should_be_five()</code>
                should have an ampersand in front of it, <code>add_test()</code>
                is a macro.
                The <code>&amp;</code> is added automatically.
            </p>
            <p>
                To run the test suite, we call <code>run_test_suite()</code> on
                it.
                This function cleans up the test suite after running it, so
                we can just write...
<c><![CDATA[
<strong>run_test_suite(our_tests(), create_text_reporter());</strong>
]]></c>
                The results of assertions are ultimately delivered as passes and
                failures to a collection of callbacks defined in a
                <code>TestReporter</code> structure.
                The only predefined one in Cgreen is the <code>TextReporter</code>
                that delivers messages in plain text.
            </p>
            <p>
                A full test script now looks like...
<c><![CDATA[
<strong>#include "cgreen/cgreen.h"
#include <string.h></strong>

static void strlen_of_hello_should_be_five() {
    assert_equal(strlen("Hello"), 5);
}

TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_should_be_five);
    return suite;
}

<strong>int main(int argc, char **argv) {
    return </strong>run_test_suite(our_tests(), create_text_reporter());<strong>
}</strong>
]]></c>
                The return value of <code>run_test_suite()</code> is a
                Unix exit code.
            </p>
            <p>
                Compliling and running gives...
<sh><![CDATA[
gcc -c strlen_test.c
gcc strlen_test.o cgreen/cgreen.a -o strlen_test
./strlen_test<strong>
Running "main"...
Completed "main": 1 pass, 0 failures, 0 exceptions.</strong>
]]></sh>
                The test messages are only shown on failure.
                If we break our test...
<c><![CDATA[
static void strlen_of_hello_should_be_five() {
    assert_equal(strlen(<strong>"Hiya"</strong>, 5);
}
]]></c>
                ...we'll get the helpful message...
<sh><![CDATA[
Running "main"...<strong>
Failure!: strlen_of_hello_should_be_five ->
        [5] shold be [4] at [strlen_test.c] line [6]</strong>
Completed "main": 0 passes, 1 failure, 0 exceptions.
]]></sh>
                Cgreen appends the location of the test failure to
                our error string.
            </p>
            <p>
                Once we have a basic test scaffold up, it's pretty easy to
                add more tests.
                Adding a test of <code>strlen()</code> with an empty string
                for example...
<c><![CDATA[
...<strong>
static void strlen_of_empty_string_should_be_zero() {
    assert_equal(strlen("\0"), 0);
}</strong>

TestSuite *<strong>our_tests</strong>() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_should_be_five);<strong>
    add_test(suite, strlen_of_empty_string_should_be_zero);</strong>
    return suite;
}
...
]]></c>
                And so on.
            </p>
        </section>
        <section name="fixtures" title="Set up and tear down">
            <p>
                It's common for test suites to have a lot of duplicate code,
                especially when setting up similar tests.
                Take this database code for example...
<c><![CDATA[
#include "cgreen/cgreen.h"
#include <stdlib.h>
#include <mysql/mysql.h>
#include "person.h"
<strong>
static void create_schema() {
    MYSQL *connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
    mysql_query(connection, "create table people (name, varchar(255) unique)");
    mysql_close(connection);
}

static void drop_schema() {
    MYSQL *connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
    mysql_query(connection, "drop table people");
    mysql_close(connection);
}</strong>

static void can_add_person_to_database() {
    <strong>create_schema();</strong>
    Person *person = create_person();
    set_person_name(person, "Fred");
    save_person(person);
    Person *found = find_person_by_name("Fred");
    assert_string_equal(get_person_name(person), "Fred");
    <strong>drop_schema();</strong>
}

static void cannot_add_duplicate_person() {
    <strong>create_schema();</strong>
    Person *person = create_person();
    set_person_name(person, "Fred");
    assert_true(save_person(person));
    Person *duplicate = create_person();
    set_person_name(duplicate, "Fred");
    assert_false(save_person(duplicate));
    <strong>drop_schema();</strong>
}

TestSuite *person_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, can_add_person_to_database);
    add_test(suite, cannot_add_duplicate_person);
    return suite;
}

int main(int argc, char **argv) {
    return run_test_suite(person_tests(), create_text_reporter());
}
]]></c>
                We have already factored out the duplicate code into it's
                own functions <code>create_scheme()</code> and <code>drop_schema()</code>,
                so things are not so bad.
                At least not yet.
                What happens when we get dozens of tests?
                For a test subject as compicated as a database
                <a href="http://www.martinfowler.com/eaaCatalog/activeRecord.html">ActiveRecord</a>,
                having dozens of tests is very likely.
            </p>
            <p>
                We can get Cgreen to do some of the work for us by
                declaring these methods as <code>setup()</code> and
                <code>teardown()</code> functions in the test suite.
            </p>
            <p>
                Here is the new version...
<c><![CDATA[
...
static void <strong>create_schema()</strong> { ... }

static void <strong>drop_schema()</strong> { ... }

static void can_add_person_to_database() {
    Person *person = create_person();
    set_person_name(person, "Fred");
    save_person(person);
    Person *found = find_person_by_name("Fred");
    assert_string_equal(get_person_name(person), "Fred");
}

static void cannot_add_duplicate_person() {
    Person *person = create_person();
    set_person_name(person, "Fred");
    assert_true(save_person(person));
    Person *duplicate = create_person();
    set_person_name(duplicate, "Fred");
    assert_false(save_person(duplicate));
}

TestSuite *person_tests() {
    TestSuite *suite = create_test_suite();<strong>
    setup(suite, create_schema);
    teardown(suite, drop_schema);</strong>
    add_test(suite, can_add_person_to_database);
    add_test(suite, cannot_add_duplicate_person);
    return suite;
}
...
]]></c>
                With this new arrangement Cgreen runs the <code>create_schema()</code>
                function before each test, and the <code>drop_schema()</code>
                function after each test.
                This saves some repetitive typing and reduces the chance of accidents.
                It also makes the tests more focused.
            </p>
            <p>
                The reason we try so hard to strip everything out of
                the test functions is that that the test suite acts
                as documentation.
                In our <em>person.h</em> example we can easily see that
                <code>Person</code> has some kind of name property, and
                that this value must be unique.
                For the tests to act like a readable specification we have
                to remove as much mechanical clutter as we can.
            </p>
            <p>
                A couple of details.
                Currently only one <code>setup()</code> and <code>teardown()</code>
                may be added to each <code>TestSuite</code>.
                Also the <code>teardown()</code> function may not be run if the
                test crashes, causing some test interference.
                This brings us nicely onto the next section...
            </p>
        </section>
        <section name="fork" title="Each test in it's own process">
            <p>
                Consider this test method...
<c><![CDATA[<strong>
void will_seg_fault() {
    int *p = NULL;
    (*p)++;
}</strong>
]]></c>
                Crashes are not something you would normally want to have
                in a test run.
                Not least because it will stop you receiving the very test output
                you need to tackle the problem.
            </p>
            <p>
                To prevent segmentation faults and other problems bringing
                down the test suites, Cgreen runs every test in it's
                own process.
            </p>
            <p>
                Just before the <code>setup()</code> call, Cgreen <code>fork()</code>'s.
                The main process wait's for the test to complete normally or die.
                This includes the <code>teardown()</code>.
                If the test process dies, an exception is reported and the main
                test process carries on.
            </p>
            <p>
                For example...
<c><![CDATA[
#include "cgreen/cgreen.h"
#include <stdlib.h>
<strong>
static void will_seg_fault() {
    int *p = NULL;
    (*p)++;
}</strong>

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test(suite, <strong>will_seg_fault</strong>);
    run_test_suite(suite, create_text_reporter());
}
]]></c>
                When built and run, this gives...
<sh><![CDATA[
Running "main"...
Exception!: will_seg_fault -> Test "will_seg_fault" failed to complete
Completed "main": 0 passes, 0 failures, 1 exception.
]]></sh>
                The obvious thing to do now is to fire up the debugger.
                Unfortunately, the constant <code>fork()</code>'ing of
                Cgreen can be an extra complication too many when debugging.
                It's enough of a problem to find the bug.
            </p>
            <p>
                To get around this, and also to allow the running of
                one test at a time, Cgreen has the <code>run_single_test()</code>
                function.
                The signatures of the two run methods are...
<ul class="api">
<li><code><![CDATA[int run_test_suite(TestSuite *suite, TestReporter *reporter);]]></code></li>
<li><code><![CDATA[int run_single_test(TestSuite *suite, char *test, TestReporter *reporter);]]></code></li>
</ul>
                The extra parameter of <code>run_single_test()</code>, the
                <code>test</code> string, is the name of the test to select.
                This could be any test, even in nested test suites (see below).
            </p>
            <p>
                Here is how we would use it to debug our crashing test...
<c><![CDATA[
int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test(suite, will_seg_fault);
    <strong>run_single_test(suite, "will_seg_fault", create_text_reporter());</strong>
}
]]></c>
                When run in this way, Cgreen will not <code>fork()</code>.
            </p>
            <p>
                This deals with the segmentation fault case, but what about a process that fails
                to complete by getting stuck in a loop?
            </p>
            <p>
                Well, Cgreen will wait forever too.
                Using the C signal handlers, we can place a time limit on the
                process by sending it an interrupt.
                To save us writing this ourselves, Cgreen includes the
                <code>die_in()</code> function to help us out.
            </p>
            <p>
                Here is an example of time limiting a test...
<c><![CDATA[
...
static void will_seg_fault() { ... }
<strong>
static void this_would_stall() {
    die_in(1);
    while(0 == 0) { }
}</strong>

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test(suite, will_seg_fault);
    add_test(suite, <strong>this_would_stall</strong>);
    run_test_suite(suite, create_text_reporter());
}
]]></c>
                When executed, the code will slow for a second, and
                then finish with...
<sh><![CDATA[
Running "main"...
Exception!: will_seg_fault -> Test "will_seg_fault" failed to complete
Exception!: will_stall -> Test "this_would_stall" failed to complete
Completed "main": 0 passes, 0 failures, 2 exceptions.
]]></sh>
                Note that you see the test results as they come in.
                Cgreen streams the results as they happen, making it easier
                to figure out where the test suite has problems.
            </p>
            <p>
                Of course, if you want to set a general time limit on all
                your tests, then you can add a <code>die_in()</code> to
                a <code>setup()</code> function.
                Cgreen will then apply the limit to all of them.
            </p>
        </section>
        <section name="suites" title="Building composite test suites">
            <p>
                The <code>TestSuite</code> is a composite structure.
                This means test suites can be added to test suites, building
                a tree structure that will be executed in order.
            </p>
            <p>
                Let's combine the <code>strlen()</code> tests with the
                <code>Person</code> tests above.
                Firstly we need to remove the <code>main()</code> calls.
                E.g...
<c><![CDATA[
#include "cgreen/cgreen.h"
#include <string.h>

static void strlen_of_hello_should_be_five() { ... }
static void strlen_of_empty_string_should_be_zero() { ... }

TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_should_be_five);
    add_test(suite, strlen_of_empty_string_should_be_zero);
    return suite;
}
]]></c>
                Then we can write a small runner script with a new
                <code>main()</code> function...
<c><![CDATA[
#include "strlen_tests.c"
#include "person_tests.c"
<strong>
TestSuite *our_tests();
TestSuite *person_tests();</strong>

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_suite(suite, our_tests());
    add_suite(suite, person_tests());<strong>
    if (argc > 1) {</strong>
        return run_single_test(suite, <strong>argv[1]</strong>, create_text_reporter());<strong>
    }</strong>
    return run_test_suite(suite, create_text_reporter());
}
]]></c>
                It's usually easier to place the <code>TestSuite</code>
                prototypes in the runner
                scripts, rather than have lot's of header files.
                This is the same reasoning that let us drop the prototypes
                for the test functions in the actual test scripts.
                We can get away with this, because the tests are more about
                documentation than encapsulation.
            </p>
            <p>
                It's sometimes handy to be able to run just a single test
                from the command line, so we added a simple <code>if</code>
                block to take the test name as an optional argument.
                The entire test suite will be searched for the named
                test.
                This trick also saves us a recomplile when we debug.
            </p>
            <p>
                We've placed each test suite in it's own file, but that
                is not necessary.
                We could build several test suites in the same file, even
                nesting them.
                We can even add mixtures of test functions and test suites
                to the same parent test suite.
                Loops will give trouble, however.
            </p>
            <p>
                If we do place several suites in the same file, then
                all the suites will be named the same
                in the breadcrumb trail in the test message.
                They will all be named after the function the create call sits in.
                If you want to get around this, or you just like to name
                your test suites, you can use <code>create_named_test_suite()</code>
                instead of <code>create_test_suite()</code>.
                This takes a single string parameter.
                In fact <code>create_test_suite()</code> is just a macro that
                inserts the <code>__func__</code> constant into
                <code>create_named_test_suite()</code>.
            </p>
            <p>
                What happens to <code>setup()</code> and <code>teardown()</code>
                in a <code>TestSuite</code> that contains other
                <code>TestSuite</code>s?
            </p>
            <p>
                Well firstly, Cgreen does not <code>fork()</code> when running
                a suite.
                It leaves it up to the child suite to <code>fork()</code>
                the individual tests.
                This means that a <code>setup()</code> and <code>teardown()</code>
                will run in the main process.
                They will be run once for each child suite.
            </p>
            <p>
                We can use this to speed up our <code>Person</code> tests
                above.
                Remember we were creating a new connection and closing it
                again in the fixtures.
                This means opening and closing a lot of connections.
                At the slight risk of some test interference, we could
                reuse the connection accross tests...
<c><![CDATA[
...
static MYSQL *connection;

static void create_schema() {<strong>
    mysql_query(connection, "create table people (name, varchar(255) unique)");</strong>
}

static void drop_schema() {<strong>
    mysql_query(connection, "drop table people");</strong>
}

static void can_add_person_to_database() { ... }
static void cannot_add_duplicate_person() { ... }
<strong>
void open_connection() {
    connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
}

void close_connection() {
    mysql_close(connection);
}
</strong>
TestSuite *person_tests() {
    TestSuite *suite = create_test_suite();
    setup(suite, create_schema);
    teardown(suite, drop_schema);
    add_test(suite, can_add_person_to_database);
    add_test(suite, cannot_add_duplicate_person);
<strong>
    TestSuite *fixture = create_named_test_suite("Mysql fixture");
    add_suite(fixture, suite);
    setup(fixture, open_connection);
    teardown(fixture, close_connection);
    return fixture;</strong>
}
]]></c>
                The trick here is creating a test suite as a wrapper
                whose sole purpose to wrap the main test suite in
                the fixture.
                This is our <code>fixture</code> pointer.
                This code is a little confusing, because we have two sets of
                fixtures in the same test script.
            </p>
            <p>
                We have the MySQL connection fixture.
                This is runs <code>open_connection()</code> and
                <code>close_connection()</code> just
                once at the beginning and end of the person tests.
                This is because the <code>suite</code> pointer is the
                only member of <code>fixture</code>.
            </p>
            <p>
                We also have the schema fixture, the <code>create_schema()</code>
                and <code>drop_schema()</code>, which is run before
                and after every test.
                Those are still attached to the inner <code>suite</code>.
            </p>
            <p>
                In the real world we would probably place the connection
                fixture in it's own file...
<c><![CDATA[
static MYSQL *connection;

MYSQL *get_connection() {
    return connection;
}

static void open_connection() {
    connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
}

static void close_connection() {
    mysql_close(connection);
}
<strong>
TestSuite *connection_fixture(TestSuite *suite) {</strong>
    TestSuite *fixture = create_named_test_suite("Mysql fixture");
    add_suite(fixture, suite);
    setup(fixture, open_connection);
    teardown(fixture, close_connection);
    return fixture;<strong>
}</strong>
]]></c>
                This allows the reuse of common fixtures across projects.
            </p>
        </section>
    </content>
    <internal>
        <link>
            Writing a <a href="#writing">Cgreen unit test</a>.
        </link>
        <link>
            The <a href="#fixtures">setup() and teardown()</a> fixtures.
        </link>
        <link>
            <a href="#fork">Each test in it's own process</a>.
        </link>
        <link>
            Creating a <a href="#suites">composite test harness</a>.
        </link>
    </internal>
    <external>
        <link>
            The <a href="http://sourceforge.net/projects/cgreen">Cgreen SourceForge page</a>.
        </link>
        <link>
            Unit testing was popularised by <a href="http://extremeprogramming.org">eXtreme Programming</a>.
        </link>
        <link>
            Cgreen is heavily influenced by the <a href="http://check.sourceforge.net/">Check</a> unit tester.
        </link>
    </external>
    <meta>
        <keywords>
            unit testing, test-first, test first, code quality, automate unit testing,
            automated testing, c programming, c tools, c unit tests, unit testing in c,
            check unit tester, test coverage, TDD, extreme programming, agile development,
            tdd in C, c regression testing, cunit, cppunit, c unit testing frameworks, sourceforge,
            open source, open source framework, gnu licence, lgpl, software testing,
            executable specification, software requirements, quality assurance,
            automated software testing for c development teams, software quality assurance
        </keywords>
    </meta>
</page>