#+Title: Testing Examples

This is the place to have examples in =core/examples= tested.

* Outputs
Before running a test the optional commands found between =#+begin_src prep= and
=#+end_src= are run to prepare the environment for the test. This is commonly
used to prepare files for the example to operate on.

The =outputs= test checks to see that the output generated by the test is
correct as compared with the content between the =#+begin_src example_output=
and =#+end_src= tags.

*NOTE:* Agent output should be *identical* between a normal execution and a dry
run. This means that some examples are poor candidates for automatic testing.

** Troubleshooting Outputs Test Failures
This section is primarily intended for CFEngine Staff as not all build output is
publicly available.

It is *very* important for the examples output to be consistent or it will
result in a failed test. Example test failures are somewhat opaque and can be
difficult to troubleshoot.

A failure will look like this in the build system:

#+BEGIN_EXAMPLE
00:13:23.974 ./04_examples/outputs/check_outputs.cf FAIL (UNEXPECTED FAILURE)
#+END_EXAMPLE

*Note:* The specific example that failed is not immediately apparent. You must
wait for the job to finish and then drill into the test results.

Where you might be looking for something like this:

#+BEGIN_EXAMPLE
   error: Finished command related to promiser "/usr/bin/perl /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/remake_outputs.pl --cfagent="/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/bin/cf-agent" --workdir=/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/tmp/TESTDIR.cfengine -c -v /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/multiple_outcomes.cf" -- an error occurred, returned 1
  notice: Q: "...r/bin/perl /hom": Test file: /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/multiple_outcomes.cf
Q: "...r/bin/perl /hom": Command: "/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/bin/cf-agent" -D_cfe_output_testing -nKf /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/tmp/TESTDIR.cfengine/multiple_outcomes.cf 2>&1
Q: "...r/bin/perl /hom": NEW OUTPUT: [[[ warning: Warning promised, need to create file "/tmp/repaired_and_not_kept.txt"
Q: "...r/bin/perl /hom": R: My promise was not kept because the template failed to render.
Q: "...r/bin/perl /hom": R: Found class: "outcome_failed"
Q: "...r/bin/perl /hom": R: Found class: "outcome_not_kept"
Q: "...r/bin/perl /hom": R: Found class: "outcome_error"
Q: "...r/bin/perl /hom": R: Found class: "outcome_reached"
Q: "...r/bin/perl /hom": ]]]
Q: "...r/bin/perl /hom": --- /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/tmp/TESTDIR.cfengine/multiple_outcomes.cf.a	2016-04-12 16:16:28.247136634 +0200
Q: "...r/bin/perl /hom": +++ /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/tmp/TESTDIR.cfengine/multiple_outcomes.cf.b	2016-04-12 16:16:28.247136634 +0200
Q: "...r/bin/perl /hom": @@ -1,10 +1,6 @@
Q: "...r/bin/perl /hom": -error: Template file "$(this.promsie_filename).broken_mustache" could not be opened for reading
Q: "...r/bin/perl /hom": -R: My promise was repaired because the file was created.
Q: "...r/bin/perl /hom": +warning: Warning promised, need to create file "/tmp/repaired_and_not_kept.txt"
Q: "...r/bin/perl /hom":  R: My promise was not kept because the template failed to render.
Q: "...r/bin/perl /hom": -R: My promise was kept because the permissions were as desired.
Q: "...r/bin/perl /hom":  R: Found class: "outcome_failed"
Q: "...r/bin/perl /hom": -R: Found class: "outcome_repaired"
Q: "...r/bin/perl /hom": -R: Found class: "outcome_error"
Q: "...r/bin/perl /hom":  R: Found class: "outcome_not_kept"
Q: "...r/bin/perl /hom": +R: Found class: "outcome_error"
Q: "...r/bin/perl /hom":  R: Found class: "outcome_reached"
Q: "...r/bin/perl /hom": -R: Found class: "outcome_kept"
Q: "...r/bin/perl /hom": /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/multiple_outcomes.cf: output differs from original...
   error: Method "verbose_output" failed in some repairs
R: test_example: checker "/usr/bin/perl /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/remake_outputs.pl --cfagent="/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/bin/cf-agent" --workdir=/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/tmp/TESTDIR.cfengine -c /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/multiple_outcomes.cf" returned error
R: test_example: ran checker "/usr/bin/perl /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/remake_outputs.pl --cfagent="/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/bin/cf-agent" --workdir=/home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/workdir/__04_examples_outputs_check_outputs_cf/tmp/TESTDIR.cfengine -c /home/jenkins/workspace/testing-enterprise-pr_core/label/PACKAGES_x86_64_linux_redhat_6/core/tests/acceptance/../../examples/multiple_outcomes.cf"
   error: Method "test_example" failed in some repairs
#+END_EXAMPLE

Note how in the above the test failed because of differing output. In this
specific case, the differences are due to some difference in output between a
normal run and a dry-run where warnings are promised.
