<chapter id="chapter.metrics">
  <title>Coverage Metrics</title>
  
  <para>
    Covered currently generates six types of code coverage metrics.
  </para>

  <sect1 id="section.metrics.line">
    <title>Line Coverage</title>
    
    <sect2 id="section.metrics.line.description">
      <title>Description</title>
      <para>
        Line coverage simply answers the question, "Was this line of code executed during simulation?"
      </para>
      <para>
        Covered will display the number of logical lines of code that exist in a particular file with the number of logical 
        lines that were executed during the simulation along with a percentage indicating the percentage of lines executed.
        If verbose mode is selected for a report, Covered will display the lines of logic that were not executed during the
        simulation run.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.line.example">
      <title>Example</title>
      <para>
        Suppose that the following module was specified in your DUT that is being analyzed:
      </para>
      <para>
        <example xml:id="example.metrics.line">
          <title>Line Coverage Code Sample</title>
          <programlisting lang="Verilog">
  module test;

    reg a, b, c;

    initial begin
      a = 0;    // Line 1
      b = 1;    // Line 2
      if( a )
        c = b;  // Line 3
      else
        c = ~b; // Line 4
    end

  endmodule
          </programlisting>
        </example>
      </para>
      <para>
        If this code were simulated as stated, the simulator would execute lines 1 and 2, and due to the value assigned 
        to a, line 4 would be executed. This means that during execution, out of a possibility of 4 lines, only 3 were 
        executed. Therefore, Covered would state that for this module the line coverage percentage would be 75% (or 3
        out of 4).
      </para>
    </sect2>
    
    <sect2 id="section.metrics.line.recommend">
      <title>Recommendations</title>
      <para>
        For a design to pass full coverage, it is recommended that the line coverage for all modules in a design receive
        100% coverage. If a line of logic is not executed during simulation, the design has not been fully exercised. Line
        coverage is useful for determining holes in the test suite.
      </para>
    </sect2>
  </sect1>
  
  <sect1 id="section.metrics.toggle">
    <title>Toggle Coverage</title>
    
    <sect2 id="section.metrics.toggle.description">
      <title>Description</title>
      <para>
        Toggle coverage answers the question, "Did this bit of this wire/register change from a value of zero (0) to one
        (1) and back from one (1) to zero (0) during simulation?"
      </para>
      <para>
        A bit is said to be fully covered when it toggles back and forth at least once. This metric does not indicate to
        the user that every value of a multi-bit vector was seen. For example, if we have a two bit vector called "foo",
        toggle coverage will not tell you that the value of foo was set to the values of 0, 1, 2 and 3. However, it will
        tell you that all bits in that vector were toggled back and forth.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.toggle.example">
      <title>Example</title>
      <para>
        Suppose that a DUT was comprised of the following module:
      </para>
      <para>
        <example xml:id="example.metrics.toggle">
          <title>Toggle Coverage Code Sample</title>
          <programlisting lang="Verilog">
  module test;

    reg [2:0] a;

    initial begin
      a = 3'b0;
      #10;
      a = 3'b110;
      #10;
      a = 3'b010;
      #10;
    end

  endmodule
          </programlisting>
        </example>
      </para>
      <para>
        The register called "a", after being simulated, would have achieved a total toggle percentage of 50% (or 3 out of
        6). Can you see which toggles are missing?
      </para>
      <para>
        Bit 0 has never toggled to 1 (and has never been at a value of 1 and toggled to 0); therefore, bit-0 has toggled
        0% (0 out of 2). Bit 1 has toggled from a value of 0 to 1, but has not toggled from a value of 1 to 0; therefore,
        bit-1 has toggled 50% (0 out of 2). Bit 2 is the only bit that has fully toggled, achieving a toggle percentage of
        100% (2 out of 2). If you add all of the possible toggles to the number of achieved toggles, you end up with 3 out
        of 6 (or 50% total toggle coverage).
      </para>
    </sect2>
    
    <sect2 id="section.metrics.toggle.recommend">
      <title>Recommendations</title>
      <para>
        For a design to pass full coverage, it is recommended that the toggle coverage for all modules in a design received
        100% coverage. If a bit is never changes value, it is usually an indication that a mode is not being exercised in
        the design or a datapath has a stuck-at issue.
      </para>
    </sect2>
  </sect1>
		
  <sect1 id="section.metrics.memory">
    <title>Memory Coverage</title>
    
    <sect2 id="section.metrics.memory.description">
      <title>Description</title>
      <para>
        Memory coverage answers a number of questions about memories/multi-dimensional arrays being used in your design, including 
        the following:
      </para>
      <para>
        <orderedlist>
          <listitem><para>Did all bits of each addressable memory element toggle from 0 to 1?</para></listitem>
          <listitem><para>Did all bits of each addressable memory element toggle from 1 to 0?</para></listitem>
          <listitem><para>Did every addressable memory element get written?</para></listitem>
          <listitem><para>Did every addressable memory element get read?</para></listitem>
        </orderedlist>
      </para>
      <para>
        Memories/multi-dimensional arrays can have two types of dimensions, packed and unpacked. Packed dimensional information is 
        specified to the left of an array declaration while unpacked dimensional information is specified to the right of an array 
        declaration. Consider the following example:
      </para>
      <para>
        <example xml:id="example.metrics.memory">
          <title>Memory Coverage Code Sample</title>
          <programlisting lang="Verilog">
  reg [3:0][2:0] foo[0:15];
          </programlisting>
        </example>
      </para>
      <para>
        The memory "foo" consists of three dimensions. The first dimension is an unpacked dimension which specifies that there are 
        16 addressable memory elements (AME). The number of AMEs are determined by multiplying the unpacked dimension of an array. 
        The second two dimensions are packed dimensions, creating a total of 12 bits for each AME.
      </para>
      <para>
        Other than getting toggle coverage information for each AME, memory coverage also allows us to easily see which AMEs were 
        written and read during simulation. If all entries of a memory have not been written during a regression run, it could indicate 
        several shortcomings of either the testbench or a logical problem with the write control logic, including, but not limited to, 
        the following:
      </para>
      <para>
        <orderedlist>
          <listitem>
            <para>
              If no AMEs for a given memory were written, it could indicate that the testsuite was never able to get out of bypass mode 
              for that memory.
            </para>
          </listitem>
          <listitem>
            <para>
              If fewer AMEs for a given memory were written than were read, it could indicate that there is a read controller issue leading 
              to an underflow issue with the memory.
            </para>
          </listitem>
          <listitem>
            <para>
              For memories that act like stacks or use lowest/highest address first write selection scheme, indicates that the memory was 
              never completely filled (may miss out on things like buffer overflow errors).
            </para>
          </listitem>
          <listitem>
            <para>
              For memories that are written in a round-robin fashion, indicates that either not enough entries were written (could miss a 
              write pointer wrap bug) or that there is a logical issue by which the write pointer did not properly increment in a 
              round-robin fashion.
            </para>
          </listitem>
          <listitem>
            <para>
              If there are "holes" in the write coverage of the AMEs (that is an AME is found to not have been written while those above 
              and below it have been), this could indicate a corner case logic bug or other testsuite issue that needs to be addressed.
            </para>
          </listitem>
        </orderedlist>
      </para>
      <para>
        If all entries of a memory have not been read during a regression run, it could indicate several other shortcomings of either the
        testbench or a logical problem with the read control logic, including, but not limited to, the following:
      </para>
      <para>
        <orderedlist>
          <listitem>
            <para>
              If no AMEs for a given memory were read, it could indicate that the testsuite was never able to get out of bypass mode for 
              that memory.
            </para>
          </listitem>
          <listitem>
            <para>
              If fewer AMEs for a given memory were read than written, it could indicate that a resource allocation issue.
            </para>
          </listitem>
          <listitem>
            <para>
              For memories that are written in a round-robin fashion, indicates that either not enough entries were read (could miss a 
              read pointer wrap bug) or that there is a logical issue by which the read pointer did not properly increment in a 
              round-robin fashion.
            </para>
          </listitem>
          <listitem>
            <para>
              If there are "holes" in the read coverage of the AMEs (that is an AME is found to not have been read while those above and 
              below it have been), this could indicate a corner case logic bug or other testsuite issue that needs to be addressed.
            </para>
          </listitem>
        </orderedlist>
      </para>
      <para>
        Currently, Covered simulates all memories/multi-dimensional arrays and its coverage is automatically accumulated during the score 
        command. To see its report information, either specify the -m m option to the report command or bring up the GUI for interactive 
        analysis.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.memory.recommend">
      <title>Recommendations</title>
      <para>
        For a design to pass full coverage, it is recommended that the write/read memory coverage for all modules in a design receive 100% 
        coverage. If every AME was not both written and read during simulation, the design has not been fully exercised. Toggle coverage 
        per AME need not receive 100% since a memory element that is written once will not cause toggle coverage to change.
      </para>
    </sect2>
  </sect1>

  <sect1 id="section.metrics.logic">
    <title>Combinational Logic Coverage</title>
    
    <sect2 id="section.metrics.logic.description">
      <title>Description</title>
      <para>
        Combinational logic coverage answers the question, "What values did an expression (or subexpression) evaluate to (or not evaluate 
        to) during the course of the simulation?"
      </para>
      <para>
        This type of coverage is extremely useful in determining logical combinations of signals that were not tried during simulation, 
        exposing potential holes in verification.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.logic.example">
      <title>Example</title>
      <para>
        Suppose the DUT comprised of the following module:
      </para>
      <para>
        <example xml:id="example.metrics.logic">
          <title>Combinational Logic Coverage Code Sample</title>
          <programlisting lang="Verilog">
  module test;

    reg  a;
    reg  b;
    wire c;

    initial begin
      a = 1'b0;
      b = 1'b1;
      #10;
      a = 1'b1;
      #10;
    end

    assign c = a | b;

  endmodule
          </programlisting>
        </example>
      </para>
      <para>
        This module comprises of only one expression: a | b. If this module were simulated it would result in a combinational coverage 
        value of 50% (2 out of 4). Can you determine which cases are not being covered?
      </para>
      <para>
        The expression "a | b" can result in two values, 0 and 1, but can do so in four combinations:
      </para>
      <para>
        <orderedlist>
          <listitem><para>a = 0, b = 0, c = 0</para></listitem>
          <listitem><para>a = 0, b = 1, c = 1</para></listitem>
          <listitem><para>a = 1, b = 0, c = 1</para></listitem>
          <listitem><para>a = 1, b = 1, c = 1</para></listitem>
        </orderedlist>
      </para>
      <para>
        Noticing the values assigned to a and b during simulation, shows that combinations (2) and (4) were hit during execution while 
        combinations (1) and (3) were not (2 out of 4 - 50%). This also shows us that the value of c was never 0 during simulation.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.logic.recommend">
      <title>Recommendations</title>
      <para>
        For a design to pass full coverage, it is recommended that the combinational logic coverage for all modules be 80% or higher. 
        If the expression coverage for an expression is not 100%, it is recommended that the verification engineer closely examine these 
        missed cases to determine if more testing is required. Sometimes certain combinations of signals are unachievable due to design 
        constraints, keeping the expression coverage from ever reaching a value of 100% but still can be considered fully covered.
      </para>
    </sect2>
  </sect1>
  
  <sect1 id="section.metrics.fsm">
    <title>Finite State Machine (FSM) Coverage</title>
    
    <sect2 id="section.metrics.fsm.description">
      <title>Description</title>
      <para>
        Finite state machine (FSM) coverage answers the question, "Did I reach all of the states and traverse all possible paths through 
        a given state machine?"
      </para>
      <para>
        There are two types of coverage detail for FSMs that Covered can handle:
      </para>
      <para>
        <orderedlist>
          <listitem>
            <para>
              State coverage - answers the question "Were all states of an FSM hit during simulation?"
            </para>
          </listitem>
          <listitem>
            <para>
              State transition coverage - answers the question "Did the FSM transition between all states (that are achievable) in 
              simulation?"
            </para>
          </listitem>
        </orderedlist>
      </para>
    </sect2>
    
    <sect2 id="section.metrics.fsm.example">
      <title>Example</title>
      <para>
        Suppose the DUT comprised of the following finite state machine:
      </para>
      <para>
        <example xml:id="example.metrics.fsm">
          <title>FSM Coverage Code Sample</title>
          <programlisting lang="Verilog"><![CDATA[
  module test( clock );
  
    input clock;

    reg [1:0] state;
    reg req, gnt;

    parameter IDLE 2'b00;
    parameter REQ  2'b01;
    parameter WAIT 2'b10;
    parameter GNT  2'b11;

    initial begin
      req = 1'b0;
      gnt = 1'b0;
      repeat(4) @(posedge clock);
      req <= 1'b1;
      @(posedge clock);
      req <= 1'b0;
      gnt <= 1'b1;
      @(posedge clock);
      gnt <= 1'b0;
      repeat(2) @(posedge clock);
    end

    always @(posedge clock)
      case( state )
        IDLE : state <= req ? REQ : IDLE;
        REQ  : state <= gnt ? GNT : WAIT;
        WAIT : state <= gnt ? GNT : WAIT;
        GNT  : state <= IDLE;
        default : state <= 2'bx;
    endcase

  endmodule
          ]]></programlisting>
        </example>
      </para>
      <para>
        To help visualize this state machine, let's also display it as a digraph.
      </para>
      <para>
        <figure id="figure.metrics.fsm">
          <title>Digraph Version of Example FSM</title>
          <mediaobject><imageobject><imagedata fileref="img/fsm_example.png" format="PNG"/></imageobject></mediaobject>
        </figure>
      </para>
      <para>
        This state machine has four states: IDLE, REQ, WAIT, GNT. During simulation, 75% of the states were hit 
        (3 out of 4) including IDLE, REQ and GNT. You can see that the WAIT state was never hit during simulation 
        since the gnt signal was asserted the clock period after the req signal asserted.
      </para>
      <para>
        Additionally, this state machine contains 7 state "arcs" which are represented in the digraph as arrowed lines. 
        They are the following:
      </para>
      <para>
        <orderedlist>
          <listitem><para><programlisting>IDLE -> IDLE  (hit)</programlisting></para></listitem>
          <listitem><para><programlisting>IDLE -> REQ   (hit)</programlisting></para></listitem>
          <listitem><para><programlisting>REQ  -> WAIT  (missed)</programlisting></para></listitem>
          <listitem><para><programlisting>REQ  -> GNT   (hit)</programlisting></para></listitem>
          <listitem><para><programlisting>WAIT -> WAIT  (missed)</programlisting></para></listitem>
          <listitem><para><programlisting>WAIT -> GNT   (missed)</programlisting></para></listitem>
          <listitem><para><programlisting>GNT  -> IDLE  (hit)</programlisting></para></listitem>
        </orderedlist>
      </para>
      <para>
        Counting the number of traversed arcs during simulation, we can see that we have covered 4 out of 7 (or 57%) of 
        the possible arcs in this state machine.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.fsm.recommend">
      <title>Recommendations</title>
      <para>
        For a design to pass full coverage, it is recommended that the FSM coverage for all finite state machines in the 
        design to receive 100% coverage for the state coverage and 100% for all achievable state transitions. Since Covered 
        will not determine which state transitions are achievable, it is up to the verification engineer to examine the 
        executed state transitions to determine if 100% of possible transitions occurred.
      </para>
    </sect2>
  </sect1>
  
  <sect1 id="section.metrics.assert">
    <title>Assertion Coverage</title>
    
    <sect2 id="section.metrics.assert.description">
      <title>Description</title>
      <para>
        Assertion coverage answers the question, "Did I hit all of the possible coverage points of the assertions that I 
        included in the design?"
      </para>
      <para>
        Currently, Covered is capable of finding all OVL (Open Verification Language) assertion modules instantiated within 
        the design (when the <emphasis role="bold">-A ovl</emphasis> option has been specified to the score command). Within 
        most OVL assertion modules are one or more built-in coverage points. When the user has specified a particular assertion 
        instance to check for a coverage point, Covered simulates this assertion module, keeping track of which coverage 
        points within the assertion have been hit during simulation (and how many times each coverage point has been hit) and 
        which have not. This allows the user to check for more complex coverage scenarios within the design.
      </para>
      <para>
        Currently, only <ulink url="http://www.accellera.org/activities/ovl">OVL version 1.6</ulink> (and newer) are supported. 
        Older versions of the OVL may not work and are not planned to be supported.
      </para>
    </sect2>
    
    <sect2 id="section.metrics.assert.recommend">
      <title>Recommendations</title>
      <para>
        For a design to pass full coverage, it is recommended that the assertion coverage for all modules in a design receive 
        100% coverage. If an assertion coverage point is not hit during simulation, the design has not been fully exercised. 
        Assertion coverage is useful for determining more specific corner case holes in the test suite.
      </para>
    </sect2>
  </sect1>
		
</chapter>
