<chapter id="chapter.rank">
  <title>The rank Command</title>
  <para>
    The rank command allows the user to specify a number of CDD files that have been created from the same DUT (presumably in
    a regression run -- but not necessarily) and rank them in order of coverage importance.  This information can be useful
    in accomplishing the following tasks:
  </para>
  <para>
    <itemizedlist>
      <listitem>
        <para>
          Understand which CDD files (simulations) can be eliminated from regression due to their inability to add
          useful coverage information that selected CDD files already hit.
        </para>
      </listitem>
      <listitem>
        <para>
          Understand the best order to run simulations in full regression.
        </para>
      </listitem>
      <listitem>
        <para>
          It can help in creating a quick (confidence) regression which is meant to hit a good deal of the overall coverage
          points in a short period of time to verify that recent changes don't massively break a previously working DUT.
          This can be accomplished by selecting the first N simulation CDD files in the ranked list.  The selection can
          easily be done by either using the simulation order, accumulated hit, hit percent or timestep column values as 
          a means of selection.
        </para>
      </listitem>
    </itemizedlist>
  </para>
  
  <sect1 id="section.rank.options">
    <title>Options</title>
    <para>
      <table id="table.rank.options">
        <title>Options to rank Command</title>
        <tgroup cols='2'>
          <thead>
            <row>
              <entry>
                Option
              </entry>
              <entry>
                Description
              </entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>
                -depth <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies the minimum number of CDD files to hit each coverage point. The value of 
                <emphasis>number</emphasis> should be a value of 1 or more.  Default is 1.
              </entry>
            </row>
            <row>
              <entry>
                -d <emphasis>directory</emphasis>
              </entry>
              <entry>
                Directory to search for CDD files to include.  This option is used in conjunction with the -ext option 
                which specifies the file extension to use for determining which files in the directory are CDD files.
              </entry>
            </row>
            <row>
              <entry>
                -ext <emphasis>extension</emphasis>
              </entry>
              <entry>
                Used in conjunction with the -d option.  If no -ext options are specified on the command-line, the 
                default value of '.cdd' is used.  Note that a period (.) should be specified.
              </entry>
            </row>
            <row>
              <entry>
                -f <emphasis>filename</emphasis>
              </entry>
              <entry>
                Name of file containing additional arguments to parse.
              </entry>
            </row>
            <row>
              <entry>
                -h
              </entry>
              <entry>
                Displays help information for the rank command.
              </entry>
            </row>
            <row>
              <entry>
                -names-only
              </entry>
              <entry>
                If specified, outputs only the needed CDD filenames that need to be run in the order they need to be run. 
                If this option is not set, a report-style output is provided with additional information.
              </entry>
            </row>
            <row>
              <entry>
                -o <emphasis>filename</emphasis>
              </entry>
              <entry>
                Name of file to output ranking information to.  Default is stdout.
              </entry>
            </row>
            <row>
              <entry>
                -require-cdd <emphasis>filename</emphasis>
              </entry>
              <entry>
                Name of CDD file to use as a required CDD file for ranking, regardless of whether it is necessary to achieve
                full coverage or not.
              </entry>
            </row>
            <row>
              <entry>
                -require-list <emphasis>filename</emphasis>
              </entry>
              <entry>
                Name of file containing list of CDD files which are required to be in the list of ranked CDDs to be run,
                regardless of whether they are necessary to achieve full coverage or not.
              </entry>
            </row>
            <row>
              <entry>
                -v
              </entry>
              <entry>
                Outputs verbose information during the run of the rank command.  Useful for selection process understanding
                and command performance.
              </entry>
            </row>
            <row>
              <entry>
                -weight-assert <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies a relative weighting for assertion coverage used to rank non-unique coverage points.
              </entry>
            </row>
            <row>
              <entry>
                -weight-comb <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies a relative weighting for combinational logic coverage used to rank non-unique coverage points.
              </entry>
            </row>
            <row>
              <entry>
                -weight-fsm <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies a relative weighting for FSM state/state transition coverage used to rank non-unique coverage 
                points.
              </entry>
            </row>
            <row>
              <entry>
                -weight-line <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies a relative weighting for line coverage used to rank non-unique coverage points.
              </entry>
            </row>
            <row>
              <entry>
                -weight-memory <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies a relative weighting for memory coverage used to rank non-unique coverage points.
              </entry>
            </row>
            <row>
              <entry>
                -weight-toggle <emphasis>number</emphasis>
              </entry>
              <entry>
                Specifies a relative weighting for toggle coverage used to rank non-unique coverage points.
              </entry>
            </row>
          </tbody>
        </tgroup>
      </table>
    </para>
  </sect1>
  
  <sect1 id="section.rank.inputting">
    <title>Methods for specifying CDDs to rank</title>
    <para>
      The rank command provides several mechanisms for specifying CDD files to rank.  Feel free to use any combination
      of methods for providing this information to Covered as it makes sense to do in your environment.  The following is 
      the list of input methods:
    </para>
    <para>
      <itemizedlist>
        <listitem>
          <para>
            Specify each CDD file after the rank command options have been specified.  This method is best when there are
            a relatively small number of CDD files to rank as the length of the command-line in a shell is typically
            limited to a certain number of characters.
          </para>
        </listitem>
        <listitem>
          <para>
            Create a file containing the list of CDD files to rank and pass this file to the rank command's 
            <emphasis role="bold">-f</emphasis> option.  Using this method, one can selectively specify a large number 
            of CDD files.  You may specify as many "-f <emphasis>filename</emphasis>" options as is necessary.
          </para>
        </listitem>
        <listitem>
          <para>
            Use shell commands to create a list of CDD files and pass them to Covered's standard input via command-line 
            piping and the <emphasis role="bold">-f -</emphasis> command-line option (the - means use standard input).
            The following line is an example of this usage:
          </para>
          <para>
            <programlisting><![CDATA[
  ls foo*.cdd | covered rank -f -          
            ]]></programlisting>
          </para>
          <para>
            You may specify only one "-f -" option to the rank command.
          </para>
        </listitem>
        <listitem>
          <para>
            Use the <emphasis role="bold">-d <emphasis>directory</emphasis></emphasis> and 
            <emphasis role="bold">-ext <emphasis>extension</emphasis></emphasis> options to tell
            Covered to retrieve all files found in the directory specified by <emphasis>directory</emphasis> that contain
            the file extension of <emphasis>extension</emphasis> (if the <emphasis role="bold">-ext</emphasis> option
            is not specified, the default extension of ".cdd" is used).  The following is an example of this usage: 
          </para>
          <para>
            <programlisting><![CDATA[
  covered rank -d /home/bubba/cdd_dir -ext .mycdd          
            ]]></programlisting>
          </para>
          <para>
            You may specify as many -d options on the command-line as is necessary; however, only one -ext option is allowed.
          </para>
        </listitem>
        <listitem>
          <para>
            Specify a single CDD file that MUST be included in the final ranking list via the
            <emphasis role="bold">-require-cdd</emphasis> option.  You can specify as many -require-cdd options to the rank
            command as necessary.
          </para>
        </listitem>
        <listitem>
          <para>
            Create a file containing a list of CDDs that MUST be included in the final ranking list where each CDD filename
            is separated by a newline or whitespace.  Pass this file to the <emphasis role="bold">-require-list</emphasis> 
            option.  You can specify as many -require-list options to the rank command as necessary.
          </para>
        </listitem>
      </itemizedlist>
    </para>
  </sect1>
  
  <sect1 id="section.rank.weighting">
    <title>Specifying metric weights</title>
    <para>
      The rank command has several options that allow the user to weight any and all of the coverage metrics that Covered 
      supplies.  Weights are used to ascribe the importance of its metric in ranking the value of a CDD file.
    </para>
    <para>
      Weight values are non-negative numbers whose values are relative to each other.  A value of 0 effectively disables 
      the associated weight from consideration in ranking.  All other values are considered relative to each other.  In
      other words, if the line weight is set to a value of 5 while the toggle weight is set to 1, line coverage is 
      considered 5 times more important than toggle coverage in ranking.
    </para>
    <para>
      By default, each metric is given an equal weight of 1.  Adjust each of the metrics as necessary to potentially
      modify which CDD files are chosen and what order those CDD files are given in the ranked list of diagnostics to run.
    </para>
  </sect1>
  
  <sect1 id="section.rank.output" xreflable="Understanding the rank command output information">
    <title>Understanding the rank command output information</title>
    <para>
      After the rank command has been executed, the output will either be displayed to standard output (if no 
      <emphasis role="bold">-o</emphasis> option was specified) or it will be output to a file.  If the 
      <emphasis role="bold">-names_only</emphasis> option was specified, the output will be a list of CDD filenames (one
      per line) which are output in the order in which they should be run to achieve full coverage in the shortest amount
      of time.  If the -names_only option was not specified, the output will be in the form of a table (similar to reports
      generated with the report command) which describes not only the order that CDD files should be run but also
      various statistical data and the names of CDD files which are considered redundant in terms of coverage information.
      The following example shows an example of this report output.
    </para>
    <para>
      <example id="example.rank.noreq">
        <title>Rank Output With No Required Files</title>
        <programlisting><![CDATA[
                                           ::::::::::::::::::::::::::::::::::::::::::::::::::::
                                           ::                                                ::
                                           ::     Covered -- Simulation Ranked Run Order     ::
                                           ::                                                ::
                                           ::::::::::::::::::::::::::::::::::::::::::::::::::::



* Reduced 3 CDD files down to 2 needed to maintain coverage ( 33% reduction,   1.5x improvement)
* Reduced 9 timesteps down to 6 needed to maintain coverage ( 33% reduction,   1.5x improvement)

-----------+-------------------------------------------+----------------------------------------------------------
           |                ACCUMULATIVE               |                             CDD
Simulation |-------------------------------------------+----------------------------------------------------------
Order      |        Hit /      Total     %   Timesteps |  R  Name               Hit /      Total     %   Timesteps
-----------+-------------------------------------------+----------------------------------------------------------
         
         1            9           17   53%           3       rank1c.cdd           9           17   53%           3 
         2           13           17   76%           6       rank1b.cdd           8           17   47%           3 

---------------------------------------  The following CDD files add no additional coverage  ----------------------------------------------

         3           13           17   76%           9       rank1a.cdd           8           17   47%           3

        ]]></programlisting>
      </example>
    </para>
    <para>
      After the header information, there is a summary of the ranking reduction (if one occurred) specified in both number
      of files and number of simulation timesteps.  <xref linkend="example.rank.req"/> shows what this information looks
      like when no file reduction has occurred.
    </para>
    <para>
      After the summary information, there is the rank output table which displays both the individual CDD information and
      the accumulative coverage information.  Each row of the table contains one CDD file where the first row contains
      the first CDD that should be run in regression for maximum coverage impact.  Each successive row contains CDD files
      that add less and less coverage to the overall coverage gained.  If there were CDD files that were found to not
      add any coverage information to the above CDD files, a line is drawn and these CDD files displayed below that line.
      These CDD files can be excluded from regression because they don't add any useful coverage information over-and-above
      the CDD files that are selected to be run in regression.
    </para>
    <para>
      The first major column is the simulation order.  This value increments by one for each successive row.  If you would
      like to create a shorter running regression that is limited to a certain number of simulations, this row information
      can be useful for figuring out which simulations to include in these type of regressions.
    </para>
    <para>
      The second major column is the accumulative coverage statistics which include the following:
    </para>
    <para>
      <itemizedlist>
        <listitem>
          <para><emphasis role="bold">Hit</emphasis></para>
          <para>
            The first row contains the number of coverage points that the first CDD file hit.  All successive rows add
            the unique coverage points hit by the CDD file listed on that line to the above hit total.
          </para>
          <para>
            For example, in <xref linkend="example.rank.noreq"/> "rank1c.cdd" contains 9 hit coverage points (accumulative 
            hit count = 9).  The second highest ranked file, "rank1b.cdd", contains 8 hit coverage points but only 4 of 
            those coverage points are different than the coverage points hit by "rank1c.cdd"; therefore, the accumulative 
            coverage hit count is 13.  The third coverage file, "rank1c.cdd", contains 8 coverage points but none of them
            add to the coverage points hit by the first two coverage files; therefore, it is considered unnecessary for
            regression.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Total</emphasis></para>
          <para>
            The "accumulative" total represents the total number of coverage points in each CDD file.  Since this number
            must be the same for each CDD file, this value will be the same for each row.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">%</emphasis></para>
          <para>
            This column represents the hit percentage and is a simple calculation of <code>hit / total</code> for each
            row.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Timesteps</emphasis></para>
          <para>
            This column represents the number of accumulated simulation timesteps.  This value represents the number of
            timesteps that need to transpire to gain the amount of accumulated coverage.  One of the goals of the
            ranking algorithm is to gain the most amount of coverage in the least amount of simulation time.
          </para>
        </listitem>
      </itemizedlist>
    </para>
    <para>
      The third major column is the individual coverage information for the CDD file listed in the row which includes
      the following information:
    </para>
    <para>
      <itemizedlist>
        <listitem>
          <para><emphasis role="bold">R</emphasis></para>
          <para>
            This column represents whether the current CDD file was a required CDD file by the user (a file listed
            via the -require-list or -require-cdd options).  If the file is a required CDD file, an asterisk (*)
            is listed in this column.  If the file was not required, nothing will be listed in this column.
            <xref linkend="example.rank.req"/> provides an example of what this output looks like.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Name</emphasis></para>
          <para>
            Filename of the CDD file that was selected for this row.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Hit</emphasis></para>
          <para>
            This column represents the number of coverage points hit by the CDD file.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Total</emphasis></para>
          <para>
            This column represents the total number of coverage points within the CDD file that could be hit.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">%</emphasis></para>
          <para>
            This column represents the hit percentage (a calculation of <code>hit / total</code>.
          </para>
        </listitem>
        <listitem>
          <para><emphasis role="bold">Timesteps</emphasis></para>
          <para>
            This column represents the number of simulation timesteps that the CDD file took to run during simulation.
            This value does not necessarily represent the ending simulation time.
          </para>
        </listitem>
      </itemizedlist>
    </para>
    <para>
      <example id="example.rank.req">
        <title>Rank Output With Required Files</title>
        <programlisting><![CDATA[
                                           ::::::::::::::::::::::::::::::::::::::::::::::::::::
                                           ::                                                ::
                                           ::     Covered -- Simulation Ranked Run Order     ::
                                           ::                                                ::
                                           ::::::::::::::::::::::::::::::::::::::::::::::::::::



No reduction occurred 

-----------+-------------------------------------------+------------------------------------------------------------
           |                ACCUMULATIVE               |                              CDD
Simulation |-------------------------------------------+------------------------------------------------------------
Order      |        Hit /      Total     %   Timesteps |  R  Name                 Hit /      Total     %   Timesteps
-----------+-------------------------------------------+------------------------------------------------------------

         1            9           17   53%           3       rank1.1c.cdd           9           17   53%           3
         2           13           17   76%           6    *  rank1.1b.cdd           8           17   47%           3
         3           13           17   76%           9    *  rank1.1a.cdd           8           17   47%           3

        ]]></programlisting>
      </example>
    </para>
  </sect1>
  
  <sect1 id="section.rank.algorithm">
    <title>Description of ranking algorithm</title>
    <para>
      This section describes the overall algorithm that is used to rank the specified CDD files. The algorithm is being 
      presented in this document for the purposes of providing an explanation for how diagnostics are selected for
      ranking and how they are ranked. Knowing this information is not necessary for using the rank command or 
      understanding its results.
    </para>
    <para>
      The algorithm makes use of three different types of structures.  The first structure is a bit-compressed structure
      where each bit corresponds to one coverage point in the associated CDD file. The purpose of this structure is to 
      store a large amount of information as concisely and immediately available as possible. In addition to coverage
      point information, each structure also contains the following information:
    </para>
    <para>
      <itemizedlist>
        <listitem>
          <para>
            Name of CDD file that the structure is associated with (for reporting purposes only).
          </para>
        </listitem>
        <listitem>
          <para>
            The number of timesteps that occurred during the CDD simulation. This is used for calculating the average 
            number of coverage points hit during a single clock period -- used for ranking selection.
          </para>
        </listitem>
        <listitem>
          <para>
            The total number of coverage points hit by the CDD. This is used for calculating the average number of 
            coverage points hit during a single clock period -- used for ranking selection.
          </para>
        </listitem>
        <listitem>
          <para>
            The total number of coverage points that were hit by this CDD that were not hit by any other CDD listed in
            the rank command-line. This information is used to figure out which CDDs are absolutely necessary to be
            included in the ranked list.
          </para>
        </listitem>
        <listitem>
          <para>
            The current score associated with the CDD file. This value is updated for each selection iteration.
          </para>
        </listitem>
      </itemizedlist>
    </para>
    <para>
      Each of these structures are maintained in an array that is iterated on during the ranking process.
    </para>
    <para>
      The second structure is an array of 16-bit values where each value corresponds to a single coverage point. Its 
      value is a sum of the number of unranked CDD files that have hit that coverage point. Each value is decremented
      by one when a CDD file is moved from the unranked list to the ranked list and that CDD file hits the given
      coverage point. If a coverage point value is a value of 1 and a CDD file has its corresponding coverage bit set,
      we know that the CDD is the only one that hits the coverage point.
    </para>
    <para>
      The third structure is similar to the second and works similarly, except that it stores the number of CDD files in
      the ranked list that hit a particular coverage point.
    </para>
    <para>
      The specified CDD files are ranked in a series of steps, some of which are reiterated for each CDD that is
      selected to be ranked. Here is the basic algorithm (in pseudo-code):
    </para>
    <para>
      <programlisting><![CDATA[
      
  calculate the size of the compressed coverage point structure
  
  foreach CDD in CDD_list {
    convert CDD to compressed coverage point structure
  }
  
  foreach CDD in CDD_list {
    if this CDD contains uniquely hit coverage points {
      move the CDD file into the ranked list
    }
  }
  
  foreach CDD in unranked list {
    foreach CDD in unranked list {
      calculate score (((total_num_cov_points_hit / num_timesteps) * 100) * metric_weight_value)
      if any unranked elements are less than the -depth option value and are hit by this CDD {
        set unique_found to true
      }
      if our score > highest score and a "unique" coverage point was found {
        set highest_score to our score
      }
    }
    move the CDD file indexed by highest_score from the unranked list to the ranked list
  }
  
  resort ranked list from most unique to least unique (if there is a tie, select the CDD that hit the most coverage
    points in the least amount of time)
      ]]></programlisting>
    </para>
  </sect1>
  
</chapter>
