=head1 NAME

heri-eval - evaluate classification algorithm

=head1 SYNOPSIS

B<heri-eval> [OPTIONS] I<dataset> [-- SVM_TRAIN_OPTIONS]

=head1 DESCRIPTION

B<heri-eval> runs training algorithm on I<dataset> and then
evaluate it using testing set, specified by option I<-e>.
If option I<-n> was applied,
cross-validation is used for evaluation, training and testing on different folds
are run in parallel, thus utilizing available CPUs. If I<-r> is used, the
dataset is splitted into training and testing datasets randomly with
the specified ratio, and then holdout is run.

=head1 OPTIONS

=over 6

=item B<-h, --help>

Display help information.

=item B<-f>

Enable output of per-fold statistics. See B<-M>I<f>.

=item B<-n> I<N>

Enable T*I<N>-fold cross-validation mode and set the number of folds to I<N>.

=item B<-r> I<ratio>

Split the dataset into training and testing parts with the specified ratio
of their sizes (in percents).

=item B<-t> I<T>

Enable I<T>*N-fold cross-validation mode and set the number of runs to I<T>
which 1 by default.

=item B<-e> I<testing_dataset>

Enable hold-out mode and set the testing dataset.

=item B<-T> I<threshold>

Set the minimum threshold for making a classification decision. If this flag is applied,
micro-average precision, recall, and F1 are calculated instead of accuracy.

=item B<-o> I<filename>

Save predictions from testing sets
to the specified file.

Format: outcome_class prediction_class [score]

=item B<-O> I<filename>

Save incorrectly classified objects
to the specified file.

Format: #object_number: outcome_class prediction_class [score])

=item B<-m> I<filename>

Save confusion matrix to the specified file.

Format: frequency : outcome_class prediction_class

=item B<-p> I<opts>

Pass the specified I<opts> to B<heri-stat(1)>.

=item B<-s> I<opts>

Pass the specified I<opts> to B<heri-split(1)>.

=item B<-M> I<chars>

Sets the output mode where chars are:
t -- output total statistics,
f -- output per-fold statistics,
c -- output cross-fold statistics.
The default is "-M tc".

=item B<-S> I<seed>

Pass the specified I<seed> to B<heri-split(1)>.

=item B<-K>

Keep temporary directory after exiting.

=item B<-D>

Turn on the debugging mode, implies -K.

=back

=head1 EXAMPLES

=over 1

 heri-eval -e testing_set.libsvm training_set.libsvm -- -s 0 -t 0

 export SVM_TRAIN_CMD='liblinear-train'
 export SVM_PREDICT_CMD='liblinear-predict'
 heri-eval -p '-mr' -n 5 training_set.libsvm -- -s 4 -q
 heri-eval -p '-mr' -n 5 training_set.libsvm -- -s 4 -q

 export SVM_TRAIN_CMD='scikit_rf-train --estimators=400'
 export SVM_PREDICT_CMD='scikit_rf-predict'
 heri-eval -p '-c' -Mt -t 50 -r 70 dataset.libsvm

=back

=head1 ENVIRONMENT

=over 6

=item I<SVM_TRAIN_CMD>

Training utility, e.g., liblinear-train
(the default is svm-train).

=item I<SVM_PREDICT_CMD>

Predicting utility, e.g., liblinear-predict
(the default is svm-predict).

=item I<SVM_HERI_STAT_CMD>

Utility for calculating statistics (the default is B<heri-stat(1)>).

=item I<SVM_HERI_STAT_ADDONS_CMD>

Utility for calculating additional statistics (the default is B<heri-stat-addons(1)>).

=item I<SVM_HERI_SPLIT_CMD>

Utility for splitting the dataset (the default is B<heri-split(1)>).

=item I<TMPDIR>

Temporary directory (the default is /tmp).

=back

=head1 HOME

L<http://github.com/cheusov/herisvm>

=head1 SEE ALSO

L<heri-split(1)>
L<heri-stat(1)>
