/*!

@file linear_regression.txt
@author James Cline
@brief Tutorial for how to use the LinearRegression class.

@page lrtutorial Linear/ridge regression tutorial (mlpack_linear_regression)

@section intro_lrtut Introduction

Linear regression and ridge regression are simple machine learning techniques
that aim to estimate the parameters of a linear model.  Assuming we have \f$n\f$
\b predictor points \f$\mathbf{x_i}, 0 \le i < n\f$ of dimensionality \f$d\f$
and \f$n\f$ responses \f$y_i, 0 \le i < n\f$, we are trying to estimate the best
fit for \f$\beta_i, 0 \le i \le d\f$ in the linear model

\f[
y_i = \beta_0 + \displaystyle\sum_{j = 1}^{d} \beta_j x_{ij}
\f]

for each predictor \f$\mathbf{x_i}\f$ and response \f$y_i\f$.  If we take each
predictor \f$\mathbf{x_i}\f$ as a row in the matrix \f$\mathbf{X}\f$ and each
response \f$y_i\f$ as an entry of the vector \f$\mathbf{y}\f$, we can represent
the model in vector form:

\f[
\mathbf{y} = \mathbf{X} \mathbf{\beta} + \beta_0
\f]

The result of this method is the vector \f$\mathbf{\beta}\f$, including the
offset term (or intercept term) \f$\beta_0\f$.

\b mlpack provides:

 - a \ref cli_lrtut "simple command-line executable" to perform linear regression or ridge regression
 - a \ref linreg_lrtut "simple C++ interface" to perform linear regression or    ridge regression

@section toc_lrtut Table of Contents

A list of all the sections this tutorial contains.

 - \ref intro_lrtut
 - \ref toc_lrtut
 - \ref cli_lrtut
   - \ref cli_ex1_lrtut
   - \ref cli_ex2_lrtut
   - \ref cli_ex3_lrtut
   - \ref cli_ex4_lrtut
 - \ref linreg_lrtut
   - \ref linreg_ex1_lrtut
   - \ref linreg_ex2_lrtut
   - \ref linreg_ex3_lrtut
   - \ref linreg_ex4_lrtut
   - \ref linreg_ex5_lrtut
 - \ref further_doc_lrtut

@section cli_lrtut Command-Line 'mlpack_linear_regression'

The simplest way to perform linear regression or ridge regression in \b mlpack
is to use the \c mlpack_linear_regression executable.  This program will perform
linear regression and place the resultant coefficients into one file.

The output file holds a vector of coefficients in increasing order of dimension;
that is, the offset term (\f$\beta_0\f$), the coefficient for dimension 1
(\f$\beta_1\f$, then dimension 2 (\f$\beta_2\f$) and so forth, as well as the
intercept.  This executable can also predict the \f$y\f$ values of a second
dataset based on the computed coefficients.

Below are several examples of simple usage (and the resultant output).  The
\c option is used so that verbose output is given.  Further documentation on
each individual option can be found by typing

@code
$ mlpack_linear_regression --help
@endcode

@subsection cli_ex1_lrtut One file, generating the function coefficients

@code
$ mlpack_linear_regression --training_file dataset.csv -v -M lr.xml
[INFO ] Loading 'dataset.csv' as CSV data.  Size is 2 x 5.
[INFO ]
[INFO ] Execution parameters:
[INFO ]   help: false
[INFO ]   info: ""
[INFO ]   input_model_file: ""
[INFO ]   lambda: 0
[INFO ]   output_model_file: lr.xml
[INFO ]   output_predictions: predictions.csv
[INFO ]   test_file: ""
[INFO ]   training_file: dataset.csv
[INFO ]   training_responses: ""
[INFO ]   verbose: true
[INFO ]   version: false
[INFO ]
[INFO ] Program timers:
[INFO ]   load_regressors: 0.000263s
[INFO ]   loading_data: 0.000220s
[INFO ]   regression: 0.000392s
[INFO ]   total_time: 0.001920s
@endcode

Convenient program timers are given for different parts of the calculation at
the bottom of the output, as well as the parameters the simulation was run with.
Now, if we look at the output model file, which is \c lr.xml,

@code
$ cat dataset.csv
0,0
1,1
2,2
3,3
4,4

$ cat lr.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="12">
<linearRegressionModel class_id="0" tracking_level="0" version="0">
  <parameters class_id="1" tracking_level="0" version="0">
    <n_rows>2</n_rows>
    <n_cols>1</n_cols>
    <n_elem>2</n_elem>
    <vec_state>1</vec_state>
    <item>-3.97205464519563669e-16</item>
    <item>1.00000000000000022e+00</item>
  </parameters>
  <lambda>0.00000000000000000e+00</lambda>
  <intercept>1</intercept>
</linearRegressionModel>
</boost_serialization>
@endcode

As you can see, the function for this input is \f$f(y)=0+1x_1\f$.  We can see
that the model we have trained catches this; in the \c <parameters> section of
\c lr.xml, we can see that there are two elements, which are (approximately) 0
and 1.  The first element corresponds to the intercept 0, and the second column
corresponds to the coefficient 1 for the variable \f$x_1\f$.  Note that in this
example, the regressors for the dataset are the second column.  That is, the
dataset is one dimensional, and the last column has the \f$y\f$ values, or
responses, for each row. You can specify these responses in a separate file if
you want, using the \c --input_responses, or \c -r, option.

@subsection cli_ex2_lrtut Compute model and predict at the same time

@code
$ mlpack_linear_regression --training_file dataset.csv --test_file predict.csv \
> -v
[INFO ] Loading 'dataset.csv' as CSV data.  Size is 2 x 5.
[INFO ] Loading 'predict.csv' as raw ASCII formatted data.  Size is 1 x 3.
[INFO ] Saving CSV data to 'predictions.csv'.
[INFO ]
[INFO ] Execution parameters:
[INFO ]   help: false
[INFO ]   info: ""
[INFO ]   input_model_file: ""
[INFO ]   lambda: 0
[INFO ]   output_model_file: ""
[INFO ]   output_predictions: predictions.csv
[INFO ]   test_file: predict.csv
[INFO ]   training_file: dataset.csv
[INFO ]   training_responses: ""
[INFO ]   verbose: true
[INFO ]   version: false
[INFO ]
[INFO ] Program timers:
[INFO ]   load_regressors: 0.000371s
[INFO ]   load_test_points: 0.000229s
[INFO ]   loading_data: 0.000491s
[INFO ]   prediction: 0.000075s
[INFO ]   regression: 0.000449s
[INFO ]   saving_data: 0.000186s
[INFO ]   total_time: 0.002731s

$ cat dataset.csv
0,0
1,1
2,2
3,3
4,4

$ cat predict.csv
2
3
4

$ cat predictions.csv
2.0000000000e+00
3.0000000000e+00
4.0000000000e+00
@endcode

We used the same dataset, so we got the same parameters. The key thing to note
about the \c predict.csv dataset is that it has the same dimensionality as the
dataset used to create the model, one.  If the model generating dataset has
\f$d\f$ dimensions, so must the dataset we want to predict for.

@subsection cli_ex3_lrtut Prediction using a precomputed model

@code
$ mlpack_linear_regression --input_model_file lr.xml --test_file predict.csv -v
[INFO ] Loading 'predict.csv' as raw ASCII formatted data.  Size is 1 x 3.
[INFO ] Saving CSV data to 'predictions.csv'.
[INFO ]
[INFO ] Execution parameters:
[INFO ]   help: false
[INFO ]   info: ""
[INFO ]   input_model_file: lr.xml
[INFO ]   lambda: 0
[INFO ]   output_model_file: ""
[INFO ]   output_predictions: predictions.csv
[INFO ]   test_file: predict.csv
[INFO ]   training_file: ""
[INFO ]   training_responses: ""
[INFO ]   verbose: true
[INFO ]   version: false
[INFO ]
[INFO ] Program timers:
[INFO ]   load_model: 0.000264s
[INFO ]   load_test_points: 0.000186s
[INFO ]   loading_data: 0.000157s
[INFO ]   prediction: 0.000098s
[INFO ]   saving_data: 0.000157s
[INFO ]   total_time: 0.001688s

$ cat lr.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE boost_serialization>
<boost_serialization signature="serialization::archive" version="12">
<linearRegressionModel class_id="0" tracking_level="0" version="0">
  <parameters class_id="1" tracking_level="0" version="0">
    <n_rows>2</n_rows>
    <n_cols>1</n_cols>
    <n_elem>2</n_elem>
    <vec_state>1</vec_state>
    <item>-3.97205464519563669e-16</item>
    <item>1.00000000000000022e+00</item>
  </parameters>
  <lambda>0.00000000000000000e+00</lambda>
  <intercept>1</intercept>
</linearRegressionModel>
</boost_serialization>

$ cat predict.csv
2
3
4

$ cat predictions.csv
2.0000000000e+00
3.0000000000e+00
4.0000000000e+00
@endcode

@subsection cli_ex4_lrtut Using ridge regression

Sometimes, the input matrix of predictors has a covariance matrix that is not
invertible, or the system is overdetermined.  In this case, ridge regression is
useful: it adds a normalization term to the covariance matrix to make it
invertible.  Ridge regression is a standard technique and documentation for the
mathematics behind it can be found anywhere on the Internet.  In short, the
covariance matrix

\f[
\mathbf{X}' \mathbf{X}
\f]

is replaced with

\f[
\mathbf{X}' \mathbf{X} + \lambda \mathbf{I}
\f]

where \f$\mathbf{I}\f$ is the identity matrix.  So, a \f$\lambda\f$ parameter
greater than zero should be specified to perform ridge regression, using the
\c --lambda (or \c -l) option.  An example is given below.

@code
$ mlpack_linear_regression --training_file dataset.csv -v --lambda 0.5 -M lr.xml
[INFO ] Loading 'dataset.csv' as CSV data.  Size is 2 x 5.
[INFO ]
[INFO ] Execution parameters:
[INFO ]   help: false
[INFO ]   info: ""
[INFO ]   input_model_file: ""
[INFO ]   lambda: 0.5
[INFO ]   output_model_file: lr.xml
[INFO ]   output_predictions: predictions.csv
[INFO ]   test_file: ""
[INFO ]   training_file: dataset.csv
[INFO ]   training_responses: ""
[INFO ]   verbose: true
[INFO ]   version: false
[INFO ]
[INFO ] Program timers:
[INFO ]   load_regressors: 0.000210s
[INFO ]   loading_data: 0.000170s
[INFO ]   regression: 0.000332s
[INFO ]   total_time: 0.001835s
@endcode

Further documentation on options should be found by using the \c --help option.

@section linreg_lrtut The 'LinearRegression' class

The 'LinearRegression' class is a simple implementation of linear regression.

Using the LinearRegression class is very simple. It has two available
constructors; one for generating a model from a matrix of predictors and a
vector of responses, and one for loading an already computed model from a given
file.

The class provides one method that performs computation:
@code
void Predict(const arma::mat& points, arma::vec& predictions);
@endcode

Once you have generated or loaded a model, you can call this method and pass it
a matrix of data points to predict values for using the model. The second
parameter, predictions, will be modified to contain the predicted values
corresponding to each row of the points matrix.

@subsection linreg_ex1_lrtut Generating a model

@code
#include <mlpack/methods/linear_regression/linear_regression.hpp>

using namespace mlpack::regression;

arma::mat data; // The dataset itself.
arma::vec responses; // The responses, one row for each row in data.

// Regress.
LinearRegression lr(data, responses);

// Get the parameters, or coefficients.
arma::vec parameters = lr.Parameters();
@endcode

@subsection linreg_ex2_lrtut Setting a model

Assuming you already have a model and do not need to create one, this is how
you would set the parameters for a LinearRegression instance.

@code
arma::vec parameters; // Your model.

LinearRegression lr(); // Create a new LinearRegression instance or reuse one.
lr.Parameters() = parameters; // Set the model.
@endcode

@subsection linreg_ex3_lrtut Load a model from a file

If you have a generated model in a file somewhere you would like to load and
use, you can use \c data::Load() to load it.

@code
std::string filename; // The path and name of your file.

LinearRegression lr;
data::Load(filename, "lr_model", lr);
@endcode

@subsection linreg_ex4_lrtut Prediction

Once you have generated or loaded a model using one of the aforementioned
methods, you can predict values for a dataset.

@code
LinearRegression lr();
// Load or generate your model.

// The dataset we want to predict on; each row is a data point.
arma::mat points;
// This will store the predictions; one row for each point.
arma::vec predictions;

lr.Predict(points, predictions); // Predict.

// Now, the vector 'predictions' will contain the predicted values.
@endcode

@subsection linreg_ex5_lrtut Setting lambda for ridge regression

As discussed in \ref cli_ex4_lrtut, ridge regression is useful when the
covariance of the predictors is not invertible.  The standard constructor can be
used to set a value of lambda:

@code
#include <mlpack/methods/linear_regression/linear_regression.hpp>

using namespace mlpack::regression;

arma::mat data; // The dataset itself.
arma::vec responses; // The responses, one row for each row in data.

// Regress, with a lambda of 0.5.
LinearRegression lr(data, responses, 0.5);

// Get the parameters, or coefficients.
arma::vec parameters = lr.Parameters();
@endcode

In addition, the \c Lambda() function can be used to get or modify the lambda
value:

@code
LinearRegression lr;
lr.Lambda() = 0.5;
Log::Info << "Lambda is " << lr.Lambda() << "." << std::endl;
@endcode

@section further_doc_lrtut Further documentation

For further documentation on the LinearRegression class, consult the
\ref mlpack::regression::LinearRegression "complete API documentation".

*/
