exp_eps: Second Order Reverse Sweep
Keywords: exp_eps: second order reverse sweep purpose mathematical form epsilon f_7 index 7: f_6 6: f_5 5: f_4 4: f_3 3: f_2 2: f_1 verification exercises

| Prev | Next |  |  |  |  |  |  |

exp_eps: Second Order Reverse Sweep
Purpose
In general, a second order reverse sweep is given the first order expansion for all of the variables in an operation sequence. Given a choice of a particular variable, it computes the derivative, of that variables first order expansion coefficient, with respect to all of the independent variables.
Mathematical Form
Suppose that we use the algorithm exp_eps.hpp to compute `exp_eps(x, epsilon)` with `x` is equal to .5 and `epsilon` is equal to .2. For this case, the mathematical function for the operation sequence corresponding to `exp_eps` is f ( x , ε ) = 1 + x + x 2 / 2 The corresponding derivative of the partial derivative with respect to x is ∂ 2 ∂ x 2 f ( x , ε ) = 1 ∂ ε ∂ x f ( x , ε ) = 0
epsilon
Since ε is an independent variable, it could included as an argument to all of the f j functions below. The result would be that all the partials with respect to ε would be zero and hence we drop it to simplify the presentation.
f_7
In reverse mode we choose one dependent variable and compute its derivative with respect to all the independent variables. For our example, we chose the value returned by exp_eps.hpp which is v 7 . We begin with the function f 7 where v 7 is both an argument and the value of the function; i.e., f 7 (v 1 ( 0 ) , v 1 ( 1 ) , … , v 7 ( 0 ) , v 7 ( 1 ) ) = v 7 ( 1 ) ∂ f 7 ∂ v 7 ( 1 ) = 1 All the other partial derivatives of f 7 are zero.
Index 7: f_6
The last operation has index 7, v 7 ( 0 ) = v 4 ( 0 ) + v 6 ( 0 ) v 7 ( 1 ) = v 4 ( 1 ) + v 6 ( 1 ) We define the function f 6 (v 1 ( 0 ) , … , v 6 ( 1 ) ) as equal to f 7 except that v 7 ( 0 ) and v 7 ( 1 ) are eliminated using this operation; i.e. f 6 = f 7 [v 1 ( 0 ) , … , v 6 ( 1 ) , v 7 ( 0 ) (v 4 ( 0 ) , v 6 ( 0 ) ) , v 7 ( 1 ) (v 4 ( 1 ) , v 6 ( 1 ) ) ] It follows that ∂ f 6 ∂ v 4 ( 1 ) = ∂ f 7 ∂ v 4 ( 1 ) + ∂ f 7 ∂ v 7 ( 1 ) * ∂ v 7 ( 1 ) ∂ v 4 ( 1 ) = 1 ∂ f 6 ∂ v 6 ( 1 ) = ∂ f 7 ∂ v 6 ( 1 ) + ∂ f 7 ∂ v 7 ( 1 ) * ∂ v 7 ( 1 ) ∂ v 6 ( 1 ) = 1 All the other partial derivatives of f 6 are zero.
Index 6: f_5
The previous operation has index 6, v 6 ( 0 ) = v 5 ( 0 ) / 2 v 6 ( 1 ) = v 5 ( 1 ) / 2 We define the function f 5 (v 1 ( 0 ) , … , v 5 ( 1 ) ) as equal to f 6 except that v 6 ( 0 ) and v 6 ( 1 ) are eliminated using this operation; i.e. f 5 = f 6 [v 1 ( 0 ) , … , v 5 ( 1 ) , v 6 ( 0 ) (v 5 ( 0 ) ) , v 6 ( 1 ) (v 5 ( 1 ) ) ] It follows that ∂ f 5 ∂ v 4 ( 1 ) = ∂ f 6 ∂ v 4 ( 1 ) = 1 ∂ f 5 ∂ v 5 ( 1 ) = ∂ f 6 ∂ v 5 + ∂ f 6 ∂ v 6 ( 1 ) * ∂ v 6 ( 1 ) ∂ v 5 ( 1 ) = 0.5 All the other partial derivatives of f 5 are zero.
Index 5: f_4
The previous operation has index 5, v 5 ( 0 ) = v 3 ( 0 ) * v 1 ( 0 ) v 5 ( 1 ) = v 3 ( 1 ) * v 1 ( 0 ) + v 3 ( 0 ) * v 1 ( 1 ) We define the function f 4 (v 1 ( 0 ) , … , v 4 ( 1 ) ) as equal to f 5 except that v 5 ( 0 ) and v 5 ( 1 ) are eliminated using this operation; i.e. f 4 = f 5 [v 1 ( 0 ) , … , v 4 ( 1 ) , v 5 ( 0 ) (v 1 ( 0 ) , v 3 ( 0 ) ) , v 5 ( 1 ) (v 1 ( 0 ) , v 1 ( 1 ) , v 3 ( 0 ) , v 3 ( 1 ) ) , ] Given the information from the forward sweep, we have v 1 ( 0 ) = 0.5 , v 3 ( 0 ) = 0.5 , v 1 ( 1 ) = 1 , v 3 ( 1 ) = 1 , and the fact that the partial of f 5 with respect to v 5 ( 0 ) is zero, we have ∂ f 4 ∂ v 1 ( 0 ) = ∂ f 5 ∂ v 1 ( 0 ) + ∂ f 5 ∂ v 5 ( 1 ) * ∂ v 5 ( 1 ) ∂ v 1 ( 0 ) = 0.5 ∂ f 4 ∂ v 1 ( 1 ) = ∂ f 5 ∂ v 1 ( 1 ) + ∂ f 5 ∂ v 5 ( 1 ) * ∂ v 5 ( 1 ) ∂ v 1 ( 1 ) = 0.25 ∂ f 4 ∂ v 3 ( 0 ) = ∂ f 5 ∂ v 3 ( 0 ) + ∂ f 5 ∂ v 5 ( 1 ) * ∂ v 5 ( 1 ) ∂ v 3 ( 0 ) = 0.5 ∂ f 4 ∂ v 3 ( 1 ) = ∂ f 3 ∂ v 1 ( 1 ) + ∂ f 5 ∂ v 5 ( 1 ) * ∂ v 5 ( 1 ) ∂ v 3 ( 1 ) = 0.25 ∂ f 4 ∂ v 4 ( 1 ) = ∂ f 5 ∂ v 4 ( 1 ) = 1 All the other partial derivatives of f 5 are zero.
Index 4: f_3
The previous operation has index 4, v 4 ( 0 ) = 1 + v 3 ( 0 ) v 4 ( 1 ) = v 3 ( 1 ) We define the function f 3 (v 1 ( 0 ) , … , v 3 ( 1 ) ) as equal to f 4 except that v 4 ( 0 ) and v 4 ( 1 ) are eliminated using this operation; i.e. f 3 = f 4 [v 1 ( 0 ) , … , v 3 ( 1 ) , v 4 ( 0 ) (v 3 ( 0 ) ) , v 4 ( 1 ) (v 3 ( 1 ) ) ] It follows that ∂ f 3 ∂ v 1 ( 0 ) = ∂ f 4 ∂ v 1 ( 0 ) = 0.5 ∂ f 3 ∂ v 1 ( 1 ) = ∂ f 4 ∂ v 1 ( 1 ) = 0.25 ∂ f 3 ∂ v 2 ( 0 ) = ∂ f 4 ∂ v 2 ( 0 ) = 0 ∂ f 3 ∂ v 2 ( 1 ) = ∂ f 4 ∂ v 2 ( 1 ) = 0 ∂ f 3 ∂ v 3 ( 0 ) = ∂ f 4 ∂ v 3 ( 0 ) + ∂ f 4 ∂ v 4 ( 0 ) * ∂ v 4 ( 0 ) ∂ v 3 ( 0 ) = 0.5 ∂ f 3 ∂ v 3 ( 1 ) = ∂ f 4 ∂ v 3 ( 1 ) + ∂ f 4 ∂ v 4 ( 1 ) * ∂ v 4 ( 1 ) ∂ v 3 ( 1 ) = 1.25
Index 3: f_2
The previous operation has index 3, v 3 ( 0 ) = v 2 ( 0 ) / 1 v 3 ( 1 ) = v 2 ( 1 ) / 1 We define the function f 2 (v 1 ( 0 ) , … , v 2 ( 1 ) ) as equal to f 3 except that v 3 ( 0 ) and v 3 ( 1 ) are eliminated using this operation; i.e. f 2 = f 3 [v 1 ( 0 ) , … , v 2 ( 1 ) , v 3 ( 0 ) (v 2 ( 0 ) ) , v 3 ( 1 ) (v 2 ( 1 ) ) ] It follows that ∂ f 2 ∂ v 1 ( 0 ) = ∂ f 3 ∂ v 1 ( 0 ) = 0.5 ∂ f 2 ∂ v 1 ( 1 ) = ∂ f 3 ∂ v 1 ( 1 ) = 0.25 ∂ f 2 ∂ v 2 ( 0 ) = ∂ f 3 ∂ v 2 ( 0 ) + ∂ f 3 ∂ v 3 ( 0 ) * ∂ v 3 ( 0 ) ∂ v 2 ( 0 ) = 0.5 ∂ f 2 ∂ v 2 ( 1 ) = ∂ f 3 ∂ v 2 ( 1 ) + ∂ f 3 ∂ v 3 ( 1 ) * ∂ v 3 ( 1 ) ∂ v 2 ( 0 ) = 1.25
Index 2: f_1
The previous operation has index 1, v 2 ( 0 ) = 1 * v 1 ( 0 ) v 2 ( 1 ) = 1 * v 1 ( 1 ) We define the function f 1 (v 1 ( 0 ) , v 1 ( 1 ) ) as equal to f 2 except that v 2 ( 0 ) and v 2 ( 1 ) are eliminated using this operation; i.e. f 1 = f 2 [v 1 ( 0 ) , v 1 ( 1 ) , v 2 ( 0 ) (v 1 ( 0 ) ) , v 2 ( 1 ) (v 1 ( 1 ) ) ] It follows that ∂ f 1 ∂ v 1 ( 0 ) = ∂ f 2 ∂ v 1 ( 0 ) + ∂ f 2 ∂ v 2 ( 0 ) * ∂ v 2 ( 0 ) ∂ v 1 ( 0 ) = 1 ∂ f 1 ∂ v 1 ( 1 ) = ∂ f 2 ∂ v 1 ( 1 ) + ∂ f 2 ∂ v 2 ( 1 ) * ∂ v 2 ( 1 ) ∂ v 1 ( 1 ) = 1.5 Note that v 1 is equal to x , so the second partial derivative of `exp_eps(x, epsilon)` at `x` equal to .5 and `epsilon` equal .2 is ∂ 2 ∂ x 2 v 7 ( 0 ) = ∂ v 7 ( 1 ) ∂ x = ∂ f 1 ∂ v 1 ( 0 ) = 1 There is a theorem about algorithmic differentiation that explains why the other partial of f 1 is equal to the first partial of `exp_eps(x, epsilon)` with respect to x .
Verification
The file exp_eps_rev2.cpp contains a routine that verifies the values computed above. It returns true for success and false for failure. It only tests the partial derivatives of f j that might not be equal to the corresponding partials of f j + 1 ; i.e., the other partials of f j must be equal to the corresponding partials of f j + 1 .
Exercises

1. Consider the case where x = .1 and we first preform a zero order forward mode sweep for the operation sequence used above (in reverse order). What are the results of a first order reverse mode sweep; i.e., what are the corresponding values for ∂ f j ∂ v k for all j , k such that ∂ f j ∂ v k ≠ 0 .
2. Create a modified version of exp_eps_rev2.cpp that verifies the values you obtained for the previous exercise. Also create and run a main program that reports the result of calling the modified version of exp_eps_rev2.cpp .

Input File: introduction/exp_apx/exp_eps.omh