The Theory of Reverse Mode
Keywords: the theory reverse mode taylor notation binary operators addition subtraction multiplication division standard math functions

| Prev | Next |  |  |  |  |  |  |

The Theory of Reverse Mode
Taylor Notation
In Taylor notation, each variable corresponds to a function of a single argument which we denote by `t` (see Section 10.2 of Evaluating Derivatives ). Here and below X ( t ) , Y ( t ) , and `Z(t)` are scalar valued functions and the corresponding `p`-th order Taylor coefficients row vectors are x , y and z ; i.e., X ( t ) = x ( 0 ) + x ( 1 ) * t + ⋯ + x ( p ) * t p + O ( t p + 1 ) Y ( t ) = y ( 0 ) + y ( 1 ) * t + ⋯ + y ( p ) * t p + O ( t p + 1 ) Z ( t ) = z ( 0 ) + z ( 1 ) * t + ⋯ + z ( p ) * t p + O ( t p + 1 ) For the purposes of this discussion, we are given the `p`-th order Taylor coefficient row vectors x , y , and z . In addition, we are given the partial derivatives of a scalar valued function G ( z ( j ) , … , z ( 0 ) , x , y ) We need to compute the partial derivatives of the scalar valued function H ( z ( j -1 ) , … , z ( 0 ) , x , y ) = G ( z ( j ) , z ( j -1 ) , … , z ( 0 ) , x , y ) where z ( j ) is expressed as a function of the `j-1`-th order Taylor coefficient row vector for Z and the vectors x , y ; i.e., z ( j ) above is a shorthand for z ( j ) ( z ( j -1 ) , … , z ( 0 ) , x , y ) If we do not provide a formula for a partial derivative of H , then that partial derivative has the same value as for the function G .
Binary Operators
Addition
The forward mode formula for addition is z ( j ) = x ( j ) + y ( j ) If follows that for k = 0 , … , j and l = 0 , … , j -1 ∂ H ∂ x ( k ) = ∂ G ∂ x ( k ) + ∂ G ∂ z ( k ) ∂ H ∂ y ( k ) = ∂ G ∂ y ( k ) + ∂ G ∂ z ( k ) ∂ H ∂ z ( l ) = ∂ G ∂ z ( l )
Subtraction
The forward mode formula for subtraction is z ( j ) = x ( j ) - y ( j ) If follows that for k = 0 , … , j ∂ H ∂ x ( k ) = ∂ G ∂ x ( k ) - ∂ G ∂ z ( k ) ∂ H ∂ y ( k ) = ∂ G ∂ y ( k ) - ∂ G ∂ z ( k )
Multiplication
The forward mode formula for multiplication is z ( j ) = ∑ k = 0 j x ( j - k ) * y ( k ) If follows that for k = 0 , … , j and l = 0 , … , j -1 ∂ H ∂ x ( j - k ) = ∂ G ∂ x ( j - k ) + ∑ k = 0 j ∂ G ∂ z ( j ) y ( k ) ∂ H ∂ y ( k ) = ∂ G ∂ y ( k ) + ∑ k = 0 j ∂ G ∂ z ( j ) x ( j - k )
Division
The forward mode formula for division is z ( j ) = 1 y ( 0 ) (x ( j ) - ∑ k = 1 j z ( j - k ) y ( k ) ) If follows that for k = 1 , … , j ∂ H ∂ x ( j ) = ∂ G ∂ x ( j ) + ∂ G ∂ z ( j ) 1 y ( 0 ) ∂ H ∂ z ( j - k ) = ∂ G ∂ z ( j - k ) - ∂ G ∂ z ( j ) 1 y ( 0 ) y ( k ) ∂ H ∂ y ( k ) = ∂ G ∂ y ( k ) - ∂ G ∂ z ( j ) 1 y ( 0 ) z ( j - k ) ∂ H ∂ y ( 0 ) = ∂ G ∂ y ( 0 ) - ∂ G ∂ z ( j ) 1 y ( 0 ) 1 y ( 0 ) (x ( j ) - ∑ k = 1 j z ( j - k ) y ( k ) ) = ∂ G ∂ y ( 0 ) - ∂ G ∂ z ( j ) 1 y ( 0 ) z ( j )
Standard Math Functions
The standard math functions have only one argument. Hence we are given the partial derivatives of a scalar valued function G ( z ( j ) , … , z ( 0 ) , x ) We need to compute the partial derivatives of the scalar valued function H ( z ( j -1 ) , … , z ( 0 ) , x ) = G ( z ( j ) , z ( j -1 ) , … , z ( 0 ) , x ) where z ( j ) is expressed as a function of the `j-1`-th order Taylor coefficient row vector for Z and the vector x ; i.e., z ( j ) above is a shorthand for z ( j ) ( z ( j -1 ) , … , z ( 0 ) , x )
Contents

exp_reverse | Exponential Function Reverse Mode Theory
log_reverse | Logarithm Function Reverse Mode Theory
sqrt_reverse | Square Root Function Reverse Mode Theory
sin_cos_reverse | Trigonometric and Hyperbolic Sine and Cosine Reverse Theory
atan_reverse | Inverse Tangent and Hyperbolic Tangent Reverse Mode Theory
asin_reverse | Inverse Sine and Hyperbolic Sine Reverse Mode Theory
acos_reverse | Inverse Cosine and Hyperbolic Cosine Reverse Mode Theory
tan_reverse | Tangent and Hyperbolic Tangent Reverse Mode Theory
erf_reverse | Error Function Reverse Mode Theory

Input File: omh/theory/reverse_theory.omh