An Important Reverse Mode Identity
Keywords: important reverse mode identity notation sweep theorem proof

| Prev | Next |  |  |  |  |  | reverse_identity |

An Important Reverse Mode Identity
The theorem and the proof below is a restatement of the results on page 236 of Evaluating Derivatives .
Notation
Given a function f ( u , v ) where u ∈ B n we use the notation ∂ f ∂ u ( u , v ) = [∂ f ∂ u 1 ( u , v ) , ⋯ , ∂ f ∂ u n ( u , v ) ]
Reverse Sweep
When using reverse mode we are given a function F : B n → B m , a matrix of Taylor coefficients x ∈ B n × p , and a weight vector w ∈ B m . We define the functions X : B × B n × p → B n , W : B × B n × p → B , and W j : B n × p → B by X ( t , x ) = x ( 0 ) + x ( 1 ) t + ⋯ + x ( p -1 ) t p -1 W ( t , x ) = w 0 F 0 [ X ( t , x ) ] + ⋯ + w m -1 F m -1 [ X ( t , x ) ] W j ( x ) = 1 j ! ∂ j ∂ t j W ( 0 , x ) where x ( j ) is the `j`-th column of x ∈ B n × p . The theorem below implies that ∂ W j ∂ x ( i ) ( x ) = ∂ W j - i ∂ x ( 0 ) ( x ) A general reverse sweep calculates the values ∂ W p -1 ∂ x ( i ) ( x ) ( i = 0 , … , p -1 ) But the return values for a reverse sweep are specified in terms of the more useful values ∂ W j ∂ x ( 0 ) ( x ) ( j = 0 , … , p -1 )
Theorem
Suppose that F : B n → B m is a p times continuously differentiable function. Define the functions Z : B × B n × p → B n , Y : B × B n × p → B m , and y ( j ) : B n × p → B m by Z ( t , x ) = x ( 0 ) + x ( 1 ) t + ⋯ + x ( p -1 ) t p -1 Y ( t , x ) = F [ Z ( t , x ) ] y ( j ) ( x ) = 1 j ! ∂ j ∂ t j Y ( 0 , x ) where x ( j ) denotes the `j`-th column of x ∈ B n × p . It follows that for all i , j such that i ≤ j < p , ∂ y ( j ) ∂ x ( i ) ( x ) = ∂ y ( j - i ) ∂ x ( 0 ) ( x )
Proof
If follows from the definitions that ∂ y ( j ) ∂ x ( i ) ( x ) = 1 j ! ∂ ∂ x ( i ) [∂ j ∂ t j ( F ∘ Z ) ( t , x ) ] t = 0 = 1 j ! [∂ j ∂ t j ∂ ∂ x ( i ) ( F ∘ Z ) ( t , x ) ] t = 0 = 1 j ! {∂ j ∂ t j [t i ( F ( 1 ) ∘ Z ) ( t , x ) ] } t = 0 For k > i , the `k`-th partial of t i with respect to t is zero. Thus, the partial with respect to t is given by ∂ j ∂ t j [t i ( F ( 1 ) ∘ Z ) ( t , x ) ] = ∑ k = 0 i ( j k ) i ! ( i - k ) ! t i - k ∂ j - k ∂ t j - k ( F ( 1 ) ∘ Z ) ( t , x ) {∂ j ∂ t j [t i ( F ( 1 ) ∘ Z ) ( t , x ) ] } t = 0 = ( j i ) i ! ∂ j - i ∂ t j - i ( F ( 1 ) ∘ Z ) ( t , x ) = j ! ( j - i ) ! ∂ j - i ∂ t j - i ( F ( 1 ) ∘ Z ) ( t , x ) ∂ y ( j ) ∂ x ( i ) ( x ) = 1 ( j - i ) ! ∂ j - i ∂ t j - i ( F ( 1 ) ∘ Z ) ( t , x ) Applying this formula to the case where j is replaced by j - i and i is replaced by zero, we obtain ∂ y ( j - i ) ∂ x ( 0 ) ( x ) = 1 ( j - i ) ! ∂ j - i ∂ t j - i ( F ( 1 ) ∘ Z ) ( t , x ) = ∂ y ( j ) ∂ x ( i ) ( x ) which completes the proof

Input File: omh/theory/reverse_identity.omh