| Prev | Next |  |  |  |  |  | cholesky_theory |

AD Theory for Cholesky Factorization
Reference
See section 3.6 of Sebastian F. Walter's Ph.D. thesis, Structured Higher-Order Algorithmic Differentiation in the Forward and Reverse Mode with Application in Optimum Experimental Design , Humboldt-Universitat zu Berlin, 2011.
Notation
Cholesky Factor
We are given a positive definite symmetric matrix A ∈ R n × n and a Cholesky factorization A = L L T where L ∈ R n × n is lower triangular.
Taylor Coefficient
The matrix A is a function of a scalar argument t . For k = 0 , … , K , we use A k for the corresponding Taylor coefficients; i.e., A ( t ) = o ( t K ) + ∑ k = 0 K A k t k where o ( t K ) / t K → 0 as t → 0 . We use a similar notation for L ( t ) .
Lower Triangular Part
For a square matrix C , lower ( C ) is the lower triangular part of C , diag ( C ) is the diagonal matrix with the same diagonal as C and low ( C ) = lower ( C ) - 1 2 diag ( C )
Forward Mode
For Taylor coefficient order k = 0 , … , K the coefficients A k ∈ R n × n , and satisfy the equation A k = ∑ ℓ = 0 k L ℓ L k - ℓ T In the case where k = 0 , the A 0 = L 0 L 0 T The value of L 0 can be computed using the Cholesky factorization. In the case where k > 0 , A k = L k L 0 T + L 0 L k T + B k where B k = ∑ ℓ = 1 k -1 L ℓ L k - ℓ T Note that B k is defined in terms of Taylor coefficients of L ( t ) that have order less than k . We also note that L 0 -1 ( A k - B k ) L 0 - T = L 0 -1 L k + L k T L 0 - T The first matrix on the right hand side is lower triangular, the second is upper triangular, and the diagonals are equal. It follows that L 0 -1 L k = low [ L 0 -1 ( A k - B k ) L 0 - T ] L k = L 0 low [ L 0 -1 ( A k - B k ) L 0 - T ] This expresses L k in term of the Taylor coefficients of A ( t ) and the lower order coefficients of L ( t ) .
Lemma 1
We use the notation C ˙ for the derivative of a matrix valued function C ( s ) with respect to a scalar argument s . We use the notation S ¯ and L ¯ for the partial derivative of a scalar value function F ¯ ( S , L ) with respect to a symmetric matrix S and an lower triangular matrix L . Define the scalar valued function F ^ ( C ) = F ¯ [ S , L ^ ( S ) ] We use S ^ for the total derivative of F ^ with respect to S . Suppose that L ^ ( S ) is such that L ˙ = L 0 low ( L 0 -1 S ˙ L 0 - T ) for any S ( s ) . It follows that S ^ = S ¯ + 1 2 ( M + M T ) where M = L 0 - T low ( L 0 T L ¯ ) T L 0 -1
Proof ∂ s F ^ [ S ( s ) , L ( s ) ] = tr ( S ¯ T S ˙ ) + tr ( L ¯ T L ˙ ) tr ( L ¯ T L ˙ ) = tr [ L ¯ T L 0 low ( L 0 -1 S ˙ L 0 - T ) ] = tr [ low ( L 0 -1 S ˙ L 0 - T ) T L 0 T L ¯ ] = tr [ L 0 -1 S ˙ L 0 - T low ( L 0 T L ¯ ) ] = tr [ L 0 - T low ( L 0 T L ¯ ) L 0 -1 S ˙ ] ∂ s F ^ [ S ( s ) , L ( s ) ] = tr ( S ¯ T S ˙ ) + tr [ L 0 - T low ( L 0 T L ¯ ) L 0 -1 S ˙ ] We now consider the ( i , j ) component function, for a symmetric matrix S ( s ) , defined by S k , ℓ ( s ) = { 1 if k = i and ℓ = j 1 if k = j and ℓ = i 0 otherwise } This shows that the formula in the lemma is correct for S ^ i , j and S ^ j , i . This completes the proof because the component ( i , j ) was arbitrary.
Lemma 2
We use the same assumptions as in Lemma 1 except that the matrix S is lower triangular (instead of symmetric). It follows that S ^ = S ¯ + lower ( M ) where M = L 0 - T low ( L 0 T L ¯ ) T L 0 -1 The proof of this lemma is identical to Lemma 2 except that component function is defined by S k , ℓ ( s ) = { 1 if k = i and ℓ = j 0 otherwise }
Reverse Mode
Case k = 0
For the case k = 0 , A ˙ 0 = L ˙ 0 L 0 T + L 0 L ˙ 0 T L 0 -1 A ˙ 0 L 0 - T = L 0 -1 L ˙ 0 + L ˙ 0 T L 0 - T low ( L 0 -1 A ˙ 0 L 0 - T ) = L 0 -1 L ˙ 0 L ˙ 0 = L 0 low ( L 0 -1 A ˙ 0 L 0 - T ) It follows from Lemma 1 that A ¯ 0 = + 1 2 ( M + M T ) where M = L 0 - T low ( L 0 T L ¯ 0 ) T L 0 -1 and A ¯ 0 is the partial before and after is before and after L 0 is removed from the scalar function dependency.
Case k > 0
In the case where k > 0 , A k = L k L 0 T + L 0 L k T + B k where B k is defined in terms of Taylor coefficients of L ( t ) that have order less than k . It follows that L ˙ k L 0 T + L 0 L ˙ k T = A ˙ k - B ˙ k - L ˙ 0 L k T - L k L ˙ 0 T L 0 -1 L ˙ k + L ˙ k T L 0 - T = L 0 -1 ( A ˙ k - B ˙ k - L ˙ 0 L k T - L k L ˙ 0 T ) L 0 - T L 0 -1 L ˙ k = low [ L 0 -1 ( A ˙ k - B ˙ k - L ˙ 0 L k T - L k L ˙ 0 T ) L 0 - T ] L ˙ k = L 0 low [ L 0 -1 ( A ˙ k - B ˙ k - L ˙ 0 L k T - L k L ˙ 0 T ) L 0 - T ] The matrix A k is symmetric, it follows that A ¯ k = + 1 2 ( M k + M k T ) where M k = L 0 - T low ( L 0 T L ¯ k ) T L 0 -1 The matrix B k is also symmetric, hence B ¯ k = - 1 2 ( M k + M k T ) We define the symmetric matrix C k ( s ) by C ˙ k = L ˙ 0 L k T + L k L ˙ 0 T and remove the dependency on C k with tr ( C ¯ k T C ˙ k ) = tr ( B ¯ k T C ˙ k ) = tr ( B ¯ k T L ˙ 0 L k T ) + tr ( B ¯ k T L k L ˙ 0 T ) = tr ( L k T B ¯ k T L ˙ 0 ) + tr ( L k T B ¯ k L ˙ 0 ) = tr [ L k T ( B ¯ k + B ¯ k T ) L ˙ 0 ] Thus, removing C k from the dependency results in the following update to L ¯ 0 : L ¯ 0 = + lower [ ( B ¯ k + B ¯ k T ) L k ] which is the same as L ¯ 0 = + 2 lower [ B ¯ k L k ] We still need to remove B k from the dependency. It follows from its definition that B ˙ k = ∑ ℓ = 1 k -1 L ˙ ℓ L k - ℓ T + L ℓ L ˙ k - ℓ T tr ( B ¯ k T B ˙ k ) = ∑ ℓ = 1 k -1 tr ( B ¯ k T L ˙ ℓ L k - ℓ T ) + tr ( B ¯ k T L ℓ L ˙ k - ℓ T ) = ∑ ℓ = 1 k -1 tr ( L k - ℓ T B ¯ k T L ˙ ℓ ) + ∑ ℓ = 1 k -1 tr ( L ℓ T B ¯ k L ˙ k - ℓ ) We now use the fact that B ¯ k is symmetric to conclude tr ( B ¯ k T B ˙ k ) = 2 ∑ ℓ = 1 k -1 tr ( L k - ℓ T B ¯ k T L ˙ ℓ ) Each of the L ˙ ℓ matrices is lower triangular. Thus, removing B k from the dependency results in the following update for ℓ = 1 , … , k -1 : L ¯ ℓ = + 2 lower ( B ¯ k L k - ℓ )

Input File: omh/appendix/theory/cholesky.omh