Introduction to Automatic Differentiation for Optimization
Shaun Forth
EUCCO06: the 2nd European Conference on Computational
Optimization. April 2nd-4th, 2007,
We review the forward and reverse mode algorithms of automatic differentiation in the context of gradient-based optimization. A short computational complexity analysis shows that using the reverse mode enables the gradient of an objective function to be calculated for a small multiple of the number of floating point operations required for the objective function itself - the cheap gradient result. We briefly review algorithms for Hessians and Jacobians - Jacobian compression may be advantageous for large, sparse Jacobians of constraints.
Download
PDF: saf_eucco07.pdf