# Optimization Algorithms on Matrix Manifolds

P.-A. Absil
R. Mahony
R. Sepulchre
Pages: 240
https://www.jstor.org/stable/j.ctt7smmk

1. Front Matter
(pp. i-vi)
(pp. vii-x)
3. List of Algorithms
(pp. xi-xii)
4. Foreword
(pp. xiii-xiv)
Paul Van Dooren

Constrained optimization is quite well established as an area of research, and there exist several powerful techniques that address general problems in that area. In this book a special class of constraints is considered, called geometric constraints, which express that the solution of the optimization problem lies on a manifold. This is a recent area of research that provides powerful alternatives to the more general constrained optimization methods. Classical constrained optimization techniques work in an embedded space that can be of a much larger dimension than that of the manifold. Optimization algorithms that work on the manifold have therefore a...

5. Notation Conventions
(pp. xv-xvi)
6. Chapter One Introduction
(pp. 1-4)

This book is about the design of numerical algorithms for computational problems posed on smooth search spaces. The work is motivated by matrix optimization problems characterized by symmetry or invariance properties in the cost function or constraints. Such problems abound in algorithmic questions pertaining to linear algebra, signal processing, data mining, and statistical analysis. The approach taken here is to exploit the special structure of these problems to develop efficient numerical procedures.

An illustrative example is the eigenvalue problem. Because of their scale invariance, eigenvectors are not isolated in vector spaces. Instead, each eigendirection defines a linear subspace of eigenvectors....

7. Chapter Two Motivation and Applications
(pp. 5-16)

The problem of optimizing a real-valued function on a matrix manifold appears in a wide variety of computational problems in science and engineering. In this chapter we discuss several examples that provide motivation for the material presented in later chapters. In the first part of the chapter, we focus on the eigenvalue problem. This application receives special treatment because it serves as a running example throughout the book. It is a problem of unquestionable importance that has been, and still is, extensively researched. It falls naturally into the geometric framework proposed in this book as an optimization problem whose natural...

8. Chapter Three Matrix Manifolds: First-Order Geometry
(pp. 17-53)

The constraint sets associated with the examples discussed in Chapter 2 have a particularly rich geometric structure that provides the motivation for this book. The constraint sets arematrix manifoldsin the sense that they are manifolds in the meaning of classical differential geometry, for which there is a natural representation of elements in the form of matrix arrays.

The matrix representation of the elements is a key property that allows one to provide a natural development of differential geometry in a matrix algebra formulation. The goal of this chapter is to introduce the fundamental concepts in this direction: manifold...

9. Chapter Four Line-Search Algorithms on Manifolds
(pp. 54-90)

Line-search methods in ℝnare based on the update formula

$x_{k + 1} = x_k + t_k \eta _k, \caption{(4.1)}$

where ηk∈ ℝnis thesearch directionand tk∈ ℝ is thestep size. The goal of this chapter is to develop an analogous theory for optimization problems posed on nonlinear manifolds.

The proposed generalization of (4.1) to a manifold 𝓜 consists of selecting ηkas a tangent vector to 𝓜 atxkand performing a search along a curve in 𝓜 whose tangent vector att= 0 is ηk. The selection of the curve relies on the concept of retraction, introduced in Section 4.1. The...

10. Chapter Five Matrix Manifolds: Second-Order Geometry
(pp. 91-110)

Many optimization algorithms make use of second-order information about the cost function. The archetypal second-order optimization algorithm is Newton’s method. This method is an iterative method that seeks a critical point of the cost functionf(i.e., a zero of gradf) by selecting the update vector atxkas the vector along which the directional derivative of gradfis equal to −gradf(xk). The second-order information on the cost function is incorporated through the directional derivative of the gradient.

For a quadratic cost function in 𝓡n, Newton’s method identifies a zero of the gradient in one step. For...

11. Chapter Six Newton’s Method
(pp. 111-135)

This chapter provides a detailed development of the archetypal second-order optimization method, Newton’s method, as an iteration on manifolds. We propose a formulation of Newton’s method for computing the zeros of a vector field on a manifold equipped with an affine connection and a retraction. In particular, when the manifold is Riemannian, this geometric Newton method can be used to compute critical points of a cost function by seeking the zeros of its gradient vector field. In the case where the underlying space is Euclidean, the proposed algorithm reduces to the classical Newton method. Although the algorithm formulation is provided...

12. Chapter Seven Trust-Region Methods
(pp. 136-167)

The plain Newton method discussed in Chapter 6 was shown to be locally convergent to any critical point of the cost function. The method does not distinguish among local minima, saddle points, and local maxima: all (nondegenerate) critical points are asymptotically stable fixed points of the Newton iteration. Moreover, it is possible to construct cost functions and initial conditions for which the Newton sequence does not converge. There even exist examples where the set of nonconverging initial conditions contains an open subset of search space.

To exploit the desirable superlinear local convergence properties of the Newton algorithm in the context...

13. Chapter Eight A Constellation of Superlinear Algorithms
(pp. 168-188)

The Newton method (Algorithm 5 in Chapter 6) applied to the gradient of a real-valued cost is the archetypal superlinear optimization method. The Newton method, however, suffers from a lack of global convergence and the prohibitive numerical cost of solving the Newton equation (6.2) necessary for each iteration. The trust-region approach, presented in Chapter 7, provides a sound framework for addressing these shortcomings and is a good choice for a generic optimization algorithm. Trust-region methods, however, are algorithmically complex and may not perform ideally on all problems. A host of other algorithms have been developed that provide lower-cost numerical iterations...

14. Appendix A Elements of Linear Algebra, Topology, and Calculus
(pp. 189-200)
15. Bibliography
(pp. 201-220)
16. Index
(pp. 221-224)