James E. Gentle, PhD, is University Professor of Computational Statistics at George Mason University. He is a Fellow of the American Statistical Association (ASA) and of the American Association for the Advancement of Science. Professor Gentle has held several national offices in the ASA and has served as editor and associate editor of journals of the ASA as well as for other journals in statistics and computing. He is author of Random Number Generation and Monte Carlo Methods (Springer, 2003) and Computational Statistics (Springer, 2009).

Matrix algebra is one of the most important areas of mathematics for data analysis and for statistical theory. This much-needed work presents the relevant aspects of the theory of matrix algebra for applications in statistics. It moves on to consider the various types of matrices encountered in statistics, such as projection matrices and positive definite matrices, and describes the special properties of those matrices. Finally, it covers numerical linear algebra, beginning with a discussion of the basics of numerical computations, and following up with accurate and efficient algorithms for factoring matrices, solving linear systems of equations, and extracting eigenvalues and eigenvectors.

Part I Linear Algebra

1 Basic Vector/Matrix Structure and Notation

1.1 Vectors

1.2 Arrays

1.3 Matrices

1.4 Representation of Data

2 Vectors and Vector Spaces

2.1 Operations on Vectors

2.1.1 Linear Combinations and Linear Independence

2.1.2 Vector Spaces and Spaces of Vectors

2.1.3 Basis Sets for Vector Spaces

2.1.4 Inner Products

2.1.5 Norms

2.1.6 Normalized Vectors

2.1.7 Metrics and Distances

2.1.9 The "One Vector"

2.2 Cartesian Coordinates and Geometrical Properties of Vectors

2.2.1 Cartesian Geometry

2.2.2 Projections

2.2.3 Angles between Vectors

2.2.4 Orthogonalization Transformations; Gram-Schmidt .

2.2.5 Orthonormal Basis Sets

2.2.6 Approximation of Vectors

2.2.7 Flats, Affine Spaces, and Hyperplanes

2.2.8 Cones

2.2.9 Cross Products in IR3

2.3 Centered Vectors and Variances and Covariances of Vectors

2.3.1 The Mean and Centered Vectors

2.3.3 Covariances and Correlations between Vectors

Exercises

3 Basic Properties of Matrices

3.1 Basic Definitions and Notation

3.1.1 Matrix Shaping Operators

3.1.2 Partitioned Matrices

3.1.3 Matrix Addition

3.1.4 Scalar-Valued Operators on Square Matrices:The Trace

3.1.5 Scalar-Valued Operators on Square Matrices:The Determinant

3.2 Multiplication of Matrices and Multiplication ofVectors and Matrices

3.2.1 Matrix Multiplication (Cayley)

3.2.3 Elementary Operations on Matrices

3.2.4 The Trace of a Cayley Product that Is Square

3.2.5 The Determinant of a Cayley Product of Square Matrices

3.2.6 Multiplication of Matrices and Vectors

3.2.7 Outer Products

3.2.8 Bilinear and Quadratic Forms; Definiteness

3.2.9 Anisometric Spaces

3.2.10 Other Kinds of Matrix Multiplication

3.3 Matrix Rank and the Inverse of a Matrix

3.3.1 The Rank of Partitioned Matrices, Products of Matrices, and Sums of Matrices

3.3.2 Full Rank Partitioning

3.3.4 Full Rank Factorization

3.3.5 Equivalent Matrices

3.3.6 Multiplication by Full Rank Matrices

3.3.7 Gramian Matrices: Products of the Form ATA

3.3.8 A Lower Bound on the Rank of a Matrix Product

3.3.9 Determinants of Inverses

3.3.10 Inverses of Products and Sums of Nonsingular Matrices

3.3.11 Inverses of Matrices with Special Forms

3.3.12 Determining the Rank of a Matrix

3.4 More on Partitioned Square Matrices: The Schur Complement

3.4.1 Inverses of Partitioned Matrices

3.4.2 Determinants of Partitioned Matrices

3.5.1 Solutions of Linear Systems

3.5.2 Null Space: The Orthogonal Complement

3.6 Generalized Inverses

3.6.1 Special Generalized Inverses; The Moore-Penrose Inverse

3.6.2 Generalized Inverses of Products and Sums of Matrices

3.6.3 Generalized Inverses of Partitioned Matrices

3.7 Orthogonality

3.8 Eigenanalysis; Canonical Factorizations

3.8.1 Basic Properties of Eigenvalues and Eigenvectors

3.8.2 The Characteristic Polynomial

3.8.3 The Spectrum

3.8.4 Similarity Transformations

3.8.6 Similar Canonical Factorization; Diagonalizable Matrices

3.8.7 Properties of Diagonalizable Matrices

3.8.8 Eigenanalysis of Symmetric Matrices

3.8.9 Positive Definite and Nonnegative Definite Matrices

3.8.10 Generalized Eigenvalues and Eigenvectors

3.8.11 Singular Values and the Singular Value Decomposition (SVD)

3.9 Matrix Norms

3.9.1 Matrix Norms Induced from Vector Norms

3.9.2 The Frobenius Norm - The "Usual" Norm

3.9.3 Other Matrix Norms

3.9.4 Matrix Norm Inequalities

3.9.6 Convergence of a Matrix Power Series

3.10 Approximation of Matrices

Exercises

4 Vector/Matrix Derivatives and Integrals

4.1 Basics of Differentiation

4.2 Types of Differentiation

4.2.1 Differentiation with Respect to a Scalar

4.2.2 Differentiation with Respect to a Vector

4.2.3 Differentiation with Respect to a Matrix

4.3 Optimization of Scalar-Valued Functions

4.3.1 Stationary Points of Functions

4.3.2 Newton's Method

4.3.3 Least Squares

4.3.4 Maximum Likelihood

4.3.5 Optimization of Functions with Constraints

4.4 Integration and Expectation: Applications to Probability Distributions

4.4.1 Multidimensional Integrals and Integrals InvolvingVectors and Matrices

4.4.2 Integration Combined with Other Operations

4.4.3 Random Variables and Probability Distributions

Exercises

5 Matrix Transformations and Factorizations

5.1 Linear Geometric Transformations

5.1.1 Transformations by Orthogonal Matrices

5.1.2 Rotations

5.1.3 Reflections

5.2 Householder Transformations (Reflections)

5.3 Givens Transformations (Rotations)

5.4 Factorization of Matrices

5.5 LU and LDU Factorizations

5.6 QR Factorization

5.6.1 Householder Reflections to Form the QR Factorization

5.6.2 Givens Rotations to Form the QR Factorization

5.6.3 Gram-Schmidt Transformations to Form theQR Factorization

5.7 Factorizations of Nonnegative Definite Matrices

5.7.1 Square Roots

5.7.2 Cholesky Factorization

5.7.3 Factorizations of a Gramian Matrix

5.9 Other Incomplete Factorizations

Exercises

6 Solution of Linear Systems

6.1 Condition of Matrices

6.1.1 Condition Number

6.1.2 Improving the Condition Number

6.1.3 Numerical Accuracy

6.2 Direct Methods…