Orthogonal projections of multidimensional ellipsoids – IV – the relation to a Schur complement of the quadratic form matrix

The first three posts in this series showed that the orthogonal projection of a (n-1)-dimensional ellipsoid in a n-dimensional Euclidean space has a surface which is a (n-2)-dimensional ellipsoid. Our original ellipsoid is defined by a quadratic form and a respective matrix. The projection can be performed along the direction of an arbitrary vector onto a (n-1)-dimensional subspace, whose vector-elements are orthogonal to the projection vector. This sub-space orthogonally complements the 1-dimensional sub-space given by an infinite line along the projection vector.
In this post I briefly touch a subject which shows us again an interesting link between geometry and linear algebra. We show that the matrix which defines the quadratic form of the border ellipsoid of the projection image is given by a Schur complement of the original ellipsoid’s matrix. This insight can be extended to orthogonal projections onto p-dimensional sub-spaces.

Previous posts

Quadratic form

We have an ellipsoid in the ℝⁿ. It is defined as a closed (n-1)-dim hypersurface E

\[ \text{vectors } \pmb{x}_e \text{ of (} n\text{-1)-dim ellipsoid } E\,: \quad \pmb{x}_{e} \in E = \left\{ \pmb{x}_e \,\, | \,\, \pmb{x}_e \, \pmb{\operatorname{\Sigma}}^{-1} \, \pmb{x}_{e} \,=\, 1 \, \right\} \subset \mathbb{R}^n \,. \tag{1} \]

As in previous posts we have named the matrix of the quadratic form Σ^-1 to keep in touch with Multivariate Normal Distributions. The matrices Σ and Σ^-1 are symmetric, invertible and positive-definite. Σ has a Cholesky decomposition

\[ \pmb{\operatorname{\Sigma}} \,=\, \pmb{\operatorname{A}} \, \pmb{\operatorname{A}}^T \,. \tag{2} \]

Matrix A generates the ellipsoid when applied to vectors u of the unit sphere 𝕊^n-1 (u ∈ 𝕊^n-1) :

\[ ||\pmb{u}|| = \sqrt{ u_1^2 + u_2^1 + … + u_n^2 \, } \,=\, 1 \,, \,\, \, \pmb{u} \in \mathbb{S}^{n-1} \, , \tag{3} \]

\[ \pmb{u} \in \mathbb{S}^{n-1} \,\Rightarrow \, \pmb{\operatorname{A}} \, \pmb{u} \in E \,. \tag{4} \]

The Cartesian Coordinate System [CCS] is defined by orthogonal unit-vectors e_i (1 ≤ i ≤ n). We write down our vectors x and the matrix in sectioned form – one component of vector z along e_n and the rest in the (n-1)-dim sub-space spanned by e_i (1 ≤ i ≤ n-1):

\[ \pmb{x} \,=\, \begin{pmatrix} \pmb{y} \\ z \end{pmatrix} \,, \,\,\, \text{with} \,\,\, \pmb{x} \in \mathbb{R}^n, \,\,\, \pmb{y} \in \mathbb{R}^{n-1}, \,\,\, z \in \mathbb{R} \,. \tag{5} \]

We can always reorganize the order of unit vectors that our chosen unit vector is the last one. Remember that the change of order of unit vectors corresponds to an orthogonal transformation to a new base of the ℝⁿ. So, without loosing generality we choose e_n to give us the projection’s direction below. To cover the effect upon the matrix we write it down partitioned into blocks:

\[ \pmb{\operatorname{\Sigma}}^{-1} \,\equiv \pmb{\operatorname{Q}} \,=\, \begin{pmatrix} \pmb{\operatorname{Q}}_{yy} & \pmb{q}_{yz} \\ (\pmb{q}_{yz})^T & q_{zz} \end{pmatrix} \,. \tag{6} \]

The introduction of Q is just to make writing simpler as we can omit the {-1}. We see

\[ \pmb{\operatorname{Q}}_{yy} \,\in \, \mathbb{R}^{(n-1)\operatorname{x}\,(n-1)} \,, \,\, \pmb{q}_{yz} \in \mathbb{R}^{(n-1)\operatorname{x}1} \,, \,\, q_{zz} \in \mathbb{R} \,. \tag{7} \]

Orthogonal projection and condition for tangential points

We introduce the projection P along e_n as a projection matrix

\[ \pmb{\operatorname{P}} \,=\, \pmb{\mathbb{I}}_n \,-\, \pmb{e}_n \, \pmb{e}_n^T \,. \tag{8} \]

What property characterizes the critical points x_e^t,

\[ \pmb{x}_e^t \,=\, \begin{pmatrix} \pmb{y}_e^t \\ z_e^t \end{pmatrix} \,, \tag{9} \]

on the ellipsoid that give us the surface of the projection image? Answer from previous posts: e_n must be tangential and orthogonal to the normal vector of the ellipsoid’s surface manifold at such a tangential point. As we have seen in previous posts that the normal vector to the ellipsoid’s surface at vector x_e is given by Qx_e. Thus the condition is:

\[ (\pmb{e}_n)^T \, \pmb{\operatorname{Q}} \,\pmb{x}_e^t \,=\, 0 \,. \tag{10} \]

Furthermore, we know

\[ \pmb{y}_e^t \,=\, \pmb{\operatorname{P}} \, \pmb{x}_e^t \,. \tag{11} \]

The vectors (y_e^t , 0)^T reside in a subspace V_p ℝⁿ whose vectors are orthogonal to e_n:

\[ (\pmb{y}_e, 0) \,, \, (\pmb{y}_e^t, 0) \in S_P\, \quad \text{with} \,\,\, S_P \,=\, \left\{ \pmb{x} \,\, | \,\,\, \pmb{x} \bullet \pmb{e}_n = 0 \,, \,\,\, \pmb{x} \perp \pmb{e}_n \, \right\} \,. \]

This sub-space is a representation of the ℝ^(n-1). So far, we learned nothing new compared to previous posts. But now let us draw conclusions regarding the structure of Q. A quick check of the matrix multiplication shows that condition (9) means

\[ \begin{align} (\pmb{q}_{yz})^T \, \pmb{y}_e^t + q_{zz}\,z_e^t \,=\, 0 \,, \tag{11} \\[10pt] z_e^t \,=\, – {1 \over q_{zz} } \, (\pmb{q}_{yz})^T \, \pmb{y}_e^t \,. \tag{12} \end{align} \]

Now we do once again the same as in previous posts. We apply condition (1) to our tangential points x_e^t, resolve the matrix multiplication and insert eq. (12) for z_e^t:

\[ \begin{align} & (\pmb{x}_e^t)^T \, \pmb{\operatorname{Q}} \, (\pmb{x}_e^t) \,=\, 1 \quad \Rightarrow \\[10pt] & (\pmb{y}_e^t)^T\, \pmb{\operatorname{Q}}_{yy} \, \pmb{y}_e^t \, +\, 2 \, \left[ (\pmb{y}_e^t)^T \, \pmb{q}_{yz}\right] * z_e^t \, +\, q_{zz}\,(z_e^t)^2 \,=\, 1 \,, \tag{13}\\[10pt] & (\pmb{y}_e^t)^T\, \pmb{\operatorname{Q}}_{yy} \, \pmb{y}_e^t \, -\, {2 \over q_{zz} } \, \left[ (\pmb{y}_e^t)^T \, \pmb{q}_{yz} \right] \, (\pmb{q}_{yz})^T \, \pmb{y}_e^t \,+\, { q_{zz} \over q_{zz}^2} \,\left( \, -\, (\pmb{q}_{yz})^T \, \pmb{y}_e^t \right)^2 \,, \tag{14} \\[10pt] & (\pmb{y}_e^t)^T\, \pmb{\operatorname{Q}}_{yy} \, \pmb{y}_e^t \, -\, {2 \over q_{zz} } \, \left[ (\pmb{y}_e^t)^T \, \pmb{q}_{yz} \right] \, (\pmb{q}_{yz})^T \, \pmb{y}_e^t \,+\, {1\over q_{zz} }\, (\pmb{y}_e^t)^T \, \pmb{q}_{yz} \, (\pmb{q}_{yz})^T \, \pmb{y}_e^t \,=\, 1 \,, \tag{15} \\[10pt] & \Rightarrow \quad (\pmb{y}_e^t)^T\, \left[ \, \pmb{\operatorname{Q}}_{yy} \,-\, {1 \over q_{zz}} \, \pmb{q}_{yz} \, (\pmb{q}_{yz})^T \, \right] \, \pmb{y}_e^t \,=\, 1\,. \tag{16} \end{align} \]

We found again a matrix mediating a quadratic form – this time working on the reduced vectors y_e^t.

The matrix of the quadratic form for the surface of the projection image is a Schur complement

Eq. (16) above gives us a certain symmetric reduced ((n-1) x (n-1)) matrix Q_P,red, which affects the projection image vectors (y_e^t) ∈ S_P of the vectors x_e^t (giving the tangential points on the ellipsoid):

\[ \begin{align} & \small{\text{Matrix of a quadratic form for vectors } \pmb{y}_e \text{ in of (} n\text{-1)-dim } } S_P \,: \\[10pt] & \pmb{\operatorname{Q}}_{P,red} \,=\, \pmb{\operatorname{Q}}_{yy} \,-\, {1 \over q_{zz}} \, \pmb{q}_{yz} \, (\pmb{q}_{yz})^T \,. \tag{17} \end{align} \]

It is relatively easy to show that this matrix is positive definite because Q was it. Just regard eigenvalues and the above analysis in an eigen-system. The matrix defined by eq. (17) is just the so called Schur complement of block q_zz of matrix Q (= Σ^-1).

Now, we could repeat the whole argumentation for our (n-2)-dimensional ellipsoid defined by eq. (16) with a further projection down to an even lower-dimensional sub-space. However, we repeat the whole process for a projection operator that eliminates (n-p) dimensions to see effects regarding the matrix structure in cases of orthogonal projections down to p-dimensional sub-space.

Generalization – orthogonal projection down to a p-dimensional sub-space

We introduce a (nxn)-projection operators Pⁿ_p and Pⁿ_(n-p)as

\[ \begin{align} \pmb{\operatorname{P}}^n_p \,&=\, \begin{pmatrix} \pmb{\mathbb{I}}_p & \quad\:\,\, \pmb{\mathbb{O}}_{(n-p)\operatorname{x}p} \\ \pmb{\mathbb{O}}_{p\operatorname{x}(n-p)} & \pmb{\mathbb{O}}_{p\operatorname{x}p} \end{pmatrix} \,=\, \pmb{\mathbb{I}}_p \,-\, \sum_{i=p+1}^n \, \pmb{e}_i \, (\pmb{e}_i)^T \,, \tag{18}\\[10pt] \pmb{\operatorname{P}}^n_{n-p} \,&=\, \pmb{\mathbb{I}}_n \, -\, \pmb{\operatorname{P}}^n_p \,. \tag{19} \end{align} \]

A matrix 𝕀_q is the (qxq)-identiy matrix. The matrices 𝕆_mxn are (mxn) zero matrices. Let us look at special sets of vectors, V_p ⊂ ℝⁿ and V_n-p ⊂ ℝⁿ:

\[ \begin{align} & \,\, V_p \, =\, \left\{ \, \pmb{y} \,=\, (y_1, …, y_p, \, 0, …., 0 )^T \,\, \in \mathbb{R}^n\,\, |\,\, \, \text{with} \,\,\,\, y_i = 0 , \,\,\, \text{for} \,\,\, i \gt p \right\} \,, \tag{20} \\[10pt] & V_{n-p} \;=\, \left\{ \, \pmb{z} \,=\, (0, …, 0, z_{p+1}, …, z_n )^T \in \mathbb{R}^n\,\, |\,\, \,\text{with} \,\,\,\, z_i = 0 , \,\,\, \text{for} \,\,\, i \le p \right\} \,, \tag{21} \\[10pt] & \, \pmb{y} \perp \pmb{z} \,\, \iff \,\, \pmb{y}^T \pmb{z} \,=\, 0 \,. \tag{22} \end{align}\]

Vectors y ∈ V_p and z ∈ V_n-p are orthogonal to each other. The sets, V_p and V_n-p, correspond to two orthogonal sub-vector-spaces, S_P and (S_P)^⊥, of the ℝⁿ. Subspace S_P is p-dimensional, subspace (S_P)^⊥ is (n-p)-dimensional. They are complementary regarding the full ℝⁿ :

\[ \mathbb{R}^n \,=\, S_P \,\oplus (S_P)^{\perp} \,. \]

We introduce respective reduced vectors

\[ \begin{align} \pmb{y}_{e,r} \,&= \, (y_1, ……… , y_p)^T \in S_P \sim \mathbb{R}^p\,, \\[10pt] \pmb{z}_{e,r} \,&=\, (z_{p+1}, …, z_n)^T \in (S_P)^{\perp} \sim \mathbb{R}^{(n-p)} \,, \\[10pt] \pmb{x}_e \,&=\, \begin{pmatrix} \pmb{y}_{e,r} \\ \pmb{z}_{e,r} \end{pmatrix} \,. \tag{23} \end{align} \]

Projection Pⁿ_p produces vectors y in V_p. It is therefore natural to write the vectors x_e ∈ E as a sum of a vector y_e ∈ V_p and a z_e ∈ V_n-p :

\[ \pmb{x}_e \,= \, \pmb{y}_e + \pmb{z}_e \,, \quad \text{with} \:\: \pmb{y}_e = \pmb{\operatorname{P}}^n_p \, \pmb{x}_e \,, \,\,\, \pmb{z}_e = \pmb{\operatorname{P}}^n_{n-p} \, \pmb{x}_e \,. \tag{24} \]

To understand matrix-operations working on the reduced vectors, we partition the (n x n) matrix Q into respective blocks with respect to the orthogonal sub-spaces (S_P ⊕ (S_P)^⊥):

\[ \pmb{\operatorname{\Sigma}}^{-1} \,\equiv \pmb{\operatorname{Q}} \,=\, \begin{pmatrix} \pmb{\operatorname{Q}}_{yy} & \pmb{Q}_{yz} \\ (\pmb{Q}_{yz})^T & \pmb{Q}_{zz} \end{pmatrix} \,. \tag{25} \]

\[ \pmb{\operatorname{Q}}_{yy} \,\in \, \mathbb{R}^{p\operatorname{x}p} \,, \,\, \pmb{\operatorname{Q}}_{yz} \in \mathbb{R}^{p\operatorname{x}(n-p)} \,, \,\, \pmb{\operatorname{Q}}_{zz} \in \mathbb{R}^{(n-p)\operatorname{x}(n-p)} \,. \tag{26} \]

We use the orthogonality condition for tangent vectors along the relevant e_i (p+1 ≤ i ≤ n) spanning V_n-p. You can imagine the resulting condition as a summary of searching for the the relevant points for surfaces of (n-p) subsequent projections, each projection along one of the series e_p+1, e_p+2, … e_n, eventually down to V_P (or regarding the reduced vectors to S_P). Note also: Due to the quadratic form, for any vector x_e on the ellipsoid the y-part is maximized if the z-related part is minimized:

\[ \begin{align} & \left(\pmb{y}_e + \pmb{z}_e \right)^T \pmb{\operatorname{Q}} \, \pmb{x}_e \,=\, 1 \quad \Rightarrow \\[10pt] &\left(\pmb{y}_e\right)^T \pmb{\operatorname{Q}} \, \pmb{x}_e \,=\, 1 \,-\, \left(\pmb{z}_e\right)^T \pmb{\operatorname{Q}} \, \pmb{x}_e \,. \tag{27} \end{align} \]

The respective minimization condition reduces the number of relevant tangential points for the surface of the projection image:

\[ \left(\pmb{z}_e^t\right)^T \pmb{\operatorname{Q}} \, \pmb{x}_e^t \, =\, 0 \,. \tag{28} \]

Note that this actually means the respective original vectors u_e^t of our x_e^t must be vertical to all back-transformed A^-1e_j (p+1 ≤ j ≤ n)

\[ \begin{align} & \pmb{u}_e^t \,=\, \pmb{\operatorname{A}}^{-1} \pmb{x}_e^t \, \\[10pt] & \pmb{u}_e^t \,\perp\, \pmb{\operatorname{A}}^{-1} \pmb{e}_j \,, \quad \forall \,j \,\,\, \text{with} \,\,\, p+1 \,\le \, j \, \le\, n \,. \tag{29} \end{align} \]

This can be used to find the relevant vectors u_e^t of the unit sphere 𝕊^n-1 from which we can construct the tangential points x_e^t with the help of matrix A. In one of the next post we will use this insight for a numerical experiment.

Let us return to the partitioned matrix Q. Expanding condition (28) and checking for relevant sub-matrices gives us:

\[ \begin{align} & \left(\pmb{z}_e^t\right)^T \pmb{\operatorname{Q}} \, \pmb{x}_e^t \,= \left(\pmb{z}_e^t\right)^T \pmb{\operatorname{Q}} \, ( \pmb{y}_e^t + \pmb{z}_e^t) \,= \\[10pt] & \left(\pmb{z}_{e,r}^t\right)^T [\pmb{\operatorname{Q}}_{yz}] ^T \, \pmb{y}_{e,r}^t \,+\, \left(\pmb{z}_{e,r}^t\right)^T \pmb{\operatorname{Q}}_{zz} \, \pmb{z}_{e,r}^t \,= \\[10pt] & \left(\pmb{z}_{e,r}^t\right)^T \left( \, [ \pmb{\operatorname{Q}}_{yz}]^T \, \pmb{y}_{e,r}^t \,+\, \pmb{\operatorname{Q}}_{zz} \, \pmb{z}_e^t \, \right) \,= \, 0 \quad \Rightarrow \\[10pt] & \pmb{z}_{e,r}^t \,=\, – \, [ \pmb{\operatorname{Q}}_{zz} ]^{-1} \, \left(\, [ \pmb{\operatorname{Q}}_{yz}]^T \, \, \pmb{y}_{e,r}^t\,\right) \,. \tag{30} \end{align} \]

Now we evaluate the quadratic form for our tangential points on the ellipsoid, which give us the surface points of the projection image,

\[ \begin{align} & (\pmb{x}_e^t)^T \pmb{\operatorname{Q}} \, \pmb{x}_e^t\,=\, 1 \,, \tag{31} \\[10pt] & \left( (\pmb{y}_{e,r}^t)^T,\, (\pmb{z}_{e,r}^t)^T \right) \, \begin{pmatrix} \pmb{\operatorname{Q}}_{yy} & \pmb{Q}_{yz} \\ (\pmb{Q}_{yz})^T & \pmb{Q}_{zz} \end{pmatrix} \,\begin{pmatrix} \pmb{y}_{e,r}^t \\ \pmb{z}_{e,r}^t \end{pmatrix} \,=\, 1\,. \tag{32} \end{align} \]

Due to our solution of condition (28), the right side of eq. (27) reduces to 1 and eq.(28) can be simplified for our tangent vectors:

\[ \left(\pmb{y}_e^t \right)^T \pmb{\operatorname{Q}} \, \pmb{x}_e^t \,=\, 1 \,. \tag{33} \]

This leads to

\[ \left( (\pmb{y}_{e,r}^t)^T,\, (\pmb{\mathbb{O}}_{n-p,1})^T \right) \, \begin{pmatrix} \pmb{\operatorname{Q}}_{yy} & \pmb{Q}_{yz} \\ (\pmb{Q}_{yz})^T & \pmb{Q}_{zz} \end{pmatrix} \,\begin{pmatrix} \pmb{y}_{e,r}^t \\ \pmb{z}_{e,r}^t \end{pmatrix} \,=\, 1\,. \tag{34} \]

The zero sub-vector on the left side leads to the following expressions on the left side to

\[ ( \pmb{y}_{e,r}^t )^T \pmb{\operatorname{Q}}_{yy} \, \pmb{y}_{e,r}^t \,+\, ( \pmb{y}_{e,r}^t )^T \pmb{\operatorname{Q}}_{yz} \, \pmb{z}_{e,r}^t \,=\, 1 \,. \tag{35} \]

We insert our result of eq. (30) to get

\[ \begin{align} (\pmb{y}_{e,r}^t )^T \pmb{\operatorname{Q}}_{yy} \, \pmb{y}_{e,r}^t \,-\, (\pmb{y}_{e,r}^t )^T \pmb{\operatorname{Q}}_{yz} \pmb{\operatorname{Q}}_{zz}^{-1} \pmb{\operatorname{Q}}_{yz}^T \, \pmb{y}_{e,r}^t \,=\, 1 \,, \\[10pt] (\pmb{y}_{e,r}^t )^T \left[ \pmb{\operatorname{Q}}_{yy} \,-\, \pmb{\operatorname{Q}}_{yz} \pmb{\operatorname{Q}}_{zz}^{-1} \pmb{\operatorname{Q}}_{yz}^T \right] \, \pmb{y}_{e,r}^t \,=\, 1 \,. \tag{36} \end{align} \]

Again, we find a reduced matrix Q_P,red which provides a quadratic form for the reduced vectors in sub-space S_P (corresponding to the target space of projection Pⁿ_p) :

\[ \pmb{\operatorname{Q}}_{P,red} \,:=\, \pmb{\operatorname{Q}}_{yy} \,-\, \pmb{\operatorname{Q}}_{yz} \pmb{\operatorname{Q}}_{zz}^{-1} \pmb{\operatorname{Q}}_{yz}^T \,\, \in \,\, \mathbb{R}^{p\,x\, p} \,. \tag{37} \]

\[ \pmb{y}_{e,r}^T \, \pmb{\operatorname{Q}}_{P,red} \, \pmb{y}_{e,r} \,=\, 1 \,. \tag{38} \]

As we projected the “tangential” vectors, this quadratic form controls the (p-1)-dimensional surface of the projection image of the original ellipsoid. This defined a (p-1)-dimensional ellipsoid E_Sp :

\[ \begin{align} & \text{(}p\text{-1)-dim ellipsoid } E_{Sp} \,\, : \\[10pt] & E_{Sp} \:=\, \left\{ \pmb{y}_{p,r} \,\, | \,\, (\pmb{y}_{p,r})^T \, \pmb{\operatorname{Q}}_{P,red} \, \pmb{y}_{p,r} \,=\, 1 \,\, \land \,\, \begin{pmatrix} \pmb{y}_{p,r} \\ \pmb{\mathbb{O}}_{n-p,1} \end{pmatrix} \in V_p \, \right\} \subset S_P \,. \tag{39} \end{align} \]

And, most importantly: The matrix Q_P,red is the Schur complement of the original matrix Q with respect to block Q_zz (based on a block partitioning with respect to the complementary sub-spaces S_P and (S_P)^⊥).

Conclusion

An orthogonal projection of an (n-1)-dimensional ellipsoid in the ℝⁿ down to a p-dimensional sub-space results in a p-dimensional volume figure. The (p-1)-dimensional surface of this figure is an ellipsoid, too. The matrix controlling the quadratic form in the projection’s target sub-space is a Schur complement of the original matrix Q (= Σ^-1). The Schur complement relates to matrix blocks which reflect complementary orthogonal p– and (n-p)-dimensional sub-spaces of the ℝⁿ.

In the next post we are going to use this result to find out, how the respective inverse matrices Q^-1= Σ and [Q_P,red]^-1=Σ_P,red are related. We shall see that this relation is rather simple. Stay tuned …