j. differential geometry 51 (1999) 375-399

AN ANALYTIC COMPACTIFICATION OF SYMPLECTIC GROUP

HONGYU L. HE

Introduction

In this paper, we propose an analytic compactification from the symplectic group Sp_{2n}(ℝ) to the symmetric space U(2n)/O_{2n}(ℝ). We obtain this compactification through the study of Bargmann-Segal model of the oscillator (metaplectic) representation.

In the theory of symmetric spaces, a Hermitian symmetric space of noncompact type can be realized as a bounded domain in a complex vector space. For example, SL(2, ℝ)/SO(2) can be realized as the Poincaré disc. Harish-Chandra studied harmonic analysis on such a domain. The studies along this line had been quite fruitful for holomorphic discrete representations. For a noncompact reductive group G, we also wish to do analysis on an appropriate compactification \overline{G} of G. If the “push forward”of matrix coefficients of unitary representations of G behaves reasonably well, we may gain a better understanding of unitary representations of G through the study of functions on \overline{G} (see [5]).

To begin with, let X be an analytic manifold. We say (i, \overline{X}) is an analytic compactification of X, if \overline{X} is a compact analytic manifold and

i : X → \overline{X}

is an analytic embedding such that i(X) is open dense in \overline{X}. Let G be the standard symplectic group. Then G has a KAK decomposition, where K is U(n) = Sp_{2n}(ℝ) ∩ SO_{2n}(ℝ) and A ≅ ℝ^{n}. Let K^{o} be the opposite group. Then G has a K × K^{o} action. For the symmetric space Y = U(2n)/O_{2n}(ℝ), one can also define a K × K action on Y, where K × K is embedded diagonally into U(2n). We define a group isomorphism τ : K × K^{o} → K × K by

Received April 7, 1998, and, in revised form, July 13, 1998.

τ(k_{1}, k_{2}) = (\overline{k_{1}}, k_{2}^{−1}) (k_{1}, k_{2} ∈ K).

Thus K × K^{o} can be identified with K × K through τ. In this paper, we prove the following theorem.

Theorem 0.1. There exists a U(n) × U(n)^{o}-equivariant analytic embedding:

H : Sp_{2n}(ℝ) → U(2n)/O_{2n}(ℝ)

such that (H, U(2n)/O_{2n}(ℝ)) is an analytic compactification.

If f is a U(n)-finite matrix coefficient of an irreducible nontrival unitary representation of Sp_{2n}(ℝ), then f vanishes at infinity (see Theorem 5.4 [3]). Thus f can automatically be extended into a continuous function f^{0} on U(2n)/O_{2n}(ℝ) by zero. It is well-known that f is analytic on Sp_{2n}(ℝ). Therefore f^{0} is analytical on H(Sp_{2n}(ℝ)). The intrigue question is whether f^{0} is analytic over the boundary. If it is the case, is the matrix coefficients’ being analytic a consequence of unitarity of the underlying representation? In other words, can there be a nonunitary irreducible representation whose U(n)-finite matrix coefficient can be extended to an analytic function on U(2n)/O(2n)? Of course, these questions are far beyond the scope of this paper, we wish to address these questions in the future.

In fact, U(2n)/O_{2n} can be realized as a space of matrices. Let S_{2n} be the space of symmetric unitary matrices of the following form

{X^{t}X | X ∈ U(2n)}.

If 2n is fixed, we will write S. Now g ∈ U(2n) acts on S by

τ(g) : s → gsg^{t} (s ∈ S).

We compute the isotropic subgroup at the identity,

U(2n)_{I} = {U^{t}U = I | U ∈ U(2n)} = O_{2n}.

Therefore S can be identified with U(2n)/O_{2n}. In this paper, we establish the compactification on the model S through the study of the Bargmann-Segal model of the oscillator representation.

Roughly speaking, the Bargmann-Segal model is the “minimal ”unitary representation of the double covering of Sp_{2n}(ℝ). The underlying Hilbert space is the space of L^{2}-analytic functions with respect to the Gaussian measure. Then the group action of { \atop Sp_{2n}(ℝ)} can be expressed as integration operators. We observe some nice structure in the integration kernel which leads to the compactification (H, S).

Recall that Sp_{2}(ℝ) = SL(2, ℝ) and SL(2, ℝ) ≅ SU(1, 1). The latter is given by

( \begin{matrix}a & b \\ c & d \end{matrix} ) → \frac{1}{2} ( \begin{matrix}1 & i \\ i & 1 \end{matrix} ) ( \begin{matrix}a & b \\ c & d \end{matrix} ) ( \begin{matrix}1 & −i \\ −i & 1 \end{matrix} ) .

The compactification of these groups are given by the following theorem:

Theorem 0.2. We have for ( \begin{matrix}a & b \\ c & d \end{matrix} ) ∈ SL(2, ℝ)

H(( \begin{matrix}a & b \\ c & d \end{matrix} )) = ( \begin{matrix}\frac{(a−d)+(b+c)i}{(a+d)−(c−b)i} & \frac{−2i}{(a+d)−(c−b)i} \\ \frac{−2i}{(a+d)−(c−b)i} & \frac{(a−d)−(b+c)i}{(a+d)−(c−b)i} \end{matrix} ) ∈ S_{2},

and for ( \begin{matrix}p & q \\ \overline{q} & \overline{p} \end{matrix} ) ∈ SU(1, 1),

H(( \begin{matrix}p & q \\ \overline{q} & \overline{p} \end{matrix} )) = ( \begin{matrix}\frac{iq}{\overline{p}} & \frac{−i}{\overline{p}} \\ \frac{−i}{\overline{p}} & \frac{−i\overline{q}}{\overline{p}} \end{matrix} ) .

Finally, let D = S − Im(H) be the boundary at infinity. In this paper, we prove

Theorem 0.3. D is an irreducible subvariety of codimension 1 in S. The generic U(n) × U(n)-orbit in D is given by

U(n) × U(n)/{(U, V) | U = diag(±1, ±1, … ± 1) = V }.

Therefore D can be regarded as the divisor at infinity of the compactification (H, S).

The following is what is covered here in this paper. In Chapter 1, we introduce the Bargmann-Segal model of Sp_{2n}(ℝ). In Chapter 2, we introduce the structure theory of Sp_{2n}(ℝ) and define a continuous and one-to-one mapping H from Sp_{2n}(ℝ) to S. In Chapter 3, we prove that dH is nondegenerate. Thus H is a local diffeomorphism. We show that H is in fact analytic. Therefore, H is an analytic embedding. In Chapter 4, we use generalized Cartan decomposition to show that the image of H is in fact dense in U(2n)/O_{2n}(ℝ). We also provide the proofs of Theorems 0.1 and 0.2. In Chapter 5, we prove Theorem 0.3.

I should make a final remark here. It can be shown that SO_{p,q}(ℝ) also possesses an analytic compactification, namely, SO_{p+q}(ℝ). This compactification is established in [4] using reductive dual pair and Corallary 3.5 in this paper. The author would like to thank Professors Helgason, Schlichtkrull and Vogan for their advices and the referee for pointing out the question regarding divisor at infinity.

1. Bargmann-Segal model

Let V be an n−dimensional complex Hilbert space with the standard inner product (∗, ∗). Let {e_{1}, e_{2}, …, e_{n}} be an orthonormal basis of V. We write

(u, v) = Re(u, v) + iIm(u, v) (u, v ∈ V).

Then Ω(u, v) = Im(u, v) is a real symplectic form on V. Notice that

iΩ(ie_{j}, e_{k}) = iIm(ie_{j}, e_{k}) = iδ_{j}^{k}.

We fix a real basis once for all

{ie_{1}, ie_{2}, …, ie_{n}, e_{1}, …, e_{n}} = {ξ_{1}, ξ_{2}, …, ξ_{n}, e_{1}, …, e_{n}}.

If we regard V as a real vector space under such a basis, then Ω is the standard symplectic form, and Re(,) is the standard (real) inner product. From now on, whenever we regard V as a complex space, we will add a subscript ℂ. For a linear endomorphism g of V, without the subscript ℂ, g will be a real linear transform. However g_{ℂ} will be a complex linear transform of V.

Let O_{2n}(ℝ) be the subgroup of GL(V) fixing Re(,), and Sp_{2n}(ℝ) be the subgroup of GL(V) fixing Ω(,). Let U(n) be the subgroup of GL(V) fixing the (complex) inner product (,). Then

U(n) = O_{2n}(ℝ) ∩ Sp_{2n}(ℝ).

In terms of real basis, the complex multiplication by imaginary i can be identified with left multiplication by

J = ( \begin{matrix}O & I \\ −I & 0 \end{matrix} ) .

For arbitrary g ∈ Sp_{2n}(ℝ), g can be decomposed into

g = C_{g} + A_{g},

where C_{g} commutes with J, and A_{g} anticommutes with J. Thus C_{g} ∈ End_{ℂ}(V), and A_{g} is complex-conjugate linear. Explicitly,

C_{g} = \frac{1}{2}(g − JgJ), A_{g} = \frac{1}{2}(g + JgJ).

It is known that C_{g} ∈ GL_{ℂ}(V) (see [8]). Let Π_{ℂ}(V) be the set of T ∈ GL_{ℂ}(V) for which Re(T v, v) is strictly positive for all nonzero v ∈ V. According to [2], Π_{ℂ}(V) is a contractible open domain of the identity in GL_{ℂ}(V). Consequently, there is a unique continuous function

detℂ^{\frac{1}{2}} : Π_{ℂ}(V) → ℂ

such that

detℂ^{\frac{1}{2}}(I) = 1, (detℂ^{\frac{1}{2}}(T))^{2} = det_{ℂ} T, (T ∈ Π_{ℂ}(V)).

Notice here det_{ℂ} T is the determinant of T as a complex matrix. Now we define

Z_{g} = C_{g}^{−1}A_{g}, (g ∈ Sp(V, Ω)).

According to [8], we have I − Z_{g_{1}}Z_{g_{2}} ∈ Π_{ℂ}(V) for g_{1}, g_{2} ∈ Sp(V, Ω).

Let Mp(V, Ω) be the double cover of Sp_{2n}(ℝ). This group is often called the metaplectic group. Sometimes, we denote it by { \atop Sp_{2n}(ℝ)}. There is in fact a nice way to represent this group (see [8]).

Theorem 1.1.

Mp(V, Ω) = {(λ, g) | g ∈ Sp(V, Ω), λ ∈ ℂ, λ^{2} det_{ℂ}(C_{g}) = 1}.

In addition, the multiplicative structure is given by

(λ_{1}, g_{1})(λ_{2}, g_{2}) = (λ_{1}λ_{2}(detℂ^{\frac{1}{2}}(I − Z_{g_{1}}Z_{g_{2}^{−1}}))^{−1}, g_{1}g_{2}).

Now we will construct the Bargmann-Segal model. Let dx be the Euclidean measure on V. Let

dμ(x) = exp(−\frac{1}{2}(x, x))dx

be the Gaussian measure. Let P_{n} or simply P be the polynomial ring on V_{ℂ}. We define an inner product on P by

(f, g) = ∫_{V} f(x)\overline{g(x)}dμ(x), (f, g ∈ P).

Let ∥f∥^{2} = (f, f). Let F be the completion of P under ∥ ∗ ∥. Then F is exactly the space of square Gaussian integrable analytic functions. In particular, ∥ ∗ ∥-covergence implies pointwise convergence. I should refer the reader to Bargmann’s original paper [1] for details.

Theorem 1.2 (Bargmann-Segal model). Let (λ, g) ∈ Mp(V, Ω). For every f ∈ F, we define

ω(λ, g)f(z) = ∫_{V} λ exp \frac{1}{4}(2(C_{g}^{−1}z, w)−(z, Z_{g^{−1}}z)−(Z_{g}w, w))f(w)dμ(w).

Then ω is a faithful unitary representation of Mp(V, Ω). Let

H(g, z, w) = 2(C_{g}^{−1}z, w) − (z, Z_{g^{−1}}z) − (Z_{g}w, w).

If g ≠ g′, then as functions of complex variables z and w

H(g, z, w) ≠ H(g′, z, w).

Proof. A proof of the first part of the theorem can be found in [8]. Suppose g ≠ g′, but

H(g, z, w) = H(g′, z, w).

Then C_{g} = C_{g′}. Let λ ∈ ℂ, such that

λ^{2} det_{ℂ}(C_{g}) = 1.

Then (λ, g), (λ, g′) ∈ Mp(V, Ω), and

ω(λ, g) = ω(λ, g′).

This implies that g = g′, a contradiction. q.e.d.

2. Some structure theory

Since K = U(n) is a maximal compact subgroup of Sp_{2n}(ℝ), we can choose

A = {diag(λ_{1}, …, λ_{n}, λ_{1}^{−1}, …, λ_{n}^{−1}) | λ_{i} ∈ ℝ^{+}}

to be the maximal split Abelian subgroup. Then Sp_{2n}(ℝ) possesses a KAK decomposition.

an analytic compactification of symplectic group 381

Theorem 2.1. For g ∈ Sp_{2n}(ℝ), let g = k_{1}ak_{2} be a KAK decomposition. Let a = exp(H), H ∈ 𝔞. Then we have

C_{g} = k_{1} cosh(H)k_{2}, A_{g} = k_{1} sinh(H)k_{2},

Z_{g} = k_{2}^{−1} tanh(H)k_{2}, Z_{g^{−1}} = −k_{1} tanh(H)k_{1}^{−1},

where cosh(H) = \frac{1}{2}[exp(H)+exp(−H)]; sinh(H) = \frac{1}{2}[exp(H)−exp(−H)]

tanh(H) = (cosh(H))^{−1} sinh(H).

Proof. In Sp_{2n}(ℝ), the action of K commutes with J. Thus

\begin{matrix}C_{g} = & \frac{1}{2}(g − JgJ) \\ = & \frac{1}{2}(k_{1}ak_{2} − Jk_{1}ak_{2}J) \\ = & \frac{1}{2}(k_{1}ak_{2} − k_{1}JaJk_{2}) \\ = & k_{1}C_{a}k_{2}. \end{matrix} (1)

Similarly, we have

A_{g} = k_{1}A_{a}k_{2}.

Thus

Z_{g} = C_{g}^{−1}A_{g} = (k_{1}C_{a}k_{2})^{−1}(k_{1}A_{a}k_{2}) = k_{2}^{−1}(C_{a}^{−1}A_{a})k_{2}.

Since g^{−1} = k_{2}^{−1}a^{−1}k_{1}^{−1}, we have

Z_{g^{−1}} = k_{1}(C_{a^{−1}})^{−1}A_{a^{−1}}k_{1}^{−1}.

Now a simple computation shows that

JaJ = −a^{−1}, (a ∈ A).

Thus

C_{a} = \frac{1}{2}[exp(H) + exp(−H)] = cosh(H),
A_{a} = \frac{1}{2}[exp(H) − exp(−H)] = sinh(H),
Z_{a} = C_{a}^{−1}A_{a} = tanh(H).

Z_{a^{−1}} = tanh(−H) = − tanh(H).

Therefore

C_{g} = k_{1} cosh(H)k_{2}, A_{g} = k_{1} sinh(H)k_{2},
Z_{g} = k_{2}^{−1} tanh(H)k_{2}, Z_{g^{−1}} = −k_{1} tanh(H)k_{1}^{−1}.

q.e.d.

We define

sech(H) = (cosh(H))^{−1} , coth(H) = (tanh(H))^{−1}.

Combined with Theorem 1.2, we have

Theorem 2.2. Let (λ, g) be an element in Mp(V, Ω), and g = k_{1} exp(H)k_{2} be a KAK decomposition. Then

H(g, z, w) =2(sech(H)k_{1}^{−1}z, k_{2}w)
+ (k_{1}^{−1}z, tanh(H)k_{1}^{−1}z)
− (tanh(H)k_{2}w, k_{2}w).

In particular, the right-hand side does not depend on the KAK decomposition.

Recall that C_{a} is always complex linear and A_{a} is always complexconjugate linear. Suppose

H = diag(H_{1}, …, H_{n}, −H_{1}, …, −H_{n}), (H_{i} ∈ ℝ).

We write

H_{ℂ} = diag(H_{1}, H_{2}, …, H_{n}).

Then

(sech(H)z, w) = (sech(H_{ℂ})z, w).

Now we want to compute (tanh(H)z, w). Let z = iy + x with x, y ∈ ℝ^{n}. Then

tanh(H)z = tanh(H)(x + iy)
=i tanh(H_{ℂ})y − tanh(H_{ℂ})x
= − tanh(H_{ℂ})\overline{z}.

Therefore

(tanh(H)z, w) = −(tanh(H_{ℂ})\overline{z}, w),
(z, tanh(H)w) = −(z, tanh(H_{ℂ})\overline{w}).

Now

H(g, z, w) =2(sech(H_{ℂ})k_{1}^{−1}z, k_{2}w) − (k_{1}^{−1}z, tanh(H_{ℂ})\overline{k_{1}^{−1}z})
+ (tanh(H_{ℂ})\overline{k_{2}w}, k_{2}w)
=2\overline{w^{t}}\overline{k_{2}^{t}}sech(H_{ℂ})k_{1}^{−1}z − z^{t}\overline{k_{1}} tanh(H_{ℂ})k_{1}^{−1}z
+ \overline{w^{t}k_{2}^{t}} tanh(H_{ℂ})\overline{k_{2}w}
=(iz^{t}, \overline{w^{t}}) ( \begin{matrix}\overline{k_{1}} & 0 \\ 0 & \overline{k_{2}^{t}} \end{matrix} ) ( \begin{matrix}tanh(H_{ℂ}) & −isech(H_{ℂ}) \\ −isech(H_{ℂ}) & tanh(H_{ℂ}) \end{matrix} )

· ( \begin{matrix}k_{1}^{−1} & 0 \\ 0 & \overline{k_{2}} \end{matrix} ) ( iz ; \overline{w} ) .

(2)

Definition 2.1. We define

H(k_{1} exp(H)k_{2})

= ( \begin{matrix}\overline{k_{1}} & 0 \\ 0 & \overline{k_{2}^{t}} \end{matrix} ) ( \begin{matrix}tanh(H_{ℂ}) & −isech(H_{ℂ}) \\ −isech(H_{ℂ}) & tanh(H_{ℂ}) \end{matrix} ) ( \begin{matrix}k_{1}^{−1} & 0 \\ 0 & \overline{k_{2}} \end{matrix} ) .

Notice that k_{1}, k_{2} are unitary. One critical observation is that the images of H are symmetric unitary matrices. Therefore this definition of H is uniquely determined by the following equation

H(g, z, w) = (iz^{t}, \overline{w^{t}})H(g) ( iz ; \overline{w} ) . (3)

Theorem 2.3. The map H is a continuous injection from Sp_{2n}(ℝ) into U(2n).

Proof. First of all, if H(g) = H(g′), then

H(g, z, w) = H(g′, z, w), (∀ z, w ∈ V).

According to Theorem 1.2, we have g = g′. Therefore H is an injection. Since the maps g → C_{g}^{−1}, g → Z_{g}, and g → Z_{g^{−1}} are all continuous, for every z, w ∈ V, the map g → H(g, z, w) is continuous. From Equation 3 and by linearity, every entry of the matrix H(g) is a continuous function of Sp_{2n}(ℝ). Therefore, H(g) is continuous as well. q.e.d.

3. Analytic properties of H

We define 𝕋_{n} in U(2n) to be the space of matrices of the following form

T (θ) = ( \begin{matrix}diag(cos θ_{1}, …, cos θ_{n}) & diag(−i sin θ_{1}, …, −i sin θ_{n}) \\ diag(−i sin θ_{1}, …, −i sin θ_{n}) & diag(cos θ_{1}, …, cos θ_{n}) \end{matrix} )

(θ = (θ_{1}, …, θ_{n})).

We want to analyze the map κ : A → 𝕋^{n}, defined to be the restriction of H on A. Without loss of generality, let n = 1. Then

κ(exp H) = ( \begin{matrix}tanh H & −isechH \\ −isechH & tanh H \end{matrix} ) ∈ 𝕋.

κ can be regarded as a homeomorphism from ℝ to (0, π). Therefore, θ can be regarded as a continuous function of H, and H can also be regarded as a continuous function of θ. Notice that from tanh H = cos θ, we obtain

(sech(H))^{2}dH = −(sin θ)dθ.

Therefore

\frac{dθ}{dH} = −sech(H) ≠ 0, \frac{dH}{dθ} = − csc θ ≠ 0.

Since all these functions are (real) analytic, κ is an analytic embedding from A to 𝕋^{n}. From the K × K^{o} action on Sp_{2n}(ℝ) one may guess that H is in fact an embedding; however, in order to prove this, knowing the fact that κ is an embedding is not sufficient.

Let S = {U^{t}U | U ∈ U(2n)} be a subset of U(2n). Then κ(A) is contained in S. Thus H(Sp_{2n}(ℝ)) is in fact contained in S. We obtain

Lemma 3.1.

• Let U(2n) act on S by

g → U^{t}gU, (g ∈ S, U ∈ U(2n)).

Then S ≅ U(2n)/O_{2n}(ℝ).

• The map H is a continuous map from Sp_{2n}(ℝ) into S.

• Let U(n)^{o} be the opposite group of U(n). Let U(n) × U(n)^{o} act on Sp_{2n}(ℝ) by left and right multiplications respectively. Let

τ : U(n) × U(n)^{o} → U(n) × U(n)

be a group isomorphism defined as follows:

τ(k_{1}, k_{2}) = (\overline{k_{1}}, k_{2}^{−1}).

If we identify these two groups through τ, then H is equivariant with respect to these two group actions.

Now we want to compute the differential of H,

dH : T Sp_{2n}(ℝ) → T S.

Let g(t) be a germ of a smooth curve in a neigborhood of g ∈ Sp_{2n}(ℝ). Let dg be the tangent vector represented by this germ g(t). Since Sp_{2n}(ℝ) is contained in the space of 2n × 2n matrices, we can engage all our discussion in the space of 2n × 2n matrices. Thus the tangent vector dg in Sp_{2n}(ℝ) can be identified with a 2n × 2n matrix. This is going to be the perspective we take in interpreting all the equations we will have. From gg^{−1} = 1, we obtain

(dg)g^{−1} + g(dg^{−1}) = 0.

Therefore we have

dg^{−1} = −g^{−1}(dg)g^{−1}.

By standard calculus, we can prove the following lemma.

Lemma 3.2.

1. dg^{−1} = −g^{−1}(dg)g^{−1};

2. dC_{g}^{−1} = −C_{g}^{−1}(dC_{g})C_{g}^{−1};

3. dC_{g^{−1}} = −\frac{1}{2}(g^{−1}(dg)g^{−1} − Jg^{−1}(dg)g^{−1}J);

4. dA_{g^{−1}} = −\frac{1}{2}(g^{−1}(dg)g^{−1} + Jg^{−1}(dg)g^{−1}J);

5. dZ_{g} = −C_{g}^{−1}(dC_{g})Z_{g} + C_{g}^{−1}(dA_{g}); where

dC_{g} = \frac{1}{2}(dg − J(dg)J), dA_{g} = \frac{1}{2}(dg + J(dg)J);

6. dZ_{g^{−1}} = −C_{g^{−1}}^{−1}(dC_{g^{−1}})Z_{g^{−1}} + C_{g^{−1}}^{−1}(dA_{g^{−1}});

7. dZ_{g} = −C_{g}^{−1}(dC_{g})Z_{g} + C_{g}^{−1}(dA_{g}); where

dC_{g} = \frac{1}{2}(dg − J(dg)J), dA_{g} = \frac{1}{2}(dg + J(dg)J);

8. dZ_{g^{−1}} = −C_{g^{−1}}^{−1}(dC_{g^{−1}})Z_{g^{−1}} + C_{g^{−1}}^{−1}(dA_{g^{−1}}).

Now we can compute dH. Let 𝔨 ⊕ 𝔭 be the Cartan decomposition of 𝔰p_{2n}(ℝ) with 𝔨 = 𝔲(n). In fact, it can be shown that:

Lemma 3.3. The space 𝔨 is complex linear and 𝔭 is complex-conjugate linear in End(V).

Now we have the following theorem.

Theorem 3.1. The map (dH)_{g} : T_{g}(Sp_{2n}(ℝ)) → T_{H(g)}(S) is bijective.

Proof. For an arbitrary g ∈ Sp_{2n}(ℝ), let g = k_{1} exp Hk_{2} be a KAK decomposition. Because of the action of U(n) × U(n)^{o}, without loss of generality, we assume that g = exp H, H ∈ 𝔞.

1. First notice that

dim(S) = dim(U(2n))−dim(O_{2n}(ℝ)) = n(2n+1) = dim(Sp_{2n}(ℝ)).

It suffices to show that the kernel of (dH)_{g} is trivial.

2. Let dg be the equivalence class of the germ g exp tk with k ∈ 𝔨. Then we may write dg = gk. We have

\begin{matrix}dC_{g} = & \frac{1}{2}(dg − J(dg)J) \\ = & \frac{1}{2}(gk − JgkJ) \\ = & \frac{1}{2}(g − JgJ)k \\ = & C_{g}k. \end{matrix} (4)

Similarly, we have

dC_{g^{−1}} = −kC_{g^{−1}},
dA_{g} = A_{g}k, dA_{g^{−1}} = −kA_{g^{−1}}.

Thus we obtain

dC_{g}^{−1} = − C_{g}^{−1}(dC_{g})C_{g}^{−1} = −kC_{g}^{−1},
dZ_{g} = − C_{g}^{−1}(dC_{g})Z_{g} + C_{g}^{−1}(dA_{g}) = −kZ_{g} + Z_{g}k,
dZ_{g^{−1}} = − C_{g^{−1}}^{−1}(dC_{g^{−1}})C_{g^{−1}}^{−1}A_{g^{−1}} + C_{g^{−1}}^{−1}dA_{g^{−1}}
=C_{g^{−1}}^{−1}kA_{g^{−1}} − C_{g^{−1}}^{−1}kA_{g^{−1}} = 0.

Now we have proved

d(z, Z_{g^{−1}}z) = 0, (5)

dH(g, z, w) = 2(−kC_{g}^{−1}z, w) + ((kZ_{g} − Z_{g}k)w, w). (6)

Since C_{g} ∈ GL_{ℂ}(V), we can see that:

\begin{matrix}dH(g, z, w) & = 0 (∀ z, w ∈ V) \\ ⟹ & (−kC_{g}^{−1}z, w) = 0(∀ z, w ∈ V) \\ ⟹ & k = 0. \end{matrix} (7)

Conversely, k = 0 implies that dH(g, z, w) = 0(∀ z, w ∈ V).

3. On the other hand, let dg be the equivalence class of the germ g exp tp with p ∈ 𝔭. Then we may write dg = gp. Since pJ = −Jp, we have

dC_{g} = \frac{1}{2}(dg − JdgJ) = \frac{1}{2}(gp − JgpJ) = \frac{1}{2}(gp + JgJp) = A_{g}p,

dA_{g} = C_{g}p,  dA_{g^{−1}} = −pC_{g^{−1}},  dC_{g^{−1}} = −pA_{g^{−1}}.

Then

dC_{g}^{−1} = −C_{g}^{−1}(dC_{g})C_{g}^{−1} = −Z_{g}pC_{g}^{−1},
dZ_{g} = −C_{g}^{−1}(dC_{g})Z_{g} + C_{g}^{−1}(dA_{g}) = −Z_{g}pZ_{g} + p,

\begin{aligned}dZ_{g^{−1}} & = −C_{g^{−1}}^{−1}(dC_{g^{−1}})Z_{g^{−1}} + C_{g^{−1}}^{−1}dA_{g^{−1}} \\ & = C_{g^{−1}}^{−1}pA_{g^{−1}}Z_{g^{−1}} − C_{g^{−1}}^{−1}pC_{g^{−1}} \\ & = C_{g^{−1}}^{−1}p(A_{g^{−1}}Z_{g^{−1}} − C_{g^{−1}}). \end{aligned} (8)

Since g = exp H, and H ∈ 𝔞, we have

A_{g^{−1}}Z_{g^{−1}} − C_{g^{−1}} = sinh(H) tanh(H) − cosh(H)
=(cosh(H))^{−1}(sinh(H)^{2} − cosh(H)^{2})
= − (cosh(H))^{−1}.

(9)

Notice that 𝔞 is commutative. Thus our computation above is valid. The above equation implies

d(z, Z_{g^{−1}}z) =(z, −sech(H)(p)sech(H)z)
= − (z, sech(H_{ℂ})(p)sech(H_{ℂ})z).

Suppose under the real basis {ie_{j}, e_{j}}_{1}^{n},

p = ( \begin{matrix}A & B \\ B & −A \end{matrix} ) , (A^{t} = A, B^{t} = B). (10)

Therefore

p(yi + x) =i(Ay + Bx) + (By − Ax) = (Bi − A)(x − iy)
=(Bi − A)(\overline{x + iy}).

We see that

d(z, Z_{g^{−1}}z) = − (z, sech(H_{ℂ})(p)sech(H_{ℂ})z)
= − (z, sech(H_{ℂ})(Bi − A)sech(H_{ℂ})\overline{z})
= − z^{t}sech(H_{ℂ})(−Bi − A)sech(H_{ℂ})z
=z^{t}sech(H_{ℂ})(A + Bi)sech(H_{ℂ})z.

(11)

Since A + Bi is a symmetric matrix and sech(H_{ℂ}) is invertible, we have

d(z, Z_{g^{−1}}z) = 0 (∀ z ∈ V) ⟺ A + Bi = 0 ⟺ p = 0. (12)

4. For an arbitrary X = k + p ∈ 𝔤, g = exp H, we fix a germ g(t) = g exp(tX). Let p be defined as in Equation 10. Suppose that dH(g, z, w) = 0. Then combined with Equation 5 and Equation 11, we see that

d(z, Z_{g^{−1}}z) = z^{t}sech(H_{ℂ})(A + Bi)sech(H_{ℂ})z = 0.

Thus we have

d(z, Z_{g^{−1}}z) = 0(∀ z ∈ V) ⟹ p = 0

Now X = k. From Equation 7, we see that k = 0. Therefore, X = 0. Thus we have proved that

dH(g, z, w) = 0 (∀ z, w ∈ V) ⟹ X = 0.

By Equation 3 we conclude that

dH(g) = 0 ⟹ X = 0.

5. Since Sp_{2n}(ℝ) is a Lie group, the tangent space T_{g}(Sp_{2n}(ℝ)) can be identified with those germs

g exp(tX) (X ∈ 𝔤).

Thus

dH|_{g} : T_{g}(Sp_{2n}(ℝ)) → T_{H(g)}(S)

is injective. Because of the left and right K-action, this is true for all g ∈ Sp_{2n}(ℝ). q.e.d.

This theorem shows that H is an immersion, locally homeomorphism. It is also one-to-one. Thus H is a homeomorphism from Sp_{2n}(ℝ) onto an open submanifold of S. In fact H is analytic.

Theorem 3.2. The map H : Sp_{2n}(ℝ) → S is analytic.

Proof. In this proof V will be regarded as a real vector space. Then S is an analytic submanifold of B(V ⊕ V, ℂ), the space of symmetric complex-valued bilinear forms on V ⊕ V. It suffices to show that

H : Sp_{2n}(ℝ) → B(V ⊕ V, ℂ)

is analytic. Recall that under the real basis {ie_{j}, (j = 1, …, n), e_{j}, (j = 1, …, n)}, multiplication by i can be regarded as left multiplication by J, and taking conjugation can be regarded as left multiplication by

B = ( \begin{matrix}−I & 0 \\ 0 & I \end{matrix} ) .

Therefore

2(C_{g}^{−1}z, w) = 2w̄^{t}C_{g}^{−1}z = 2w̄^{t}C_{g}^{−1}(−J)iz,

(z, Z_{g^{−1}}z) = \overline{z^{t}}\overline{Z_{g^{−1}}}^{t}z = −(iz^{t})B\overline{Z_{g^{−1}}}^{t}(iz),
(Z_{g}w, w) = \overline{w^{t}}Z_{g}w = \overline{w^{t}}Z_{g}B\overline{w}.

Since the maps g → g^{−1}, g → C_{g}^{−1}, g → Z_{g} are all real analytic, we conclude that

H : Sp_{2n}(ℝ) → B(V ⊕ V, ℂ)

is analytic. q.e.d.

Notice that the maps g → g^{−1}, g → C_{g}^{−1}, g → Z_{g} are all rational functions. Therefore by the same argument, we have

Theorem 3.3. H : Sp_{2n}(ℝ) → S is a rational function.

Now we have shown that dH_{g} is bijective and H : Sp_{2n}(ℝ) → S is analytic and one-to-one. From the classical theorem on inverse functions (see page 21 [9]), we obtain the following theorem.

Theorem 3.4. The map H : Sp_{2n}(ℝ) → S is an analytic embedding.

In a more general setting, we have

Theorem 3.5. Let G be an arbitrary Lie group with a faithful representation into Sp_{2n}(ℝ). Suppose the closure of H(G), denoted by \overline{G}, is a compact smooth submanifold of S. Then (H|_{G}, \overline{G}) is an analytic compactification of G.

4. Generalized Cartan decomposition and some remarks

Let G be a connected compact Lie group. For a subgroup H of G, let N_{G}(H), Z_{G}(H) be the normalizer and centralizer of H in G. For a Lie subalgebra 𝔥 of 𝔤, let N_{G}(𝔥) and Z_{G}(𝔥) be the normalizer and centralizer of 𝔥 in G. Suppose G is a compact connected Lie group. Let σ, τ be a pair of commuting involutions of G. Let K and H be the fixed point sets of σ and τ respectively. Let 𝔭 be the −1 eigenspace of σ, and 𝔮 the −1 eigenspace of τ. Let 𝔱_{𝔭𝔮} be the maximal Abelian subspace of 𝔭 ∩ 𝔮. Let T_{𝔭𝔮} be the analytic group of 𝔱_{𝔭𝔮}. We define the Weyl group

W_{𝔭𝔮} = N_{K}(𝔱_{𝔭𝔮})/Z_{K}(𝔱_{𝔭𝔮}) ≅ N_{H}(𝔱_{𝔭𝔮})/Z_{H}(𝔱_{𝔭𝔮}).

Theorem 4.1 (Generalized Cartan Decomposition). The group G possesses a KT_{𝔭𝔮}H decomposition. In other words,

m : K × T_{𝔭𝔮} → G/H

is surjective. In addition, for g = kth, t is unique up to the action of W_{𝔭𝔮} and a multiplication of T_{𝔭𝔮} ∩ Z_{K}(𝔱_{𝔭𝔮})Z_{H}(𝔱_{𝔭𝔮}).

This theorem is essentially due to Hoogenboom (see p. 194 in [7]). Now for G = U(2n), let

σ(x) = ( \begin{matrix}I_{n} & 0 \\ 0 & −I_{n} \end{matrix} ) x ( \begin{matrix}I_{n} & 0 \\ 0 & −I_{n} \end{matrix} ) (x ∈ U(2n)),

τ(x) = \overline{x}, (x ∈ U(2n)).

It is obvious that

τσ = στ

and K = U(n) × U(n) H = O_{2n}(ℝ),

𝔭 = \{( \begin{matrix}0 & A \\ −\overline{A^{t}} & 0 \end{matrix} ) | A ∈ gl(n, ℂ)\},

𝔮 = {iB | B^{t} = B, B ∈ gl(2n, ℝ)}.

Thus

𝔭 ∩ 𝔮 = \{( \begin{matrix}0 & iA \\ iA^{t} & 0 \end{matrix} ) | A ∈ gl(n, ℝ)\}

We may choose T_{𝔭𝔮} = 𝕋^{n} ⊆ U(2n). Then

𝔱_{𝔭𝔮} =\{𝔱_{θ} = ( \begin{matrix}0 & −diag(iθ_{1}, …, iθ_{n}) \\ −diag(iθ_{1}, …, iθ_{n}) & 0 \end{matrix} )
· | θ_{i} ∈ ℝ, i ∈ [1, n]}.

Hence W_{𝔭𝔮} is simply the Weyl group of type B_{n} Lie algebra. More precisely, W_{𝔭𝔮} acts on 𝔱_{θ} by permuting θ_{i}’s and changing the signs of θ_{i}’s. We identify 𝕋^{n} with (𝕋)^{n}. According to the generalized Cartan decomposition, we have

Theorem 4.2. The group U(2n) possesses a K𝕋^{n}H decomposition, where K is U(n) × U(n) embedded diagonally, and H is O_{2n}(ℝ). In addition, for g = kth, t = exp 𝔱_{θ} is unique up to a reordering of (𝕋)^{n} and conjugations on any factor 𝕋 in 𝕋^{n}. If we define ψ : K × 𝕋^{n} → S by

ψ(k, t) = ktk^{t} ∈ S ≅ U(2n)/H (k ∈ K, t ∈ 𝕋^{n}),

then ψ is surjective.

In particular, due to the action of Weyl group, we may assume that sin θ_{i} ≥ 0 for every i ∈ [1, n], i.e.,

θ_{i} ∈ [0, π] (i ∈ [1, n]).

We observe that the set

Im(H) = ψ(K × (H(A)) = ψ(K × {T_{θ} | θ_{i} ∈ (0, π)})

is dense in S. Combined with Theorem 3.4 we have shown

Theorem 4.3 (Compactification of Sp_{2n}(ℝ)). (H, S) is an analytic compactification of Sp_{2n}(ℝ).

For any function f ∈ C(Sp_{2n}(ℝ), let f^{0} be the push-forward of f, defined to be

f^{0}(s) = f(H^{−1}(s)), (s ∈ Im(H)),

and zero otherwise. Let (π, H) be a nontrivial irreducible unitary representation of Sp_{2n}(ℝ). Suppose now f is a U(n)-finite matrix coefficient of π. In other words, f(g) = (π(g)u, v) with u, v U(n)−finite. It is well known that f(g) vanishes at infinity (see Theorem 5.4 [3]). Let D = S −Im(H) be the boundary. Since the push forward f^{0} vanishes on the boundary D, f^{0} is continuous on D. f^{0} is also analytic on Im(H). Therefore f^{0} is continuous on S. For the trivial representation, the matrix coefficients are constant functions. We can simply extend the constant function to S.

We will compute the exact formula for the compactification of Sp_{2}(ℝ) = SL(2, ℝ).

Theorem 4.4.

H(( \begin{matrix}a & b \\ c & d \end{matrix} )) = ( \begin{matrix}\frac{(a−d)+(b+c)i}{(a+d)−(c−b)i} & \frac{−2i}{(a+d)−(c−b)i} \\ \frac{−2i}{(a+d)−(c−b)i} & \frac{(a−d)−(b+c)i}{(a+d)−(c−b)i} \end{matrix} ) ∈ S_{2}.

Proof. Let

g = ( \begin{matrix}a & b \\ c & d \end{matrix} ) , g^{−1} = ( \begin{matrix}d & −b \\ −c & a \end{matrix} ) .

Then

C_{g} = \frac{1}{2} ( \begin{matrix}a + d & b − c \\ c − b & a + d \end{matrix} ) , A_{g} = \frac{1}{2} ( \begin{matrix}a − d & b + c \\ b + c & d − a \end{matrix} ) .

an analytic compactification of symplectic group 393

Thus we obtain

det C_{g} = \frac{1}{4}(a^{2} + b^{2} + c^{2} + d^{2} + 2ad − 2bc) = \frac{1}{4}(a^{2} + b^{2} + c^{2} + d^{2} + 2).

Let ξ = a^{2} + b^{2} + c^{2} + d^{2} + 2. Then

C_{g}^{−1} = \frac{2}{ξ} ( \begin{matrix}a + d & c − b \\ b − c & a + d \end{matrix} ) ,

Z_{g} = C_{g}^{−1}A_{g} = \frac{1}{ξ} ( \begin{matrix}a^{2} − d^{2} + c^{2} − b^{2} & 2(ab + cd) \\ 2(ab + cd) & b^{2} − c^{2} + d^{2} − a^{2} \end{matrix} ) .

Recalling that the real basis of V = ℂ is i, 1, we have

2(C_{g}^{−1}z, w) = \frac{4}{ξ}((a + d) + (c − b)i)z\overline{w} = \frac{−4i}{(a + d) + (b − c)i}(iz)\overline{w},

(Z_{g}w, w) =\frac{1}{ξ}((b^{2} − c^{2} + d^{2} − a^{2}) + 2(ab + cd)i)\overline{w}\overline{w}
=\frac{(b + ai)^{2} + (d + ci)^{2}}{ξ} \overline{w}\overline{w}
=\frac{(d − a) + (c + b)i}{(d + a) − (c − b)i}\overline{w}\overline{w}.

Interchanging a ↔ d, b ↔ −b, c ↔ −c gives

(Z_{g^{−1}}z, z) = \frac{(a − d) − (c + b)i}{(a + d) − (b − c)i}\overline{z}\overline{z}.

Thus

(z, Z_{g^{−1}}z) = \overline{(Z_{g^{−1}}z, z)} = \frac{(a − d) + (b + c)i}{(a + d) + (b − c)i}zz.

From

H(g, z, w) = (iz, \overline{w})H(g) ( iz, \overline{w} )

it follows that

H(g) = ( \begin{matrix}\frac{(a−d)+(b+c)i}{(a+d)+(b−c)i} & \frac{−2i}{(a+d)+(b−c)i} \\ \frac{−2i}{(a+d)+(b−c)i} & \frac{(a−d)−(c+b)i}{(d+a)+(b−c)i} \end{matrix} ) .

It is easy to check that H(g) ∈ S. q.e.d.

5. Divisor at infinity

Now let us look at the boundary D. The first question one may ask is whether D is a subvariety of codimension 1 in S. If this is the case, is D irreducible? What is the defining function for this divisor? Let us first recall that

Im(H) = ψ(K × (H(A)) = ψ(K × {T_{θ} | θ_{i} ∈ (0, π) ∀ i ∈ [1, n]}).

Then

D = ψ(K × {T (θ) | ∃ i ∈ [1, n], θ_{i} ∈ {0, π}})

Observe that

D ∩ 𝕋^{n} = ∪_{i=1}^{n}𝕋_{±,i}^{n},

where 𝕋_{±,i}^{n} = {T (θ) | cos(θ_{i}) = ±1}. In fact, each 𝕋_{±,i}^{n} can be identified with 𝕋^{n−1}, hence is irreducible. We start with the following lemma.

Lemma 5.1. The geometric dimension of D is 2n^{2} + n − 1.

Proof. To compute the dimension of D, we compute the isotropic algebra of the action of 𝔲(n) × 𝔲(n) on T (θ) (previously denoted by ψ). Suppose (U, V) ∈ 𝔲(n) × 𝔲(n) such that

( \begin{matrix}U & 0 \\ 0 & V \end{matrix} ) ( \begin{matrix}cos θ & −i sin θ \\ −i sin θ & cos θ \end{matrix} )

+ ( \begin{matrix}cos θ & −i sin θ \\ −i sin θ & cos θ \end{matrix} ) ( \begin{matrix}U^{t} & 0 \\ 0 & V^{t} \end{matrix} ) = 0,

which implies that

( \begin{matrix}U cos θ & −iU sin θ \\ −iV sin θ & V cos θ \end{matrix} ) = ( \begin{matrix}cos θ\overline{U} & −i sin θ\overline{V} \\ −i sin θ\overline{U} & cos θ\overline{V} \end{matrix} ) .

Thus

U cos θ = cos θ\overline{U}, V cos θ = cos θ\overline{V} ,

and therefore it follows that

∀ (i, j) ∈ [1, n],  U_{i,j} cos θ_{j} = cos θ_{i}\overline{U_{i,j}}  V_{i,j} cos θ_{j} = cos θ_{i}\overline{V_{i,j}},

so that

∀ (i, j) ∈ [1, n],  \overline{U_{i,j}} = ±U_{i,j}  \overline{V_{i,j}} = ±V_{i,j}.

Combining with the former equation yields

∀ (i, j) ∈ [1, n],  U_{i,j}(cos θ_{j} ± cos θ_{i}) = 0,  V_{i,j}(cos θ_{j} ± cos θ_{i}) = 0.

We assume that

∀ (i ≠ j) ∈ [1, n], cos θ_{i} ± cos θ_{j} ≠ 0.

Then

∀ (i ≠ j) ∈ [1, n],  U_{ij} = 0,  V_{ij} = 0.

For i = j, since \overline{U_{i,i}} = −U_{i,i}, we assume that

cos θ_{i} ≠ 0.

Under these two assumptions on T (θ), we must have U = 0 and V = 0. Therefore, the isotropy algebra of T (θ) is trivial. This shows that if

cos θ_{i} ± cos θ_{j} ≠ 0, (∀ i ≠ j ∈ [1, n]),

and cos(θ_{i}) ≠ 0 for all i, then

dim ψ((U(n) × U(n)) × T (θ)) = 2n^{2}.

Since

dim(𝕋_{±,i}^{n}∩{T (θ) | ∀ i ≠ j ∈ [1, n], cos θ_{i}±cos θ_{j} ≠ 0; cos θ_{i} ≠ 0}) = n−1,

we have

dim(D) = 2n^{2} + n − 1.

q.e.d.

One may now ask the question, what does the (generic) U(n)×U(n)- orbit look like in the boundary D? Without loss of generality, we assume that cos θ_{1} = 1 and

∀ (i ≠ j) ∈ [1, n], cos θ_{i} ± cos θ_{j} ≠ 0; cos θ_{i} ≠ 0.

Suppose that X = (U, V) ∈ U(n) × U(n) such that XT (θ)X^{t} = T (θ). Then XT (θ) = T (θ)\overline{X}, and we obtain the following equations:

U cos θ = cos θ\overline{U}, V cos θ = cos θ\overline{V} ,
U sin θ = sin θ\overline{V} , V sin θ = sin θ\overline{U}.

For i ≠ j, applying the argument in the proof of the lemma, we obtain

U_{ij} = 0, V_{ij} = 0.

Since U and V are diagonal, we see that

U = diag(±1, ±1, …, ±1) = V.

Now we have proved the following theorem:

Theorem 5.1. The generic U(n) × U(n)-orbit on D is given by

U(n) × U(n)/{(U, V) | U = diag(±1, ±1, …, ±1) = V }.

From a purely algebraic point of view, all these U(n) × U(n)-orbits are closed and algebraic. Therefore they can all be defined by algebraic equations. The proof of the next theorem should give us some flavor about how we can construct the defining functions.

Theorem 5.2. There exists an algebraic function f ∈ O_{S} such that zero(f) = D. Therefore, D is a subvariety of S. In addition, D is an irreducible divisor.

Before I go ahead proving this theorem, I should say that one can give a much easier proof for the existence of f. But such a proof cannot be generalized to produce an algorithm to compute the defining functions for any closed U(n) × U(n)-subvariety of S. Therefore I choose a more general construction here.

Proof.

1. We will construct a U(n) × U(n)-invariant function f ∈ O_{S} such that

f(T (θ)) = ∏_{i=1}^{n} sin^{2} θ_{i}.

Therefore

zero(f) = ψ(K × {T (θ) | ∃ i ∈ [1, n], θ_{i} ∈ {0, π}}) = D.

Thus D is a closed subvareity.

2. We observe that

• ∏_{i=1}^{n} sin^{2} θ_{i} = ∏_{i=1}^{n} −(\frac{1}{2}(exp(iθ_{i}) − exp(−iθ_{i})))^{2}

= ∏_{i=1}^{n} \frac{1}{2}(1 − cos(2θ_{i})).

an analytic compactification of symplectic group 397

• The following two linear independent bases {1, exp(2iα) + exp(−2iα), …, exp(2kiα) + exp(−2kiα)}, {1, exp(2iα) + exp(−2iα), …, (exp(2iα) + exp(−2iα))^{k}} span the same vector space. In other words,

{1, cos(2α), …, cos(2kα)},

{1, cos(2α), …, (cos(2α))^{k}}

span the same vector space.

• By the theory of symmetric functions, the function

∏_{i=1}^{n}(1 − cos(2θ_{i}))

can be written as a function of

{s_{k} = ∑_{i=1}^{n}(cos(2θ_{i}))^{k} | k ∈ ℕ}.

• Therefore ∏_{i=1}^{n} sin^{2} θ_{i} can be expressed as a function of

{ ∑_{i=1}^{n} cos(2kθ_{i}) | k ∈ ℕ}.

3. Now let B = diag(−I_{n}, I_{n}). We look at the function ϕ_{1}(s) = T r(B\overline{s}Bs). Since for every g ∈ U(n) × U(n), we have ϕ_{1}(gsg^{t}) = T r(B\overline{g}\overline{s}\overline{g^{t}}Bgsg^{t}) = T r((g^{t}B\overline{g})\overline{s}(\overline{g^{t}}Bg)s) = T r(B\overline{s}Bs), which shows that the function ϕ_{1} is U(n) × U(n)-invariant. On the other hand we have

B\overline{T (θ)}BT (θ) = BT (−θ)BT (θ) = T (2θ),

ϕ_{1}(T (θ)) = ∑_{i=1}^{n} 2 cos(2θ_{i}).

Similarly, we may define ϕ_{k}(s) = T r((B\overline{s}Bs)^{k}).

Then ϕ_{k}(s) is U(n) × U(n)-invariant, and

ϕ_{k}(T (θ)) = ∑_{i=1}^{n} 2 cos(2kθ_{i}).

Now we can conclude that f can be expressed as a function of

{ϕ_{k}(s) | k ∈ ℕ}.

It is not difficult to see that each ϕ_{k} is U(2n)-finite. Therefore ϕ_{k} ∈ O_{S}. Thus f is algebraic and D is a closed subvareity.

4. From the lemma, we have

dim(D) = 2n^{2} + n − 1.

Hence the subvariety D is of codimension 1.

5. Now suppose that D = ∑ D_{j} with D_{j} irreducible. Since the group U(n) × U(n) acts on S algebraically, U(n) × U(n) acts on the irreducible components of D. Since U(n) × U(n) is connected, it acts on the set of irreducible components trivially. Therefore, U(n) × U(n) acts on each D_{j}. Now the intersection of D_{j} with 𝕋^{n} must be a W_{𝔭q}-stable subvariety. However D ∩ 𝕋^{n} = ∪_{i=1}^{n}𝕋_{±,i}^{n} is the only W_{𝔭q}-stable subvariety of codimension 1 of 𝕋^{n}; that is contained in D ∩ 𝕋^{n}. Therefore there is only one irreducible component of D. This implies that D is irreducible. q.e.d.

We should make a final remark. In the setting of Cartan decomposition for a compact group G, we have the map

ψ : K × A → G/K.

Even though the singular points in the maximal torus A is of codimension 1, the singular points in G/K is always of codimension less or equal to two (see Ch VII.3 in [6]). The reason that in our case D is of codimension 1 is that D is only related to the sign changes in the Weyl group W_{𝔭q}.

References

[1] V. Bargmann, On a Hilbert space of analytic functions and an associated integral transform, Comm. Pure Appl. Math. 14 (1961) 187-214.

[2] R. J. Blattner, The metalinear geometry of nonreal polarization, Lecture in Math. 570 (1977) 11-45.

[3] A. Borel & N. Wallach, Continuous cohomology, discrete subgroups, and representations of reductive groups, Ann. of Math. Stud., No. 94, Princeton Univ. Press, 1980.

[4] Hongyu. He, Rank and dual pair correspondence in semistable range, Ph.D. Thesis, M.I.T., 1998.

[5] , Functions on symmetric spaces and dual pair representations: I: Signature of genuine representations of metaplectic group, in process.

[6] S. Helgason, Differential geometry and symmetric spaces, Academic Press, New York, 1962.

[7] G. Heckman & H. Schlichtkrull, Harmonic analysis and special functions on symmetric spaces, Perspectives Math., Academic Press, New York, 1994.

[8] P. L. Robinson & J. H. Rawnsley, The metaplectic representation, Mp^{c} structures and geometric quantization, Ch. 1.2, Mem. Amer. Math., No. 410, 1989.

[9] V. S. Varadarajan, Lie groups, Lie algebras, and their representations, Springer, Berlin, 1984.

Massachusetts Institute of Technology