j. differential geometry

64 (2003) 111-170

SURFACE GROUP REPRESENTATIONS AND U(p, q)-HIGGS BUNDLES

STEVEN B. BRADLOW^{1,2}, OSCAR GARCÍA-PRADA^{1,3,5,6} & PETER B. GOTHEN^{1,4,5}

Abstract

Using the L^{2} norm of the Higgs field as a Morse function, we study the moduli spaces of U(p, q)-Higgs bundles over a Riemann surface. We require that the genus of the surface be at least two, but place no constraints on (p, q). A key step is the identification of the function’s local minima as moduli spaces of holomorphic triples. In a companion paper [7] we prove that these moduli spaces of triples are nonempty and irreducible.

Because of the relation between flat bundles and fundamental group representations, we can interpret our conclusions as results about the number of connected components in the moduli space of semisimple PU(p, q)- representations. The topological invariants of the flat bundles are used to label subspaces. These invariants are bounded by a Milnor–Wood type inequality. For each allowed value of the invariants satisfying a certain coprimality condition, we prove that the corresponding subspace is nonempty and connected. If the coprimality condition does not hold, our results apply to the closure of the moduli space of irreducible representations.

1 Members of VBAC (Vector Bundles on Algebraic Curves), which is partially supported by EAGER (EC FP5 Contract no. HPRN-CT-2000-00099) and by EDGE (EC FP5 Contract no. HPRN-CT-2000-00101).

2 Partially supported by the National Science Foundation under grant DMS-0072073.

3 Partially supported by the Ministerio de Ciencia y Tecnología (Spain) under grant BFM2000-0024.

4 Partially supported by the Fundação para a Ciência e a Tecnologia (Portugal) through the Centro de Matemática da Universidade do Porto and through grant no. SFRH/BPD/1606/2000.

5 Partially supported by the Portugal/Spain bilateral Programme Acciones Integradas, grant nos. HP2000-0015 and AI-01/24.

6 Partially supported by a British EPSRC grant (October-December 2001). Received 11/27/2002.

1. Introduction

The relation between Higgs bundles and fundamental group representations provides a vivid illustration of the interaction between geometry and topology. On the topological side we have a closed oriented surface X and the moduli space (or character variety) of representations of π_{1}X in a Lie group G. We cross over to complex geometry by fixing a complex structure on X, thereby turning it into a Riemann surface. The space of representations, or equivalently the space of flat G-bundles, then emerges as a complex analytic moduli space of G-Higgs bundles. In this guise, the moduli space carries a natural proper function whose restriction to the smooth locus is a Morse-Bott function. We can therefore use this function to determine topological properties of the moduli space of representations. Our goal in this paper is to pursue these ideas in the case where the group G is the real Lie group PU(p, q), the adjoint form of the noncompact group U(p, q).

The relevant Higgs bundles in our situation are U(p, q)-Higgs bundles. These can be seen as a special case of the G-Higgs bundles defined by Hitchin in [22], where G is a real form of a complex reductive Lie group. Such objects provide a natural generalization of holomorphic vector bundles, which correspond to the case G = U(n) and zero Higgs field. In particular, they permit an extension to other groups of the Narasimhan and Seshadri theorem ([26]) on the relation between unitary representations of π_{1}X and stable vector bundles. By embedding U(p, q) in GL(p+q) we can give a concrete description of a U(p, q)-Higgs bundle as a pair

(V ⊕ W, Φ = ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} )) (1.1)

where V and W are holomorphic vector bundles of rank p and q respectively, β is a section in H^{0}(Hom(W, V)⊗K), and γ ∈ H^{0}(Hom(V, W)⊗ K), so that Φ ∈ H^{0}(End(V ⊕ W) ⊗ K).

By the work of Hitchin [22, 23] Donaldson [12], Simpson [29, 30, 31, 32] and Corlette [10], we can define moduli spaces of polystable Higgs bundles, and these can be identified with moduli spaces of solutions to natural gauge theoretic equations. Moreover, since the gauge theory equations amount to a projective flatness condition, these moduli spaces correspond to moduli spaces of flat structures. In the case of U(p, q)-Higgs bundles, the flat structures correspond to semi-simple representations of π_{1}X into the group PU(p, q). The Higgs bundle moduli spaces can thus be used, in a way which we make precise in Sections 2 and 3, to study the representation variety

R(PU(p, q)) = Hom^{+}(π_{1}X, PU(p, q))/PU(p, q),

where Hom^{+}(π_{1}X, PU(p, q)) denotes the set of semi-simple representations of π_{1}X in PU(p, q), and the quotient is by the adjoint action.

Our main tool for studying the topology of the Higgs moduli space is the function which measures the L^{2}-norm of the Higgs field. When the moduli space is smooth, this turns out to provide a suitably nondegenerate Bott-Morse function which is, moreover, a proper map. In some cases (cf. [22, 18, 20]) the critical submanifolds are well enough understood to allow the extraction of topological information as detailed as the Poincaré polynomial. In our case our understanding is confined to the local minima of the function. This is sufficient to allow us to count the number of components of the Higgs moduli spaces, and thus of the representation varieties. A trivial but important observation is that the properness of the function allows us to draw conclusions about connected components also in the non-smooth case; we shall henceforth, somewhat imprecisely, refer to the function as the “Morse Function”, whether or not the moduli space is smooth.

The criterion we use for finding the local minima can be applied more generally, for instance if U(p, q) is replaced by any real form of a complex reductive group. This should provide an important tool for future research. In the present case, this criterion allows us to identify the subspaces of local minima as moduli spaces in their own right, namely as moduli spaces of the holomorphic triples introduced in [4]. In a companion paper [7] we develop the theory of such objects and their moduli spaces. Using the results of [7] we are able to deduce several results about the Higgs moduli spaces and also about the corresponding representation spaces.

The relation between Higgs bundles and surface group representations has been successfully exploited by others, going back originally to the work of Hitchin and Simpson on complex reductive groups. The use of Higgs bundle methods to study R(G) for real G was pioneered by Hitchin in [23], and further developed in [18, 19]. It has also been used by Xia and Xia-Markman (in [34, 35, 36, 24]) to study various special cases of G = PU(p, q). None of these, though, address the general case of PU(p, q), as we do in this paper.

We now give a brief summary of the contents and main results of this paper.

In Sections 2 and 3 we give some background and describe the basic objects of our study. In Section 2 we describe the natural invariants associated with representations of π_{1}X into PU(p, q). We also discuss the invariants associated with representations of Γ, the universal central extensions of π_{1}X, into U(p, q). The space of such representations is denoted by R_{Γ}(U(p, q)). In both cases, these involve a pair of integers (a, b) which can be interpreted respectively as degrees of rank p and rank q vector bundles over X. In the case of the PU(p, q) representations, the pair is well-defined only as a class in a quotient ℤ ⊕ ℤ/(p, q)ℤ. This leads us to define subspaces R[a, b] ⊂ R(PU(p, q)) and R_{Γ}(a, b) ⊂ R_{Γ}(U(p, q)). For fixed (a, b), the space R_{Γ}(a, b) fibers over R[a, b] with connected fibers.

In Section 3 we define U(p, q)-Higgs bundles and their moduli spaces and establish their essential properties. Thinking of a U(p, q)-Higgs bundle as a pair (V ⊕ W, Φ), the parameters (a, b) appear here as the degrees of the bundles V and W. The moduli space of polystable U(p, q)- Higgs bundles with deg(V) = a and deg W = b, which we denote by M(a, b), is the space that can be identified with the component R_{Γ}(a, b) of R_{Γ}(U(p, q)). This, together with the fibration over R_{Γ}(U(p, q)) are the crucial links between the Higgs moduli and the surface group representation varieties.

Fixing p, q, a and b, we begin the Morse theoretic analysis of M(a, b) in Section 4. The basic results we need (cf. Proposition 4.3) are that the L^{2}-norm of the Higgs field has a minimum on each connected component of M(a, b), and hence if the subspace of local minima is connected then so is M(a, b). We identify the local minima, the loci of which we denote by N (a, b), and prove (cf. Theorem 4.6 and Proposition 4.8) that these correspond precisely to holomorphic triples in the sense of [4]. A full treatment of holomorphic triples and their moduli spaces is given in [7]. We summarize the salient features of these moduli spaces in Section 5.

In Section 6 we knit together all the strands. Using the properties of the moduli spaces of triples, we establish the key (for our purposes) topological properties of the strata N (a, b). These lead directly to our main results for the moduli spaces M(a, b). Some of the results depend on (a, b) only in the combination

τ = τ(a, b) = 2\frac{aq − bp}{p + q},

known as the Toledo invariant. Indeed, (a, b) is constrained by the bounds 0 ⩽ |τ| ⩽ τ_{M}, where τ_{M} = 2 min{p, q}(g − 1). Originally proved by Domic and Toledo in [11], these bounds emerge naturally from our point of view (cf. Corollary 3.27 and Remark 5.7). Bounds on invariants of this type, for representations of finitely generated groups in U(p, q), have also recently been studied using techniques from ergodic theory (see [9]). Summarizing our main results, we prove:

Theorem A (Theorems 6.1 and 6.5). Fix positive integers (p,q). Take (a, b) ∈ ℤ⊕ℤ and let τ(a, b) be the Toledo invariant. Let M^{s}(a, b) ⊆ M(a, b) denote the moduli space of strictly stable U(p, q)-Higgs bundles.

(1) M(a, b) is nonempty if and only if 0 ⩽ |τ(a, b)| ⩽ τ_{M}. If τ(a, b) = 0, or |τ(a, b)| = τ_{M} and p ≠ q then M^{s}(a, b) is empty; otherwise it is nonempty whenever M(a, b) is nonempty.

(2) If |τ(a, b)| = 0 or |τ(a, b)| = τ_{M} and p ≠ q then M(a, b) is connected.

(3) Whenever nonempty, the moduli space M^{s}(a, b) is a smooth manifold of the expected dimension (i.e., 1 + (p + q)^{2}(g − 1)), with connected closure \overline{M}^{s}(a, b) ⊆ M(a, b). In these cases, if M(a, b) has more than one connected component, then GCD(p + q, a + b) ≠ 1 and, if p = q, 0 < |τ| ⩽ (p − 1)(2g − 2).

Theorem B (Theorem 3.32). Suppose that p ≠ q and (a, b) ∈ ℤ⊕ℤ are such that |τ(a, b)| = τ_{M}. To be specific, suppose that p < q and τ(a, b) = p(2g − 2). Then every element in M(a, b) decomposes as the direct sum of a polystable U(p, p)-Higgs bundle with maximal Toledo invariant and a polystable vector bundle of rank q − p. Thus

(1.2) M(p, q, a, b) ≅ M(p, p, a, a − p(2g − 2)) × M(q − p, b − a + p(2g − 2)).

In particular, the smooth locus in M(p, q, a, b) has dimension 2 + (q^{2} + 5p^{2} − 2pq)(g − 1). This is strictly smaller than the expected dimension if g ⩾ 2.

(A similar result holds if p > q, as well as if τ = −p(2g − 2).)

Since we identify M(a, b) = R_{Γ}(a, b), we can translate these results directly into statements about R_{Γ}(a, b) (given in Theorems 6.6 and 6.7).^{7} The subspace in R_{Γ}(a, b) which corresponds to M^{s}(a, b) ⊆ M(a, b) is denoted by R_{Γ}^{∗}(a, b). The representations it labels include all the simple representations. Defining R_{Γ}^{∗}(U(p, q)) ⊂ R_{Γ}(U(p, q)) to be the union over all (a, b) of the components R_{Γ}^{∗}(a, b) we thus obtain:

7Note added in proof: after this paper was submitted, M. Burger, A. Iozzi and A. Wienhard published the note Surface group representations with maximal Toledo invariant, Comptes Rendus 336 (2003) 387–390, in which they prove a result similar to our Theorem 6.7.

Theorem C (Corollary 6.16). The moduli space R_{Γ}^{∗}(U(p, q)) has

2(p + q) min{p, q}(g − 1) + GCD(p, q)

connected components.

Since R_{Γ}(a, b) fibers over R[a, b] with connected fibers, we can apply our results to the latter. The results are given in Theorems 6.10 and 6.11.

The above results fall just short of saying that the full moduli spaces M(a, b) (= R(a, b)) and R[a, b] are connected for all allowed choices of (a, b). They show however that if any one is not connected then it has one (nonempty) connected component which contains all the irreducible objects. Any other components must thus consist entirely of reducible (or strictly semisimple) elements. Theorem B and its analogs for R_{Γ}(a, b) and R[a, b] generalize rigidity results of Toledo [33] (when p = 1) and Hernández [21] (when p = 2).

This paper, together with its companion [7] form a substantially revised version of the preprint [6]. The main results proved in this paper were announced in the note [5]. In that note we claim (without proof) that the connectedness results for the moduli spaces R(a, b) and R[a, b] hold without the above qualifications. This is a reasonable conjecture, which we hope to come back to in a future publication.

We note, finally, that our methods surely apply more widely than to U(p, q)-Higgs bundles and PU(p, q) representations (see, for example, Remark 4.16). Moreover, careful scrutiny of the Lie algebra properties used in the proofs suggests certain aspects can be generalized to representations in any real group G for which G/H is hermitian symmetric, where H ⊂ G is a maximal compact subgroup. This will be addressed in a future publication.

Acknowledgements. We thank the mathematics departments of the University of Illinois at Urbana-Champaign, the University Autónoma of Madrid and the University of Aarhus, the Department of Pure Mathematics of the University of Porto, the Mathematical Sciences Research Institute of Berkeley and the Mathematical Institute of the University of Oxford, and the Erwin Schrödinger International Institute for Mathematical Physics in Vienna for their hospitality during various stages of this research. We thank Fran Burstall, Bill Goldman, Nigel Hitchin, Eyal Markman, S. Ramanan, Domingo Toledo, and Eugene Xia, for many insights and patient explanations.

2. Representations of surface groups

In this section we record some general facts about representations of a surface group in U(p, q) or PU(p, q) and set up our notation. A very useful reference for the general theory is Goldman’s paper [16].

2.1 Moduli spaces of representations

Let X be a closed oriented surface of genus g ⩾ 2. By definition U(p, q) is the subgroup of GL(n, ℂ) (with n = p + q) which leaves invariant a hermitian form of signature (p, q). It is a noncompact real form of GL(n, ℂ) with center U(1) and maximal compact subgroup U(p)×U(q). The quotient U(p, q)/(U(p)×U(q)) is a hermitian symmetric space. The adjoint form PU(p, q) is given by the exact sequence of groups

1 ⟶ U(1) ⟶ U(p, q) ⟶ PU(p, q) ⟶ 1,

and we have a standard inclusion PU(p, q) ⊂ PGL(n, ℂ).

Definition 2.1. By a representation of π_{1}X in PU(p, q) we mean a homomorphism ρ: π_{1}X → PU(p, q). We say that a representation of π_{1}X in PU(p, q) is semi-simple if the induced (adjoint) representation on the Lie algebra of PU(p, q) is semi-simple. The group PU(p, q) acts on the set of representations via conjugation. Restricting to the semisimple representations, we get the moduli space of representations,

(2.1) R(PU(p, q)) = Hom^{+}(π_{1}X, PU(p, q))/PU(p, q).

The moduli space of representations can be described more concretely as follows. From the standard presentation

π_{1}X = ⟨A_{1}, B_{1}, …, A_{g}, B_{g} | ∏_{i=1}^{g}[A_{i}, B_{i}] = 1⟩

we see that Hom^{+}(π_{1}X, PU(p, q)) can be embedded in PU(p, q)^{2g} via

Hom^{+}(π_{1}X, PU(p, q)) → PU(p, q)^{2g}
ρ ↦ (ρ(A_{1}), … ρ(B_{g})).

We give Hom^{+}(π_{1}X, PU(p, q)) the subspace topology and R(PU(p, q)) the quotient topology. This topology is Hausdorff because we have restricted attention to semi-simple representations.

Clearly any representation of π_{1}X in U(p, q) gives rise to a representation in PU(p, q); however, not all representations in PU(p, q) lift to U(p, q). We are thus motivated to consider representations of the central extension

0 ⟶ ℤ ⟶ Γ ⟶ π_{1}X ⟶ 1. (2.2)

Such extensions are defined (as in [1]) by the generators A_{1}, B_{1}, …, A_{g}, B_{g} and a central element J subject to the relation ∏_{i=1}^{g}[A_{i}, B_{i}] = J. With Γ thus defined, any representation of π_{1}X in PU(p, q) can be lifted to a representation of Γ in U(p, q).

In analogy with Definition 2.1 we make the following definition.

Definition 2.2. We define the moduli space of semi-simple representations of Γ in U(p, q) by

(2.3) R_{Γ}(U(p, q)) = Hom^{+}(Γ, U(p, q))/U(p, q),

where semi-simplicity is defined with respect to the induced adjoint representation. This space is topologized in the same way as R(PU(p, q)).

2.2 Invariants

Our basic objective is to study the number of connected components of the spaces R(PU(p, q)) and R_{Γ}(U(p, q)). The first step in the study of topological properties of these spaces is to identify the appropriate topological invariant of a representation ρ: π_{1}X → G. For a general connected Lie group G the relevant invariant is an obstruction class in H^{2}(X, π_{1}G) ≅ π_{1}G (see Goldman [16, 17]). In the following we give an explicit description of this invariant in our case, using characteristic classes of the flat bundles associated to representations of the fundamental group. In fact we shall not need the more general description of the invariant.

We begin by considering the case G = U(p, q). By the same argument as in [1]^{8}, R_{Γ}(U(p, q)) can be identified with the moduli space of connections with central curvature on a fixed U(p, q)-bundle on X.

8While [1] gives the argument for U(n) and PU(n), there are no essential changes to be made in order to adapt for the case of U(p, q) and PU(p, q).

Taking a reduction to the maximal compact U(p) × U(q), we thus associate to each class ρ̃ ∈ R_{Γ}(U(p, q)) a vector bundle of the form V ⊕ W, where V and W are rank p and q respectively, and thus a pair of integers (a, b) = (deg(V), deg(W)). There is thus a map

c̃: R_{Γ}(U(p, q)) → ℤ ⊕ ℤ

given by c̃(ρ̃) = (a, b). The corresponding map on Hom^{+}(Γ, U(p, q)) is clearly continuous and thus locally constant. Since U(p, q) is connected, the map c̃ is likewise continuous and thus constant on connected components. We make the following definition.

Definition 2.3. The subspace of R_{Γ}(U(p, q)) corresponding to representations with invariants (a, b) is denoted by

R_{Γ}(a, b) = c^{−1}(a, b)
= {ρ̃ ∈ R_{Γ}(U(p, q)) | c̃(ρ̃) = (a, b) ∈ ℤ ⊕ ℤ}.

Note that R_{Γ}(a, b) is a union of connected components, because c̃ is constant on each connected component.

Next we consider the case G = PU(p, q). Any flat PU(p, q)-bundle lifts to a U(p, q)-bundle with a connection with constant central curvature. This lift is, however, not uniquely determined: in fact two such U(p, q)-bundles give rise to the same flat PU(p, q)-bundle if and only if one can be obtained from the other by twisting with a line bundle L with a unitary connection of constant curvature. If the invariant of the U(p, q)-bundle is (a, b) and the degree of L is l, then the invariant associated to the twisted bundle is (a+pl, b+ql). There is thus a well-defined map

c: R(PU(p, q)) ⟶ (ℤ ⊕ ℤ)/(p, q)ℤ, (2.4)

where (ℤ ⊕ ℤ)/(p, q)ℤ denotes the quotient of ℤ ⊕ ℤ by the ℤ-action l·(a, b) = (a+pl, b+ql). Notice that (ℤ⊕ℤ)/(p, q)ℤ can be identified with π_{1}(PU(p, q)). The invariant defined by c is the same as the obstruction class defined by Goldman [16, 17].

Definition 2.4. Denote the image of (a, b) in (ℤ ⊕ ℤ)/(p, q)ℤ by [a, b]. The subspace of R(PU(p, q)) corresponding to representations with invariant [a, b] is denoted by

R[a, b] = c^{−1}[a, b]
= {ρ ∈ R(PU(p, q)) | c(ρ) = [a, b] ∈ (ℤ ⊕ ℤ)/(p, q)ℤ}.

The space R[a, b] is a union of connected components in the same way as R_{Γ}(a, b). In order to compare the spaces R_{Γ}(a, b) and R[a, b] notice that we have surjective maps

R_{Γ}(a, b) → R[a, b]. (2.5)

Moreover, the preimage

π^{−1}(R[a, b]) = ⋃_{(a,b)} R_{Γ}(a, b) (2.6)

where the union is over all (a, b) in the class [a, b] ∈ (ℤ ⊕ ℤ)/(p, q)ℤ. As mentioned above, tensoring by line bundles of degree l with constant curvature connections gives an isomorphism

R_{Γ}(a, b) ⟶^{≅} R_{Γ}(a + pl, b + ql).

Notice that if c(ρ) = [a, −a] for a representation ρ ∈ R(PU(p, q)), then the associated U(p, q)-bundle can be taken to have degree zero and the projectively flat connection is actually flat. Then ρ defines a representation of π_{1}X in U(p, q). Under the correspondence between R(PU(p, q)) and R_{Γ}(U(p, q)), ρ corresponds to a Γ representation in which the central element J acts trivially. Furthermore, the subspaces R_{Γ}(a, −a) ⊂ R_{Γ}(U(p, q)) can be identified with components of R(U(p, q)) (the moduli space for representations of π_{1}X in U(p, q)). Indeed, defining

R(a) = R_{Γ}(a, −a), (2.7)

we see that R(U(p, q)) is a union over a ∈ ℤ of the subspaces R(a).

Finally, we observe that the moduli space of flat degree zero line bundles acts by tensor product of bundles on R_{Γ}(a, b). Since this moduli space is isomorphic to the torus U(1)^{2g}, we get the following relation between connected components.

Proposition 2.5. The map R_{Γ}(a, b) → R[a, b] given in (2.5) defines a U(1)^{2g}-fibration. Thus the subspace R[a, b] ⊆ R(PU(p, q)) is connected if R_{Γ}(a, b) is connected.

3. Higgs bundles and flat connections

We study the moduli spaces of representations by choosing a complex structure on X. This allows us to identify these spaces with certain moduli spaces of Higgs bundles. In this section we explain this correspondence and recall some general facts about Higgs bundles. Following this, we describe the special class of Higgs bundles relevant for the study of representations in PU(p, q) and U(p, q) and derive some basic results about these moduli spaces.

3.1 GL(n, ℂ)-Higgs bundles

Give X the structure of a Riemann surface. We recall (from [10, 12, 22, 29, 31, 32]) the following definition and basic facts about GL(n, ℂ)-Higgs bundles.

Definition 3.1.

(1) A GL(n, ℂ)-Higgs bundle on X is a pair (E, Φ), where E is a rank n holomorphic vector bundle over X and Φ ∈ H^{0}(End(E)⊗K) is a holomorphic endomorphism of E twisted by the canonical bundle K of X.

(2) The GL(n, ℂ)-Higgs bundle (E, Φ) is stable if the slope stability condition

μ(E′) < μ(E) (3.1)

holds for all proper Φ-invariant subbundles E′ of E. Here the slope is defined by μ(E) = deg(E)/ rk(E) and Φ-invariance means that Φ(E′) ⊂ E′ ⊗ K. Semistability is defined by replacing the above strict inequality with a weak inequality. A Higgs bundle is called polystable if it is the direct sum of stable Higgs bundles with the same slope.

(3) Given a hermitian metric on E, let A denote the unique unitary connection compatible with the holomorphic structure, and let F_{A} be its curvature. Hitchin’s equations on (E, Φ) are

(3.2)  F_{A} + [Φ, Φ^{∗}] = −\sqrt{−1}μId_{E}ω,
\overline{∂}_{A}Φ = 0,

where μ is a constant, Id_{E} is the identity on E, \overline{∂}_{A} is the antiholomorphic part of the covariant derivative d_{A} and ω is the Kähler form on X. If we normalize ω so that ∫_{X} ω = 2π then, taking the trace and integrating over X in the first equation, one sees that μ = μ(E). A solution to Hitchin’s equations is irreducible if there is no proper subbundle of E preserved by A and Φ.

Theorem 3.2.

(1) Let (E, Φ) be a GL(n, ℂ)-Higgs bundle. Then (E, Φ) is polystable if and only if it admits a hermitian metric such that Hitchin’s equations (3.2) are satisfied. Moreover, (E, Φ) is stable if and only if the corresponding solution is irreducible.

(2) Fix a hermitian metric in a smooth rank n complex vector bundle on X, then there is a gauge theoretic moduli space of pairs (A, Φ), consisting of a unitary connection A and an endomorphism valued (1, 0)-form Φ, which are solutions to Hitchin’s equations (3.2), modulo U(n)-gauge equivalence.

(3) The moduli space of rank n degree d polystable Higgs bundles is a quasi-projective variety of complex dimension 2(1 + n^{2}(g − 1)). There is a map from the gauge theoretic moduli space to this moduli space given by taking a solution (A, Φ) to Hitchin’s equations to the Higgs bundle (E, Φ), where the holomorphic structure on E is given by \overline{∂}_{A}. This map is a homeomorphism, and a diffeomorphism on the smooth locus.

(4) If we define a Higgs connection (as in [31]) by

D = d_{A} + θ (3.3)

where θ = Φ + Φ^{∗}, then Hitchin’s equations are equivalent to the conditions

(3.4)  F_{D} = − \sqrt{−1}μId_{E}ω,
d_{A}θ = 0,
d_{A}^{∗}θ = 0.

In particular, D is a projectively flat connection. If deg(E) = 0 then D is actually flat. It follows that in this case the pair (E, D) defines a representation of π_{1}X in GL(n, ℂ). If deg(E) ≠ 0, then the pair (E, D) defines a representation of π_{1}X in PGL(n, ℂ), or equivalently, a representation of Γ in GL(n, ℂ). By the theorem of Corlette ([10]), every semisimple representation of Γ (and therefore every semisimple representation of π_{1}X) arises in this way.

(5) This correspondence gives rise to a homeomorphism between the moduli space of polystable Higgs bundles of rank n and the moduli space of semisimple representations of Γ in GL(n, ℂ). If the degree of the Higgs bundle is zero, then the moduli space is homeomorphic to the moduli space of representations of π_{1}X in GL(n, ℂ).

3.2 U(p, q)-Higgs bundles

If we fix integers p and q such that n = p + q, then we can isolate a special class of GL(n, ℂ)-Higgs bundles by the requirements that

(3.5)  E = V ⊕ W
Φ = ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} )

where V and W are holomorphic vector bundles of rank p and q respectively and the nonzero components in the Higgs field are

β ∈ H^{0}(Hom(W, V) ⊗ K), and γ ∈ H^{0}(Hom(V, W) ⊗ K).

The form of the Higgs field is determined by the Lie theory of the symmetric space U(p, q)/(U(p) × U(q)). Recall that for any real form G of a complex reductive group G^{ℂ}, with maximal compact subgroup H, there is an Ad-invariant decomposition

𝔤 = 𝔥 + 𝔪

where 𝔤=Lie(G), 𝔥=Lie(H) is the +1 eigenspace of the Cartan involution and 𝔪 is the −1 eigenspace. This induces a decomposition

𝔤^{ℂ} = 𝔥^{ℂ} + 𝔪^{ℂ} (3.6)

of 𝔤^{ℂ}=Lie(G^{ℂ}). In the case of G = U(p, q), where H = U(p) × U(q) and thus 𝔥^{ℂ} = 𝔤𝔩(p, ℂ) ⊕ 𝔤𝔩(q, ℂ), the decomposition (3.6) becomes

𝔤𝔩(n, ℂ) = (𝔤𝔩(p, ℂ) ⊕ 𝔤𝔩(q, ℂ)) + 𝔪^{ℂ}. (3.7)

If we identify 𝔤𝔩(p, ℂ) ⊕ 𝔤𝔩(q, ℂ) with the block diagonal elements in 𝔤𝔩(n, ℂ), then 𝔪^{ℂ} corresponds to the off diagonal matrices.

We can now describe the above Higgs bundles more intrinsically as follows. Let P_{GL(p,ℂ)} and P_{GL(q,ℂ)} be the principal frame bundles for V and W respectively. Let P = P_{GL(p,ℂ)} × P_{GL(q,ℂ)} be the fiber product, and let Ad P = P ×_{Ad} 𝔤𝔩(n, ℂ) be the adjoint bundle, where GL(p, ℂ) × GL(q, ℂ) ⊂ GL(n, ℂ) acts by the adjoint action on the Lie algebra of GL(n, ℂ). This defines a subbundle

P_{𝔪^{ℂ}} = P ×_{Ad} 𝔪^{ℂ} ⊂ Ad P. (3.8)

We can then make the following definition.

Definition 3.3. A U(p, q)-Higgs bundle^{9} on X is a pair (P, Φ) where P is a holomorphic principal GL(p, ℂ) × GL(q, ℂ) bundle, and Φ is a holomorphic section of the vector bundle P_{𝔪^{ℂ}} ⊗ K (where P_{𝔪^{ℂ}} is the bundle defined in (3.8)).

Remark 3.4. We can always write P = P_{GL(p,ℂ)} × P_{GL(q,ℂ)}. If we let V and W be the standard vector bundles associated to P_{GL(p,ℂ)} and P_{GL(q,ℂ)} respectively, then any Φ ∈ H^{0}(P_{𝔪^{ℂ}} ⊗ K) can be written as in (3.5). We will usually adopt the vector bundle description of U(p, q)- Higgs bundles.

Remark 3.5. Definition 3.3 is compatible with the definitions in [23] and [18], where G-Higgs bundles are defined for any real form G of a complex reductive Lie group G^{ℂ}. There, using the above notation, a G-Higgs bundle is a pair (P, Φ), where P is a principal H^{ℂ}-bundle and Φ is a holomorphic section of (P ×_{Ad} 𝔪^{ℂ}) ⊗ K. From a different perspective, Definition 3.3 defines an example of a principal pair in the sense of [2] and [25]. Strictly speaking, since the canonical bundle K plays the role of a fixed ‘twisting bundle’, what we get is a principal pair in the sense of [8]. The defining data for the pair are then the principal GL(p, ℂ)×GL(q, ℂ)×GL(1)-bundle P_{GL(p,ℂ)}×P_{GL(q,ℂ)}×P_{K} (where P_{K} is the frame bundle for K), and the associated vector bundle P_{𝔪^{ℂ}} ⊗ K.

Lemma 3.6. Let (E = V ⊕ W, Φ) be a U(p, q)-Higgs bundle with a hermitian metric such that V ⊕W is a unitary orthogonal decomposition. Let A be a unitary connection and let D = d_{A} + θ be the corresponding Higgs connection, where θ = Φ + Φ^{∗}. Then D is a U(p, q)-connection, i.e., in any unitary local frame the connection 1-form takes its values in the Lie algebra of U(p, q).

Proof. Fix a local unitary frame. Then D = d + A + θ, where A takes its values in 𝔲(p) ⊕ 𝔲(q) ⊂ 𝔲(p, q), while θ takes its values in 𝔪, where

𝔲(p, q) = 𝔲(p) ⊕ 𝔲(q) + 𝔪

is the eigenspace decomposition of the Cartan involution. q.e.d.

Definition 3.7. Let (E, Φ) be a U(p, q)-Higgs bundle with E = V ⊕ W and Φ = ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} ). We say (E, Φ) is a stable U(p, q)-Higgs bundle if the slope stability condition μ(E′) < μ(E), is satisfied for all Φinvariant subbundles of the form E′ = V′ ⊕ W′, i.e., for all subbundles V′ ⊂ V and W′ ⊂ W such that

9The reason for the name is explained by Remark 3.5 and Lemma 3.6

(3.9)  β : W′ ⟶ V′ ⊗ K
(3.10)  γ : V′ ⟶ W′ ⊗ K.

Semistability for U(p, q)-Higgs bundles is defined by replacing the above strict inequality with a weak inequality, and (E, Φ) is polystable if it is a direct sum of stable U(p, q)-Higgs bundles all of the same slope. We shall say that a polystable U(p, q)-Higgs bundle which is not stable is reducible. A morphism between two U(p, q)-Higgs bundles (V ⊕ W, Φ) and (V′⊕W′, Φ′) is given by maps g_{V} : V → V′ and g_{W} : W → W′ which intertwine Φ and Φ′, i.e., such that (g_{V} ⊕ g_{W}) ⊗ I_{K} ∘ Φ = Φ′ ∘ (g_{V} ⊕ g_{W}) where I_{K} is the identity on K. In particular we have a natural notion of isomorphism of U(p, q)-Higgs bundles.

Remark 3.8. The stability condition for a U(p, q)-Higgs bundle is a priori weaker than the stability condition given in Definition 3.1 for GL(n, ℂ)-Higgs bundles. However, it is shown in [19, Section 2.3] that the weaker condition is in fact equivalent to the ordinary stability of (E, Φ).

Proposition 3.9. Let (E, Φ) be a U(p, q)-Higgs bundle with E = V ⊕W and Φ = ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} ). Then (E, Φ) is polystable if and only if it admits a hermitian metric such that E = V ⊕W is an orthogonal decomposition and such that Hitchin’s equations (3.2) are satisfied.

Proof. This is a special case of the correspondence invoked in [23] for G-Higgs bundles where G is a real form of a reductive Lie group. By Remark 3.5 it can also be seen as a special case of the Hitchin– Kobayashi correspondence for principal pairs (cf. [2] and [25] and [8]). We note finally that in one direction the result follows immediately from Theorem 3.2 (1): if (V ⊕ W, Φ) supports a compatible metric such that (3.2) is satisfied, then it is polystable as a GL(n, ℂ)-Higgs bundle, and hence it is U(p, q)-polystable. q.e.d.

Definition 3.10. We define M(a, b) to be the moduli space of polystable U(p, q)-Higgs bundles with deg(V) = a and deg W = b. We denote by M^{s}(a, b) the subspace parameterizing the strictly stable U(p, q)-Higgs bundles.

The construction of M(a, b) is essentially the same as in Section 9 of [32]. There the moduli space of G-Higgs bundles is constructed for any reductive group G. We take G = GL(p, ℂ) × GL(q, ℂ). The difference between a U(p, q)-Higgs bundle and a GL(p, ℂ) × GL(q, ℂ)-Higgs bundle is entirely in the nature of the Higgs fields. Taking the standard embedding of GL(p, ℂ) × GL(q, ℂ) in GL(p + q, ℂ) we see that in a GL(p, ℂ) × GL(q, ℂ)-Higgs bundle the Higgs field Φ takes its values in the subspace (𝔤𝔩(p) ⊕ 𝔤𝔩(q)) ⊂ 𝔤𝔩(p + q), while in a U(p, q)-Higgs bundle the Higgs field Φ takes its values in the complementary subspace 𝔪^{ℂ} (as in (3.7)). Since both subspaces are invariant under the adjoint action of GL(p, ℂ) × GL(q, ℂ), the same method of construction works for the moduli spaces of both types of Higgs bundle.

We can describe the gauge theory version of the moduli space M(a, b) using standard methods; see Hitchin [22] for a construction in the case of ordinary rank 2 Higgs bundles. To adapt to our case we proceed as follows. Let E = V ⊕ W be a smooth complex vector bundle with a hermitian metric such that the direct sum decomposition is orthogonal. We let A denote the space of connections on E which are direct sums of unitary connections on V and W and we let Ω denote the space of Higgs fields Φ ∈ Ω^{1,0}(End(E)) of the form Φ = ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} ). The correspondence between unitary connections and holomorphic structures via \overline{∂}-operators turns A × Ω into a complex affine space which acquires a hermitian metric using the metric on E and integration over X. The group G of U(p) × U(q)-gauge transformations acts on the configuration space C ⊆ A × Ω of solutions (A, Φ) to Hitchin’s equations (3.2). The quotient C/G is, by definition, the gauge theory moduli space. As in [22], the open subset of C/G corresponding to irreducible solutions has a Kähler manifold structure.

To see that the gauge theory moduli space is homeomorphic to M(a, b) we can consider this latter space from the complex analytic point of view (cf. Remark 3.23 below): consider triples (\overline{∂}_{V} , \overline{∂}_{W} , Φ), where \overline{∂}_{V} and \overline{∂}_{W} are \overline{∂}-operators on V and W, respectively, and Φ ∈ Ω. Let C_{ℂ} be the set of such triples for which Φ is holomorphic and the associated U(p, q)-Higgs bundle is polystable. We can then view M(a, b) as the quotient of C_{ℂ} by the complex gauge group. We clearly have an inclusion C ↪ C_{ℂ} which descends to give a continuous map from the gauge theory moduli space to M(a, b). The Hitchin-Kobayashi correspondence of Proposition 3.9 now shows that this map is in fact a homeomorphism.

For a third perspective, we observe that provided that V and W are not isomorphic bundles, i.e., provided p ≠ q or a ≠ b, we can view M^{s}(a, b) as a subvariety of a moduli space of stable GL(p + q)-Higgs bundle. If V ≃ W, then M^{s}(a, b) is a finite cover of a subvariety in the larger moduli space:

Proposition 3.11. With n = p+q and d = a+b, let M^{s}(d) denote the moduli space of stable GL(n, ℂ)-Higgs bundles of degree d. If p ≠ q or a ≠ b then M^{s}(a, b) embeds as a closed subvariety in M^{s}(d). If p = q and a = b, then there is an involution on M^{s}(a, a) such that the quotient injects into M^{s}(d).

Proof. Let [V ⊕W, Φ]_{p,q} denote the point in M^{s}(a, b) represented by the U(p, q)-Higgs bundle (V ⊕ W, Φ). Then (E = V ⊕ W, Φ) is a stable GL(n, ℂ)-Higgs bundle and the map M^{s}(a, b) → M(d) is defined by

[V ⊕ W, Φ]_{p,q} ↦ [E, Φ]_{n},

where [,]_{n} denotes the isomorphism class in M(d). The only question is whether this map is injective. Suppose that (E = V ⊕ W, Φ) and (E′ = V′ ⊕ W′, Φ′) are isomorphic as GL(n, ℂ)-Higgs bundles. Let the isomorphism be given by a complex gauge transformation g : E → E′. If g is not of the form ( \begin{matrix}g_{V} & 0 \\ 0 & g_{W} \end{matrix} ) then the off diagonal components determine morphisms ξ : V → W′ and σ : W → V′. Let N = ker(ξ) ⊕ ker(σ) be the subbundle of V ⊕ W determined by the kernels of ξ and σ. If p ≠ q then N is a nontrivial proper subbundle. Moreover, using the fact that gΦ = Φ′g, we see that it is Φ-invariant. Since (V ⊕ W, Φ) is stable, it follows that

μ(N) < μ(E). (3.11)

Similarly, the images of ξ and σ determine a proper Φ′-invariant subbundle of E′, say I, for which

μ(I) < μ(E′). (3.12)

But if μ(E) = μ(E′) then (3.11) and (3.12) cannot both be satisfied. Thus ξ and σ must both vanish and hence [V ⊕W, Φ]_{p,q} = [V′⊕W′, Φ′]_{p,q}.

If p = q, then this argument can fail, but only if ξ and σ are both isomorphisms. In that case, N = 0 and I = E. This also requires a = b. Under these conditions, if V and W are non-isomorphic, then [V ⊕W, ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} )]_{n} = [W ⊕V, ( \begin{matrix}0 & γ \\ β & 0 \end{matrix} )]_{n} but the Higgs bundles are not isomorphic as U(p, q)-Higgs bundles. Hence the last statement of the Proposition follows taking the involution [V ⊕ W, ( \begin{matrix}0 & β \\ γ & 0 \end{matrix} )] ↦ [W ⊕ V, ( \begin{matrix}0 & γ \\ β & 0 \end{matrix} )] on M^{s}(a, a). q.e.d.

Proposition 3.12. If GCD(p + q, a + b) = 1 then M^{s}(a, b) = M(a, b).

Proof. If GCD(p + q, a + b) = 1 then for purely numerical reasons there are no strictly semistable U(p, q)-Higgs bundles in M(a, b). q.e.d.

The link to moduli spaces of representations is provided by the next result.

Proposition 3.13. There is a homeomorphism M(a, b) ≅ R_{Γ}(a, b).

Proof. Suppose that (E = V ⊕ W, Φ) represents a point in M(a, b), i.e., suppose that it is a U(p, q)-polystable Higgs bundle, and suppose that E has a hermitian metric such that the direct sum decomposition is orthogonal and Hitchin’s equations (3.2) are satisfied. Rewriting the equations in terms of the Higgs connection D = d_{A} + θ, where A is the metric connection and θ = Φ+Φ^{∗}, we see that D is projectively flat. By Lemma 3.6 it is a projectively flat U(p, q)-connection, and thus defines a point in R_{Γ}(a, b). Conversely by Corlette’s theorem [10], every representation in Hom^{+}(π_{1}X, PU(p, q)), or equivalently every representation in Hom^{+}(Γ, U(p, q)), arises in this way. The fact that this correspondence gives a homeomorphism follows by the same argument as the one given in [32] for ordinary Higgs bundles. q.e.d.

Definition 3.14. Define the subspace R_{Γ}^{∗}(a, b) to be the subspace corresponding to M^{s}(a, b) via the homeomorphism in Proposition 3.13. Using the fibration of R_{Γ}(a, b) over R[a, b], define R^{∗}[a, b] ⊂ R[a, b] to be the image of R_{Γ}^{∗}(a, b).

Remark 3.15. Thus R_{Γ}^{∗}(a, b) parameterizes the representations which give rise to stable U(p, q)-Higgs bundles. Recall from Remark 3.8 that a U(p, q)-Higgs bundle is stable (in the sense of Definition 3.7) if and only if its is stable as an ordinary GL(n, ℂ)-Higgs bundle. Now, a GL(n, ℂ)-Higgs bundle is stable if and only if the corresponding representation of Γ on ℂ^{n} is irreducible (cf. Corlette [10]). Hence we see that the subspace R_{Γ}^{∗}(a, b) corresponds to the representations of Γ in U(p, q) which are irreducible as GL(n, ℂ) representations. Similarly, the subspace R^{∗}[a, b] corresponds to the representations of π_{1}X which are irreducible as PGL(n, ℂ) representations.

We point out, moreover, that the subspace R_{Γ}^{∗}(a, b) includes as a dense open set the representations whose induced adjoint representations on the Lie algebra of PU(p, q) are irreducible. It may also contain some representations whose induced adjoint representation is reducible for the following reason. If (E = V ⊕ W, Φ) is the U(p, q)-Higgs bundle corresponding to a representation in R_{Γ}^{∗}(a, b), then (End(E), Φ) is a polystable Higgs bundle but it is not necessarily stable. The representations with reducible induced adjoint representation are the ones for which (End(E), Φ) is strictly polystable.

3.3 Deformation theory

The results of Biswas and Ramanan [3] and Hitchin [23] readily adapt to describe the deformation theory of U(p, q)-Higgs bundles.

Definition 3.16. Let (E = V ⊕ W, Φ) be a U(p, q)-Higgs bundle. We introduce the following notation:

U = End(E),
U^{+} = End(V) ⊕ End(W),
U^{−} = Hom(W, V) ⊕ Hom(V, W).

With this notation, U = U^{+} ⊕ U^{−}, Φ ∈ H^{0}(U^{−} ⊗ K), and ad(Φ) interchanges U^{+} and U^{−}. We consider the complex of sheaves

C^{•} : U^{+} ⟶^{ad(Φ)} U^{−} ⊗ K. (3.13)

Lemma 3.17. Let (E, Φ) be a stable U(p, q)-Higgs bundle. Then

(3.14)  ker(ad(Φ): H^{0}(U^{+}) → H^{0}(U^{−} ⊗ K)) = ℂ,
(3.15)  ker(ad(Φ): H^{0}(U^{−}) → H^{0}(U^{+} ⊗ K)) = 0.

Proof. By Remark 3.8 (E, Φ) is stable as a GL(n, ℂ)-Higgs bundle. Hence it is simple, that is, its only endomorphisms are the nonzero scalars. Thus,

ker(ad(Φ): H^{0}(U) → H^{0}(U ⊗ K)) = ℂ.

Since U = U^{+} ⊕ U^{−} and ad(Φ) interchanges these two summands, the statements of the lemma follow. q.e.d.

Proposition 3.18 (Biswas-Ramanan [3]).

(1) The space of endomorphisms of (E, Φ) is isomorphic to the zeroth hypercohomology group ℍ^{0}(C^{•}).

(2) The space of infinitesimal deformations of (E, Φ) is isomorphic to the first hypercohomology group ℍ^{1}(C^{•}).

(3) There is a long exact sequence

(3.16) 0 ⟶ ℍ^{0}(C^{•}) ⟶ H^{0}(U^{+}) ⟶ H^{0}(U^{−} ⊗ K) ⟶ ℍ^{1}(C^{•})

⟶ H^{1}(U^{+}) ⟶ H^{1}(U^{−} ⊗ K) ⟶ ℍ^{2}(C^{•}) ⟶ 0,

where the maps H^{i}(U^{+}) ⟶ H^{i}(U^{−} ⊗ K) are induced by ad(Φ).

Proposition 3.19. Let (E, Φ) be a stable U(p, q)-Higgs bundle, then:

(1) ℍ^{0}(C^{•}) = ℂ (in other words (E, Φ) is simple) and

(2) ℍ^{2}(C^{•}) = 0.

Proof. (1) Follows immediately from Lemma 3.17 and Proposition 3.18 (3).

(2) We have natural ad-invariant isomorphisms U^{+} ≅ (U^{+})^{∗} and U^{−} ≅ (U^{−})^{∗}. Thus

ad(Φ): H^{1}(U^{+}) → H^{1}(U^{−} ⊗ K)

is Serre dual to ad(Φ): H^{0}(U^{−}) → H^{0}(U^{+} ⊗ K). Hence Lemma 3.17 and (3) of Proposition 3.18 show that ℍ^{2}(C^{•}) = 0. q.e.d.

Proposition 3.20. The moduli space of stable U(p, q)-Higgs bundles is a smooth complex variety of dimension 1 + (p + q)^{2}(g − 1).

Proof. By Proposition 3.19 (2) ℍ^{2}(C^{•}) = 0 at all points in the moduli space of stable U(p, q)-Higgs bundles. Smoothness is thus a consequence of the results of [3], as follows. Let e ∈ M(a, b) be the point corresponding to a stable U(p, q)-Higgs bundle (E, Φ) and let F be the infinitesimal deformation functor of (E, Φ) as in [3]. Then the completion of the local ring O_{e} pro-represents F (cf. Schlessinger [28]). Now Proposition 3.19 and Theorem 3.1 of [3] show that the completion of O_{e} is regular and hence O_{e} is itself regular. Thus M(a, b) is smooth at e.

Using (2) and (3) of Proposition 3.18, Proposition 3.19 and the Riemann-Roch Theorem, the dimension of the moduli space is given by

dim ℍ^{1}(C^{•}) = 1 − χ(U^{+}) + χ(U^{−} ⊗ K)
= 1 + (p^{2} + q^{2})(g − 1) + 2pq(g − 1)
= 1 + (p + q)^{2}(g − 1).

q.e.d.

Remark 3.21. The dimension of the moduli space of stable U(p, q)- Higgs bundles is half that of the moduli space of stable GL(p + q, ℂ)- Higgs bundles.

Remark 3.22. By Proposition 3.12 M(a, b) is smooth if GCD(p + q, a + b) = 1.

Remark 3.23. As an alternative to the algebraic arguments of [3], the fact that the deformation theory of a U(p, q)-Higgs bundle is controlled by the complex of sheaves (3.13) can be seen from the complex analytic point of view as follows. As in the gauge theory construction of M(a, b) (cf. Section 3.2) let V ⊕ W be a smooth complex vector bundle, and consider a U(p, q)-Higgs bundle as being given by a triple (\overline{∂}_{V} , \overline{∂}_{W} , Φ). Now write down a Dolbeault resolution of the complex C^{•}:

Ω^{0}(U^{+}) ⟶^{ad(Φ)} Ω^{1,0}(U^{−}) \sqrt{∂}  ↓_{−\overline{∂}} Ω^{0,1}(U^{+}) ⟶^{ad(Φ)} Ω^{1,1}(U^{−}) ↓  ↓ 0  ⟶  0.

Consider the associated total complex C^{0} →^{D^{0}} C^{1} →^{D^{1}} C^{2}. Then C^{0} is the Lie algebra of the GL(p, ℂ) × GL(q, ℂ)-gauge group and C^{1} is the tangent space to the affine space of triples (\overline{∂}_{V} , \overline{∂}_{W} , Φ). Furthermore, D^{0} is the infinitesimal action of the complex gauge group, while D^{1} is the derivative of the holomorphicity condition: this gives the desired interpretation of the deformation complex C^{•} in complex analytic terms.

To conclude this line of thought we give an alternative argument for the smoothness of the moduli space of stable U(p, q)-Higgs bundles: suppose that (\overline{∂}_{V} , \overline{∂}_{W} , Φ) corresponds to a stable U(p, q)-Higgs bundle (E, Φ). Proposition 3.19 shows that ℍ^{0}(C^{•}) = ℂ and ℍ^{2}(C^{•}) = 0. The differential of the holomorphicity condition is thus surjective and (E, Φ) has no nontrivial automorphisms. It follows by standard arguments that the moduli space can be constructed as a smooth complex manifold near (E, Φ).

3.4 Bounds on the topological invariants

In this section we show how the Higgs bundle point of view provides an easy proof of a result of Domic and Toledo [11] which allows us to bound the topological invariants deg(V) and deg(W) for which U(p, q)-Higgs bundles may exist. The lemma is a slight variation on the results of [19, Section 3] (cf. also Lemma 3.6 of Markman and Xia [24]).

Lemma 3.24. Let (E, Φ) be a semistable U(p, q)-Higgs bundle. Then

(3.17)  p(μ(V) − μ(E)) ⩽ rk(γ)(g − 1),
(3.18)  q(μ(W) − μ(E)) ⩽ rk(β)(g − 1).

If equality occurs in (3.17) then either (E, Φ) is strictly semistable or p = q and γ is an isomorphism. If equality occurs in (3.18) then either (E, Φ) is strictly semistable or p = q and β is an isomorphism.

Proof. If γ = 0 then V is Φ-invariant. By stability, μ(V) ⩽ μ(E) and equality can only occur if (E, Φ) is strictly semistable. This proves (3.17) in the case γ = 0. We may therefore assume that γ ≠ 0. Let N = ker(γ) ⊆ V and let I = im(γ) ⊗ K^{−1} ⊆ W. Then

rk(N) + rk(I) = p (3.19)

and, since γ induces a nonzero section of det((V/N)^{∗} ⊗ I ⊗ K),

(3.20) deg(N) + deg(I) + rk(I)(2g − 2) ⩾ deg(V).

The bundles N and V ⊕ I are Φ-invariant subbundles of E and hence we obtain by semistability that μ(N) ⩽ μ(E) and μ(V ⊕ I) ⩽ μ(E) or, equivalently, that

(3.21)  deg(N) ⩽ μ(E) rk(N),
(3.22)  deg(I) ⩽ μ(E)(p + rk(I)) − deg(V).

Adding (3.21) and (3.22) and using (3.19) we obtain

(3.23) deg(N) + deg(I) ⩽ 2μ(E)p − deg(V).

Finally, combining (3.20) and (3.23) we get

deg(V) − rk(I)(2g − 2) ⩽ 2μ(E)p − deg(V),

which is equivalent to (3.17) since rk(γ) = rk(I). Note that equality can only occur if we have equality in (3.21) and (3.22) and thus either (E, Φ) is strictly semistable or neither of the subbundles N and V ⊕ I is proper and nonzero. In the latter case, clearly N = 0 and I = W and therefore p = q; furthermore we must also have equality in (3.20) implying that γ is an isomorphism. An analogous argument applied to β proves (3.18). q.e.d.

Remark 3.25. The proof also shows that if we have equality in, say, (3.17) then γ: V/N → I ⊗ K is an isomorphism. In particular, if

p < q and μ(V) − μ(E) = g − 1 then γ: V ⟶^{≅} I ⊗ K.

We can re-formulate Lemma 3.24 to obtain the following corollary.

Corollary 3.26. Let (E, Φ) be a semistable U(p, q)-Higgs bundle. Then

(3.24)  q(μ(E) − μ(W)) ⩽ rk(γ)(g − 1),
(3.25)  p(μ(E) − μ(V)) ⩽ rk(β)(g − 1).

Proof. Use μ(W) − μ(E) = \frac{p}{q}(μ(E) − μ(V)) to see that (3.24) is equivalent to (3.17). Similarly (3.25) is equivalent to (3.18). q.e.d.

An important corollary of the lemma above is the following Milnor– Wood type inequality for U(p, q)-Higgs bundles (due to Domic and Toledo [11], improving on a bound obtained by Dupont [13] in the case G = SU(p, q)). This result gives bounds on the possible values of the topological invariants deg(V) and deg(W).

Corollary 3.27. Let (E, Φ) be a semistable U(p, q)-Higgs bundle. Then

\frac{pq}{p + q} |μ(V) − μ(W)| ⩽ min\{p, q\}(g − 1). (3.26)

Proof. Since μ(E) = \frac{p}{p+q} μ(V) + \frac{q}{p+q} μ(W) we have μ(V) − μ(E) = \frac{q}{p+q} (μ(V) − μ(W) and therefore (3.17) gives

\frac{pq}{p + q} (μ(V) − μ(W)) ⩽ rk(γ)(g − 1).

A similar argument using (3.18) shows that

\frac{pq}{p + q} (μ(W) − μ(V)) ⩽ rk(β)(g − 1).

But, obviously, rk(β) and rk(γ) are both less than or equal to min{p, q}. q.e.d.

Definition 3.28. The Toledo invariant of the representation corresponding to (E = V ⊕ W, Φ) is

τ = τ(a, b) = 2\frac{qa − pb}{p + q} (3.27)

where a = deg(V) and b = deg(W).

Remark 3.29. Since

τ = 2\frac{pq}{p + q} (μ(V) − μ(W)) = −2p(μ(E) − μ(V)) = 2q(μ(E) − μ(W)),

the inequalities in Lemma 3.24 and Corollary 3.26 can be written as

(3.28)  \frac{τ}{2} ⩽ rk(γ)(g − 1),
(3.29)  −\frac{τ}{2} ⩽ rk(β)(g − 1).

Similarly the inequality (3.26) can be written |τ| ⩽ τ_{M}, where

τ_{M} = min{p, q}(2g − 2). (3.30)

3.5 Rigidity and extreme values of the Toledo invariant

If |τ| = τ_{M} then the moduli space M(a, b) has special features. These depend on whether p = q or p ≠ q.

Consider first the case p = q. Notice that if p = q then τ(a, b) = a−b and τ_{M} = p(2g − 2). We thus examine the moduli space M(a, b) when |a − b| = p(2g − 2). Before giving a description we review briefly the notion of L-twisted Higgs pairs. Let L be a line bundle. An L-twisted Higgs pair (V, θ) consists of a holomorphic vector bundle V and an Ltwisted homomorphism θ : V ⟶ V ⊗ L. The notions of stability, semistability and polystability are defined as for Higgs bundles. The moduli space of semistable L-twisted Higgs pairs has been constructed by Nitsure using Geometric Invariant Theory [27]. Let M_{L}(n, d) be the moduli space of polystable L-twisted Higgs pairs of rank n and degree d.

Proposition 3.30. Let p = q and |a − b| = p(2g − 2). Then

M(a, b) ≅ M_{K^{2}}(p, a) ≅ M_{K^{2}}(p, b).

Proof. Let (E = V ⊕ W, Φ) ∈ M(a, b). Suppose for definiteness that b − a = p(2g − 2). From (3.18) it follows that γ : V ⟶ W ⊗ K is an isomorphism. We can then compose β : W ⟶ V ⊗ K with γ ⊗ Id_{K} : V ⊗ K ⟶ W ⊗ K^{2} to obtain a K^{2}-twisted Higgs pair θ_{W} : W ⟶ W ⊗ K^{2}. Similarly, twisting β : W ⟶ V ⊗ K with K and composing with γ, we obtain a K^{2}-twisted Higgs pair θ_{V} : V ⟶ V ⊗K^{2}. Conversely, given an isomorphism γ : V ⟶ W ⊗ K, we can recover β from θ_{V} as well as from θ_{W}. It is clear that the (poly)stability of (E, Φ) is equivalent to the (poly)stability of (V, θ_{V}) and to the (poly)stability of (W, θ_{W}), proving the claim. q.e.d.

Remark 3.31. The moduli space M_{K^{2}}(p, a) contains an open (irreducible) subset consisting of a vector bundle over the moduli space of stable bundles of rank p and degree a. This is because the stability of V implies the stability of any K^{2}-twisted Higgs pair (V, θ_{V}), and H^{1}(End V ⊗ K^{2}) = 0. The rank of the bundle is determined by the Riemann–Roch Theorem.

Now consider the case p ≠ q. For definiteness, we assume p < q. We use the more precise notation M(p, q, a, b) for the moduli space of U(p, q)-Higgs bundles such that deg(V) = a, and deg(W) = b, and write the Toledo invariant as

τ = τ(p, q, a, b) = 2\frac{qa − pb}{p + q}. (3.31)

Theorem 3.32. Suppose (p, q, a, b) are such that p < q and |τ(p, q, a, b)| = p(2g − 2). Then every element in M(p, q, a, b) is strictly semistable and decomposes as the direct sum of a polystable U(p, p)-Higgs bundle with maximal Toledo invariant and a polystable vector bundle of rank (q − p). If τ = p(2g − 2), then

(3.32) M(p, q, a, b) ≅ M(p, p, a, a − p(2g − 2)) × M(q − p, b − a + p(2g − 2)),

where M(q − p, b − a + p(2g − 2)) denotes the moduli space of polystable bundles of rank q − p and degree b − a + p(2g − 2). In particular, the dimension at a smooth point in M(p, q, a, b) is 2+(p^{2}+5q^{2}−2pq)(g−1), and it is hence strictly smaller than the expected dimension.

(A similar result holds if τ = −p(2g − 2) and also if p > q.)

Proof. Let (E = V ⊕W, Φ) ∈ M(p, q, a, b) and suppose τ(p, q, a, b) = p(2g − 2). Then μ(V) − μ(E) = g − 1 and μ(E) − μ(W) = \frac{p}{q}(g − 1). Since rk(β) and rk(γ) are at most p, it follows from (3.17) and (3.25) that rk(β) = rk(γ) = p. Let W_{γ} = im(γ) ⊗ K^{−1} and let W_{β} = ker(β). Then V ⊕ W_{γ} is a Φ-invariant subbundle of V ⊕ W, and μ(V ⊕ W_{γ}) = μ(0 ⊕ W_{β}) = μ(E). We see that (E, Φ) is strictly semistable (as we already knew from Lemma 3.24). Since it is polystable it must split as

(V ⊕ W_{γ}, Φ) ⊕ (0 ⊕ W/W_{γ}, 0).

It is clear that (V ⊕ W_{γ}, Φ) ∈ M(p, p, a, a − p(2g − 2)) and that (V ⊕ W_{γ}, Φ) has maximal Toledo invariant, that is, τ(p, p, a, a − p(2g − 2)) = 2p(g − 1). Also, using

0 ⟶ ker(Φ) ⟶ V ⊕ W ⟶ (V ⊕ W_{γ}) ⊗ K ⟶ 0.

we see that W/W_{γ} ∈ M(q − p, b − a + p(2g − 2)). To complete the proof we observe that

dim M^{s}(p, p, a, a − p(2g − 2)) + dim M^{s}(q − p, b − a + p(2g − 2))

= 1 + (2p)^{2}(g − 1) + 1 + (q − p)^{2}(g − 1) = 2 + (p^{2} + 5q^{2} − 2pq)(g − 1).

Since q > 1, this is smaller than 1 + (p + q)^{2}(g − 1), the dimension of M(p, q, a, b) when the Toledo invariant is not maximal. q.e.d.

Remark 3.33. The fact the moduli space has smaller dimension than expected may be viewed as a certain kind of rigidity. This phenomenon (for large Toledo invariant) has been studied from the point of view of representations of the fundamental group by D. Toledo [33] when p = 1 and L. Hernández [21] when p = 2. We deal here with the general case which, as far as we know, has not appeared previously in the literature.

Corollary 3.34. Fix (p, q, a, b) such that p < q and τ(p, q, a, b) = p(2g − 2). Then

M(p, q, a, b) ≅ M_{K^{2}}(p, a − p(2g − 2)) × M(q − p, b − a + p(2g − 2)).

Proof. It follows from Theorem 3.32 and Proposition 3.30. q.e.d.

4. Morse theory

Morse theoretic techniques for studying the topology of moduli spaces of Higgs bundles were introduced by Hitchin [22, 23]. Though standard Morse theory cannot be applied to M(a, b) when it is not smooth, as we shall see in the following, we can still use Morse theory ideas to count connected components. Throughout this section we assume that p and q are any positive integers and that (a, b) ∈ ℤ ⊕ ℤ is such that |τ| ⩽ τ_{M}, where τ is as in Definition 3.28 and τ_{M} is given by (3.30).

4.1 The Morse function

Consider the moduli space M(a, b) from the gauge theory point of view (cf. Section 3.2). We can then define a real positive function

(4.1)  f: M(a, b) → ℝ
[A, Φ] ↦ \frac{1}{π} ∥Φ∥^{2},

where the L^{2}-norm of Φ is ∥Φ∥^{2} = \frac{\sqrt{−1}}{2} ∫_{X} tr(ΦΦ^{∗}).

We have the following result due to Hitchin [22].

Proposition 4.1.

(1) The function f is proper.

(2) The restriction of f to M^{s}(a, b) is a moment map (up to a constant) for the Hamiltonian circle action [A, Φ] ↦ [A, e^{iθ}Φ].

(3) If M(a, b) is smooth, then f is a perfect Bott-Morse function.

Thus, if the moduli space is smooth, then its number of connected components is bounded by the number of connected components of the subspace of local minima of f. However, even if M(a, b) is not smooth, f can be used to obtain information about the connected components of M(a, b) using the following elementary result.

Proposition 4.2. Let Z be a Hausdorff space and let f: Z → ℝ be proper and bounded below. Then f attains a minimum on each connected component of Z and, furthermore, if the subspace of local minima of f is connected then so is Z.

In particular this applies to our situation, giving:

Proposition 4.3. The function f: M(a, b) → ℝ defined in (4.1) has a minimum on each connected component of M(a, b). Moreover, if the subspace of local minima of f is connected then so is M(a, b).

Definition 4.4. Let

N (a, b) = {(E, Φ) ∈ M(a, b) | β = 0 or γ = 0}. (4.2)

Proposition 4.5. For all (E, Φ) ∈ M(a, b)

f(E, Φ) ⩾ \frac{|τ(a, b)|}{2} (4.3)

with equality if and only if (E, Φ) ∈ N (a, b).

Proof. Writing out the first of Hitchin’s equations (3.2) for a U(p, q)- Higgs bundle (E, Φ) in its componenents on V and W we get the pair of equations

F (A_{V}) + ββ^{∗} + γ^{∗}γ = −\sqrt{−1}μ Id_{V} ω,
F (A_{W}) + γγ^{∗} + β^{∗}β = −\sqrt{−1}μ Id_{W} ω,

where A_{V} and A_{W} are the components on V and W, respectively, of the unitary connection A on E = V ⊕ W. Taking the trace and integrating over X in the first of these equations we get from Chern-Weil theory

deg(V) = μp − \frac{1}{π} ∥β∥^{2} + \frac{1}{π} ∥γ∥^{2},

where we have used ∫_{X} ω = 2π. Since μ = μ(E), this is equivalent to

\frac{1}{π} ∥β∥^{2} − \frac{1}{π} ∥γ∥^{2} = p(μ(E) − μ(V)) = −\frac{τ}{2} .

But f(E, Φ) = \frac{1}{π} ∥β∥^{2} + \frac{1}{π} ∥γ∥^{2} and thus

(4.4)  f(E, Φ) = \frac{2}{π} ∥γ∥^{2} − \frac{τ}{2}

= \frac{2}{π} ∥β∥^{2} + \frac{τ}{2} ,

from which the result is immediate. q.e.d.

The above proposition identifies N (a, b) as the set of global minima of f. The following theorem, which is of fundamental importance to our approach, shows that there are no other local minima.

Theorem 4.6. Let (E, Φ) be a polystable U(p, q)-Higgs bundle in M(a, b). Then (E, Φ) is a local minimum of f: M(a, b) → ℝ if and only if (E, Φ) belongs to N (a, b).

Proof. This follows directly from Proposition 4.5 above and Propositions 4.17 and 4.20, which are given in Sections 4.4 and 4.5, respectively.

q.e.d.

Remark 4.7. This Theorem was already known to hold when p, q ⩽ 2 (by the results of [19], Hitchin [22], and Xia [36]), and also when p = q and (p − 1)(2g − 2) < |τ| ⩽ p(2g − 2) by Markman-Xia [24].

Which section actually vanishes for a minimum is given by the following.

Proposition 4.8. Let (E, Φ) ∈ N (a, b). Then:

(1) γ = 0 if and only if a/p ⩽ b/q (i.e., τ ⩽ 0). In this case,

f(N (a, b)) = b − q(a + b)/(p + q) = −\frac{τ}{2} .

(2) β = 0 if and only if a/p ⩾ b/q (i.e., τ ⩾ 0). In this case,

f(N (a, b)) = a − p(a + b)/(p + q) = \frac{τ}{2} .

In particular, β = γ = 0 if and only if a/p = b/q (i.e., τ = 0) and, in this case, f(E, Φ) = 0.

Proof. The relation between the conditions on τ and those on a/p − b/q follows directly from the definition of τ (cf. (3.27)). The rest follows immediately from (4.4) and the fact that f is, by definition, nonnegative. Alternatively one can argue algebraically, using Lemma 3.24 and polystability. q.e.d.

Corollary 4.9. If a/p = b/q then N (a, b) ≅ M(p, a) × M(q, b).

Proof. If a/p = b/q, then any (E, Φ) ∈ N (a, b) has E = V ⊕ W and Φ = 0. Polystability of (E, Φ) is thus equivalent to the polystability of V and W. q.e.d.

4.2 Critical points of the Morse function

In this section we recall Hitchin’s method [22, 23] for determining the local minima of f and spell out how this works in the case of U(p, q)- Higgs bundles.

Since f is a moment map, a smooth point of the moduli space is a critical point if and only if it is a fixed point of the circle action. To determine the fixed points, note that, if (A, Φ) represents a fixed point then there must be a 1-parameter family of gauge transformations g(θ) taking (A, Φ) to (A, e^{iθ}Φ). This gives an infinitesimal U(p)×U(q)-gauge transformation ψ = ġ which is covariantly constant (i.e., d_{A}ψ = 0) and such that [ψ, Φ] = iΦ. (Note that we can take ψ to be trace-free.) It follows that we can decompose E in holomorphic subbundles F_{λ} on which ψ acts as iλ and furthermore that Φ maps F_{λ} to F_{λ+1} ⊗ K. We thus have the following result.

Proposition 4.10. A U(p, q)-Higgs bundle (E, Φ) in M(a, b) represents a fixed point of the circle action if and only if it is a system of Hodge bundles, that is,

E = F_{1} ⊕ ⋯ ⊕ F_{m} (4.5)

for holomorphic vector bundles F_{i} such that the restriction

Φ_{i} := Φ_{|F_{i}} ∈ H^{0}(Hom(F_{i}, F_{i+1}) ⊗ K),

and the F_{i} are direct sums of bundles contained in V and W. Furthermore, each F_{i} is an eigenbundle for an infinitesimal trace-free gauge transformation ψ. If Φ_{i} ≠ 0, then the weight of ψ on F_{i+1} is one plus the weight of ψ on F_{i}. Moreover, if (E, Φ) is stable, then each restriction Φ_{i} is nonzero and the F_{i} are alternately contained in V and W.

Proof. Only the last statement requires a proof. But if some component of Φ vanished, or if some F_{i} had a nonzero component in both V and W, then (E, Φ) would be reducible and hence not stable. q.e.d.

When (E, Φ) is stable the decomposition E = F_{1} ⊕ ⋯ ⊕ F_{m} gives a corresponding decomposition of the bundle U = End(E) into eigenbundles for the adjoint action of ψ:

U = ⨁_{k=−m+1}^{m−1} U_{k},

where U_{k} = ⨁_{i−j=k} Hom(F_{j}, F_{i}) is the eigenbundle corresponding to the eigenvalue ik.

By Hitchin’s calculations in [23, §8] (see also [18, Section 2.3.2]) the eigenvalues of the Hessian of f at a smooth critical point can be determined in the following way.

Proposition 4.11. Let (E, Φ) be a stable U(p, q)-Higgs bundle which represents a critical point of f. Then the eigenspace of the Hessian of f corresponding to the eigenvalue −k is ℍ^{1} of the following complex:

C_{k}^{•} : U_{k}^{+} ⟶^{ad(Φ)} U_{k+1}^{−} ⊗ K, (4.6)

where we use the notation

U_{k}^{+} = U_{k} ∩ U^{+},
U_{k}^{−} = U_{k} ∩ U^{−},

with U^{+} and U^{−} as defined in Definition 3.16. In particular (E, Φ) corresponds to a local minimum of f if and only if

ℍ^{1}(C_{k}^{•}) = 0

for all k ⩾ 1.

Remark 4.12. When (E, Φ) is a stable U(p, q)-Higgs bundle, we know from Proposition 4.10 that the F_{i} are alternately contained in V and W. Thus we have

U^{+} = ⨁_{k even} U_{k} ; U^{−} = ⨁_{k odd} U_{k}. (4.7)

In particular all the eigenvalues of the Hessian of f are even.

Remark 4.13. The description in Proposition 4.11 of the eigenspace of the Hessian of f gives rise to the long exact sequence

0 ⟶ ℍ^{0}(C_{k}^{•}) ⟶ H^{0}(U_{k}^{+}) ⟶ H^{0}(U_{k+1}^{−} ⊗ K) ⟶ ℍ^{1}(C_{k}^{•})

⟶ H^{1}(U_{k}^{+}) ⟶ H^{1}(U_{k+1}^{−} ⊗ K) ⟶ ℍ^{2}(C_{k}^{•}) ⟶ 0.

Suppose that (E, Φ) is a stable U(p, q)-Higgs bundle. The vanishing result of Proposition 3.19 shows that ℍ^{0}(C_{k}^{•}) = ℍ^{2}(C_{k}^{•}) = 0 for k ≠ 0 (while ℍ^{0}(C_{0}^{•}) = ℂ and ℍ^{2}(C_{0}^{•}) = 0). Hence one can use this exact sequence, Remark 4.12, and the Riemann–Roch formula to calculate the dimension of ℍ^{1}(C_{k}^{•}) for any k in terms of the ranks and the degrees of the F_{i}. This provides a method for calculating the Morse index of f at a critical point. However, we shall omit the formula since we have no need for it.

4.3 Local minima and the adjoint bundle

In this section we give a criterion for (E, Φ) to be a local minimum in terms of the adjoint bundle. This is the key step in the proof of Theorem 4.6. We use the notation introduced in Section 4.2.

Consider the complex C_{k}^{•} defined in (4.6) and let

χ(C_{k}^{•}) = dim ℍ^{0}(C_{k}^{•}) − dim ℍ^{1}(C_{k}^{•}) + dim ℍ^{2}(C_{k}^{•}).

Proposition 4.14. Let (E, Φ) be a polystable U(p, q)-Higgs bundle which is a fixed point of the S^{1}-action on M(a, b). Then χ(C_{k}^{•}) ⩽ 0 and equality holds if and only if

ad(Φ): U_{k}^{+} → U_{k+1}^{−} ⊗ K

is an isomorphism.

Proof. For simplicity we shall adopt the notation

Φ_{k}^{±} = ad(Φ)_{|U_{k}^{±}}: U_{k}^{±} ⟶ U_{k+1}^{∓} ⊗ K.

The key fact we need is that there is a natural ad-invariant isomorphism U ≅ U^{∗} under which we have U^{+} ≅ (U^{+})^{∗}, U^{−} ≅ (U^{−})^{∗} and U_{k}^{±} ≅ (U_{−k}^{±})^{∗}. Since ad(Φ)^{t} = ad(Φ) ⊗ 1_{K^{−1}} under this isomorphism we have

(Φ_{k}^{±})^{t} = Φ_{−k−1}^{∓} ⊗ 1_{K^{−1}}. (4.8)

We have the short exact sequence

0 ⟶ ker(Φ_{k}^{+}) ⟶ (U_{k+1}^{−} ⊗ K)^{∗} ⟶ im(Φ_{k}^{+}) ⟶ 0.

From (4.8) we have ker(Φ_{k}^{+,t}) ≅ ker(Φ_{−k−1}^{−}) ⊗ K^{−1}. Thus, tensoring the above sequence by K, we obtain the short exact sequence

0 ⟶ ker(Φ_{−k−1}^{−}) ⟶ (U_{k+1}^{−})^{∗} ⟶ im(Φ_{k}^{+}) ⊗ K ⟶ 0.

It follows that

deg(im(Φ_{k}^{+})) ⩽ deg(U_{k+1}^{−}) + (2g − 2) rk(Φ_{k}^{+}) + deg(ker(Φ_{−k−1}^{−})).

Combining this inequality with the fact that

(4.9) deg(U_{k}^{+}) ⩽ deg(ker(Φ_{k}^{+})) + deg(im(Φ_{k}^{+})), we obtain

(4.10)  deg(U_{k}^{+}) ⩽ deg(U_{k+1}^{−}) + (2g − 2) rk(Φ_{k}^{+})
+ deg(ker(Φ_{−k−1}^{−})) + deg(ker(Φ_{k}^{+})).

Since (E, Φ) is semistable, so is the Higgs bundle (End(E), ad(Φ)). Clearly the kernel ker(Φ_{k}^{±}) ⊆ End(E) is Φ-invariant and hence, from semistability,

deg(ker(Φ_{k}^{±})) ⩽ 0,

for all k. Substituting this inequality in (4.10), we obtain

(4.11) deg(U_{k}^{+}) ⩽ deg(U_{k+1}^{−}) + (2g − 2) rk(Φ_{k}^{+}).

From the long exact sequence (4.6) and the Riemann–Roch formula we obtain

χ(C_{k}^{•}) = χ(U_{k}^{+}) − χ(U_{k+1}^{−} ⊗ K)
= (1 − g)(rk(U_{k}^{+}) + rk(U_{k+1}^{−})) + deg(U_{k}^{+}) − deg(U_{k+1}^{−}).

Using this identity and the inequality (4.11) we see that

χ(C_{k}^{•}) ⩽ (g − 1)(2 rk(Φ_{k}^{+}) − rk(U_{k}^{+}) − rk(U_{k+1}^{−})).

Hence χ(C_{k}^{•}) ⩽ 0. Furthermore, if equality holds we have

rk(Φ_{k}^{+}) = rk(U_{k}^{+}) = rk(U_{k+1}^{−})

and also equality must hold in (4.11) and so deg(im(Φ_{k}^{+})) = deg(U_{k+1}^{−} ⊗ K), showing that Φ_{k}^{+} is an isomorphism as claimed. q.e.d.

Corollary 4.15. Let (E, Φ) be a stable U(p, q)-Higgs bundle which represents a critical point of f. This critical point is a local minimum if and only if

ad(Φ): U_{k}^{+} → U_{k+1}^{−} ⊗ K

is an isomorphism for all k ⩾ 1.

Proof. By Proposition 3.19 we have ℍ^{0}(C_{k}^{•}) = ℍ^{2}(C_{k}^{•}) = 0 for k ⩾ 1. Hence we have −χ(C_{k}^{•}) = ℍ^{1}(C_{k}^{•}) and the result follows from Propositions 4.11 and 4.14. q.e.d.

Remark 4.16. Let (P, Φ) be a G-Higgs bundle as defined in Remark 3.5 and define

U = P ×_{Ad} 𝔤^{ℂ},
U^{+} = P ×_{Ad} 𝔥^{ℂ},
U^{−} = P ×_{Ad} 𝔪^{ℂ}.

Then U = U^{+} ⊕ U^{−} and if (P, Φ) is fixed under the circle action we can write U = ⨁ U_{k} as a direct sum of eigenbundles for an infinitesimal gauge transformation as before. Thus we can define a complex C_{k}^{•} as in (4.6). If (P, Φ) is a stable G-Higgs bundle, then the Higgs vector bundle (U, ad(Φ)) is semistable and so the proof of Proposition 4.14 goes through unchanged. Thus this key result is valid in the more general setting.

4.4 Stable Higgs bundles

In this section we prove Theorem 4.6 for stable Higgs bundles. The reducible (polystable) ones are dealt with in the next section. We continue to use the notation of Section 4.2.

Proposition 4.17. Let (E, Φ) = (F_{1} ⊕ ⋯ ⊕ F_{m}, Φ) be a stable U(p, q)-Higgs bundle representing a critical point of f such that m ⩾ 3. Then (E, Φ) is not a local minimum of f.

Proof. Note that U_{k} = 0 for |k| ⩾ m; in particular U_{m} = 0. We shall consider the cases when m is odd and even separately.

The case m odd. In this case m − 1 is even and so, using Remark 4.12 we see that U_{m−1}^{+} = U_{m−1} ≠ 0 while U_{m}^{−} ⊆ U_{m} = 0. Hence ad(Φ): U_{m−1}^{+} → U_{m}^{−} ⊗ K cannot be an isomorphism and we are done by Corollary 4.15.

The case m even. From Remark 4.12 we see that

U_{m−1}^{−} = U_{m−1} = Hom(F_{1}, F_{m}),
U_{m−2}^{+} = U_{m−2} = Hom(F_{1}, F_{m−1}) ⊕ Hom(F_{2}, F_{m}).

Thus, by Corollary 4.15 it suffices to prove that

ad(Φ): U_{m−2} → U_{m−1} ⊗ K

is not an isomorphism. In fact the restriction of ad(Φ) to a fiber cannot even be injective. Indeed, if it were, then its restriction to Hom(F_{1}, F_{m−1}) would be injective and hence Φ_{m−1} would also be injective. Take a nonzero element η ∈ Hom(F_{2}, F_{m}) whose image is contained in the image of Φ_{m−1}. Define ζ = Φ^{−1}ηΦ ∈ Hom(F_{1}, F_{m−1}). Then ad(Φ)(η + ζ) = 0 which is a contradiction. q.e.d.

Remark 4.18. Let (E, Φ) be a stable U(p, q)-Higgs bundle with β = 0 or γ = 0. Then, as pointed out above, Proposition 4.5 shows that (E, Φ) is a local minimum of f. This can also be seen from the Morse theory point of view, as follows. Such a Higgs bundle either has β = γ = 0 or it is a Hodge bundle of length 2. In the former case, clearly we have End(E) = U_{0}. In the latter case, E = F_{1} ⊕ F_{2} with F_{1} = V and F_{2} = W (if β = 0) or vice-versa (if γ = 0). Hence End(E) = U_{−1} ⊕ U_{0} ⊕ U_{1}. Hence, in both cases U_{k} = 0 for |k| > 1. It follows that the complex C_{k}^{•} is zero for any k > 0 and hence all eigenvalues of the Hessian of f are positive.

4.5 Reducible Higgs bundles

In this section we shall finally conclude the proof of Theorem 4.6 by showing that it also holds for reducible Higgs bundles. First we shall show that a reducible Higgs bundle which is not of the form given in Theorem 4.6 cannot be a local minimum of f; for this we use an argument similar to the one given by Hitchin [23, §8] for the case of G = PSL(n, ℝ).

Let (E, Φ) be a strictly polystable U(p, q)-Higgs bundle which is a local minimum of f. Since f(E, Φ) is the sum of the values of f on each of the stable direct summands (on the corresponding lower rank moduli space), it follows that each stable direct summand must be a local minimum in its moduli space and, therefore, a fixed point of the circle action. Hence (E, Φ) is itself fixed and thus (cf. Proposition 4.10)

E = ⨁ F_{λ},

where each F_{λ} is an iλ-eigenbundle for an infinitesimal trace-free U(p)× U(q)-gauge transformation ψ. Moreover, if Φ_{|F_{λ}} ≠ 0, then its image is contained in F_{λ+1} ⊗ K. In analogy with the case of stable U(p, q)-Higgs bundles we write

End E = ⨁ U_{μ},

where U_{μ} is the iμ-eigenbundle for the adjoint action of ψ. Let

U_{μ}^{+} = U_{μ} ∩ U^{+},
U_{μ}^{−} = U_{μ} ∩ U^{−},

then we can define a complex of sheaves

C_{>0}^{•} : ⨁_{μ>0} U_{μ}^{+} ⟶^{ad(Φ)} ⨁_{μ>1} U_{μ}^{−} ⊗ K. (4.12)

In this language Hitchin’s criterion [23, §8] for showing that a given fixed point is not a local minimum can be expressed as follows.

Lemma 4.19. Let (E_{t}, Φ_{t}) be a 1-parameter family of polystable U(p, q)-Higgs bundles such that (E_{0}, Φ_{0}) is a fixed point of the circle action. If the tangent vector (Ė, Φ̇) at 0 is nontrivial and lies in the subspace

ℍ^{1}(C_{>0}^{•})

of the infinitesimal deformation space ℍ^{1}(C^{•}) of (E_{0}, Φ_{0}), then (E_{0}, Φ_{0}) is not a local minimum of f.

Proposition 4.20. Let (E, Φ) be a reducible U(p, q)-Higgs bundle. If β ≠ 0 and γ ≠ 0 then (E, Φ) is not a local minimum of f.

Proof. As we noted above, each stable direct summand of (E, Φ) is a local minimum on its moduli space and therefore (by Proposition 4.17) it has β = 0 or γ = 0. Hence we can choose two stable direct summands (E′ = V′⊕W′, Φ′) and (E′′ = V′′⊕W′′, Φ′′) such that γ′ ≠ 0 and β′′ ≠ 0 and β′ = γ′′ = 0. It is clearly sufficient to show that (E′ ⊕ E′′, Φ′ ⊕ Φ′′) is not a local minimum of f on the corresponding moduli space and we can therefore assume that (E, Φ) = (E′ ⊕ E′′, Φ′ ⊕ Φ′′) without loss of generality. We shall construct a family of deformations (E_{t}, Φ_{t}) of (E, Φ) satisfying the conditions of Lemma 4.19.

By Lemma 4.21 both H^{1}(Hom(W′′, W′)) and H^{1}(Hom(V′, V′′)) are non-vanishing, so let η ∈ H^{1}(Hom(V′, V′′)) and σ ∈ H^{1}(Hom(W′′, W′)) be nonzero. We can then define a deformation of (E, Φ) by using that η defines an extension

0 ⟶ V′′ ⟶ V^{η} ⟶ V′ ⟶ 0,

while σ defines an extension

0 ⟶ W′ ⟶ W^{σ} ⟶ W′′ ⟶ 0.

Let E^{(η,σ)} = V^{η} ⊕ W^{σ} and define Φ^{(η,σ)} by the compositions

b^{(η,σ)} : W^{σ} ⟶ W′′ ⟶^{β′′} V′′ → V^{η},
c^{(η,σ)} : V^{η} ⟶ V′ ⟶^{γ′} W′ ⟶ W^{σ}.

Note that (E^{0}, Φ^{0}) = (E, Φ) (the Higgs fields agree since β′ = γ′′ = 0). It is then easy to see that (E^{η,σ}, Φ^{η,σ}) is stable: the essential point is that the destabilizing subbundles V′ and W′′ of (E, Φ) are not subbundles of the deformed Higgs bundle; we leave the details to the reader.

Now define the family (E_{t}, Φ_{t}) = (E^{(ηt,σt)}, Φ^{(ηt,σt)}). It is clear that the induced infinitesimal deformation of E is

Ė = (η, σ) ∈ H^{1}(Hom(V′, V′′)) ⊕ H^{1}(Hom(W′′, W′)) ⊆ H^{1}(End(E)).

Considering the holomorphic structure as given by a \overline{∂}-operator on the underlying smooth bundle, our definition of (E^{(η,σ)}, Φ^{(η,σ)}) did not change the Higgs field but only the holomorphic structure on E. Thus, taking a Dolbeault representative (cf. Remark 3.23) for (Ė, Φ̇) ∈ ℍ^{1}(C^{•}) we see that the weights of ψ on (Ė, Φ̇) are given by its weights on Ė. From Proposition 4.10 we have decompositions E′ = ⨁ F′_{k} and E′′ = ⨁ F′′_{k} into eigenspaces of infinitesimal trace-free gauge transformations ψ′ and ψ′′. Note that the infinitesimal gauge transformation producing the decomposition of E is ψ = ψ′ + ψ′′. Clearly we have

F′_{1} = V′,  F′_{2} = W′,
F′′_{1} = W′′,  F′′_{2} = V′′.

Let λ′_{V} and λ′_{W} be the weights of the action of ψ′ on V′ and W′ respectively, and analogously for E′′. We then have that

λ′_{W} = λ′_{V} + 1,  λ′′_{V} = λ′′_{W} + 1.

and, since tr ψ′ = tr ψ′′ = 0,

λ′_{V} p′ + λ′_{W} q′ = 0,
λ′′_{V} p′′ + λ′′_{W} q′′ = 0,

where p′ = rk(V′), q′ = rk(W′), p′′ = rk(V′′) and q′′ = rk(W′′). From these equations we conclude that

λ′_{W} − λ′′_{W} = \frac{p′}{p′ + q′} + \frac{p′′}{p′′ + q′′} > 0,
λ′′_{V} − λ′_{V} = \frac{q′′}{p′′ + q′′} + \frac{q′}{p′ + q′} > 0.

It follows that the weights of ψ on H^{1}(Hom(W′′, W′)) and H^{1}(Hom(V′, V′′)) are both positive and hence that (Ė, Φ̇) lies in a direct sum of positive weight spaces of ψ. This concludes the proof of the proposition.

q.e.d.

Lemma 4.21 Let (E′ = V′ ⊕ W′, Φ′) and (E′′ = V′′ ⊕ W′′, Φ′′) be stable U(p, q)-Higgs bundles of the same slope. Then the cohomology groups H^{1}(Hom(V′, V′′)) and H^{1}(Hom(W′′, W′)) are both non-vani-shing.

Proof. Since γ′′ = 0, V′′ is a Φ-invariant subbundle of E′′. Thus μ(V′′) < μ(E′′). Using the Riemann–Roch formula and the equality μ(E′′) = μ(E′) we obtain

h^{0}(Hom(V′, V′′) − h^{1}(Hom(V′, V′′) = p′p′′(1 − g + μ(V′′) − μ(V′))
< p′p′′(1 − g + μ(E′) − μ(V′)).

Since rk(β′) ⩽ p′ the inequality (3.25) of Corollary 3.26 shows that μ(E′) − μ(V′) ⩽ g − 1 and we therefore deduce that

h^{0}(Hom(V′, V′′) − h^{1}(Hom(V′, V′′) < 0,

from which it follows that H^{1}(Hom(V′, V′′) ≠ 0.

Similarly one sees that H^{1}(Hom(W′′, W′)) ≠ 0. q.e.d.

4.6 Local minima and connectedness

In this section we obtain connectedness results on M^{s}(a, b) and its closure \overline{M}^{s}(a, b). We denote by N^{s}(a, b) ⊆ N (a, b) the subspace consisting of stable U(p, q)-Higgs bundles, and denote its closure by \overline{N}^{s}(a, b).

The invariants (a, b) will be fixed in the following and we shall occasionally drop them from the notation and write M = M(a, b), etc.

Proposition 4.22. The closure of N^{s} in M coincides with \overline{N}^{s} and

\overline{N}^{s} = \overline{M}^{s} ∩ N .

Proof. Clear. q.e.d.

Now consider the restriction of the Morse function to \overline{M}^{s},

f: \overline{M}^{s} → ℝ.

Proposition 4.23. The restriction of f to \overline{M}^{s} is proper and the subspace of local minima of this function coincides with \overline{N}^{s}.

Proof. Properness of the restriction follows from properness of f and the fact that \overline{M}^{s} is closed in M. By Proposition 4.5 f is constant on N and its value there is its global minimum on M. Thus \overline{N}^{s} is contained in the subspace of local minima of f.

It remains to see that f has no other local minima on \overline{M}^{s}. We already know that the subspace of local minima on M^{s} is N^{s}. Thus, since M^{s} is open in \overline{M^{s}}, there cannot be any additional local minima on M^{s}. We need to prove therefore that there are no local minima in (\overline{M^{s}} ∖ M^{s}) ∖ \overline{N}^{s}. So let (E, Φ) be a strictly poly-stable U(p, q)-Higgs bundle representing a point in this space. From Proposition 4.22 we see that β ≠ 0 and γ ≠ 0. In the proof of Proposition 4.20 we constructed a family (E_{t}, Φ_{t}) of U(p, q)-Higgs bundles such that (E, Φ) = (E_{0}, Φ_{0}) and (E_{t}, Φ_{t}) is stable for t ≠ 0. Furthermore we showed that the restriction of f to this family does not have a local minimum at (E_{0}, Φ_{0}). It follows that (E, Φ) is not a local minimum of f on \overline{M^{s}}. q.e.d.

Proposition 4.24.

(1) If N (a, b) is connected, then so is M(a, b).

(2) If N^{s}(a, b) is connected, then so is \overline{M}^{s}(a, b).

Proof. (1) In view of Proposition 4.3, this follows from Theorem 4.6.

(2) If N^{s}(a, b) is connected, then so is its closure \overline{N}^{s}(a, b). But from Proposition 4.23, \overline{N}^{s}(a, b) is the subspace of local minima of the proper positive map f: \overline{M}^{s}(a, b) → ℝ. Hence the result follows from Proposition 4.2. q.e.d.

5. Local minima as holomorphic triples

The next step is to identify the spaces N (a, b) and N^{s}(a, b) as moduli spaces in their own right. By definition (cf. Definition 4.4), the Higgs bundles in N (a, b) all have β = 0 or γ = 0 in their Higgs fields. Suppose first that (E, Φ) is a U(p, q)-Higgs bundle with γ = 0. Then (E, Φ) determines the triple T = (E_{1}, E_{2}, ϕ) where

E_{1} = V ⊗ K,
E_{2} = W,
ϕ = β.

Conversely, given two holomorphic bundles E_{1}, E_{2} of rank p and q respectively, together with a bundle endomorphism Φ ∈ H^{0}(Hom(E_{2}, E_{1})), we can use the above relations to define a U(p, q)-Higgs bundle with γ = 0. Similarly, there is a bijective correspondence between U(p, q)- Higgs bundles with β = 0 and holomorphic triples in which

E_{1} = W ⊗ K,
E_{2} = V,
ϕ = γ.

The triples (E_{1}, E_{2}, Φ) are examples of the holomorphic triples studied in [4] and [15].

5.1 Holomorphic triples

We briefly recall the relevant definitions, referring to [4] and [15] for details. A holomorphic triple on X, T = (E_{1}, E_{2}, ϕ), consists of two holomorphic vector bundles E_{1} and E_{2} on X and a holomorphic map ϕ: E_{2} → E_{1}. Denoting the ranks E_{1} and E_{2} by n_{1} and n_{2}, and their degrees by d_{1} and d_{2}, we refer to (n, d) = (n_{1}, n_{2}, d_{1}, d_{2}) as the type of the triple.

A homomorphism from T′ = (E′_{1}, E′_{2}, ϕ′) to T = (E_{1}, E_{2}, ϕ) is a commutative diagram

E′_{2} ⟶^{ϕ′} E′_{1}
↓  ↓
E_{2} ⟶^{ϕ} E_{1}.

T′ = (E′_{1}, E′_{2}, ϕ′) is a subtriple of T = (E_{1}, E_{2}, ϕ) if the homomorphisms of sheaves E′_{1} → E_{1} and E′_{2} → E_{2} are injective.

For any α ∈ ℝ the α-degree and α-slope of T are defined to be

deg_{α}(T) = deg(E_{1}) + deg(E_{2}) + α rk(E_{2}),
μ_{α}(T) = \frac{deg_{α}(T)}{rk(E_{1}) + rk(E_{2})}

= μ(E_{1} ⊕ E_{2}) + α\frac{rk(E_{2})}{rk(E_{1}) + rk(E_{2})}.

The triple T = (E_{1}, E_{2}, ϕ) is α-stable if

μ_{α}(T′) < μ_{α}(T) (5.1) for any proper sub-triple T′ = (E′_{1}, E′_{2}, ϕ′). Define α-semistability by replacing (5.1) with a weak inequality. A triple is called α-polystable if it is the direct sum of α-stable triples of the same α-slope. It is strictly α-semistable (polystable) if it is α-semistable (polystable) but not α-stable.

We denote the moduli space of isomorphism classes of α-polystable triples of type (n_{1}, n_{2}, d_{1}, d_{2}) by

N_{α} = N_{α}(n, d) = N_{α}(n_{1}, n_{2}, d_{1}, d_{2}). (5.2)

Using Seshadri S-equivalence to define equivalence classes, this is the moduli space of equivalence classes of α-semistable triples. The isomorphism classes of α-stable triples form a subspace which we denoted by N_{α}^{s}.

Proposition 5.1 ([4, 15]). The moduli space N_{α}(n_{1}, n_{2}, d_{1}, d_{2}) is a complex analytic variety, which is projective when α is rational. A necessary condition for N_{α}(n_{1}, n_{2}, d_{1}, d_{2}) to be nonempty is

\begin{cases}0 ⩽ α_{m} ⩽ α ⩽ α_{M} & if n_{1} ≠ n_{2} \\ 0 ⩽ α_{m} ⩽ α & if n_{1} = n_{2} \end{cases} (5.3)

where

(5.4)  α_{m} = μ_{1} − μ_{2},
(5.5)  α_{M} = (1 + \frac{n_{1} + n_{2}}{|n_{1} − n_{2}|}) (μ_{1} − μ_{2})

and μ_{1} = \frac{d_{1}}{n_{1}} , μ_{2} = \frac{d_{2}}{n_{2}} .

Within the allowed range for α there is a discrete set of critical values. These are the values of α for which it is numerically possible to have a subtriple T′ = (E′_{1}, E′_{2}, ϕ′) such that μ(E′_{1} ⊕ E′_{2}) ≠ μ(E_{1} ⊕ E_{2}) but μ_{α}(T′) = μ_{α}(T′). All other values of α are called generic. The critical values of α are precisely the values for α at which the stability properties of a triple can change, i.e., there can be triples which are strictly α-semistable, but either α′-stable or α′-unstable for α′ ≠ α.

Strict α-semistability can, in general, also occur at generic values for α, but only if there can be subtriples with μ(E′_{1} ⊕ E′_{2}) = μ(E_{1} ⊕ E_{2}) and \frac{n′_{2}}{n′_{1}+n′_{2}} = \frac{n_{2}}{n_{1}+n_{2}} . In this case the triple is strictly α-semistable for all values of α. We refer to this phenomenon as α-independent semistability. This cannot happen if GCD(n_{2}, n_{1} + n_{2}, d_{1} + d_{2}) = 1.

5.2 Identification of N (a, b)

The following result relates the stability conditions for holomorphic triples and that for U(p, q)-Higgs bundles.

Proposition 5.2. A U(p, q)-Higgs bundle (E, Φ) with β = 0 or γ = 0 is (semi)stable if and only if the corresponding holomorphic triple is α-(semi)stable for α = 2g − 2.

Proof. Let T = (E_{1}, E_{2}, ϕ) be the triple corresponding to the Higgs bundle (V ⊕ W, Φ). For definiteness we shall assume that γ = 0 (of course, the same argument applies if β = 0). Thus E_{1} = V ⊗ K and E_{2} = W and, hence,

deg(E_{1}) = deg(V) + p(2g − 2).

Since p = rk(E_{1}) and q = rk(E_{2}) it follows that

μ_{α}(T) = μ(E) + \frac{p}{p + q} (2g − 2) + \frac{q}{p + q} α. (5.6)

If we set α = 2g − 2 we therefore have

μ_{α}(T) = μ(E) + 2g − 2. (5.7)

Clearly the correspondence between holomorphic triples and U(p, q)- Higgs bundles gives a correspondence between sub-triples T′ = (E′_{1}, E′_{2}, ϕ′) and Φ-invariant subbundles of E which respect the decomposition E = V ⊕ W (i.e., subbundles E′ = V′ ⊕ W′ with V′ ⊆ V and W′ ⊆ W). Now, it follows from (5.7) that μ(E′) < μ(E) if and only if μ_{α}(T′) < μ_{α}(T) (and similarly for semistability), thus concluding the proof. q.e.d.

We thus have the following important characterization of the subspace of local minima of f on M(a, b).

Theorem 5.3. Let N (a, b) be the subspace of local minima of f on M(a, b) and let τ be the Toledo invariant as defined in Definition 3.28.

If a/p ⩽ b/q, or equivalently if τ ⩽ 0, then N (a, b) can be identified with the moduli space of α-polystable triples of type (p, q, a+p(2g−2), b), with α = 2g − 2.

If a/p ⩾ b/q, or equivalently if τ ⩾ 0, then N (a, b) can be identified with the moduli space of α-polystable triples of type (q, p, b+q(2g−2), a), with α = 2g − 2.

That is,

N (a, b)

≅ \begin{cases}N_{2g−2}(p, q, a + p(2g − 2), b) & if a/p ⩽ b/q (equivalently τ ⩽ 0) \\ N_{2g−2}(q, p, b + q(2g − 2), a) & if a/p ⩾ b/q (equivalently τ ⩾ 0). \end{cases}

Proof. This follows from Theorem 4.6, Proposition 4.8, and Proposition 5.2. q.e.d.

Thus, combining Proposition 4.24 and Theorem 5.3, we get:

Theorem 5.4.

(1) Suppose a/p ⩽ b/q. If N_{2g−2}(p, q, a+p(2g−2), b) is connected then M(a, b) is connected. If N_{2g−2}^{s}(p, q, a + p(2g − 2), b) is connected then \overline{M}^{s}(a, b) is connected.

(2) Suppose a/p ⩾ b/q. If N_{2g−2}(q, p, b+q(2g−2), a) is connected then M(a, b) is connected. If N_{2g−2}^{s}(q, p, b + q(2g − 2), a) is connected then \overline{M}^{s}(a, b) is connected.

5.3 The Toledo invariant, 2g−2, and α-stability for triples

In view of Theorems 5.3 and 5.4, it is important to understand where 2g − 2 lies in relation to the range (given by Proposition 5.1) for the stability parameter α. Recall that for given (p, q, a, b), the Toledo invariant (Definition 3.28) is constrained by 0 ⩽ |τ| ⩽ τ_{M}, where (see (3.30)) τ_{M} = min{p, q}(2g −2). Recall also that α is constrained by the bounds given in Proposition 5.1. Whenever necessary we shal indicate the dependence of α_{m} and α_{M} on (p, q, a, b) by writing α_{m} = α_{m}(p, q, a, b), and similarly for α_{M}.

Lemma 5.5. Fix (p, q, a, b). Then

α_{m}(p, q, a, b) = (2g − 2) − \frac{p + q}{2pq} |τ| (5.8)

where τ is the Toledo invariant. If p ≠ q then

α_{M}(p, q, a, b) = ( \frac{2 max\{p, q\}}{|p − q|}) α_{m}(p, q, a, b). (5.9)

If p = q then α_{M}(p, q, a, b) = ∞.

Proof. By Theorem 5.3 the type of the triple is determined by the sign of τ. The result thus follows by applying (5.3) and (5.4) to triples of type (p, q, a + p(2g − 2), b) (if τ ⩽ 0) or type (q, p, b + q(2g − 2), a) (if τ ⩾ 0). q.e.d.

Proposition 5.6. Fix (p, q, a, b). Then

0 ⩽ |τ| ⩽ τ_{M} ⇔ \begin{cases}0 < α_{m}(p, q, a, b) ⩽ 2g − 2 ⩽ α_{M}(p, q, a, b) & if p ≠ q \\ 0 ⩽ α_{m}(p, q, a, b) ⩽ 2g − 2 & if p = q \end{cases} (5.10)

Furthermore,

τ = 0 ⇔ 2g − 2 = α_{m} (5.11)

and

|τ| = τ_{M} ⇔ \begin{cases}2g − 2 = α_{M} & if p ≠ q \\ α_{m} = 0 & if p = q. \end{cases} (5.12)

Proof. Using (5.8) and (5.9) we see that 0 ⩽ |τ| ⩽ τ_{M} is equivalent to

2g − 2 ⩾ α_{m} ⩾ (\frac{|p − q|}{2 max\{p, q\}}) (2g − 2), (5.13)

and hence also (assuming p ≠ q) to

(\frac{2 max\{p, q\}}{|p − q|}) (2g − 2) ⩾ α_{M} ⩾ (2g − 2). (5.14)

In both (5.13) and (5.14), we get equality in the first place if and only if τ = 0, and in the second place if and only if |τ| = τ_{M}. Notice that \frac{|p−q|}{2 max\{p,q\}} is strictly positive if p ≠ q and is zero if p = q. The results follow. q.e.d.

These results are summarized in Figure 1, which can be used as follows. For any allowed value of τ, draw a horizontal line at height τ. The corresponding range for α and the relative location of 2g − 2 are then read off from the α-axis.

Remark 5.7. The above proposition gives another explanation for the Milnor–Wood inequality in Corollary 3.27. Using the fact that the

[figure: p ≠ q; τ; −τ_{M}; 0; τ_{M}; α; 0; \frac{|p−q|}{2 max\{p,q\}}(2g − 2); 2g − 2; α_{m}; α_{M}; p = q; τ; −τ_{M}; 0; τ_{M}; α; 0; 2g − 2; α_{m}; d_{1} − d_{2}]

Figure 1: Range for the stability parameter α for triples in N (a, b), displayed as functions of τ = \frac{2pq}{p+q} (\frac{a}{p} − \frac{b}{q}), and showing the relative location of 2g − 2. non-emptiness of M(a, b) is equivalent to the non-emptiness of N (a, b) and hence to that of either N_{2g−2}(p, q, a+p(2g −2), b) or N_{2g−2}(q, p, b+ q(2g − 2), a), we see that the Milnor–Wood inequality is equivalent to the condition that 2g−2 lies within the range where α-polystable triples of the given kind exist.

5.4 Moduli spaces of triples

Proposition 5.6 shows that in order to study N (a, b) for different values of the Toledo invariant, we need to understand the moduli spaces of triples for values of α that may lie anywhere (including at the extremes α_{m} and α_{M}) in the α-range given in Proposition 5.1. The information we need can be found in [7]. From the results in [7] we get the following for triples of type (n_{1}, n_{2}, d_{1}, d_{2}).

Theorem 5.8 (Theorem A in [7]).

(1) A triple T = (E_{1}, E_{2}, ϕ) of type (n_{1}, n_{2}, d_{1}, d_{2}) is α_{m}-polystable if and only if ϕ = 0 and E_{1} and E_{2} are polystable. We thus have

N_{α_{m}}(n_{1}, n_{2}, d_{1}, d_{2}) ≅ M(n_{1}, d_{1}) × M(n_{2}, d_{2}).

where M(n, d) denotes the moduli space of polystable bundles of rank n and degree d. In particular, N_{α_{m}}(n_{1}, n_{2}, d_{1}, d_{2}) is nonempty and irreducible.

(2) If α > α_{m} is any value such that 2g − 2 ⩽ α (and α < α_{M} if n_{1} ≠ n_{2})) then N_{α}^{s}(n_{1}, n_{2}, d_{1}, d_{2}) is nonempty and irreducible. Moreover:

• If n_{1} = n_{2} = n then N_{α}^{s}(n, n, d_{1}, d_{2}) is birationally equivalent to a ℙ^{N}-fibration over M^{s}(n, d_{2})×Sym^{d_{1}−d_{2}}(X), where M^{s}(n, d_{2}) denotes the subspace of stable bundles of type (n, d_{2}), Sym^{d_{1}−d_{2}}(X) is the symmetric product, and the fiber dimension is N = n(d_{1} − d_{2}) − 1.

• If n_{1} > n_{2} then N_{α}^{s}(n_{1}, n_{2}, d_{1}, d_{2}) is birationally equivalent to a ℙ^{N}-fibration over M^{s}(n_{1} − n_{2}, d_{1} − d_{2}) × M^{s}(n_{2}, d_{2}), where the fiber dimension is N = n_{2}d_{1} − n_{1}d_{2} + n_{2}(n_{1} − n_{2})(g − 1) − 1. The birational equivalence is an isomorphism if GCD(n_{1}−n_{2}, d_{1}−d_{2}) = 1 and GCD(n_{2}, d_{2}) = 1.

• If n_{1} < n_{2} then N_{α}^{s}(n_{1}, n_{2}, d_{1}, d_{2}) is birationally equivalent to a ℙ^{N}-fibration over M^{s}(n_{2} − n_{1}, d_{2} − d_{1}) × M^{s}(n_{1}, d_{1}), where the fiber dimension is N = n_{2}d_{1} − n_{1}d_{2} + n_{1}(n_{2} − n_{1})(g − 1) − 1. The birational equivalence is an isomorphism if GCD(n_{2}−n_{1}, d_{2}−d_{1}) = 1 and GCD(n_{1}, d_{1}) = 1.

In particular, if n_{1} ≠ n_{2} then N_{α}^{s}(n_{1}, n_{2}, d_{1}, d_{2}) is a smooth manifold of dimension (g − 1)(n_{1}^{2} + n_{2}^{2} − n_{1}n_{2}) − n_{1}d_{2} + n_{2}d_{1} + 1.

(3) If n_{1} ≠ n_{2} then N_{α_{M}} (n_{1}, n_{2}, d_{1}, d_{2}) is nonempty and irreducible. Moreover

N_{α_{M}} (n_{1}, n_{2}, d_{1}, d_{2}) (5.15)

≅ \begin{cases}M(n_{2}, d_{2}) × M(n_{1} − n_{2}, d_{1} − d_{2}) & if n_{1} > n_{2} \\ M(n_{1}, d_{1}) × M(n_{2} − n_{1}, d_{2} − d_{1}) & if n_{1} < n_{2}. \end{cases}

Theorem 5.9 (Corollary 8.2 and Theorem 8.10 in [7]). If n_{1} = n_{2} = n then:

(1) If α_{m} = 0, i.e., if d_{1} = d_{2} (= d), then N_{α}(n, n, d, d) ≅ M(n, d) for all α > 0. In particular N_{α}(n, n, d, d) is nonempty and irreducible.

(2) If 0 < d_{1} − d_{2} < α, then N_{α}(n, n, d_{1}, d_{2}) is nonempty and irreducible.

Remark 5.10. Notice that if n_{1} = n_{2} and α_{m} = 0, then N_{α}(n, n, d,

d) ≅ M(n, d) for all α > 0, while N_{0}(n, n, d, d) ≅ M(n, d) × M(n, d). The picture is quite different if we restrict to the stable points in the moduli spaces. In fact there are no stable points in N_{0}(n, n, d, d), i.e., N_{0}^{s}(n, n, d, d) is empty, while N_{α}^{s}(n, n, d, d) ≅ M^{s}(n, d) for α > 0.

Proposition 5.11 (Proposition 2.6 and Lemma 2.7 in [7]).

(1) If α ∈ [α_{m}, α_{M}] is generic and GCD(n_{1}, n_{1} + n_{2}, d_{1} + d_{2}) = 1,

then

N_{α}(n_{1}, n_{2}, d_{1}, d_{2}) = N_{α}^{s}(n_{1}, n_{2}, d_{1}, d_{2}).

In particular, the moduli space N_{α}(n_{1}, n_{2}, d_{1}, d_{2}) is nonempty and irreducible if in addition 2g − 2 ⩽ α.

(2) Let m ∈ ℤ be such that GCD(n_{1} + n_{2}, d_{1} + d_{2} − mn_{1}) = 1. Then α = m is not a critical value and there are no α-independent semistable triples.

6. Main results

We now use the results of Section 5.4, applied to the case α = 2g − 2, to deduce our main results on the moduli spaces of U(p, q)- Higgs bundles, and hence for the representation spaces R(PU(p, q)) and R_{Γ}(U(p, q)) (defined in Section 2). Recall that we identified components of R(PU(p, q)) labeled by [a, b] ∈ ℤ⊕ℤ/(p+q)ℤ, and similarly identified components of R_{Γ}(U(p, q)) labeled by (a, b) ∈ ℤ ⊕ ℤ. Our arguments proceed along the following lines:

• By Proposition 2.5 R_{Γ}(a, b) is a U(1)^{2g}-fibration over R[a, b]. The number of connected components of R_{Γ}(a, b) is thus greater than or equal to that of R[a, b]. In particular, R[a, b] is connected whenever R_{Γ}(a, b) is.

• By Proposition 3.13 there is a homeomorphism between R_{Γ}(a, b) and the moduli space M(a, b) of U(p, q)-Higgs bundles. This restricts to give a homeomorphism between R_{Γ}^{∗}(a, b) and M^{s}(a, b).

• By Proposition 4.3 the number of connected components of M(a, b) is bounded above by the number of connected components in the subspace of local minima for the Bott-Morse function defined in Section 4.1. By Proposition 4.24 the same conclusion holds for M^{s}(a, b).

• By Theorems 4.6 and 5.3 we can identify the subspace of local minima as a moduli space of α-stable triples, with α = 2g − 2.

Summarizing, we have:

|π_{0}(R[a, b])| ⩽ |π_{0}(R_{Γ}(a, b))| = |π_{0}(M(a, b))|
⩽ |π_{0}(N (a, b))| = |π_{0}(N_{2g−2}(n_{1}, n_{2}, d_{1}, d_{2}))|

where |π_{0}(·)| denotes the number of components, and (in the notation of Section 5) the moduli space of triples which appears in the last line is either N_{2g−2}(p, q, a + p(2g − 2), b) (if a/p ⩽ b/q) or N_{2g−2}(q, p, b + q(2g − 2), a). Similarly, we get that

|π_{0}(\overline{R}^{∗}[a, b])| ⩽ |π_{0}(\overline{R}_{Γ}^{∗}(a, b))| = |π_{0}(\overline{M}^{s}(a, b))|
⩽ |π_{0}(\overline{N}^{s}(a, b))| = |π_{0}(\overline{N} _{2g−2}^{s}(n_{1}, n_{2}, d_{1}, d_{2}))|

In particular, if the moduli spaces of triples are connected, then so are the Higgs moduli spaces and the moduli spaces of representations.

6.1 Moduli spaces of Higgs bundles

We begin with results for the U(p, q)-Higgs moduli spaces. Recall from Proposition 3.20 that, whenever the moduli space M^{s}(a, b) of stable U(p, q)-Higgs bundles with invariants (a, b) is nonempty, it is a smooth complex manifold of dimension 1 + (p + q)^{2}(g − 1). We shall refer to this dimension as the expected dimension in the following.

Theorem 6.1. Let (p, q) be any pair of positive integers and let (a, b) ∈ ℤ ⊕ ℤ be such that 0 ⩽ |τ(a, b)| ⩽ τ_{M}.

(1) If either of the following sets of conditions apply, then the moduli space M^{s}(a, b) is a nonempty smooth manifold of the expected dimension, with connected closure \overline{M}^{s}(a, b):

(i) 0 < |τ(a, b)| < τ_{M},

(ii) |τ(a, b)| = τ_{M} and p = q.

(2) If any one of the following sets of conditions apply, then the moduli space M(a, b) is nonempty and connected:

(i) τ(a, b) = 0,

(ii) |τ(a, b)| = τ_{M} and p ≠ q,

(iii) (p − 1)(2g − 2) < |τ| ⩽ τ_{M} = p(2g − 2) and p = q.

Proof. (1) By Proposition 5.6 condition (i) implies that α_{m} < 2g − 2 < α_{M} for the triples corresponding to points in N (a, b). Thus Theorem 5.8(2) (together with Theorem 5.3) implies that N (a, b) is nonempty and connected. Similarly, condition (ii) implies that α_{m} = 0, and we can apply Theorem 5.9(1). The rest follows from Theorem 5.4.

(2) By Proposition 5.6, the conditions in (i) and (ii) are equivalent to α_{m} = 2g − 2 and α_{M} = 2g − 2 respectively. It follows by parts (1) and (3) of Theorem 5.8 (together with Theorem 5.3) that N (a, b) is nonempty and connected. The rest follows from Theorem 5.4.

For (iii), we use the fact that |τ| = |b − a| if p = q. The condition on |τ| is thus equivalent to d_{1} − d_{2} < 2g − 2 for the triples corresponding to points in N (a, b). The result thus follows by Theorem 5.9(2). q.e.d.

Remark 6.2. Combining (1) and (i)-(ii) of (2) in Theorem 6.1, we see that the moduli space M(a, b) is nonempty for all (p, q, a, b) such that 0 ⩽ |τ| ⩽ τ_{M}.

Remark 6.3. In Theorem 3.32 we gave a detailed description for M(a, b) in the case that p ≠ q and |τ(a, b)| = τ_{M}. The description was complete, provided that the space was nonempty. By the previous remark we can now remove this caveat.

In general, the stable locus M^{s}(a, b) is not the full moduli space and the full moduli space M(a, b) is not smooth. Singularities can occur at points representing strictly semistable objects, and these can also account for singularities in N (a, b), the space of local minima (as in Section 5). These types of singularities are prevented by a certain coprimality condition:

Proposition 6.4. Suppose that GCD(p + q, a + b) = 1. Then:

(1) M(a, b) is smooth.

(2) α = 2g −2 is not a critical value for triples of type (p, q, a+p(2g − 2), b) or (q, p, b + q(2g − 2), a).

(3) The moduli spaces N_{2g−2}(p, q, a + p(2g − 2), b) and N_{2g−2}(q, p, b + q(2g − 2), a) are nonempty, smooth and irreducible.

Proof.

(1) This is simply a re-statement of Proposition 3.12.

(2) Apply Proposition 5.11 (2) with (n_{1}, n_{2}, d_{1}, d_{2}) equal to (p, q, a + p(2g − 2), b) or (q, p, b + q(2g − 2), a) and m = 2g − 2.

(3) Since GCD(p+q, a+b) = 1 implies GCD(p, p+q, b+a+q(2g−2)) = 1 (or GCD(q, p + q, b + a + p(2g − 2)) = 1), the result follows from (2) and Proposition 5.11 (1).

q.e.d.

Theorem 6.5. Let (p, q) be any pair of positive integers and let (a, b) be such that 0 ⩽ |τ(a, b)| ⩽ τ_{M}. Suppose also that GCD(p + q, a + b) = 1. Then the moduli space M(a, b) is a (nonempty) smooth, connected manifold of the expected dimension.

Proof. Combine Proposition 6.4 and Theorem 5.4. q.e.d.

Theorems 6.1 plus 6.5 are equivalent to Theorem A in the Introduction.

6.2 Moduli spaces of representations

Using Proposition 3.13 we can translate the results of Section 6.1 into results about the representation spaces R_{Γ}(a, b) and R_{Γ}^{∗}(a, b) (for U(p, q) representations of the surface group Γ). We denote the closure of R_{Γ}^{∗}(a, b) in R_{Γ}(a, b) by \overline{R}_{Γ}^{∗}(a, b).

Theorem 6.6. Let (p, q) be any pair of positive integers and let (a, b) ∈ ℤ ⊕ ℤ be such that 0 ⩽ |τ(a, b)| ⩽ τ_{M}.

(1) The moduli space R_{Γ}(a, b) is nonempty.

(2) If either of the following sets of conditions apply, then the moduli space R_{Γ}^{∗}(a, b) is a nonempty smooth manifold of the expected dimension, with connected closure \overline{R}_{Γ}^{∗}(a, b) in R_{Γ}(a, b):

(i) 0 < |τ(a, b)| < τ_{M}, (ii) |τ(a, b)| = τ_{M} and p = q.

(3) If any one of the following sets of conditions apply, then the moduli space R_{Γ}(a, b) is connected:

(i) τ(a, b) = 0,

(ii) |τ(a, b)| = τ_{M} and p ≠ q, (iii) (p − 1)(2g − 2) < |τ| ⩽ τ_{M} = p(2g − 2) and p = q, (iv) GCD(p + q, a + b) = 1

(4) If GCD(p + q, a + b) = 1 then R_{Γ}(a, b) is a smooth manifold of the expected dimension.

Proof. By Proposition 3.13, this follows from Theorem 6.1 and 6.5. q.e.d.

Theorem 6.7. Let (p, q) be any pair of positive integers such that p ≠ q, and let (a, b) be such that |τ(a, b)| = τ_{M}. Then every representation in R_{Γ}(a, b) is reducible (i.e., R_{Γ}^{∗}(a, b) is empty). If p < q, then every such representation decomposes as a direct sum of a semisimple representation of Γ in U(p, p) with maximal Toledo invariant and a semisimple representation in U(q −p). Thus, if τ = p(2g −2) then there is an isomorphism

R_{Γ}(p, q, a, b) ≅ R_{Γ}(p, p, a, a − p(2g − 2)) × R_{Γ}(q − p, b − a + p(2g − 2)), where the notation R_{Γ}(p, q, a, b) indicates the moduli space of representations of Γ in U(p, q) with invariants (a, b), and R_{Γ}(n, d) denotes the moduli space of degree d representations of Γ in U(n).

(A similar result holds if p > q, as well as if τ = −p(2g − 2).)

Proof. Proposition 3.13 and Theorem 3.32. q.e.d.

As observed in Section 2.2 (cf. (2.7)), the spaces R(a) = R_{Γ}(a, −a) can be identified with components of R(U(p, q)), i.e., with components of the moduli space for representations of π_{1}X in U(p, q). Applying Theorems 6.6 and 6.7, together with the observation that τ(a, −a) = 2a in the special case where b = −a, we thus obtain the following results for R(U(p, q)). Notice that the condition GCD(p + q, a + b) = 1 is never satisfied if a + b = 0.

Theorem 6.8. Let (p, q) be any pair of positive integers and let a ∈ ℤ ⊕ ℤ be such that |a| ⩽ min{p, q}(g − 1).

(1) The moduli space R_{Γ}(a) is nonempty.

(2) If either of the following sets of conditions apply, then the moduli space R^{∗}(a) is a nonempty, smooth manifold of the expected dimension, with connected closure \overline{R}^{∗}(a) in R(a):

(i) 0 < |a| < min{p, q}(g − 1), or

(ii) |a| = p(g − 1) and p = q.

(3) If any one of the following sets of conditions apply, then the moduli space R(a) is connected:

(i) a = 0,

(ii) |a| = min{p, q}(g − 1) and p ≠ q,

(iii) (p − 1)(g − 1) < |a| ⩽ p(g − 1) and p = q.

Theorem 6.9. Let (p, q) be any pair of positive integers such that p ≠ q. If |a| = min{p, q}(g − 1) then R^{∗}(a) is empty and every representation in R(a) is reducible. If p < q, then every such representation decomposes as a direct sum of a semisimple representation of Γ in U(p, p) with maximal Toledo invariant and a semisimple representation in U(q − p). Thus, if a = p(g − 1) then there is an isomorphism

R(a) ≅ R_{Γ}(p, p, a, a − p(2g − 2)) × R_{Γ}(q − p, p(2g − 2)),

where the notation R_{Γ}(p, q, a, b) indicates the moduli space of representations of Γ in U(p, q) with invariants (a, b), and R_{Γ}(n, d) denotes the moduli space of degree d representations of Γ in U(n).

(A similar result holds if p > q, as well as if a = −p(g − 1).)

From Theorem 6.6 and Proposition 2.5 we obtain the following theorem about the moduli spaces for PU(p, q) representations of π_{1}X. Note that the closure \overline{R}^{∗}[a, b] in R[a, b] is the image of \overline{R}_{Γ}^{∗}(a, b) under the map of Proposition 2.5, hence these two spaces have the same number of connected components.

Theorem 6.10. Let (p, q) be any pair of positive integers and let (a, b) ∈ ℤ ⊕ ℤ be such that 0 ⩽ |τ(a, b)| ⩽ τ_{M}.

(1) The moduli space R[a, b] is nonempty.

(2) If either of the following sets of conditions apply, then the moduli space R^{∗}[a, b] is a nonempty smooth manifold of the expected dimension, with connected closure \overline{R}^{∗}[a, b] in R[a, b]:

(i) 0 < |τ(a, b)| < τ_{M}, or (ii) |τ(a, b)| = τ_{M} and p = q.

(3) If any one of the following sets of conditions apply, then the moduli space R[a, b] of all semi-simple representations is connected:

(i) τ(a, b) = 0,

(ii) |τ(a, b)| = τ_{M} and p ≠ q, (iii) (p − 1)(2g − 2) < |τ| ⩽ τ_{M} = p(2g − 2) and p = q, (iv) GCD(p + q, a + b) = 1.

Theorem 6.11. Let (p, q) be any pair of positive integers such that p ≠ q, and let (a, b) be such that |τ(a, b)| = τ_{M}. Then R^{∗}[a, b] is empty. If p < q, then every such representation reduces to a semisimple representation of π_{1}X in P(U(p, p) × U(q − p)), such that the PU(p, p) representation induced via projection on the first factor has maximal Toledo invariant. (A similar result holds if p > q.)

Remark 6.12. As explained by Hitchin in [22, Section 5], the moduli space of irreducible representations in the adjoint form of a Lie group is liable to acquire singularities, because of the existence of stable vector bundles which are fixed under the action of tensoring by a finite order line bundle. For this reason we do not make any smoothness statements in Theorem 6.10.

6.3 Total number of components and coprimality conditions

We end with some elementary observations about the total number of components in the decomposition R(PU(p, q)) = ⋃_{(a,b)} R[a, b], and about the number of such components for which the coprime condition GCD(p+q, a+b) = 1 apply. We begin with the number of components.

By definition, τ(a, b) takes values in \frac{2}{n}ℤ, where n = p + q.

Proposition 6.13. Suppose that GCD(p, q) = k. Then the map

τ: ℤ ⊕ ℤ/(p, q)ℤ ⟶ \frac{2}{n}ℤ
[a, b] ⟼ \frac{2}{n}(aq − bp)

fits in an exact sequence

0 ⟶ ℤ/kℤ ⟶^{σ} ℤ ⊕ ℤ/(p, q)ℤ ⟶^{τ} \frac{2k}{n} ℤ ⟶ 0

where the map σ is [t] ↦ [t\frac{p}{k}, t\frac{q}{k}]. In particular, τ is a k : 1 map onto the subset \frac{2k}{n}ℤ ⊂ \frac{2}{n}ℤ.

Proof. The map σ is clearly injective, and τ ∘ σ = 0. To see that ker(τ) = im(σ), observe that if τ[a, b] = 0 then either a = b = 0 or \frac{a}{b} = \frac{p}{q}, i.e., [a, b] = [t\frac{p}{k}, t\frac{q}{k}] for some t ∈ ℤ. Finally, if a_{0}q −b_{0}p = k, then for any l ∈ ℤ we have τ[la_{0}, lb_{0}] = \frac{2k}{n}l. Thus τ is surjective onto \frac{2k}{n}ℤ.

q.e.d.

Remark 6.14. Proposition 6.13 shows why^{10} we must use [a, b] rather than τ to label the components of R(PU(p, q)) or of R_{Γ}(U(p, q)).

Definition 6.15. Suppose that GCD(p, q) = k. Define

C = τ^{−1}([−τ_{M}, τ_{M}] ∩ \frac{2k}{n} ℤ), (6.1)

where τ is the map defined in Proposition 6.13.

The following is then an immediate corollary of Proposition 6.13.

10Unless p and q are coprime, in which case there is a bijective correspondence between [a, b] and τ.

Corollary 6.16. Suppose that GCD(p, q) = k and C is as above. Then C is precisely the set of all the points in ℤ ⊕ ℤ/(p, q)ℤ which label components R[a, b] in R(PU(p, q)). The cardinality of C is

|C| = 2n min{p, q}(g − 1) + k

= |([−τ_{M}, τ_{M}] ∩ \frac{2}{n}ℤ)| + GCD(p, q) − 1.

Proof. The first statement is a direct consequence of Proposition 6.13 and the bound on τ. Suppose for definiteness that min{p, q} = p. Then since τ_{M} = 2 min\{p, q\}(g − 1) = \frac{2k}{n}(n\frac{p}{k}(g − 1)) ∈ \frac{2k}{n}ℤ, the number of points in [−τ_{M}, τ_{M}] ∩ \frac{2k}{n}ℤ is 2n\frac{p}{k}(g − 1) + 1. The second statement now follows from the fact that τ is a k : 1 map. The proof is similar if min{p, q} = q. q.e.d.

Finally, we examine the coprime condition GCD(p+q, a+b) = 1. We regard p and q as fixed, but allow [a, b] to vary. The coprime condition GCD(p + q, a + b) = 1 can thus be satisfied on some components but not on others.

Definition 6.17. Fix p and q and let C ⊂ ℤ ⊕ ℤ/(p + q)ℤ be as in Definition 6.15. Define C_{1} to be the subset of classes [a, b] ∈ C for which the condition GCD(p + q, a + b) = 1 is satisfied.

Proposition 6.18. Fix p and q and let C and C_{1} be as above. Both C_{1} and its complement in C are nonempty.

Proof. If a = p and b = q − 1 then GCD(p + q, a + b) = 1. Also, τ(p, q −1) = \frac{2p}{p+q}, which is in [−τ_{M}, τ_{M}]∩ \frac{2k}{n}ℤ. Thus [p, q −1] is in C_{1}. It is similarly straightforward to see that (a, b) = (0, 0) defines an element in C − C_{1}, as does (a, b) = (p, −p) if p ⩽ q or (a, b) = (q, −q) if q ⩽ p.

q.e.d.

It seems somewhat complicated to go beyond this result and completely enumerate the elements in C_{1}. The following result is, however, a step in that direction.

Definition 6.19. Let Ω ⊂ ℝ⊕ℝ be the region depicted in Figure 2, i.e., the region bounded by (i) the ray b = q and a ⩽ p, (ii) the ray a = p and b ⩽ q, (iii) the ray a = 0 and b ⩽ 0, (iv) the ray b = 0 and a ⩽ 0,

(v) the line aq − bp = \frac{n}{2}τ_{M}, and (vi) the line aq − bp = −\frac{n}{2}τ_{M}, and including all the boundary lines except the first two rays. Let Ω_{ℤ} be the set of integer points in Ω, i.e. Ω_{ℤ} = Ω ⋂ ℤ ⊕ ℤ. We refer to Ω as the fundamental region for (p, q) (see Figure 2). Then Ω_{ℤ} is the integer lattice inside the fundamental region.

[figure: a; −\frac{p+q}{2q} τ_{M}; 0; p; b; −\frac{p+q}{2p} τ_{M}; 0; q; τ = 0; τ = τ_{M}; τ = −τ_{M}; d = q; d = 0; d = −\frac{p+q}{2q} τ_{M}]

Figure 2: Fundamental region for (a, b). Components of R(PU(p, q)) correspond to the integer points in this region. Illustrative lines of constant τ (at τ = −τ_{M}, 0, τ_{M}) and lines of constant d (at d = −\frac{n}{2q} τ_{M}, 0, q) are shown.

Proposition 6.20. Suppose that p and q are integers with GCD(p, q) = k.

(1) There is a bijection between C and Ω_{ℤ}.

(2) If (a, b) lies in Ω_{ℤ} then d = a + b satisfies the bounds

−n(g − 1) ⩽ d < n. (6.2)

All values of d in this range occur.

(3) Let l_{t} denote the line aq − bp = tk. Then the points on l_{t} ⋂ Ω_{ℤ} define the locus of points (a, b) for which τ(a, b) = t\frac{2k}{n}.

(4) The line l_{t} intersects Ω_{ℤ} for −\frac{n}{2k}τ_{M} ⩽ t ⩽ \frac{n}{2k}τ_{M}. For each integer t in this range, there are k points on l_{t} ⋂ Ω_{ℤ}.

(5) For a fixed t, GCD(a+b, \frac{n}{k}) is the same for all integer points (a, b) on l_{t} ⋂ Ω_{ℤ}.

(6) Fix t and let (a, b) be any point in the set l_{t} ⋂ Ω_{ℤ}. If GCD(a + b, \frac{n}{k}) ≠ 1 then GCD(a′ + b′, n) ≠ 1 for all (a′, b′) ∈ l_{t} ⋂ Ω_{ℤ}.

Proof. (1) Suppose first that \frac{a}{p} ⩾ \frac{b}{q}. Pick l such that 0 ⩽ a + lp ⩽ p. Then b + lq ⩽ q, so that (a + lp, b + lq) is in the fundamental region. Similarly, if \frac{a}{p} ⩽ \frac{b}{q} then we pick l such that 0 ⩽ b + lq ⩽ q and see that a + lp ⩽ p. In this way we get a well-defined map from C to the fundamental region. The map is clearly injective. To see that it is surjective, notice that the boundary lines aq − bp = ±\frac{n}{2}τ_{M} correspond to the conditions τ = ±τ_{M}.

(2) This is clear from a sketch of the fundamental region (see Figure 2), in which the loci of points with constant value of d = a + b are straight lines of slope −1. The maximal value for d corresponds to the line passing through the top right corner of the region, i.e., through (p, q). Thus d_{max} = p + q = n. The minimal value for d corresponds either to the line passing through (−\frac{n}{2q} τ_{M}, 0) or to the line through (0, −\frac{n}{2p}τ_{M}), depending on which yields the smaller value for d. Since τ_{M} = 2 min{p, q}(g − 1), we find in all cases that d_{min} = −n(g − 1). It is straightforward to see that all intermediate values for d occur.

(3)-(4) This is simply a restatement of Proposition 6.13.

(5)-(6) Both follow from the fact that for any two points (a, b) and (a′, b′) on l_{t}, we get d′ = d + s\frac{n}{k} for some s ∈ ℤ. q.e.d.

Remark 6.21. Part (6) says that for fixed t, if GCD(a + b, \frac{n}{k}) ≠ 1 for any point (a, b) ∈ l_{t} ⋂ Ω_{ℤ}, then GCD(a + b, n) ≠ 1 for all points (a, b) ∈ l_{t} ⋂ Ω_{ℤ}. That is, we can detect the non-coprimality of (a + b, n) for all (a, b) ∈ l_{t} ⋂ Ω_{ℤ} by checking the non-coprimality of (a + b, \frac{n}{k}) at any one (a, b) ∈ l_{t} ⋂ Ω_{ℤ}. We cannot however check for coprimality in the same way. If GCD(a + b, \frac{n}{k}) = 1, it is possible that GCD(a′ + b′, n) ≠ 1 for some (a′, b′) ∈ l_{t} ⋂ Ω_{ℤ}. For example, take p = 2, q = 4, a = −1, b = 0, a′ = 0, b′ = 2, and t = −2. Then GCD(a′ + b′, n) = 2 while GCD(a + b, \frac{n}{k}) = 1.

References

[1] M.F. Atiyah & R. Bott, The Yang-Mills equations over Riemann surfaces, Philos. Trans. Roy. Soc. London Ser. A 308 (1983) 523–615, MR 85k:14006 Zbl 0509.14014.

[2] D. Banfield, Stable pairs and principal bundles, Q. J. Math. 51 (2000) 417–436, MR 2001k:53036, Zbl 0979.53028.

[3] I. Biswas & S. Ramanan, An infinitesimal study of the moduli of Hitchin pairs, J. London Math. Soc. (2) 49 (1994) 219–231, MR 94k:14006, Zbl 0819.58007.

[4] S.B. Bradlow & O. García-Prada, Stable triples, equivariant bundles and dimensional reduction, Math. Ann. 304 (1996) 225–252, MR 96m:32033, Zbl 0852.32016.

[5] S.B. Bradlow, O. García-Prada & P.B. Gothen, Representations of the fundamental group of a surface in PU(p, q) and holomorphic triples, C.R. Acad. Sci. Paris Sér. I Math. 333 (2001) 347–352, MR 2002q:14013, Zbl 0991.32012.

[6] S.B. Bradlow, O. García-Prada & P.B. Gothen, Surface group representations, Higgs bundles and holomorphic triples, preprint, math.AG/0206012.

[7] S.B. Bradlow, O. García-Prada & P.B. Gothen, Moduli spaces of holomorphic triples over compact Riemann surfaces, math.AG/0211428, to appear in Math. Ann.

[8] S.B. Bradlow, O. García-Prada & I. Mundet i Rierra, Relative Hitchin–Kobayashi correspondences for principal pairs, Quart. J. Math. 54 (2003) 171-208.

[9] M. Burger & A. Iozzi, Bounded Kahler class rigidity of actions on Hermitian symmetric spaces, preprint, 2002.

[10] K. Corlette, Flat G-bundles with canonical metrics, J. Differential Geom. 28 (1988) 361–382, MR 89k:58066, Zbl 0676.58007.

[11] A. Domic & D. Toledo, The Gromov norm of the Kaehler class of symmetric domains, Math. Ann. 276 (1987) 425–432, MR 88e:32057, Zbl 0595.53061.

[12] S.K. Donaldson, Twisted harmonic maps and the self-duality equations, Proc. London Math. Soc. (3) 55 (1987) 127–131, MR 88q:58040, Zbl 0634.53046.

[13] J.L. Dupont, Bounds for characteristic numbers of flat bundles in ‘Algebraic topology’, J.L. Dupont & I. Madsen (eds.), Aarhus 1978, Lecture Notes in Mathematics, 763, 109–119, Berlin, Springer, 1978, MR 81f:57021, Zbl 0511.57018.

[14] T. Frankel, Fixed points and torsion on Kähler manifolds, Ann. of Math. (2) 70 (1959) 1–8, MR 24 #A1730, Zbl 0088.38002

[15] O. García-Prada, Dimensional reduction of stable bundles, vortices and stable pairs, Int. J. Math. 5 (1994) 1–52, MR 95d:32035, Zbl 0799.32022.

[16] W.M. Goldman, Representations of fundamental groups of surfaces, in ‘Geometry and topology’, Proceedings, J. Alexander & J. Harer (eds.), University of Maryland 1983–1984, Lecture Notes in Mathematics, 1167, 95–117. Berlin-Heidelberg-New York, Springer, 1985, MR 87j:32068, Zbl 0575.57027.

[17] W.M. Goldman, Topological components of spaces of representations, Invent. Math. 93 (1988) 557–607, MR 89m:57001, Zbl 0655.57019.

[18] P.B. Gothen, The topology of Higgs bundle moduli spaces, Ph.D. Thesis, Mathematics Institute, University of Warwick, 1995.

[19] P.B. Gothen, Components of spaces of representations and stable triples, Topology 40 (2001) 823–850, MR 2002k:14017.

[20] P.B. Gothen, Topology of U(2, 1) representation spaces, Bull. London Math. Soc. 34 (2002) 729–738, MR 2003f:14036.

[21] L. Hernández, Maximal representations of surface groups in bounded symmetric domains, Transactions Amer. Math. Soc. 324 (1991) 405–420, MR 91f:32040, Zbl 0733.32024.

[22] N.J. Hitchin, The self-duality equations on a Riemann surface, Proc. London Math. Soc. 55 (1987) 59–126, MR 89a:32021, Zbl 0634.53045.

[23] N.J. Hitchin, Lie groups and Teichmüller space, Topology 31 (1992) 449–473, MR 93e:32023, Zbl 0769.32008.

[24] E. Markman & E.Z. Xia, The moduli of flat PU(p, p) structures with large Toledo invariants, Math. Z. 240 (2002) 95–109, MR 2003i:14009, Zbl 1008.32006.

[25] I. Mundet i Riera, A Hitchin–Kobayashi correspondence for Kahler fibrations, J. Reine Angew. Math. 528 (2000) 41–80, MR 2002b:53035, Zbl 1002.53057.

[26] M.S. Narasimhan & C.S. Seshadri, Stable and unitary bundles on a compact Riemann surface, Ann. of Math., 82, 540–564 (1965), MR 30 #588, Zbl 0171.04803.

[27] N. Nitsure, Moduli spaces of semistable pairs on a curve, Proc. London Math. Soc. 62 (1991) 275–300, MR 92a:14010, Zbl 0733.14005.

[28] M. Schlessinger, Functors on Artin rings, Trans. Amer. Math. Soc. 130 (1968) 208–222, MR 36 #184, Zbl 0167.49503.

[29] C.T. Simpson, Constructing variations of Hodge structure using Yang-Mills theory and applications to uniformization, J. Amer. Math. Soc. 1 (1988) 867–918, MR 90e:58026, Zbl 0669.58008.

[30] C.T. Simpson, Higgs bundles and local systems, Inst. Hautes Études Sci. Publ. Math. 75 (1992) 5–95, MR 94d:32027, Zbl 0814.32003.

[31] C.T. Simpson, Moduli of representations of the fundamental group of a smooth projective variety I, Inst. Hautes Études Sci. Publ. Math. 79 (1994) 47–129 MR 96e:14012, Zbl 0891.14005.

[32] C.T. Simpson, Moduli of representations of the fundamental group of a smooth projective variety II, Inst. Hautes Études Sci. Publ. Math. 80 (1994) 5–79, MR 96e:14013, Zbl 0891.14006.

[33] D. Toledo, Representations of surface groups in complex hyperbolic space, J. Differential Geometry 29 (1989) 125–133, MR 90a:57016, Zbl 0676.57012.

[34] E.Z. Xia, Components of Hom(π_{1}, PGL(2, R)), Topology 36 (1997) 481–499, MR 97j:57002, Zbl 0872.57004.

[35] E.Z. Xia, The moduli of flat U(p, 1) structures on Riemann surfaces, preprint, 1999, math.AG/9910037.

[36] E.Z. Xia, The moduli of flat PU(2, 1) structures on Riemann surfaces, Pacific Journal of Mathematics 195 (2000) 231–256, MR 2001q:32033.

Department of Mathematics

University of Illinois Urbana, IL 61801

Instituto de Matemáticas y Física Fundamental Consejo Superior de Investigaciones Científicas

28006 Madrid, Spain

Departamento de Matemática Pura

Universidade do Porto, 4169-007 Porto, Portugal