Klyachko Models for Ladder Representations

Arnab Mitra, Omer Offen, Eitan Sayag^{1}

Received: June 7, 2016

Communicated by Dan Ciubotaru

Abstract. We give a new proof for the existence of Klyachko models for unitary representations of GL_{n}(F) over a non-archimedean local field F. Our methods are purely local and are based on studying distinction within the class of ladder representations introduced by Lapid and Mínguez. We classify those ladder representations that are distinguished with respect to Klyachko models. We prove the hereditary property of these models for induced representations from arbitrary finite length representations. Finally, in the other direction and in the context of admissible representations induced from ladder, we study the relation between distinction of the parabolic induction with respect to the symplectic groups and distinction of the inducing data.

Contents

1. Introduction  612
2. Notation and preliminaries  619
3. Non-degenerate skew-symmetric matrices and parabolic orbits  623
4. The geometric lemma  627
5. First applications of the geometric lemma to Sp-distinction  628
6. Representations of GL_{n}(F)  631
7. A necessary condition for Sp-distinction of Z(𝔪)  633
8. Multi-sets of segments  636
9. On Sp-distinction of standard modules  639
10. The Sp-distinguished ladder representations  641
11. On sums of ladders of Speh type  644
12. On distinction of representations induced from ladder  646

1Arnab Mitra, partially supported by postdoctoral fellowships funded by the Skirball Foundation via the Center for Advanced Studies in Mathematics at Ben-Gurion University of the Negev and the Department of Mathematics, Technion. Omer Offen, partially supported by ISF grant No. 1394/12Eitan Sayag, partially supported by ISF grant No. 1138/10.

13. On distinction by Klyachko subgroups  650
14. Klyachko models for ladder representations  653
References  655

1. Introduction

Let G be a totally disconnected locally compact group and H a closed subgroup. A smooth, complex valued representation (π, V) of G is called H-distinguished if there exists a non-zero linear form ℓ on V such that ℓ(π(h)v) = ℓ(v) for all h ∈ H and v ∈ V. If π is irreducible, then such a linear form realizes π in a space of functions on G, to wit,

π ≃ {g ↦ ℓ(π(g^{−1})v) : v ∈ π} ⊆ C^{∞}(G/H).

The class of H-distinguished representations plays an important role in the harmonic analysis of the homogeneous space G/H (see [Ber88] for instance). Furthermore, distinguished representations are crucial for the global theory of period integrals of automorphic forms, have applications to the study of special values of L-functions and to the description of the image of functorial lifts in the Langlands program. This paper continues the study of [OS08a] of distinguished representations of GL_{n} over a non-archimedean local field F with respect to Klyachko subgroups. A Klyachko model is an induced representation Ind_{H_{2k,r}}^{G}(ψ) of G = GL_{n}(F). Here, n = 2k + r, H_{2k,r} is the subgroup of G consisting of matrices of the form ( \begin{matrix}h & X \\ 0 & u \end{matrix} ) where h ∈ Sp_{2k}(F), X ∈ M_{2k×r}(F), u is an upper-triangular unipotent matrix, and ψ is the character of H_{2k,r} trivially extending a non-degenerate character on the upper-triangular unipotent matrices in GL_{r}(F). For a fixed n, as k varies, these models ‘interpolate’ between the well known Whittaker model (the case k = 0) and, if n is even, the symplectic model (the case k = n/2). When F is a finite field, they were introduced by Klyachko in [Kly83]. Together they form a complete model in that case ([IS91]), that is, they satisfy

⊕_{k=0}^{⌊n/2⌋} Ind_{H_{2k,r}}^{G}(ψ) = ⊕_{π∈Ĝ}π.

Klyachko models over non-archimedean local fields were first studied in [HR90] where it was observed, among other things, that some irreducible representations do not imbed into a Klyachko model. In [OS08b] it is proved that the sum ⊕_{k=0}^{⌊n/2⌋} Ind_{H_{2k,r}}^{G}(ψ) is multiplicity free. We shall say that an irreducible representation π admits a Klyachko model if it can be embedded into ⊕_{k=0}^{⌊n/2⌋} Ind_{H_{2k,r}}^{G}(ψ). The main result in [OS08a] prescribes a Klyachko model for any irreducible representation in the unitary dual of GL_{n}(F) based on Tadić classification. This was achieved by first prescribing a Klyachko model to any Speh representation. For a general unitary representation, a hereditary property of Klyachko models for representations parabolically induced from the Speh representations is applied. Recently, Lapid and Mínguez introduced a class of irreducible representations of GL_{n} over a non-archimedean local field [LM14]. Inspired by their presentation in the Zelevinsky classification scheme, they called them ladder representations. (See §10.1.1 for the definition). The class of ladder representations contains the Speh representations, the building blocks of the unitary dual (see Tadić classification of the unitary dual of GL_{n}(F) [Tad86]). Thus any irreducible unitarizable representation of GL_{n}(F) is a product of some ladder representations. In [OS09] we found a connection between the Klyachko model and a partition naturally obtained from the Langlands parameter of a representation. Inspired by this relation, we hope to extend our study of Klyachko models to the entire admissible dual. The present paper provides a collection of results regarding distinction of representations of finite length with respect to the Klyachko groups. In particular, in Theorem 1.3 below we classify the distinguished ladder representations in the context of Klyachko models over a non-archimedean local field F of characteristic different than 2. The special case, when G = GL_{2n}(F) and H = Sp_{2n}(F) is described in Theorem 1.2 below. These results, together with the hereditary property established in Thereom 1.1 below, recovers, using only local methods, our recipe for the Klyachko model of any representation in the unitary dual of a general linear group. To help understand the motivation for our results and techniques, we mention two general strategies that one could employ to approach the problem of classifying distinguished admissible representations in the context of a reductive p-adic group. The first strategy is based on Langlands classification, the second based on the notion of imprimitive representations. We start with the strategy based on the Langlands classification and the notion of standard modules. The smooth dual of G was classified by Langlands in terms of tempered representations of Levi subgroups: Every irreducible smooth representation of G is the unique irreducible quotient of a unique standard module. Clearly, every non-zero H-invariant linear form on an irreducible representation produces such a linear form on its standard module. A possible strategy for classifying H-distinguished representations is based on the Langlands classification:

(1) Classify all H-distinguished standard modules;

(2) Determine if an H-invariant linear form on the standard module descends to one on the irreducible quotient.

To implement the first part of this strategy one can use the geometric lemma of Bernstein and Zelevinsky to analyze distinction of induced representations. We refer to [FLO12] and [Gur15] for cases where a complete classification of distinguished standard modules was achieved.

Implementing the second step turns out to be subtler. An H-invariant linear form on a standard module will induce such a linear form on some irreducible component. To determine whether the irreducible quotient admits an H-invariant linear form is equivalent to determining whether one of the H-invariant linear forms on the standard module descends to the irreducible quotient. This problem can be approached by studying the distinction properties of the maximal proper submodule. However, in general, not enough is known about its structure. The second strategy is based on the concept of imprimitive representations. An irreducible representation of G is called imprimitive if it is not parabolically induced from any proper parabolic subgroup. Any irreducible representation is induced from an imprimitive one. Thus an approach to the classification problem of H-distinction on the smooth dual of G could be:

(1) Classify all H-distinguished imprimitive representations

(2) Determine the relation between H-distinction and parabolic induction. We now focus on the case where G = GL_{n}(F) or a product of general linear groups. In this case, the second step might be more accessible. We further propose to carry this step in two stages:

• Hereditary property: Showing that H-distinction is compatible with parabolic induction

• Purity lemma: Showing that an H-distinguished representation that is induced from a parabolic subgroup, must be induced from a distinguished representation of the Levi subgroup of that parabolic.

The second strategy is problematic even for GL_{n} as the classification of imprimitive representations is an open problem. Nevertheless, within the class of ladder representations, the imprimitive representations are easy to describe and one of the contributions of this work is to pursue this second approach for the problem of distinction with respect to Klyachko models. Moreover, since the maximal proper subrepresentation of the standard module associated to a ladder representation has a particularly simple description we can implement the steps in the first strategy for these representations. This makes distinction problems for the class of irreducible representations parabolically induced from the ladder representations more accessible.

1.1. Main results. To simplify our exposition we omit from the introduction, the results whose formulation will require heavy notation. The interested reader should look in the body of the paper for further results of interest. We state our main results in the form of Theorem 1.1, Theorem 1.2 and Theorem 1.3. Additionally we will formulate a conditional Theorem 1.4. Theorem 1.1 concerns the distinction of representations of finite length with respect to Klyachko subgroups while Theorem 1.2 (resp. Theorem 1.3) provides a classification of the distinguished ladder representations with respect to the symplectic (resp. general Klyachko) subgroup. Theorem 1.4, conditional on a certain combinatorial assumption (Hypothesis 8.5), provides a complete classification of representations induced from ladder representations that are distinguished with respect to the symplectic group. Furthermore, assuming Hypothesis 8.5, we provide a necessary consition for a standard module to be distinguished by the symplectic group. For the sake of notational simplification let us say that a smooth finite length representation π of GL_{n}(F) is Sp-distinguished if n is even and π is Sp_{n}(F)- distinguished. We also require some of the notation and beautiful results of Zelevinsky [Zel80]. We recall that for an irreducible cuspidal representation ρ of GL_{n}(F) and integers a ≤ b one considers the segment

Δ = Δ_{ρ} = [ν^{a}ρ, ν^{b}ρ] = {ν^{i}ρ : i = a, …, b},

where ν(g) = | det(g)| for g ∈ GL_{n}(F). We set b(Δ) = a for the beginning, e(Δ) = b for the end and ℓ(Δ) = b − a + 1 for the length of Δ. To Δ one associates a representation Z(Δ) and a representation L(Δ) as follows: Z(Δ) is the unique irreducible subrepresentation while L(Δ) is the unique irreducible quotient of the Bernstein-Zelevinsky product ν^{a}ρ × ⋯ × ν^{b}ρ. We remark that the L(Δ)’s are the essentially square integrable representations. To a multi-set 𝔪 = {Δ_{1}, …, Δ_{t}} (a set with possible repetitions) of segments of irreducible cuspidal representations one associates an irreducible representation Z(𝔪) and an irreducible representation L(𝔪) as follows (see §6.1): Z(𝔪) is the unique irreducible submodule of the product Z(Δ_{1}) × Z(Δ_{2}) × ⋯ × Z(Δ_{t}) where we have arranged the segments Δ ∈ 𝔪 in a standard form (see §6.1.6). Analogously, the representation L(𝔪) is the unique irreducible quotient of the standard module λ(𝔪) = L(Δ_{1}) × L(Δ_{2}) × ⋯ × L(Δ_{t}). The Zelevinsky classification implies that the map 𝔪 ↦ Z(𝔪) is a bijection between the set of such multi-sets of segments and the disjoint union of admissible duals of GL_{n}(F) for all n, while the Langlands classification implies that the map 𝔪 ↦ L(𝔪) is a bijection between these sets.

Theorem 1.1. (1) A necessary condition for Sp-distinction (see Proposition 7.5): If Z(𝔪) is Sp-distinguished then ℓ(Δ) is even for all Δ ∈ 𝔪.

(2) Hereditary property for Klyachko models (see Proposition 13.3): Let π_{i} be representations of finite length and n_{i} = 2k_{i} + r_{i} be such that π_{i} is (H_{2k_{i},r_{i}}, ψ)-distinguished for i = 1, …, t. Then π = π_{1} × ⋯ × π_{t} is (H_{2k,r}, ψ)-distinguished where k = k_{1} + ⋯ + k_{t} and r = r_{1} + ⋯ + r_{t}.

(3) Reduction to cuspidal lines (see Proposition 13.4): Let π_{i} be representations of finite length, i = 1, …, t, such that their cuspidal supports, Supp(π_{i}) and Supp(π_{j}) are totally disjoint for all i ≠ j (see §2.3.3). Then π = π_{1} × ⋯ × π_{t} admits a Klyachko model if and only if π_{i} admits a Klyachko model for all i = 1, …, t.

The Zelevinsky classification implies that any irreducible representation π of GL_{n}(F) can be written as a product π = π_{1} × ⋯ × π_{t} where the cuspidal support Supp(π_{i}) is contained in a cuspidal line and those cuspidal lines are disjoint for i ≠ j. This is sometimes called a decomposition of π into a product of irreducible rigid representations. Now, using Theorem 1.1 (3), the study of distinction with respect to the Klyachko groups is reduced to the study of distinction within the class of rigid irreducible representations. We say that a rigid irreducible representation π = L(𝔪) is a ladder representation if the multi-set of segments 𝔪 = {Δ_{1}, …, Δ_{t}} satisfies the conditions b(Δ_{1}) > ⋯ > b(Δ_{t}) and e(Δ_{1}) > ⋯ > e(Δ_{t}). The next theorems provide the classification of distinguished ladder representations in terms of Langlands classification. We begin with Sp-distinguished representations since the result is easier to formulate.

Theorem 1.2. Sp-distinguished ladder representations (see Theorem 10.3): Let L(𝔪) be a ladder representation with 𝔪 = {Δ_{1}, …, Δ_{t}}. Then L(𝔪) is Sp-distinguished if and only if, t is even and Δ_{2i−1} = νΔ_{2i} for all i = 1, …, t/2.

The condition on 𝔪 in Theorem 1.2 is equivalent to the existence of a multi-set of segments 𝔫 such that 𝔪 = 𝔫 + ν𝔫. We call such 𝔪 a multi-set of Speh type (see §8.0.7). For the next theorem we need the notion of right-alignment (see §14.2). For segments Δ = [ν^{a}ρ, ν^{b}ρ] and Δ′ = [ν^{a′}ρ, ν^{b′}ρ] we say that Δ′ is right-aligned with Δ and write Δ′ ⊢ Δ if a ≥ a′+1 and b = b′+1. When ρ is a representation of GL_{d}(F) we label this relation by the integer r = d(a − a′ − 1) and write Δ′ ⊢_{r} Δ. Our description of ladder representations distinguished with respect to Klyachko groups will be given in two steps. We will say that a ladder representation π = L(𝔪) is a proper ladder if for all i = 1, …, t−1 we have e(Δ_{i+1}) ≥ b(Δ_{i})−1. The proper ladder representations are imprimitive and every ladder representation is a product of proper ladders in an essentially unique way. This decomposition into proper ladders is explicit in terms of the underlying multi-set of segments associated to the ladder representation.

Theorem 1.3. Ladder representations distinguished with respect to Klyachko groups (see Proposition 14.5 and Theorem 14.7):

(1) Let L(𝔪) be a proper ladder representation of GL_{n}(F) with 𝔪 =

{Δ_{1}, …, Δ_{t}} and let n = 2k + r.

• If t is even then the representation L(𝔪) is (H_{2k,r}, ψ)-distinguished if and only if Δ_{t−2i} ⊢_{r_{i}} Δ_{t−2i−1} for some r_{i} (i = 0, …, t/2 − 1) and r = r_{0} + ⋯ + r_{t/2−1}.

• If t is odd, let s be such that L(Δ_{1}) is an irreducible representation of GL_{s}(F). The representation L(𝔪) is (H_{2k,r}, ψ)-distinguished if and only if Δ_{t−2i} ⊢_{r_{i}} Δ_{t−2i−1} for some r_{i} (i = 0, …, (t − 3)/2) and r = r_{0} + ⋯ + r_{(t−3)/2} + s.

(2) Let π = π_{1} × ⋯ × π_{t} be the decomposition of the ladder representation π into proper ladder representations π_{i}, i = 1, …, t. Then π admits a Klyachko model if and only if the proper ladder representations π_{i} admit Klyachko models for all i = 1, …, t.

Our last main result contains in its formulation a certain combinatorial property of multi-sets of segments that we call Hypothesis 8.5 (see §8). Roughly speaking, it says that the restrictions imposed by the geometric lemma on Spdistinction of a standard module λ(𝔪) imply that 𝔪 is of Speh type. For more details see §1.2.

Theorem 1.4. (1) On Sp-distinguished standard modules (see Proposition 9.1): Suppose λ(𝔪) is rigid and Sp-distinguished. Assume further that 𝔪 satisfies Hypothesis 8.5. Then 𝔪 is of Speh type. In particular, if L(𝔪) is Sp-distinguished and 𝔪 satisfies Hypothesis 8.5 then 𝔪 is of Speh type.

(2) Distinction for irreducible products of ladder representations (see Proposition 12.5): Assume Hypothesis 8.6 holds true for all multisegement. Let π_{1}, …, π_{k} be ladder representations such that π = π_{1} × ⋯ × π_{k} is an irreducible representation. If π is Spdistinguished then π_{i} is Sp-distinguished for all i = 1, …, k.

(3) Purity of symplectic distinction within ladder class (see Corollary 12.6): Let π_{1} and π_{2} be ladder representations such that π = π_{1} × π_{2} is irreducible. If π is Sp-distinguished then π_{1} and π_{2} are Sp-distinguished.

We emphasize that Theorem 1.4 (3) is unconditional. We further show in Proposition 8.7 that Hypothesis 8.5 is satisfied by multi-sets that are in fact sets. This yields the following unconditional result.

Theorem 1.5. (see Corollary 9.2): Let λ(𝔪) = L(Δ_{1}) × ⋯ × L(Δ_{t}) be a standard module such that Δ_{i} ≠ Δ_{j} for all i ≠ j. If λ(𝔪) is Sp-distinguished then 𝔪 is of Speh type.

1.2. Proofs and methods. Let us now elaborate a bit on the techniques used in the proofs. The filtration of the geometric lemma allows us to study Spdistinction of induced representations from the parabolic subgroup P in terms of the geometry of P -orbits on the symmetric space GL_{2n}(F)/ Sp_{2n}(F). In particular we show that an induced Sp-distinguished representation admits a P - orbit which is relevant. Analyzing the relevant orbits together with the Jacquet module calculations of segment representations allows us to prove Theorem 1.1 (1). For Theorem 1.1 (2), we combine the theory of derivatives with a meromorphic continuation technique of Blanc and Delorme. The first is used to reduce the problem to the case of Sp-distinction and the second to construct Sp-invariant linear forms on families of induced representations. A key ingredient for the proof of Theorems 1.2 and 1.4 is the necessary condition for a standard module to be Sp-distinguished provided by the geometric lemma. This allows us to reduce the problem to a purely combinatorial one on multi-sets of segments. We address it under a technical hypothesis that we can prove only for certain multi-sets (in particular, whenever they are sets). The hypothesis can be interpreted as a statement that certain orbits, of the natural action of a parabolic subgroup P of GL_{2n}(F) on GL_{2n}(F)/ Sp_{2n}(F), do not contribute a non-trivial Sp_{2n}(F)-invariant linear form. Explicitly, assuming the hypothesis we show that if a representation, irreducibly induced from ladder representations, is Sp-distinguished then each ladder representation in the inducing data also admits a symplectic model. This partial result along with the description of the maximal proper subrepresentation of the standard module associated to a ladder representation allows us to finish the proof of Theorem 1.2. The proof of Theorem 1.3 is obtained by using the explicit knowledge of the structure of a Jacquet module of a ladder representation and the classification of Sp-distinguished ladders given by Theorem 1.2. To prove Theorem 1.4 (2) we apply a recent irreducibility result from [LM], as well as the invariance of the class of ladder representations with respect to the Zelevinsky involution, to construct an inductive set-up in the context of representations irreducibly induced from ladder representations. We hope that the techniques developed in doing so will be useful more generally. Most notably, when we attempt to study distinction by other closed subgroups of GL_{n}(F) for representations that are irreducibly induced from ladders.

1.3. Related works. We mention here a few works where the results or the tools used have some intersection with the present work. The present work began as an attempt to extend the distinction results of [OS07] and [OS08a] from the unitary dual of GL_{n}(F) to the admissible dual of GL_{n}(F). We emphasize that in [OS07] and [OS08a] obtaining a model for a Speh representation, in particular the classification of Speh representations admitting a symplectic model, was based on the global theory of period integrals of Eisenstein series and their residues obtained in [Off06a] and [Off06b]. A novel aspect of this work is that our method of proof is purely local, and therefore, independently provides a local proof for the results of the aforementioned works. The methods employed here are very different from those works and are, in fact, closer in spirit to the techniques of [HR90], or to that of [Mit14] which studies admissible representations distinguished with respect to a symplectic group in small rank cases. The focus on distinction problems within the class of ladder representations was made in [FLO12] and later in [Gur15] and Theorem 1.2 of the introduction could be considered as an analogue to their results. A study parallel to our study of the distinction problem for standard modules can be found in these two references. In [GOSS12] the existence of Klyachko models is proved for unitary representations of GL_{n}(ℝ) and GL_{n}(ℂ). The methods there are parallel to those in the works of the second and third author in the non-archimedean case. In particular, the proof of existence of those models for Speh representations was based on the theory of periods of automorphic forms. Recently, in [GSS15] a local construction of these invariant functionals is provided, based on tools from the theory of distributions and D-modules. Some of the results of the present work can be considered as a non-archimedean analogue of the main result of [GSS15].

1.4. Structure of the paper. Let us now delineate the contents of this paper. A large part of it (§2-12) concerns symplectic models. After setting up the general notation for this work in §2, we recall some well known results concerning GL_{2n}(F)/ Sp_{2n}(F) in §3, especially the structure of orbits of the natural action of a parabolic subgroup of GL_{2n}(F). Our main tool for studying Sp-distinction of induced representations is an application of the geometric lemma of Bernstein and Zelevinsky. This is recalled in §4. In §5.1 we obtain some immediate consequences for Sp-distinction of certain induced representations. They come from contributions to the open and to the closed orbits of the aforementioned action. In §5.2 we reduce the classification of Sp-distinguished irreducible representations to those supported in a single cuspidal line viz. the rigid representations. In order to study distinction for rigid representations, we recall in §6 the segment notation of Zelevinsky and the classification of the admissible dual. In §7 we provide a necessary condition for an irreducible representation to be Sp-distinguished in terms of the Zelevinsky classification (see Proposition 7.5). We then turn to the study of distinction of standard modules. A necessary condition for a standard module (and for an irreducible representation) to be Sp-distinguished is reduced in §9 to a combinatorial problem. This problem is formulated in §8 as Hypothesis 8.5. §8 is written in a way completely independent from the rest of the paper, is accessible to any mathematician, and presents a problem with applications to the study of Sp-distinction. Our partial results suffice in order to obtain a complete classification of Spdistinguished ladder representations. This is Theorem 10.3. In §12 we obtain our results on Sp-distinction for the class of representations irreducibly induced from ladder. These are conditional on Hypothesis 8.5. Again these results require some purely combinatorial lemmas that we prove in §11. In §13 we turn to the study of Klyachko models in general. We prove the hereditary property with respect to parabolic induction (Proposition 13.3) and reduce the problem to rigid representations (Proposition 13.4). Finally the classification of ladder representation with a given Klyachko model is obtained in §14.

1.5. Acknowledgments. The authors are grateful to Erez Lapid for sharing with them his insights on ladder representations and irreducibility results.

2. Notation and preliminaries

We set the general notation in this section. More particular notation is defined in the section where it first occurs.

2.1. Generalities. Let G be a totally disconnected, locally compact group.

2.1.1. Let δ_{G} be the modulus function of G with the convention that δ_{G}(g)dg is a right-invariant Haar measure if dg is a left-invariant Haar measure on G.

2.1.2. Let H be a closed subgroup of G and σ a smooth, complex-valued representation of H. We denote by Ind_{H}^{G}(σ) the normalized induced representation. It is the representation of G by right translations on the space of functions f from G to the space of σ satisfying

f(hg) = (δ_{H}^{1/2}δ_{G}^{−1/2})(h)σ(h)f(g), h ∈ H, g ∈ G

and f is right invariant by some open subgroup of G. The representation of G on the subspace of functions with compact support modulo H is denoted by ind_{H}^{G}(σ).

2.1.3. This paper is concerned with distinguished representations in the following sense.

Definition 2.1. Let π be a smooth, complex-valued representation of G and H a closed subgroup of G.

• We say that π is H-distinguished if there exists a non-zero H-invariant linear form ℓ on the space of π, i.e., ℓ(π(h)v) = ℓ(v) for all h ∈ H and v in the space of π. We denote by Hom_{H}(π, 1) the space of H-invariant linear forms on π.

• More generally, for a character χ of H we say that π is (H, χ)- distinguished if the space Hom_{H}(π, χ) of H-equivariant linear forms on π is non-zero.

By Frobenius reciprocity we have a natural linear isomorphism

(1)  Hom_{H}(π, χδ_{H}^{1/2}δ_{G}^{−1/2}) ≃ Hom_{G}(π, Ind_{H}^{G}(χ)).

2.1.4. We state the following simple observation.

Lemma 2.2. Let π and σ be smooth, complex-valued representations of G so that σ is a quotient of π, H is a closed subgroup of G and χ is a character of H. If σ is (H, χ)-distinguished then π is (H, χ)-distinguished.

Proof. Note that composition with the projection π → σ defines an imbedding Hom_{H}(σ, χ) ↪ Hom_{H}(π, χ). The lemma follows. □

The lemma allows us to reduce some distinction questions to induced representations (e.g. using the Langlands classification). Its converse need not be true.

2.1.5. We record here another simple observation related to the converse problem, distinction of subquotients of a distinguished representation.

Lemma 2.3. Let π be a smooth, complex-valued representation of G, H a closed subgroup of G and χ a character of H. Let 0 = π_{0} ⊆ π_{1} ⊆ ⋯ ⊆ π_{k} = π be a filtration of π by sub-representations. If π is (H, χ)-distinguished, then there exists i ∈ {1, …, k} such that π_{i}/π_{i−1} is (H, χ)-distinguished.

In particular, if π is of finite length and (H, χ)-distinguished then there exists an irreducible subquotient σ of π that is (H, χ)-distinguished.

Proof. If 0 ≠ ℓ ∈ Hom_{H}(π, χ) then there exists i ∈ {1, …, k} minimal such that ℓ|_{π_{i}} ≠ 0. Thus, ℓ defines a non-zero element of Hom_{H}(π_{i}/π_{i−1}, χ). Since a finite length representation has such a finite filtration with irreducible quotients the rest of the lemma follows. □

2.1.6. Let Π(G) be the category of complex valued, smooth, admissible representations of G of finite length and Irr(G) the class of irreducible representations in Π(G). Let π^{∨} denote the contragredient of a representation π ∈ Π(G). Then (π^{∨})^{∨} ≃ π and π ∈ Irr(G) if and only if π^{∨} ∈ Irr(G).

2.2. Notation for GL_{n}(F). Let F be a non-archimedean local field of characteristic different than two. For n ∈ ℕ, let G_{n} = GL_{n}(F). By convention, let G_{0} be the trivial group.

2.2.1. Fix n and let G = G_{n}. Let B = T ⋉N be the standard Borel subgroup of G consisting of uppertriangular matrices with its standard Levi decomposition. Here T is the subgroup of diagonal matrices and N = N_{n} is the unipotent radical of B.

2.2.2. A parabolic subgroup of G that contains B is called standard. Standard parabolic subgroups of G are in bijection with compositions of n. For a composition α = (n_{1}, …, n_{k}) of n let P_{α} = M_{α} ⋉ U_{α} be the standard parabolic subgroup of G consisting of block uppertriangular matrices with standard Levi subgroup

M_{α} = {diag(g_{1}, …, g_{k}) : g_{i} ∈ G_{n_{i}}, i = 1, …, k} ≃ G_{n_{1}} × ⋯ × G_{n_{k}}

and unipotent radical U_{α}.

2.2.3. The Weyl group N_{G}(T)/T of G is isomorphic to the permutation group S_{n} of n elements. We identify it with the subgroup W = W_{G} of permutation matrices in G. Let w_{n} = (δ_{i,n+1−j}) ∈ W be the longest Weyl element. By the Bruhat decompositon W is a complete set of representatives for the double coset space B\G/B.

2.2.4. More generally, for a composition α = (n_{1}, …, n_{k}) of n, W_{M_{α}} = W ∩ M_{α} ≃ S_{n_{1}} × ⋯ × S_{n_{k}} is the Weyl group of M_{α}. If P = M ⋉ U and Q = L ⋉ V are standard parabolic subgroups of G with their standard Levi decompositions then w ↦ P wQ defines a bijection

W_{M}\W/W_{L} ≃ P \G/Q.

Furthermore, every double coset in W_{M}\W/W_{L} contains a unique element of minimal length. Denote by _{M}W_{L} the set of elements w ∈ W that are of minimal length in W_{M}wW_{L}. Then _{M}W_{L} is a complete set of representatives for P \G/Q.

For every w ∈ _{M}W_{L} the group

P (w) = M(w) ⋉ U(w) = M ∩ wQw^{−1}

is a standard parabolic subgroup of M with its standard Levi decomposition, where

M(w) = M ∩ wLw^{−1} and U(w) = M ∩ wV w^{−1}.

2.3. Representations of GL_{n}(F). We recall some well known facts and set the notation for representations of G_{n}. Let Π be the disjoint union of Π(G_{n}) for all n ∈ ℤ_{≥0}. Let Irr be the subset of irreducible representations in Π and Cusp be the subset of cuspidal representations in Irr.

2.3.1. Parabolic induction. Set G = G_{n}. Let P = M ⋉ U and Q = L ⋉ V be standard parabolic subgroups of G with their standard Levi decompositions. Assume further that Q is a subgroup of P. The functor i_{M,L} : Π(L) → Π(M) of normalized parabolic induction is defined as follows. As noted above M ∩ Q = L ⋉ (M ∩ V) is a standard parabolic subgroup of M. For ρ ∈ Π(L) we consider ρ as a representation of M ∩ Q trivial on its unipotent radical M ∩ V and set

i_{M,L}(ρ) = ind_{M∩Q}^{M}(ρ).

The functor i_{M,L} is exact and we have

i_{M,L}(ρ)^{∨} ≃ i_{M,L}(ρ^{∨}).

Let α = (n_{1}, …, n_{k}) be a composition of n. Assume that M = M_{α} and let ρ_{i} ∈ Π(G_{n_{i}}), i = 1, …, k. Then ρ = ρ_{1} ⊗ ⋯ ⊗ ρ_{k} ∈ Π(M). Set

ρ_{1} × ⋯ × ρ_{k} = i_{G,M}(ρ).

2.3.2. Jacquet module. The functor i_{M,L} admits a left adjoint, namely, the normalized Jacquet functor r_{L,M} : Π(M) → Π(L). For σ ∈ Π(M), r_{L,M}(σ) is the representation of L on the space of V ∩ M-coinvariants of σ induced by the action δ_{Q∩M}^{−1/2}σ. It is also an exact functor and for σ ∈ Π(M) and ρ ∈ Π(L) we have the natural linear isomorphism (Frobenius reciprocity):

(2)  Hom_{M}(σ, i_{M,L}(ρ)) ≃ Hom_{L}(r_{L,M}(σ), ρ).

Let β_{i} be the composition of n_{i}, i = 1, …, k so that L = M_{(β_{1},...,β_{k})}. For representations π_{i} ∈ Π(G_{n_{i}}), i = 1, …, k we have

(3)  r_{L,M}(π_{1} ⊗ ⋯ ⊗ π_{k}) = r_{M_{β_{1}} ,G_{n_{1}}} (π_{1}) ⊗ ⋯ ⊗ r_{M_{β_{k}} ,G_{n_{k}}} (π_{k}).

2.3.3. The cuspidal support. For every π ∈ Irr there exist ρ_{1}, …, ρ_{k} ∈ Cusp, unique up to rearrangement, so that π is isomorphic to a subrepresentation of ρ_{1} × ⋯ × ρ_{k}. Let Supp(π) = {ρ_{i} : i = 1, …, k} be the support of π.^{2} For σ_{1}, …, σ_{k} ∈ Irr let Supp(σ_{1} ⊗ ⋯ ⊗ σ_{k}) = ∪_{i=1}^{k}Supp(σ_{i}). For any standard Levi subgroup M of G and π ∈ Π(M) let {π_{1}, …, π_{t}} be the set of irreducible components (subquotients) of π and set

Supp(π) = ∪_{i=1}^{t}Supp(π_{i}).

2The support is often considered as a multi-set. Only the underlying set is relevant to us.

As a simple consequence of the geometric lemma of Bernstein and Zelevinsky [BZ77, §2.12] and exactness we have

(4)  Supp(r_{M,G}(π)) ⊆ Supp(π), π ∈ Π(G).

2.3.4. Generic representations. Let ψ be a non-trivial character of F. We further denote by ψ = ψ_{n} the character of N_{n} defined by

ψ(u) = ψ(∑_{i=1}^{n−1} u_{i,i+1}), u = (u_{i,j}) ∈ N_{n}.

Definition 2.4. A representation π ∈ Π(G_{n}) is called generic if it is (N_{n}, ψ)- distinguished.

3. Non-degenerate skew-symmetric matrices and parabolic orbits

We recall here the analysis of double cosets and related data that are relevant to the study of induced representations of G_{2n} that are distinguished by the symplectic group.

3.1. The symmetric space. Fix n ∈ N and let G = G_{2n}.

3.1.1. Let

H = H_{n} = Sp_{2n}(F) = {g ∈ G : ^{t}gJg = J}

where

J = J_{n} = (\begin{matrix}& w_{n} \\ −w_{n} \end{matrix}) .

Note that H = G^{θ} is the group of fixed points in G of the involution θ defined by

θ(g) = J ^{t}g^{−1}J^{−1}.

3.1.2. For π ∈ Π(G_{n}), since n will not always be specified, we adopt throughout the following convention. We say that π is Sp-distinguished if n is even and π is Sp_{n}(F)-distinguished. If in addition π ∈ Irr we say that π admits a symplectic model (see (1)). By a result of Gelfand and Kazhdan, [GK75], we have π^{θ} ≃ π^{∨} for every π ∈ Irr. We therefore have

Lemma 3.1. A representation π ∈ Irr is Sp-distinguished if and only if π^{∨} is Sp-distinguished. □

3.1.3. Consider the symmetric space

X = {x ∈ G : xθ(x) = I_{2n}}

with the G-action

g · x = gxθ(g)^{−1}.

Note that XJ is the space of skew-symmetric matrices in G and

(g · x)J = g(xJ)^{t}g.

Therefore X is a homogeneous G-space. The map (g ↦ g·I_{2n}) : G → X defines an isomorphism G/H ≃ X of G-spaces.

3.1.4. For a subgroup Q of G and a Q-invariant subspace Y of X we denote by Q\Y the set of Q-orbits in Y. For x ∈ X let Q_{x} = {g ∈ Q : g · x = x} be the stabilizer in Q of x. Applications of the geometric lemma to the study of H-distinguished induced representations of G require the study of orbits in X by the group from which we induce. Since parabolic induction is central to the classification of Irr we recall next the study of orbits in X under a standard parabolic subgroup P = M ⋉U. Of particular interest for these applications are choices of orbit representatives x for which we can provide explicit description of the stabilizer M_{x} and the restriction to M_{x} of δ_{P}^{1/2}δ_{P_{x}}^{−1}. We refer to [Off06a, §3] and [Off06b, §3.1] for proofs of the results presented in §3.2 and §3.3.

3.2. Borel orbits in X. We begin with the Borel orbits.

3.2.1. Note that both B and T are θ stable. In particular θ defines an involution on W that we continue to denote by θ. We have

θ(w) = w_{2n}ww_{2n}^{−1}, w ∈ W.

It also follows that the map (B · x ↦ BxB) : B\X → B\G/B is well defined. By the Bruhat decomposition this defines a map from B\X to W. Since θ(BxB) = (BxB)^{−1} it follows that every w in the image of this map satisfies wθ(w) = e (the identity element of W). We refer to such permutations as twisted involutions.

3.2.2. Let

[w_{2n}] = {ww_{2n}w^{−1} : w ∈ W }

be the W -conjugacy class of the longest Weyl element. It is the set of involutions without fixed points in W. Note that the set of twisted involutions in W is precisely [w_{2n}]w_{2n}. In fact we have (see [Off06a, Proposition 3.2 and Corollary 3.3]):

Lemma 3.2. The map (B · x ↦ B · x ∩ T) : B\X → T \(X ∩ N_{G}(T)) and the natural map from T \(X ∩ N_{G}(T)) to [w_{2n}]w_{2n} are both bijective. □

3.2.3. Fix a Borel orbit B · x ∈ B\X. We may and do assume that x ∈ X ∩ N_{G}(T) and let w ∈ [w_{2n}] be such that x ∈ T ww_{2n}. Below is a description of T_{x} and of the restriction to T_{x} of δ_{B}^{1/2}δ_{B_{x}}^{−1} (see §3.1.4 for notation). Note first that

T_{I_{2n}} = T ∩ H = {diag(a_{1}, …, a_{n}, a_{n}^{−1}, …, a_{1}^{−1}) : a_{i} ∈ F^{∗}, i = 1, …, n}

and

(δ_{B}^{−1/2}δ_{B_{I_{2n}}})(diag(a_{1}, …, a_{n}, a_{n}^{−1}, …, a_{1}^{−1})) = |a_{1}|_{F} ⋯ |a_{n}|_{F} .

We summarize the relevant results.

Lemma 3.3. For every orbit in B\X there exists a unique τ ∈ [w_{2n}] and a representative x ∈ N_{G}(T) such that

(1) T_{x} = {diag(a_{1}, …, a_{2n}) ∈ T : a_{τ(i)} = a_{i}^{−1}, i = 1, …, n}

(2) (δ_{B}^{−1/2}δ_{B_{x}})(t) = ∏_{i<τ(i)} |a_{i}|_{F} for every t = diag(a_{1}, …, a_{2n}) ∈ T_{x}. □

3.3. P -orbits in X. Let α = (n_{1}, …, n_{k}) be a composition of 2n and let P = P_{α} = M ⋉ U. Note that θ(P) = P_{(n_{k},...,n_{1})} and θ(M) = w_{2n}Mw_{2n}^{−1} = M_{(n_{k},...,n_{1})}.

3.3.1. The P -orbits in X are in bijection with certain twisted involutions.

Lemma 3.4. The map (P · x ↦ P xθ(P)) : P \X → P \G/θ(P) ≃ _{M}W_{θ(M)} defines a bijection

P \X ≃ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n}.
□

3.3.2. Let

ı_{M} : _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} → P \X

denote the bijection of Lemma 3.4. Recall that for w ∈ _{M}W_{θ(M)}

M(w) = M ∩ wθ(M)w^{−1}

is a standard parabolic subgroup of M.

Lemma 3.5. For every w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} we have that ı_{M}(w) ∩ M(w)w is a single M(w)-orbit. In particular, it is not empty. □

3.3.3. Admissible orbits.

Definition 3.6. We say that w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} (or the corresponding P -orbit ı_{M}(w)) is M-admissible if M(w) = M, i.e., if ww_{2n} ∈ N_{G}(M).

Thus, w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} is M-admissible if and only if the intersection ı_{M}(w) ∩ N_{G}(M)w_{2n} is not empty. In particular ı_{M} restricts to a bijection

(_{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} ∩ N_{G}(M)w_{2n}) ≃ (M − admissible orbits in X).

The P -orbits in X are studied in terms of certain L-admissible orbits for Levi subgroups L of M. More precisely, to the P -orbit ı_{M}(w) we associate a certain M(w)-admissible orbit. We therefore begin by describing the relevant data for M-admissible orbits. Let

S_{2}[α] = {τ ∈ S_{k} : τ^{2} = e, n_{τ(i)} = n_{i}, i = 1, …, k and n_{i} is even if τ(i) = i}.

The admissible M-orbits are in bijection with S_{2}[α]. Before we state the general results we provide examples of prototypes of admissible orbits.

3.3.4. Assume that k = s + 2t, n_{i} = n_{k+1−i}, i = 1, …, t and n_{i} is even for i = t + 1, …, t + s, i.e., α is of the form

α = (n_{1}, …, n_{t}, 2m_{1}, …, 2m_{s}, n_{t}, …, n_{1}).

Let

x = diag(I_{N}, J_{(m_{1},...,m_{s})}J_{m}^{−1}, I_{N}) = (\begin{matrix}& & w_{N} \\ & J_{(m_{1},...,m_{s})} \\ −w_{N} \end{matrix} ) J_{n}^{−1} ∈ X

where

J_{(m_{1},...,m_{s})} = diag(J_{m_{1}}, …, J_{m_{s}}), N = n_{1} +⋯+n_{t} and m = m_{1} +⋯+m_{s}.

Note that xJ_{n} is a skew-symmetric matrix in N_{G}(M) and therefore P · x is M-admissible. For every d ∈ ℕ consider the involution g ↦ g^{∗} on G_{d} defined by g^{∗} = w_{d} ^{t}g^{−1}w_{d}^{−1}. We have

M_{x} = {diag(g_{1}, …, g_{t}, h_{1}, …, h_{s}, g_{t}^{∗}, …, g_{1}^{∗}) :

: g_{1}, …, g_{t} ∈ G_{n_{i}}, h_{1}, …, h_{s} ∈ H_{m_{j}}}

and

(δ_{P}^{−1/2}δ_{P_{x}})(diag(g_{1}, …, g_{t}, h_{1}, …, h_{s}, g_{t}^{∗}, …, g_{1}^{∗})) = ∏_{i=1}^{t} |det g_{i}|_{F} .

3.3.5. We now return to the general setting where α is any composition of 2n.

Lemma 3.7. There is a bijection between the M-admissible P -orbits in X and S_{2}[α] that satisfies the following properties. Let w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} ∩ N_{G}(M)w_{2n} and let τ ∈ S_{2}[α] correspond to ı_{M}(w). Then, there exists x = x_{M,w} ∈ X ∩ Mw such that:

(1) M_{x} = {diag(g_{1}, …, g_{k}) : g_{τ(i)} = g_{i}^{∗} if τ(i) ≠ i and g_{i} ∈ H_{n_{i}/2} if τ(i) = i}

(2) (δ_{P}^{−1/2}δ_{P_{x}})(diag(g_{1}, …, g_{k})) = ∏_{i<τ(i)} |det g_{i}|_{F}. □

3.3.6. Every τ ∈ S_{k} defines a unique w_{τ} ∈ W such that for every g = diag(g_{1}, …, g_{k}) ∈ M we have

w_{τ} gw_{τ}^{−1} = diag(g_{τ^{−1}(1)}, …, g_{τ^{−1}(k)}).

Remark 3.8. In fact, the relation between w and τ in the above lemma is characterized by diag(w_{n_{1}}, …, w_{n_{k}})w_{τ} w_{2n} = w.

3.3.7. General orbits. Fix w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} and let L = M(w) be the standard Levi subgroup of M we associated with w. Let β = (β_{1}, …, β_{k}) be such that L = M_{β} where β_{i} = (m_{i,1}, …, m_{i,k_{i}}) is a composition of n_{i}. On the set of indices

ℑ = {(i, j) : i = 1, …, k, j = 1, …, k_{i}}

we consider the lexicographic order (i, j) ≺ (i′, j′) if either i < i′ or i = i′ and j < j′. We further consider the partial order (i, j) ≪ (i′, j′) if i < i′. Recall that by Lemmas 3.4 and 3.5, X ∩ Lw is an L-orbit. Note that w is L-admissible. Furthermore, for x ∈ X ∩ Lw we have M_{x} = L_{x} and P_{x} = Q_{x} where Q = P_{β} is the standard parabolic subgroup of G with Levi subgroup L. We may therefore apply Lemma 3.7 with M replaced by L. We consider S_{2}[β] as a set of involutions on ℑ, by identifying (ℑ, ≺) with the linearly ordered set {1, 2, …, |ℑ|}. Let τ ∈ S_{2}[β] be the involution associated with ı_{L}(w). Since w ∈ _{M}W_{θ(M)} there are more restrictions on τ, it must satisfy

(5)  τ(i, j + 1) ≪ τ(i, j), i = 1, …, k, j = 1, …, k_{i} − 1.

This implies in particular that for every i there is at most one j such that τ(i, j) = (i, j).

4. The geometric lemma

We recall here a special case of the geometric lemma [BZ77, Theorem 5.2] (see also [BD08, Proposition 1.17]). As in the previous section, fix n ∈ ℕ and let G = G_{2n} and H = H_{n}. Let α = (n_{1}, …, n_{k}) be a composition of 2n and let P = P_{α} = M ⋉ U. Consider the functor Res_{H} ∘i_{G,M} from Π(M) to the category of smooth representations of H where Res_{H} stands for restriction to H.

4.1. The H-filtration. For every σ ∈ Π(M) we recall here the existence of an H-filtration on Res_{H} ∘i_{G,M}(σ) parameterized by P \X and explicate the factors of the filtration.

4.1.1. By [BZ76, §1.5] (see also [BD08, Lemma 3.1]) there is a linear ordering MW_{θ(M)} ∩ [w_{2n}]w_{2n} = {w_{1}, …, w_{m}} so that

X_{i} = ∪_{j=1}^{i}ı_{M}(w_{i})

is open in X for all i = 1, …, m.

4.1.2. The orbit of the identity ι_{M}(I_{2n}) = P · I_{2n} is closed in X and we may assume that w_{m} = I_{2n}. Furthermore, if n_{i} is even for all i = 1, …, k then the orbit P · x_{M} where

x_{M} = J_{(n_{1}/2,...,n_{k}/2)}J_{n}^{−1}

is open in X and we may assume that ı_{M}(w_{1}) = P · x_{M}. Furthermore, in this case, w_{1} is M-admissible.

4.1.3. Let σ ∈ Π(M) and let V be the representation space of i_{G,M}(σ). Set

V_{i} = {φ ∈ V : Supp(φ) ⊆ X_{i}}, i = 1, …, m

then V_{0} := 0 ⊆ V_{1} ⊆ ⋯ ⊆ V_{m} = V is a filtration of Res_{H}(i_{G,M}(σ)). For every i choose (by Lemma 3.5) x_{i} ∈ ı_{M}(w_{i}) ∩ M(w_{i})w_{i} and η_{i} ∈ G such that η_{i} · I_{2n} = x_{i}. For a subgroup A of a group B, b ∈ B and a representation ρ of A we denote by ρ^{b} the representation of b^{−1}Ab on the space of ρ defined by ρ^{b}(b^{−1}ab) = ρ(a). By [Off06b, Proposition 3] we have

Lemma 4.1. For every i = 1, …, m we have the isomorphism of representations of H

V_{i}/V_{i−1} ≃ ind_{H∩η_{i}^{−1}P_{x_{i}} η_{i}}^{H}(δ_{P_{x_{i}}}^{−1/2}(Res_{P_{x_{i}}} (δ_{P}^{1/2}σ))^{η_{i}}).
□

4.1.4. Relevant orbits.

Definition 4.2. We say that w_{i} ∈ _{M}W_{θ(M)} ∩[w_{2n}]w_{2n} (or ı_{M}(w_{i})) is relevant for σ if

Hom_{H}(V_{i}/V_{i−1}, 1) ≠ 0.

4.1.5. The following makes this property more explicit. By [Off06b, Corollary 1] we have

Lemma 4.3. Fix i and let w = w_{i}, x = x_{i}, η = η_{i}, L = M(w_{i}) and Q the standard parabolic subgroup of G with Levi subgroup L. Then

Hom_{H}(ind_{H∩η^{−1}P_{x}η}^{H}(δ_{P_{x}}^{−1/2}(Res_{P_{x}}(δ_{P}^{1/2}σ))^{η}), 1) ≃ Hom_{L_{x}}(r_{L,M}(σ), δ_{Q}^{−1/2}δ_{Q_{x}}).

□

4.1.6. Combining Lemmas 4.1 and 4.3 we have

Corollary 4.4. Let w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n}. With the above notation w is relevant for σ if and only if

Hom_{L_{x}}(r_{L,M}(σ), δ_{Q}^{−1/2}δ_{Q_{x}}) ≠ 0.
□

4.1.7. Finally, by choosing the orbit representative x as in Lemma 3.7 (where M is replaced by L) we explicate the condition Hom_{L_{x}}(ρ, δ_{Q}^{−1/2}δ_{Q_{x}}) ≠ 0 for certain pure tensor representations ρ ∈ Π(L). For a representation π ∈ Π(G_{r}), let π^{∗} ∈ Π(G_{r}) be the representation on the space of π defined by π^{∗}(g) = π(g^{∗}). By a result of Gelfand and Kazhdan ([GK75]) we have π^{∗} ≃ π^{∨} for all π ∈ Irr. In the notation of §3.3.7 let

ρ = ⊗_{ı∈(ℑ,≺)}ρ_{ı}

and let τ ∈ S_{2}[β] be the involution on ℑ associated to w by Lemma 3.7 applied with L replacing M. Assume that ρ_{ı} ∈ Irr(G_{n_{ı}}) whenever τ(ı) ≠ ı. Then by Lemma 3.7 we have

(6) Hom_{L_{x}}(ρ, δ_{Q}^{−1/2}δ_{Q_{x}}) ≠ 0 if and only if for all ı ∈ ℑ we have

ρ_{ı} ≃ νρ_{τ(ı)} whenever ı ≺ τ(ı) and ρ_{ı} is H_{n_{ı}} − distinguished if τ(ı) = ı.

4.1.8. Combining Corollary 4.4, (6) and Lemma 2.3 we have

Corollary 4.5. Let w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n}. With the above notation, if w is relevant for σ then there exists an irreducible component ρ = ⊗_{ı∈(ℑ,≺)}ρ_{ı} ∈ Irr(L) of r_{L,M}(σ) such that

ρ_{ı} ≃ νρ_{τ(ı)} whenever ı ≺ τ(ı) and ρ_{ı} is H_{n_{ı}}-distinguished if τ(ı) = ı.

□

5. First applications of the geometric lemma to Sp-distinction

In this section we apply the contribution of the open and closed orbits of the filtration defined in §4 in order to show that certain induced representations are Sp-distinguished. We further apply §4 to reduce the study of Sp-distinction on Irr to representations supported on a single cuspidal line.

5.1. Distinction and relevant orbits.

5.1.1. The variant of the geometric lemma discussed in §4 is often applied to show that certain induced representations are not distinguished. This is based on the following simple observation, which is an immediate consequence of Lemma 2.3 (applied with G = H).

Lemma 5.1. Let M be a standard Levi subgroup of G. If σ ∈ Π(M) is such that i_{G,M}(σ) is H-distinguished then there exists a P -orbit in X that is relevant to σ. □

The reverse implication need not be true. However, there are two cases in which the geometric lemma indicates distinction.

5.1.2. Assume that n_{i} is even for all i. Then the open P -orbit in X is ı_{M}(w_{1}) = P · x_{M} (see §4.1.2) and

M_{x_{M}} = {diag(h_{1}, …, h_{k}) : h_{i} ∈ H_{n_{i}/2}, i = 1, …, k}.

Let σ_{i} ∈ Π(G_{n_{i}}) be H_{n_{i}/2}-distinguished and 0 ≠ ℓ_{i} ∈ Hom_{H_{n_{i}/2}}(σ_{i}, 1) for all i = 1, …, k. Let σ = σ_{1} ⊗ ⋯ ⊗ σ_{k} and ℓ = ℓ_{1} ⊗ ⋯ ⊗ ℓ_{k} ∈ Hom_{M_{x_{M}}} (σ, 1). The integral

(7)  ℓ̃(φ) = ∫_{(H∩η_{M}^{−1}M_{x_{M}} η_{M})\H} φ(η_{M}h) dh

where η_{M} ∈ G is such that η_{M} · I_{2n} = x_{M}, defines a non-zero linear form ℓ̃∈ Hom_{H}(V_{1}, 1). It does not necessarily extend to an H-invariant linear form on i_{G,M}(σ), but it lies in a holomorphic family of linear forms that do extend meromorphically. Note that G_{x_{M}} = G^{θ_{x_{M}}} is the fixed point group of the involution θ_{x_{M}} (g) = x_{M}θ(g)x_{M}^{−1} and that M = P ∩ θ_{x_{M}} (P). For λ = (λ_{1}, …, λ_{k}) ∈ ℂ^{k} let σ[λ] be the representation on the space of σ defined by

σ[λ](diag(g_{1}, …, g_{k})) = |det g_{1}|_{F}^{λ_{1}} ⋯ |det g_{k}|_{F}^{λ_{k}} σ(diag(g_{1}, …, g_{k})).

The representations i_{G,M}(σ[λ]) can all be realized in the same space V and then the H-filtration {V_{i}}_{i=1}^{n} is independent of λ. The following follows from [BD08, Theorem 2.8].

Lemma 5.2. With the above notation and assumptions, there is a nonzero meromorphic function (λ ↦ ℓ_{λ}) : ℂ^{k} → V^{∗} that satisfies ℓ_{λ} ∈ Hom_{H}(i_{G,M}(σ[λ]), 1) whenever holomorphic at λ.

5.1.3. Hereditary property of Sp-distinction. This implies the hereditary property of Sp-distinction.

Corollary 5.3. Assume that n_{i} is even for all i = 1, …, k. Let σ_{i} ∈ Π(G_{n_{i}}) be H_{n_{i}/2}-distinguished for all i = 1, …, k. Then σ_{1}×⋯×σ_{k} is H-distinguished.

Proof. This is immediate from Lemma 5.2 by taking a leading term at λ = 0 of ℓ_{λ} at a complex line through zero in a generic direction. □

5.1.4. Distinction by the closed orbit. When a closed orbit is relevant, the geometric lemma directly implies distinction.

Lemma 5.4. Let σ_{1}, …, σ_{t} ∈ Irr, ρ_{1}, …, ρ_{s} ∈ Π and assume that ρ_{i} is Spdistinguished for i = 1, …, s (allow the case s = 0). Then

νσ_{1} × ⋯ × νσ_{k} × ρ_{1} × ⋯ × ρ_{s} × σ_{k} × ⋯ × σ_{1}

is Sp-distinguished.

Proof. It follows from Corollary 5.3 that ρ = ρ_{1} × ⋯ × ρ_{s} is Sp-distinguished. Let G = G_{2n} and M its standard Levi subgroup so that

σ = νσ_{1} ⊗ ⋯ ⊗ νσ_{k} ⊗ ρ ⊗ σ_{k} × ⋯ ⊗ σ_{1}

is a representation of M. Then i_{G,M}(σ) ∈ Π(G). Let P be the standard parabolic subgroup with Levi subgroup M. Then the closed orbit P · I_{2n} is relevant to σ by (6). By §4.1.2, Hom_{H}(V/V_{m−1}, 1) ≠ 0 and therefore by Lemma 2.2 (applied with G = H), i_{G,M}(σ) is Sp-distinguished.

□

5.2. Reduction to cuspidal ℤ-lines. Let ν = |det|_{F} on G_{n} for any n ∈ ℕ.

5.2.1. For ρ ∈ cusp let ρ^{ℤ} = {ν^{n}ρ : n ∈ ℤ} be the ℤ-line through ρ. Denote by < the order on ρ^{ℤ} induced by the standard order on ℤ (so that ρ < νρ).

Definition 5.5. A representation π ∈ Π is called rigid if Supp(π) ⊆ ρ^{ℤ} for some ρ ∈ Cusp.

5.2.2. Every element of Irr has a unique decomposition as a product of rigid representations supported on disjoint cuspidal lines. Indeed, by [Zel80, Proposition 8.6] we have

Lemma 5.6. For every π ∈ Irr there exist ρ_{1}, …, ρ_{k} ∈ Cusp, so that ρ_{i}^{ℤ}∩ρ_{j}^{ℤ} = ∅ for all i ≠ j, and π_{1}, …, π_{k} ∈ Irr so that Supp(π_{i}) ⊆ ρ_{i}^{ℤ} and π = π_{1} × ⋯ × π_{k}. □

5.2.3. Another application of the geometric lemma will allow us to reduce the question of Sp-distinction of irreducible representations to those supported on a single cuspidal line. Indeed, if π = π_{1} × ⋯ × π_{k} is a decomposition as in Lemma 5.6 then

(8)  Hom_{Sp}(π, 1) ≃ Hom_{Sp}(π_{1}, 1) ⊗ ⋯ ⊗ Hom_{Sp}(π_{k}, 1).

Here, we write Hom_{Sp}(π, 1) = Hom_{Sp_{2n}(F)}(π, 1) for any π ∈ Π(G_{2n}). In fact, we prove (8) for a slightly more general setting for which we need to introduce some more terminology.

5.2.4. Consider the graph E with Cusp as the set of vertices and an edge between ρ and νρ for every ρ ∈ Cusp. For every finite subset V ⊆ Cusp let E_{V} be the induced graph on the set of vertices V and 𝔪_{V} the set of connected components of E_{V}. Every connected component Δ ∈ 𝔪_{V} is of the form Δ = {ν^{i}ρ : i = a, …, b} for some ρ ∈ Cusp and integers a ≤ b.

Definition 5.7. We say that finite subsets V, V′ ⊆ Cusp are totally disjoint if either V and V′ are contained in disjoint cuspidal ℤ-lines or they satisfy the following property. For every Δ ∈ 𝔪_{V} and Δ′ ∈ 𝔪_{V′} we have that either νρ < ρ′ for all ρ ∈ Δ and ρ′ ∈ Δ′ or νρ′ < ρ for all ρ ∈ Δ and ρ′ ∈ Δ′. (Equivalently, Δ ∪ Δ′ is not connected in E_{V ∪V′}.)

As a consequence of [Zel80, Proposition 8.5] we have

Lemma 5.8. If π_{1}, …, π_{k} ∈ Irr are such that Supp(π_{i}) and Supp(π_{j}) are totally disjoint for all i ≠ j then π_{1} × ⋯ × π_{k} ∈ Irr. □

5.2.5. We now show that (8) holds for totally disjoint decompositions as in Lemma 5.8. The following is a small generalization of [Mit14, Lemma 3.4].

Lemma 5.9. Let π_{1}, …, π_{k} ∈ Π be such that Supp(π_{i}) and Supp(π_{j}) are totally disjoint for all i ≠ j. Then π = π_{1} × ⋯ × π_{k} is Sp-distinguished if and only if π_{i} is Sp-distinguished for all i = 1, …, k. In particular, if π ∈ Irr then (8) holds.

Proof. The ‘only if’ part follows from Corollary 5.3. We prove the ‘if’ part. Let σ = π_{1}⊗⋯⊗π_{k}, n be such that π_{1}×⋯×π_{k} ∈ Π(G_{n}) and α the composition of n such that σ ∈ Π(M_{α}). Set G = G_{n} and M = M_{α}. Assume that π_{1} × ⋯ × π_{k} is Sp-distinguished. By Lemma 5.1, σ admits a relevant orbit, let w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} be relevant to σ. Apply the notation of §3.3.7. By Corollary 4.5, there exists an irreducible component ρ of r_{L,M}(σ) that satisfies (6). Then ρ = ⊗_{ı∈(ℑ,≺)}ρ_{ı} where ρ_{i,1} ⊗ ⋯ ⊗ ρ_{i,k_{i}} is an irreducible component of r_{M_{β_{i}} ,G_{n_{i}}} (π_{i}) for all i = 1, …, k (see (3)). In particular, Supp(ρ_{i,j}) ⊆ Supp(r_{M_{β_{i}} ,G_{n_{i}}} (π_{i})) ⊆ Supp(π_{i}) for all i (see (4)). Assume that there exists ı ∈ ℑ such that ı ≺ τ(ı) and let ı = (i, j) and τ(ı) = (i′, j′). Then ρ_{ı} ≃ νρ_{τ(ı)} and by (5), i ≠ i′. In particular, there exists ρ′ ∈ Supp(ρ_{τ(ı)}) ⊆ Supp(π_{i′}) such that νρ′ ∈ Supp(ρ_{ı}) ⊆ Supp(π_{i}). This contradicts the total disjointness of Supp(π_{i}) and Supp(π_{i′}). Therefore, τ is the trivial involution. Now (5) implies that w is M-admissible and π_{i} is Sp-distinguished for all i = 1, …, k as required. The isomorphism (8) now follows from [HR90, Theorem 2.4.2] (local multiplicity one for symplectic models). □

6. Representations of GL_{n}(F)

Before we continue with further applications of the geometric lemma to Spdistinction, we need to introduce the segment notation of Zelevinsky, and the Langlands and Zelevinsky classifications of Irr. We refer to [Zel80] for the results stated in this section.

6.1. Segment representations. By a segment of cuspidal representations we mean a set

[a, b]_{(ρ)} = {ν^{i}ρ : i = a, a + 1, …, b}

where ρ ∈ Cusp and a ≤ b are integers. By convention, the empty set is also considered a segment.

6.1.1. For a segment Δ = [a, b]_{(ρ)} as above, the representation ν^{a}ρ × ν^{a+1}ρ × ⋯ × ν^{b}ρ has a unique irreducible subrepresentation that we denote by Z(Δ) and a unique irreducible quotient that we denote by L(Δ).

6.1.2. We remark that Δ ↦ L(Δ) is a bijection between the set of segments of cuspidal representations and the subset of essentially square-integrable representations in Irr.

6.1.3. Also, Z(Δ) is the unique irreducible quotient and L(Δ) is the unique irreducible subrepresentation of ν^{b}ρ × ⋯ × ν^{a+1}ρ × ν^{a}ρ. By convention, if the segment Δ is empty, then both L(Δ) and Z(Δ) are taken to be the trivial representation of the trivial group.

6.1.4. We denote by b(Δ) = ν^{a}ρ the beginning, e(Δ) = ν^{b}ρ the end and ℓ(Δ) = |Δ| = b − a + 1 the length of Δ. Let νΔ = [a + 1, b + 1]_{(ρ)}.

6.1.5. Let Δ and Δ′ be segments of cuspidal representations. We say that Δ precedes Δ′ and write Δ ≺ Δ′ if both Δ and Δ′ are contained in some ℤ-line ρ^{ℤ} ⊆ Cusp, b(Δ) < b(Δ′), e(Δ) < e(Δ′) and e(Δ) ≥ b(Δ′) − 1. The segments Δ and Δ′ are called linked if either Δ ≺ Δ′ or Δ′ ≺ Δ. Equivalently, Δ and Δ′ are linked if and only if neither of them is contained in the other and their union is a segment.

6.1.6. For our conventions regarding multi-sets see §8.0.1. Let O be the set of multi-sets of segments of cuspidal representatons. An order 𝔪 = {Δ_{1}, …, Δ_{t}} ∈ O on a multi-set 𝔪 is of standard form if Δ_{i} ⊀ Δ_{j} for all i < j. Every 𝔪 ∈ O admits at least one standard order. Indeed, if for example e(Δ_{1}) ≥ ⋯ ≥ e(Δ_{t}) then {Δ_{1}, …, Δ_{t}} is in standard form.

6.1.7. The Zelevinsky classification. Let 𝔪 = {Δ_{1}, …, Δ_{t}} ∈ O be ordered in standard form. The representation

ζ(𝔪) = Z(Δ_{1}) × ⋯ × Z(Δ_{t})

is independent of the choice of order of standard form. It has a unique irreducible submodule that we denote by Z(𝔪). The Zelevinsky classification says that the map (𝔪 ↦ Z(𝔪)) : O → Irr is a bijection.

6.1.8. The representation

ζ̃(𝔪) = Z(Δ_{t}) × ⋯ × Z(Δ_{1}).

is also independent of the choice of standard order on 𝔪 and Z(𝔪) is the unique irreducible quotient of ζ̃(𝔪).

6.1.9. The Langlands classification. Let 𝔪 = {Δ_{1}, …, Δ_{t}} ∈ O be ordered in standard form. The representation

λ(𝔪) = L(Δ_{1}) × ⋯ × L(Δ_{t})

is independent of the choice of order of standard form. It has a unique irreducible quotient that we denote by L(𝔪). The Langlands classification says that the map (𝔪 ↦ L(𝔪)) : O → Irr is a bijection.

6.1.10. The Zelevinsky involution. It follows from §6.1.7 and §6.1.9 that for any 𝔪 ∈ O there exists a unique 𝔪^{t} ∈ O such that Z(𝔪) = L(𝔪^{t}). The function 𝔪 ↦ 𝔪^{t} is an involution on O. For π = Z(𝔪) ∈ Irr let π^{t} = L(𝔪). Then π ↦ π^{t} is the corresponding involution on Irr.

6.1.11. For 𝔪 ∈ O let Supp(𝔪) = {ρ ∈ Cusp : ρ ∈ Δ for some Δ ∈ 𝔪} be the support of 𝔪. (Note that Supp(𝔪) = Supp(Z(𝔪)) = Supp(L(𝔪)).) A multi-set 𝔪 ∈ O is called rigid if Supp(𝔪) ⊆ ρ^{ℤ} for some ρ ∈ Cusp. Let

O_{ρ} = {𝔪 ∈ O : Supp(𝔪) ⊆ ρ^{ℤ}}

be the set of rigid multi-sets supported on ρ^{ℤ}.

6.1.12. Lemma 5.9 reduces the study of Sp-distinguished representations in Irr to those supported on a cuspidal ℤ-line. From now on fix ρ ∈ Cusp once and for all. We will study Sp-distinction of certain rigid representations supported on ρ^{ℤ}.

7. A necessary condition for Sp-distinction of Z(𝔪)

In this section we show that if 𝔪 ∈ O_{ρ} is such that Z(𝔪) is Sp-distinguished then all segments in 𝔪 are of even length. The main tool is the geometric lemma of §4. We also apply a result of Heumos and Rallis that we first recall.

7.1. Results of Heumos and Rallis.

7.1.1. The following disjointness of models is [HR90, Theorem 3.2.2].

Lemma 7.1. If π ∈ Irr is generic then it is not Sp-distinguished. □

7.1.2. We also recall [HR90, Theorem 11.1] that provides first examples of irreducible, Sp-distinguished representations that are not necessarily one dimensional.

Lemma 7.2. Let Δ be a segment in Cusp. Then L({Δ, νΔ}) is Spdistinguished.

For the convenience of the reader, and since the proof below will be generalized in the sequel, we recall here the argument given by Heumos and Rallis. Recall that L({Δ, νΔ}) lies in an exact sequence

0 → π → λ({Δ, νΔ}) = νL(Δ) × L(Δ) → L({Δ, νΔ}) → 0

where π ∈ Irr is generic. The representation νL(Δ) × L(Δ) is Sp-distinguished by Lemma 5.4, whereas π is not Sp-distinguished by Lemma 7.1. Therefore, L({Δ, νΔ}) is Sp-distinguished.

Remark 7.3. Representations of the form L({Δ, νΔ, …, ν^{n−1}Δ}) are often referred to as Speh representations. In [OS07], Lemma 7.2 is generalized, showing that the Speh representation L({Δ, νΔ, …, ν^{n−1}Δ}) is Sp-distinguished if and only if n is even. This characterization of Sp-distinguished Speh representations was based on a global argument involving the period integrals of certain Eisenstein series. In this paper we provide a local proof of a generalization. Since Lemmas 7.1 and 7.2 will be applied in the sequel, we emphasize that their proofs in [HR90] are purely local.

7.2. On Sp-distinction of Z(𝔪).

7.2.1. Let Δ = [a, b]_{(ρ)} be a segment in ρ^{ℤ}. A special case of Remark 7.3 says that Z(Δ) = L({ν^{a}ρ}, {ν^{a+1}ρ}, …, {ν^{b}ρ}) is Sp-distinguished if and only if ℓ(Δ) is even. For the sake of completeness of our local proof, we now provide a proof of the easy part of this equivalence. A local proof of the other implication will be a part of Corollary 10.5.

Lemma 7.4. If the representation Z(Δ) is Sp-distinguished then ℓ(Δ) is even.

Proof. Assume that Z(Δ) is Sp-distinguished. By Lemma 2.2, ν^{b}ρ × ⋯ × ν^{a}ρ is also Sp-distinguished. Let d be such that ρ ∈ Π(G_{d}), G = G_{d(b−a+1)} and M = M_{(d,...,d)} the standard Levi subgroup of G so that σ = ν^{b}ρ ⊗ ⋯ ⊗ ν^{a}ρ ∈ Π(M). By Lemma 5.1, there exists an orbit, relevant to σ. Since ρ is cuspidal, it follows from (3) that r_{L,M}(σ) = 0 for every proper Levi subgroup L of M. It therefore follows from Corollary 4.4 that an orbit relevant to σ is M-admissible. Since all elements of Cusp are generic, it now follows from Lemma 7.1 and (6) (for L = M) that there exists, in particular, an involution on Δ without fixed points. Therefore ℓ(Δ) is even. □

7.2.2. This can be generalized to products of Z(Δ)’s, but we first need to explicate their Jacquet modules. We start with the Jacquet module of Z(Δ) itself. Let Δ = [a, b]_{(ρ)} be a segment in Cusp. Recall the description of the Jacquet module of Z(Δ) following [Zel80, §3.4]. Suppose that ρ ∈ Π(G_{d}) and let n = (b − a + 1)d so that Z(Δ) ∈ Π(G_{n}). Let M = M_{α} for a composition α = (n_{1}, …, n_{k}) of n. Then r_{M,G_{n}}(Z(Δ)) = 0 unless d|n_{i}, i = 1, …, k in which case

r_{M,G_{n}}(Z(Δ)) = Z(Δ_{1}) ⊗ ⋯ ⊗ Z(Δ_{k})

where Δ_{i} = [a_{i}, b_{i}]_{(ρ)}, a_{1} = a, a_{i+1} = b_{i} +1, i = 1, …, k −1 and d(b_{i} −a_{i} +1) = n_{i}, i = 1, …, k.

7.2.3. Suppose that β is a refinement of a composition α of n and let M = M_{α} and L = M_{β}. For the parts of the compositions and the ordered index set ℑ we apply the notation of §3.3.7. Let Δ_{1}, …, Δ_{k} be segments of cuspidal representations so that σ = Z(Δ_{1}) ⊗ ⋯⊗Z(Δ_{k}) is an irreducible representation of M. It follows from (3) and §7.2.2 that, whenever non-zero,

r_{L,M}(σ) = ⊗_{ı∈(ℑ,≺)} Z(Δ_{ı})

where r_{M_{β_{i}} ,G_{n_{i}}} (Z(Δ_{i})) = Z(Δ_{i,1}) ⊗ ⋯ ⊗ Z(Δ_{i,k_{i}}) is prescribed by §7.2.2.

7.2.4.

Proposition 7.5. Let 𝔪 ∈ O_{ρ}. If ζ̃(𝔪) is Sp-distinguished then Z(Δ) is Spdistinguished and, in particular, ℓ(Δ) is even for all Δ ∈ 𝔪. In particular, if Z(𝔪) is Sp-distinguished then ℓ(Δ) is even for all Δ ∈ 𝔪.

Proof. Fix an order 𝔪 = {Δ_{1}, …, Δ_{k}} so that b(Δ_{1}) ≤ ⋯ ≤ b(Δ_{k}) and note that {Δ_{k}, …, Δ_{1}} is a standard order on 𝔪, i.e., that

ζ̃(𝔪) ≃ Z(Δ_{1}) × ⋯ × Z(Δ_{k}).

Let σ = Z(Δ_{1}) ⊗ ⋯ ⊗ Z(Δ_{k}), G = G_{n} and M = M_{α} a standard Levi subgroup of G such that Z(Δ_{1}) × ⋯ × Z(Δ_{k}) = i_{G,M}(σ). Assume that ζ̃(𝔪) is Spdistinguished. By Lemma 5.1 there exists w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n} that is relevant to σ. We show that Z(Δ_{i}) is Sp-distinguished for all i. Assume, by contradiction, the contrary and use the notation of §3.3.7. It follows from Corollary 4.5 that τ is not the trivial involution. Let ı ∈ ℑ be minimal such that τ(ı) ≠ ı. Then ı ≺ τ(ı) and it follows from (5) that ı = (i, 1) for some 1 ≤ i ≤ k. But then the condition Δ_{ı} = νΔ_{τ(ı)} (from Corollary 4.5) contradicts our choice of order on 𝔪 and §7.2.3. It therefore follows that Z(Δ_{i}) is Sp-distinguished and from Lemma 7.4 that ℓ(Δ_{i}) is even for i = 1, …, k. The last part of the proposition follows from Lemma 2.2. □

Applications of the geometric lemma to the study of Sp-distinction of representations of the form λ(𝔪), 𝔪 ∈ O (standard modules) lead to certain combinatorial problems that we are only able to solve in special cases. In the next section we formulate these problems and present the proofs for our partial results. The section is written in such a way that it is independent of the rest of the paper and elementary. The problem we raise is accessible to every mathematician.

8. Multi-sets of segments

We formulate an elementary problem on multi-sets of segments of integers that has applications to representation theory. We are only able to provide a partial solution.

8.0.1. By a multi-set f of elements in a set X we mean a function f : X → ℤ_{≥0} of finite support. The support of f is also referred to as the set underlying f. If f takes value in {0, 1} then we identify f with its support and say that it is a set. For example, for x ∈ X the set of one element {x} is the characteristic function of x. Denote by |f| = ∑_{x∈X} f(x) the size of the multi-set f. By abuse of notation, we sometimes write f = {x_{1}, …, x_{t}} where t = |f| and x ∈ X equals x_{i} for exactly f(x) indices i. We refer to the presentation {x_{1}, …, x_{t}} as an order on f. We write x ∈ f if x is in the support of f.

8.0.2. By a segment of integers we mean a set [a, b] = {a, a + 1, …, b} where a ≤ b are integers. By convention, the empty set is a segment. Let Sgm denote the set of all segments of integers. Consider the operation

ν[a, b] = [a + 1, b + 1], [a, b] ∈ Sgm

and define the following relation on Sgm. For [a, b], [a′, b′] ∈ Sgm we say that [a, b] precedes [a′, b′] and write [a, b] ≺ [a′, b′] if a < a′, b < b′ and b ≥ a′ − 1. By a decomposition of [a, b] ∈ Sgm we mean a k-tuple of segments ([a_{1}, b_{1}], …, [a_{k}, b_{k}]) ∈ Sgm^{k}, k ∈ ℕ, such that b_{1} = b, a_{k} = a and b_{i+1} = a_{i}−1, i = 1, …, k − 1. The decomposition is called trivial if k = 1.

8.0.3. Let O_{ℤ} be the set of multi-sets of segments of integers. We say that a multi-set 𝔪 = {Δ_{1}, …, Δ_{k}} is ordered in standard form if Δ_{i} ⊀ Δ_{j} for all i < j. Given an ordered multi-set 𝔪 = {Δ_{1} …, Δ_{k}} ∈ O_{ℤ}, by a decomposition of 𝔪 we mean a decomposition of Δ_{i} for all i = 1, …, k. The decomposition is called trivial if the decomposition of each Δ_{i} is trivial. It will be convenient to index a decomposition of an ordered multi-set as follows. If (Δ_{i,1}, …, Δ_{i,k_{i}}) is the decomposition of Δ_{i}, i = 1, …, k let (ℑ, ≺) be the linearly ordered set

ℑ = {(i, j) : i = 1, …, k, j = 1, …, k_{i}}

with the lexicographic order (i, j) ≺ (i′, j′) if either i < i′ or i = i′ and j < j′. We further consider the partial order (i, j) ≪ (i′, j′) if i < i′. Thus, a decomposition of an ordered multi-set 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ℤ} produces a new ordered multi-set {Δ_{ı} : ı ∈ (ℑ, ≺)}.

8.0.4. Relevant decompositions.

Definition 8.1. Let 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ℤ} be an ordered multi-set. We say that an ordered decomposition {Δ_{ı} : ı ∈ (ℑ, ≺)} (of the order {Δ_{1}, …, Δ_{k}} of 𝔪) is relevant to {Δ_{1}, …, Δ_{k}} if there exists an involution τ on ℑ satisfying the following properties:

(1) τ(i, j + 1) ≪ τ(i, j), i = 1, …, k, j = 1, …, k_{i} − 1;

(2) τ(ı) ≠ ı, ı ∈ ℑ;

(3) Δ_{ı} = νΔ_{τ(ı)} whenever ı ≺ τ(ı).

8.0.5. The involutions τ in the above definition must satisfy the following property.

Lemma 8.2. Let τ be an involution on ℑ satisfying conditions (1) and (2) of Definition 8.1. Then, there exist i_{1} > ⋯ > i_{k_{1}} > 1 such that τ(1, j) = (i_{j}, k_{i_{j}}), j = 1, …, k_{1}.

Proof. Let τ(1, j) = (i_{j}, r_{j}). The inequalities i_{1} > ⋯ > i_{k_{1}} > 1 are immediate from conditions (1) and (2) of Definition 8.1. If r_{j} < k_{i_{j}} then, again by the same condition, τ(i_{j}, r_{j} + 1) ≪ (1, j) which is impossible. Therefore r_{j} = k_{i_{j}}. □

8.0.6. Distinguished multi-set.

Definition 8.3. A multi-set 𝔪 ∈ O_{ℤ} is called distinguished if every standard order of 𝔪 admits a relevant decomposition.

8.0.7. Speh type. For 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ℤ} and n ∈ ℤ let ν^{n}𝔪 = {ν^{n}Δ_{1}, …, ν^{n}Δ_{k}}. It is easy to see that the following conditions are equivalent for 𝔪 ∈ O_{ℤ}:

• 𝔪 is of the form 𝔫 + ν𝔫 for some 𝔫 ∈ O_{ℤ};

• the trivial decomposition of 𝔪 is relevant to some standard order of 𝔪;

• the trivial decomposition of 𝔪 is relevant to any standard order of 𝔪.

Definition 8.4. We say that 𝔪 ∈ O_{ℤ} is of Speh type if it satisfies the above equivalent conditions.

8.0.8. The main hypothesis. Consider the following property of a multi-set 𝔪 ∈ O_{ℤ}.

Hypothesis 8.5. If 𝔪 is distinguished then 𝔪 is of Speh type.

Unfortunately, we do not have enough information to determine whether Hypothesis 8.5 is satisfied for all 𝔪 ∈ O_{ℤ}. We therefore cautiously refrain from stating it as a conjecture. We will prove that it holds in some special cases and in particular when 𝔪 is a set.

8.0.9. In fact, for some of the representation theoretic applications we have in mind it suffices to consider a slightly weaker property. For [a, b] ∈ Sgm let [a, b]^{∨} = [−b, −a] and for 𝔪 ∈ O_{ℤ} let 𝔪^{∨} ∈ O_{ℤ} be defined by 𝔪^{∨}(Δ) = 𝔪(Δ^{∨}), Δ ∈ Sgm. Consider the following property of a multi-set 𝔪 ∈ O_{ℤ}.

Hypothesis 8.6. If 𝔪 and 𝔪^{∨} are both distinguished then 𝔪 is of Speh type.

8.0.10. In order to prove that a given multi-set 𝔪 satisfies the Hypothesis 8.5 it is enough to show that if 𝔪 is distinguished then the trivial decomposition is relevant to some standard order. In particular, it is enough to show that for some standard order, no non-trivial decomposition is relevant. We can only prove Hypothesis 8.5 for some special cases by proving this stronger version. For this purpose we define a certain standard order on multi-sets.

8.0.11. For Δ = [a, b] let b(Δ) = a be the beginning and e(Δ) = b the end of Δ. For 𝔪 ∈ O_{ℤ} let c_{1} > ⋯ > c_{s} be such that we have the identity of sets {c_{1}, …, c_{s}} = {e(Δ) : Δ ∈ 𝔪}. Let 𝔪[i] ∈ O_{ℤ} be defined by

𝔪[i](Δ) = \begin{cases}𝔪(Δ) & e(Δ) = c_{i} \\ 0 & otherwise \end{cases}

for i = 1, …, s so that

𝔪 = 𝔪[1] + ⋯ + 𝔪[s]

and all segments in the support of 𝔪[i] end at c_{i}. Note that any linear order on 𝔪 that extends the relation, Δ < Δ′ whenever Δ ∈ 𝔪[i] and Δ′ ∈ 𝔪[j] for all 1 ≤ i < j ≤ s, is in standard form.

8.0.12. In order to fix such an order, we need to linearly order each 𝔪[i]. We make a particular such choice recursively. Let 𝔪[1] be ordered by 𝔪[1] = {Δ_{1}, …, Δ_{k}} where b(Δ_{1}) ≤ ⋯ ≤ b(Δ_{k}) and set 𝔪[1]′ = 𝔪[1]. Suppose that 𝔪[1], …, 𝔪[i] are ordered and that 𝔪[1]′, …, 𝔪[i]′ are defined for some i < s. We order 𝔪[i + 1] and define 𝔪[i + 1]′ as follows. Set 𝔪[i + 1] = {Δ_{1}, …, Δ_{k}, Δ′_{1}, …, Δ′_{m}} where b(Δ_{1}) ≤ ⋯ ≤ b(Δ_{k}), b(Δ′_{1}) ≥ ⋯ ≥ b(Δ′_{m}) and min(𝔪[i + 1], ν^{−1}𝔪[i]′) = {Δ′_{1}, …, Δ′_{m}} and let 𝔪[i + 1]′ = {Δ_{1}, …, Δ_{k}}. Here min is the minimum of two ℤ_{≥0}-valued functions (see §8.0.1).

8.0.13. We now prove that Hypothesis 8.5 holds for sets.

Proposition 8.7. Let 𝔪 ∈ O_{ℤ} be a set. Then, no non-trivial decomposition is relevant to the order on 𝔪 defined in §8.0.12. In particular, Hypothesis 8.5 holds for 𝔪.

Proof. We prove the statement by induction on |𝔪|. If |𝔪| ≤ 2 then it follows from Lemma 8.2 that conditions (1) and (2) of Definition 8.1 cannot be satisfied by any non-trivial decomposition of 𝔪 and the proposition is therefore true.

Let 𝔪 = {Δ_{1}, …, Δ_{k}} and assume by contradiction that there exists a nontrivial decomposition {Δ_{ı} : ı ∈ ℑ} that is relevant to {Δ_{1}, …, Δ_{k}}. Let τ be the involution on ℑ satisfying properties (1), (2), (3) of Definition 8.1 and let i_{1} > ⋯ > i_{k_{1}} > 1 be given by Lemma 8.2 so that τ(1, j) = (i_{j}, k_{i_{j}}), j = 1, …, k_{1}. By the property of the order chosen e(Δ_{1}) ≥ e(Δ_{i_{k_{1}}} ) ≥ ⋯ ≥ e(Δ_{i_{1}}). The condition (3) of Definition 8.1 implies that e(Δ_{i_{1}}) ≥ e(Δ_{1}) − 1 and b(Δ_{i_{1}}) > ⋯ > b(Δ_{i_{k_{1}}} ) = b(Δ_{1}) − 1. Since the equality e(Δ_{i_{k_{1}}} ) = e(Δ_{1}) contradicts the order chosen on 𝔪[1] we must have e(Δ_{i_{k_{1}}} ) = e(Δ_{1}) − 1, i.e., Δ_{i_{k_{1}}} = ν^{−1}Δ_{1} and in the notation of §8.0.11, c_{2} = c_{1} − 1 and Δ_{i_{j}} ∈ 𝔪[2] for all j = 1, …, k_{1}. Since 𝔪 is a set, we get that Δ_{i_{k_{1}}} ∉ 𝔪[2]′. Taking the order chosen on 𝔪[2] into consideration, now implies that k_{1} = 1 = k_{i_{1}}. Again since 𝔪 is a set, it is easy to see that the order defined on 𝔪 − {Δ_{1}} − {Δ_{i_{1}}} by §8.0.12 is that inherited from 𝔪, i.e., the order {Δ_{2}, …, Δ_{i_{1}−1}, Δ_{i_{1}+1}, …, Δ_{k}}. Furthermore, τ|_{ℑ\{(1,1),(i_{1},1)}} shows that {Δ_{ı} : ı ∈ ℑ \ {(1, 1), (i_{1}, 1)}} is a non-trivial decomposition relevant to {Δ_{2}, …, Δ_{i_{1}−1}, Δ_{i_{1}+1}, …, Δ_{k}}. This contradicts the induction hypothesis. □

8.0.14. The same proof gives another family of multi-sets satisfying Hypothesis 8.5.

Proposition 8.8. Let 𝔪 ∈ O_{ℤ} and set 𝔪 = 𝔪[1] + ⋯ + 𝔪[s] as in §8.0.11. Assume that |𝔪[i]| ≤ 2 for all i = 1, …, s. Then, no non-trivial decomposition is relevant to the order on 𝔪 defined in §8.0.12. In particular, Hypothesis 8.5 holds for 𝔪.

Proof. The first three paragraphs of the proof of Proposition 8.7 hold for any multi-set and we apply the conclusions and the notation in this case. Since |𝔪[2]| ≤ 2 we conclude that k_{1} ≤ 2. Furthermore, if k_{1} = 2 then Δ_{i_{2}} = ν^{−1}Δ_{1}, Δ_{i_{1}} = ν^{−1}Δ_{1,1} and 𝔪[2] = {Δ_{i_{2}}, Δ_{i_{1}}}. But this shows that Δ_{i_{2}} ∉ 𝔪[2]′ and contradicts the order on 𝔪[2]. We therefore have k_{1} = 1 = k_{i_{1}}. Let 𝔫 = 𝔪 − {Δ_{1}} − {Δ_{i_{1}}}. Again, it is easily observed that the order on 𝔫 defined in §8.0.12 is the one inherited by 𝔪 and the proposition follows by induction as in the last paragraphs of the proof of Proposition 8.7. □

9. On Sp-distinction of standard modules

The bijection [a, b] ↦ [a, b]_{(ρ)} from segments of integers to segments in ρ^{ℤ} induces a bijection on multi-sets from O_{ℤ} to O_{ρ}. We refer to this bijection as ρ-labeling and to its inverse as unlabeling. In this section we show that if 𝔪 ∈ O_{ρ} is such that λ(𝔪) is Sp-distinguished then the unlabeling of 𝔪 is distinguished in the sense of Definition 8.3. The results and hypotheses of §8 therefore become relevant to the study of Sp-distinction of standard modules.

From now on we adopt the following convention. We say that a multi-set 𝔪 ∈ O_{ρ} satisfies a property defined on multi-sets in O_{ℤ} if its unlabeling satisfies this property.

9.0.1. Let Δ = [a, b]_{(ρ)} be a segment in ρ^{ℤ} and δ = L(Δ). We first recall the description of the Jacquet module of δ following [Zel80, §9.5]. Suppose that ρ is a representation of G_{d} and let n = (b − a + 1)d so that δ ∈ Π(G_{n}). Let M = M_{α} for a composition α = (n_{1}, …, n_{k}) of n. Then r_{M,G_{n}}(δ) = 0 unless d|n_{i}, i = 1, …, k in which case

r_{M,G_{n}}(δ) = δ_{1} ⊗ ⋯ ⊗ δ_{k}

where δ_{i} = L(Δ_{i}), Δ_{i} = [a_{i}, b_{i}]_{(ρ)}, b_{1} = b, b_{i+1} = a_{i} − 1, i = 1, …, k − 1 and d(b_{i} − a_{i} + 1) = n_{i}, i = 1, …, k.

9.0.2. Suppose that β is a refinement of a composition α of n and let M = M_{α} and L = M_{β}. For the parts of the compositions and the ordered index set ℑ we apply the notation of §3.3.7. Let δ = δ_{1} ⊗ ⋯ ⊗ δ_{k} be an irreducible, essentially square-integrable representation of M (i.e., δ_{i} = L(Δ_{i}) for some segment of cuspidal representations, i = 1, …, k). It follows from (3) that, whenever non-zero,

r_{L,M}(δ) = ⊗_{ı∈(ℑ,≺)} δ_{ı}

where r_{M_{β_{i}} ,G_{n_{i}}} (δ_{i}) = δ_{i,1} ⊗ ⋯ ⊗ δ_{i,k_{i}} is prescribed by §9.0.1.

9.0.3. Let M and δ be as above and w ∈ _{M}W_{θ(M)} ∩ [w_{2n}]w_{2n}. Recall that δ_{i} is generic i = 1, …, k. In the notation of §3.3.7 it follows from (6), Lemma 7.1 and §9.0.2 that

w is relevant for δ if and only if τ(ı) ≠ ı

for all ı ∈ ℑ and δ_{ı} ≃ νδ_{τ(ı)} whenever ı ≺ τ(ı).

(9)

9.0.4. The following is an immediate consequence.

Proposition 9.1. Let 𝔪 ∈ O_{ρ} satisfy Hypothesis 8.5. If λ(𝔪) is Spdistinguished then 𝔪 is of Speh type. In particular, if L(𝔪) is Sp-distinguished then 𝔪 is of Speh type.

Proof. If λ(𝔪) is Sp-distinguished then for any standard order on 𝔪 = {Δ_{1}, …, Δ_{k}}, the induced representation L(Δ_{1}) × ⋯ × L(Δ_{k}) is Spdistinguished. Therefore, by Lemma 5.1, L(Δ_{1})⊗⋯⊗L(Δ_{k}) admits a relevant orbit. Combining the condition 5 with (9), §9.0.3 says that 𝔪 is distinguished in the sense of Definition 8.3. Hypothesis 8.5 therefore implies that 𝔪 is of Speh type. The last part of the proposition follows from Lemma 2.2. □

9.0.5. Combining Proposition 9.1 with Propositions 8.7 and 8.8 we obtain

Corollary 9.2. Let 𝔪 ∈ O_{ρ} either be a set or, in the notation of §8.0.11, satisfy |𝔪[i]| ≤ 2 for all i. If λ(𝔪) is Sp-distinguished then 𝔪 is of Speh type. In particular, if L(𝔪) is Sp-distinguished then 𝔪 is of Speh type. □

9.0.6. We also point out the weaker consequence of Hypothesis 8.6.

Proposition 9.3. Let 𝔪 ∈ O_{ρ} satisfy Hypothesis 8.6. If L(𝔪) is Spdistinguished then 𝔪 is of Speh type.

Proof. If L(𝔪) is Sp-distinguished then L(𝔪^{∨}) is Sp-distinguished by Lemma 3.1. Therefore, by Lemma 2.2, both λ(𝔪) and λ(𝔪^{∨}) are Sp-distinguished. It now follows, as in the proof of Proposition 9.1, that both 𝔪 and 𝔪^{∨} are distinguished in the sense of Definition 8.3. Hypothesis 8.6 therefore implies that 𝔪 is of Speh type. □

9.0.7. We end this section by providing an example which demonstrates that the necessary condition for distinction obtained by combining Propositions 7.5 and 9.3 is not sufficient.

Example 9.4. Let π = L(𝔪) = Z(𝔪^{t}) where

𝔪 = {[ν^{4}, ν^{4}], [ν^{3}, ν^{3}], [ν^{3}, ν^{3}], [ν^{2}, ν^{2}], [ν, ν^{2}], [1, ν]}.

Using the combinatorial algorithm of Mœglin and Waldspurger ([MW86]), it is easy to see that

𝔪^{t} = {[ν^{2}, ν^{3}], [ν, ν^{4}], [1, ν]}.

We will now show that π is not Sp-distinguished. Assume the contrary, if possible. Let π_{1} = Z([ν^{2}, ν^{3}], [1, ν]) and π_{2} = Z([ν, ν^{4}]). The representation π = Z(𝔪^{t}) is the unique irreducible quotient of ζ̃(𝔪^{t}) and so it is also the unique irreducible quotient of π_{1} × π_{2}. Thus, by Lemma 2.2, π_{1} × π_{2} is Spdistinguished. Apply the notation of §3.3.7 with k = 2 for an orbit that is relevant to π_{1} ⊗ π_{2} (by Lemma 5.1). Since k = 2, note that k_{2} ≤ 2. From Corollary 4.5 and (3) it follows that there exist irreducible components σ_{1} of r_{M_{β_{1}} ,G_{n_{1}}} (π_{1}) and σ_{2} of r_{M_{β_{2}} ,G_{n_{2}}} (π_{2}) such that writing

σ_{i} = σ_{i,1} ⊗ ⋯ ⊗ σ_{i,k_{i}}, i = 1, 2, σ_{i,j} ∈ Irr, j = 1, …, k_{i}

we have σ_{ı} = νσ_{τ(ı)} whenever ı ≺ τ(ı) and σ_{ı} is Sp-distinguished if τ(ı) = ı. Also it follows from §7.2.2 that ν^{4} ∈ Supp(σ_{2,k_{2}}). Since ν^{5} ∉ Supp(π_{1}), we deduce that τ(2, k_{2}) = (2, k_{2}). By (5) it follows that k_{2} = k_{1} = 1, and so τ(1, 1) = (1, 1). In other words, Z([ν^{2}, ν^{3}], [1, ν]) is Sp-distinguished. Using the algorithm of Mœglin and Waldspurger again we get that

Z([ν^{2}, ν^{3}], [1, ν]) ≅ L([ν^{3}, ν^{3}], [ν, ν^{2}], [1, 1]).

By Corollary 9.2 we obtain a contradiction.

10. The Sp-distinguished ladder representations

We classify Sp-distinguished representations in the class of ladder representations introduced by Lapid and Mínguez in [LM14].

10.1. Distinction of ladder representations-the L aspect. We classify Sp-distinction in the class of ladder representations defined below.

10.1.1. Ladder representations.

Definition 10.1. Let ρ ∈ Cusp. The set {Δ_{1}, …, Δ_{k}} ∈ O_{ρ} is called a ladder if

b(Δ_{1}) > ⋯ > b(Δ_{k}) and e(Δ_{1}) > ⋯ > e(Δ_{k}).

A representation π ∈ Irr is called a ladder representation if π = L(𝔪) where 𝔪 ∈ O_{ρ} is a ladder .

Whenever we say that 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ρ} is a ladder, we implicitly assume that 𝔪 is already ordered as in the definition above.

10.1.2. The following property allows us to show that certain ladder representations are Sp-distinguished. By convention, let L([a, a − 1]_{(ρ)}) be the trivial representation of the trivial group and let L([a, b]_{(ρ)}) = 0 if b < a − 1. Let 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ρ} be a ladder, with Δ_{i} = [a_{i}, b_{i}]_{(ρ)} and for every i = 1, …, k − 1 let

K_{i} = L(Δ_{1}) × ⋯ × L(Δ_{i−1}) × L([a_{i+1}, b_{i}]_{(ρ)}) × L([a_{i}, b_{i+1}]_{(ρ)})× ×L(Δ_{i+2}) × ⋯ × L(Δ_{k}).

(Thus, K_{i} = 0 if a_{i} > b_{i+1} + 1.) By [LM14, Theorem 1] we have

Theorem 10.2. With the above notation let K be the kernel of the projection λ(𝔪) → L(𝔪). Then K = ∑_{i=1}^{k−1} K_{i}. □

10.1.3. The following is the characterization of Sp-distinguished ladder representations.

Theorem 10.3. Let 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ρ} be a ladder. Then the following conditions are equivalent

(1) L(𝔪) is Sp-distinguished;

(2) k is even and Δ_{2i−1} = νΔ_{2i} for all i = 1, …, k/2;

(3) 𝔪 is of Speh type.

Proof. The equivalence of the last two conditions is obvious. If L(𝔪) is Spdistinguished then 𝔪 is of Speh type by Corollary 9.2. Assume now that k = 2m is even and Δ_{2i−1} = νΔ_{2i} for all i = 1, …, m. Let π_{i} = L(Δ_{2i−1}, Δ_{2i}), i = 1, …, m and π = π_{1} × ⋯ × π_{m}. Note that π is a quotient of λ(𝔪). It follows from Lemma 7.2 that π_{i} is Sp-distinguished for all i = 1, …, m. Therefore, by Corollary 5.3, π is Sp-distinguished and by Lemma 2.2, λ(𝔪) is Sp-distinguished. Apply the notation of §10.1.2. In order to show that L(𝔪) is Sp-distinguished it is enough to show that K is not Sp-distinguished (by Lemma 2.3). By Theorem 10.2 it is enough to show that K_{i} is not Sp-distinguished for all i = 1, …, 2m−1. Note that

𝔪_{i} = {Δ_{1}, …, Δ_{i−1}, [a_{i+1}, b_{i}]_{(ρ)}, [a_{i}, b_{i+1}]_{(ρ)}, Δ_{i+2}, …, Δ_{k}} ∈ O_{ρ}

is ordered by strictly decreasing end points and is therefore, in particular, a set and in standard form. Thus K_{i} ≃ λ(𝔪_{i}). By Corollary 9.2, it is enough to show that 𝔪_{i} is not of Speh type for all i = 1, …, 2m − 1. Assume by contradiction that 𝔪_{i} is of Speh type. Let Δ′_{j} = Δ_{j} for j ≠ i, i + 1, Δ′_{i} = [a_{i+1}, b_{i}]_{(ρ)} and Δ′_{i+1} = [a_{i}, b_{i+1}]_{(ρ)}. Since 𝔪_{i} = {Δ′_{1}, …, Δ′_{2m}} is ordered by strictly decreasing end points we clearly must have Δ′_{2j−1} = νΔ′_{2j} for all j = 1, …, m. If i is odd this implies that a_{i+1} = a_{i} + 1 contradicting the inequality a_{i} > a_{i+1}. If i is even this implies that a_{i−1} = a_{i+1} +1 contradicting the fact that a_{i−1} > a_{i} > a_{i+1} are integers. The theorem follows. □

10.2. Distinction of ladder representations-the Z aspect. The class of ladder representations is closed under Zelevinsky involution. We now reinterpret the classification above in order to characterize the ladders 𝔪 ∈ O_{ρ} so that Z(𝔪) is Sp-distinguished.

10.2.1. Recall that in [MW86], Mœglin and Waldspurger describe a combinatorial algorithm to compute 𝔪^{t} for 𝔪 ∈ O_{ρ}. This algorithm takes a particularly simple form if 𝔪 is a ladder, as described in [LM14, §3.2]. In particular, Lapid and Mínguez observe that 𝔪 ∈ O_{ρ} is a ladder if and only if 𝔪^{t} is a ladder. Thus, Z(𝔪) is a ladder representation for a ladder 𝔪 ∈ O_{ρ} and every ladder representation is of this form for some ladder 𝔪. In §11.0.3 we give a recursive characterization of the Mœglin and Waldspurger algorithm for ladders based on [LM14, §3.2].

10.2.2. The following is another elementary observation that follows from §11.0.3. We omit the simple proof.

Lemma 10.4. Let 𝔪 ∈ O_{ρ} be a ladder and let 𝔪^{t} = {Δ_{1}, …, Δ_{k}} be ordered as a ladder. Then 𝔪 is of Speh type if and only if ℓ(Δ_{i}) is even for all i = 1, …, k and ℓ(Δ_{i} ∩ Δ_{i+1}) is odd for all 1 ≤ i ≤ k − 1 such that Δ_{i} ∪ Δ_{i+1} is a segment. □

10.2.3. Combining Lemma 10.4 and Theorem 10.3 we obtain another classification of Sp-distinguished ladder representations.

Corollary 10.5. Let ρ ∈ Cusp and 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ρ} be a ladder. Then Z(𝔪) is Sp-distinguished if and only if we have

(1) ℓ(Δ_{i}) is even for all i = 1, …, k and

(2) ℓ(Δ_{i} ∩ Δ_{i+1}) is odd for all 1 ≤ i ≤ k − 1 such that Δ_{i} ∪ Δ_{i+1} is a

segment.

□

Remark 10.6. In [OS08b], we obtained a classification of the Sp-distinguished unitary dual in terms of Tadic’s classification ([Tad86]). Recall (see Remark 7.3) that a Speh representation L({Δ, νΔ, …, ν^{n−1}Δ}) is Sp-distinguished if and only if it is even (i.e. n is even). Any unitary representation is a product of Speh representations and it was already proved in [OS07] that a product of even Speh representations is Sp-distinguished. The disjointness of Klyachko models obtained in [OS08b], together with a prescribed model for any irreducible unitary representation [OS08a, Theorem 3.7] imply that if an irreducible product of Speh representations is Sp-distinguished then all Speh representations in the product are even. Based on our current results, we can reprove this implication without reference to Klyachko models as follows. If Δ ⊆ ρ^{ℤ} for ρ ∈ Cusp and ℓ = ℓ(Δ) then it is well known that L({Δ, νΔ, …, ν^{n−1}Δ}) = Z({Δ′, νΔ, …, ν^{ℓ−1}Δ′}) for some segment Δ′ ⊆ ρ^{ℤ} with ℓ(Δ′) = n. If Z(𝔪) = π_{1} × ⋯ × π_{k} ∈ Irr is Sp-distinguished and π_{i} = Z(𝔪_{i}) is a Speh representation for all i = 1, …, k, then 𝔪 = 𝔪_{1} + ⋯ + 𝔪_{k} and it follows from Proposition 7.5 and the results of §5.2 that all segments in 𝔪 are of even length, i.e., that all π_{i}’s are even Speh representations.

10.2.4. Our next goal is to study the Sp-distinguished representations in the class of representations in Irr that are induced from ladder representations. We only obtain a classification of this class conditional to Hypothesis 8.5. Our proof is based on certain combinatorial statements concerning the multi-sets in O_{ρ} that are obtained as sums of ladders in the above context. It is more convenient, to formulate these technical results by unlabeling. We therefore, now use the convention that 𝔪 ∈ O_{ℤ} satisfies a property on O_{ρ} if its ρ-labeling satisfies this property. The next section collects the required results on multisets in O_{ℤ}.

11. On sums of ladders of Speh type

11.0.1. For segments Δ, Δ′ ∈ Sgm, write Δ ≤_{b} Δ′ if either b(Δ) < b(Δ′) or b(Δ) = b(Δ′) and e(Δ) ≤ e(Δ′). Thus, ≤_{b} is a linear order on Sgm.

11.0.2. Let ℓ ∈ ℕ and 𝔪 ∈ O_{ℤ} be such that ℓ(Δ) = ℓ for all Δ ∈ 𝔪. If Δ ∈ 𝔪 is minimal with respect to ≤_{b} then we can express 𝔪 as a linear combination

𝔪 = ∑_{n=0}^{N} a_{n}{ν^{n}Δ}

with a_{0} ∈ ℕ and a_{n} ∈ ℤ_{≥0} for all n = 1, …, N for some large enough N ∈ ℕ.

Lemma 11.1. With the above assumptions and notation, if 𝔪 is of Speh type then

(10)  ∑_{n=0}^{N}(−1)^{N−n}a_{n} = 0.

Proof. Note that

𝔪 = ∑_{n=0}^{N−1} b_{n}({ν^{n}Δ} + {ν^{n+1}Δ}) + b_{N}{ν^{N}Δ}

where b_{0} = a_{0} and b_{n} = ∑_{i=0}^{n}(−1)^{n−i}a_{i}. If 𝔪 is of Speh type, then it follows that b_{n} ≥ 0 for all n and that, as required, b_{N} = 0. □

11.0.3. As observed by Mœglin and Waldspurger the Zelevinsky involution π^{t} = L(𝔪^{t}) of a rigid representation π = L(𝔪) ∈ Irr where 𝔪 ∈ O_{ρ} is ‘blind’ to the cuspidal line ρ^{ℤ} where it is supported. Indeed, the Moeglin-Waldspurger algorithm defines an involution on O_{ℤ} that gives 𝔪 ↦ 𝔪^{t} via ρ-labeling. We now explicate a recursive characterization of the Moeglin-Waldspurger algorithm of the Zelevinsky involution for ladders based on [LM14, §3.2]. For Δ = [a, b] ∈ Sgm we have {Δ}^{t} = ∑_{c=a}^{b}{c} = {{b}, …, {a}}. Let 𝔪 = {Δ_{1}, …, Δ_{k}, Δ_{k+1}} be a ladder and let 𝔪′ = {Δ_{1}, …, Δ_{k}}. Write Δ_{i} = [a_{i}, b_{i}]. Let (𝔪′)^{t} = {Δ′_{1}, …, Δ′_{s}} be ordered as a ladder. If b_{k+1} + 1 < a_{k} then 𝔪^{t} = (𝔪′)^{t} + {Δ_{k+1}}^{t} and therefore

𝔪 = {Δ′_{1}, …, Δ′_{s}, {b_{k+1}}, …, {a_{k+1}}}

is ordered as a ladder. Otherwise, let c = s − (b_{k+1} − a_{k} + 2). Then c ≥ 0 and

(11)  𝔪^{t} = {Δ′_{1}, …, Δ′_{c},^{+}Δ′_{c+1}, …,^{+}Δ′_{s}, {a_{k} − 2}, …, {a_{k+1}}}

where ^{+}[a, b] = [a − 1, b]. In other words, in order to obtain the ladder 𝔪^{t} from (𝔪′)^{t} one has to perform the following steps. If Δ_{k+1} ⊀ Δ_{k} then add to 𝔪′ at the tail of the ladder, the ladder {Δ_{k+1}}^{t}, i.e., the ℓ(Δ_{k+1}) length one segments consisting of elements of Δ_{k+1} in decreasing order. Otherwise, b_{k+1} − a_{k} + 2 ≥ 1. Starting with (𝔪′)^{t}, replace Δ by ^{+}Δ for each of the last b_{k+1} − a_{k} + 2 segments of (𝔪′)^{t} and then add at the tail of the resulting ladder, the a_{k} − a_{k+1} − 1 length one segments consisting of elements of [a_{k+1}, a_{k} − 2] in decreasing order.

11.0.4. Let 𝔪_{1}, …, 𝔪_{k} ∈ O_{ℤ} be ladders and let Δ_{0} be minimal in 𝔪 = 𝔪_{1} + ⋯ + 𝔪_{k} with respect to ≤_{b}. Let 𝔪_{i}^{†} = 𝔪_{i} if Δ_{0} ∉ 𝔪_{i} and 𝔪_{i}^{†} = 𝔪_{i} − {Δ_{0}} otherwise.

Lemma 11.2. With the above notation, suppose that the ladder (𝔪_{i}^{†})^{t} is of Speh type for all i = 1, …, k and that ℓ(Δ_{0}) is even. If 𝔫 = 𝔪_{1}^{t} + ⋯ + 𝔪_{k}^{t} is of Speh type then 𝔪_{i}^{t} is of Speh type for all i = 1, …, k.

Proof. If 𝔪_{i} = 𝔪_{i}^{†} then 𝔪_{i}^{t} is of Speh type. If 𝔪_{i} ≠ 𝔪_{i}^{†}, let Δ_{i} be the minimal segment in 𝔪_{i}^{†} with respect to ≤_{b}. If Δ_{0} ⊀ Δ_{i} then it follows from §11.0.3 that 𝔪_{i}^{t} = (𝔪_{i}^{†})^{t} + {Δ_{0}}^{t}. Since ℓ(Δ_{0}) is even it follows that 𝔪_{i}^{t} is of Speh type. If Δ_{0} ≺ Δ_{i}, then Δ_{0} ∪ Δ_{i} is a segment and by Lemma 10.4 (applied once with 𝔪 = 𝔪_{i}^{t} and once with 𝔪 = (𝔪_{i}^{†})^{t}), 𝔪_{i}^{t} is of Speh type if and only if ℓ(Δ_{0} ∩ Δ_{i}) is odd. Let A = {1 ≤ i ≤ k : 𝔪_{i}^{t} is not of Speh type} and 𝔫 = 𝔫_{1} + 𝔫_{2} where 𝔫_{1} = ∑_{i∈A} 𝔪_{i}^{t}. Assume by contradiction, that 𝔫 is of Speh type and 𝔫_{1} ≠ 0 (i.e. A ≠ ∅). Let i ∈ A and c_{i} = s_{i} − (e(Δ_{0}) − b(Δ_{i}) + 2) where s_{i} = |(𝔪_{i}^{†})^{t}|. Note that by the above remarks, s_{i} − c_{i} = e(Δ_{0}) − b(Δ_{i}) + 2 = ℓ(Δ_{0} ∩ Δ_{i}) + 1 ≥ 1 is odd. Since (𝔪_{i})^{t} is of Speh type, s_{i} is even, hence c_{i} is also odd. Furthermore, d_{i} = b(Δ_{i}) − b(Δ_{0}) − 1 is odd and by (11) 𝔪_{i}^{t} ordered as a ladder has the form

(12) 𝔪_{i}^{t} = {νΔ′_{1}, Δ′_{1}, …, νΔ′_{(c_{i}−1)/2}, Δ′_{(c_{i}−1)/2}, νΔ′_{(c_{i}+1)/2},^{+}Δ′_{(c_{i}+1)/2},

νΔ′′_{1}, Δ′′_{1}, …, νΔ′′_{(s_{i}−c_{i}−1)/2}, Δ′′_{(s_{i}−c_{i}−1)/2}, {x_{0} + d_{i} − 1}, …, {x_{0} + 1}, {x_{0}}}

where ℓ(Δ′′_{i}) > 1 for i = 1, …, (s_{i} − c_{i} − 1)/2 and x_{0} = b(Δ_{0}). Note further that b(^{+}Δ′_{(c_{i}+1)/2})) = e(Δ_{0}) is independent of i ∈ A. Thus, we may decompose

𝔪_{i}^{t} = 𝔞_{i} + {νΓ_{i},^{+}Γ_{i}} + 𝔟_{i} + {x_{0}}

where 𝔞_{𝔦} and 𝔟_{i} are of Speh type, b(Δ) > e(Δ_{0})+2 for all Δ ∈ 𝔞_{i}, b(Δ) < e(Δ_{0}) for all Δ ∈ 𝔟_{i} and Γ_{i} ∈ Sgm is such that b(^{+}Γ_{i}) = e(Δ_{0}) and therefore also b(νΓ_{i}) = e(Δ_{0}) + 2. In particular, b(Δ) ≠ e(Δ_{0}) + 1 for all Δ ∈ 𝔪_{i}^{t}. For ℓ ∈ ℕ and a multi-set 𝔞 ∈ O_{ℤ} let 𝔞(ℓ) = δ(ℓ) · 𝔞 where δ(ℓ) is the characteristic function of all segments of length ℓ. Clearly, 𝔞 is of Speh type if and only if 𝔞(ℓ) is of Speh type for all ℓ ∈ ℕ. Fix i_{0} ∈ A and let ℓ = ℓ(^{+}Γ_{i_{0}}) > 1, B = {i ∈ A : ℓ = ℓ(^{+}Γ_{i})} and C = {i ∈ A : ℓ = ℓ(νΓ_{i})}. Since ℓ(^{+}Γ_{i}) = ℓ(νΓ_{i}) + 1, B and C are disjoint. We have

𝔪_{i}^{t}(ℓ) = \begin{cases}𝔞_{i}(ℓ) + \{^{+}Γ_{i}\} + 𝔟_{i}(ℓ) & i ∈ B \\ 𝔞_{i}(ℓ) + \{νΓ_{i}\} + 𝔟_{i}(ℓ) & i ∈ C \\ 𝔞_{i}(ℓ) + 𝔟_{i}(ℓ) & i ∈ A \ (B ∪ C). \end{cases}

Set Δ = ^{+}Γ_{i_{0}} and note further that for i ∈ B we have ^{+}Γ_{i} = Δ and for i ∈ C we have νΓ_{i} = ν^{2}Δ. It follows that

𝔫(ℓ) = 𝔫_{1}(ℓ) + 𝔫_{2}(ℓ) = (|B| {Δ} + |C| {ν^{2}Δ}) + ∑_{i∈A}(𝔞_{i}(ℓ) + 𝔟_{i}(ℓ)) + 𝔫_{2}(ℓ).

By assumption 𝔫(ℓ) and ∑_{i∈A}(𝔞_{i}(ℓ) + 𝔟_{i}(ℓ)) + 𝔫_{2}(ℓ) are both of Speh type and therefore, by Lemma 11.1, each of them satisfies the linear condition (10). It follows that |B| {Δ} + |C| {ν^{2}Δ} satisfies the same linear condition, i.e., that |B| + |C| = 0. But since i_{0} ∈ B this is a contradiction. □

12. On distinction of representations induced from ladder

We now study distinction in the class of representations in Irr that are induced from ladder representations. For a product of more than two ladder representations, our results are only conditional on Hypothesis 8.6.

12.0.1. We recall [LM, Lemma 5.17]. It reduces the reducibility of a product of ladder representations to induction from a maximal parabolic.

Lemma 12.1. Let π_{1}, …, π_{k} be ladder representations and let π = π_{1} ×⋯×π_{k}. Then π ∈ Irr if and only if π_{i} × π_{j} ∈ Irr for all i < j. □

12.0.2. For a maximal parabolic, the following criterion is a combination of [LM, Proposition 5.21 and Lemma 5.22].

Definition 12.2. Let 𝔪 = {Δ_{1}, …, Δ_{t}} and 𝔫 = {Δ′_{1}, …, Δ′_{s}} be ladders in O_{ρ}. We say that the condition NC(𝔪, 𝔫) is satisfied if there exist k ≥ 0, 1 ≤ i ≤ t and 1 ≤ j ≤ s such that i + k ≤ t, j + k ≤ s and the following properties are satisfied:

(1) Δ_{i+l} ≺ Δ′_{j+l} for all l = 0, …, k;

(2) ν^{−1}Δ_{i−1} ⊀ Δ′_{j} if i > 1;
(3) ν^{−1}Δ_{i+k} ⊀ Δ′_{j+k+1} if j + k + 1 ≤ s.

Proposition 12.3. In the above notation Z(𝔪) × Z(𝔫) ∈ Irr if and only if neither NC(𝔪, 𝔫) nor NC(𝔫, 𝔪) hold.

The main result of this section requires some preparation.

12.0.3. For segments Δ, Δ′ ∈ O_{ρ}, write Δ ≤_{b} Δ′ if either b(Δ) < b(Δ′) or b(Δ) = b(Δ′) and e(Δ) ≤ e(Δ′). Thus, ≤_{b} is a linear order on O_{ρ}.

Lemma 12.4. Let 𝔪_{1}, …, 𝔪_{k} ∈ O_{ρ} be ladders such that Z(𝔪_{1}) × ⋯ × Z(𝔪_{k}) ∈ Irr. Let Δ_{0} be a minimal segment in 𝔪_{1} + ⋯ + 𝔪_{k} with respect to ≤_{b} and let

𝔪_{i}^{†} = \begin{cases}𝔪_{i} − \{Δ_{0}\} & Δ_{0} ∈ 𝔪_{i} \\ 𝔪_{i} & otherwise. \end{cases}

Then Z(𝔪_{1}^{†}) × ⋯ × Z(𝔪_{k}^{†}) ∈ Irr.

Proof. By Lemma 12.1, it is enough to prove the lemma for the case k = 2. Note that if Δ_{0} ∉ 𝔪_{i}, i = 1, 2, then 𝔪_{i} = 𝔪_{i}^{†} and we have nothing to prove. Thus we assume that Δ_{0} belongs to at least one of the two multi-sets. By the symmetry of the irreducibility criterion of Proposition 12.3, we may assume without loss of generality that Δ_{0} ∈ 𝔪_{1}. Write 𝔪_{1}^{†} = {Δ_{1}, …, Δ_{t}} and 𝔪_{2}^{†} = {Δ′_{1}, …, Δ′_{s}} ordered as ladders. Note that 𝔪_{1} = {Δ_{1}, …, Δ_{t}, Δ_{0}} is ordered as a ladder. Assume by contradiction that Z(𝔪_{1}^{†}) × Z(𝔪_{2}^{†}) reduces. It follows from Proposition 12.3 that either NC(𝔪_{1}^{†}, 𝔪_{2}^{†}) or NC(𝔪_{2}^{†}, 𝔪_{1}^{†}) is satisfied. We separate into two cases and show that in each case this implies that either NC(𝔪_{1}, 𝔪_{2}) or NC(𝔪_{2}, 𝔪_{1}) holds. Since this is a contradiction to Proposition 12.3 the lemma will follow. Consider first the case that Δ_{0} ∉ 𝔪_{2} (i.e., 𝔪_{2} = 𝔪_{2}^{†}) and NC(𝔪_{1}^{†}, 𝔪_{2}^{†}) holds. Let i, j, k be the indices satisfying (1)-(3) of Definition 12.2 for (𝔪_{1}^{†}, 𝔪_{2}^{†}). Then the same indices i, j, k show that NC(𝔪_{1}, 𝔪_{2}) holds. Next consider the case Δ_{0} ∈ 𝔪_{2} or NC(𝔪_{1}^{†}, 𝔪_{2}^{†}) doesn’t hold. If Δ_{0} ∉ 𝔪_{2} then by assumption NC(𝔪_{2}^{†}, 𝔪_{1}^{†}) holds. If Δ_{0} ∈ 𝔪_{2} then by symmetry between 𝔪_{1} and 𝔪_{2}, without loss of generality, we may also assume that NC(𝔪_{2}^{†}, 𝔪_{1}^{†}) holds. If the indices i, j, k satisfy (1)-(3) of Definition 12.2 for (𝔪_{2}^{†}, 𝔪_{1}^{†}) then for the same indices (1) and (2) are automatic for (𝔪_{2}, 𝔪_{1}), while (3) is automatic unless j + k = t in which case (3) reads ν^{−1}Δ′_{i+k} ⊀ Δ_{0}. By the minimality of Δ_{0} and since Δ_{0} ∉ 𝔪_{2}^{†} it follows that ν^{−1}Δ′ ⊀ Δ_{0} for all Δ′ ∈ 𝔪_{2}^{†} and in particular for Δ′ = Δ′_{i+k}. It therefore follows that NC(𝔪_{2}, 𝔪_{1}) holds. □

12.0.4. Define an operation 𝔪 ↦ 𝔪′ on O_{ρ} as follows. For 𝔪 ∈ O_{ρ}, let Δ_{0} be minimal in 𝔪 with respect to ≤_{b} and 𝔪′ = 𝔪 − 𝔪(Δ_{0}){Δ_{0}}.

Proposition 12.5. Let Ω ⊆ O_{ρ} be a subset closed under the operation 𝔪 ↦ 𝔪′ and such that Hypothesis 8.6 holds for 𝔪^{t} for all 𝔪 ∈ Ω. Let π_{1}, …, π_{k} be ladder representations such that π = π_{1} × ⋯ × π_{k} ∈ Irr. Let 𝔪 ∈ O_{ρ} be such that π = Z(𝔪) and assume that 𝔪 ∈ Ω. If π is Sp-distinguished then π_{i} is Sp-distinguished for all i = 1, …, k.

Proof. Let 𝔪_{1}, …, 𝔪_{k} be ladders such that π_{i} = Z(𝔪_{i}), i = 1, …, k. Since π is irreducible, we have 𝔪 = 𝔪_{1} + ⋯ + 𝔪_{k}. The proof is by induction on |𝔪|. For 𝔪 = 0 there is nothing to prove. Let Δ_{0} be the minimal segment in 𝔪 with respect to ≤_{b}. Let 𝔫_{0} = 𝔪(Δ_{0}){Δ_{0}} so that 𝔪 = 𝔪′ + 𝔫_{0} and let 𝔪_{i}^{†} = min{𝔪_{i}, 𝔪′}, i = 1, …, k. Note that

𝔪_{i}^{†} = \begin{cases}𝔪′_{i} & Δ_{0} ∈ 𝔪_{i} \\ 𝔪_{i} & otherwise \end{cases}

and 𝔪′ = 𝔪_{1}^{†} + ⋯ + 𝔪_{k}^{†}. By the definition of 𝔪, π is the unique irreducible quotient of ζ̃(𝔪). Since no segment in 𝔪′ precedes Δ_{0} we have ζ̃(𝔪) = ζ(𝔫_{0}) × ζ̃(𝔪′) and therefore π is also the unique irreducible quotient of Z(𝔫_{0}) × Z(𝔪′). Thus, by Lemma 2.2, Z(𝔫_{0}) × Z(𝔪′) is Sp-distinguished. Apply the notation of §3.3.7 with k = 2 for an orbit that is relevant to Z(𝔫_{0}) ⊗ Z(𝔪′) (by Lemma 5.1). It follows from Corollary 4.5 and (3) that there exist irreducible components σ_{1} of r_{M_{β_{1}} ,G_{n_{1}}} (Z(𝔫_{0})) and σ_{2} of r_{M_{β_{2}} ,G_{n_{2}}} (Z(𝔪′)) such that writing

σ_{i} = σ_{i,1} ⊗ ⋯ ⊗ σ_{i,k_{i}}, i = 1, 2, σ_{i,j} ∈ Irr, j = 1, …, k_{i}

we have σ_{ı} = νσ_{τ(ı)} whenever ı ≺ τ(ı) and σ_{ı} is Sp-distinguished if τ(ı) = ı. By [Zel80, Theorem 4.2] we have

Z(𝔫_{0}) = \overbrace{Z(Δ_{0}) × ⋯ × Z(Δ_{0})}^{𝔪(Δ_{0})−times} .

In particular, it follows from the geometric lemma of Bernstein and Zelevinsky [BZ77, §2.12] and §7.2.3 that σ_{1,1} = Z(Δ) for some segment satisfying b(Δ) = b(Δ_{0}). If τ(1, 1) ≠ (1, 1) then, by (5) and the fact that k = 2, we must have τ(1, 1) = (2, k_{2}) and therefore ν^{−1}σ_{1,1} = σ_{2,k_{2}}. But since Supp(σ_{(2,k_{2})}) ⊆ Supp(σ_{2}) ⊆ Supp(Z(𝔪′)) (see (4)) we have ν^{−1}b(Δ_{0}) ∈ Supp(ν^{−1}σ_{(1,1)}) \ Supp(σ_{(2,k_{2})}) which is a contradiction. It follows that τ(1, 1) = (1, 1) and since k = 2, τ must be trivial. In other words, both Z(𝔫_{0}) and Z(𝔪′) are Sp-distinguished. It follows from Proposition 7.5 that ℓ(Δ_{0}) is even. Let π′_{i} = Z(𝔪_{i}^{†}), i = 1, …, k. It follows from Lemma 12.4 that π′ = π′_{1} × ⋯ × π′_{k} ∈ Irr and therefore π′ = Z(𝔪′).

By the assumption on Ω and the induction hypothesis, π′_{i} = L((𝔪_{i}^{†})^{t}) is Spdistinguished for all i = 1, …, k. By Theorem 10.3, (𝔪_{i}^{†})^{t} is of Speh type for i = 1, …, k. Since π = L(𝔪^{t}) is Sp-distinguished and 𝔪^{t} satisfies Hypothesis 8.6, by Proposition 9.3, 𝔪^{t} is of Speh type. It now follows from Lemma 11.2 that 𝔪_{i}^{t} is of Speh type and therefore, again by Theorem 10.3, that π_{i} = L(𝔪_{i}^{t}) is Sp-distinguished for all i = 1, …, k. □

12.0.5. Let Ω_{k} be the set of all 𝔪 ∈ O_{ρ} that are obtained as sums of at most k ladders, i.e. 𝔪 = 𝔪_{1} + ⋯ + 𝔪_{k} where 𝔪_{i} is either zero or a ladder, and such that Z(𝔪_{1}) × ⋯ × Z(𝔪_{k}) ∈ Irr. Since both ladder representations and Irr are closed under Zelevinsky involution and in the Grothendick group it is multiplicative, it follows that Ω_{k} is closed under Zelevinsky involution. It further follow from Lemma 12.4 that Ω_{k} is closed under the operation 𝔪 ↦ 𝔪′ defined in §12.0.4. For a product of two ladder representations this gives the following unconditional result.

Corollary 12.6. Let π_{1} and π_{2} be ladder representations such that π = π_{1} × π_{2} ∈ Irr. If π is Sp-distinguished then π_{1} and π_{2} are Sp-distinguished.

Proof. Note that |𝔪[i]| ≤ 2 for all i and all 𝔪 ∈ Ω_{2}. Since, as remarked above, Ω_{2} is closed under Zelevinsky involution, it follows from Proposition 8.8 that 𝔪^{t} satisfies Hypothesis 8.5 for all 𝔪 ∈ Ω_{2}. Since we also observed above that Ω_{2} is closed under the operation 𝔪 ↦ 𝔪′ defined in §12.0.4, the statement follows from Proposition 12.5. □

12.0.6. We conclude this section with an example of a family of imprimitive, Sp-distinguished representations that are not ladders.

Definition 12.7. Let F denote a set of irreducible representations of the form Z(𝔪) such that the multi-set 𝔪 = {Δ_{1}, Δ_{2}, Δ_{3}} satisfies the following properties

(1) ℓ(Δ_{i}) is even for all i,

(2) Δ_{1} ⊆ νΔ_{2} and Δ_{1} ⊆ ν^{−1}Δ_{2},

(3) ℓ(Δ_{3} ∩ Δ_{1}) and ℓ(Δ_{3} ∩ Δ_{2}) are both odd.

Note that (2) implies that Δ_{1} ⊆ Δ_{2} which in particular implies that none of these representations are ladders. Further note that F consists of only rigid representations and π ∈ F if and only if π^{∨} ∈ F. The conditions on the length of the segments in 𝔪 imply that the pairs {Δ_{1}, Δ_{3}} and {Δ_{2}, Δ_{3}} are linked. A simple example of a representation in F is Z([ν^{3}, ν^{4}], [ν, ν^{6}], [1, ν^{3}]). The next lemma shows that indeed any representation in F has the desired properties. Before we proceed, recall that an elementary operation on an arbitrary multi-set 𝔪 is to choose a pair of linked segments in it and replace the pair by their union and their intersection. By [Zel80, Theorem 7.1] any irreducible subquotient of ζ(𝔪) is of the form Z(𝔫) where 𝔫 is a multi-set obtained from 𝔪 by a sequence of elementary operations on it.

Lemma 12.8. Let π ∈ F. Then π is Sp-distinguished and imprimitive.

Proof. Let 𝔪 = {Δ_{1}, Δ_{2}, Δ_{3}} be a multi-set satisfying the conditions (1), (2) and (3) of Definition 12.7 and π = Z(𝔪). By Corollary 10.5, Z(Δ_{i}) is Spdistinguished, i = 1, 2, 3 and therefore by Corollary 5.3,

I(𝔪) = Z(Δ_{1}) × Z(Δ_{2}) × Z(Δ_{3})

is Sp-distinguished. Applying [Zel80, Theorem 1.9] to ζ(𝔪) we get that π occurs as a subquotient of I(𝔪). We now analyze the other possible irreducible subquotients of I(𝔪) using [Zel80, Theorem 1.9 and Theorem 7.1]. Since Δ_{1} ⊆ Δ_{2}, any elementary operation on 𝔪 is performed on either {Δ_{1}, Δ_{3}} or on {Δ_{2}, Δ_{3}}. The result will respectively contain either Δ_{1} ∩ Δ_{3} or Δ_{2} ∪ Δ_{3}, which are of odd length. Since the first is contained in and the second contains all three segments any further sequence of operations will result in a multi-set containing one of them. Thus by Proposition 7.5 none of these subquotients are Sp-distinguished. The Sp-distinction of π now follows from Lemma 2.3. Next we show that π is imprimitive. Assume, if possible, that it is not so. Then there exists indices i, j, k such that {i, j, k} = {1, 2, 3} and π ≅ Z(Δ_{i}) × Z(Δ_{j}, Δ_{k}). By considering the multi-set 𝔪^{∨} instead of 𝔪 if required, assume further that Δ_{3} ≺ Δ_{2} and hence Δ_{3} ≺ Δ_{1}. Note that Z(Δ_{1}, Δ_{2}) × Z(Δ_{3}) ≅ I(𝔪) which is reducible by [Zel80, Theorem 4.2]. Thus π ≅ Z(Δ_{i}, Δ_{3}) × Z(Δ_{j}) where {i, j} = {1, 2}. It follows from Proposition 12.3 and condition (2) of Definition 12.7 that this product is reducible which is a contradiction. □

13. On distinction by Klyachko subgroups

We continue the study of Klyachko models for representations of GL_{n}(F), following [HR90], [OS08a] and [OS08b]. Over finite fields Klyachko models were introduced in [Kly83]. In that case, it is a disjoint family of models and their direct sum contains every irreducible representation with multiplicity one [IS91]. Over a non-archimedean local field, Heumos and Rallis observed that some (non-unitary) representations do not admit a Klyachko model. The second and the third authors showed that the direct sum of all Klyachko models is multiplicity free and prescribed a model to any representation in the unitary dual. In this section, we reduce the study of Klyachko models on the admissible dual to rigid representations and prove that models behave well with respect to parabolic induction.

13.1. The Klyachko model setting.

13.1.1. Let G = G_{n}. For a decomposition n = 2k + r let

H_{2k,r} = \{(\begin{matrix}h & X \\ 0 & u \end{matrix}) : h ∈ Sp_{2k}(F), X ∈ M_{2k×r}(F), u ∈ N_{r}\}

and ψ = ψ_{2k,r} be defined by

ψ((\begin{matrix}h & X \\ 0 & u \end{matrix})) = ψ(u).

(See §2.3.4 for the definition of N_{r} and its character ψ.) For any π ∈ Π, being (H_{2k,r}, ψ)-distinguished is independent of the choice of non-trivial character ψ of F. Indeed, for any other character ψ′ ≠ 1 there is a diagonal matrix a ∈ G normalizing H_{2k,r} such that ψ′_{2k,r}(h) = ψ_{2k,r}(aha^{−1}), h ∈ H_{2k,r}.

13.1.2. Let τ be the involution on G defined by g^{τ} = w^{−1}^{t}g^{−1}w where w =

( \begin{matrix}0 & I_{r} \\ I_{2k} & 0 \end{matrix} ) and let

H′_{r,2k} = H_{2k,r}^{τ} = \{(\begin{matrix}u & X \\ 0 & h \end{matrix}) : h ∈ Sp_{2k}(F), X ∈ M_{r×2k}(F), u ∈ N_{r}\}.

In [HR90], [OS08a] and [OS08b] we studied distinction by (H′_{r,2k}, ψ). Clearly, π ∈ Π is (H_{2k,r}, ψ)-distinguished if and only if π^{τ} is (H′_{r,2k}, ψ)-distinguished. If π ∈ Irr then π^{τ} ≃ π^{∨}, by [GK75], and we get a natural isomorphism

(13)  Hom_{H_{2k,r}}(π, ψ) ≃ Hom_{H′_{r,2k}}(π^{∨}, ψ).

In particular, π is (H_{2k,r}, ψ)-distinguished if and only if π^{∨} is (H′_{r,2k}, ψ)- distinguished. More generally, if π_{1}, …, π_{t} ∈ Irr then (π_{1} × ⋯ × π_{t})^{τ} ≃ π_{t}^{∨} × ⋯ × π_{1}^{∨} and therefore

(14)  Hom_{H_{2k,r}}(π_{1} × ⋯ × π_{t}, ψ) ≃ Hom_{H′_{r,2k}}(π_{t}^{∨} × ⋯ × π_{1}^{∨}, ψ).

In particular, π_{1} ×⋯×π_{t} is (H_{2k,r}, ψ)-distinguished if and only if π_{t}^{∨} ×⋯×π_{1}^{∨} is (H′_{r,2k}, ψ)-distinguished.

Remark 13.1. We remark that the proof of [OS08a, Theorem 3.7] applied [OS08a, Lemma 3.1], where (13) was mistakenly formulated for any representation π. The isomorphism (14) suffices to fill the gap. In any case, we provide in the sequel an independent generalization of [OS08a, Theorem 3.7].

13.1.3. Let π ∈ Irr ∩ Π(G_{n}). If π is (H_{2k,r}, ψ)-distinguished for some decomposition n = 2k + r then by (1) it imbeds in Ind_{H_{2k,r}}^{G_{n}}(ψ) and we say that it admits a Klyachko model. By the uniqueness and disjointness of Klyachko models, [OS08b, Theorem 1], both the imbedding (up to a constant multiple) and the decomposition n = 2k+r are uniquely determined by π. (Indeed, in the main result on distributions [OS08b, Proposition 1] implying [OS08b, Theorem 1] H_{2k,r} and H′_{r,2k} are in symmetry.) In that case we denote by

r(π) = r

the Klyachko type of π.

13.1.4. The main tool in our study of Klyachko models is the theory of derivatives of representations of G_{n} developed in [GK75], [BZ76], [BZ77] and [Zel80]. It allows a reduction of many of the problems concerned with Klyachko models to the study of Sp-distinction and generic representations.

For π ∈ Π(G_{n}) and any r = 0, 1, …, n we denote by π^{(r)} the r-th derivative of π as defined in [BZ77, §3.5 and §4.3]. It is a functor from Π(G_{n}) to Π(G_{n−r}). As a consequence of [BZ77, Lemma 4.7(a)] we have

Lemma 13.2. Let π ∈ Π(G_{n}). Then, Supp(π^{(r)}) ⊆ Supp(π) for all 0 ≤ r ≤ n. □

13.1.5. As observed in [OS08a, (3.2)], it follows from [Zel80, Proposition 3.7] that for n = 2k + r and π ∈ Π(G_{n}) there is a natural linear isomorphism

(15)  Hom_{H_{2k,r}}(π, ψ) ≃ Hom_{Sp_{2k}(F)}(π^{(r)}, 1).

This is the reason that we prefer H_{2k,r} to H′_{r,2k}. Note that, in particular, π is generic if and only if π^{(n)} ≠ 0.

13.2. Hereditary property of Klyachko models. As we observe below, Klyachko models behave well with respect to parabolic induction.

Proposition 13.3. Let π_{i} ∈ Π(G_{n_{i}}) and n_{i} = 2k_{i} + r_{i} be such that π_{i} is (H_{2k_{i},r_{i}}, ψ)-distinguished for i = 1, …, t. Then π = π_{1} × ⋯ × π_{t} is (H_{2k,r}, ψ)- distinguished where k = k_{1} + ⋯ + k_{t} and r = r_{1} + ⋯ + r_{t}.

Proof. Induction on t reduces the statement to the case t = 2 that we now assume. Let π_{s} = ν^{s}π_{1} × π_{2} for s ∈ ℂ, so that π = π_{0}. Recall that by the Leibnitz rule, [BZ77, Lemma 4.5], π_{s}^{(r)} admits a filtration with factors (ν^{s}π_{1})^{(i)} × π_{2}^{(r−i)} ≃ ν^{s}π_{1}^{(i)} × π_{2}^{(r−i)}, i = 0, …, r. Note, that there exists a small enough punctured neighborhood U of s = 0, so that for i ≠ j the central characters of the (finitely many) irreducible components of ν^{s}π_{1}^{(i)} × π_{2}^{(r−i)} and of ν^{s}π_{1}^{(j)} × π_{2}^{(r−j)} are disjoint. It follows that for s ∈ U we have,

π_{s}^{(r)} ≃ ⊕_{i=0}^{r}(ν^{s}π_{1}^{(i)} × π_{2}^{(r−i)}).

In fact, when realizing all π_{s} in the representation space of π, this direct sum decomposition is independent of s ∈ U. It follows, that there is a meromorphic map P_{s} : π_{s}^{(r)} → ν^{s}π_{1}^{(r_{1})} × π_{2}^{(r_{2})}, that is surjective for s ∈ U. By (15), π_{i}^{(r_{i})} is Sp_{2k_{i}}(F)-distinguished, i = 1, 2 and therefore, by Lemma 5.2, there exists a non-zero, meromorphic family of linear forms ℓ_{s} such that in a possibly smaller punctured neighborhood U_{0} of s = 0 we have ℓ_{s} ∈ Hom_{Sp_{2k}(F)}(ν^{s}π_{1}^{(r_{1})} × π_{2}^{(r_{2})}, 1). Therefore, ℓ_{s} ∘ P_{s} ∈ Hom_{Sp_{2k}(F)}(π_{s}^{(r)}, 1) is non-zero for s ∈ U_{0}. As in Corollary 5.3, a leading term argument implies that π^{(r)} = π_{0}^{(r)} is Sp_{2k}(F)-distinguished and therefore, by (15), π is (H_{2k,r}, ψ)- distinguished.

□

13.3. Reduction to cuspidal lines. We reduce the study of (H_{2k,r}, ψ)- distinguished representations in Π to rigid representations, in fact, more generally to totally disjoint supports (Definition 5.7).

Proposition 13.4. Let π_{i} ∈ Π, i = 1, …, t be such that Supp(π_{i}) and Supp(π_{j}) are totally disjoint for all i ≠ j. Then π = π_{1}×⋯×π_{t} admits a Klyachko model if and only if π_{i} admits a Klyachko model for all i = 1, …, t. More precisely, π is (H_{2k,r}, ψ)-distinguished if and only if π_{i} is (H_{2k_{i},r_{i}}, ψ)-distinguished, i = 1, …, t for some decomposition r = r_{1} + ⋯ + r_{t}.

Proof. The ‘if’ part is immediate from Proposition 13.3. Assume that π ∈ Π(G_{n}), n = 2k + r and π is (H_{2k,r}, ψ)-distinguished. By (15), π^{(r)} is Sp_{2k}(F)- distinguished. Therefore, by the Leibnitz rule, [BZ77, Lemma 4.5], and Lemma 2.3, there exists a decomposition r = r_{1} + ⋯ + r_{t} such that π_{1}^{(r_{1})} × ⋯ × π_{t}^{(r_{t})} is Sp_{2k}(F)-distinguished. Since, by Lemma 13.2, Supp(π_{i}^{(r_{i})}) ⊆ Supp(π_{i}) it now follows from Lemma 5.9 that π_{i}^{(r_{i})} is Sp-distinguished for all i = 1, …, t. The lemma now follows from (15). □

14. Klyachko models for ladder representations

We classify all ladder representations that admit, any given, Klyachko model.

14.1. Klyachko models for proper ladders.

14.1.1. Proper ladders.

Definition 14.1. A ladder, 𝔪 = {Δ_{1}, …, Δ_{k}} ∈ O_{ρ} is called a proper ladder if Δ_{i+1} ≺ Δ_{i}, i = 1, …, k − 1. If 𝔪 is a proper ladder then L(𝔪) is called a proper ladder representation.

In fact, if 𝔪 ∈ O_{ρ} is a proper ladder then 𝔪^{t} is also a proper ladder, hence Z(𝔪) is a proper ladder representation, but this fact will not be used in the sequel.

14.1.2. Note that if 𝔪 ∈ O_{ρ} is a ladder then it can be decomposed uniquely (up to order) as a sum 𝔪 = 𝔪_{1} + ⋯ + 𝔪_{t} where 𝔪_{i} is a proper ladder for all i = 1, …, t and Δ ⊀ Δ′ for all i ≠ j, Δ ∈ 𝔪_{i} and Δ′ ∈ 𝔪_{j}. Therefore, Supp(L(𝔪_{i})) and Supp(L(𝔪_{j})) are totally disjoint for all i ≠ j and, by Lemma 5.8,

L(𝔪) = L(𝔪_{1}) × ⋯ × L(𝔪_{t}).

In other words, any ladder representation is a product of proper ladder representations uniquely determined up to order.

14.1.3. Right aligned segments. We define the following relation on segments of cuspidal representations.

Definition 14.2. For segments Δ = [a, b]_{(ρ)} and Δ′ = [a′, b′]_{(ρ)} we say that Δ′ is right-aligned with Δ and write Δ′ ⊢ Δ if

• a ≥ a′ + 1 and

• b = b′ + 1.

We label this relation by the integer r = d(a − a′ − 1) where ρ ∈ Π(G_{d}) and write Δ′ ⊢_{r} Δ.

Note, in particular, that Δ′ ⊢_{0} Δ means that Δ = νΔ′.

Example 14.3. Let Δ = [4, 7]_{(ρ)} and Δ′ = [0, 6]_{(ρ)} be segments. Then

∘^{4}  ∘^{5}  ∘^{6}  ∘^{7} ∘^{0}  ∘^{1}  ∘^{2}  ∘^{3}  ∘^{4}  ∘^{5}  ∘^{6}

illustrates the relation Δ′ ⊢_{3d} Δ if ρ ∈ Π(G_{d}).

14.1.4. Before characterizing the proper ladder representations admitting Klyachko models we need the following technical result.

Lemma 14.4. Let d be such that ρ ∈ Π(G_{d}) and 𝔪 = {Δ_{1}, …, Δ_{t}} ∈ O_{ρ} a proper ladder. Write Δ_{i} = [a_{i}, b_{i}]_{(ρ)}. Suppose that c_{1} > ⋯ > c_{t} are integers such that a_{i} − 1 ≤ c_{i} ≤ b_{i}, i = 1, …, t and let 𝔪_{1} = {[c_{1} + 1, b_{1}]_{(ρ)}, …, [c_{t} + 1, b_{t}]_{(ρ)}} and 𝔪_{2} = {[a_{1}, c_{1}]_{(ρ)}, …, [a_{t}, c_{t}]_{(ρ)}} be the associated ladders. If either 𝔪_{1} = 0 or L(𝔪_{1}) is Sp-distinguished and either 𝔪_{2} = 0 or L(𝔪_{2}) is generic then c_{t−2i} + 1 = c_{t−2i−1} = a_{t−2i−1} − 1 and Δ_{t−2i} ⊢_{r_{i}} Δ_{t−2i−1} where r_{i} = d(a_{t−2i−1} − a_{t−2i} − 1) for all i = 0, …, ⌊t/2⌋ − 1. Moreover, if t is odd then c_{1} = b_{1}.

Proof. If t = 1 then the lemma follows from the fact that if 𝔪_{1} ≠ 0 then L(𝔪_{1}) is generic, and Lemma 7.1. Assume that t > 1. Suppose that 1 < i ≤ t is such that c_{i} = b_{i} (in particular, [a_{i}, c_{i}]_{(ρ)} is not empty). Then, since 𝔪 is a proper ladder, we have a_{i−1} − 1 ≤ b_{i} = c_{i} < c_{i−1} and therefore [a_{i−1}, c_{i−1}]_{(ρ)} is non-empty. But then [a_{i}, c_{i}]_{(ρ)} ≺ [a_{i−1}, c_{i−1}]_{(ρ)} are both in 𝔪_{2}. By [Zel80, Theorem 9.7] this contradicts the assumption that L(𝔪_{2}) is generic. Therefore, c_{i} < b_{i} for all i > 1. By the assumption that L(𝔪_{1}) is Sp-distinguished and Theorem 10.3, 𝔪_{1} is of Speh type. That is, c_{1} < b_{1} if and only if t is even and either way,

[c_{t−2i−1} + 1, b_{t−2i−1}]_{(ρ)} = ν[c_{t−2i} + 1, b_{t−2i}]_{(ρ)}, i = 0, …, ⌊t/2⌋ − 1.

To complete the proof it is only left to show that c_{t−2i−1} = a_{t−2i−1} − 1 for all i = 0, …, ⌊t/2⌋ − 1. But if c_{t−2i−1} ≥ a_{t−2i−1} then c_{t−2i} = c_{t−2i−1} − 1 ≥ a_{t−2i−1} − 1, i.e., [a_{t−2i}, c_{t−2i}]_{(ρ)} ≺ [a_{t−2i−1}, c_{t−2i−1}]_{(ρ)} in 𝔪_{2} which, again by [Zel80, Theorem 9.7], is a contradiction. The lemma follows. □

14.1.5. We now determine the proper ladder representations that admit any particular Klyachko model.

Proposition 14.5. Let 𝔪 = {Δ_{1}, …, Δ_{t}} ∈ O_{ρ} be a proper ladder, so that L(𝔪) ∈ Π(G_{n}) and let n = 2k+r. If t is odd, let s be such that L(Δ_{1}) ∈ Π(G_{s}), otherwise, set s = 0. Then L(𝔪) is (H_{2k,r}, ψ)-distinguished if and only if Δ_{t−2i} ⊢_{r_{i}} Δ_{t−2i−1} for some r_{i}, i = 0, …, ⌊t/2⌋−1 and r = r_{0}+⋯+r_{⌊t/2⌋−1}+s.

Proof. Let π = L(𝔪) and note that Ind_{Sp_{2k}(F)×N_{r}}^{M_{(2k,r)}}(1⊗ψ) = Ind_{H_{2k,r}}^{P_{(2k,r)}}(ψ)|_{M_{(2k,r)}}. By (1), (2) and transitivity of induction we have

(16) Hom_{H_{2k,r}}(π, ψ) ≃ Hom_{G_{n}}(π, Ind_{H_{2k,r}}^{G_{n}}(ψ)) ≃

≃ Hom_{M_{(2k,r)}}(r_{M_{(2k,r)},G_{n}}(π), Ind_{Sp_{2k}(F)×N_{r}}^{M_{(2k,r)}}(1 ⊗ ψ)) ≃

≃ Hom_{Sp_{2k}(F)×N_{r}}(r_{M_{(2k,r)},G_{n}}(π), 1 ⊗ ψ).

Assume first that π is (H_{2k,r}, ψ)-distinguished. By (16) and Lemma 2.3 there is an irreducible component σ_{1} ⊗ σ_{2} of r_{M_{(2k,r)},G_{n}}(π), (where σ_{1} ∈ Π(G_{2k}) and σ_{2} ∈ Π(G_{r})) so that σ_{1} is Sp-distinguished and σ_{2} is generic. If Δ_{i} = [a_{i}, b_{i}]_{(ρ)} then it follows from [KL12, Theorem 2.1] that there exist c_{1} > ⋯ > c_{t} such that σ_{1} = L(𝔪_{1}) and σ_{2} = L(𝔪_{2}) where 𝔪_{1} = {[c_{1} + 1, b_{1}]_{(ρ)}, …, [c_{t} + 1, b_{t}]_{(ρ)}} and 𝔪_{2} = {[a_{1}, c_{1}]_{(ρ)}, …, [a_{t}, c_{t}]_{(ρ)}}. The ‘only if’ part of the proposition therefore follows from Lemma 14.4. Assume that Δ_{t−2i} ⊢_{r_{i}} Δ_{t−2i−1}, i = 0, …, ⌊t/2⌋ − 1 and r = r_{0} + ⋯ + r_{⌊t/2⌋−1} + s. Let c_{t−2i} + 1 = c_{t−2i−1} = a_{t−2i−1} − 1, i = 0, …, ⌊t/2⌋ − 1. If t is odd, further let c_{1} = b_{1}. Let σ_{1} = L(𝔪_{1}) and σ_{2} = L(𝔪_{2}) where 𝔪_{1} = {[c_{1} + 1, b_{1}]_{(ρ)}, …, [c_{t} + 1, b_{t}]_{(ρ)}} and 𝔪_{2} = {[a_{1}, c_{1}]_{(ρ)}, …, [a_{t}, c_{t}]_{(ρ)}}. Note that Δ ⊀ Δ′ for any two segments in the ladder 𝔪_{2} and therefore σ_{2} is generic by [Zel80, Theorem 9.7]. It is also clear from the above definitions that 𝔪_{1} is of Speh type and therefore σ_{1} is Sp-distinguished by Theorem 10.3. By [KL12, Corollary 2.2], σ_{1} ⊗ σ_{2} is a direct summand (and in particular a quotient) of r_{M_{(2k,r)},G_{n}}(π). Therefore (16) and Lemma 2.2 complete the proof of the proposition. □

Remark 14.6. If π is a proper ladder representation then Proposition 14.5 provides a recipe for computing r(π) and in particular, directly implies the uniqueness of r(π). The same is true more generally for ladder representations.

14.2. Klyachko models of ladder representations.

Theorem 14.7. Let π be a ladder representation and assume that π = π_{1}×⋯× π_{t} is the unique decomposition of π as a product of proper ladder representations (see §14.1.2). Then π admits a Klyachko model if and only if π_{i} admits a Klyachko model for all i = 1, …, t. Furthermore, in that case r(π) = r(π_{1}) + ⋯ + r(π_{t}).

Proof. This is immediate from Proposition 13.4 and §14.1.2. □

Remark 14.8. Based on (13), the classification of ladder representations that are (H′_{r,2k}, ψ)-distinguished is obtained by ‘reflecting all segments along the origin of their ℤ-line’.

References

[BD08] Philippe Blanc and Patrick Delorme, Vecteurs distributions Hinvariants de représentations induites, pour un espace symétrique réductif p-adique G/H, Ann. Inst. Fourier (Grenoble) 58 (2008), no. 1, 213–261. MR 2401221 (2009e:22015)

[Ber88] Joseph N. Bernstein, On the support of Plancherel measure, J. Geom. Phys. 5 (1988), no. 4, 663–710 (1989). MR 1075727 (91k:22027)

[BZ76] I. N. Bernšteĭn and A. V. Zelevinskiĭ, Representations of the group GL(n, F), where F is a local non-Archimedean field, Uspehi Mat. Nauk 31 (1976), no. 3(189), 5–70. MR 0425030 (54 #12988)

[BZ77] I. N. Bernstein and A. V. Zelevinsky, Induced representations of reductive 𝔭-adic groups. I, Ann. Sci. École Norm. Sup. (4) 10 (1977), no. 4, 441–472. MR 0579172 (58 #28310)

[FLO12] Brooke Feigon, Erez Lapid, and Omer Offen, On representations distinguished by unitary groups, Publ. Math. Inst. Hautes Études Sci. (2012), 185–323. MR 2930996

[GK75] I. M. Gel′fand and D. A. Kajdan, Representations of the group GL(n, K) where K is a local field, Lie groups and their representations (Proc. Summer School, Bolyai János Math. Soc., Budapest, 1971), Halsted, New York, 1975, pp. 95–118. MR 0404534 (53 #8334)

[Gur15] Maxim Gurevich, On a local conjecture of Jacquet, ladder representations and standard modules, Math. Z. 281 (2015), no. 3-4, 1111–1127. MR 3421655

[GSS15] Gourevitch, Dmitry and Sahi, Siddhartha and Sayag, Eitan, Invariant functionals on Speh representations , Transformation groups 20 (2015), no. 4, 1023–1042. MR 3416438

[GOSS12] Gourevitch, Dmitry and Offen, Omer and Sahi, Siddhartha and Sayag, Eitan, Existence of Klyachko models for GL(n, ℝ) and GL(n, ℂ) , Journal of Functional Analysis 8 (2012), no. 8, 3585–3601. MR 2889169

[HR90] Michael J. Heumos and Stephen Rallis, Symplectic-Whittaker models for Gl_{n}, Pacific J. Math. 146 (1990), no. 2, 247–279. MR 1078382 (91k:22036)

[IS91] N. F. J. Inglis and J. Saxl, An explicit model for the complex representations of the finite general linear groups, Arch. Math. (Basel) 57 (1991), no. 5, 424–431. MR 1129515 (92j:20039)

[KL12] Arno Kret and Erez Lapid, Jacquet modules of ladder representations, C. R. Math. Acad. Sci. Paris 350 (2012), no. 21-22, 937–940. MR 2996769

[Kly83] A. A. Klyachko, Models for complex representations of groups GL(n, q), Mat. Sb. (N.S.) 120(162) (1983), no. 3, 371–386. MR MR691984 (84j:20014)

[LM] Erez Lapid and Alberto Mínguez, On parabolic induction on inner forms of the general linear group over a non-archimedean local field, Selecta Math. (N.S.) to appear, arXiv:1411.6310.

[LM14] , On a determinantal formula of Tadić, Amer. J. Math. 136 (2014), no. 1, 111–142. MR 3163355 [Mit14] Arnab Mitra, On representations of GL_{2n}(F) with a symplectic period,

Pacific J. Math. 268 (2014), no. 2, 435–463. MR 3227442

[MW86] C. Mœglin and J.-L. Waldspurger, Sur l’involution de Zelevinski, J. Reine Angew. Math. 372 (1986), 136–177. MR 863522 (88c:22019)

[Off06a] Omer Offen, On symplectic periods of the discrete spectrum of GL_{2n}, Israel J. Math. 154 (2006), 253–298. MR 2254544 (2007h:11064)

[Off06b] , Residual spectrum of GL_{2n} distinguished by the symplectic group, Duke Math. J. 134 (2006), no. 2, 313–357. MR 2248833 (2007h:11063)

[OS07] Omer Offen and Eitan Sayag, On unitary representations of GL_{2n} distinguished by the symplectic group, J. Number Theory 125 (2007), no. 2, 344–355. MR 2332593 (2009a:22012)

[OS08a] , Global mixed periods and local Klyachko models for the general linear group, Int. Math. Res. Not. IMRN (2008), no. 1, Art. ID rnm 136, 25. MR 2417789 (2009e:22017)

[OS08b] , Uniqueness and disjointness of Klyachko models, J. Funct. Anal. 254 (2008), no. 11, 2846–2865. MR 2414223 (2009e:22018)

[OS09] , The SL(2)-type and base change, Representation Theory. An Electronic Journal of the American Mathematical Society 13 (2009), no. 11, 228–235. MR 2515933

[Tad86] Marko Tadić, Classification of unitary representations in irreducible representations of general linear group (non-Archimedean case), Ann. Sci. École Norm. Sup. (4) 19 (1986), no. 3, 335–382. MR 870688 (88b:22021)

[Zel80] A. V. Zelevinsky, Induced representations of reductive 𝔭-adic groups. II. On irreducible representations of GL(n), Ann. Sci. École Norm. Sup.

(4) 13 (1980), no. 2, 165–210. MR 584084 (83g:22012)

Arnab Mitra Department of Mathematics Technion – Israel Institute of Technology Haifa 3200003 Israel 00.arnab.mitra@gmail.com Omer Offen Department of Mathematics Technion – Israel Institute of Technology Haifa 3200003 Israel offen@tx.technion.ac.il

Eitan Sayag

Department of Mathematics Ben-Gurion University of the Negev P.O.B. 653 B’eer Sheva 84105 ISRAEL eitan.sayag@gmail.com 658

Documenta Mathematica 22 (2017)