Séminaire Lotharingien de Combinatoire 81 (2020), Article B81k

A GENERALIZATION OF SCHUR’S P - AND Q-FUNCTIONS

SOICHI OKADA

Dedicated to Christian Krattenthaler on the occasion of his 60th birthday

Abstract. We introduce and study a generalization of Schur’s P -/Q-functions associated with a polynomial sequence, which can be viewed as “Macdonald’s ninth variation” for P -/Q-functions. This variation includes as special cases Schur’s P -/Q-functions, Ivanov’s factorial P -/Q-functions and the t = −1 specialization of Hall–Littlewood functions associated with the classical root systems. We establish several identities and properties such as generalizations of Schur’s original definition of Schur’s Q-functions, a Cauchy-type identity, a generalization of the Józefiak–Pragacz–Nimmo formula for skew Q-functions, and a Pieri-type rule for multiplication.

Contents

1. Introduction

2. Several expressions for generalized P -functions 4 3. Dual P -functions and Cauchy-type identity 11 4. Generalized skew P -functions 16 5. Pieri-type rule 24 6. Applications to factorial P - and Q-functions 28 7. P -functions associated with classical root systems 39 Appendix A. Pfaffian formulas 44 References 49

1. Introduction

Schur (S-)functions and Schur P -/Q-functions are two important families of symmetric functions, and they appear in several parallel situations. For example, in the representation theory of the symmetric groups, Schur functions describe the characters of irreducible linear representations, while Schur Q-functions describe the characters of irreducible projective representations (see [22]). In cohomology theory, Schur functions represent the Schubert classes of Grassmannians, while Schur Q-functions represent the

2010 Mathematics Subject Classification. Primary 05E05; Secondary 15A15. Key words and phrases. Schur P -/Q-functions, factorial P -/Q-functions, Macdonald’s ninth variation,

Hall–Littlewood functions, Pfaffian.

This work was partially supported by JSPS Grants-in-Aid for Scientific Research No. 24340003, No. 15K13425 and No. 18K03208.

Schubert classes of Lagrangian Grassmannians (see [20]). Moreover, some identities for Schur functions have their counterparts for Schur P -/Q-functions.

There are several generalizations, variations or deformations of Schur functions, such as Hall–Littlewood functions, Macdonald functions and factorial Schur functions. The generalization relevant to this paper is Macdonald’s ninth variation ([11], see also [16]) associated with a polynomial sequence, which is defined as follows.

Let F = {f_{d}}_{d=0}^{∞} be a sequence of polynomials f_{d}(u) ∈ K[u], where K is a ground field of characteristic 0, such that deg f_{d} = d for d ≥ 0. Given a partition λ of length l ≤ n, we define the generalized Schur function s_{λ}^{F}(x_{1}, …, x_{n}) as the ratio of two alternants,

s_{λ}^{F}(x_{1}, …, x_{n}) = \frac{det (f_{λ_{j}+n−j}(x_{i}))_{1≤i,j≤n}}{det (f_{n−j}(x_{i}))_{1≤i,j≤n}} ,  (1.1)

where λ_{l+1} = ⋯ = λ_{n} = 0. The original Schur functions s_{λ}(x) are recovered by setting f_{d}(u) = u^{d} for d ≥ 0. The factorial Schur functions s_{λ}(x|a) with factorial parameters a = (a_{0}, a_{1}, …) are obtained by taking f_{d}(u) = (u|a)^{d} = ∏_{i=0}^{d−1}(u − a_{i}). Moreover, classical group characters are special cases of generalized Schur functions. For example, if the polynomial sequence F = {f_{d}}_{d=0}^{∞} is defined by

f_{d}(x + x^{−1}) = \frac{x^{d+1} − x^{−d−1}}{x − x^{−1}} (d ≥ 0),

then it is not difficult to see that the generalized Schur function s_{λ}^{F}(x_{1} +x_{1}^{−1}, …, x_{n} +x_{n}^{−1}) equals the irreducible character of the symplectic group Sp_{2n}(ℂ) with highest weight λ.

Generalized Schur functions share many of the same properties as the original Schur functions. For example, they satisfy the modified Jacobi–Trudi identity and the Giambelli identity,

s_{λ}^{F}(x_{1}, …, x_{n}) = det (s_{(λ_{i}−i+j)}^{F}(x_{j}, …, x_{n}))_{1≤i,j≤l}
= det (s_{(α_{i}|β_{j})}^{F}(x_{1}, …, x_{n}))_{1≤i,j≤r},

where λ is a partition of length l ≤ n and λ = (α_{1}, …, α_{r}|β_{1}, …, β_{r}) in Frobenius notation.

The aim of this paper is to introduce and study the “ninth variation” of Schur P -/Q-functions, which we call generalized P -functions associated with polynomial sequences. We define generalized P -functions in terms of Nimmo-type formula and derive Pfaffian identities and basic properties by following a linear algebraic approach similar to [19].

We use the following terminology on polynomial sequences.

Definition 1.1. Let F = {f_{d}}_{d=0}^{∞} be a sequence of polynomials f_{d}(u) ∈ K[u]. We say that F is admissible if it satisfies the conditions

f_{0}(u) = 1, deg f_{d} = d (d ≥ 1).  (1.2)

An admissible sequence F is called constant-term free if f_{d}(0) = 0 for any d ≥ 1.

In this article, a partition of length l is a weakly decreasing sequence λ = (λ_{1}, …, λ_{l}) of positive integers. We write l = l(λ) and |λ| = ∑_{i=1}^{l} λ_{i}. A partition λ of length l is called strict if λ_{1} > ⋯ > λ_{l}. The empty sequence ∅ is the unique strict partition of length 0.

A GENERALIZATION OF SCHUR’S P - AND Q-FUNCTIONS 3

For a sequence x = (x_{1}, …, x_{n}) of n indeterminates, we put

A(x) = (\frac{x_{j} − x_{i}}{x_{j} + x_{i}} )_{1≤i,j≤n} , Δ(x) = ∏_{1≤i<j≤n} \frac{x_{j} − x_{i}}{x_{j} + x_{i}} .  (1.3)

Now we give a definition of generalized P -functions associated with polynomial sequences in terms of a Nimmo-type formula (see [18, (A13)]).

Definition 1.2. Let n be a positive integer and x = (x_{1}, …, x_{n}) a sequence of n indeterminates. For an admissible sequence F = {f_{d}}_{d=0}^{∞} of polynomials and a sequence α = (α_{1}, …, α_{r}) of nonnegative integers, let V_{α}^{F}(x) be the n × r matrix given by

V_{α}^{F}(x) = (f_{α_{j}}(x_{i}))_{1≤i≤n,1≤j≤r}.

Given a strict partition λ of length l, we define the corresponding generalized P -function P_{λ}^{F}(x) associated with F by putting

P_{λ}^{F}(x) = \begin{cases}\frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & V_{λ}^{F}(x) \\ −^{t}V_{λ}^{F}(x) & O \end{matrix}) , & if n + l is even, \\ \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & V_{λ^{0}}^{F}(x) \\ −^{t}V_{λ^{0}}^{F}(x) & O \end{matrix}) , & if n + l is odd, \end{cases}  (1.4)

where λ^{0} = (λ_{1}, …, λ_{l}, 0). We simply write V_{α}(x) and P_{λ}(x) for V_{α}^{F}(x) and P_{λ}^{F}(x) if there is no confusion, e.g., in the proofs.

Note that (see Proposition A.1)

Δ(x) = \begin{cases}Pf A(x), & if n is even, \\ Pf (\begin{matrix}A(x) & 1 \\ −^{t}1 & 0 \end{matrix}) , & if n is odd, \end{cases}

where 1 is the all-one column vector of appropriate size. Hence our definition (1.4) can be regarded as a counterpart of the definition (1.1) of generalized Schur functions.

Example 1.3. (1) It follows from Nimmo’s formula [18, (A13)] that we recover the original Schur P -function P_{λ}(x) and Schur Q-function Q_{λ}(x) by setting f_{d}(u) = u^{d} and f_{d}(u) = 2u^{d}, respectively.

(2) It follows from the Nimmo-type formula [7, Theorem 3.2] that Ivanov’s factorial P -function P_{λ}(x|a) and Q-function Q_{λ}(x|a) are obtained by taking f_{d}(u) = (u|a)^{d} and f_{d}(u) = 2(u|a)^{d}, respectively.

(3) As we will see in Section 7, our generalized P -functions include the t = −1 specializations of Hall–Littlewood functions associated with the root systems of type B, C and D.

Ikeda and Naruse [4], and Nakagawa and Naruse [17] introduced other generalizations of factorial P - and Q-functions from the viewpoint of Schubert calculus. In a very recent paper [2], Foley and King give a combinatorial generalization of Schur Q-functions in terms of shifted tableaux and prove several Pfaffian formulas.

The organization and main results of this paper are as follows. In Section 2, we relate our definition of generalized P -functions (Definition 1.2) with generalizations of two other definitions of Schur P -/Q-functions. Namely, we prove that P_{λ}^{F}(x) is also obtained by setting t = −1 in the generalized Hall–Littlewood function associated with a polynomial sequence (see Theorem 2.3), and that P_{λ}^{F}(x) is expressed as the Pfaffian of the skew-symmetric matrix with entries P_{(λ_{i},λ_{j})}^{F}(x) (see Theorem 2.6). In Section 3, we introduce the notion of generalized dual P -functions P̂_{λ}^{F}(x) and prove a corresponding Cauchy-type identity. In Section 4, we define generalized skew P -functions P_{λ/μ,p}^{F}(x) in terms of a Józefiak–Pragacz–Nimmo-type Pfaffian and prove that P_{λ/μ,p}^{F}(x) appears as the coefficient of P_{μ}^{F}(y) in the expansion of P_{λ}^{F}(x, y) (see Theorem 4.2). In Section 5, we consider the modified Pieri coefficients in the expansion of the product P_{μ}^{F}(x) · Q_{(r)}(x) and obtain a determinant formula for the generating function of modified Pieri coefficients (see Theorem 5.3). Section 6 focuses on Ivanov’s factorial P -/Q-functions. We derive a determinant formula for the factorial skew P -function in one variable (see Theorem 6.5), and an explicit product formula for the generating function of modified Pieri coefficients (see Theorem 6.6). In Section 7, we show that the Hall–Littlewood functions at t = −1 associated with the classical root systems can be written as generalized P -functions associated with certain polynomial sequences (see Theorem 7.2). Appendix A collects some Schur-type Pfaffian evaluations and useful formulas.

2. Several expressions for generalized P -functions

In this section, we give several expressions for generalized P -functions associated with an admissible polynomial sequence, and we study their basic properties.

2.1. Hall–Littlewood-type expression. In this subsection, we prove that our generalized P -functions are obtained as the t = −1 specialization of Hall–Littlewood-type functions.

We begin with the following proposition.

Proposition 2.1. Let F be an admissible sequence of polynomials and x = (x_{1}, …, x_{n}). Then we have:

(1) For the empty partition ∅, we have P_{∅}^{F}(x) = 1.

(2) If λ is a strict partition of length > n, then we have P_{λ}^{F}(x) = 0.

Proof. By using the definition (1.4), we can derive (1) from the Pfaffian evaluations (A.3) and (A.4), and (2) from Proposition A.5. □

We define a generalization of Hall–Littlewood polynomials associated with an admissible polynomial sequence.

Definition 2.2. Let n be a positive integer and x = (x_{1}, …, x_{n}). Given a partition λ of length l ≤ n, we regard λ as a sequence (λ_{1}, …, λ_{l}, 0, …, 0) of length n, and define a polynomial v_{λ}^{(n)}(t) by putting

v_{λ}^{(n)}(t) = ∏_{k≥0}[m_{k}]_{t}!,

where m_{k} = {i : 1 ≤ i ≤ n, λ_{i} = k} and [m]_{t}! = ∏_{j=1}^{m}(1 − t^{j})/(1 − t). For an admissible polynomial sequence F and a partition λ of length ≤ n, we define the generalized Hall– Littlewood function ℙ_{λ}^{F}(x; t) corresponding to λ by putting

ℙ_{λ}^{F}(x; t) = \frac{1}{v_{λ}^{(n)}(t)} ∑_{w∈S_{n}} w (∏_{i=1}^{n} f_{λ_{i}}(x_{i}) ∏_{1≤i<j≤n} \frac{x_{i} − tx_{j}}{x_{i} − x_{j}} ) ,  (2.1)

where S_{n} is the symmetric group acting on K(t)[x_{1}, …, x_{n}] by permuting variables. We write ℙ_{λ}(x; t) for ℙ_{λ}^{F}(x; t) when there is no confusion.

Setting f_{d}(u) = u^{d} for d ≥ 0, we recover the original Hall–Littlewood polynomials. The following is the main theorem of this subsection.

Theorem 2.3. For an admissible sequence F and a strict partition λ of length l ≤ n, we have

P_{λ}^{F}(x) = ℙ_{λ}^{F}(x; −1).  (2.2)

Note that Equation (2.2) with f_{d}(u) = u^{d} is the definition of Schur P -function adopted in [12, III.8]. For the sake of completeness and later use, we give a proof of this theorem, which follows the argument in [18, Appendix]. As a first step, we show the following lemma.

Lemma 2.4. For a strict partition λ of length l ≤ n, we have

ℙ_{λ}^{F}(x; −1) = ∑_{u∈S_{n}/S_{n−l}} u (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} )  (2.3)

= \frac{1}{(n − l)!} ∑_{u∈S_{n}} u (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} ) ,  (2.4)

where S_{n−l} is the symmetric group on the last n − l variables x_{l+1}, …, x_{n}.

Proof. Since f_{0}(u) = 1 and the product ∏_{1≤i<j≤n,i≤l}(x_{i} − tx_{j})/(x_{i} − x_{j}) is invariant under S_{n−l}, we have

ℙ_{λ}(x; t)

= \frac{1}{v_{λ}^{(n)}(t)} ∑_{w′∈S_{n}/S_{n−l}} w′ (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} − tx_{j}}{x_{i} − x_{j}} ∑_{w′′∈S_{n−l}} w′′ ( ∏_{l+1≤i<j≤n} \frac{x_{i} − tx_{j}}{x_{i} − x_{j}} )) .

By using (see [10, Theorem 2.8])

∑_{w′′∈S_{n−l}} w′′ ( ∏_{l+1≤i<j≤n} \frac{x_{i} − tx_{j}}{x_{i} − x_{j}} ) = [n − l]_{t}! = v_{λ}^{(n)}(t),

we have

ℙ_{λ}(x; t) = ∑_{w′∈S_{n}/S_{n−l}} w′ (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} − tx_{j}}{x_{i} − x_{j}} ) .

By specializing t = −1, we obtain (2.3), from which (2.4) follows. □ Proof of Theorem 2.3. Since Δ(x) is alternating in x_{1}, …, x_{n}, it follows from (2.3) that

ℙ_{λ}(x; −1) = \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}}}{Δ(x)} ∑_{v∈S_{n}/S_{n−l}} sgn(v)v (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{l+1≤i<j≤n} \frac{x_{j} − x_{i}}{x_{j} + x_{i}} ) .

Since ∏_{l+1≤i<j≤n}(x_{j} − x_{i})/(x_{j} + x_{i}) is invariant under the symmetric group S_{l} acting on the first l variables x_{1}, …, x_{l}, we have

ℙ_{λ}(x; −1)

= \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}}}{Δ(x)} ∑_{v′∈S_{n}/(S_{l}×S_{n−l})} sgn(v′)v′ ( ∑_{v′′∈S_{l}} sgn(v′′)v′′ (∏_{i=1}^{l} f_{λ_{i}}(x_{i})) ∏_{l+1≤i<j≤n} \frac{x_{j} − x_{i}}{x_{j} + x_{i}} )

= \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}}}{Δ(x)} ∑_{v′∈S_{n}/(S_{l}×S_{n−l})} sgn(v′)v′ (det (f_{λ_{j}}(x_{i}))_{1≤i,j≤l} ∏_{l+1≤i<j≤n} \frac{x_{j} − x_{i}}{x_{j} + x_{i}} ) .

We take R = {u ∈ S_{n} : u(1) < ⋯ < u(l), u(l + 1) < ⋯ < u(n)} as a complete set of coset representatives of S_{n}/(S_{l} × S_{n−l}). We note that the correspondence u ↦ {u(l + 1), …, u(n)} gives a bijection between the coset representatives R and the set \binom{[n]}{n−l} of all (n − l)-element subsets of [n] = \{1, …, n\}.

First we consider the case where n − l is even. In this case, by using Schur’s Pfaffian evaluation (A.3), we have

ℙ_{λ}(x; −1) = \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}}}{Δ(x)} ∑_{u∈S_{n}/S_{l}×S_{n−l}} sgn(u)u( det V_{λ}(x_{[l]}) Pf A(x_{[n]\[l]})),

where [l] = {1, …, l}, [n]\[l] = {l+1, …, n} and x_{J} = (x_{j_{1}}, …, x_{j_{m}}) for J = {j_{1}, …, j_{m}} with j_{1} < ⋯ < j_{m}. On the other hand, by applying Proposition A.5 (a Pfaffian version of the Laplace expansion) to the matrices Z = A(x) and W = V_{λ}(x), we obtain

Pf (\begin{matrix}A(x) & V_{λ}(x) \\ −^{t}V_{λ}(x) & O \end{matrix}) = ∑_{I∈\binom{[n]}{n−l}}(−1)^{Σ(I)−\binom{n}{2}} Pf A(x_{I}) det V_{λ}(x_{[n]\I}),

where Σ(I) = ∑_{i∈I} i. Since n − l is even, we can see that, if u ∈ R corresponds to I ∈ \binom{[n]}{n−l}, then the inversion number of u is given by

inv(u) = \binom{n + 1}{2} − \binom{l + 1}{2} − Σ(I) ≡ \binom{n}{2} − \binom{l}{2} + Σ(I) mod 2.

Hence we have

Pf (\begin{matrix}A(x) & V_{λ}(x) \\ −^{t}V_{λ}(x) & O \end{matrix}) = ∑_{u∈S_{n}/(S_{l}×S_{n−l})}(−1)^{\binom{l}{2}} sgn(u)u( det V_{λ}(x_{[l]}) Pf A(x_{[n]\[l]})),

and

ℙ_{λ}(x; −1) = \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}+\binom{l}{2}}}{Δ(x)} Pf (\begin{matrix}A(x) & V_{λ}(x) \\ −^{t}V_{λ}(x) & O \end{matrix}) .

Now we can use the relation \binom{n}{2} − \binom{l}{2} − \binom{n−l}{2} = l(n − l) ≡ 0 mod 2 to complete the proof of (2.2) in the case where n − l is even.

Next we consider the case where n − l is odd. In this case, by using (A.4), we see that

ℙ_{λ}(x; −1) = \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}}}{Δ(x)} ∑_{u∈S_{n}/(S_{l}×S_{n−l})} sgn(u)u (det V_{λ}(x_{[l]}) Pf (\begin{matrix}A(x_{[n]\[l]}) & 1 \\ −^{t}1 & 0 \end{matrix})) ,

where 1 is the all-one column vector. On the other hand, by applying Proposition A.5 to the matrices

Z = (\begin{matrix}A(x) & 1 \\ −^{t}1 & 0 \end{matrix}) , W = (\begin{matrix}V_{λ}(x) \\ O \end{matrix}) ,

we see that

Pf (\begin{matrix}A(x) & 1 & V_{λ}(x) \\ −^{t}1 & 0 & O \\ −^{t}V_{λ}(x) & O & O \end{matrix}) = ∑_{I}(−1)^{Σ(I)−\binom{n+1}{2}} Pf Z(I) det W ([n + 1] \ I; [l]),

where I runs over all (n + 1 − l)-element subsets of [n + 1]. If n + 1 ∉ I, then we have

det W ([n + 1] \ I; [l]) = det (\begin{matrix}V_{λ}(x_{[n]\I}) \\ O \end{matrix}) = 0.

Hence we have

Pf (\begin{matrix}A(x) & 1 & V_{λ}(x) \\ −^{t}1 & 0 & O \\ −^{t}V_{λ}(x) & O & O \end{matrix}) = ∑_{I∈\binom{[n]}{n−l}}(−1)^{Σ(I∪\{n+1\})−\binom{n+1}{2}} Pf (\begin{matrix}A(x_{I}) & 1 \\ −^{t}1 & 0 \end{matrix}) det V_{λ}(x_{[n]\I}).

Since n − l is odd, we see that, if u ∈ R corresponds to I ∈ \binom{[n]}{n−l}, then we have

inv(u) = \binom{n + 1}{2} − \binom{l + 1}{2} − Σ(I) ≡ \binom{n + 1}{2} − \binom{l}{2} − Σ(I ∪ \{n + 1\}) mod 2.

Moreover, by permuting rows/columns we have

Pf (\begin{matrix}A(x) & 1 & V_{λ}(x) \\ −^{t}1 & 0 & O \\ −^{t}V_{λ}(x) & O & O \end{matrix}) = (−1)^{l} Pf (\begin{matrix}A(x) & V_{λ^{0}}(x) \\ −^{t}V_{λ^{0}}(x) & O \end{matrix}) .

Hence we have

Pf (\begin{matrix}A(x) & V_{λ^{0}}(x) \\ −^{t}V_{λ^{0}}(x) & O \end{matrix})

= ∑_{u∈S_{n}/(S_{l}×S_{n−l})}(−1)^{\binom{l}{2}+l} sgn(u)u( Pf (\begin{matrix}A(x_{[n]\[l]}) & 1 \\ −^{t}1 & 0 \end{matrix}) det V_{λ}(x_{[l]})),

and

ℙ_{λ}(x; −1) = \frac{(−1)^{\binom{n}{2}+\binom{n−l}{2}+\binom{l}{2}+l}}{Δ(x)} Pf (\begin{matrix}A(x) & V_{λ^{0}}(x) \\ −^{t}V_{λ^{0}}(x) & O \end{matrix}) .

Now we can complete the proof in the case where n − l is odd by using the congruence relation \binom{n}{2} − \binom{l}{2} − \binom{n−l}{2} = l(n − l) ≡ l mod 2. □

By combining Theorem 2.3 and Lemma 2.4, we obtain the following corollary.

Corollary 2.5. For a strict partition λ of length l ≤ n, we have

P_{λ}^{F}(x) = ∑_{u∈S_{n}/S_{n−l}} u (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} )  (2.5)

= \frac{1}{(n − l)!} ∑_{u∈S_{n}} u (∏_{i=1}^{l} f_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} ) .  (2.6)

2.2. Schur-type Pfaffian formula. In this subsection, we use the definition (1.4) and a Pfaffian version of the Sylvester formula (Proposition A.4) to derive a Schur-type Pfaffian formula for P_{λ}^{F}(x), which generalizes (a part of) Schur’s original definition of Schur Qfunctions [22, §35] and a similar formula for factorial Q-functions [7, Theorem 9.1]. We use the following conventions:

P_{(0)}^{F}(x) = 1,  (2.7)
P_{(s,r)}^{F}(x) = −P_{(r,s)}^{F}(x), P_{(r,0)}^{F}(x) = −P_{(0,r)}^{F}(x) = P_{(r)}^{F}(x), P_{(0,0)}^{F}(x) = 0,  (2.8)

where r and s are positive integers.

Theorem 2.6. Let F be an admissible sequence. For a sequence α = (α_{1}, …, α_{r}) of nonnegative integers, let S_{α}^{F}(x) be the r × r skew-symmetric matrix defined by

S_{α}^{F}(x) = (P_{(α_{i},α_{j})}^{F}(x))_{1≤i,j≤r}.  (2.9)

Then, for a strict partition λ of length l, we have

P_{λ}^{F}(x) = \begin{cases}Pf S_{λ}^{F}(x), & if l is even, \\ Pf S_{λ^{0}}^{F}(x), & if l is odd, \end{cases}  (2.10)

where λ = (λ_{1}, …, λ_{l}) and λ^{0} = (λ_{1}, …, λ_{l}, 0).

In order to prove this theorem, we can use the same argument as in [19, Theorem 4.1 (3) and Remark 4.3]. As we will see in Proposition 2.7, the generalized P -functions do not have the stability property, so we cannot reduce the proof to the case where n is even.

Proof. By applying Proposition A.4 to the matrix X given by

X = \begin{cases}(\begin{matrix}A(x) & V_{λ}(x) \\ −^{t}V_{λ}(x) & O \end{matrix}) , & if n is even and l is even, \\ (\begin{matrix}A(x) & V_{λ^{0}}(x) \\ −^{t}V_{λ^{0}}(x) & O \end{matrix}) , & if n is even and l is odd, \\ (\begin{matrix}A(x) & 1 & V_{λ}(x) \\ −^{t}1 & 0 & O \\ −^{t}V_{λ}(x) & O & O \end{matrix}) , & if n is odd and l is even, \\ (\begin{matrix}A(x) & 1 & V_{λ}(x) & 0 \\ −^{t}1 & 0 & O & −1 \\ −^{t}V_{λ}(x) & O & O & O \\ 0 & 1 & O & 0 \end{matrix}) , & if n is odd and l is odd. \end{cases}

If n is even, then we have

\frac{Pf X}{Pf X([n])} = P_{λ}(x), \frac{Pf X([n] ∪ \{n + i, n + j\})}{Pf X([n])} = P_{(λ_{i},λ_{j})}(x).

If n is odd and l is even, then, by permuting rows/columns, we see that

\frac{Pf X}{Pf X([n + 1])} = P_{λ}(x), \frac{Pf X([n + 1] ∪ \{n + 1 + i, n + 1 + j\})}{Pf X([n + 1])} = P_{(λ_{i},λ_{j})}(x).

If n is odd and l is odd, then by expanding the Pfaffians along the last row/column, we have

\frac{Pf X}{Pf X([n + 1])} = P_{λ}(x), \frac{Pf X([n + 1] ∪ \{n + 1 + i, n + 1 + (l + 1)\})}{Pf X([n + 1])} = P_{(λ_{i})}(x),

and by permuting rows/columns we see that

\frac{Pf X([n + 1] ∪ \{n + 1 + i, n + 1 + j\})}{Pf X([n + 1])} = P_{(λ_{i},λ_{j})}(x).

Now Theorem 2.6 follows immediately from Proposition A.4. □

2.3. Stability. The Schur P -functions have the stability property (see [12, III, (2.5)])

P_{λ}(x_{1}, …, x_{n}, 0) = P_{λ}(x_{1}, …, x_{n}).

Our generalizations P_{λ}^{F}(x) do not have the stability property in general. For example we can show that

P_{(r)}^{F}(x_{1}, …, x_{n}, 0) = P_{(r)}^{F}(x_{1}, …, x_{n}) + (−1)^{n}f_{r}(0)

for r ≥ 1. The following “mod 2 stability property” was given by [3, Proposition 8.1] for factorial P -functions.

Proposition 2.7. Let F be an admissible sequence, and λ a strict partition. (1) In general, we have

P_{λ}^{F}(x_{1}, …, x_{n}, 0, 0) = P_{λ}^{F}(x_{1}, …, x_{n}).

(2) If F is constant-term free, then we have

P_{λ}^{F}(x_{1}, …, x_{n}, 0) = P_{λ}^{F}(x_{1}, …, x_{n}).

Proof. (1) Let x = (x_{1}, …, x_{n}) and x̃ = (x_{1}, …, x_{n}, x_{n+1}). It follows from the definition (1.4) that

P_{λ}(x_{1}, …, x_{n}, x_{n+1}, 0) = \frac{1}{(−1)^{n+1}Δ(x̃)} Pf (\begin{matrix}A(x̃) & −1 & V_{λ^{∗}}(x̃) \\ 1^{t} & 0 & V_{λ^{∗}}(0) \\ −^{t}V_{λ^{∗}}(x̃) & −^{t}V_{λ^{∗}}(0) & O \end{matrix}) ,

where λ^{∗} = λ or λ^{0} according to whether n + l is even or odd, and V_{α}(0) is the row vector (f_{α_{1}}(0), …, f_{α_{r}}(0)). Hence, if we put x_{n+1} = 0 in the above formula, we have

P_{λ}(x_{1}, …, x_{n}, 0, 0) = \frac{1}{(−1)^{n+1} · (−1)^{n}Δ(x)} Pf (\begin{matrix}A(x) & −1 & −1 & V_{λ^{∗}}(x) \\ 1^{t} & 0 & −1 & V_{λ^{∗}}(0) \\ 1^{t} & 1 & 0 & V_{λ^{∗}}(0) \\ −^{t}V_{λ^{∗}}(x) & −^{t}V_{λ^{∗}}(0) & −^{t}V_{λ^{∗}}(0) & O \end{matrix}) .

By subtracting the (n + 1)st row/column from the (n + 2)nd row/column and then by expanding the resulting Pfaffian along the (n + 2)nd row/column, we see that

P_{λ}(x_{1}, …, x_{n}, 0, 0) = \frac{1}{(−1)^{n+1} · (−1)^{n}Δ(x)} · (−1) Pf (\begin{matrix}A(x) & V_{λ^{∗}}(x) \\ −^{t}V_{λ^{∗}}(x) & O \end{matrix}) = P_{λ}(x).

(2) By the definition (1.4), we have

P_{λ}(x_{1}, …, x_{n}, 0) = \frac{1}{(−1)^{n}Δ(x)} Pf (\begin{matrix}A(x) & −1 & V_{λ^{∗}}(x) \\ 1^{t} & 0 & V_{λ^{∗}}(0) \\ −^{t}V_{λ^{∗}}(x) & −^{t}V_{λ^{∗}}(0) & O \end{matrix}) ,

where λ^{∗} = λ^{0} or λ according to whether n + l is even or odd. If n + l is even, by adding the (n + 1)st row/column to the last row/column and then by expanding the resulting Pfaffian along the last row/column, we see that P_{λ}(x, 0) = P_{λ}(x). If n + l is odd, then by permuting rows/columns, we obtain P_{λ}(x, 0) = P_{λ}(x). □

2.4. Relation with generalized Schur functions. We conclude this section by proving a relation between generalized P -functions and generalized Schur functions.

Proposition 2.8. Let F be an admissible sequence and x = (x_{1}, …, x_{n}). For a partition μ of length m ≤ n, let μ + δ_{n} be the strict partition obtained from (μ_{1} + n − 1, μ_{2} + n − 2, …, μ_{n−1} + 1, μ_{n}) by removing 0s, where μ_{m+1} = ⋯ = μ_{n} = 0. Then we have

P_{μ+δ_{n}}^{F}(x) = ∏_{1≤i<j≤n}(x_{i} + x_{j}) · s_{μ}^{F}(x),

where s_{μ}^{F}(x) is the generalized Schur function given by (1.1).

Proof. Since the strict partition μ + δ_{n} has length n − 1 or n, it follows from (2.6) that

P_{μ+δ_{n}}^{F}(x) = ∑_{w∈S_{n}} w (∏_{i=1}^{n} f_{μ_{i}+n−i}(x_{i}) ∏_{1≤i<j≤n} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} )

= ∏_{1≤i<j≤n} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} ∑_{w∈S_{n}} sgn(w)w (∏_{i=1}^{n} f_{μ_{i}+n−i}(x_{i}))

= ∏_{1≤i<j≤n}(x_{i} + x_{j}) · s_{μ}^{F}(x).  □

3. Dual P -functions and Cauchy-type identity

In this section, we introduce the dual of generalized P -functions and prove a Cauchytype identity for generalized P -functions.

3.1. Dual sequences. For a nonzero formal power series g(v) = ∑_{i=0}^{∞} b_{i}v^{i} ∈ K[[v]], the order ord g of g is defined to be the minimum integer k such that b_{k} ≠ 0. Let ⟨ , ⟩ : K[u] × K[[v]] → K be the non-degenerate bilinear pairing defined by

⟨u^{i}, v^{j}⟩ = \begin{cases}1, & if i = j = 0, \\ 1/2, & if i = j > 0, \\ 0, & if i ≠ j. \end{cases}

Lemma 3.1. Let F = {f_{d}}_{d=0}^{∞} be an admissible sequence of polynomials.

(1) Let F̂ = {f̂_{d}}_{d=0}^{∞} be a sequence of formal power series f̂_{d}(v) ∈ K[[v]] satisfying ord f̂_{d} = d for d ≥ 0. Then ⟨f_{k}, f̂_{l}⟩ = δ_{k,l} for any k, l ≥ 0 if and only if

∑_{k=0}^{∞} f_{k}(u)f̂_{k}(v) = \frac{1 + uv}{1 − uv} .  (3.1)

(2) There exists a unique sequence F̂ = {f̂_{d}}_{d=0}^{∞} satisfying ord f̂_{d} = d for d ≥ 0 and the equivalent conditions in (1). We call such a sequence F̂ the dual of F.

(3) If F̂ = {f̂_{d}}_{d=0}^{∞} is the dual of F, then F is constant-term free if and only if f̂_{0} = 1.

Proof. (1), (2) We write f_{d}(u) = ∑_{i≥0} a_{d,i}u^{i} and f̂_{d}(v) = ∑_{i≥0} b_{d,i}v^{i}. Since deg f_{d} = d and ord f̂_{d} = d, we have a_{d,i} = 0 for i ≥ d + 1 and b_{d,i} = 0 for i ≤ d − 1. We define b′_{d,i} by putting

b′_{d,i} = \begin{cases}b_{d,0}, & if i = 0, \\ b_{d,i}/2, & if i > 0. \end{cases}

Fix a nonnegative integer N and consider two (N+1)×(N+1) matrices A = (a_{i,j})_{0≤i,j≤N} and B′ = (b′_{i,j})_{0≤i,j≤N}. Then it is easy to see that

(a) ⟨f_{k}, f̂_{l}⟩ = δ_{k,l} for all 0 ≤ k, l ≤ N if and only if A ·^{t}B′ = I_{N+1};

(b) ∑_{k≥0} f_{k}(u)f̂_{k}(v) = (1 + uv)/(1 − uv) in the quotient ring K[[u, v]]/(u^{N+1}, v^{N+1}) if

and only if ^{t}A · B′ = I_{N+1}. The claims (1) and (2) follow from this observation.

(3) If f_{d}(0) = 0 for any d ≥ 1, then by substituting u = 0 in (3.1) we obtain f̂_{0}(v) = 1. Conversely, if f̂_{0} = 1, then ⟨f_{d}, f̂_{0}⟩ = δ_{d,0} is equal to the constant term of f_{d}. □

For example, if f_{d}(u) = u^{d} for d ≥ 0, then the dual of F is given by

f̂_{d}(v) = \begin{cases}1, & if d = 0, \\ 2v^{d}, & if d ≥ 1. \end{cases}

See Lemma 6.1 for the dual of the sequence {(u|a)^{d}}_{d=0}^{∞} of factorial monomials.

3.2. Generating functions of generalized P -functions. For a sequence of variables x = (x_{1}, …, x_{n}) and another variable z, we put

Π_{z}(x) = ∏_{i=1}^{n} \frac{1 + x_{i}z}{1 − x_{i}z} .  (3.2)

Then the generating functions of Schur Q-functions Q_{(r)}(x) and Q_{(r,s)}(x) are expressed as

∑_{r≥0} Q_{(r)}(x)z^{r} = Π_{z}(x),  (3.3)

∑_{r,s≥0} Q_{(r,s)}(x)z^{r}w^{s} = \frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1),  (3.4)

respectively (see [12, III, (8.1)] and [23, p. 117]). We can generalize these generating functions in terms of the dual sequence F̂.

Proposition 3.2. Let F = {f_{d}}_{d=0}^{∞} be an admissible sequence of polynomials, and F̂ = {f̂_{d}}_{d=0}^{∞} the dual sequence of F. Then we have:

(1) The generating function of generalized P -functions P_{(r)}^{F}(x) is given by

∑_{r≥0} P_{(r)}^{F}(x)f̂_{r}(z) = \begin{cases}Π_{z}(x), & if n is odd, \\ Π_{z}(x) + f̂_{0}(z) − 1, & if n is even, \end{cases}  (3.5)

under the convention (2.7).

(2) The generating function of generalized P -functions P_{(r,s)}^{F}(x) is given by

∑_{r,s≥0} P_{(r,s)}^{F}(x)f̂_{r}(z)f̂_{s}(w)

= \{ \begin{matrix}\frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1) + (f̂_{0}(w) − 1)Π_{z}(x) − (f̂_{0}(z) − 1)Π_{w}(x), \\ if n is odd, \\ \frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1), if n is even, \end{matrix} (3.6)

under the convention (2.8).

If we set f_{d}(u) = (u|a)^{d} (factorial monomial) with a_{0} = 0, then it follows from Lemma 6.1 that the formulas (3.5) and (3.6) reduce to the formulas given in [7, Theorem 8.2] and [7, Theorem 8.4] for factorial P -functions.

Proof. The idea of the proof is similar to that of [19, Theorem 4.1 (1) and (2)]. Let B_{z}(x) be the column vector with ith entry (1 + x_{i}z)/(1 − x_{i}z). By (3.1) we have

∑_{r≥0} f̂_{r}(z)V_{(r)}(x) = B_{z}(x), ∑_{r≥1} f̂_{r}(z)V_{(r)}(x) = B_{z}(x) − f̂_{0}(z)1.

(1) If n is odd, then by the definition (1.4) and the multilinearity of Pfaffians, we have

∑_{r≥0} P_{(r)}(x)f̂_{r}(z) = \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{z}(x) \\ −^{t}B_{z}(x) & O \end{matrix}) ,

which equals Π_{z}(x) by (A.5). Similarly, if n is even, then we have

∑_{r≥0} P_{(r)}(x)f̂_{r}(z) = f̂_{0}(z) + \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{z}(x) − f̂_{0}(z)1 & 1 \\ −^{t}B_{z}(x) + f̂_{0}(z)^{t}1 & 0 & 0 \\ −^{t}1 & 0 & 0 \end{matrix}) .

By adding the last row/column multiplied by f̂_{0}(z) to the second to last row/column, and by then using (A.6) with w = 0, we obtain

∑_{r≥0} P_{(r)}(x)f̂_{r}(z) = f̂_{0}(z) + \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{z}(x) & 1 \\ −^{t}B_{z}(x) & 0 & 0 \\ −^{t}1 & 0 & 0 \end{matrix}) = f̂_{0}(z) + (Π_{z}(x) − 1) .

(2) If n is even, then we have

∑_{r,s≥0} P_{(r,s)}(x)f̂_{r}(z)f̂_{s}(w) = \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) \\ −^{t}B_{z}(x) & 0 & 0 \\ −^{t}B_{w}(x) & 0 & 0 \end{matrix}) ,

and this equals (z − w)/(z + w) · (Π_{z}(x)Π_{w}(x) − 1) by (A.6). If n is odd, then we have

∑_{r,s≥0} P_{(r,s)}(x)f̂_{r}(z)f̂_{s}(w)

= ∑_{r,s>0} P_{(r,s)}(x)f̂_{r}(z)f̂_{s}(w) + ∑_{r≥0} P_{(r,0)}(x)f̂_{r}(z)f̂_{0}(w) + ∑_{s≥0} P_{(0,s)}(x)f̂_{0}(z)f̂_{s}(w)

= \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{z}(x) − f̂_{0}(z)1 & B_{w}(x) − f̂_{0}(w)1 & 1 \\ −^{t}B_{z}(x) + f̂_{0}(z)^{t}1 & 0 & 0 & 0 \\ −^{t}B_{w}(x) + f̂_{0}(w)^{t}1 & 0 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix})

+ f̂_{0}(w) · \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{z}(x) − f̂_{0}(z)1 \\ −^{t}B_{z}(x) + f̂_{0}(z)^{t}1 & 0 \end{matrix})

− f̂_{0}(z) · \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & B_{w}(x) − f̂_{0}(w)1 \\ −^{t}B_{w}(x) + f̂_{0}(w)^{t}1 & 0 \end{matrix}) .

By adding the last row/column multiplied by f̂_{0}(z) (respectively f̂_{0}(w)) to the third (respectively second) to last row/column in the first Pfaffian, we see that

Pf (\begin{matrix}A(x) & B_{z}(x) − f̂_{0}(z)1 & B_{w}(x) − f̂_{0}(w)1 & 1 \\ −^{t}B_{z}(x) + f̂_{0}(z)^{t}1 & 0 & 0 & 0 \\ −^{t}B_{w}(x) + f̂_{0}(w)^{t}1 & 0 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix})

= Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 1 \\ −^{t}B_{z}(x) & 0 & 0 & 0 \\ −^{t}B_{w}(x) & 0 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix}) .

By using the multilinearity of Pfaffians, we have

Pf (\begin{matrix}A(x) & B_{z}(x) − f̂_{0}(z)1 \\ −^{t}B_{z}(x) + f̂_{0}(z)^{t}1 & 0 \end{matrix})

= Pf (\begin{matrix}A(x) & B_{z}(x) \\ −^{t}B_{z}(x) & 0 \end{matrix}) − f̂_{0}(z) Pf (\begin{matrix}A(x) & 1 \\ −^{t}1 & 0 \end{matrix}) ,

Pf (\begin{matrix}A(x) & B_{w}(x) − f̂_{0}(w)1 \\ −^{t}B_{w}(x) + f̂_{0}(w)^{t}1 & 0 \end{matrix})

= Pf (\begin{matrix}A(x) & B_{w}(x) \\ −^{t}B_{w}(x) & 0 \end{matrix}) − f̂_{0}(w) Pf (\begin{matrix}A(x) & 1 \\ −^{t}1 & 0 \end{matrix}) .

Hence we can use (A.7), (A.5) and (A.4) to evaluate these Pfaffians and complete the proof. □

3.3. Dual P -functions and Cauchy-type identity. In this section, we introduce the dual P -functions and prove a Cauchy-type identity.

Definition 3.3. Let F = {f_{d}}_{d=0}^{∞} be an admissible sequence and F̂ = {f̂_{d}}_{d=0}^{∞} the dual of F. For a sequence α = (α_{1}, …, α_{r}) of nonnegative integers, let V̂_{α}^{F}(x) be the n × r matrix given by

V̂_{α}^{F}(x) = (f̂_{α_{j}}(x_{i}))_{1≤i≤n,1≤j≤r} .

Given a strict partition λ of length l, we define the generalized dual P -functions P̂_{λ}^{F}(x) by putting

P̂_{λ}^{F}(x) = \begin{cases}\frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & V̂_{λ}^{F}(x) \\ −^{t}V̂_{λ}^{F}(x) & O \end{matrix}) , & if n + l is even, \\ \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & V̂_{λ^{0}}^{F}(x) \\ −^{t}V̂_{λ^{0}}^{F}(x) & O \end{matrix}) , & if n + l is odd, \end{cases}  (3.7)

where λ^{0} = (λ_{1}, …, λ_{l}, 0).

If f_{d}(u) = u^{d} for d ≥ 0, then f̂_{d}(v) = 2v^{d} for d ≥ 0. Hence P_{λ}^{F}(x) = P_{λ}(x) is the original Schur P -function, and the dual P -function P̂_{λ}^{F}(x) = Q_{λ}(x) is the original Schur Q-function.

By the same arguments as in Section 2, we obtain the following proposition.

Proposition 3.4. Suppose that F = {f_{d}}_{d=0}^{∞} is a constant-term free admissible sequence with dual F̂ = {f̂_{d}}_{d=0}^{∞}. Then we have:

(1) For a strict partition of length l ≤ n, we have

P̂_{λ}^{F}(x) = \frac{1}{(n − l)!} ∑_{u∈S_{n}} u (∏_{i=1}^{l} f̂_{λ_{i}}(x_{i}) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} ) .  (3.8)

(2) For a sequence α = (α_{1}, …, α_{r}) of nonnegative integers, let Ŝ_{α}^{F}(x) be the r × r skew-symmetric matrix defined by

Ŝ_{α}^{F}(x) = (P̂_{(α_{i},α_{j})}^{F}(x))_{1≤i,j≤r} ,

where we use the same convention as (2.8). Then, for a strict partition λ of length l, we have

P̂_{λ}^{F}(x) = \begin{cases}Pf Ŝ_{λ}^{F}(x), & if l is even, \\ Pf Ŝ_{λ^{0}}^{F}(x), & if l is odd. \end{cases}  (3.9)

(3) For a strict partition λ, we have

P̂_{λ}^{F}(x_{1}, …, x_{n}, 0) = P̂_{λ}^{F}(x_{1}, …, x_{n}).  (3.10)

Proof. By Lemma 3.1 (3), we have f̂_{0} = 1 for a constant-term free admissible sequence F. Hence the proofs of Theorems 2.3, 2.6 and Proposition 2.7 (2) work literally in this dual setting. □

For Schur’s P - and Q-functions we have the following Cauchy-type identity (see [12, III, (8.13)], and [19, Theorem 5.1] for a linear algebraic proof):

∑_{λ} P_{λ}(x)Q_{λ}(y) = ∏_{i,j=1}^{n} \frac{1 + x_{i}y_{j}}{1 − x_{i}y_{j}} ,

where x = (x_{1}, …, x_{n}), y = (y_{1}, …, y_{n}), and the summation is taken over all strict partitions of length ≤ n. We can use the notion of dual P -functions to formulate a Cauchy-type identity for generalized P -functions.

Theorem 3.5. Let F be an admissible sequence of polynomials, and let x = (x_{1}, …, x_{n}) and y = (y_{1}, …, y_{n}) be two sequences of indeterminates. Then we have

∑_{λ} P_{λ}^{F}(x)P̂_{λ}^{F}(y) = ∏_{i,j=1}^{n} \frac{1 + x_{i}y_{j}}{1 − x_{i}y_{j}} ,

where λ runs over all strict partitions.

Proof. We use the same argument as in the proof of [19, Theorem 5.1]. Apply a Pfaffian version of the Cauchy–Binet formula (A.16) to the matrices

A = A(x), B = A(y), S = (f_{k}(x_{i}))_{1≤i≤n,k≥0}, T = (f̂_{k}(y_{i}))_{1≤i≤n,k≥0}.

Strict partitions λ are in bijection with subsets of ℕ satisfying #I = n mod 2 via the correspondence λ ↦ I = {λ_{1}, …, λ_{l(λ)}} or {λ_{1}, …, λ_{l(λ)}, 0}. Furthermore, we have

P_{λ}^{F}(x) = \frac{(−1)^{\binom{#I}{2}}}{Δ(x)} Pf (\begin{matrix}A(x) & S([n]; I) \\ −^{t}S([n]; I) & O \end{matrix}) ,

P̂_{λ}^{F}(y) = \frac{(−1)^{\binom{#I}{2}}}{Δ(y)} Pf (\begin{matrix}A(y) & T ([n]; I) \\ −^{t}T ([n]; I) & O \end{matrix}) ,

where S([n]; I) and T ([n]; I) are the submatrices of S and T, respectively, obtained by picking up the columns indexed by I. Since the (i, j) entry of S^{t}T is equal to (1 + x_{i}y_{j})/(1 − x_{i}y_{j}) by (3.1), we can complete the proof by using the Pfaffian evaluation (A.5). □

4. Generalized skew P -functions

In this section, we introduce generalized skew P -functions in terms of a Józefiak– Pragacz–Nimmo-type Pfaffian, and study their properties.

4.1. Józefiak–Pragacz–Nimmo-type formula. First we define generalized skew P - functions associated with an admissible sequence.

Definition 4.1. Let F = {f_{d}}_{d=0}^{∞} be an admissible sequence. For a pair of nonnegative integers r and k, we define a symmetric polynomial R_{r/k}^{F}(x) by the relation

P_{(r)}^{F}(x_{1}, …, x_{n}, y) = ∑_{k=0}^{∞} R_{r/k}^{F}(x_{1}, …, x_{n})f_{k}(y).  (4.1)

For two sequences α = (α_{1}, …, α_{r}) and β = (β_{1}, …, β_{s}) of nonnegative integers, let M_{α/β}(x) be the r × s matrix given by

M_{α/β}^{F}(x) = (R_{α_{i}/β_{s+1−j}}^{F}(x))_{1≤i≤r,1≤j≤s}.  (4.2)

For a pair of strict partitions λ of length l and μ of length m and a positive integer p, we define the generalized skew P -function P_{λ/μ,p}^{F}(x) by putting

P_{λ/μ,p}^{F}(x) = \begin{cases}Pf (\begin{matrix}S_{λ}^{F}(x) & M_{λ/μ}^{F}(x) \\ −^{t}M_{λ/μ}^{F}(x) & O \end{matrix}) , & if l ≡ p and m ≡ p mod 2, \\ Pf (\begin{matrix}S_{λ}^{F}(x) & M_{λ/μ^{0}}^{F}(x) \\ −^{t}M_{λ/μ^{0}}^{F}(x) & O \end{matrix}) , & if l ≡ p and m ≢ p mod 2, \\ Pf (\begin{matrix}S_{λ^{0}}^{F}(x) & M_{λ^{0}/μ}^{F}(x) \\ −^{t}M_{λ^{0}/μ}^{F}(x) & O \end{matrix}) , & if l ≢ p and m ≡ p mod 2, \\ Pf (\begin{matrix}S_{λ^{0}}^{F}(x) & M_{λ^{0}/μ^{0}}^{F}(x) \\ −^{t}M_{λ^{0}/μ^{0}}^{F}(x) & O \end{matrix}) , & if l ≢ p and m ≢ p mod 2, \end{cases}  (4.3)

where S_{α}^{F}(x) is the skew-symmetric matrix defined in (2.9), and λ^{0} = (λ_{1}, …, λ_{l}, 0), μ^{0} = (μ_{1}, …, μ_{m}, 0).

The main result of this section is the following theorem, which is a generalization of the Józefiak–Pragacz–Nimmo formula for Schur P /Q functions (see [21, Theorem 1] and [18, (2.22)]).

Theorem 4.2. For two sequences of variables x = (x_{1}, …, x_{n}) and y = (y_{1}, …, y_{p}), we have

P_{λ}^{F}(x, y) = ∑_{μ} P_{λ/μ,p}^{F}(x)P_{μ}^{F}(y),  (4.4)

where μ runs over all strict partitions.

We postpone the proof of this theorem to the next subsection. Before the proof, we derive several properties of generalized skew P -functions from the definition (4.3). We begin with the following property of R_{r/k}^{F}(x).

Lemma 4.3. For a positive integer r, the generalized P -function P_{(r)}^{F}(x, y) has degree at most r in y. Hence we have R_{r/k}^{F}(x) = 0 unless r ≥ k.

Proof. The coefficient of z^{r} in Π_{z}(x, y) = ∏_{i=1}^{n}(1 + x_{i}z)/(1 − x_{i}z) · (1 + yz)/(1 − yz) has degree at most r in y. On the other hand, since ord f̂_{r} = r, the coefficient of z^{r} in ∑_{r≥0} P_{(r)}^{F}(x, y)f̂_{r}(z) is a linear combination of P_{(0)}^{F}(x, y), …, P_{(r)}^{F}(x, y) with nonzero coefficient for P_{(r)}^{F}(x, y). Hence, by using (3.5) and the induction on r, we can conclude that P_{(r)}^{F}(x, y) has degree at most r in y. □

This lemma can be used to prove the following vanishing property of generalized skew P -functions. For a strict partition λ, we define its shifted diagram S(λ) by putting

S(λ) = {(i, j) ∈ ℤ^{2} : 1 ≤ i ≤ l(λ), i ≤ j ≤ λ_{i} + i − 1}.

For two strict partitions λ and μ, we write λ ⊃ μ if S(λ) ⊃ S(μ).

Proposition 4.4. Let F be an admissible sequence. For two strict partitions λ and μ, we have P_{λ/μ,p}^{F}(x) = 0 unless λ ⊃ μ.

Proof. Suppose that there exists an index k such that λ_{k} < μ_{k}. Then, if i ≥ k and j ≤ k, we have λ_{i} ≤ λ_{k} < μ_{k} ≤ μ_{j} and R_{λ_{i}/μ_{j}}(x) = 0 by Lemma 4.3. Hence the skew-symmetric matrices X appearing in the definition (4.3) of P_{λ/μ,p}^{F}(x) are of the form

X = (\begin{matrix}Z & W \\ −^{t}W & O \end{matrix} ) ,

where W has k columns and at most (k −1) nonzero rows. Since all the k ×k minors of W vanish, it follows from the Laplace expansion (Proposition A.5) that P_{λ/μ,p}^{F}(x) = Pf X = 0. □

If the polynomial sequence F is constant-term free, then the skew P -function P_{λ/μ,p}^{F} is independent of p and some formulas have simple forms.

Proposition 4.5. Suppose that F is a constant-term free admissible sequence. (1) R_{0/0}^{F}(x) = 1 and R_{r/0}^{F}(x) = P_{(r)}^{F}(x) for a positive integer r.

(2) For two strict partitions λ of length l and μ of length m, we have

P_{λ/μ,p}^{F}(x) = \{ \begin{matrix}Pf (\begin{matrix}S_{λ}^{F}(x) & M_{λ/μ}^{F}(x) \\ −^{t}M_{λ/μ}^{F}(x) & O \end{matrix}) , if l ≡ m mod 2, \\ Pf (\begin{matrix}S_{λ}^{F}(x) & M_{λ/μ^{0}}^{F}(x) \\ −^{t}M_{λ/μ^{0}}^{F}(x) & O \end{matrix}) = Pf (\begin{matrix}S_{λ^{0}}^{F}(x) & M_{λ^{0}/μ}^{F}(x) \\ −^{t}M_{λ^{0}/μ}^{F}(x) & O \end{matrix}) , \\ if l ≢ m mod 2. \end{matrix}

In particular, P_{λ/μ,p}^{F}(x) is independent of p. We write P_{λ/μ}^{F}(x) in this case.

Proof. (1) Substituting y = 0 in the definition (4.1), we obtain

P_{(r)}(x, 0) = ∑_{k=0}^{r} R_{r/k}(x)f_{k}(0).

Since P_{(r)}(x, 0) = P_{(r)}(x) by the stability property (Proposition 2.7 (2)), and f_{r}(0) = δ_{r,0} by assumption, we have R_{r/0}(x) = P_{(r)}(x).

(2) By using (1), we see that

(\begin{matrix}S_{λ}(x) & M_{λ/μ^{0}}(x) \\ −^{t}M_{λ/μ^{0}}(x) & O \end{matrix}) = (\begin{matrix}S_{λ^{0}}(x) & M_{λ^{0}/μ}(x) \\ −^{t}M_{λ^{0}/μ}(x) & O \end{matrix}) .

It remains to show that, if l(λ) ≢ p and l(μ) ≢ p mod 2, then

Pf (\begin{matrix}S_{λ^{0}}(x) & M_{λ^{0}/μ^{0}}(x) \\ −^{t}M_{λ^{0}/μ^{0}}(x) & O \end{matrix}) = Pf (\begin{matrix}S_{λ}(x) & M_{λ/μ}(x) \\ −^{t}M_{λ/μ}(x) & O \end{matrix}) .

Let T_{λ}(x) (respectively T_{λ/0}(x)) be the column vector with ith entry P_{(λ_{i})}(x) (respectively R_{λ_{i}/0}(x)). Since R_{0/0}(x) = 1 and R_{0/k}(x) = 0 for k ≥ 1, we have

(\begin{matrix}S_{λ^{0}}(x) & M_{λ^{0}/μ^{0}}(x) \\ −^{t}M_{λ^{0}/μ^{0}}(x) & O \end{matrix}) = (\begin{matrix}S_{λ}(x) & T_{λ}(x) & T_{λ/0}(x) & M_{λ/μ}(x) \\ −^{t}T_{λ}(x) & 0 & 1 & O \\ −^{t}T_{λ/0}(x) & −1 & 0 & O \\ −^{t}M_{λ/μ}(x) & O & O & O \end{matrix}) .

Since T_{λ}(x) = T_{λ/0}(x) by (1), we add the (l + 1)st row/column multiplied by −1 to the (l + 2)nd row/column and expand the resulting Pfaffian along the (l + 2)nd row/column to obtain

Pf (\begin{matrix}S_{λ^{0}}(x) & M_{λ^{0}/μ^{0}}(x) \\ −^{t}M_{λ^{0}/μ^{0}}(x) & O \end{matrix}) = Pf (\begin{matrix}S_{λ}(x) & M_{λ/μ}(x) \\ −^{t}M_{λ/μ}(x) & O \end{matrix}) .  □

If x consists of a single variable, then we have the following proposition.

Proposition 4.6. Let F be an admissible sequence and x = (x) a single variable. (1) For two strict partitions λ and μ, we have P_{λ/μ,p}^{F}(x) = 0 unless l(λ) − l(μ) ≤ 2.

(2) If F is constant-term free, then we have P_{λ/μ}^{F}(x) = 0 unless l(λ) − l(μ) ≤ 1, and

P_{λ/μ}^{F}(x) = det (R_{λ_{i}/μ_{j}}^{F}(x))_{1≤i,j≤l(λ)}.  (4.5)

Proof. Put l = l(λ) and m = l(μ).

(1) Since P_{(λ_{i},λ_{j})}(x) = 0 for λ_{i} > λ_{j} > 0 by Proposition 2.1, we have

P_{λ/μ,p}(x) = \begin{cases}Pf (\begin{matrix}O_{l} & M_{λ/μ}(x) \\ −^{t}M_{λ/μ}(x) & O_{m} \end{matrix}) , & if l ≡ p and m ≡ p mod 2, \\ Pf (\begin{matrix}O_{l} & M_{λ/μ^{0}}(x) \\ −^{t}M_{λ/μ^{0}}(x) & O_{m+1} \end{matrix}) , & if l ≡ p and m ≢ p mod 2, \\ Pf (\begin{matrix}O_{l} & T_{λ}(x) & M_{λ/μ}(x) \\ −^{t}T_{λ}(x) & 0 & O \\ −^{t}M_{λ/μ}(x) & O & O_{m} \end{matrix}) , & if l ≢ p and m ≡ p mod 2, \\ Pf (\begin{matrix}O_{l} & T_{λ}(x) & T_{λ/0}(x) & M_{λ/μ}(x) \\ −^{t}T_{λ}(x) & 0 & 1 & O \\ −^{t}T_{λ/0}(x) & −1 & 0 & O \\ −^{t}M_{λ/μ}(x) & O & O & O_{m} \end{matrix}) , & if l ≢ p and m ≢ p mod 2, \end{cases}

where T_{λ}(x) (respectively T_{λ/0}(x)) is the column vector with ith entry P_{(λ_{i})}(x) (respectively R_{λ_{i}/0}(x)). By using Proposition A.5, we see that P_{λ/μ}(x) = 0 unless

l = m,  if l ≡ p and m ≡ p mod 2,
l = m + 1,  if l ≡ p and m ≢ p mod 2,
l = m + 1,  if l ≢ p and m ≡ p mod 2,
l = m or m + 2, if l ≢ p and m ≢ p mod 2.

Here we note that m ≤ l ≤ m+2 if and only if l ≤ (l+m+2)/2 and m+1 ≤ (l+m+2)/2. (2) By using Proposition 4.5 (2) and Proposition A.5, we have

P_{λ/μ,p}^{F}(x) = \begin{cases}Pf (\begin{matrix}O_{l} & M_{λ/μ}^{F}(x) \\ −^{t}M_{λ/μ}^{F}(x) & O \end{matrix}) = (−1)^{\binom{l}{2}} det M_{λ/μ}(x), & if l = m, \\ Pf (\begin{matrix}O_{l} & M_{λ/μ^{0}}^{F}(x) \\ −^{t}M_{λ/μ^{0}}^{F}(x) & O \end{matrix}) = (−1)^{\binom{l}{2}} det M_{λ/μ^{0}}(x), & if l = m + 1. \end{cases}

By permuting columns of M_{λ/μ}(x) and M_{λ/μ^{0}}(x), we obtain (4.5). □

4.2. Proof of Theorem 4.2. We give a proof of Theorem 4.2 by using the same idea as in the proof of [19, Theorem 6.1] for Schur Q-functions. A key is played by the following proposition, which interpolates the Nimmo-type formula (1.4) (n = 0 case) and the Schurtype formula (2.10) (p = 0 case).

Proposition 4.7. Let F be an admissible polynomial sequence and x = (x_{1}, …, x_{n}) and y = (y_{1}, …, y_{p}) two sequences of indeterminates. For a nonnegative integer sequence α = (α_{1}, …, α_{r}), let N_{α}^{F}(x|y) be the r × p matrix defined by

N_{α}^{F}(x|y) = (P_{α_{i}}^{F}(x, y_{j}))_{1≤i≤r,1≤j≤p}.

Then, for a strict partition λ of length l, we have

P_{λ}^{F}(x, y) = \begin{cases}\frac{1}{Δ(y)} Pf (\begin{matrix}S_{λ}^{F}(x) & N_{λ}^{F}(x|y) \\ −^{t}N_{λ}^{F}(x|y) & A(y) \end{matrix}) , & if l + p is even, \\ \frac{1}{Δ(y)} Pf (\begin{matrix}S_{λ^{0}}^{F}(x) & N_{λ^{0}}^{F}(x|y) \\ −^{t}N_{λ^{0}}^{F}(x|y) & A(y) \end{matrix}) , & if l + p is odd, \end{cases}  (4.6)

where λ^{0} = (λ_{1}, …, λ_{l}, 0), and S_{α}^{F}(x) and A(y) are given by (2.9) and (1.3), respectively.

We denote by P′_{λ}(x|y) the right-hand side of (4.6). First we prove that P′_{λ}(x|y) satisfies a Schur-type Pfaffian formula.

Lemma 4.8. For a strict partition λ of length l, we have

P′_{λ}(x|y) = \begin{cases}Pf (P′_{(λ_{i},λ_{j})}(x|y))_{1≤i,j≤l} , & if l is even, \\ Pf (P′_{(λ_{i},λ_{j})}(x|y))_{1≤i,j≤l+1} , & if l is odd, \end{cases}

where λ_{l+1} = 0 if l is odd, and we use a convention similar to (2.8).

Proof. The proof is similar to that of Theorem 2.6, so we omit the details. We apply a Pfaffian analogue of the Sylvester formula (Proposition A.4) to the following matrix X (after permuting rows/columns):

X = \begin{cases}(\begin{matrix}S_{λ}(x) & N_{λ}(x|y) \\ −^{t}N_{λ}(x|y) & A(y) \end{matrix}) , & if l is even and p is even, \\ (\begin{matrix}S_{λ^{0}}(x) & N_{λ^{0}}(x|y) \\ −^{t}N_{λ^{0}}(x|y) & A(y) \end{matrix}) , & if l is odd and p is even, \\ (\begin{matrix}S_{λ}(x) & T_{λ}(x) & N_{λ}(x|y) \\ −^{t}T_{λ}(x) & 0 & 1^{t} \\ −^{t}N_{λ}(x|y) & −1 & A(y) \end{matrix}) , & if l is even and p is even, \\ (\begin{matrix}S_{λ}(x) & T_{λ}(x) & 0 & N_{λ}(x|y) \\ −^{t}T_{λ}(x) & 0 & 1 & O \\ 0 & −1 & 0 & 1^{t} \\ −^{t}N_{λ}(x|y) & O & −^{t}1 & A(y) \end{matrix}) , & if l is odd and p is odd, \end{cases}

where T_{λ}(x) is the column vector with ith entry P_{(λ_{i})}(x). □ Proof of Proposition 4.7. By comparing Theorem 2.6 with Lemma 4.8, the proof is reduced to showing

P_{(r)}(x, y) = P′_{(r)}(x|y), P_{(r,s)}(x, y) = P′_{(r,s)}(x|y).

By considering the generating functions and using Proposition 3.2, it is enough to prove

∑_{r≥0} P′_{(r)}(x|y)f̂_{r}(z) = \begin{cases}Π_{z}(x, y), & if n + p is odd, \\ Π_{z}(x, y) + f̂_{0}(z) − 1, & if n + p is even, \end{cases}  (4.7)

and

A GENERALIZATION OF SCHUR’S P - AND Q-FUNCTIONS 21

∑_{r,s≥0} P′_{(r,s)}(x|y)f̂_{r}(z)f̂_{s}(w)

= \{ \begin{matrix}\frac{z − w}{z + w}(Π_{z}(x, y)Π_{w}(x, y) − 1) + (f̂_{0}(w) − 1)Π_{z}(x, y) − (f̂_{0}(z) − 1)Π_{w}(x, y), \\ if n + p is odd, \\ \frac{z − w}{z + w}(Π_{z}(x, y)Π_{w}(x, y) − 1), if n + p is even. \end{matrix} (4.8)

First we prove (4.7). We put

F_{z}(x) = ∑_{r≥0} P_{(r)}(x)f̂_{r}(x), F_{z}^{+}(x) = ∑_{r≥1} P_{(r)}(x)f̂_{r}(x) = F_{z}(x) − f̂_{0}(z).

Let U_{z}(x|y) (respectively U_{z}^{+}(x|y)) be the row vector with jth entry ∑_{r≥0} P_{(r)}(x, y_{j})f̂_{r}(z) (respectively ∑_{r≥1} P_{(r)}(x, y_{j})f̂_{r}(z)). Then, by the definition of P′_{(r)}(x) and the multilinearity of Pfaffians, we have

∑_{r≥0} P′_{(r)}(x|y)f̂_{r}(z) = \begin{cases}\frac{1}{Δ(y)} Pf (\begin{matrix}0 & U_{z}(x|y) \\ −^{t}U_{z}(x|y) & A(y) \end{matrix}) , & if p is odd, \\ f̂_{0}(z) + \frac{1}{Δ(y)} Pf (\begin{matrix}0 & F_{z}^{+}(x) & U_{z}^{+}(x|y) \\ −F_{z}^{+}(x) & 0 & 1^{t} \\ −^{t}U^{+}z(x|y) & −1 & A(y) \end{matrix}) , & if p is even. \end{cases}

By adding the 2nd row/column multiplied by f̂_{0}(z) to the 1st row/column, we see that

Pf (\begin{matrix}0 & F_{z}^{+}(x) & U_{z}^{+}(x|y) \\ −F_{z}^{+}(x) & 0 & 1^{t} \\ −^{t}U^{+}z(x|y) & −1 & A(y) \end{matrix}) = Pf (\begin{matrix}0 & F_{z}^{+}(x) & U_{z}(x|y) \\ −F_{z}^{+}(x) & 0 & 1^{t} \\ −^{t}U_{z}(x|y) & −1 & A(y) \end{matrix}) .

By Proposition 3.2, we have

U_{z}(x|y) = \begin{cases}Π_{z}(x)^{t}B_{z}(y), & if n is even, \\ Π_{z}(x)^{t}B_{z}(y) + (f̂_{0}(z) − 1)^{t}1, & if n is odd. \end{cases}  (4.9)
F_{z}^{+}(x) = \begin{cases}Π_{z}(x) − 1, & if n is even, \\ Π_{z}(x) − f̂_{0}(z), & if n is odd, \end{cases}  (4.10)

where B_{z}(x) is the column vector with ith entry (1 + x_{i}z)/(1 − x_{i}z). Now we distinguish four cases according to the parity of p and n.

If p is odd and n is even, then, by using (A.5), we have

∑_{r≥0} P′_{(r)}(x|y)f̂_{r}(z) = \frac{1}{Δ(y)} · Π_{z}(x) Pf (\begin{matrix}0 & ^{t}B_{z}(y) \\ −B_{z}(y) & A(y) \end{matrix}) = Π_{z}(x)Π_{z}(y).

If p is odd and n is odd, then, by using the multilinearity of Pfaffians and (A.5), (A.4), we have

∑_{r≥0} P′_{(r)}(x|y)f̂_{r}(z) = \frac{1}{Δ(y)} [Π_{z}(x) Pf (\begin{matrix}0 & ^{t}B_{z}(y) \\ −B_{z}(y) & A(y) \end{matrix}) + (f̂_{0}(z) − 1) Pf (\begin{matrix}0 & 1^{t} \\ −1 & A(y) \end{matrix})]
= Π_{z}(x, y) + f̂_{0}(z) − 1.

If p is even and n is even, then, by using the multilinearity and the expansion of Pfaffians, (A.5) (with w = 0) and (A.4), we have

∑_{r≥0} P′_{(r)}(x|y)f̂_{r}(z)

= f̂_{0}(z)

+ \frac{1}{Δ(y)} [(Π_{z}(x) − 1) Pf (\begin{matrix}0 & 1 & 0^{t} \\ −1 & 0 & 1^{t} \\ 0 & −1 & A(y) \end{matrix}) + Π_{z}(x) Pf (\begin{matrix}0 & 0 & ^{t}B_{z}(y) \\ 0 & 0 & 1^{t} \\ −B_{z}(y) & −1 & A(y) \end{matrix})]

= Π_{z}(x, y) + f̂_{0}(z) − 1.

If p is even and n is odd, then similarly we have

∑_{r≥0} P′_{(r)}(x|y)f̂_{r}(z) = f̂_{0}(z) + \frac{1}{Δ(y)} [(Π_{z}(x) − f̂_{0}(z)) Pf (\begin{matrix}0 & 1 & 0 \\ −1 & 0 & 1^{t} \\ 0 & −1 & A(y) \end{matrix})

+ Π_{z}(x) Pf (\begin{matrix}0 & 0 & ^{t}B_{z}(y) \\ 0 & 0 & 1^{t} \\ −B_{z}(y) & −1 & A(y) \end{matrix})]

= Π_{z}(x, y).

Next we prove (4.8). We put

G_{z,w}(x) = ∑_{r,s≥0} P_{(r,s)}(x)f̂_{r}(z)f̂_{s}(w), G_{z,w}^{++}(x) = ∑_{r,s≥1} P_{(r,s)}(x)f̂_{r}(z)f̂_{s}(w).

By the definition of P′_{(r,s)}(x) and the multilinearity of Pfaffians, we see that, if p is even, then

∑_{r,s≥0} P′_{(r,s)}(x|y)f̂_{r}(z)f̂_{s}(w) = \frac{1}{Δ(y)} Pf (\begin{matrix}0 & G_{z,w}(x) & U_{z}(x|y) \\ −G_{z,w}(x) & 0 & U_{w}(x|y) \\ −^{t}U_{z}(x|y) & −^{t}U_{w}(x|y) & A(y) \end{matrix}) ,

and, if p is odd, then

∑_{r,s≥0} P′_{(r,s)}(x|y)f̂_{r}(z)f̂_{s}(w)

= ∑_{r,s≥1} P′_{(r,s)}(x|y)f̂_{r}(z)f̂_{s}(w) + ∑_{r≥1} P′_{(r,0)}(x|y)f̂_{r}(z)f̂_{0}(w) + ∑_{s≥1} P′_{(0,s)}(x|y)f̂_{0}(z)f̂_{s}(w)

= \frac{1}{Δ(y)} Pf (\begin{matrix}0 & G_{z,w}^{++}(x) & F_{z}^{+}(x) & U_{z}^{+}(x|y) \\ −G_{z,w}^{++}(x) & 0 & F_{w}^{+}(x) & U_{w}^{+}(x|y) \\ −F_{z}^{+}(x) & −F_{w}^{+}(x) & 0 & 1^{t} \\ −^{t}U_{z}^{+}(x|y) & −^{t}U_{w}^{+}(x|y) & −1 & A(y) \end{matrix})

+ f̂_{0}(w) · \frac{1}{Δ(y)} Pf (\begin{matrix}0 & U_{z}^{+}(x|y) \\ −^{t}U_{z}^{+}(x|y) & A(y) \end{matrix})

− f̂_{0}(z) · \frac{1}{Δ(y)} Pf (\begin{matrix}0 & U_{w}^{+}(x|y) \\ −^{t}U_{w}^{+}(x|y) & A(y) \end{matrix}) .

Here we note

G_{z,w}(x) = G_{z,w}^{++}(x) + f̂_{0}(w)F_{z}^{+}(x) − f̂_{0}(z)F_{w}^{+}(x),

and

U_{z}(x|y) = U_{z}^{+}(x|y) + f̂_{0}(z)^{t}1, U_{w}(x|y) = U_{w}^{+}(x|y) + f̂_{0}(w)^{t}1.

By adding the 3rd row/column multiplied by f̂_{0}(z) (respectively f̂_{0}(w)) to the 1st (respectively 2nd) row/column in the first Pfaffian, we see that

Pf (\begin{matrix}0 & G_{z,w}^{++}(x) & F_{z}^{+}(x) & U_{z}^{+}(x|y) \\ −G_{z,w}^{++}(x) & 0 & F_{w}^{+}(x) & U_{w}^{+}(x|y) \\ −F_{z}^{+}(x) & −F_{w}^{+}(x) & 0 & 1^{t} \\ −^{t}U_{z}^{+}(x|y) & −^{t}U_{w}^{+}(x|y) & −1 & A(y) \end{matrix})

= Pf (\begin{matrix}0 & G_{z,w}(x) & F_{z}^{+}(x) & U_{z}(x|y) \\ −G_{z,w}(x) & 0 & F_{w}^{+}(x) & U_{w}(x|y) \\ −F_{z}^{+}(x) & −F_{w}^{+}(x) & 0 & 1^{t} \\ −^{t}U_{z}(x|y) & −^{t}U_{w}(x|y) & −1 & A(y) \end{matrix}) .

By Proposition 3.2, we have

G_{z,w}(x)

= \begin{cases}\frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1), & if n is even, \\ \frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1) + (f̂_{0}(w) − 1)Π_{z}(x) − (f̂_{0}(z) − 1)Π_{w}(x), & if n is odd. \end{cases} (4.11)

Now we distinguish four cases according to the parity of p and n, and in each case we evaluate Pfaffians by using (4.9), (4.10) and (4.11) together with Propositions A.1 and A.2. The rest of the proof is done by straightforward computation, so we omit it. □

Now we are in the position to give a proof of Theorem 4.2. Proof of Theorem 4.2. We put l = l(λ).

First we consider the case where l ≡ p mod 2. Then we apply the Pfaffian analogue of the Cauchy–Binet formula (A.15) to the matrices given by

A = S_{λ}(x), S = (R_{λ_{i}/k}(x))_{1≤i≤l,k≥0},
B = A(y), T = (f_{k}(x_{i}))_{1≤i≤p,k≥0}.

Strict partitions μ are in bijection with subsets of ℕ satisfying #I = p mod 2 via the correspondence μ ↦ I = {μ_{1}, …, μ_{l(μ)}} or {μ_{1}, …, μ_{l(μ)}, 0}, and we see that

Pf (\begin{matrix}S_{λ}(x) & S([l]; I) \\ −^{t}S([l]; I) & O \end{matrix}) = P_{λ/μ,p}(x),
Pf (\begin{matrix}A(y) & T ([p]; I) \\ −^{t}T ([p]; I) & O \end{matrix}) = (−1)^{\binom{#I}{2}}Δ(y)P_{μ}(y).

By (4.1), the (i, j) entry of S^{t}T is equal to

∑_{k≥0} R_{λ_{i}/k}(x)f_{k}(y_{j}) = P_{(λ_{i})}(x, y_{j}).

Hence, by applying (A.15), we have

∑_{μ} P_{λ/μ,p}(x)P_{μ}(y) = \frac{1}{Δ(y)} Pf (\begin{matrix}S_{λ}(x) & N_{λ}(x, y) \\ −^{t}N_{λ}(x, y) & A(y) \end{matrix}) .

If l ≢ p mod 2, then we apply (A.15) to the matrices given by

A = S_{λ^{0}}(x), S = (R_{λ_{i}/k}(x))_{1≤i≤l+1,k≥0},
B = A(y), T = (f_{k}(y_{i}))_{1≤i≤p,k≥0}.

Then, by an argument similar to above, we obtain

∑_{μ} P_{λ/μ,p}(x)P_{μ}(y) = \frac{1}{Δ(y)} Pf (\begin{matrix}S_{λ^{0}}(x) & N_{λ^{0}}(x, y) \\ −^{t}N_{λ^{0}}(x, y) & A(y) \end{matrix}) .

Now the proof of Theorem 4.2 can be completed by using Proposition 4.7. □

5. Pieri-type rule

In this section, we give a Pieri-type rule for the product of any generalized P -function P_{λ}^{F}(x) with a Schur Q-function Q_{(r)}(x) corresponding to a one-row partition.

5.1. The ring of Schur P - and Q-functions. Let Γ^{(n)} be the subring of the ring of symmetric polynomials Λ^{(n)} = K[x_{1}, …, x_{n}]^{S_{n}} defined by

Γ^{(n)} = {f ∈ K[x_{1}, …, x_{n}]^{S_{n}} : f(t, −t, x_{3}, …, x_{n}) is independent of t}.

Then it is known that Schur P -functions {P_{λ}(x) : λ ∈ S^{(n)}} form a basis of Γ^{(n)}, where S^{(n)} is the set of all strict partitions of length ≤ n (see [20, Theorem 2.11]).

We give a relation between two families of generalized P -functions associated with different admissible sequences.

Proposition 5.1. Let F = {f_{d}}_{d=0}^{∞} and G = {g_{d}}_{d=0}^{∞} be two admissible sequences. For a strict partition λ of length l ≤ n, the generalized P -function P_{λ}^{F}(x) associated with F can be written as a K-linear combination of the generalized P -functions P_{μ}^{G}(x) associated with G in the following form:

P_{λ}^{F}(x) = a_{λ,λ}P_{λ}^{G}(x) + ∑_{μ⊊λ} a_{λ,μ}P_{μ}^{G}(x),

where a_{λ,λ} ≠ 0, and μ runs over all strict partitions satisfying μ ⊊ λ.

Proof. We write f_{k} = ∑_{i=0}^{∞} a_{k,l}g_{l} for k ≥ 0. Then by the assumption (1.2) we have a_{k,l} = 0 for k < l, a_{k,k} ≠ 0, and a_{0,0} = 1.

If n + l is even, then by using the multilinear and alternating property of Pfaffians we have

P_{λ}^{F}(x) = \frac{1}{Δ(x)} ∑_{α∈ℕ^{l}} a_{λ_{1},α_{1}} … a_{λ_{l},α_{l}} Pf (\begin{matrix}A(x) & V_{α}^{G}(x) \\ −^{t}V_{α}^{G}(x) & O \end{matrix})

= ∑_{μ_{1}>···>μ_{l}≥0} det (a_{λ_{i},μ_{j}})_{1≤i,j≤l}P_{μ}^{G}(x),

where μ runs over all strict partitions of length l − 1 or l. Similarly, if n + l is odd, then we have

P_{λ}^{F}(x) = \frac{1}{Δ(x)} ∑_{α∈ℕ^{l+1}} a_{λ_{1},α_{1}} … a_{λ_{l},α_{l}}a_{0,α_{l+1}} Pf (\begin{matrix}A(x) & V_{α}^{G}(x) \\ −^{t}V_{α}^{G}(x) & O \end{matrix}) .

Since a_{0,l} = 0 for l > 0, we see that

P_{λ}^{F}(x) = ∑_{μ_{1}>···>μ_{l}>0} det (a_{λ_{i},μ_{j}})_{1≤i,j≤l}P_{μ}^{G}(x),

where μ runs over all strict partitions of length l. For a strict partition μ of length l − 1 or l, we put

a_{λ,μ} = det (a_{λ_{i},μ_{j}})_{1≤i,j≤l},

where μ_{l} = 0 if l(μ) = l − 1. We prove that a_{λ,λ} ≠ 0 and a_{λ,μ} = 0 unless λ ⊃ μ. Since the matrix (a_{λ_{i},λ_{j}})_{1≤i,j≤l} is upper-triangular, we have a_{λ,λ} = ∏_{i=1}^{l} a_{λ_{i},λ_{i}} ≠ 0. If there is an index k such that λ_{k} < μ_{k}, then we have λ_{i} ≤ λ_{k} < μ_{k} ≤ μ_{j} and a_{λ_{i},μ_{j}} = 0 for i ≥ k and j ≤ k and thus a_{λ,μ} = 0. Hence we obtain the desired result. □

Corollary 5.2. The generalized P -functions {P_{λ}^{F}(x) : λ ∈ S^{(n)}} associated with a fixed sequence F form a basis of Γ^{(n)}.

5.2. Pieri-type rule. Let q_{r}(x) = Q_{(r)}(x) be the Schur Q-function corresponding to a one-row partition (r), and set q_{0}(x) = 1. It is known (see [12, III.(8.5)]) that Γ^{(n)} is generated by q_{r}(x) (r ≥ 1). Thus the algebra structure of Γ^{(n)} is governed by the multiplication rule for q_{r}(x)s.

Theorem 5.3. Let F = {f_{d}}_{d=0}^{∞} be an admissible sequence. We define formal power series b_{r}^{s}(z) by the relation

f_{r}(u) · \frac{1 + uz}{1 − uz} = ∑_{s=0}^{∞} b_{r}^{s}(z)f_{s}(u).  (5.1)

We define the modified Pieri coefficients c_{μ,r}^{λ} by the relation

P_{μ}^{F}(x) · q_{r}(x) = ∑_{λ} c_{μ,r}^{λ}P_{λ}^{F}(x),  (5.2)

where the summation is taken over all strict partitions. Then the generating function of modified Pieri coefficients is given by

∑_{r=0}^{∞} c_{μ,r}^{λ}z^{r} = \begin{cases}det B_{μ}^{λ}, & if n + l(μ) is even and l(λ) = l(μ), \\ det B_{μ}^{λ^{0}}, & if n + l(μ) is even and l(λ) = l(μ) − 1, \\ det B_{μ^{0}}^{λ}, & if n + l(μ) is even and l(λ) = l(μ) + 1, \\ det B_{μ^{0}}^{λ^{0}}, & if n + l(μ) is odd and l(λ) = l(μ), \\ det B_{μ^{0}}^{λ}, & if n + l(μ) is odd and l(λ) = l(μ) + 1, \\ 0, & otherwise, \end{cases}  (5.3)

where B_{β}^{α} = (b_{β_{j}}^{α_{i}}(z))_{1≤i,j≤r} for α = (α_{1}, …, α_{r}) and β = (β_{1}, …, β_{r}).

If we put

c_{μ}^{λ}(z) = ∑_{r=0}^{∞} c_{μ,r}^{λ}z^{r},

then it follows from (3.3) and (5.2) that

P_{μ}^{F}(z) · Π_{z}(x) = ∑_{λ} c_{μ}^{λ}(z)P_{λ}^{F}(z).  (5.4)

In order to prove the above theorem, we derive a Nimmo-type expression for the product P_{μ}^{F}(x) · Π_{z}(x) in terms of a Pfaffian.

Lemma 5.4. For a sequence α = (α_{1}, …, α_{r}) of nonnegative integers, we put

W_{α}(x) = (f_{α_{j}}(x_{i}) · \frac{1 + x_{i}z}{1 − x_{i}z} )_{1≤i≤n,1≤j≤r} .

Then, for a strict partition μ of length m, we have

P_{μ}^{F}(x) · Π_{z}(x) = \begin{cases}\frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & W_{μ}(x) & W_{(0)}(x) & 1 \\ −^{t}W_{μ}(x) & 0 & 0 & 0 \\ −^{t}W_{(0)}(x) & 0 & 0 & 1 \\ −^{t}1 & 0 & −1 & 0 \end{matrix}) , & if n + m is even, \\ \frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & W_{μ^{0}}(x) \\ −^{t}W_{μ^{0}}(x) & 0 \end{matrix}) , & if n + m is odd. \end{cases}

Proof. The method of the proof is the same as in the proof of Theorem 2.3, so we will only give a sketch of the proof.

Since Π_{z}(x) = ∏_{i=1}^{n}(1 + x_{i}z)/(1 − x_{i}z) is invariant under the symmetric group S_{n}, it follows from Corollary 2.5 that

P_{μ}(x) · Π_{z}(x)

= ∑_{w∈S_{n}/S_{n−m}} w (∏_{i=1}^{m} f_{μ_{i}}(x_{i}) ∏_{i=1}^{n} \frac{1 + x_{i}z}{1 − x_{i}z} ∏_{\begin{matrix}1≤i<j≤n \\ i≤m \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} )

= \frac{(−1)^{\binom{n}{2}+\binom{n−m}{2}}}{Δ_{n}(x)}

× ∑_{w∈S_{n}/S_{n−m}} sgn(w)w (∏_{i=1}^{m} (f_{μ_{i}}(x_{i})\frac{1 + x_{i}z}{1 − x_{i}z} ) ∏_{i=m+1}^{n} \frac{1 + x_{i}z}{1 − x_{i}z} ∏_{m+1≤i<j≤n} \frac{x_{j} − x_{i}}{x_{j} + x_{i}} ) .

By using (A.5) with p = 2, y_{1} = z, y_{2} = 0, or p = 1, y_{1} = z, we obtain

P_{μ}(x) · Π_{z}(x) = \frac{(−1)^{\binom{n}{2}+\binom{n−m}{2}}}{Δ(x)} ∑_{w∈S_{n}/S_{m}×S_{n−m}} sgn(w)w( det W_{μ}(x_{[m]}) Pf X),

where the skew-symmetric matrix X is given by

X = \begin{cases}(\begin{matrix}A(x_{[n]\[m]}) & W_{(0)}(x_{[n]\[m]}) & 1 \\ −^{t}W_{(0)}(x_{[n]\[m]}) & 0 & 1 \\ −^{t}1 & −1 & 0 \end{matrix}) , & if n + m is even, \\ (\begin{matrix}A(x_{[n]\[m]}) & W_{(0)}(x_{[n]\[m]}) \\ −^{t}W_{(0)}(x_{[n]\[m]}) & 0 \end{matrix}) , & if n + m is odd. \end{cases}

Now we can use a Pfaffian analogue of Laplace expansion (Proposition A.5) to complete the proof. □ Proof of Theorem 5.3. The argument is similar to that in the proof of Proposition 5.1.

First we consider the case where n + m is even. In this case, by using the multilinearity and the expansion along the last row/column, we have

Pf (\begin{matrix}A(x) & W_{μ}(x) & W_{(0)}(x) & 1 \\ −^{t}W_{μ}(x) & O & O & O \\ −^{t}W_{(0)}(x) & O & 0 & 1 \\ −^{t}1 & O & −1 & 0 \end{matrix})

= Pf (\begin{matrix}A(x) & W_{μ^{0}}(x) & 1 \\ −^{t}W_{μ^{0}}(x) & O & O \\ −^{t}1 & O & 0 \end{matrix}) + (−1)^{n+m} Pf (\begin{matrix}A(x) & W_{μ}(x) \\ −^{t}W_{μ}(x) & 0 \end{matrix}) .

Since W_{(r)}(x) = ∑_{s=0}^{∞} b_{r}^{s}(z)V_{(s)}(x) by (5.1), we can use the multilinear and alternating property to obtain

Pf (\begin{matrix}A(x) & W_{μ^{0}}(x) & 1 \\ −^{t}W_{μ^{0}}(x) & O & O \\ −^{t}1 & O & 0 \end{matrix})

= ∑_{α∈ℕ^{m+1}} ∏_{i=1}^{m+1} b_{μ_{i}}^{α_{i}}(z) Pf (\begin{matrix}A(x) & V_{α}(x) & 1 \\ −^{t}V_{α}(x) & O & O \\ −^{t}1 & O & 0 \end{matrix})

= ∑_{λ_{1}>···>λ_{m+1}≥0} det (b_{μ_{j}}^{λ_{i}}(z))_{1≤i,j≤m+1} Pf (\begin{matrix}A(x) & V_{λ}(x) & 1 \\ −^{t}V_{λ}(x) & O & O \\ −^{t}1 & O & 0 \end{matrix}) .

If λ_{m+1} = 0, then the last column of V_{λ}(x) coincides with 1, so the corresponding Pfaffian vanishes. Hence we have

\frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & W_{μ^{0}}(x) & 1 \\ −^{t}W_{μ^{0}}(x) & O & O \\ −^{t}1 & O & 0 \end{matrix}) = ∑_{λ} det B_{μ^{0}}^{λ}P_{λ}(x),

where λ runs over all strict partitions of length m + 1. Similarly we have

\frac{1}{Δ(x)} Pf (\begin{matrix}A(x) & W_{μ}(x) \\ −^{t}W_{μ}(x) & 0 \end{matrix}) = ∑_{λ} det B_{μ}^{λ^{∗}}P_{λ}(x),

where λ runs over all strict partitions of length m − 1 or m, and λ^{∗} = λ^{0} or λ. The case where n + m is odd can be treated in a similar manner. □ If F is constant-term free, then we have a simpler formula.

Corollary 5.5. If F is constant-term free, then we have

c_{μ}^{λ}(z) = \begin{cases}det B_{μ}^{λ}, & if l(λ) = l(μ), \\ det B_{μ^{0}}^{λ}, & if l(λ) = l(μ) + 1, \\ 0, & otherwise, \end{cases}

under the same notation as in Theorem 5.3.

Proof. By substituting t = 0 in (5.1) and using the assumption f_{d}(0) = δ_{d,0}, we see that b_{r}^{0}(z) = δ_{r,0}. Hence we have

det B_{μ^{0}}^{λ^{0}} = det B_{μ}^{λ}, det B_{μ}^{λ^{0}} = 0,

and we obtain the corollary. □

6. Applications to factorial P - and Q-functions

In this section, we focus on Ivanov’s factorial P - and Q-functions.

6.1. Factorial P - and Q-functions. Recall the definition of Ivanov’s factorial P - and Q-functions. Let a = (a_{0}, a_{1}, …) be parameters, called factorial parameters. We define the factorial monomial (u|a)^{d} by putting

(u|a)^{d} = ∏_{i=0}^{d−1}(u − a_{i}).

Then the factorial P -function P_{λ}(x|a) is defined to be the generalized P -function P_{λ}^{F}(x) associated with F = {(u|a)^{d}}_{d=0}^{∞} (see Definition 1.2). The factorial Q-function Q_{λ}(x|a) is defined by

Q_{λ}(x|a) = 2^{l(λ)}P_{λ}(x|a).

The factorial Q-function Q_{λ}(x|a) is also the generalized P -function P_{λ}^{F′}(x) associated with the sequence F′ = {f′_{d}}_{d=0}^{∞} given by

f′_{d}(u) = \begin{cases}1, & if d = 0, \\ 2(u|a)^{d}, & if d ≥ 1. \end{cases}

Note that the sequence F = {(u|a)^{d}}_{d=0}^{∞} is constant-term free if and only if a_{0} = 0.

Since the factorial P - and Q-functions are special cases of our generalized P -functions, we can recover some formulas in [6] and [7] from the results of Section 2. For example, we recover [7, Theorem 9.1] without assumption a_{0} = 0:

Q_{λ}(x|a) = Pf (Q_{(λ_{i},λ_{j})}(x|a))_{1≤i,j≤r} ,

where r = l(λ) if l(λ) is even and l(λ) + 1 if l(λ) is odd, and λ_{l(λ)+1} = 0. (We use the convention (2.8).)

Next we compute explicitly the dual of F = {(t|a)^{d}}_{d=0}^{∞} introduced in Section 3.

Lemma 6.1. Let F̂ = {f̂_{d}}_{d=0}^{∞} be the dual of F = {(u|a)^{d}}_{d=0}^{∞}. Then we have

f̂_{d}(v) = \begin{cases}\frac{1 + a_{0}v}{1 − a_{0}v} , & if d = 0, \\ \frac{2v^{d}}{∏_{i=0}^{d}(1 − a_{i}v)}, & if d ≥ 1. \end{cases}

Proof. Put f_{d}(u) = (u|a)^{d}. The sequence {f̂_{d}}_{d=0}^{∞} is uniquely determined by the relation (3.1). Since f_{0} = 1, we see that f̂_{0}(v) is determined by substituting u = a_{0} in (3.1), and we obtain f̂_{0}(v) = (1 + a_{0}v)/(1 − a_{0}v). Let r > 0. Since f_{k}(a_{r}) = 0 for k > r and f_{r}(a_{r}) ≠ 0, we see that f̂_{r}(t) is determined inductively by the relation

∑_{k=0}^{r} f̂_{k}(v)f_{k}(a_{r}) = \frac{1 + a_{r}v}{1 − a_{r}v} .

Hence it is enough to show

\frac{1 + a_{r}v}{1 − a_{r}v} = \frac{1 + a_{0}v}{1 − a_{0}v} + ∑_{k=1}^{r} \frac{2v^{r}}{∏_{i=0}^{k}(1 − a_{i}v)} ∏_{j=0}^{k−1}(a_{r} − a_{j}).  (6.1)

By using

\frac{1 + a_{r}v}{1 − a_{r}v} − \frac{1 − a_{0}v}{1 − a_{0}v} = \frac{2v(a_{r} − a_{0})}{(1 − a_{0}v)(1 − a_{r}v)},

and cancelling the common factor 2z(a_{r} − a_{0})/(1 − a_{0}z), we see that (6.1) is equivalent to

\frac{1}{1 − a_{r}v} = ∑_{k=1}^{r} \frac{v^{k−1}}{∏_{i=1}^{k}(1 − a_{i}v)} ∏_{j=1}^{k−1}(a_{r} − a_{j}).  (6.2)

We proceed by induction on r to prove (6.2). The case r = 1 is trivial. If r > 1, then by the induction hypothesis with factorial parameters (a_{2}, …, a_{r}), we have

\frac{1}{1 − a_{r}v} = ∑_{k=2}^{r} \frac{v^{k−2}}{∏_{i=2}^{k}(1 − a_{i}v)} ∏_{j=2}^{k−1}(a_{r} − a_{j}).

Hence we have

∑_{k=1}^{r} \frac{v^{k−1}}{∏_{i=1}^{k}(1 − a_{i}v)} ∏_{j=1}^{k−1}(a_{r} − a_{i}) = \frac{1}{1 − a_{1}v} + \frac{v(a_{r} − a_{1})}{1 − a_{1}v} ∑_{k=2}^{r} \frac{v^{k−2}}{∏_{i=2}^{k}(1 − a_{i}v)} ∏_{j=2}^{k−1}(a_{r} − a_{j})

= \frac{1}{1 − a_{1}v} + \frac{v(a_{r} − a_{1})}{1 − a_{1}v} · \frac{1}{1 − a_{r}v}

= \frac{1}{1 − a_{r}v} .

This completes the proof. □

We denote by P̂_{λ}(x|a) the dual P -function P̂_{λ}^{F}(x) associated with F = {(u|a)^{d}}_{d=0}^{∞}. If the first parameter a_{0} is equal to 0, then we can recover Korotkikh’s dual P -function (see [8, Definition 2] and (3.8)) given by

P̂_{λ}(x|a) = \frac{2^{l}}{(n − l)!} ∑_{w∈S_{n}} w (∏_{i=1}^{l} \frac{x_{i}^{λ_{i}}}{∏_{k=0}^{λ_{i}}(1 − a_{k}x_{i})} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{x_{i} + x_{j}}{x_{i} − x_{j}} )

and the Cauchy-type identity (see [8, Theorem 8])

∑_{λ} P_{λ}(x|a)P̂_{λ}(x|a) = ∏_{i,j=1}^{n} \frac{1 + x_{i}y_{j}}{1 − x_{i}y_{j}} .

6.2. Factorial skew P -functions. For two strict partitions λ and μ and a positive integer p, we denote by P_{λ/μ,p}(x|a) the generalized skew P -functions P_{λ/μ,p}^{F}(x) associated with F = {(u|a)^{d}}_{d=0}^{∞}, and call it the factorial skew P -function (see Definition 4.1). Since (u|a)^{r} depends only on a_{0}, a_{1}, …, a_{r−1}, it follows from the definition (1.4) that P_{(r)}(x|a) also depends only on a_{0}, a_{1}, …, a_{r−1}. So we write P_{(r)}(x|a_{0}, a_{1}, …, a_{r−1}) for P_{(r)}(x|a).

Proposition 6.2. We define R_{r/k}(x|a) by the relation

P_{(r)}(x, y|a) = ∑_{k=0}^{∞} R_{r/k}(x|a)(y|a)^{k}.

Then we have we have

R_{r/k}(x|a) = \begin{cases}P_{(r)}(x| − a_{0}, a_{1}, …, a_{r−1}), & if k = 0, \\ P_{(r−k)}(x|0, a_{k+1}, …, a_{r−1}), & if 1 ≤ k ≤ r − 1, \\ 1, & if k = r, \\ 0, & if k > r. \end{cases}  (6.3)

In particular, if a_{0} = 0, then we have

R_{r/k}(x|a) = P_{(r−k)}(x|0, a_{k+1}, …, a_{r−1}).  (6.4)

If x = (x) consists of a single variable, then P_{(r)}(x|a) = (x|a)^{r}. Hence we obtain

Corollary 6.3. If x = (x) consists of a single variable, then we have

R_{r/k}(x|a) = \begin{cases}(x + a_{0}) ∏_{i=1}^{r−1}(x − a_{i}), & if k = 0, \\ x ∏_{i=k+1}^{r−1}(x − a_{i}), & if 1 ≤ k ≤ r − 1, \\ 1, & if k = r, \\ 0, & if k > r. \end{cases}

In the proof of Proposition 6.2, we need the following relations for elementary symmetric polynomials e_{r}(x).

Lemma 6.4. (1) If k > 0 and l > 0, then we have

∑_{m=1}^{r−1} e_{m−k}(x_{1}, …, x_{m})e_{r−m−l}(x_{m+2}, …, x_{r}) = e_{r−k−l}(x_{1}, …, x_{r}).

(2) If l > 0, then we have

2 ∑_{m=1}^{r−1} e_{m}(x_{1}, …, x_{m})e_{r−m−l}(x_{m+2}, …, x_{r}) + e_{r−l}(−x_{1}, x_{2}, …, x_{r}) = e_{r−l}(x_{1}, x_{2}, …, x_{r}).

Proof. For 1 ≤ a < b ≤ r, we put [a, b] = \{a, a + 1, …, b\} and denote by \binom{[a,b]}{p} the set of p-element subsets of [a, b]. If we put x_{I} = ∏_{i∈I} x_{i} for I ⊂ [a, b], then we have

e_{p}(x_{a}, …, x_{b}) = ∑_{I∈\binom{[a,b]}{p}} x_{I}.

(1) We define a map

φ : ⨆_{m=1}^{r−1} \binom{[1, m]}{m − k} × \binom{[m + 2, r]}{r − m − l} → \binom{[1, r]}{r − k − l}

by φ(I, J) = I ⊔ J. Given K ∈ \binom{[1,r]}{r−k−l}, let m + 1 be the (k + 1)st smallest element in the (k + l)-element subset [r] \ K and put I = K ∩ [1, m] and J = K ∩ [m + 2, r]. Then the correspondence K ↦ (I, J) gives the inverse map of φ, and we obtain the desired identity.

(2) Since we have

e_{r−l}(−x_{1}, x_{2}, …, x_{r}) = − ∑_{\begin{matrix}I∈\binom{[1,r]}{r−l} \\ 1∈I \end{matrix}} x_{I} + ∑_{\begin{matrix}I∈\binom{[1,r]}{r−l} \\ 1∉I \end{matrix}} x_{I},

it is enough to show that

∑_{m=1}^{r−1} e_{m}(x_{1}, …, x_{m})e_{r−m−l}(x_{m+2}, …, x_{r}) = ∑_{\begin{matrix}K∈\binom{[1,r]}{r−l} \\ 1∈K \end{matrix}} x_{K}.

We define a map

ψ : ⨆_{m=1}^{r−1} \binom{[m + 2, r]}{r − m − l} → \{K ∈ \binom{[1, r]}{r − l} : 1 ∈ K\}

by ψ(J) = [1, m] ⊔ J. Given K ∈ \binom{[1,r]}{r−l} with 1 ∈ K, let m be the maximum integer m satisfying [1, m] ⊂ K, and put J = K \ [1, m]. Then the correspondence K ↦ J gives the inverse map of ψ, and we obtain the desired identity. □

Now we prove Proposition 6.2.

Proof of Proposition 6.2. We need to show that

P_{(r)}(x, y|a_{0}, …, a_{r−1}) = P_{(r)}(x| − a_{0}, a_{1}, …, a_{r−1})

+ ∑_{k=1}^{r−1} P_{(r−k)}(x|0, a_{k+1}, …, a_{r−1})(y|a_{0}, …, a_{k−1})^{k}

+ (y|a_{0}, …, a_{r−1})^{r}.  (6.5)

We compare the coefficients of P_{(k)}(x)y^{l} in the expansions of both sides, where P_{(k)}(x) is the Schur P -function. We denote by a_{k,l} and b_{k,l} the coefficients of P_{(k)}(x)y^{l} on the leftand right-hand sides, respectively.

Plugging (u|a)^{r} = ∑_{m=0}^{r}(−1)^{r−m}e_{r−m}(a_{0}, …, a_{r−1})u^{m} into the definition (1.4) and using the multilinearity of Pfaffians, we have

P_{(r)}(x|a) = \begin{cases}∑_{m=1}^{r}(−1)^{r−m}e_{r−m}(a_{0}, …, a_{r−1})P_{(m)}(x), & if n is even, \\ ∑_{m=0}^{r}(−1)^{r−m}e_{r−m}(a_{0}, …, a_{r−1})P_{(m)}(x), & if n is odd. \end{cases}  (6.6)

Since Q_{(r)}(x) = 2P_{(r)}(x), it follows from (3.3) that

1 + 2 ∑_{r=1}^{∞} P_{(r)}(x, y)z^{r} = ∏_{i=1}^{n} \frac{1 + x_{i}z}{1 − x_{i}z} · \frac{1 + yz}{1 − yz} = (1 + 2 ∑_{r=1}^{∞} P_{(r)}(x)z^{r}) (1 + 2 ∑_{r=1}^{∞} y^{r}z^{r}) .

Equating coefficients of z^{r}, we get

P_{(r)}(x, y) = P_{(r)}(x) + 2 ∑_{h=1}^{r−1} P_{(r−h)}(x)y^{h} + y^{r}.  (6.7)

Using (6.6) and (6.7), we have

P_{(r)}(x, y|a)

= \{ \begin{matrix}e_{r}(a_{0}, …, a_{r−1}) \\ + ∑_{m=1}^{r}(−1)^{r−m}e_{r−m}(a_{0}, …, a_{r−1}) (P_{(m)}(x) + 2 ∑_{l=1}^{m−1} P_{(m−l)}(x)y^{l} + y^{m}) , \\ if n is even, \\ ∑_{m=1}^{r}(−1)^{r−m}e_{r−m}(a_{0}, …, a_{r−1}) (P_{(m)}(x) + 2 ∑_{l=1}^{m−1} P_{(m−l)}(x)y^{l} + y^{m}) , \\ if n is odd. \end{matrix}

Hence the coefficient a_{k,l} of P_{(k)}(x)y^{l} on the left-hand side of (6.5) is given by

a_{k,l} = \begin{cases}e_{r}(a_{0}, …, a_{r−1}), & if k = 0, l = 0 and n is even, \\ 0, & if k = 0, l = 0 and n is odd, \\ (−1)^{r−l}e_{r−l}(a_{0}, …, a_{r−1}), & if k = 0 and l > 0, \\ 2(−1)^{r−k−l}e_{r−k−l}(a_{0}, …, a_{r−1}), & if k > 0. \end{cases}

In a similar manner, we can compute the coefficient b_{k,l} on the right-hand side of (6.5) and see that (a) if k = l = 0, then

b_{0,0} = \begin{cases}e_{r}(a_{0}, …, a_{r−1}), & if n is even, \\ 0, & if n is odd, \end{cases}

(b) if k = 0 and l > 0, then

b_{0,l} = (−1)^{r−l}e_{r−l}(a_{0}, …, a_{r−1}),

(c) if k > 0 and l = 0, then

b_{k,0} = (−1)^{r−k} ( \begin{matrix}e_{r−k}(−a_{0}, a_{1}, …, a_{r−1}) \\ + ∑_{m=1}^{r−1} e_{r−m−k}(0, a_{m+1}, …, a_{r−1})e_{m}(a_{0}, …, a_{m−1}) \end{matrix} ) ,

(d) if k > 0 and l > 0, then

b_{k,l} = (−1)^{r−k−l} ∑_{m=1}^{r−1} e_{r−m−k}(0, a_{m+1}, …, a_{r−1})e_{m−l}(a_{0}, …, a_{m−1}).

Now, by using Lemma 6.4, we see that a_{k,l} = b_{k,l} and obtain (6.5). □

By an argument similar to the one in the proof of Proposition 4.6, we can derive a determinant formula for P_{λ/μ,p}(x|a) for a single variable x.

Theorem 6.5. Let a = (a_{0}, a_{1}, …) be factorial parameters. For two strict partitions λ of length l and μ of length m, the factorial skew P -function P_{λ/μ,p}(x|a) in a single variable x is given as follows:

(1) We have P_{λ/μ,p}(x|a) = 0 unless λ ⊃ μ and m = l or l − 1.

(2) If λ ⊃ μ and m = l or l − 1, then we have

P_{λ/μ,p}(x|a) = det (R_{λ_{i}/μ_{j}}(x|a))_{1≤i,j≤l} .

Proof. It follows from Proposition 4.4 that P_{λ/μ,p}(x|a) = 0 unless λ ⊃ μ. By an argument similar to the one in the proof of Proposition 4.5, we can show:

(a) if l ≡ p and m ≡ p mod 2, then

P_{λ/μ,p}(x|a) = \begin{cases}det (R_{λ_{i}/μ_{j}}(x|a))_{1≤i,j≤l} , & if l = m, \\ 0, & otherwise, \end{cases}

(b) if l ≡ p and m ≢ p mod 2, or if l ≢ p and m ≡ p mod 2, then

P_{λ/μ,p}(x|a) = \begin{cases}det (R_{λ_{i}/μ_{j}}(x|a))_{1≤i,j≤l} , & if l = m + 1, \\ 0, & otherwise, \end{cases}

where μ_{l} = 0.

It remains to consider the case where l ≢ p and m ≢ p mod 2. By the definition (4.3) and the multilinearity of Pfaffians we have

P_{λ/μ,p}(x|a) = Pf (\begin{matrix}O_{l} & T_{λ}(x) & T_{λ/0}(x) & M_{λ/μ}(x) \\ −^{t}T_{λ}(x) & 0 & 1 & O \\ −^{t}T_{λ/0}(x) & −1 & 0 & O \\ −^{t}M_{λ/μ}(x) & O & O & O_{m} \end{matrix})

= Pf (\begin{matrix}O_{l} & T_{λ}(x) & T_{λ/0}(x) & M_{λ/μ}(x) \\ −^{t}T_{λ}(x) & 0 & 0 & O \\ −^{t}T_{λ/0}(x) & 0 & 0 & O \\ −^{t}M_{λ/μ}(x) & O & O & O_{m} \end{matrix})

+ Pf (\begin{matrix}O_{l} & T_{λ}(x) & 0 & M_{λ/μ}(x) \\ −^{t}T_{λ}(x) & 0 & 1 & O \\ 0 & −1 & 0 & O \\ −^{t}M_{λ/μ} & O & O & O_{m} \end{matrix}) ,

where T_{λ}(x) and T_{λ/0}(x) are the column vectors with ith entry P_{(λ_{i})}(x|a) and R_{λ_{i}/0}(x|a), respectively. Since P_{(r)}(x|a) = (x−a_{0}) ∏_{i=1}^{r−1}(x−a_{i}) and R_{r/0}(x|a) = (x+a_{0}) ∏_{i=1}^{r−1}(x−a_{i}) by Corollary 6.3, we see that T_{λ}(x) and T_{λ/0}(x) are linearly independent. Hence the first Pfaffian vanishes. By expanding the second Pfaffian along the (l + 2)nd row/column, we have

P_{λ/μ,p}(x) = Pf (\begin{matrix}O_{l} & M_{λ/μ}(x) \\ −^{t}M_{λ/μ}(x) & O_{m} \end{matrix}) = \begin{cases}det (R_{λ_{i}/μ_{j}}(x))_{1≤i,j≤l} , & if l = m, \\ 0, & otherwise. \end{cases}

This completes the proof. □

We can use this theorem to provide a lattice path proof of the tableau description of factorial P - and Q-functions given in [1, Theorem 2.1] and [7, Theorem 4.3] (the case where a_{0} = 0).

6.3. Modified Pieri coefficients. In this subsection, we give a combinatorial description of the modified Pieri coefficients for factorial P -functions. Recall that the skew shifted diagram S(λ/μ) = S(λ) \ S(μ) is called a border strip if it is connected and contains no 2 × 2 block of cells.

Theorem 6.6. Let a = (a_{0}, a_{1}, …) be factorial parameters. We define the modified Pieri coefficient c_{μ,r}^{λ}(a) by the relation

P_{μ}(x|a) · q_{r}(x) = ∑_{λ} c_{μ,r}^{λ}(a)P_{λ}(x|a),  (6.8)

where q_{r}(x) = Q_{(r)}(x) is Schur’s Q-function, and λ runs over all strict partitions. For two strict partitions λ and μ, we consider the generating function of modified Pieri coefficients

c_{μ}^{λ}(z|a) = ∑_{r=0}^{∞} c_{μ,r}^{λ}(a)z^{r}.

Then we have:

(1) If the skew shifted diagram S(λ/μ) contains a 2 × 2 block of cells, then we have c_{μ}^{λ}(z|a) = 0.

(2) Suppose that S(λ/μ) contains no 2 × 2 block of cells. Let

S(λ/μ) = ⨆_{i=1}^{r} S((λ_{m(i)}, …, λ_{M(i)})/(μ_{m(i)}, …, μ_{M(i)}))

be the decomposition of S(λ/μ) into a disjoint union of border strips, where m(1) ≤ M(1) < m(2) ≤ M(2) < ⋯ < m(r) ≤ M(r). Then we have

c_{μ}^{λ}(z|a) = ∏_{k∈K} \frac{1 + a_{μ_{k}}z}{1 − a_{μ_{k}}z} ∏_{i=1}^{r} \frac{2z^{λ_{m(i)}−μ_{M(i)}}}{∏_{j=μ_{M(i)}}^{λ_{m(i)}}(1 − a_{j}z)},  (6.9)

where

K = \begin{cases}\{k : 1 ≤ k ≤ l(μ), λ_{k} = μ_{k}\}, & if n + l(μ) is even, \\ \{k : 1 ≤ k ≤ l(μ) + 1, λ_{k} = μ_{k}\}, & if n + l(μ) is odd. \end{cases}

(3) In particular, the modified Pieri coefficient c_{μ,r}^{λ}(a) is a polynomial in the factorial parameters a_{0}, a_{1}, … with nonnegative integer coefficients.

Example 6.7. Let λ = (8, 6, 4, 3, 2) and μ = (6, 5, 4, 2, 1). Then the skew shifted diagram S(λ/μ) is decomposed into a disjoint union of border strips as follows:

S(λ/μ) =

= S((8, 6)/(6, 5)) ⊔ S((3, 2)/(2, 1)).

Since we have

K = \begin{cases}\{3\}, & if n is even, \\ \{3, 6\}, & if n is odd, \end{cases}

we obtain

c_{(6,5,4,2,1)}^{(8,6,4,3,2)}(z|a) = \begin{cases}\frac{1 + a_{4}z}{1 − a_{4}z} · \frac{2z^{8−5}}{∏_{i=5}^{8}(1 − a_{i}z)} · \frac{2z^{3−1}}{∏_{i=1}^{3}(1 − a_{i}z)}, & if n is even, \\ \frac{1 + a_{4}z}{1 − a_{4}z} · \frac{1 + a_{0}z}{1 − a_{0}z} · \frac{2z^{8−5}}{∏_{i=5}^{8}(1 − a_{i}z)} · \frac{2z^{3−1}}{∏_{i=1}^{3}(1 − a_{i}z)}, & if n is odd. \end{cases}

Setting a_{0} = a_{1} = ⋯ = 0, we recover the Pieri rule for Schur P -functions.

Corollary 6.8 (Morris [15, Theorem 1]). For a strict partition μ and a nonnegative integer k, we have

P_{μ}(x) · q_{r}(x) = ∑_{λ} 2^{a(λ,μ)}P_{λ}(x),

where λ runs over all strict partitions such that |λ|−|μ| = r and S(λ/μ) contains no 2×2 block, and where a(λ, μ) is the number of connected components of S(λ/μ).

Now we use Theorem 5.3 to give a proof of Theorem 6.6. Let b_{r}^{s}(z|a) be the coefficient in the expansion

(u|a)^{r} · \frac{1 + uz}{1 − uz} = ∑_{s≥0} b_{r}^{s}(z|a)(u|a)^{s}.  (6.10)

The following lemma gives an explicit formula for b_{r}^{s}(z|a).

Lemma 6.9. For two nonnegative integers r and s, we have

b_{r}^{s}(z|a) = \begin{cases}\frac{1 + a_{r}z}{1 − a_{r}z} , & if s = r, \\ \frac{2z^{s−r}}{∏_{j=r}^{s}(1 − a_{j}z)}, & if s > r, \\ 0, & otherwise. \end{cases}

Proof. We need to prove

(u|a)^{r} \frac{1 + uz}{1 − uz} = \frac{1 + a_{r}u}{1 − a_{r}u}(u|a)^{r} + ∑_{s=r+1}^{∞} \frac{2z^{s−r}}{∏_{j=r}^{s}(1 − a_{j}z)}(u|a)^{s}.  (6.11)

By dividing both sides of (6.11) by ∏_{i=0}^{r−1}(u − a_{i}) and then shifting the indices of factorial parameters, we may assume r = 0. The latter case follows from Lemma 6.1. □

We prove Theorem 6.6 by computing the determinant given in (5.3) of Theorem 5.3. Proof of Theorem 6.6. By (5.3), we see that a nonzero c_{μ}^{λ}(z|a) is equal to the determinant whose (i, j) entry is equal to b_{μ_{j}}^{λ_{i}} = b_{μ_{j}}^{λ_{i}}(z|a).

Claim 6.10. We have c_{μ}^{λ}(z|a) = 0 unless S(λ) ⊃ S(μ).

Proof. If there exists an index k such that λ_{k} < μ_{k}, then we have λ_{i} < λ_{k} < μ_{k} < μ_{j} for i ≥ k and j ≤ k. By Lemma 6.9, we have b_{μ_{j}}^{λ_{i}} = 0 for i ≥ k and j ≤ k, thus c_{μ}^{λ} = 0. □

In what follows we assume that S(λ) ⊃ S(μ). In this case, by Theorem 5.3, we have

c_{μ}^{λ}(z|a) = \begin{cases}det B_{μ}^{λ}, & if n + l(μ) is even and l(λ) = l(μ), \\ det B_{μ^{0}}^{λ^{0}}, & if n + l(μ) is odd and l(λ) = l(μ), \\ det B_{μ^{0}}^{λ}, & if l(λ) = l(μ) + 1, \\ 0, & otherwise. \end{cases}

If n + l(μ) is odd and l(λ) = l(μ) = l, then b_{μ_{j}}^{0} = 0 for 1 ≤ j ≤ l by Lemma 6.9. By expanding the determinant along the last row, we have

det B_{μ^{0}}^{λ^{0}} = b_{0}^{0} · det B_{μ}^{λ}.

Hence it is enough to compute the determinant det B_{μ^{∗}}^{λ}, where μ^{∗} = μ or μ^{0}. By abuse of notation, we write simply B_{μ}^{λ} for B_{μ}^{λ} and B_{μ^{0}}^{λ} in the following.

First we assume that S(λ/μ) is not connected. In this case, there exists an index k such that λ_{k+1} < μ_{k} or λ_{k} = μ_{k}.

Claim 6.11. Suppose that S(λ) ⊃ S(μ).

(1) If there exists an index k such that λ_{k+1} < μ_{k}, then we have

det B_{μ}^{λ} = det B_{μ′}^{λ′} · det B_{μ′′}^{λ′′},

where

′ = (λ_{1}, …, λ_{k}), μ′ = (μ_{1}, …, μ_{k}),
λ′′ = (λ_{k+1}, …, λ_{l}), μ′′ = (μ_{k+1}, …, μ_{l}).

(2) If there exists an index k such that λ_{k} = μ_{k} > 0, then we have

det B_{μ}^{λ} = det B_{μ′}^{λ′} · b_{μ_{k}}^{λ_{k}} · det B_{μ′′}^{λ′′},

where

′ = (λ_{1}, …, λ_{k−1}), μ′ = (μ_{1}, …, μ_{k−1}),
λ′′ = (λ_{k+1}, …, λ_{l}), μ′′ = (μ_{k+1}, …, μ_{l}).

Proof. (1) If i ≥ k + 1 and j ≤ k, then we have λ_{i} ≤ λ_{k+1} < μ_{k} ≤ μ_{j} and b_{μ_{j}}^{λ_{i}} = 0, thus

det B_{μ}^{λ} = det (\begin{matrix}B_{μ′}^{λ′} & ∗ \\ O & B_{μ′′}^{λ′′} \end{matrix}) = det B_{μ′}^{λ′} · det B_{μ′′}^{λ′′}.

(2) By a similar consideration, we have

det B_{μ}^{λ} = det (\begin{matrix}B_{μ′}^{λ′} & ∗ & ∗ \\ O & b_{μ_{k}}^{λ_{k}} & ∗ \\ O & 0 & B_{μ′′}^{λ′′} \end{matrix} ) = det B_{μ′}^{λ′} · b_{μ_{k}}^{λ_{k}} · det B_{μ′′}^{λ′′}.  □

Now we consider the case where S(λ/μ) is connected.

Claim 6.12. Suppose that S(λ) ⊃ S(μ) and S(λ/μ) is connected. If S(λ/μ) contains a 2 × 2 block of cells, then we have det B_{μ}^{λ} = 0.

Proof. If S(λ/μ) contains a 2×2 square, then there exists an index k such that λ_{k+1} > μ_{k}. We take the smallest such index k. Then we have

λ_{1} > μ_{1} = λ_{2} > μ_{2} = λ_{3} > ⋯ > μ_{k−1} = λ_{k} > λ_{k+1} > μ_{k}.

(Since S(λ/μ) is connected, we have λ_{i+1} ≥ μ_{i} if λ_{i+1} > 0.) It follows from Lemma 6.9 that, if t > s > r, then

b_{r}^{t} = b_{r}^{s} · \frac{z^{t−s}}{(1 − a_{s+1}z) ⋯ (1 − a_{t}z)}.  (6.12)

We proceed by induction on k. If k = 1, then the first row of B_{μ}^{λ} is a scalar multiple of the second row by (6.12), and det B_{μ}^{λ} = 0. If k > 1, then by subtracting the (k + 1)st row multiplied by z^{λ_{k}−λ_{k+1}}/(1 − a_{λ_{k+1}+1}z) ⋯ (1 − a_{λ_{k}}z) from the kth row in det B_{μ}^{λ}, and then expanding the resulting determinant along the kth row, we have

det B_{μ}^{λ} = (−1)^{k+(k−1)}b_{μ_{k−1}}^{λ_{k}} det B_{μ′}^{λ′},

where λ′ (respectively μ′) is the strict partition obtained from λ (respectively μ) by removing λ_{k} (respectively μ_{k−1}), i.e.,

λ′ = (λ_{1}, …, λ_{k−1}, λ_{k+1}, …, λ_{l}), μ′ = (μ_{1}, …, μ_{k−2}, μ_{k}, …, μ_{l}).

Since det B_{μ′}^{λ′} = 0 by the induction hypothesis, we obtain det B_{μ}^{λ} = 0. □ Now it remains to compute det B_{μ}^{λ} when S(λ/μ) is a border strip.

Claim 6.13. If S(λ/μ) is a border strip, i.e.,

λ_{1} > μ_{1} = λ_{2} > μ_{2} = λ_{3} > ⋯ > μ_{l−1} = λ_{l} > μ_{l},

then we have

det B_{μ}^{λ} = \frac{2z^{λ_{1}−μ_{l}}}{∏_{i=μ_{l}}^{λ_{1}}(1 − a_{i}z)}.

Proof. We proceed by induction on l. A direct computation verifies the cases l = 1 and l = 2. So we assume l ≥ 3. By Lemma 6.9, we have

b_{μ_{l}}^{λ_{i}} = \frac{z^{μ_{l−1}−μ_{l}}}{∏_{j=μ_{l}}^{μ_{l−1}−1}(1 − a_{j}z)} · b_{μ_{l−1}}^{λ_{i}} (1 ≤ i ≤ l − 1),

b_{μ_{l}}^{λ_{l}} = \frac{z^{μ_{l−1}−μ_{l}}}{∏_{j=μ_{l}}^{μ_{l−1}−1}(1 − a_{j}z)} · \frac{2}{1 − a_{λ_{l}}} · b_{μ_{l−1}}^{λ_{l}}.

Factor out the common term z^{μ_{l−1}−μ_{l}}/ ∏_{j=μ_{l}}^{μ_{l−1}−1}(1 − a_{j}z) from the last column of det B_{μ}^{λ}, and then subtract the lth column from the (l − 1)st column. Since we have

b_{μ_{l−1}}^{λ_{l}} − \frac{2}{1 − a_{λ_{l}}} · b_{μ_{l−1}}^{λ_{l}} = −1,

we expand the resulting determinant along the (l − 1)st column to see

det B_{μ}^{λ} = \frac{z^{μ_{l−1}−μ_{l}}}{∏_{j=μ_{l}}^{μ_{l−1}−1}(1 − a_{j}z)} · det B_{μ′}^{λ′},

where λ′ = (λ_{1}, …, λ_{l−1}) and μ′ = (μ_{1}, …, μ_{l−1}). Using the induction hypothesis, we obtain Claim 6.13. □

Combining the above claims together completes the proof of the theorem. □ Based on Theorem 6.6 (3) and some experimental evidence, we propose the following

conjecture.

Conjecture 6.14. We define f_{μ,ν}^{λ}(a) by the formula

P_{μ}(x|a)P_{ν}(x) = ∑_{λ} f_{μ,ν}^{λ}(a)P_{λ}(x|a),  (6.13)

where λ runs over all strict partitions. Then the coefficient f_{μ,ν}^{λ}(a) is a polynomial in a = (a_{0}, a_{1}, …) with nonnegative integer coefficients. More generally, if we expand the product of factorial P -functions corresponding to different factorial parameters a = (a_{0}, a_{1}, …) and −b = (−b_{0}, −b_{1}, …) as a linear combination of P_{λ}(x|a)s,

P_{μ}(x|a)P_{ν}(x| − b) = ∑_{λ} f_{μ,ν}^{λ}(a, b)P_{λ}(x|a),  (6.14)

then the coefficient f_{μ,ν}^{λ}(a, b) is a polynomial in a and b with nonnegative integer coefficients.

Cho and Ikeda [1, Theorem 4.6] gave a combinatorial formula for the Pieri-type coefficients f_{μ,(r)}^{λ}(a, −a), which implies that f_{μ,(r)}^{λ}(a, −a) is a polynomial in a_{i} ± a_{j} with i > j with nonnegative integer coefficients.

Remark 6.15. Let s_{λ}(x|a) be the factorial Schur function with factorial parameters a, and expand

s_{μ}(x|a)s_{ν}(x| − b) = ∑_{λ} m_{μ,ν}^{λ}(a, b)s_{λ}(x|a).

Then Molev and Sagan [14, Theorem 3.1] gave a combinatorial formula for the coefficient m_{μ,ν}^{λ}(a, b), which implies that m_{μ,ν}^{λ}(a, b) is a polynomial in a and b with nonnegative integer coefficients.

7. P -functions associated with classical root systems

In this section, we show that the Hall–Littlewood functions at t = −1 associated with the classical root systems can be written as generalized P -functions associated with certain polynomial sequences.

7.1. Hall–Littlewood function associated with root systems. Macdonald [13] generalized the definition of Hall–Littlewood functions to any root system. Let Φ be a root system in a Euclidean vector space V and fix a positive system Φ^{+}. We denote by Λ and Λ^{+} the weight lattice and the set of dominant weights, respectively. Let K = ℚ(t) be the rational function field in an indeterminate t, and let K[Λ] be the group algebra of Λ with basis {e^{λ} : λ ∈ Λ} over K. Let W be the Weyl group of Φ. Then the Hall–Littlewood functions associated with the root system Φ are defined as follows.

Definition 7.1. The Hall–Littlewood function ℙ_{λ}^{Φ} ∈ K[Λ] corresponding to a dominant weight λ ∈ Λ^{+} is defined by

ℙ_{λ}^{Φ} = \frac{1}{W_{λ}(t)} ∑_{w∈W} w (e^{λ} ∏_{α∈Φ^{+}} \frac{1 − te^{−α}}{1 − e^{−α}} ) ,  (7.1)

where W_{λ}(t) = ∑_{w∈W_{λ}} t^{l(w)} is the Poincaré polynomial of W_{λ}, with W_{λ} = {w ∈ W : wλ = λ} being the stabilizer of λ in W.

In this section, we consider the root systems of types X_{n} = B_{n}, C_{n} and D_{n}. Let V be the n-dimensional Euclidean vector space with orthonormal basis ε_{1}, …, ε_{n}. We put x_{i} = e^{ε_{i}} for 1 ≤ i ≤ n and write ℙ_{λ}^{Φ} = ℙ_{λ}^{Φ}(x; t). Let Φ(X_{n}) ⊂ V be the root system of type X_{n} with the positive system Φ^{+}(X_{n}) given by

Φ^{+}(B_{n}) = {ε_{i} ± ε_{j} : 1 ≤ i < j ≤ n} ∪ {ε_{i} : 1 ≤ i ≤ n},
Φ^{+}(C_{n}) = {ε_{i} ± ε_{j} : 1 ≤ i < j ≤ n} ∪ {2ε_{i} : 1 ≤ i ≤ n},
Φ^{+}(D_{n}) = {ε_{i} ± ε_{j} : 1 ≤ i < j ≤ n}.

Then the set Λ^{+}(X_{n}) of dominant weights is given by

Λ^{+}(B_{n}) = {∑_{i=1}^{n} λ_{i}ε_{i} : (λ_{i})_{i=1}^{n} ∈ ℤ^{n} ∪ (ℤ + 1/2)^{n}, λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{n} ≥ 0} ,

Λ^{+}(C_{n}) = {∑_{i=1}^{n} λ_{i}ε_{i} : (λ_{i})_{i=1}^{n} ∈ ℤ^{n}, λ_{1} ≥ λ_{2} ≥ ⋯ ≥ λ_{n} ≥ 0} ,

Λ^{+}(D_{n}) = {∑_{i=1}^{n} λ_{i}ε_{i} : (λ_{i})_{i=1}^{n} ∈ ℤ^{n} ∪ (ℤ + 1/2)^{n}, λ_{1} ≥ λ_{2} ≥ … λ_{n−1} ≥ |λ_{n}|} ,

where ℤ + 1/2 = {r + 1/2 : r ∈ ℤ}. We identify a partition λ of length l ≤ n with a dominant weight λ_{1}ε_{1} + ⋯ + λ_{l}ε_{l} ∈ Λ^{+}(X_{n}).

We note that a reciprocal Laurent polynomial g(z) = ∑_{i=−d}^{d} a_{i}z^{i} with a_{i} = a_{−i} and a_{d} ≠ 0 can be written as g(z) = f(z + z^{−1}) for some polynomial of degree d. We use the notation x + x^{−1} = (x_{1} + x_{1}^{−1}, …, x_{n} + x_{n}^{−1}). The following is the main theorem of this section.

Theorem 7.2. (1) Let F^{B} = {f_{d}^{B}}_{d=0}^{∞} be the sequence of polynomials defined by

f_{0}^{B} = 1, f_{d}^{B}(x + x^{−1}) = (x^{d} − x^{−d})\frac{x^{1/2} + x^{−1/2}}{x^{1/2} − x^{−1/2}} (d ≥ 1).

For a strict partition λ of length l ≤ n, we have

ℙ_{λ}^{Φ(B_{n})}(x; −1) = P_{λ}^{F^{B}}(x + x^{−1}).  (7.2)

(2) Let F^{C} = {f_{d}^{C}}_{d=0}^{∞} be the sequence of polynomials defined by

f_{0}^{C} = 1, f_{d}^{C}(x + x^{−1}) = (x^{d} − x^{−d})\frac{x + x^{−1}}{x − x^{−1}} (d ≥ 1).

For a strict partition λ of length l ≤ n, we have

ℙ_{λ}^{Φ(C_{n})}(x; −1) = P_{λ}^{F^{C}}(x + x^{−1}).  (7.3)

(3) Let F^{D} = {f_{d}^{D}}_{d=0}^{∞} be the sequence of polynomials defined by

f_{0}^{D} = 1, f_{d}^{D}(x + x^{−1}) = x^{d} + x^{−d} (d ≥ 1).

For a strict partition λ of length l < n, we have

ℙ_{λ}^{Φ(D_{n})}(x; −1) = P_{λ}^{F^{D}}(x + x^{−1}),  (7.4)

and, for a strict partition λ of length n, we have

ℙ_{λ}^{Φ(D_{n})}(x; −1) + ℙ_{λ′}^{Φ(D_{n})}(x; −1) = P_{λ}^{F^{D}}(x + x^{−1}),  (7.5)

where λ′ = λ_{1}ε_{1} + ⋯ + λ_{n−1}ε_{n−1} − λ_{n}ε_{n}.

The first few terms of the sequences F^{B}, F^{C} and F^{D} are

f_{1}^{B}(u) = u + 2, f_{2}^{B}(u) = u^{2} + 2u, f_{3}^{B}(u) = u^{3} + 2u^{2} − u − 2,
f_{1}^{C}(u) = u,  f_{2}^{C}(u) = u^{2},  f_{3}^{C}(u) = u^{3} − u,
f_{1}^{D}(u) = u,  f_{2}^{D}(u) = u^{2} − 2, f_{3}^{D}(u) = u^{3} − 3u.

See Lemma 7.7 for the generating functions of f_{d}^{X}(x + x^{−1}).

Definition 7.3. Let x = (x_{1}, …, x_{n}) be indeterminates and λ a strict partition of length l ≤ n. We define symmetric Laurent polynomials P_{λ}^{X}(x) and Q_{λ}^{X}(x) of type X_{n}, where X ∈ {B, C, D}, by putting

P_{λ}^{X}(x) = P_{λ}^{F^{X}}(x + x^{−1}), Q_{λ}^{X}(x) = 2^{l}P_{λ}^{F^{X}}(x + x^{−1}),

where F^{X} is the polynomial sequence given in Theorem 7.2. We call Q_{λ}^{B}(x), Q_{λ}^{C}(x) and Q_{λ}^{D}(x) the odd orthogonal Q-function, symplectic Q-function and even orthogonal Q-function, respectively.

Note that Q_{λ}^{X}(x) is obtained as the generalized P -function P_{λ}^{G}(x+x^{−1}) associated with the sequence G^{X} = {g_{d}^{X}}_{d=0}^{∞} given by

g_{d}^{X}(u) = \begin{cases}1, & if d = 0, \\ 2f^{X}(u), & if d ≥ 1. \end{cases}

In order to prove Theorem 7.2, we recall the structure of the Weyl groups of type B_{n}, C_{n} and D_{n}. Let T_{n} be the abelian group of order 2^{n} generated by t_{1}, …, t_{n} subject to the relations t_{i}^{2} = 1 (1 ≤ i ≤ n) and t_{i}t_{j} = t_{j}t_{i} (1 ≤ i, j ≤ n), and W_{n} = T_{n} ⋊ S_{n} the semidirect product of T_{n} with the symmetric group S_{n}, where S_{n} acts on T_{n} by permuting t_{1}, …, t_{n}. Put T′_{n} = {t_{1}^{u_{1}} … t_{n}^{u_{n}} : ∑_{i=1}^{n} u_{i} ≡ 0 mod 2} and W′_{n} = T′_{n} ⋊ S_{n} the semidirect product of T′_{n} with S_{n}. Then W_{n} is isomorphic to the Weyl groups of type B_{n} and C_{n}, and W′_{n} is isomorphic to the Weyl group of type D_{n}. The natural action of S_{n} on V and K[x_{1}^{±1}, …, x_{n}^{±1}] is extended to W_{n} by

t_{i}ε_{k} = \begin{cases}−ε_{i}, & if k = i, \\ ε_{k}, & if k ≠ i, \end{cases} t_{i}x_{k} = \begin{cases}x_{i}^{−1}, & if k = i, \\ x_{k}, & if k ≠ i. \end{cases}

If λ is a strict partition of length l, then the stabilizer W_{λ} of λ_{1}ε_{1} +⋯+λ_{l}ε_{l} is isomorphic to W_{n−l} for types B_{n} and C_{n}, and to W′_{n−l} for type D_{n}.

Lemma 7.4. For a strict partition λ of length l, we have

ℙ_{λ}^{Φ(B_{n})}(x; t) = ∑_{w∈W_{n}/W_{n−l}} w (∏_{i=1}^{l} x_{i}^{λ_{i}} ∏_{i=1}^{l} \frac{1 − tx_{i}^{−1}}{1 − x_{i}^{−1}} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 − tx_{i}^{−1}x_{j})(1 − tx_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})}) ,

ℙ_{λ}^{Φ(C_{n})}(x; t) = ∑_{w∈W_{n}/W_{n−l}} w (∏_{i=1}^{l} x_{i}^{λ_{i}} ∏_{i=1}^{l} \frac{1 − tx_{i}^{−2}}{1 − x_{i}^{−2}} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 − tx_{i}^{−1}x_{j})(1 − tx_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})}) ,

ℙ_{λ}^{Φ(D_{n})}(x; t) = ∑_{w∈W′_{n}/W′_{n−l}} w (∏_{i=1}^{l} x_{i}^{λ_{i}} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 − tx_{i}^{−1}x_{j})(1 − tx_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})}) .

Proof. For a general root system Φ with Weyl group W, we have (see [10, Theorem 2.8])

∑_{w∈W} w ( ∏_{α∈Φ^{+}} \frac{1 − te^{−α}}{1 − e^{−α}} ) = ∑_{w∈W} t^{l(w)}.

We can use the same argument as the one in the proof of Lemma 2.4 to prove this lemma. □

Proof of Theorem 7.2. (1) We can take {wu : w ∈ S_{n}/S_{n−l}, u ∈ T_{l}} as a complete set of coset representatives of W_{n}/W_{n−l}, where T_{l} = ⟨t_{1}, …, t_{l}⟩. Since the product

∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})}

is invariant under T_{l}, we see that

∑_{w∈T_{l}} w (∏_{i=1}^{l} x_{i}^{λ_{i}} \frac{1 + x_{i}^{−1}}{1 − x_{i}^{−1}} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})})

= ∏_{i=1}^{l} (x_{i}^{λ_{i}} \frac{1 + x_{i}^{−1}}{1 − x_{i}^{−1}} + x_{i}^{−λ_{i}} \frac{1 + x_{i}^{1}}{1 − x_{i}^{1}} ) ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})}.

By using

\frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})} = \frac{(x_{i} + x_{i}^{−1}) + (x_{j} + x_{j}^{−1})}{(x_{i} + x_{i}^{−1}) − (x_{j} + x_{j}^{−1})},

we have

ℙ_{λ}^{Φ(B_{n})}(x; −1)

= ∑_{w∈S_{n}/S_{n−l}} w (∏_{i=1}^{l} (x_{i}^{λ_{i}} − x_{i}^{−λ_{i}}) \frac{x_{i}^{1/2} + x_{i}^{−1/2}}{x_{i}^{1/2} − x_{i}^{−1/2}} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(x_{i} + x_{i}^{−1}) + (x_{j} + x_{j}^{−1})}{(x_{i} + x_{i}^{−1}) − (x_{j} + x_{j}^{−1})}) .

Comparing this with (2.5), we obtain (7.2).

(2) can be shown in the same manner as (1).

(3) Suppose that l < n. Since W_{n} = W′_{n} ⊔ W′_{n}t_{n}, W_{n−l} = W′_{n−l} ⊔ W′_{n−l}t_{n} and

∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})}

is invariant under t_{n}, we see that

ℙ_{λ}^{Φ(D_{n})}(x; −1) = ∑_{w∈W_{n}/W_{n−l}} w (∏_{i=1}^{l} x_{i}^{λ_{i}} ∏_{\begin{matrix}1≤i<j≤n \\ i≤l \end{matrix}} \frac{(x_{i} + x_{i}^{−1}) + (x_{j} + x_{j}^{−1})}{(x_{i} + x_{i}^{−1}) − (x_{j} + x_{j}^{−1})}) .

If l = n, then the stabilizer W_{λ} is trivial and we have

ℙ_{λ}^{Φ(D_{n})}(x; −1) + ℙ_{λ′}^{Φ(D_{n})}(x; −1)

= ∑_{w∈W′_{n}} w (∏_{i=1}^{n−1} x_{i}^{λ_{i}}x_{n}^{λ_{n}} ∏_{1≤i<j≤n} \frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})})

+ ∑_{w∈W′_{n}} w (∏_{i=1}^{n−1} x_{i}^{λ_{i}}x_{n}^{−λ_{n}} ∏_{1≤i<j≤n} \frac{(1 + x_{i}^{−1}x_{j})(1 + x_{i}^{−1}x_{j}^{−1})}{(1 − x_{i}^{−1}x_{j})(1 − x_{i}^{−1}x_{j}^{−1})})

= ∑_{w∈W_{n}} w (∏_{i=1}^{n} x_{i}^{λ_{i}} ∏_{1≤i<j≤n} \frac{(x_{i} + x_{i}^{−1}) + (x_{j} + x_{j}^{−1})}{(x_{i} + x_{i}^{−1}) − (x_{j} + x_{j}^{−1})}) .

The rest of the proof is the same as in the proof of (1). □

7.2. Generating functions. Since the Q-functions Q_{λ}^{X}(x) of type X are special cases of generalized P -functions, we have a Schur-type Pfaffian formula.

Proposition 7.5. For a strict partition λ of length l, we have

Q_{λ}^{X}(x) = Pf (Q_{(λ_{i},λ_{j})}^{X}(x))_{1≤i,j≤r} ,

where r = l or l+1 according to whether l is even or odd, and where we use the convention (2.8).

Hence, in order to obtain Q_{λ}^{X}(x) for a general strict partition λ, we need to know Q_{(r)}^{X}(x) and Q_{(r,s)}^{X}(x). We compute the generating functions for them. In order to state formulas, we introduce formal power series φ^{X}(z) and ψ^{X}(z) by putting

φ^{X}(z) = \begin{cases}\frac{(1 + z)^{2}}{1 + z^{2}} , & if X = B, \\ 1, & if X = C, \\ \frac{1 − z^{2}}{1 + z^{2}} , & if X = D, \end{cases} ψ^{X}(z) = \begin{cases}\frac{2z}{1 + z^{2}} , & if X = B, \\ 0, & if X = C, \\ −\frac{2z^{2}}{1 + z^{2}} , & if X = D. \end{cases}

Furthermore, we put

Π̃_{z}(x) = ∏_{i=1}^{n} \frac{(1 + x_{i}z)(1 + x_{i}^{−1}z)}{(1 − x_{i}z)(1 − x_{i}^{−1}z)}.

Then we have the following formulas.

Proposition 7.6. (1) The generating function for Q_{(r)}^{X}(x) is given by

∑_{r=0}^{∞} Q_{(r)}^{X}(x)z^{r} = φ^{X}(z)Π̃_{z}(x) − (−1)^{n}ψ^{X}(z).

(2) The generating function for Q_{(r,s)}^{X}(x) is given by

∑_{r,s≥0} Q_{(r,s)}^{X}(x)z^{r}w^{s} = \frac{(z − w)(1 − zw)}{(z + w)(1 + zw)} · φ^{X}(z)φ^{X}(w) (Π̃_{z}(x)Π̃_{w}(x) − 1)

+ (−1)^{n} (φ^{X}(z)ψ^{X}(w)Π̃_{z}(x) − φ^{X}(w)ψ^{X}(z)Π̃_{w}(x))
+ ψ^{X}(z) − ψ^{X}(w).

By a straightforward case-by-case computation, we can show the following lemma.

Lemma 7.7. We have

1 + 2 ∑_{r=1}^{∞} f_{r}^{X}(x + x^{−1})z^{r} = φ^{X}(z) · \frac{(1 + xz)(1 + x^{−1}z)}{(1 − xz)(1 − x^{−1}z)} + ψ^{X}(z).

Proof of Proposition 7.6. By Theorem 7.2 and (1.4), we see that the Q-function Q_{λ}^{X}(x) corresponding to a strict partition λ of length l is expressed as

Q_{λ}^{X}(x) = \frac{1}{Δ̃(x)} Pf (\begin{matrix}Ã(x) & Ṽ_{λ^{∗}}^{X}(x) \\ −^{t}Ṽ_{λ^{∗}}^{X}(x) & O \end{matrix}) ,

where λ^{∗} = λ or λ^{0} according to whether n + l is even or odd, and

Ã(x) = (\frac{(x_{j} + x_{j}^{−1}) − (x_{i} + x_{i}^{−1})}{(x_{j} + x_{j}^{−1}) + (x_{i} + x_{i}^{−1})})_{1≤i,j≤n} = (\frac{(x_{i} − x_{j})(1 − x_{i}x_{j})}{(x_{i} + x_{j})(1 + x_{i}x_{j})})_{1≤i,j≤n} ,

Δ̃(x) = ∏_{1≤i<j≤n} \frac{(x_{j} + x_{j}^{−1}) − (x_{i} + x_{i}^{−1})}{(x_{j} + x_{j}^{−1}) + (x_{i} + x_{i}^{−1})} = ∏_{1≤i<j≤n} \frac{(x_{i} − x_{j})(1 − x_{i}x_{j})}{(x_{i} + x_{j})(1 + x_{i}x_{j})},

Ṽ_{(α_{1},...,α_{r})}^{X}(x) = (χ(α_{j})f_{α_{j}}^{X}(x_{i} + x_{i}^{−1}))_{1≤i≤n,1≤j≤r} , χ(d) = \begin{cases}1, & if d = 0, \\ 2, & if d ≥ 1. \end{cases}

Now, by an argument similar to the one in the proof of Proposition 3.2, we can establish this proposition by using the Pfaffian evaluations in Proposition A.3 together with Lemma 7.7 and the relation φ^{X}(z) − ψ^{X}(z) = 1. The details are left to the readers. □

Appendix A. Pfaffian formulas

In this appendix, we collect several useful Pfaffian identities.

A.1. Pfaffians. Recall the definition and some properties of Pfaffians (see [5] for an exposition). Let X = (x_{ij})_{1≤i,j≤2m} be a skew-symmetric matrix of order 2m. The Pfaffian of X, denoted by Pf(X), is defined by

Pf(X) = ∑_{σ∈F_{2m}} sgn(σ) ∏_{i=1}^{m} x_{σ(2i−1),σ(2i)},  (A.1)

where F_{2m} is the set of permutations σ ∈ S_{2m} satisfying σ(1) < σ(3) < ⋯ < σ(2m − 1) and σ(2i − 1) < σ(2i) for 1 ≤ i ≤ m.

Pfaffians are multilinear and alternating in the following sense. Let X = (x_{ij})_{1≤i,j≤n} be an even-sized skew-symmetric matrix and fix a row/column index k. If the entries of the kth row and kth column of X are written as x_{i,j} = αx′_{i,j} + βx′′_{i,j} for i = k or j = k, then

Pf X = α Pf X′ + β Pf X′′,

where X′ (respectively X′′) is the skew-symmetric matrix obtained from X by replacing the entries x_{ij} for i = k or j = k with x′_{ij} (respectively x′′_{ij}). Moreover, for a permutation σ ∈ S_{n}, we have

Pf(x_{σ(i),σ(j)})_{1≤i,j≤n} = sgn(σ) Pf(x_{i,j})_{1≤i,j≤n}.

It follows that, if Y is the skew-symmetric matrix obtained from X by adding the kth row multiplied by a scalar α to the lth row and then adding the kth column multiplied by α to the lth column, then we have Pf Y = Pf X.

We use the following notations for submatrices. For a positive integer n, we put [n] = {1, 2, …, n}. Given a subset I ⊂ [n], we put Σ(I) = ∑_{i∈I} i. For an M × N matrix X = (x_{i,j})_{1≤i≤M,1≤j≤N} and subsets I ⊂ [M] and J ⊂ [N], we denote by X(I; J) the submatrix of X obtained by picking up rows indexed by I and columns indexed by J. If X is a skew-symmetric matrix, then we write X(I) for X(I; I). We use the convention that det X(∅; ∅) = 1 and Pf X(∅) = 1.

For an n×n skew-symmetric matrix X = (x_{i,j})_{1≤i,j≤n}, we have the following expansion formula along the kth row/column:

Pf X = ∑_{i=1}^{k−1}(−1)^{k+i−1}x_{i,k} Pf X([n] \ {i, k}) + ∑_{i=k+1}^{n}(−1)^{k+i−1}x_{k,i} Pf X([n] \ {k, i}). (A.2)

A.2. Schur’s Pfaffian evaluation and its variations. Recall that

A(x) = (\frac{x_{j} − x_{i}}{x_{j} + x_{i}} )_{1≤i,j≤n} , Δ(x) = ∏_{1≤i<j≤n} \frac{x_{j} − x_{i}}{x_{j} + x_{i}}

for a sequence x = (x_{1}, …, x_{n}) of indeterminates. The evaluation of the Pfaffian in (A.3) below originates from [22], and its simple proof can be found in [9]. Equation (A.4) is derived from (A.3) by specializing the last indeterminate to 0.

Proposition A.1. If n is even, then we have

Pf A(x) = Δ(x).  (A.3)

If n is odd, then we have

Pf (\begin{matrix}A(x) & 1 \\ −^{t}1 & 0 \end{matrix}) = Δ(x),  (A.4)

where 1 is the all-one column vector.

For two sequences x = (x_{1}, …, x_{n}) and y = (y_{1}, …, y_{p}) of indeterminates, we put

B(x; y) = (\frac{1 + x_{i}y_{j}}{1 − x_{i}y_{j}} )_{1≤i≤n,1≤j≤p} , Π(x; y) = ∏_{i=1}^{n} ∏_{j=1}^{p} \frac{1 + x_{i}y_{j}}{1 − x_{i}y_{j}} .

Let B_{z}(x) = B(x; (z)) be the column vector with ith entry (1 + x_{i}z)/(1 − x_{i}z) and set Π_{z}(x) = Π(x; (z)) = ∏_{i=1}^{n}(1 + x_{i}z)/(1 − x_{i}z). Then we have the following variations of Schur’s Pfaffian evaluation.

Proposition A.2. (1) If n + p is even, then we have

Pf (\begin{matrix}A(x) & B(x; y) \\ −^{t}B(x; y) & −A(y) \end{matrix}) = (−1)^{\binom{p}{2}}Δ(x)Δ(y)Π(x; y).  (A.5)

(2) If n is even, then we have

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) \\ −^{t}B_{z}(x) & 0 & 0 \\ −^{t}B_{w}(x) & 0 & 0 \end{matrix}) = Δ(x) · \frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1).  (A.6)

(3) If n is odd, then we have

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 1 \\ −^{t}B_{z}(x) & 0 & 0 & 0 \\ −^{t}B_{w}(x) & 0 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix})

= Δ(x) · \{\frac{z − w}{z + w}(Π_{z}(x)Π_{w}(x) − 1) − Π_{z}(x) + Π_{w}(x)\} . (A.7)

Proof. (1) Apply (A.3) to the indeterminates (x_{1}, …, x_{n}, −1/y_{1}, …, −1/y_{p}). (2) By applying (1) with p = 2 and (y_{1}, y_{2}) = (z, w), we have

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 \end{matrix}) = Δ(x)Π_{z}(x)Π_{w}(x) · \frac{z − w}{z + w}.

By using the multilinearity of Pfaffians, we obtain

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 \end{matrix})

= Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) \\ −^{t}B_{z}(x) & 0 & 0 \\ −^{t}B_{w}(x) & 0 & 0 \end{matrix}) + \frac{z − w}{z + w} · Pf (\begin{matrix}A(x) & B_{z}(x) & 0 \\ −^{t}B_{z}(x) & 0 & 1 \\ 0 & −1 & 0 \end{matrix}) .

The last Pfaffian is shown to be equal to Δ(x) by expanding along the last row/column and using (A.3). Thus we obtain (A.6).

(3) Applying (1) with p = 3 and (y_{1}, y_{2}, y_{3}) = (z, w, 0), we obtain

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & −1 \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} & −1 \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 & −1 \\ 1^{t} & 1 & 1 & 0 \end{matrix}) = Δ(x)Π_{z}(x)Π_{w}(x) · \frac{z − w}{z + w} · (−1)^{n+2}.

By using the multilinearity we see that

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & −1 \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} & −1 \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 & −1 \\ 1^{t} & 1 & 1 & 0 \end{matrix})

= − Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 1 \\ −^{t}B_{z}(x) & 0 & 0 & 0 \\ −^{t}B_{w}(x) & 0 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix}) − \frac{z − w}{z + w} Pf (\begin{matrix}A(x) & B_{z}(x) & 0 & 1 \\ −^{t}B_{z}(x) & 0 & 1 & 0 \\ −^{t}B_{w}(x) & −1 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix})

− Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 0 \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} & 1 \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 & 0 \\ 0 & −1 & 0 & 0 \end{matrix}) − Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 0 \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} & 0 \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 & 1 \\ 0 & 0 & −1 & 0 \end{matrix}) .

The last three Pfaffians can be evaluated by expanding them along a row/column and then using (A.4) and (A.5) with p = 1 as follows:

Pf (\begin{matrix}A(x) & B_{z}(x) & 0 & 1 \\ −^{t}B_{z}(x) & 0 & 1 & 0 \\ −^{t}B_{w}(x) & −1 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix}) = Pf (\begin{matrix}A(x) & 1 \\ −^{t}1 & 0 \end{matrix}) = Δ(x),

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 0 \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} & 1 \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 & 0 \\ 0 & −1 & 0 & 0 \end{matrix}) = − Pf (\begin{matrix}A(x) & B_{w}(x) \\ −^{t}B_{w}(x) & 0 \end{matrix}) = −Δ(x)Π_{w}(x),

Pf (\begin{matrix}A(x) & B_{z}(x) & B_{w}(x) & 0 \\ −^{t}B_{z}(x) & 0 & \frac{z − w}{z + w} & 0 \\ −^{t}B_{w}(x) & −\frac{z − w}{z + w} & 0 & 1 \\ 0 & 0 & −1 & 0 \end{matrix}) = Pf (\begin{matrix}A(x) & B_{z}(x) \\ −^{t}B_{z}(x) & 0 \end{matrix}) = Δ(x)Π_{z}(x).

If these evaluations are combined, the proof of (A.7) is completed. □

The following Pfaffian evaluations are used in Section 7. For x = (x_{1}, …, x_{n}) and y = (y_{1}, …, y_{p}), we put

Ã(x) = (\frac{(x_{j} + x_{j}^{−1}) − (x_{i} + x_{i}^{−1})}{(x_{j} + x_{j}^{−1}) + (x_{i} + x_{i}^{−1})})_{1≤i,j≤n} = (\frac{(x_{i} − x_{j})(1 − x_{i}x_{j})}{(x_{i} + x_{j})(1 + x_{i}x_{j})})_{1≤i,j≤n} ,

Δ̃(x) = ∏_{1≤i<j≤n} \frac{(x_{j} + x_{j}^{−1}) − (x_{i} + x_{i}^{−1})}{(x_{j} + x_{j}^{−1}) + (x_{i} + x_{i}^{−1})} = ∏_{1≤i<j≤n} \frac{(x_{i} − x_{j})(1 − x_{i}x_{j})}{(x_{i} + x_{j})(1 + x_{i}x_{j})},

B̃(x; y) = (\frac{(1 + x_{i}y_{j})(1 + x_{i}^{−1}y_{j})}{(1 − x_{i}y_{j})(1 − x_{i}^{−1}y_{j})})_{1≤i≤n,1≤j≤p} ,

Π̃(x; y) = ∏_{i=1}^{n} ∏_{j=1}^{p} \frac{(1 + x_{i}y_{j})(1 + x_{i}^{−1}y_{j})}{(1 − x_{i}y_{j})(1 − x_{i}^{−1}y_{j})}.

We write B̃_{z}(x) = B̃(x; (z)) and Π̃_{z}(x) = Π̃(x; (z)). Then we have the following result.

Proposition A.3. (1) If n is even, then we have

Pf Ã(x) = Δ̃(x).  (A.8)

(2) If n is odd, then we have

Pf (\begin{matrix}Ã(x) & 1 \\ −^{t}1 & 0 \end{matrix}) = Δ̃(x).  (A.9)

(3) If n + p is even, then we have

Pf (\begin{matrix}Ã(x) & B̃(x; y) \\ −^{t}B̃(x; y) & Ã(y) \end{matrix}) = Δ̃(x)Δ̃(y)Π̃(x; y).  (A.10)

(4) If n is even, then we have

Pf (\begin{matrix}Ã(x) & B̃_{z}(x) & B̃_{w}(x) \\ −^{t}B̃_{z}(x) & 0 & 0 \\ −^{t}B̃_{w}(x) & 0 & 0 \end{matrix}) = Δ̃(x)\frac{(z − w)(1 − zw)}{(z + w)(1 + zw)}(Π̃_{z}(x)Π̃_{w}(x) − 1). (A.11)

(5) If n is odd, then we have

Pf (\begin{matrix}Ã(x) & B̃_{z}(x) & B̃_{w}(x) & 1 \\ −^{t}B̃_{z}(x) & 0 & 0 & 0 \\ −^{t}B̃_{w}(x) & 0 & 0 & 0 \\ −^{t}1 & 0 & 0 & 0 \end{matrix})

= Δ̃(x) \{\frac{(z − w)(1 − zw)}{(z + w)(1 + zw)}(Π̃_{z}(x)Π̃_{w}(x) − 1) − Π̃_{z}(x) + Π̃_{w}(x)\} . (A.12)

Proof. (1) and (2) are obtained by replacing x_{i} by x_{i}+x_{i}^{−1} in (A.3) and (A.4), respectively.

(3) is obtained by applying (A.3) with (x_{1}+x_{1}^{−1}, …, x_{n}+x_{n}^{−1}, −(y_{1}+y_{1}^{−1}), …, −(y_{p}+y_{p}^{−1})).

(4) and (5) are derived from (3) by an argument similar to the ones in the proof of (A.6) and (A.7), respectively. □

A.3. Useful formulas for Pfaffians. The following propositions are Pfaffian analogues of the Sylvester identity, the Laplace expansion formula, and the Cauchy–Binet formula for determinants.

Proposition A.4 ([9, (2.5)]). Let n and m be even integers. If X is an (n+m)×(n+m) skew-symmetric matrix such that Pf X([n]) ≠ 0, then we have

Pf (\frac{Pf X([n] ∪ \{n + i, n + j\})}{Pf X([n])})_{1≤i,j≤m} = \frac{Pf X}{Pf X([n])}.  (A.13)

Proposition A.5 ([19, Corollary 2.4 (1)]). Let m and n be nonnegative integers with the same parity. If Z is an m × m skew-symmetric matrix and W an m × n matrix, then we have

Pf (\begin{matrix}Z & W \\ −^{t}W & O_{n,n} \end{matrix}) = \begin{cases}∑_{I}(−1)^{Σ(I)+\binom{m}{2}} Pf Z(I) det W ([m] \ I; [n]), & if m > n, \\ (−1)^{\binom{m}{2}} det W, & if m = n, \\ 0, & if m < n, \end{cases} (A.14)

where I runs over all (m − n)-element subsets of [n].

Proposition A.6 ([19, Theorem 3.2]). Let m and n be nonnegative integers with the same parity. Let A and B be m × m and n × n skew-symmetric matrices, and let S and T be m × l and n × l matrices, respectively. Then we have

∑_{I}(−1)^{\binom{#I}{2}} Pf (\begin{matrix}A & S([m]; I) \\ −^{t}S([m]; I) & O \end{matrix}) Pf (\begin{matrix}B & T ([n]; I) \\ −^{t}T ([n]; I) & O \end{matrix})

= Pf (\begin{matrix}A & S^{t}T \\ −T^{t}S & B \end{matrix} ) , (A.15)

∑_{I} Pf (\begin{matrix}A & S([m]; I) \\ −^{t}S([m]; I) & O \end{matrix}) Pf (\begin{matrix}B & T ([n]; I) \\ −^{t}T ([n]; I) & O \end{matrix})

= (−1)^{\binom{n}{2}} Pf (\begin{matrix}A & S^{t}T \\ −T^{t}S & −B \end{matrix}) , (A.16)

where I runs over all subsets of [l] with #I ≡ m ≡ n mod 2.

References

[1] S. Cho and T. Ikeda, Pieri rule for the factorial Schur P -functions, in “Schubert Varieties, Equivariant Cohomology and Characteristic Classes”, eds. J. Buczyński, M. Michałek, and E. Postinghel, EMS Ser. Congr. Rep., Eur. Math. Soc., 2018, pp. 25–48.

[2] A. M. Foley and R. C. King, Determinantal and Pfaffian identities for ninth variation skew Schur functions and Q-functions, preprint, arXiv:2002.11796.

[3] T. Ikeda and H. Naruse, Excited Young diagrams and equivariant Schubert calculus, Trans. Amer. Math. Soc. 361 (2009), 5193–5221.

[4] T. Ikeda and H. Naruse, K-theoretic analogues of factorial Schur P - and Q-functions, Adv. Math. 243 (2013), 22–66.

[5] M. Ishikawa and S. Okada, Identities for determinants and Pfaffians, and their applications, Sugaku Expositions 27 (2014), 85–116.

[6] V. N. Ivanov, Combinatorial formula for factorial Schur Q-functions, J. Math. Sci. (N.Y.) 107 (2001), 4195–4211.

[7] V. N. Ivanov, Interpolation analogues of Schur Q-functions, J. Math. Sci. (N.Y.) 131 (2005), 5495–5507.

[8] S. Korotkikh, Dual multiparameter Schur Q-functions, J. Math. Sci. (N.Y.) 224 (2) (2017), 263– 268.

[9] D. E. Knuth, Overlapping Pfaffians, Electron. J. Combin., 3 (no. 2, The Foata Festschrift) (1996), Art. #R5, 13 pp.

[10] I. G. Macdonald, The Poincaré series of a Coxeter group, Math. Ann. 199 (1972), 161–174.

[11] I. G. Macdonald, Schur functions: Theme and variations, Sém. Lothar. Combin. 28 (1992), 5–39.

[12] I. G. Macdonald, “Symmetric Functions and Hall Polynomials, 2nd edition”, Oxford Univ. Press, 1995.

[13] I. G. Macdonald, Orthogonal polynomials associated with root systems, Sém. Lothar. Combin. 45 (2000/01), Art. B45a, 40 pp.

[14] A. I. Molev and B. E. Sagan, A Littlewood–Richardson rule for factorial Schur functions, Trans. Amer. Math. Soc. 351 (1999), 4429–4443.

[15] A. O. Morris, A note on the multiplication of Hall functions, J. London Math. Soc. 39 (1964), 481–488.

[16] J. Nakagawa, M. Noumi, M. Shirakawa, and Y. Yamada, Tableau representation for Macdonald’s ninth variation of Schur functions, in “Physics and Combinatorics 2000, Proceedings of the Nagoya 2000 International Workshop” (A. N. Kirillov and N. Liskova, eds.), World Scientific, 2001, pp. 180–195,

[17] M. Nakagawa and H. Naruse, Generalized (co)homology of the loop spaces of classical groups and the universal factorial Schur P - and Q-functions, in “Schubert Calculus — Osaka 2012” (H. Naruse, T. Ikeda, M. Masuda and T. Tanisaki, eds.), Adv. Stud. Pure Math. 171 (2016), pp. 337–417.

[18] J. J. C. Nimmo, Hall–Littlewood symmetric functions and the BKP equation, J. Phys. A 23 (1990), 751–760.

[19] S. Okada, Pfaffian formulas and Schur Q-function identities, Adv. Math. 353 (2019), 446–470.

[20] P. Pragacz, Algebro-geometric applications of Schur S- and Q-polynomials, in “Topics in Invariant Theory”, Séminaire d’Algèbre Dubreil–Malliavin 1989-90, Lecture Notes in Math. 1478, Springer– Verlag, 1991, pp. 130–191.

[21] P. Pragacz and T. Józefiak, A determinantal formula for skew Q-functions, J. London Math. Soc. (2) 43 (1991), 76–90.

[22] I. Schur, Über die Darstellung der symmetrischen und der alternierenden Gruppe durch gebrochene lineare Substitutionen, J. Reine Angew. Math. 139 (1911), 155–250.

[23] J. R. Stembridge, Nonintersecting paths, pfaffians, and plane partitions, Adv. Math. 83 (1990), 96–131.

Graduate School of Mathematics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya 464-8602, Japan

Email address: okada@math.nagoya-u.ac.jp