j. differential geometry 56 (2000) 67-92

SEMIALGEBRAIC SARD THEOREM FOR GENERALIZED CRITICAL VALUES

K. KURDYKA, P. ORRO & S. SIMON

Abstract

We prove that a semialgebraic differentiable mapping has a generalized critical values set of measure zero. Moreover, if the mapping is C^{2} we obtain, by a generalisation of Ehresmann’s fibration theorem due to P. J. Rabier [20], a locally trivial fibration over the complement of this set. In the complex case, we prove that the set of generalized critical values of a polynomial mapping is a proper algebraic set.

1. Introduction

The usual Sard’s theorem says that the set K_{0}(f) of critical values of a C^{p} map f : ℝ^{n} → ℝ^{k} has zero Lebesgue measure when p ≥ max(1, n − k + 1). The Ehresmann’s fibration theorem asserts that a proper submersion is a locally trivial fibration. Thus K_{0}(f) is a bifurcation set of a proper map and is a small set.

The fibration theorem has been generalized in different ways:

• For general non-proper functions, R. S. Palais introduced a condition, known as the (C) condition of Palais. Roughly speaking it means that the norm of the differential of f is separated from zero uniformly fibrewise. Palais proved that a function from a Hilbertian manifold M to ℝ satisfying condition (C) is still a fibration outside of K_{0}(f). Later he generalized this result in the case M is a complete Finsler manifold (cf. [15]).

• For general non-proper mappings f : M → N, where M and N are Finsler manifolds, P. J. Rabier [20] introduced the notion of strong submersion which generalizes condition (C) of Palais. The norm of the differential is replaced by ν(df) (as we explain in Section 2, the function ν of Rabier is simply the distance to the set of singular operators). Under the hypothesis of completeness of M and some technical assumption (which always holds for Hilbert manifolds) Rabier proves that every strong submersion is a fibration.

Received November 22, 1999.

• For polynomials f : ℂ^{n} → ℂ, it is well known that f is a fibration outside the bifurcation set B(f), which is a finite set consisting of points in K_{0}(f) and critical points at infinity. Many authors tried to make precise the bifurcation set: [16], [19], [20], [22] — introducing different conditions at infinity such as quasi-tame, Malgrange condition, M-tame, ….

In this paper we are interested in the case of semialgebraic mappings from ℝ^{n} to ℝ^{k} (or polynomial from ℂ^{n} to ℂ^{k}). We prove that the set of generalized critical values K(f) = K_{0}(f) ∪ K_{∞}(f), where

K_{∞}(f) := {y ∈ 𝕂^{k} : ∃x_{l} ∈ 𝕂^{n}, |x_{l}| → ∞
s.t. f(x_{l}) → y and (1 + |x_{l}|)ν(df(x_{l})) → 0},

is a zero-measure semialgebraic subset of 𝕂^{k} (constructible if 𝕂 = ℂ).

Using Rabier’s results [20], this gives a fibration theorem for f over the connected components of 𝕂^{k} − K(f). The main point for the fibration result is the completeness of 𝕂^{n}. If we consider a map f defined on an open set of 𝕂^{n}, a similar result is valid if f is without fibers adherent to ∂U. Another way of proving a fibration result for maps defined on open sets is to take a complete metric on U and define a new K_{∞}-set relative to this metric.

The set K_{∞}(f) is called the set of asymptotic critical values (we use the notation of [20] but we take the definition given in Remark 6.10 [20], see our Section 2). The fact that K_{∞}(f) is finite, for semialgebraic functions f : ℝ^{n} → ℝ, plays a crucial role in a proof of the Gradient Conjecture of R. Thom (cf. [12], [13]). A different proof of the finiteness of K_{∞}(f) is given by D. d’Acunto ([1]) in a more general setting of o-minimal structures.

The proof of the main result, i.e., that for a semialgebraic mapping f : ℝ^{n} → ℝ^{k}, the set K_{∞}(f) has zero measure (more precisely that dim K_{∞}(f) < k) is inspired by Y. Yomdin’s quantitative Sard theorem for polynomials [23], see also S. K. Donaldson [4, Section 5]. We decompose the set on which |x|ν(df(x)) is small in a parametric family of L-regular cells (a L-regular cell is a subset of ℝ^{n} on which the geodesic distance is equivalent to the euclidian distance); and then, by giving an estimate for length of the image by a component of f of each cell (Lemma 3.6), we deduce the result. We give also an alternative proof of Lemma 3.6 based on a result of B. Teissier [21].

The theory of generalized critical values appears as an alternative to the stratification theory. Actually our main result (in the real case) can be also proved using the (w) condition of Kuo and Verdier. However our approach via L-regular sets is conceptually simpler, and moreover gives an explicit bound for the number of points in K_{∞}(f) in term of the degree of f for k = 1 (cf. Remark 3.2).

For a polynomial mapping f : 𝕂^{n} → 𝕂^{k} let J_{f} ⊂ 𝕂^{k} denote the set of points where f is not proper, we call it the Jelonek set of f. It was proved by Z. Jelonek that J_{f} is 𝕂-uniruled (i.e., by each point passes a curve with a polynomial parametrization), if n = k and f is dominant. Moreover in the complex case J_{f} is a hypersurface or an empty set. In Proposition 3.1 we prove that K_{∞}(f) = J_{f}, if f is a C^{1} semialgebraic mapping from ℝ^{n} to ℝ^{n}. We also prove (Theorem 3.4) that for polynomial dominant mappings from ℝ^{n} to ℝ^{k}, with compact regular fibers, the equality K_{∞}(f) = J_{f} holds as well. Moreover, by a result of Z. Jelonek [8], the set J_{f} is also ℝ-uniruled in this case. If the generic fibers are not compact, then of course K_{∞}(f) ≠ J_{f}. We conjecture that for a polynomial mapping f : 𝕂^{n} → 𝕂^{k} the set K_{∞}(f) is 𝕂-uniruled.

In Section 2 we recall the definition of the ν function of Rabier and we give several equivalent definitions of ν. In particular we prove that this function is equal to the distance to the set of singular operators, at our knowledge this fact was only known in the finite dimensional case when the space of linear operators is endowed with the L^{2}-norm. We decided to write down this part in the setting of Banach spaces since the theorem of Rabier has potentially a lot of applications in this case. Finally we recall another way for measuring the distance to the set of singular operators (valid for finite dimensional target), the so called Kuo distance which was introduced by T. C. Kuo in [14]. It turns out to be equivalent to ν and is more convenient in our construction. We discuss also two possible definitions of the set of generalized critical values and we recall some examples which show that these definitions are not equivalent.

The proofs of our main results are given in Section 3 in the real case, Theorem 3.1, and in Section 4, Theorem 4.1, in the complex case.

We are happy to thank P. J. Rabier and Z. Jelonek for valuable remarks on our paper.

2. Preliminaries and notations

2.1 Distance to the set of singular mappings

Let X, Y be Banach spaces (over 𝕂 = ℝ or ℂ). We will denote by L(X, Y) the Banach space of linear continuous mappings from X to Y, and by Σ the singular set of L(X, Y) that is the subset of operators which are not onto. The distance function to Σ is defined as usual by

dist(A, Σ) = inf_{B∈Σ} ∥A − B∥.

If Y = 𝕂, it is customary to write X′ for L(X, 𝕂). For A ∈ L(X, Y), A^{∗} stands for the algebraic adjoint operator in L(Y′, X′).

Definition 2.1 ([20]). Let A ∈ L(X, Y). Set

ν(A) = inf_{∥φ∥=1} ∥A^{∗}φ∥.

Notice that, since X, Y are Banach spaces, ν(A) > 0 ⇔ A is onto. The properties in the next proposition are well known and easy to establish, see for instance [20]

Proposition 2.1. Let A ∈ L(X, Y).

1. If Y = 𝕂 then ν(A) = ∥A∥.

2. If A ∈ GL(X, Y) then ν(A) = ∥A^{−1}∥^{−1}.

3. ν is 1-Lipschitz.

The two functions ν and dist(·, Σ) vanish on Σ. We prove that they are in fact the same. We begin by a lemma.

Lemma 2.1. Let A ∈ L(X, Y) be a surjective map and ϵ > 0. Then

dist(A, Σ) ≤ ν(A) + ϵ.

Proof. By definition of ν(A), there exists some φ_{ϵ} ∈ Y′, ||ϕ_{ϵ}|| = 1, such that ∥φ_{ϵ} ∘ A∥ ≤ ν(A) + ϵ. Now, consider the topological sum Y = ker φ_{ϵ} ⊕ ⟨y_{ϵ}⟩, where y_{ϵ} is any vector satisfying φ_{ϵ}(y_{ϵ}) = 1. Then, denote by p_{ϵ} the canonical projection onto ⟨y_{ϵ}⟩ and define B_{ϵ} = A − p_{ϵ} ∘ A ∈ Σ. Therefore,

∥A − B_{ϵ}∥ = ∥p_{ϵ} ∘ A∥ = sup_{∥x∥=1} ∥p_{ϵ} ∘ A(x)∥ = ∥y_{ϵ}∥.∥φ_{ϵ} ∘ A∥.

So that, \frac{dist(A, Σ)}{∥y_{ϵ}∥} ≤ ν(A) + ϵ.

Now, remember that if ψ ∈ Y′−{0} then for all y ∈ Y, dist(y, ker ψ) =

\frac{|ψ(y)|}{∥ψ∥}. It follows that dist(0, ker φ_{ϵ} + y_{ϵ}) = 1. Consequently, there ex-

ists a sequence u_{n} in ker φ_{ϵ} such that y_{ϵ}^{n} := y_{ϵ} + u_{n} satisfies φ_{ϵ}(y_{ϵ}^{n}) = 1 for all n and ∥y_{ϵ}^{n}∥ ↘ 1. At last, apply the preceding construction in substituting y_{ϵ} by y_{ϵ}^{n} and get the result. q.e.d.

Proposition 2.2. Let A ∈ L(X, Y). We have

ν(A) = dist(A, Σ).

Proof. The inequality ν(A) ≤ dist(A, Σ) is a consequence of the fact that ν is 1-Lipschitz, whereas ν(A) ≥ dist(A, Σ) follows from Lemma 2.1.

q.e.d.

Remark 2.1. Note that ν(A) is the maximum radius R such that the open ball of center A and radius R does not intersect Σ.

Another characterization of ν is as follows:

Proposition 2.3. Let A ∈ L(X, Y). Then

ν(A) = sup{r ≥ 0 : B(0, r) ⊂ A(B(0, 1))}.

Proof. Let R denote sup{r ≥ 0 : B(0, r) ⊂ A(B(0, 1))}.

We first prove that ν(A) ≥ R. Let r < R and φ ∈ Y′ such that ∥φ∥ = 1. For all ϵ > 0, there exists y ∈ S_{Y} (0, 1) with |φ(y)| ≥ 1 − ϵ. Then, there exists x ∈ B_{X}(0, 1) verifying |φ(Ax)| ≥ r|φ(y)| ≥ r(1 − ϵ). Hence ∥φ ∘ A∥ ≥ r(1 − ϵ), so ν(A) ≥ r(1 − ϵ), and finally, ν(A) ≥ R.

Now, we prove that ν(A) ≤ R. Let ϵ > 0 and y ∈ B_{Y} (0, R + ϵ) \ A(B_{X}(0, 1)). Denote H = A^{−1}(⟨y⟩). Then A_{|H} defines λ ∈ H′ by A(h) = λ(h)y for h ∈ H. Applying Hahn-Banach theorem, we extend λ to X and denote this extension by λ̃ which is such that ∥λ̃∥ = ∥λ∥ < 1. Now, B_{y} = A − λ̃y ∈ Σ and ∥A − B_{y}∥ ≤ R + ϵ. Thus, ν(A) ≤ R + ϵ.

q.e.d.

Assume that X, Y are complex normed vector spaces, then we have on them the induced structures of real normed vector spaces. Let A : X → Y be a continuous ℂ-linear map. We can consider ν(A) with respect to both structures. It follows immediatly from Proposition 2.3 that ν(A) is the same in the real and complex case. In particular, by Proposition 2.2 we have

(2.1) ν(A) = dist(A, Σ_{ℂ}) = dist(A, Σ_{ℝ}),

where Σ_{ℂ} (resp. Σ_{ℝ} ) is the set of nonsurjective ℂ-linear (resp. ℝ-linear) continuous maps from X to Y.

Till the end of Subsection 2.2 we will consider only the finite dimensional case.

Proposition 2.4. If the norms on ℝ^{n} and ℝ^{k} are semialgebraic, then ν is semialgebraic on L(ℝ^{n}, ℝ^{k}).

Proof. Since Σ is a semialgebraic set in this case, the result is a consequence of Proposition 2.2 and the classical fact (cf. [2]) that the distance function to a semialgebraic set is semialgebraic (i.e., that its graph is semialgebraic). q.e.d.

Let 𝕂 = ℝ or ℂ, we will always consider 𝕂^{n} equipped with an hermitian scalar product, which will be denoted by ( . | . ).

The first and well known characterization of ν(A) is the following:

Proposition 2.5. Let A ∈ L(𝕂^{n}, 𝕂^{k}). Then

ν(A) = min{|μ| : μ^{2} is an eigenvalue of AA^{∗}}.

We shall need an expression of ν in terms of gradients of components of a linear mapping. To this end we introduce the Kuo distance κ which is actually equivalent to ν, as we show in Proposition 2.6.

2.2 The Kuo distance

Definition 2.2 ([14]). Let η_{1}, …, η_{k} ∈ 𝕂^{n} for k ∈ ℕ^{∗}. The Kuo distance between these vectors is defined by

κ(η_{1}, …, η_{k}) = min_{1≤i≤k} dist(η_{i}, ⟨(η_{j})_{j≠i}⟩),

where ⟨(η_{j})_{j≠i}⟩ is the vector space generated by the vectors (η_{j})_{j≠i}.

Proposition 2.6. Let A = (A_{1}, …, A_{k}) ∈ L(𝕂^{n}, 𝕂^{k}). For i ∈ {1, …, k}, denote by η_{i} the gradient of A_{i}, then

ν(A) ≤ κ(η_{1}, …, η_{k}) ≤ \sqrt{k} ν(A).

Proof. For 1 ≤ i ≤ k, set V_{i} = {x ∈ 𝕂^{n} : (x|η_{j}) = 0, j ≠ i}. Then V_{i}^{⊥} = ⟨(η_{j})_{j≠i}⟩. So dist(η_{i}, V_{i}^{⊥}) = |p_{V_{i}}(η_{i})| = sup{|(x|η_{i})| : x ∈ V_{i}, |x| ≤ 1} where p_{V_{i}} : 𝕂^{n} → V_{i} is the projection. Denote ξ_{i} = f(x_{i}) with x_{i} ∈ V_{i} ∩ S(1) (S(r) is the sphere of radius r centered at 0) such that |ξ_{i}| = |p_{V_{i}}(η_{i})|. Then (ξ_{i})_{1≤i≤k} is an orthogonal family in 𝕂^{k}. We conclude using the next Lemma 2.2. q.e.d.

Lemma 2.2. Let (ξ_{1}, …, ξ_{k}) be an orthogonal basis of 𝕂^{k} and u ∈ S(1), then

i) there exists i_{0} ∈ \{1, …, k\} such that \frac{1}{\sqrt{k}} ∥ξ_{i_{0}}∥ ≤ |(u|ξ_{i_{0}})|.

ii) let (e_{1}, …, e_{k}) be an orthonormal basis of 𝕂^{k}, and denote by E

the ellipsoid defined in this basis by \{x : ∑_{i=1}^{k} |\frac{x_{i}}{a_{i}} |^{2} = 1\} where

0 < |a_{1}| ≤ |a_{2}| ≤ … ≤ |a_{k}| ∈ ℝ_{+}^{∗} for 1 ≤ i ≤ k. Suppose that

ξ_{i} ∈ E for 1 ≤ i ≤ k. Then min_{1≤i≤k} ∥ξ_{i}∥ ≤ \sqrt{k} |a_{1}|.

Proof. Write ∥u∥^{2} = 1 = ∑_{i=1}^{k} |(u | \frac{ξ_{i}}{∥ξ_{i}∥} )|^{2}. Then, there necessarily

exits i_{0} ∈ \{1, …, k\} such that |(u | \frac{ξ_{i_{0}}}{∥ξ_{i_{0}}∥} )|^{2} ≥ \frac{1}{k}, which gives i).

Using i), take i_{0} ∈ \{1, …, k\} such that \frac{1}{\sqrt{k}} ∥ξ_{i_{0}}∥ ≤ |(e_{1}|ξ_{i_{0}})|. Now,

|(e_{1}|ξ_{i_{0}})| ≤ |a_{1}| because ξ_{i_{0}} ∈ E. q.e.d.

It is convenient to use the Kuo distance in the following way: let A = (A_{1}, …, A_{k}) ∈ L(𝕂^{n}, 𝕂^{k}) and let η_{i} be the gradient of A_{i}. For i ∈ {1, …, k} denote V_{i} = ⋂_{j≠i} ker A_{j}. Observe that

dist (η_{i}, ⟨(η_{j})_{j≠i}⟩) = ∥A_{i}|_{V_{i}}∥,

where A_{i}|_{V_{i}} is the restriction of A_{i} to V_{i}. Recall that ⟨(η_{j})_{j≠i}⟩ is the vector space generated by the vectors (η_{j})_{j≠i}, hence it is the orthogonal complement to V_{i}. So

κ(η_{1}, …, η_{k}) = min_{1≤i≤k} ∥A_{i}|_{V_{i}}∥ (2.2)

and we denote this number by κ(A).

From Proposition 2.6 we obtain immediately:

Corollary 2.1. Let A = (A_{1}, …, A_{k}) ∈ L(𝕂^{n}, 𝕂^{k}), then

ν(A) ≤ κ(A) ≤ \sqrt{k} ν(A).

2.3 Asymptotic critical values and fibration theorem

Let M, N be C^{p≥2} manifolds and U an open subset of M. Suppose that M is complete and N is connected—although most definitions and results of this section are valid in the case of Finsler manifolds we will always suppose that M and N are Hilbert spaces. Suppose given a C^{p≥2}-map f : U → N, satisfying the condition that:

(2.3) There is no sequence x_{l} in U such that

lim_{l→∞} x_{l} ∈ ∂U and lim_{l→∞} f(x_{l}) ∈ N.

Then we will denote by K̃(f) the following set:

K̃(f) = {y ∈ N : ∃x_{l} ∈ U f(x_{l}) → y and ν(df(x_{l})) → 0}.

The set of critical values of f say

K_{0}(f) = {y ∈ N : ∃x ∈ f^{−1}(y) s.t. ν(df(x)) = 0}

is contained in K̃(f). In fact K̃(f) is the union of K_{0}(f) and K̃_{∞}(f), an asymptotic critical set for f, which is defined by (NCS means no converging subsequences in \overline{U})

K̃_{∞}(f) = {y ∈ N : ∃x_{l} ∈ U with NCS f(x_{l}) → y and ν(df(x_{l})) → 0}.

Clearly K̃(f) is closed, and K̃_{∞}(f) = ∅ if f is a proper map. Note that K̃_{∞}(f) is also closed if f is defined on an open subset U in a finite dimensional space.

In [20] one can find the following generalisation of Ehresmann’s theorem:

Theorem 2.1 ([20]). Let V ⊂ N be a connected component of N − K̃(f), then either f^{−1}(V) = ∅ or f : f^{−1}(V) → V is a locally trivial C^{p}-fibration.

As a particular case this gives the classical Erhesmann’s theorem for proper mappings.

In the case of finite dimensional manifolds, the classical Sard’s theorem says that K_{0}(f) is a subset of N of zero measure. In the infinite dimensional setting, if the spaces M and N are separable, one has a similar result for C^{p}-functions if p ≥ max(k, 2) assuming k is an integer such that dim ker f′′(x) ≤ k for all x; also for C^{r}-Fredholm maps with r > max(index(f), 0) one can show that the set K_{0}(f) has empty interior.

Without some extra hypothesis the set K̃(f), as well as K̃_{∞}(f), may be quite large. Using the fact that K̃(f) is closed it is not difficult to construct a C^{∞} function f : ℝ → ℝ such that K̃(f) = ℝ. This may happen also for polynomials in more than 2 variables as the example below shows.

Example 2.1 ([17]). Consider the polynomial

f(x, y, z) = x + x^{2}y + x^{4}yz.

Then K_{0}(f) = ∅ and K̃_{∞}(f) = ℝ. To see this take any point a ∈ ℝ^{∗} and the curve

γ_{a}(s) = (s, \frac{2a}{s^{2}}, −\frac{(1 + (4a)^{−1}s)}{2s^{2}}) .

We have

f(γ_{a}(s)) = a + s (1 + \frac{1}{4a}) → a

and

∇f(γ_{a}(s)) = (0, s^{2} (\frac{1}{2} − \frac{s}{8a}) , 2as^{2}) → 0

as s → 0. Observe nevertheless that f is a fibration over each connected component of ℝ^{∗}; see Remark 2.2 and Theorem 3.1 below.

Recall that a value y of the map f is called typical if f is a C^{∞} fibration near y and atypical otherwise. The set B(f) of atypical values, called the bifurcation set of f, is contained in K̃(f) but in general is not equal to it—see Example 2.1 and also the following:

Example 2.2 ([22]). The polynomial 2x^{2}y^{3} − 9xy^{2} + 12y has no critical points i.e., K_{0}(f) is empty, and 0 ∈ K̃_{∞}(f). But f is a fibration over ℝ.

Our main goal is to prove that the set of generalized critical values of a semialgebraic mapping f : 𝕂^{n} → 𝕂^{k} is nowhere dense in 𝕂^{k}. So we have to modify the definition of the set K̃_{∞}(f). As noticed in [20, Remark 6.1], Theorem 2.1 remains true if, in the definition of K̃_{∞}(f), we replace the condition ν(df(x_{l})) → 0 by the following one: ω(|x_{l}|)ν(df(x_{l})) → 0, where ω : [0, ∞) → [0, ∞) is a continuous function

with ∫_{0}^{∞} \frac{du}{ω(u)} = ∞. We take ω(t) = 1 + t. Let f : 𝕂^{n} → 𝕂^{k} be C^{1}

mapping, we define the set of generalized critical values of f as

K(f) = {y ∈ 𝕂^{k} : ∃x_{l} ∈ 𝕂^{n}
s.t. f(x_{l}) → y and (1 + |x_{l}|)ν(df(x_{l})) → 0}.

Clearly K(f) = K_{0}(f) ∪ K_{∞}(f), where the set

K_{∞}(f) = {y ∈ 𝕂^{k} : ∃x_{l} ∈ 𝕂^{n}, |x_{l}| → ∞
s.t. f(x_{l}) → y and (1 + |x_{l}|)ν(df(x_{l})) → 0}

will be called the set of asymptotic critical values of f.

Note that for polynomial functions f : ℂ^{n} → ℂ the condition t_{0} ∉ K_{∞}(f) means that t_{0} satisfies the Malgrange condition. Let us recall a characterisation of this condition due to Parusiński.

Theorem 2.2 ([16],[17]). Let f : ℂ^{n} → ℂ be a polynomial with isolated singularities at infinity and t_{0} a regular value (i.e., t_{0} ∉ K_{0}(f)). Then the following are equivalent:

(i) t_{0} ∉ B(f).

(ii) The Malgrange condition for t_{0} is satisfied, i.e., ∃δ > 0 : |x||∇f(x)| ≥ δ for x large and f(x) close to t_{0}.

(iii) χ(f^{−1}(t)) is constant near t_{0}.

In particular, this theorem characterizes the bifurcation set for all polynomials if n = 2, in this case we have K_{∞}(f) = K̃_{∞}(f) by [5].

In the remainder of this section we discuss some examples and related notions.

Remark 2.2. For the polynomial of Example 2.1 we have K̃_{∞}(f) = ℝ but K_{∞}(f) = {0}.

To see this consider

∇f(x, y, z) = (1 + 2xy + 4x^{3}yz, x^{2} + x^{4}z, x^{4}y) =: (A, B, C).

We already know that 0 ∈ K_{∞}(f).

Take any t ≠ 0. It is easy to see that x → 0 if ∇f → 0: Since B and C must be bounded, A cannot go to 0 if x is unbounded; and y and z must be bounded if x goes to a limit not equal to 0. Now, since x(1 + xy + x^{3}yz) → t, if y is bounded then A = 1 + xy + x^{3}yz + xy + x^{3}yz + 2x^{3}yz → ∞ thus y → ∞. Put m = (x, y, z) and suppose |m|∇f(m) → 0. The component |m|C → 0 so |x^{2}y| = ϵ(m) → 0, and since x + x^{2}y + x^{4}yz → t we also have x^{4}yz = t + δ(m) with δ(m) → 0. To finish, consider the component |m|A; we have |m|A =

|m| + \frac{|m|}{|x|}(4|t| + 4δ(m) + ϵ(m)) ≥ (5|t| + 1)|m| and so |m|A cannot go

to 0.

Remark 2.3. For the polynomial of Example 2.2 we have K(f) = {0}.

Let us recall some classes of polynomials “without critical points at infinity”:

Denote by M(f) = {x ∈ ℂ^{n} : ∃λ ∈ ℂ : ∇f(x) = λx}. A polynomial f is M-tame if for any sequence x_{k} ∈ M(f) such that x_{k} → ∞ we have lim_{k→∞} f(x_{k}) = ∞

A polynomial f is called quasi-tame if for any sequence x_{k} ∈ ℂ^{n} such that x_{k} → ∞ and ∇f(x_{k}) → 0 we have lim_{k→∞} |f(x_{k}) − (x_{k}|∇f(x_{k}))| = ∞.

It is well known, and easy to prove, that quasi-tame implies Malgrange condition and that Malgrange condition at any point (which is the same as K(f) = ∅) implies M-tame. Finally we will give an example showing that K(f) is in general different from B(f)

Example 2.3 ([19]). The polynomial x + y − 2x^{2}y^{3} + x^{3}y^{6} + zy^{3} − z^{2}y^{5} + ty^{5} is a fibration over ℂ that is B(f) = ∅. But it is not M-tame at 0, and thus K_{∞}(f) ≠ ∅—in fact K_{∞}(f) = {0}.

3. Main result. Real case

From now on we will restrict attention to the finite dimensional case. Our aim is to prove the following theorem:

Theorem 3.1. Let f : ℝ^{n} → ℝ^{k} be a differentiable semialgebraic map. Then K(f) is a closed semialgebraic set of dimension less than k. Moreover, if f is of class C^{2}, then f : ℝ^{n} \ f^{−1}(K(f)) → ℝ^{k} \ K(f) is a fibration over each connected component of ℝ^{k} \ K(f). In particular B(f) ⊂ K(f).

Here B(f) means the smallest closed subset of ℝ^{k} such that f : ℝ^{n} \ f^{−1}(B(f)) → ℝ^{k} \ B(f) is a fibration over each connected component of ℝ^{k} \ B(f).

Remark 3.1. The theorem remains valid in the case of a mapping defined on an open subset U of ℝ^{n}. That is K(f) is still a closed semialgebraic set of dimension less than k and, if f satisfies Condition (2.3), f : U − f^{−1}(K(f)) → ℝ^{k} − K(f) is a fibration over each connected component of ℝ^{k} − K(f).

The main argument of our proof is based on a fact, due to K. Kurdyka ([10]), that any semialgebraic set A ⊂ ℝ^{n} is a finite union A = ∪_{i}L^{i}, where each L^{i} has the Whitney property with constant M: any two points x, y ∈ L^{i} can be joined in L^{i} by a piecewise smooth arc of length ≤ M|x − y|. (Actually any M > 1 will do, by [9]). It is crucial for us that the constant M = M(n) depends only on n. In [18] A. Parusinski obtained a similar decomposition of semialgebraic sets, but without any estimate on M.

What we need actually is a uniform version of the above decomposition, for families parameterized by finite dimensional spaces : if B ⊂ ℝ^{n} × ℝ^{p} and t ∈ ℝ^{p}, we write B_{t} = {x ∈ ℝ^{n} : (x, t) ∈ B}. Then from the method of [10] (see also [11], [12, Chap. 2]), we obtain the following theorem:

Theorem 3.2. There exists M = M(n) > 0 such that any semialgebraic set A ⊂ ℝ^{n} × ℝ^{p} can be decomposed into a finite (and disjoint) union A = ∪_{i∈I}L^{i}, such that for each t ∈ ℝ^{p}, every set L_{t}^{i} has the Whitney property with constant M. So, in particular A_{t} = ∪_{i∈I}L_{t}^{i} for each t ∈ ℝ^{p}. (Clearly, for a fixed t ∈ ℝ^{p} some of L_{t}^{i} may be empty.)

In fact for our purpose we may use a weaker result due to B. Teissier [21], we explain it after the proof of the main theorem.

The proof of Theorem 3.1 will be given in Section 3.3. The main steps of the proof are as follows:

• We observe that points in K_{∞}(f) are associated to sequences x_{l} satisfying, for some N not depending on y,

(1 + |x_{l}|)^{1+ \frac{1}{N}} ν(df(x_{l})) → 0.

• We consider, in a sphere with large radius r, the set D_{i} of points where the Kuo distance κ(df(x)) is attained by ∇f_{i}(x). We decompose D_{i} ∩ W_{b} relatively to r using Theorem 3.2, where W_{b} is the level surface (f_{1}, …, f̂_{i}, …, f_{k})^{−1}(b).

• We prove that vol_{k}K_{∞}(f_{|D_{i}}) = 0 using that

|∇f_{i}_{|W_{b}}(x)| ≤ |x|^{−(1+ \frac{1}{N})}.

3.1 Lemma on K_{∞}(f)

In order to prove our Sard theorem, we shall use the fact that for a fixed mapping f, the convergence of ν(df(x_{l})) in the definition of K_{∞}(f) is actually faster than |x_{l}|^{−1}. To make this precise, for a differentiable function f : ℝ^{n} → ℝ^{k} and any N ∈ ℕ^{∗} we define

K_{∞}^{N}(f) = {y ∈ ℝ^{k} : ∃x_{l} ∈ ℝ^{n}, |x_{l}| → ∞
s.t. f(x_{l}) → y and |x_{l}|^{1+ \frac{1}{N}} ν(df(x_{l})) → 0\}.

We have

Lemma 3.1. Let f : ℝ^{n} → ℝ^{k} be a differentiable semialgebraic function. Then, there exists N ∈ ℕ^{∗} such that

K_{∞}(f) = K_{∞}^{N}(f).

Proof. Let i_{n} : ℝ^{n} → S^{n} ⊂ ℝ^{n+1} be the inverse of the stereographic projection. Clearly, the graph of i is semialgebraic. Denote {∞} = S^{n} \ i_{n}(ℝ^{n}). In the sequel, we shall consider that ℝ^{n} = i_{n}(ℝ^{n}) ⊂ S^{n}. We compactify also ℝ^{k} = i_{k}(ℝ^{k}) ⊂ S^{k} and ℝ ⊂ S^{1} in the similar way. Let

σ(x) = |x|ν(df(x)) for x ∈ ℝ^{n}.

By Proposition 2.4, σ is a semialgebraic mapping. Finally, consider the mapping φ : ℝ^{n} → ℝ^{k} × ℝ ⊂ S^{k} × S^{1} defined by φ(x) = (f(x), σ(x)).

Recall now the classical lemma (cf. [2, Chap. 9], [3])

Lemma 3.2 (Wing Lemma). Let Ω and B be two semialgebraic subsets of ℝ^{m}. Assume that B = \overline{B} ⊂ \overline{Ω} \ Ω. Then, there exists a semialgebraic set A ⊂ Ω such that

B = \overline{A} ∩ (\overline{Ω} \ Ω).

We apply the Wing Lemma for Ω = graph(φ) ⊂ S^{n} ×S^{k} ×S^{1}, and B the closure of {∞}×K_{∞}(f)×{0} in S^{n}×S^{k}×S^{1}. By Proposition 2.4 and the definition of K_{∞}(f) it follows that B is a closed semialgebraic set. Clearly B ⊂ \overline{Ω} \ Ω. So, there exists a semialgebraic set A ⊂ graph(φ) such that

\overline{A} ∩ (\{∞\} × S^{k} × S^{1}) = B. (3.1)

Let Γ = π(A), where π : ℝ^{n} × ℝ^{k} × ℝ → ℝ^{n} is the projection. Equality (3.1) means that {∞} × y × {0} ∈ B if and only if there exists a sequence x_{l} ∈ Γ with |x_{l}| → ∞, f(x_{l}) → y and σ(x_{l}) → 0. Let us define

θ(r) = r sup_{{x∈Γ \atop |x|=r}} ν(df(x)).

with the convention that θ(r) = 0 if Γ ∩ S(r) = ∅, where S(r) is the sphere of radius r centered at 0. Clearly θ is semialgebraic. Note that for any sequence x_{l} ∈ Γ, |x_{l}| → ∞ we have σ(x_{l}) → 0, since B is compact. This implies that θ(r) → 0 as r → ∞. So, by Puiseux, θ(r) ≤ cr^{−α} at ∞ for some α ∈ ℚ_{+}^{∗} and c > 0. Let N ∈ ℕ^{∗} be such that \frac{1}{N} < α, then of course

r^{\frac{1}{N}} θ(r) → 0 when r → ∞.

So K_{∞}(f) ⊂ K_{∞}^{N}(f), this completes the proof of Lemma 3.1 since obviously the reverse inclusion is satisfied. q.e.d.

3.2 Semialgebraic arcs at infinity

We shall use the following version of the curve selection lemma for semialgebraic sets (it can be easily obtained using a semialgebraic compactification of ℝ^{n} and the classical curve selection lemma, see [2], [3]).

Lemma 3.3 (Curve selection at infinity). Let A ⊂ ℝ^{n} and let ϕ : A → ℝ^{q} be a semialgebraic map. Assume that there exists a sequence x_{l} ∈ A such that |x_{l}| → ∞ and ϕ(x_{l}) → y, for some y ∈ ℝ^{q}. Then there exists a semialgebraic arc γ : (α, β) → ℝ^{n} such that γ(t) ∈ A, lim_{t→β} |γ(t)| = +∞ and lim_{t→β} ϕ(γ(t)) = y.

Now consider such a semialgebraic arc γ : (α, β) → ℝ^{n}. Since |γ′(t)| > 0 for t close to β, we may reparametrize γ in such a way that β = +∞ and |γ(r)| = r. Under this assumption we have:

Lemma 3.4. lim_{r→∞} |γ′(r)| = 1; in particular, γ′(r) is bounded for r > 0 large enough.

Proof. Since γ is semialgebraic, lim_{r→∞} \frac{γ(r)}{|γ(r)|} and lim_{r→∞} \frac{γ′(r)}{|γ′(r)|} exist.

Hence, it is easily seen that these limits are equal. In other words

cos α(r) → 1, as r → ∞, where α(r) is the angle between \frac{γ(r)}{|γ(r)|} = \frac{γ(r)}{|r|}

and \frac{γ′(r)}{|γ′(r)|}. Differentiation of |γ(r)|^{2} = r^{2} yields |γ′(r)| = \frac{1}{cos α(r)}.

This implies the lemma. q.e.d.

3.3 Proof of main Theorem 3.1

Let us fix N ∈ ℕ such that K_{∞}(f) = K_{∞}^{N}(f)—cf. Lemma 3.1. By Proposition 2.6, we may replace the distance ν of Rabier by the distance κ of Kuo. For each i ∈ {1, …, k}, we define

D_{i} = {x ∈ ℝ^{n} : κ(df(x)) = dist(∇f_{i}(x), V_{i}(x))}

where V_{i}(x) is the vector space generated by ∇f_{j}(x), j = 1, …, k and j ≠ i.

Clearly each D_{i} is semialgebraic in ℝ^{n} and ℝ^{n} = ∪_{i=1}^{k}D_{i}, so

K_{∞}(f) = ⋃_{i=1}^{k} K_{∞}(f_{|D_{i}})

where

K_{∞}(f_{|D_{i}}) = {y ∈ ℝ^{k} : ∃x_{l} ∈ D_{i}, |x_{l}| → ∞, f(x_{l}) → y
and |x_{l}|ν(df(x_{l})) → 0}.

We shall prove the following

Lemma 3.5. vol_{k}(K_{∞}(f_{|D_{i}})) = 0 for each i ∈ {1, …, k}. In particular dim K_{∞}(f) < k.

Proof. We will give the proof for i = 1, we write D = D_{1}, \overline{f} = (f_{2}, …, f_{k}).

Let us fix B an open ball in ℝ^{k−1} and (α, β) an open bounded interval in ℝ. The lemma is clearly a consequence of

vol_{k}(K_{∞}(f_{|D}) ∩ (α, β) × B) = 0. (3.2)

In order to prove Equality (3.2), we construct a family of sets Δ_{r} such that

\overline{Δ_{r}} ⊃ K_{∞}(f_{|D}) ∩ (α, β) × B and vol_{k}(\overline{Δ_{r}}) → 0 as r → ∞.

We first define

Σ̃_{r} = \{x ∈ D : |x| ≥ r, f_{1}(x) ∈ (α, β), \overline{f}(x) ∈ B and |x|^{1+ \frac{1}{N}} κ(df(x)) ≤ 1\}

where r > 0, and put

Δ_{r} = f(Σ̃_{r}) and finally Δ = ⋂_{r>0} \overline{Δ}_{r}.

Every Δ_{r} is semialgebraic, hence we have vol_{k}(\overline{Δ}_{r}) = vol_{k}(Δ_{r}) and consequently

vol_{k}(Δ) = lim_{r→∞} vol_{k}(Δ_{r})

since the family (Δ_{r})_{r>0} is decreasing with respect to r → ∞. It is clear that

K_{∞}(f_{|D}) ∩ (α, β) × B ⊂ Δ,

so it is enough to prove that vol_{k}(Δ) = 0. First, using Fubini’s theorem we write

vol_{k}(Δ_{r}) = ∫_{B} m_{r}(b)db

where db stands for the Lebesgue measure on ℝ^{k−1}, and

m_{r}(b) = vol_{1}({y_{1} ∈ ℝ : (y_{1}, b) ∈ Δ_{r}}).

Clearly, each m_{r} is measurable. Moreover, for fixed b ∈ B, the function r ↦ m_{r}(b) ≥ 0 is decreasing. Let

m(b) = lim_{r→∞} m_{r}(b).

By Lebesgue’s theorem on bounded convergence, we obtain

vol_{k}(Δ) = ∫_{B} m(b)db.

Now the final point in the proof of Equality (3.2) is the fact that m ≡ 0, which follows from the next lemma. q.e.d.

Lemma 3.6. There exists a constant c > 0 such that, for r large enough

m_{r}(b) ≤ cr^{− \frac{1}{N}} .

Proof. To prove Lemma 3.6, we introduce the semialgebraic family

Σ_{r,b} = Σ̃_{r} ∩ \overline{f}^{−1}(b) ∩ S(r),

where b ∈ B, r > 0, and next we write

Σ̃_{r,b} = Σ̃_{r} ∩ \overline{f}^{−1}(b) = ⋃_{s≥r} Σ_{s,b}.

Note that

m_{r}(b) = vol_{1}(f_{1}(Σ̃_{r,b})).

It follows from Theorem 3.2 that there exists a finite family L^{i} ⊂ ℝ^{n} × ℝ × ℝ^{k−1}, i ∈ I of semialgebraic sets such that

Σ_{r,b} = ⋃_{i∈I} L_{r,b}^{i}.

Each L_{r,b}^{i} has the Whitney property with constant M (some of L_{r,b}^{i} may be empty).

Recall that the condition |x|^{1+ \frac{1}{N}} κ(df(x)) ≤ 1 for x ∈ \overline{f}^{−1}(b) = W_{b} means that

|∇f_{1}_{|W_{b}}(x)| ≤ |x|^{−(1+ \frac{1}{N})}. (3.3)

Hence, by the mean value theorem f_{1}(L_{r,b}^{i}) is a segment of length d(r) where

d(r) ≤ 2Mr sup_{L_{r,b}^{i}} |∇f_{1}_{|W_{b}}| ≤ 2Mr^{− \frac{1}{N}} . (3.4)

Fix b ∈ B, i ∈ I and assume that L_{r,b}^{i} ≠ ∅ for any r large enough. Applying the curve selection lemma at infinity, we obtain a semialgebraic arc γ : [r, +∞) → ℝ^{n} such that γ(ζ) ∈ L_{ζ,b}^{i}. In particular,

γ(ζ) ∈ \overline{f}^{−1}(b) = W_{b} and |γ(ζ)| = ζ.

By Lemma 3.4, we may suppose that |γ′(ζ)| ≤ 2. So we can easily compute length of f_{1} ∘ γ([r, +∞)); namely, by (3.3) we have

∫_{r}^{+∞} |(f_{1} ∘ γ)′(ζ)|dζ ≤ 2 ∫_{r}^{+∞} ζ^{−(1+ \frac{1}{N})}dζ = 2Nr^{− \frac{1}{N}} . (3.5)

Thus, by (3.4) and (3.5), f_{1}(⋃_{ζ≥r} L_{ζ,b}^{i}) is contained in a segment of length

(4M + 2N)r^{− \frac{1}{N}} .

Therefore f_{1}(Σ̃_{r,b}) is contained in #I segments of this length. Put c = (#I)(4M + 2N); we have

m_{r}(b) ≤ cr^{− \frac{1}{N}}

and Lemma 3.6 follows. q.e.d.

As we already seen the map φ = (f, σ) is semialgebraic, and so K(f) is a semialgebraic subset of ℝ^{k}.

From Lemma 3.5 and the usual semialgebraic Sard’s theorem (see [3]) it follows that dim K(f) < k.

The fact that f is a fibration is a consequence of Theorem 2.1. This ends the proof of Theorem 3.1.

3.4 An alternative proof of Lemma 3.6 via Teissier’s theorem

Let us recall the following result due to B. Teissier [21]. We state only the semialgebraic version which is of our interest

Theorem 3.3 (Teissier). Let B ⊂ ℝ^{n} × ℝ^{p} be a semialgebraic set. Assume that for any t ∈ ℝ^{p} the set B_{t} = {x ∈ ℝ^{n} : (x, t) ∈ B} is contained in the unit closed ball in ℝ^{n}. Then there exists a constant M > 0, depending on B, such for any t ∈ ℝ^{p}: any two points in the same connected component of B_{t} can be joined in B_{t} by a piecewise smooth curve of length at most M.

The original proof of Teissier (for subanalytic sets) used Hironaka’s resolution of singularities, but now there are many simplified proofs by R. Hardt, the first named author, Y. Yomdin [23]. S. K. Donaldson [4] obtained an explicit estimate for M in some special cases.

We obtain the following from Theorem 3.3 as an immediate corollary:

Corollary 3.1. Let A ⊂ ℝ^{n} × ℝ^{p} be a semialgebraic set. Assume that for any t ∈ ℝ^{p} the set A_{t} = {x ∈ ℝ^{n} : (x, t) ∈ A} is contained in the ball {x ∈ ℝ^{n} : |x| ≤ r(t)}, where r : ℝ^{p} → ℝ is some semialgebraic (not necessarily continuous) positive function. Then there exists a constant M > 0, depending on A, such that, for any t ∈ ℝ^{p}, any two points in the same connected component of A_{t} can be joined in A_{t} by a piecewise smooth curve of length at most Mr(t).

Indeed it is enough to apply Teissier’s theorem to the semialgebraic family

B_{t} = \frac{1}{r(t)}A_{t}, t ∈ ℝ^{p}.

Let us come back to the proof of Lemma 3.6. Recall that

Σ_{r,b} = Σ̃_{r} ∩ \overline{f}^{−1}(b) ∩ S(r).

It follows from Hardt’s semialgebraic triviality theorem (see [2, Chap. 9]) that there exists a finite family L^{i} ⊂ ℝ^{n}×ℝ×ℝ^{k−1}, i ∈ I of semialgebraic sets such that

Σ_{r,b} = ⋃_{i∈I} L_{r,b}^{i}.

But this time L_{r,b}^{i} denote a connected component of Σ_{r,b} or an empty set. By Corollary 3.1 any two points in L_{r,b}^{i} can be joined (in L_{r,b}^{i}) by a piecewise smooth curve of length at most Mr. So we can obtain estimate (3.4) by the same argument as before. The rest of the proof of Lemma 3.6 need not to be modified.

Remark 3.2. We have actually proved that, for generic b ∈ ℝ^{k−1}, the set Δ ∩ {x ∈ ℝ^{k} : (x_{2}, …, x_{k}) = b} has at most s points, where s is a bound for the number of connected components of any Σ_{r,b}. This number s can be easily estimated from above (cf. [3]) in terms of the degrees of the polynomials describing the graph of f. In particular, if f_{1}, …, f_{k} are polynomials of degree less than d, then, by [3, Proposition 4.4.5],

s = s(n, d, k) ≤ (2d + 1)(4d − 3)^{n}.

Hence, if f : ℝ^{n} → ℝ is a polynomial of degree d then

#K_{∞}(f) ≤ (2d + 1)(4d − 3)^{n}.

3.5 The set J_{f} of Z. Jelonek

To end this section we will prove that, in some cases, our K_{∞}(f) is equal to the set J_{f} of Z. Jelonek.

Let us consider a continuous mapping f : 𝕂^{n} → 𝕂^{n}. We say (cf. [7]) that f is proper at a point y ∈ 𝕂^{n} if there exists an open neighborhood U of y such that the restriction f|_{f^{−1}(U)} : f^{−1}(U) → U is a proper map. We denote by J_{f} the set of points at which f is not proper, we always have the inclusion K_{∞}(f) ⊂ J_{f}.

Now we prove the following:

Proposition 3.1. Let f : ℝ^{n} → ℝ^{n} be a C^{1} semialgebraic mapping, then J_{f} = K_{∞}(f).

Proof. Observe that y ∈ J_{f} implies that there exists a sequence x_{l} ∈ ℝ^{n} such that |x_{l}| → ∞ and f(x_{l}) → y. Since f is semialgebraic, using a standard curve selection (see Subsection 3.2) we can find a C^{1} semialgebraic arc γ : [a, ∞) → ℝ^{n} such that |γ(r)| = r → ∞ and f(γ(r)) → y. Hence in particular |γ′(r)| ≥ 1. Note that a continuous semialgebraic arc η : [α, β) → ℝ^{n} has a limit (in ℝ^{n}) at β, if and only if η is of finite length.

Assume that y ∈ J_{f} and y ∉ K_{∞}(f). It means that for some b, c > 0 we have

|df(γ(r))(γ′(r))| ≥ \frac{c}{r}

for any r ≥ b. But this implies that length of f(γ) is not less than

c ∫_{b}^{∞} \frac{d r}{r} = ∞, since f is injective on γ for r large enough. So f(γ(r))

has no limit in ℝ^{n} as r → ∞, which is a contradiction. q.e.d.

We shall use the above proposition in a discussion (in the next section) on the dimension of K_{∞}(f) in the complex case.

Recall that we have always K_{∞}(f) ⊂ K̃_{∞}(f) ⊂ J_{f}. Thus Proposition 3.1 yields

Corollary 3.2. Let f : ℝ^{n} → ℝ^{n} be a C^{1} semialgebraic mapping, then K_{∞}(f) = K̃_{∞}(f).

This is in strong contrast with a case, discussed in Section 2, where the dimension of the target was smaller than the dimension of the source. Corollary 3.2 seems to be a new result also in the complex case (even for n = 2) and may be of some interest for Jacobian Conjecture.

Proposition 3.1 holds actually in a more general case. It is enough to assume that the generic fibers of f : ℝ^{n} → ℝ^{k} are compact. More precisely we have

Theorem 3.4. Let f : ℝ^{n} → ℝ^{k}, k ≤ n be a C^{1} semialgebraic mapping. Assume that the set of regular points of f is dense and that f^{−1}(y) is compact for any y ∈ ℝ^{k} \K_{0}(f). Then K_{∞}(f) = K̃_{∞}(f) = J_{f}.

Proof. Clearly, by previous remarks, it is enough to prove that J_{f} ⊂ K_{∞}(f). Assume this is not the case, it means that there exists y ∉ K_{∞}(f) and a sequence x_{l} ∈ ℝ^{n} such that |x_{l}| → ∞ and f(x_{l}) → y. By the curve selection lemma we obtain a C^{1} semialgebraic arc γ : [a, ∞) → ℝ^{n} such that |γ(r)| → ∞ and f(γ(r)) → y. For a large enough the set Γ = f(γ(a, ∞)) is a smooth curve of finite length. Moreover, since y ∉ K_{∞}(f) there exists a constant c > 0 such that

(1 + |x|)ν(df(x)) ≥ c (3.6)

for any x such that f(x) ∈ Γ and |x| ≥ R, where R > 0 is large enough. Since the set of regular values of f is dense we can suppose that Γ ∩ K_{0}(f) = ∅. Recall that f^{−1}(f(γ(a))) is compact, so we may assume that it is contained in B(0, R).

We are going to prove that f^{−1}(Γ) is bounded which will be a contradiction.

Denote by g the map f_{|f^{−1}(Γ)}, and by δ(g(x)) the unit tangent vector field to Γ in the direction of y. Using the right inverse s(x) of dg(x) (see Section 4 of [20]), we can lift δ to a C^{0} vector field X(x) = s(x)δ(g(x))

on f^{−1}(Γ). In fact X = \frac{∇g}{|∇g|^{2}} , where ∇g is the gradient of g with

respect to the induced metric on f^{−1}(Γ).

Since ν(dg(x)) = ν(df(x)_{|f^{−1}(Γ)}) ≥ ν(df(x)) (use Proposition 2.3), Condition (3.6) with f replaced by g is still true outside of B(0, R).

Take a point z ∈ (ℝ^{n} \ B(0, R)) ∩ f^{−1}(Γ), by Peano’s theorem there exists a local integral curve of the field −X starting at z; denote it by ϕ(t) with t ∈ [α, β). Using Gronwall lemma and the bound

|X(x)| ≤ \frac{2(1 + |x|)}{c}

which comes from (3.6), we easily obtain that

|ϕ(t)| ≤ |z| exp(β − α) + exp(\frac{2}{c}(β − α)) − 1

for t ∈ [α, β). This inequality implies that ϕ is Lipschitz on [α, β) and so ϕ has a limit at β and can be extended.

Observe that t is the arc length on Γ, since dg(ϕ(t))ϕ′(t) = δ_{g(ϕ(t))}. This implies that ϕ can be extended in such a way that in finite time it goes into B(0, R), since g^{−1}(g(γ(a))) ⊂ B(0, R). But, reversing time, we see (again by Gronwall lemma) that ϕ(t) has to stay in the ball of

radius R exp(length(Γ)) + exp (\frac{2}{c} length(Γ)) − 1.

Thus f^{−1}(f(γ(b))) is uniformly bounded irrespective of b. Hence the set f^{−1}(Γ) is bounded, which is a contradiction. q.e.d.

Remark 3.3. P. J. Rabier pointed out to us that the same result is valid for general C^{2} mappings from ℝ^{n} to ℝ^{k}, if f^{−1}(y) is compact for y in a dense subset of ℝ^{k}: By standard transversality arguments one can construct a perturbation with compact support of f for which y becomes a regular value. This does not change K_{∞}(f) and J_{f} near y, thus we can conclude using Rabier’s fibration theorem. Combining this with our proof we obtain that, in fact, Theorem 3.4 is valid for any C^{1} mappings with generically compact fibers.

Remark 3.4. In a recent paper [8] Z. Jelonek proved that if a polynomial mapping satysfies the assumptions of Theorem 3.4, then the corresponding semialgebraic set K_{∞}(f) = J_{f} is ℝ-uniruled—it means that by every point pass a curve (in J_{f}) with polynomial parametrization.

4. The complex case

Suppose now that f : ℂ^{n} → ℂ^{k} is a complex polynomial mapping. We know, by Theorem 3.1, that K_{∞}(f) is a nowhere dense semialgebraic subset of ℂ^{k}. But in this case, we prove more:

Theorem 4.1. If f : ℂ^{n} → ℂ^{k} is a complex polynomial mapping, then K_{∞}(f) is a complex algebraic set and dim_{ℂ} K_{∞}(f) < k.

Proof. We shall prove that K_{∞}(f) is a complex algebraic set. We can assume that f is dominant, that is the generic rank of f is k, as otherwise K_{∞}(f) is equal to the closure of the image of f. Since the closure is the same in the strong and in Zariski topology, we deduce that K_{∞}(f) is a complex algebraic set.

As before we replace Rabier’s ν invariant by the invariant κ of Kuo. Let f = (f_{1}, …, f_{k}). For any j = 1, …, k we denote

f̃_{j} = (f_{1}, …, f_{j−1}, f_{j+1}, …, f_{k}) : ℂ^{n} → ℂ^{k−1}.

Let V_{j}(z) be the kernel of df̃_{j}(z) and let w_{j}(z) denote the restriction of df_{j}(z) to the linear subspace V_{j}(z). Recall that by the definition (2.2) of κ

κ(df(z)) = min_{1≤j≤k} ∥w_{j}(z)∥. (4.1)

Let z = (z_{1}, …, z_{n}) ∈ ℂ^{n}. For each s = 1, …, n we will use the following change of coordinates:

τ_{s}(z) = (\frac{z_{1}}{z_{s}} , …, \frac{z_{s−1}}{z_{s}} , \frac{1}{z_{s}} , \frac{z_{s+1}}{z_{s}} , …, \frac{z_{n}}{z_{s}} ) ,

which sends (z_{s} = 0) to ∞. Denote by 𝔾 = 𝔾_{n−k+1}(ℂ^{n} × ℂ) the Grassmanian of (n − k + 1)-planes in ℂ^{n} × ℂ. For fixed j and s we consider rational mappings

A_{s}^{j} : ℂ^{n} \ {z_{s} = 0} → ℂ^{k} × 𝔾

given by the formula

A_{s}^{j}(z) = (f(τ_{s}(z)), W_{s}^{j}(z)).

Here W_{s}^{j}(z) stands for the graph of \frac{1}{z_{s}} w_{j}(z). To be more precise A_{s}^{j}(z)

is well defined for those z for which V_{j}(z) is of dimension (n − k + 1). Recall that we assumed that f is dominant, hence this condition holds outside a nowhere dense algebraic set. Let Λ = 𝔾_{n−k+1}(ℂ^{n} × 0) ⊂ 𝔾. Note that each λ ∈ Λ can be seen as a graph of constant (equal to 0) mapping on the (n − k + 1)-plane λ. Put

B_{s}^{j} = \overline{graph(A_{s}^{j})} ∩ \{z ∈ ℂ^{n} : z_{s} = 0\} × ℂ^{k} × Λ.

The closure is taken in the strong topology, but it is equal to the Zariski closure. Hence B_{s}^{j} is an algebraic set. Let π : ℂ^{n} × ℂ^{k} × 𝔾 → ℂ^{k} denote the projection. Finally let

K_{s}^{j} = π(B_{s}^{j}).

Clearly each K_{s}^{j} is a constructible set. Assume that we have proved:

Lemma 4.1. K_{∞}(f) = ⋃_{s=1,j=1}^{n,k} K_{s}^{j}.

This, of course, implies that K_{∞}(f) is constructible. On the other hand, the set K_{∞}(f) is closed, so it must be algebraic.

So we are left with proving Lemma 4.1. Suppose that y ∈ K_{s}^{j}, then we can find a real Puiseux arc η = (η_{1}, …, η_{n}) : (0, ε) → ℂ^{n}, |η_{s}(t)| = t such that

f ∘ τ_{s} ∘ η(t) → y, when t → 0

and

W_{s}^{j}(τ_{s}(η(t)) → W_{s}^{j}, when t → 0

for some W_{s}^{j} ∈ Λ. By (4.1) this means that y ∈ K_{∞}(f).

The reverse inclusion is easily obtained. Take y ∈ K_{∞}(f), then there is an arc going to infinity, say z(t) = (z_{1}(t), …, z_{n}(t)), such that f(z(t)) → y and

|z(t)|κ(df(z(t))) → 0. (4.2)

Choose a coordinate z_{s} in such a way that z_{s}(t) goes to infinity faster than the other coordinates, then choose j such that κ(df(z(t)) = ∥w_{j}(z(t))∥ and proceed using A_{s}^{j}. Since 𝔾 is compact there exists a limit W_{s}^{j} of graphs of z_{s}(t)w_{j}(z(t)). It follows, by (4.2), that W_{s}^{j} ∈ Λ. So (0, y, W_{s}^{j}) ∈ B_{s}^{j} and consequently y ∈ K_{s}^{j}. q.e.d.

Recall that, by a result of Z. Jelonek [7], if f : ℂ^{n} → ℂ^{n} is a polynomial dominant mapping with nonempty J_{f}, then J_{f} is a hypersurface (actually this is a ℂ-uniruled hypersurface, its degree is effectively bounded). So, by our Proposition 3.1, K_{∞}(f) is also a ℂ-uniruled hypersurface in this case. One could conjecture that, for a general polynomial dominant mapping f : ℂ^{n} → ℂ^{k} with k < n, K_{∞}(f) must be a hypersurface or an empty set. The following example (suggested by Z. Jelonek) shows that this not the case.

Example 4.1. Consider the polynomial mapping f : ℂ^{3} → ℂ^{2}, given by f(x, y, z) = (xy, xz). Using Proposition 2.5 it is easily seen that ν(df(ξ)) = |x| for ξ = (x, y, z). Let ξ_{n} = (x_{n}, y_{n}, z_{n}) be a sequence which tends to infinity and assume that

|ξ_{n}||x_{n}| = |ξ_{n}|ν(df(ξ_{n})) → 0.

Hence |x_{n}| → 0 and therefore f(ξ_{n}) → 0. So 0 is the only asymptotic critical value of f.

However we may still conjecture that K_{∞}(f) is ℂ-uniruled in the sense that by each point of K_{∞}(f) passes a curve wich has a polynomial (possibly constant) parametrization.

Remark 4.1. After the first version of the paper was written (november 1999) we learned that T. Gaffney [6] studied the set K_{∞}(f) in the complex setting. In particular he proved (without using Rabier’s result) that the mapping is a locally trivial fibration over the complement of K_{∞}(f). However his definition of ν is not exactly the same as ours. We leave to the reader to prove that in fact both definitions are equivalent.

References

[1] D. d’Acunto, Valeurs critiques asymptotiques de fonctions définissables dans une structure o-minimale, Ann. Polon. Math. (to appear).

[2] J. Bochnak, M. Coste & M.-F. Roy, Real algebraic geometry, E.M.G., Springer, Vol. 36, 1998.

[3] R. Benedetti & J.-J. Risler, Real algebraic and semialgebraic sets, Hermann, 1990.

[4] S. K. Donaldson, Symplectic submanifolds and almost-complex geometry, J. Differential Geom. 44 (1996) 666–705.

[5] H.V. Hà, Nombres de Łojasiewicz et singularités à l’infini des polynômes de deux variables complexes, C. R. Acad. Sci. Paris, Série I Math. 311 (1990) 429–432.

[6] T. Gaffney, Fibers of polynomial mappings at infinity and a generalized Malgrange Condition, Compositio Math. 119 (1999) 157–167.

[7] Z. Jelonek, Testing sets for polynomial mapping, Math. Ann. 315 (1999) 1–35.

[8] , Geometry of real polynomial mappings, Math. Z. (to appear).

[9] K. Kurdyka & P. Orro, Distance géodésique sur un sous-analytique, Revista mathemática de la Universidad Computense de Madrid, Vol. 10, 1997, num. supl.

[10] K. Kurdyka, On a subanalytic stratification satisfying a Whitney property with exponent 1, Real algebraic geometry, Proc., Rennes 1991, L.N.M. 1524 (1992) 316–322.

[11] , Linear bounds for resolutions of 1-boundaries in subanalytic sets, Real analytic and algebraic geometry, Proc., Trento 1992, Walter de Gruyter, 1995, 189–196.

[12] K. Kurdyka & T. Mostowski, On the gradient conjecture of R. Thom, Preprint, 1998.

[13] K. Kurdyka, T. Mostowski & A. Parusiński, Proof of the gradient conjecture of R. Thom, Ann. of Math. 152 (2000) 763–792.

[14] T. C. Kuo, Characterizations of V -sufficiency of jets, Topology 11 (1972) 115–131.

[15] R. S. Palais, Lusternik-Schnirelman theory on Banach manifolds, Topology 5 (1966) 115–132.

[16] A. Parusiński, On the bifurcation set of complex polynomial with isolated singularities at infinity, Compositio Math. 97 (1995) 369–384.

[17] , A note on singularities at infinity of complex polynomials, Banach center publication, Vol. 39, 1997, 131–141.

[18] , Lipschitz properties of semianalytic sets, Ann. Inst. Fourier 38 (1988) 189–213.

[19] L. Paunescu & A. Zaharia, Remarks on the Milnor fibration at infinity, Preprint, 1999.

[20] P. J. Rabier, Ehresmann’s fibrations and Palais-Smale conditions for morphisms of Finsler manifolds, Ann. of Math. 146 (1997) 647–691.

[21] B. Teissier, Sur trois questions de finitude en géométrie analytique réelle, appendix to F. Tréves, on the local solvability and the local integrability of systems of vector fields, Acta Math. 151 (1983) 38–48.

[22] M. Tibăr & A. Zaharia, Asymptotic behaviour of families of curves, Manuscripta Math. 99 (1999) 383–393.

[23] Y. Yomdin, The geometry of critical and near-critical values of differentiable mappings, Math. Ann. 4 (1983) 495–515.

Université de Savoie, Campus scientifique, France Uniwersytet Jagielloński, Poland