Limit transition between hypergeometric functions of type BC and type A

<jats:title>Abstract</jats:title><jats:p>Let <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline1" /><jats:tex-math>${F}_{BC} (\lambda , k; t)$</jats:tex-math></jats:alternatives></jats:inline-formula> be the Heckman–Opdam hypergeometric function of type BC with multiplicities <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline2" /><jats:tex-math>$k= ({k}_{1} , {k}_{2} , {k}_{3} )$</jats:tex-math></jats:alternatives></jats:inline-formula> and weighted half-sum <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline3" /><jats:tex-math>$\rho (k)$</jats:tex-math></jats:alternatives></jats:inline-formula> of positive roots. We prove that <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline4" /><jats:tex-math>${F}_{BC} (\lambda + \rho (k), k; t)$</jats:tex-math></jats:alternatives></jats:inline-formula> converges as <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline5" /><jats:tex-math>${k}_{1} + {k}_{2} \rightarrow \infty $</jats:tex-math></jats:alternatives></jats:inline-formula> and <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline6" /><jats:tex-math>${k}_{1} / {k}_{2} \rightarrow \infty $</jats:tex-math></jats:alternatives></jats:inline-formula> to a function of type A for <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline7" /><jats:tex-math>$t\in { \mathbb{R} }^{n} $</jats:tex-math></jats:alternatives></jats:inline-formula> and <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline8" /><jats:tex-math>$\lambda \in { \mathbb{C} }^{n} $</jats:tex-math></jats:alternatives></jats:inline-formula>. This limit is obtained from a corresponding result for Jacobi polynomials of type BC, which is proven for a slightly more general limit behavior of the multiplicities, using an explicit representation of Jacobi polynomials in terms of Jack polynomials. Our limits include limit transitions for the spherical functions of non-compact Grassmann manifolds over one of the fields <jats:inline-formula><jats:alternatives><jats:inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" mime-subtype="gif" xlink:type="simple" xlink:href="S0010437X13007045_inline9" /><jats:tex-math>$ \mathbb{F} = \mathbb{R} , \mathbb{C} , \mathbb{H} $</jats:tex-math></jats:alternatives></jats:inline-formula> when the rank is fixed and the dimension tends to infinity. The limit functions turn out to be exactly the spherical functions of the corresponding infinite-dimensional Grassmann manifold in the sense of Olshanski.</jats:p>