2.1 Representation theory of p-adic groups

Let F be a nonarchimedean local field of characteristic $0$ and residual characteristic $p>0$ , with ring of integers $\mathcal {O}$ and maximal ideal $\mathfrak {p}$ . Let q be the order of the residue field. We normalize the absolute value on F so that its value is $q^{-1}$ on any generator of $\mathfrak {p}$ . We fix a nontrivial additive character $\psi :F\rightarrow \mathbb {C}^{\times }$ .

Let G be the group of F-points of a connected reductive group. Let $\mathcal {M}(G)$ be the category of smooth G-representations and let $\mathrm {Irr}(G)$ be the set of isomorphism classes of irreducible objects. Given $\pi \in \mathcal {M}(G)$ we write $\widetilde {\pi }$ for the smooth contragradient representation of $\pi $ .

If $H\subset G$ is a closed subgroup and $(\sigma ,W)$ is a smooth representation of H. We write $\mathrm {Ind}_{H}^{G}(\sigma )$ for the space of right G-smooth functions $f:G\rightarrow W$ such that for any $g\in G$ and $h\in H$ we have $f(hg)=\sigma (h)f(g)$ . This is a G-representation with the action $(g\cdot f)(g^{\prime })=f(g^{\prime }g)$ . We write $\mathrm {ind}_{H}^{G}(\sigma )\subset \mathrm {Ind}_{H}^{G}(\sigma )$ for the G-submodule of functions with compact support mod H.

Now suppose $P=MN\subset G$ is a parabolic subgroup with a Levi decomposition, and $(\sigma ,W)$ is a smooth representation of M inflated to P. We fix $dn$ a Haar measure on N and let $\delta _{P}$ be the modular character of P defined by $d(pnp^{-1})=\delta _{P}(p)dn$ . We write $i_{M}^{G}(\sigma )=\mathrm {Ind}_{P}^{G}(\delta _{P}^{1/2}\otimes \sigma )$ for normalized parabolic induction.

Let $(\pi ,V)$ be a smooth G-representation. If $H\subset G$ is a subgroup with a character $\chi :H\rightarrow \mathbb {C}$ , let $V_{(H,\chi )}$ denote the space of $(H,\chi )$ -coinvariants. This space can be realized as the quotient of V by the subspace $\mathrm {span}\{h\cdot v-\chi (h)v|v\in V,\,h\in H\}$ and is a representation of the subgroup of the normalizer of H that fixes $\chi $ , which we write as $\mathrm {Stab}_{G}(\chi )$ . If $\chi $ is trivial we write $V_{H}=V_{(H,\chi )}$ . We write $r_{P}(V)=\delta _{P}^{-1/2}\otimes V_{N}$ for the normalized Jacquet module of a parabolic subgroup $P=MN\subset G$ .

2.2 Composition algebras

The theta-lift examined in this paper is based on exceptional dual pairs that can be constructed using composition and Jordan algebras. We begin by collecting some information on these algebras.

Let C be a composition algebra over F with quadratic norm form $n_{C}$ . We write $B_{C}(x,y)= n_{C}(x+y)-n_{C}(x)-n_{C}(y)$ for the bilinear form associated to $n_{C}$ and $\overline {\phantom {x}}:C\rightarrow C$ as $x\mapsto \overline {x}$ for conjugation ([Reference Springer and Veldkamp28], Section 1.3). The trace of an element of $x\in C$ is $\mathrm {Tr}_{C}(x)=x+\overline {x}$ and $n_{C}(x)=x\overline {x}$ . Note that $\mathrm {Tr}_{C}(xy)=-B_{C}(x,y)$ , for all $x,y\in C$ .

We write $C^{0}$ for the subspace of trace $0$ elements of C. The group of F-algebra automorphisms $\mathrm {Aut}(C)$ preserves the norm and acts on $C^{0}$ . Thus $\mathrm {Aut}(C)$ is contained in $O(C^{0},n_{C})$ the orthogonal group of the norm form.

Recall that $\mathrm {dim}_{F}(C)=1,2,4,8$ . Over the p-adic field F the possible composition algebras can be described explicitly. When $\mathrm {dim}_{F}(C)=2$ , then C is either a quadratic field extension of F, or C is isomorphic to the split quadratic algebra $F\oplus F$ with norm form $(x,y)\mapsto xy$ . In either case, the automorphism group of $\mathrm {Aut}(C)$ is generated by the conjugate map $x\mapsto \overline {x}$ and so is isomorphic to $\mu _{2}=\{\pm 1\}$ .

When $\mathrm {dim}(C)=4$ , C is either isomorphic to the split quaternion algebra, which can be realized as the algebra of $2\times 2$ matrices $M(2,F)$ with norm form given by the determinant; or C is isomorphic to D, the unique (up to isomorphism) quaternion division algebra over F. The group of $\mathrm {Aut}(C)$ consists of inner automorphisms (Skolem–Noether Theorem) and so is isomorphic to $PC^{\times }$ . When $C\cong M(2,F)$ , then $\mathrm {Aut}(C)\cong \mathrm {PGL}_{2}(F)$ .

When $\mathrm {dim}_{F}(C)=8$ , then C is isomorphic to $\mathbb {O}$ the split octonion algebra over F. Its automorphism group is the F-points of an algebraic group of type $G_{2}$ .

For more information on composition algebras the reader can refer to [Reference Jacobson12, Reference Springer and Veldkamp28].

2.3 Jordan algebras

Next we describe a family of Jordan algebras indexed by a composition algebra C. For more details, see Pollack [Reference Pollack22, Chapter 2, Section 2].

Let

$$ \begin{align*} \mathcal{J}=\mathcal{J}_{C}=\Big\{X=\left(\begin{smallmatrix} c_{1}& x_{3} & \overline{x_{2}}\\ \overline{x_{3}} & c_{2} & x_{1}\\ x_{2} & \overline{x}_{1} & c_{3} \end{smallmatrix}\right)|c_{j}\in F,\, x_{j}\in C\Big\}. \end{align*} $$

Given $A,B\in \mathcal {J}_{C}$ the Jordan multiplication is defined by

$$ \begin{align*} A* B=\frac{AB+BA}{2}, \end{align*} $$

where $AB$ , $BA$ denotes usual matrix multiplication. The algebra $\mathcal {J}_{C}$ is equipped with a cubic norm form

$$ \begin{align*} N_{\mathcal{J}}(X)=c_{1}c_{2}c_{3}-c_{1}n_{C}(x_{1})-c_{2}n_{C}(x_{2})-c_{3}n_{C}(x_{3})+\mathrm{Tr}(x_{1}x_{2}x_{3}). \end{align*} $$

The norm form uniquely defines a symmetric trilinear form $(-,-,-)_{\mathcal {J}}:\mathcal {J}\times \mathcal {J}\times \mathcal {J}\rightarrow F$ normalized so that $(X,X,X)_{\mathcal {J}}=6N_{\mathcal {J}}(X)$ .

We write $\mathrm {Tr}_{\mathcal {J}}:\mathcal {J}\rightarrow F$ for the map defined by $\mathrm {Tr}_{\mathcal {J}}(X)=c_{1}+c_{2}+c_{3}$ . From this we define a nondegenerate pairing $\langle -,-\rangle _{\mathcal {J}}:\mathcal {J}\times \mathcal {J}\rightarrow F$ by $\langle X,X^{\prime }\rangle _{\mathcal {J}}=\mathrm {Tr}_{\mathcal {J}}(X*X^{\prime })$ .

There is also a map $\phantom {X}^{\#}:\mathcal {J}\rightarrow \mathcal {J}$ defined by

$$ \begin{align*} X^{\#}= \left(\begin{smallmatrix} c_{2}c_{3}-n_{C}(x_{1})& \overline{x}_{2}\overline{x}_{1}-c_{3}x_{3} & x_{3}x_{1}-c_{2}\overline{x_{2}}\\ x_{1}x_{2}-c_{3}\overline{x}_{3} & c_{1}c_{3}-n_{C}(x_{2}) & \overline{x}_{3}\overline{x}_{2}-c_{1}x_{1}\\ \overline{x}_{1}\overline{x}_{3}-c_{2}x_{2} & x_{2}x_{3}-c_{1}\overline{x_{1}} & c_{1}c_{2}-n_{C}(x_{3}) \end{smallmatrix}\right). \end{align*} $$

This map can be used to define the cross product

(2.1) $$ \begin{align} X\times Y=(X+Y)^{\#}-X^{\#}-(Y)^{\#}. \end{align} $$

Alternatively $X\times Y \in \mathcal {J}$ is the unique element such that for all $Y\in \mathcal {J}$

(2.2) $$ \begin{align} \langle X\times Y, Z \rangle_{\mathcal{J}}= (X, Y ,Z)_{\mathcal{J}}. \end{align} $$

There is a notion of rank for elements in $\mathcal {J}$ . Every element $X\in \mathcal {J}$ has rank at most $3$ . If $N(X)=0$ , then X has rank at most $2$ . If $X^{\#}=0$ , then x has rank at most 1. If $X=0$ , then X has rank $0$ . (Pollack [Reference Pollack22, Chapter 3, Section 3])

We write $H_{C}$ for the group of invertible linear transformations of $\mathcal {J}=\mathcal {J}_{C}$ that scale the norm form $N_{\mathcal {J}}$ (i.e., the group of similitudes of the cubic form), and $H_{C}^{1}$ for the subgroup preserving the norm form. We have a subgroup ${\mathrm {Aut}}(C) \times {\mathrm {GL}}_3(F) \rightarrow H_{C}$ where $g\in {\mathrm {Aut}}(C)$ acts naturally on entries of elements of $\mathcal J$ , while $h\in {\mathrm {GL}}_3(F)$ acts on $X\in \mathcal J$ by

$$\begin{align*}\det(h) \cdot (h^{-1})^{\top} X h^{-1}, \end{align*}$$

where $h^{\top }$ denotes the transpose of h. The similitude character of this transformation of $\mathcal J$ is $\det (h)$ . If $C=F$ , then $H_F \cong {\mathrm {GL}}_3(F)$ . In general, ${\mathrm {Aut}} (C) \times {\mathrm {GL}}_3(F)$ preserves the decomposition

$$\begin{align*}\mathcal J_{C} =\mathcal J_F \oplus \mathcal J_{C^0} \end{align*}$$

where $\mathcal J_{C^0}$ is the subspace consisting of

$$ \begin{align*} J(x)=\left(\begin{smallmatrix} 0& x_{3} & \overline{x_{2}}\\ \overline{x_{3}} & 0 & x_{1}\\ x_{2} & \overline{x}_{1} & 0 \end{smallmatrix}\right), \end{align*} $$

where $x=(x_{1},x_{2},x_{3})\in (C^{0})^3$ . The following proposition in essence restates the known fact that the dual of the standard three-dimensional representation of ${\mathrm {GL}}_3$ is isomorphic to the exterior square of the standard representation twisted by determinant inverse. In any case it is easy to check.

2.4 Construction of exceptional Lie algebras

Let $\mathfrak {h}_{C}$ be the Lie algebra of $H_{C}^{1}$ . We define vector spaces

$$ \begin{align*} \mathfrak{g}_{0,C}&=\mathfrak{sl}(3,F)\oplus \mathfrak{h}_{C},\\\mathfrak{g}_{1,C}&=V_{3}\otimes \mathcal{J}_{C},\\\mathfrak{g}_{-1,C}&=V_{3}^{*}\otimes \mathcal{J}_{C}^{*}, \end{align*} $$

where $V_3$ is the standard representation of $\mathfrak {sl}_3$ and $V_3^*$ the dual of $V_3$ . We identify $\mathcal {J}_{C}^{*}$ with $\mathcal {J}_{C}$ using the trace form. Consider the vector space

$$ \begin{align*} \mathfrak{g}_{C}=\mathfrak{g}_{0,C}\oplus\mathfrak{g}_{1,C}\oplus\mathfrak{g}_{-1,C}. \end{align*} $$

The space $\mathfrak {g}_{C}$ can be given the structure of a Lie algebra that extends the Lie algebra structure on $\mathfrak {g}_{0,C}$ and the natural action of $\mathfrak {g}_{0,C}$ on $\mathfrak {g}_{1,C}\oplus \mathfrak {g}_{-1,C}$ . (See [Reference Rumelhart24], Section 1.3.) We write $\langle -,-\rangle _{C}$ for the Killing form of $\mathfrak {g}_{C}$ . The Lie algebra $\mathfrak {g}_{F}$ is the split simple Lie algebra of type $F_{4}$ . The Lie algebra $\mathfrak {g}_{C}$ is a simple Lie algebra of type $E_{n}$ , where $n=6,7,8$ when $\mathrm {dim}_{F}(C)=2,4,8$ , respectively.

Let $Y\in \mathfrak {sl}(3,F)$ and $X\in \mathcal {J}_{C}$ , then $X\mapsto YX+XY^{\top }$ defines a Lie algebra action of $\mathfrak {sl}(3,F)$ on $\mathcal {J}_{C}$ extending the analogous action of $\mathfrak {sl}(3,F)$ on $\mathcal {J}_{F}$ . This action induces an inclusion of Lie algebras $\mathfrak {h}_{F}\cong \mathfrak {sl}(3,F)\hookrightarrow \mathfrak {h}_{C}$ , which induces an inclusion of Lie algebras $\mathfrak {g}_{F}\hookrightarrow \mathfrak {g}_{C}$ .

Next we describe a Heisenberg parabolic subalgebra in $\mathfrak {g}_{C}$ . Let $\mathfrak {t}$ be the subalgebra of diagonal matrices in $\mathfrak {sl}(3,F)\subset \mathfrak {g}_{0,C}$ . The adjoint action of $\mathfrak {t}$ on $\mathfrak {g}_{C}$ provides a decomposition

$$ \begin{align*} \mathfrak{g}_{C}=\bigoplus_{\gamma\in \mathfrak{t}^{*}}\mathfrak{g}_{\gamma}, \end{align*} $$

where $\mathfrak {g}_{\gamma }=\{X\in \mathfrak {g}_{C}|[h,X]=\gamma (h)X\text { for all }h\in \mathfrak {t}\}$ . The weights $\gamma \neq 0$ such that $\mathfrak {g}_{\gamma }\neq 0$ form a relative root system $\underline {\Phi }$ of type $G_{2}$ ([Reference Gan and Savin7, Sections 9.2, 10.8]). Note that the long relative root spaces are all isomorphic to F and sit in $\mathfrak {sl}(3,F)$ while the short relative root spaces are isomorphic to $\mathcal {J}_{C}$ or $\mathcal {J}_{C}^{*}$ . Finally, observe that $\mathfrak g_0= \mathfrak {t} \oplus \mathfrak {h}_C$ .

We let $\{\alpha ,\beta \}$ be a set of simple roots in the $G_{2}$ relative root system so that $\alpha $ is long, $\beta $ is short. Then the maximal root $\alpha _{\max }=2\alpha +3\beta $ is a long root. Thus, without loss of generality, we can assume that $h_{\alpha _{\max }}=\mathrm {diag}(1,0,-1)\in \mathfrak {sl}(3,F)$ .

The element $h_{\alpha _{\max }}$ defines a $\mathbb {Z}$ -grading on $\mathfrak {g}_{C}$ supported on $\{0,\pm 1,\pm 2\}$ . For $j\in \mathbb {Z}$ , let

$$ \begin{align*} \mathfrak{g}_{C}(j)=\{x\in \mathfrak{g}_{C}~| ~[h_{\alpha_{\max}},x]=jx\}. \end{align*} $$

Let $\mathfrak {p}=\oplus _{j\geq 0}\mathfrak {g}_{C}(j)$ . Then $\mathfrak {p}$ is a Heisenberg parabolic subalgebra with Levi subalgebra

$$\begin{align*}\mathfrak{m}=\mathfrak{g}_{C}(0)= \mathfrak{t}\oplus\mathfrak{h}_{C}\oplus\mathfrak{g}_{\beta}\oplus\mathfrak{g}_{-\beta} \end{align*}$$

and nilpotent radical $\mathfrak {n}=\oplus _{j>0}\mathfrak {g}_{C}(j)$ with one-dimensional center

$$\begin{align*}\mathfrak{z}=\mathfrak{g}_{C}(2)= \mathfrak{g}_{\alpha_{\max}}. \end{align*}$$

Let $\mathcal {G}_{C}=\mathrm {Aut}(\frak {g}_{C})$ . If $C\neq F$ then the connected component of $\mathcal {G}_{C}$ is an adjoint group of type $E_{n}$ . The group $\mathcal G_F$ is $F_4$ . We omit the subscript C when no confusion can arise. Then the maximal parabolic subalgebra $\mathfrak p$ corresponds to a maximal parabolic subgroup $\mathcal P=\mathcal M \mathcal N$ in $\mathcal G$ . Let $\mathcal Z$ be the center of $\mathcal N$ . Then $\mathcal M$ acts on $\mathcal N/\mathcal Z \cong \mathfrak {n}/\mathfrak {z}$ . The space $\mathfrak {n}/\mathfrak {z}$ admits a symplectic and a quartic form, and $\mathcal M$ acts as a group of similitudes of these two forms.

2.5 A symplectic space

Using Pollack [Reference Pollack22, Chapter 3] we give an explicit construction of the reductive group $\mathcal M$ and its representation on the symplectic space $\mathfrak {n}/\mathfrak {z}$ . In terms of the restricted root system, we have

$$\begin{align*}\mathfrak{n}/\mathfrak{z}\cong \mathfrak g_{\alpha} \oplus \mathfrak g_{\alpha+\beta} \oplus \mathfrak g_{\alpha+2\beta} \oplus \mathfrak g_{\alpha+3\beta}. \end{align*}$$

We identify $\mathcal J$ and $\mathcal J^*$ using the trace form, so $\mathfrak g_{\gamma }\cong \mathcal J$ for any short root $\gamma $ . Thus we can identify $\mathfrak {n}/\mathfrak {z}$ with

$$ \begin{align*} \mathbb{W}=\mathbb{W}_{C}=F\oplus \mathcal{J}\oplus\mathcal{J}\oplus F. \end{align*} $$

So any element $w\in \mathbb {W}$ is a quadruple $w=(a,b,c,d)$ , where $a,d\in F$ and $b,c\in \mathcal {J}$ . The space $\mathbb W$ comes with a symplectic form

$$ \begin{align*} \langle(a,b,c,d),(a^{\prime},b^{\prime},c^{\prime},d^{\prime})\rangle_{\mathbb{W}}=ad^{\prime}-\mathrm{Tr}(b*c^{\prime})+\mathrm{Tr}(c*b^{\prime})-da^{\prime}, \end{align*} $$

and a quartic form

$$ \begin{align*} q(a,b,c,d)=(ad-\mathrm{Tr}(b*c))^{2}+4aN(c)+4dN(b)-4\mathrm{Tr}(b^{\#}*c^{\#}). \end{align*} $$

Let $(-,-,-,-)_{\mathbb {W}}$ be the unique symmetric $4$ -linear form on $\mathbb {W}$ such that $(v,v,v,v)=2q(v)$ . Then $\mathcal M$ is isomorphic to the group of similitudes

$$ \begin{align*} M_{C}=\{(g,\nu)\in \mathrm{GL}(\mathbb{W})\times \mathrm{GL}_1(F)|\langle gv,gv^{\prime}\rangle=\nu\langle v,v^{\prime}\rangle,\,q(gv)=\nu^{2}q(v)\text{ for all }v,v^{\prime}\in \mathbb{W}\}. \end{align*} $$

We write $M_{C}^{1}$ for the subgroup of elements where the similitude factor $\nu $ is equal to $1$ .

We highlight a few subgroups of $M^1_{C}$ . If $h\in H_{C}$ with similitude factor $\lambda $ , then the map $(a,b,c,d)\mapsto (\lambda a,hb,\tilde {h}c,\lambda ^{-1}d)$ , where the action of $\tilde {h}$ on $\mathcal {J}$ is defined through the identification of $\mathcal {J}$ with $\mathcal {J}^{*}$ via the trace pairing, defines an element of $M^1_{C}$ . We abuse notation and let $H_C$ denote this subgroup.

For $x\in \mathcal {J}$ let $n(x)$ be the map defined by

(2.3) $$ \begin{align} n(x)(a,b,c,d)=(a,b+ax,c+b\times x+ax^{\#},d+\mathrm{Tr}(c*x)+\mathrm{Tr}(b*x^{\#})+aN(x)). \end{align} $$

The map $n(x)\in M_{\mathcal {J}}$ and has similitude factor equal to $1$ . The group generated by these elements is isomorphic to the unipotent group $\exp (\mathfrak g_{\beta }) \subset \mathcal M$ .

Similarly, for $x\in \mathcal {J}$ let $\overline {n}(x)$ be the map defined by

$$ \begin{align*} \overline{n}(x)(a,b,c,d)=(a+\mathrm{Tr}(b*x)+\mathrm{Tr}(c*x^{\#})+dN(x),b+c\times x+dx^{\#},c+dx,d). \end{align*} $$

The map $\overline {n}(x)\in M_{C}$ and has similitude factor equal to $1$ . The group generated by these elements is isomorphic to the unipotent group $\exp (\mathfrak g_{-\beta }) \subset \mathcal M$ .

The two abelian groups generated by $n(x)$ and $\overline n(x)$ , respectively, are unipotent radicals of two opposite maximal parabolic subgroups in $M_{C}^1$ with the Levi factor $H_{C}$ . These two parabolic groups are conjugate by

(2.4) $$ \begin{align} (a,b,c,d)\mapsto (-d,c,-b,a). \end{align} $$

If $\lambda \in \mathrm {GL}_{1}(F)$ , then

$$\begin{align*}s_{\lambda}:(a,b,c,d)\mapsto (\lambda^{2},\lambda b,c,\lambda^{-1}d) \end{align*}$$

and

$$\begin{align*}s^*_{\lambda}:(a,b,c,d)\mapsto (\lambda^{-1}, b,\lambda c,\lambda^{2}d) \end{align*}$$

are two elements of $M_C$ with the similitude factor $\nu =\lambda $ . Thus $M_C$ is generated by $M^1_C$ and any of the two one-parameter groups s or $s^{\ast }$ . We have written down both of these two groups for the sake of symmetry but also because they generate a two-dimensional torus whose Lie algebra is $\mathfrak t \subset \mathfrak m$ .

Observe that ${\mathrm {Aut}}(C) \subset M_C$ where ${\mathrm {Aut}}(C)$ acts on the coordinates of $\mathbb W_C$ . The centralizer of ${\mathrm {Aut}}(C)$ in $M_C$ is $M_F$ , the Levi of the Heisenberg maximal parabolic of $F_4$ . This group is isomorphic to $\mathrm {GSp}_6(F)$ , as one can see from root data, for example. We shall fix an isomorphism $M_F\cong \mathrm {GSp}_6(F)$ as follows. Recall that $\mathbb W_C$ is a symplectic space. Under the action of ${\mathrm {Aut}}(C)$ it decomposes as

$$\begin{align*}\mathbb W_C=\mathbb W_F \oplus (\mathcal J_{C^0} \oplus \mathcal J_{C^0}). \end{align*}$$

If $V_6$ is a six-dimensional symplectic space then $C^0\otimes V_6$ is a symplectic space obtaining by tensoring the quadratic space $C^0$ and the symplectic space $V_6$ . We pick $V_6$ so that

(2.5) $$ \begin{align} \mathcal J_{C^0} \oplus \mathcal J_{C^0}\cong C^0\otimes V_6, \end{align} $$

given by $(J(x),J(y))\mapsto (x_1,x_2,x_3,y_1,y_2,y_3)$ , is an isomorphism of symplectic spaces. Since $M_F$ commutes with ${\mathrm {Aut}}(C)$ , and ${\mathrm {Aut}}(C)$ acts on $C^0$ irreducibly, $M_F$ must act on $V_6$ , giving an identification with $\mathrm {GSp}_6(F)$ . Let $\mathrm {sim}$ denote the usual similitude character of $\mathrm {GSp}_6(F)$ . Observe that the similitude character of $M_{C}$ restricts to $\mathrm {sim}$ under the identification.

2.6 Orbits

We now describe orbits of $M_{\mathcal J}$ acting on $\mathbb W=\mathbb W_{C}$ . Given $v=(a,b,c,d)\in \mathbb {W}_{\mathcal {J}}$ define $v^{\flat }=(a^{\flat },b^{\flat },c^{\flat },d^{\flat })$ ([Reference Pollack22, Proposition 1.0.3]), where

$$ \begin{align*} a^{\flat}&=-a(ad-\mathrm{Tr}(b*c))-2N(b);\\b^{\flat}&=-2c\times b^{\#}+2ac^{\#}-(ad-\mathrm{Tr}(b*c))b;\\c^{\flat}&=2b\times c^{\#}-2bd^{\#}+(ad-\mathrm{Tr}(b*c))c;\\d^{\flat}&=d(ad-\mathrm{Tr}(b*c))+2N(c). \end{align*} $$

Over the algebraic closure the orbits are classified by the rank for elements in $\mathbb {W}$ , defined as follows. Let $v\in \mathbb {W}$ . The element v has rank at most 4. If $q(v)=0$ , then v has rank at most 3. If $v^{\flat }=0$ , then v has rank at most 2. If $(v,v,w,w^{\prime })=0$ for all $w,w^{\prime }\in (v)^{\perp }$ (the orthogonal complement with respect to $\langle -,-\rangle _{\mathbb {W}}$ ), then v has rank at most 1. If $v=0$ , then v has rank 0.

We need the following proposition [Reference Gan and Savin8, Proposition 8.1] and a simple corollary.

2.7 The dual pair ${\mathrm {Aut}}(C)\times \mathcal G_F$ in $\mathcal G_C$

Observe that ${\mathrm {Aut}}(C)$ naturally acts on $\mathcal J_C$ preserving the norm $N_{\mathcal J}$ . Thus $\mathrm {Der}(C)$ , the Lie algebra of ${\mathrm {Aut}}(C)$ , is a subalgebra of $\mathfrak h_C$ . From the construction of $\frak {g}_{C}$ in Subsection 2.4 it is evident that the centralizer of $\mathrm {Der}(C)$ in $\mathfrak {g}_C$ contains $\mathfrak {g}_F$ , the Lie algebra of $\mathcal G_F$ , the group of type $F_4$ . This group is simply connected with trivial center. Thus the inclusion of Lie algebras lifts to an inclusion $\mathcal G_F\subseteq \mathcal G_C$ .

Proof. Observe that, under the adjoint action restricted to $\mathrm {Der}(C) \oplus \mathfrak g_{F}$ ,

$$\begin{align*}\mathfrak g_C= \mathrm{Der}(C) \oplus \mathfrak g_{F} \oplus C^0\otimes V_{26} \end{align*}$$

where $C^0$ and $V_{26}$ are irreducible representations of $\mathrm {Der}(C)$ and $\mathfrak g_{F}$ respectively. The irreducibility of $C^0\otimes V_{26}$ implies that $\mathrm {Der}(C) \oplus \mathfrak g_{F}$ is a maximal subalgebra of $\mathfrak g$ .

First assume $\dim (C)\neq 2$ . Then ${\mathrm {Aut}}(C) \times \mathcal G_F$ is connected. Since $\mathrm {Der}(C) \oplus \mathfrak g_{F}$ is maximal, any proper subgroup H of $\mathcal {G}_{C}$ containing ${\mathrm {Aut}}(C) \times \mathcal G_F$ must have ${\mathrm {Aut}}(C) \times \mathcal G_F$ as its connected component of identity. Now ${\mathrm {Aut}}(C) \times \mathcal G_F$ has no outer automorphisms, so if H is disconnected, then there exists a semisimple finite-order element $z\in \mathcal {G}_C$ centralizing ${\mathrm {Aut}}(C) \times \mathcal G_F$ . Since $C^0\otimes V_{26}$ is irreducible, z would have to act on it as $-1$ . But the centralizer of the semisimple element z must contain a maximal torus. Since ${\mathrm {Aut}}(C) \times \mathcal G_F$ has rank $5$ and $\mathcal {G}_{C}$ has rank 7, this is a contradiction.

Second, assume $\dim C=2$ . Note that in this case $\mathcal {G}_{C}$ is disconnected and the component group is generated by the outerautomorphism of the $E_{6}$ Dynkin diagram, which has order $2$ . As above we can see that $\mathcal {G}_{F}$ is maximal in $\mathcal {G}_{C}^{\circ }$ , the connected component of the identity. Since the generator of ${\mathrm {Aut}}(C)=\mu _2$ is the outerautomorphism in $\mathcal {G}_{C}$ , the conclusion of the lemma holds in this case, too.

We are now in a position to understand rational $\mathcal G_C$ -conjugacy classes of ${\mathrm {Aut}}(C) \times \mathcal G_F\subset \mathcal {G}_C$ over F. Over $\bar F$ , there is one conjugacy class, that is, any subgroup of $\mathcal G_C$ isomorphic to ${\mathrm {Aut}}(C) \times \mathcal {G}_F$ is conjugate to ${\mathrm {Aut}}(C) \times \mathcal {G}_F$ , this is due to Dynkin. By Lemma 2.4, the normalizer of ${\mathrm {Aut}}(C) \times \mathcal G_F$ in $\mathcal {G}_C$ is ${\mathrm {Aut}}(C) \times \mathcal G_F$ , hence conjugacy classes over F correspond to the kernel of the map of pointed sets

$$\begin{align*}H^1(F, {\mathrm{Aut}}(C)) \times H^1(F, \mathcal G_F) \rightarrow H^1(F, \mathcal{G}_C). \end{align*}$$

If F is p-adic, then $H^1(F, \mathcal G_F)$ is trivial, hence we are reduced to the map $H^1(F, {\mathrm {Aut}}(C)) \rightarrow H^1(F, \mathcal {G}_C)$ . This maps sends a rational form $C'$ of C (i.e., an element of $H^1(F, {\mathrm {Aut}}(C))$ ) to $\mathcal G_{C'}$ . This map is clearly injective, hence we have only one conjugacy class. Furthermore, injectivity implies that ${\mathrm {Aut}}(C) \times \mathcal G_F$ can be a subgroup of $\mathcal G_{C'}$ only when $C\cong C'$ .

In this paper we use two different constructions of the dual pair $\mathrm {Aut}(C)\times \mathcal {G}_{F}\subseteq \mathcal {G}_{C}$ to investigate the theta correspondence. From the above these two subgroups are conjugate. Thus the results obtained using each construction are compatible.

2.8 Degenerate principal series on $F_{4}$

In this section, we collect the results of Choi–Jantzen [Reference Choi and Jantzen4] that describe the structure of degenerate principal series on $F_{4}$ . Henceforth we write $G=\mathcal G_F$ for the unique group of type $F_4$ over F.

We note that in this paper we use the Bourbaki labeling of simple roots of $F_{4}$ [Reference Bourbaki3, Plate VIII], but Choi–Jantzen [Reference Choi and Jantzen4] use the reverse order, that is, the two labelings are related by the permutation $(14)(23)$ , written in disjoint cycle notation.

2.9 $\mathrm {PGL}_{2}(F)$

In this subsection, we recall basic facts about $\mathscr G= \mathrm {PGL}_{2}(F)$ .

Let $\mathscr {B}=\mathscr T \mathscr {U}$ be a Borel subgroup of $\mathscr G$ and let $\overline {\mathscr {B}}=\mathscr T \overline {\mathscr {U}}$ be its opposite. We fix an identification $\mathscr T\cong F^{\times }$ so that $\mathscr T$ acts on $\overline {\mathscr {U}}$ by multiplication. In particular, the modular character is $\delta _{\overline {\mathscr {B}}}(t)=|t|$ . Let $\chi $ be a character of $\mathscr {T}$ and define $i_{\overline {\mathscr {B}}}^{\mathscr {G}}(\chi )=\mathrm {Ind}^{\mathscr {G}}_{\overline {\mathscr {B}}}(\delta _{\overline {\mathscr {B}}}^{1/2}\cdot \chi )$ . Let $\mathrm {St}$ denote the Steinberg representation of $\mathscr G$ . We have the following well-known result:

2.10 Theta lifting preliminaries

In this subsection we establish preliminaries to discuss a theta lift for $\mathrm {Aut}(C)\times F_{4}\subset E_{7}$ .

Recall that associated to a quaternion algebra C we have the following groups: $\mathcal {G}$ is the F-points of a connected semisimple adjoint group of type $E_{7}$ ; G is the F-points of a connected semisimple group of type $F_{4}$ ; $\mathscr {G}$ is the F-points of the automorphism group of C. Let $(\Pi ,\mathcal {V})=(\Pi _{\mathrm {min}},\mathcal {V}_{\mathrm {min}})$ be the minimal representation of $\mathcal {G}$ . (See [Reference Kazhdan and Savin14] for split groups; [Reference Gan and Savin7] for nonsplit.)

To begin we define the big theta lift. Let $\tau $ be an irreducible smooth $\mathscr {G}$ -representation. The maximal $\tau $ -isotypic quotient of $\mathcal {V}$ is naturally a $\mathscr {G}\times G$ -representation $\mathcal {V}_{\tau }$ . Furthermore, $\mathcal {V}_{\tau }$ admits a factorization $\mathcal {V}_{\tau }\cong \tau \otimes \Theta (\tau )$ , where $\Theta (\tau )$ is a G-representation. We call $\Theta (\tau )$ the big theta lift of $\tau $ with respect to the restriction of $\mathcal {V}$ to $\mathscr {G}\times G$ . (For details, see [Reference Moeglin, Vignéras and Waldspurger20, Chapter 2, Section 3].) Let $\theta (\tau )$ be the maximal semisimple quotient (cosocle) of $\Theta (\tau )$ .

The main objective of this work is to investigate $\Theta (\tau )$ and $\theta (\tau )$ . The following simple lemma is important for our analysis. If V is a vector space, we write $V^{*}$ for its linear dual.

Lemma 2.8. Let $\tau \in \mathrm {Irr}(\mathscr {G})$ . Let $U\subset G$ be a unipotent subgroup with a character $\Psi $ and let $H=\mathrm {Stab}_{G}(U,\Psi )$ . There is an H-module isomorphism

$$ \begin{align*} (\Theta(\tau)_{(U,\Psi)})^{*}\cong \mathrm{Hom}_{\mathscr{G}}(\mathcal{V}_{(U,\Psi)},\tau), \end{align*} $$

where the H acts on $\mathrm {Hom}_{\mathscr {G}}(\mathcal {V}_{(U,\Psi )},\tau )$ by $(h\cdot f)(v)=f(h^{-1}\cdot v)$ .

In particular, there is an isomorphism of G-modules

$$ \begin{align*} \Theta(\tau)^{*}\cong \mathrm{Hom}_{\mathscr{G}}(\mathcal{V},\tau). \end{align*} $$

Proof. Because $\tau $ is irreducible, it follows that $\Theta (\tau )\cong (\mathcal {V}\otimes \tilde {\tau })_{\mathscr {G}}$ as G-modules. Taking $(U,\Psi )$ -coinvariants gives $\Theta (\tau )_{(U,\Psi )}\cong [(\mathcal {V}\otimes \tilde {\tau })_{\mathscr {G}}]_{(U,\Psi )}$ as H-modules. Since H and $\mathscr {G}$ commute we can commute the coinvariants to get an H-module isomorphism $\Theta (\tau )_{(U,\Psi )}\cong (\mathcal {V}_{(U,\Psi )}\otimes \tilde {\tau })_{\mathscr {G}}$ . Next we take the linear dual and apply the $\otimes $ -Hom adjunction to get an isomorphism of H-modules

$$ \begin{align*} (\Theta(\tau)_{(U,\Psi)})^{*}\cong \mathrm{Hom}((\mathcal{V}_{(U,\Psi)}\otimes \tilde{\tau})_{\mathscr{G}},\mathbb{C})\cong \mathrm{Hom}_{\mathscr{G}}(\mathcal{V}_{(U,\Psi)},(\tilde{\tau})^{*}). \end{align*} $$

Since $\mathcal {V}_{(U,\Psi )}$ is a smooth $\mathscr {G}$ -module we have $\mathrm {Hom}_{\mathscr {G}}(\mathcal {V}_{(U,\Psi )}, (\tilde {\tau })^{*})\cong \mathrm {Hom}_{\mathscr {G}}(\mathcal {V}_{(U,\Psi )},\tilde {\tilde {\tau }})$ . Since $\tau $ is irreducible we have $\tilde {\tilde {\tau }}\cong \tau $ as $\mathscr {G}$ -modules. Thus as H-modules $(\Theta (\tau )_{(U,\Psi )})^{*}\cong \mathrm {Hom}_{\mathscr {G}}(\mathcal {V}_{(U,\Psi )},\tau )$ .

3.1 Fiber calculation

The main objective of this section is to compute $C_{c}^{\infty }(\Omega )_{(\overline {N},\Psi )}$ , where $\Psi $ is of rank 3 or rank 0. This is accomplished in Proposition 3.7 for rank 3, and Proposition 3.11 for rank 0. The rank of $\Psi $ will be defined in terms of the rank of elements of $\mathbb {W}_{F}$ (Subsection 2.5). We explain this after setting up some notation.

The map $\mathcal {N}/Z\rightarrow \mathrm {Hom}(\overline {\mathcal {N}}/\overline {Z},\mathbb {C}^{\times })$ defined by $n\mapsto \psi (\langle n, - \rangle )$ defines an isomorphism of $\mathcal {N}/Z$ with the Pontryagin dual of $\overline {\mathcal {N}}/\overline {Z}$ . Similarly, by restriction this map defines an isomorphism between $N/Z$ and the Pontryagin dual of $\overline {N}/\overline {Z}$ .

We identify $\mathcal {N}/Z$ with $\mathbb {W}_{C}$ and $\mathcal M$ with $M_C$ so that the adjoint action of $\mathcal {M}$ on $\mathcal {N}/Z$ corresponds to the action of $M_C$ on $\mathbb {W}_{C}$ . This also fixes an identification of $N/Z$ with $\mathbb {W}_{F}= \mathbb {W}_{C}^{{\mathrm {Aut}}(C)}$ and of $M\subset \mathcal M$ with $M_F \subset M_C$ . Thus we can view the character $\Psi $ as an element of $\mathbb {W}_{F}$ . We define the rank of $\Psi $ to be the rank of its associated element in $\mathbb {W}_{F}$ .

Now we reinterpret the set $\mathcal {N}(\Psi )$ as the fiber of a map $\mathbf {F}$ , defined below.

Let $\mathbf {f}:\mathcal {J}_{C}\rightarrow \mathcal {J}_{F}$ be the map defined by

$$ \begin{align*} \left(\begin{smallmatrix} a & x & \overline{z}\\ \overline{x} & b & y\\ z & \overline{y} & c \end{smallmatrix}\right) \mapsto \left(\begin{smallmatrix} a & \frac{\mathrm{Tr}(x)}{2} & \frac{\mathrm{Tr}(z)}{2}\\ \frac{\mathrm{Tr}(x)}{2} & b & \frac{\mathrm{Tr}(y)}{2}\\ \frac{\mathrm{Tr}(z)}{2} & \frac{\mathrm{Tr}(y)}{2} & c \end{smallmatrix}\right). \end{align*} $$

Let $\mathbf {F}:\mathbb {W}_{C}\rightarrow \mathbb {W}_{F}$ be defined by

$$ \begin{align*} (a,b,c,d)\mapsto (a,\mathbf{f}(b),\mathbf{f}(c),d). \end{align*} $$

With the identifications above, the natural restriction map from the Pontryagin dual of $\overline {\mathcal {N}}/\overline {Z}$ to the Pontryagin dual of $\overline {N}/\overline {Z}$ is realized as the map $\mathbf {F}:\mathbb {W}_{C}\rightarrow \mathbb {W}_{F}$ . Viewing $\Psi $ as an element of $\mathbb {W}_{F}$ , we have $\mathcal {N}(\Psi )=\mathbf {F}^{-1}(\Psi )$ . Thus our next objective is to describe the intersection of the fibers of $\mathbf {F}$ with $\Omega $ .

Proposition 3.4. Let C be any composition algebra. Let $\xi =(1,0,c,d)\in \mathbb {W}_{F}$ . The set $\mathbf {F}^{-1}(\xi )\cap \Omega $ consists of

$$\begin{align*}(1, J(x), J(x)^{\#} , N_{\mathcal J}(J(x))) \end{align*}$$

for all $x=(x_{1},x_{2},x_{3})\in (C^{0})^3$ such that

1. 1. $c= \frac {1}{2}(\mathrm {Tr} (x_ix_j) ) $

2. 2. $d=\mathrm {Tr}(x_{1}x_{2}x_{3})$ .

Proof. Let $\Xi =(a^{\prime },b^{\prime },c^{\prime },d^{\prime })\in \mathbf {F}^{-1}(\xi )\cap \Omega $ .

Since $\Xi \in \mathbf {F}^{-1}(\xi )$ , it follows that $\Xi =(1,b^{\prime },c^{\prime },d)$ , where

$$ \begin{align*} b^{\prime}&=J(x_{1},x_{2},x_{3}), \text{ with } x_{j}\in C^{0}; \\ \mathbf{f}(c^{\prime}) &= c. \end{align*} $$

Since $\Xi \in \Omega $ , by Proposition 2.2, $d=N(b^{\prime })=\mathrm {Tr}(x_{1}x_{2}x_{3})$ and

$$ \begin{align*} c^{\prime}=(b^{\prime})^{\#}=\left( \begin{smallmatrix} -N(x_{1}) & \overline{x_{2}}\overline{x_{1}} & x_{3}x_{1}\\ x_{1}x_{2} & -N(x_{2}) & \overline{x_{3}}\overline{x_{2}}\\ \overline{x_{1}}\overline{x_{3}} & x_{2}x_{3} & -N(x_{3}) \end{smallmatrix} \right)=\left( \begin{smallmatrix} x_{1}^{2} & x_{2}x_{1} & x_{3}x_{1}\\ x_{1}x_{2} & x_{2}^{2} & x_{3}x_{2}\\ x_{1}x_{3} & x_{2}x_{3} & x_{3}^{2} \end{smallmatrix} \right). \end{align*} $$

Thus

$$ \begin{align*} c=\mathbf{f}(c^{\prime})=\frac{1}{2}\left( \begin{smallmatrix} \mathrm{Tr}(x_{1}^{2}) & \mathrm{Tr}(x_{1}x_{2}) & \mathrm{Tr}(x_{1}x_{3})\\ \mathrm{Tr}(x_{1}x_{2}) & \mathrm{Tr}(x_{2}^{2}) & \mathrm{Tr}(x_{2}x_{3})\\ \mathrm{Tr}(x_{1}x_{3}) & \mathrm{Tr}(x_{2}x_{3}) & \mathrm{Tr}(x_{3}^{2}) \end{smallmatrix} \right).\\[-49pt] \end{align*} $$

Now we describe $\mathbf {F}^{-1}((1,0,c,d))\cap \Omega $ , where $c\in \mathcal {J}_{F}$ has rank 3 and $\mathrm {dim}C=4$ .

Proof. We start with a lemma.

Lemma 3.6. Let $x=(x_1,x_2,x_3)\in (C^0)^{3}$ . Then we have the following identity of sextic polynomials

$$\begin{align*}-4\det (\frac{1}{2} \mathrm{Tr}(x_ix_j) )= [ \mathrm{Tr}(x_1x_2x_3)]^2. \end{align*}$$

Proof. Let $g\in {\mathrm {GL}}_3(F)$ . Let $y=(y_1,y_2,y_3)\in (C^{0})^{3}$ defined by $y=xg$ . It is clear that

$$\begin{align*}\det ( \mathrm{Tr}(y_iy_j) ) = \det(g)^2\cdot \det ( \mathrm{Tr}(x_ix_j) ). \end{align*}$$

On the other hand, $\mathrm {Tr}(x_1x_2x_3)$ is a nontrivial trilinear, skew-symmetric form. Since $C^{0}$ has dimension 3, the form induces an isomorphism of $\wedge ^3 C^{0}$ and F. Hence

$$\begin{align*}\mathrm{Tr}(y_1y_2y_3) =\det(g)\cdot \mathrm{Tr}(x_1x_2x_3). \end{align*}$$

Since ${\mathrm {GL}}_3(F)$ acts transitively on the open set of all bases $(x_1,x_2,x_3)$ of $C^0$ , it suffices now to check the identity on one basis of $C^{0}$ . So let us take usual $i,j,k$ such that $i^2=a$ , $j^2=b$ , $ij=k$ and $k^2=-ab$ . Then both sides of the proposed identity are equal to $(2ab)^2$ .

Now, the if and only if statement is a simple combination of the lemma and Proposition 3.4. For the last statement, on the structure of the fiber, observe that the set of x such that $c=\frac 12( \mathrm {Tr}(x_ix_j) )$ is a principal homogeneous space for $\mathrm {O}(C^0)$ . Since $\mathrm {O}(C^{0})\cong \mathrm {SO}(C^{0})\times \{\pm 1\}$ and $\mathrm {Tr}((-x_{1})(-x_{2})(-x_{3}))= -\mathrm {Tr}(x_{1}x_{2}x_{3})$ , the additional equation $d=\mathrm {Tr}(x_{1}x_{2}x_{3})$ assures that the fiber $\Omega _{\xi }$ is a principal homogenous space for $\mathrm {SO}(C^{0})={\mathrm {Aut}}(C)$ .

We shed some light on $\mathrm {Stab}_M(\Psi )$ for rank 3 characters. Recall that we have ${\mathrm {GL}}_3(F) \subset M$ such that $g\in {\mathrm {GL}}_3(F)$ acts on $\xi =\xi _{\Psi }= (1,0,c,d)\in \mathbb W_F$ by

$$\begin{align*}(\det(g), 0, \det(g)^{-1} g c g^{\top} , \det(g)^{-1} d). \end{align*}$$

Hence $g\in \mathrm {Stab}_M(\Psi )$ if and only if $\det (g)=1$ and $gcg^{\top }=c$ . In other words the stabilizer of $\xi $ in ${\mathrm {GL}}_3(F)$ is the group $\mathrm {SO}(3,c)$ . Thus we have an action of ${\mathrm {Aut}}(C) \times \mathrm {SO}(3,c)$ on $\Omega _{\xi }$ . Explicitly, $(g,h) \in {\mathrm {Aut}}(C) \times \mathrm {SO}(3,c)$ acts on $x=(x_1,x_2,x_3)\in \Omega _{\xi }$ by

$$ \begin{align*} x\mapsto (gx_1, gx_2,gx_3) h^{\top}. \end{align*} $$

We have the following corollary to Proposition 3.5.

Corollary 3.7. Assume $\mathrm {dim}C=4$ . Let $\Psi $ be a rank $3$ character of $\overline {N}$ corresponding to ${\xi = (1,0,c,d)\in \mathbb {W}_{F}}$ such that $\mathrm {SO}(3,c) \cong {\mathrm {Aut}}(C)$ , where $\mathrm {SO}(3,c) \subseteq \mathrm {Stab}_M(\Psi )$ described above. Then there are isomorphisms of ${\mathrm {Aut}}(C) \times \mathrm {SO}(3,c) $ -modules

$$ \begin{align*} \mathcal{V}_{(\overline{N},\Psi)}\cong C^{\infty}_{c}(\Omega)_{(\overline{N},\Psi)} \cong C^{\infty}_{c}(\Omega_{\xi}) \cong C^{\infty}_{c}(\mathrm{Aut}(C)) \cong C^{\infty}_{c}( \mathrm{SO}(3,c)) \end{align*} $$

where the last two isomorphisms depend on a choice of a point in $\Omega _\xi $ , giving identifications of $\Omega _{\xi }$ with ${\mathrm {Aut}}(C)$ and $\mathrm {SO}(3,c)$ , and an isomorphism $\mathrm {SO}(3,c)\cong {\mathrm {Aut}}(C)$ .

Next we give the analog of Proposition 3.5, where C is a quadratic composition algebra.

Now we discuss the rank $0$ case, that is, $\Psi $ is trivial. We begin by computing $\mathbf {F}^{-1}(0)\cap \Omega $ . Observe that $\mathbf {F}^{-1}(0)=\mathcal {J}_{C^{0}}\oplus \mathcal {J}_{C^{0}}$ . Recall that we have identified $M\cong {\mathrm {GSp}}(V_6)$ such that

$$\begin{align*}\mathcal{J}_{C^{0}}\oplus\mathcal{J}_{C^{0}}\cong C^0\otimes V_6 \end{align*}$$

via the map $(J(x),J(y))\mapsto (x_1,x_2,x_3,y_1,y_2,y_3)$ (Subsection 2.5).

Proof. Let $\Xi \in \mathbf {F}^{-1}(0)=\mathcal {J}_{C^{0}}\oplus \mathcal {J}_{C^{0}}$ , then $\Xi =(0,b,c,0)$ , where

$$ \begin{align*} b&=J(\beta_{1},\beta_{2},\beta_{3})\in \mathcal{J}_{C^{0}},\\ c&=J(\gamma_{1},\gamma_{2},\gamma_{3})\in \mathcal{J}_{C^{0}}. \end{align*} $$

If $\Xi \in \Omega $ , then by Lemma 2.2 we know that b or c is not equal to $0$ and

$$ \begin{align*} b^{\#}&=0,\\ c^{\#}&=0,\\ b*c&=0. \end{align*} $$

The equation $b^{\#}=0$ implies that $\beta _{i}^{2}=\beta _{j}^{2}=\beta _{i}\beta _{j}=0$ . Similarly, the equation $c^{\#}=0$ implies that $\gamma _{i}^{2}=\gamma _{j}^{2}=\gamma _{i}\gamma _{j}=0$ . If $C^0$ is anisotropic then there are no nonzero nilpotent elements. This proves the first claim. Now assume $C=\mathrm {M}(2,F)$ , the algebra of $2\times 2$ matrices. The equation $b\ast c=0$ implies $\beta _i\gamma _{j}+\gamma _{j}\beta _i=0$ for all i and j. We need the following lemma.

Lemma 3.10. Let $\beta ,\gamma \in \mathrm {M}(2,F)$ . If $\beta ^2=0$ , $\gamma ^2=0$ and $\beta \gamma +\gamma \beta =0$ . Then $\beta $ and $\gamma $ are proportional.

Proof. This is trivial if $\beta $ or $\gamma $ is $0$ , so suppose not. Then $\ker \beta =\mathrm {Im}\beta $ and $\ker \gamma =\mathrm {Im}\gamma $ are one dimensional. The equation $\beta \gamma =-\gamma \beta $ implies that $\gamma $ acts on $\ker \beta $ . Thus $\ker \beta =\mathrm {Im}\beta =\ker \gamma =\mathrm {Im}\gamma $ and so $\beta $ and $\gamma $ are proportional.

It follows that all $\beta _{i}$ and $\gamma _{j}$ are linearly dependent, proving the proposition.

Now we can describe $\mathcal {V}_{\overline {N}}$ .

Proof. By Theorem 3.1, $\mathcal V_{\overline {N}}$ has a filtration with quotient $\mathcal V_{\overline {\mathcal N}}$ and submodule

$$\begin{align*}C_c^{\infty}(\Omega)_{\overline{N}}\cong C_c^{\infty}(\Omega_0). \end{align*}$$

Now we apply by Proposition 3.9. If $C^0$ is anisotropic then $\Omega _0$ is empty and we are done. So suppose $C=\mathrm {M}(2,F)$ . Then $\Omega _0\subset C^0\otimes V_6$ consists of nonzero pure tensors $x\otimes v$ such that $x^2=0$ . Fix $\omega =x\otimes v$ . The stabilizer in $\mathscr {G}$ of the line through x is a Borel subgroup $\mathscr {B}$ , and the stabilizer in $\mathrm {GSp}_6$ of the line through v is a maximal parabolic subgroup Q. The stabilizer of $x\otimes v$ is a subgroup of $\mathscr {B}\times Q$ such that the quotient is $F^{\times }$ . Observe that $C_c^{\infty }(\Omega _0)$ , as a $\mathscr {G}\times \mathrm {GSp}_6(F)$ -module, is obtained by compact induction of the trivial representation of the stabilizer of $x\otimes v$ . Hence, using induction in stages,

$$\begin{align*}C_c^{\infty}(\Omega_0)\cong \mathrm{Ind}_{\mathscr{B}\times Q}^{\mathscr{G}\times \mathrm{GSp}_6} (C_c^{\infty}(F^{\times})) \end{align*}$$

where the induction is not normalized. This completes the proof, after taking into account additional twisting by the character of $\mathcal M$ in Theorem 3.1.

We remark that the variant of the previous proposition, when C is an octonion algebra, was obtained in [Reference Savin and Woodbury27].

3.2 Fourier–Jacobi functor

Now we recall the definition of the Fourier–Jacobi functor. (For more details, see Weissman [Reference Weissman30].) By the Stone-Von-Neumann theorem, the group N has a unique irreducible smooth representation with central character $\psi $ , denoted by $(\rho ^{N}_{\psi },W_{\psi })$ . By [Reference Weissman30, Proposition 2.5], there is a unique extension of $(\rho ^{N}_{\psi },W_{\psi })$ to a projective representation of $M_{1}N$ , where $M_{1}$ is the commutator subgroup of M. Furthermore, $\widetilde {\mathrm {Sp}}(14,F)$ , the two-fold cover of the symplectic group $\mathrm {Sp}(14,F)$ where $14=\mathrm {dim}(N/Z)$ , also acts on $W_{\psi }$ via the Weil representation. So, $\widetilde {M}_{1}\cong \widetilde {\mathrm {Sp}}(6,F)$ , the metaplectic double cover of $M_{1}\cong \mathrm {Sp}(6,F)$ , acts on $W_{\psi }$ through the Weil representation of $\widetilde {\mathrm {Sp}}(14,F)$ .

We now make a brief comment on why the embedding $M_{1}\hookrightarrow \mathrm {Sp}(14,F)$ induced by the action of $M_{1}$ on $N/Z$ induces an embedding $\widetilde {M}_{1}\hookrightarrow \widetilde {Sp}(14,F)$ . The representation of $M_{1}$ on $N/Z$ is irreducible and corresponds to the third fundamental weight of $M_{1}\cong \mathrm {Sp}(6,F)$ (Bourbaki labeling). Let $H\subset M_{1}$ be a long-root $\mathrm {SL}(2)$ subgroup. Then the restriction of the $M_{1}$ -representation $N/Z$ to H decomposes into four copies of the trivial representation and five copies of the unique two-dimensional representation of $\mathrm {SL}(2)$ . Thus the image of H in $\mathrm {Sp}(14,F)$ sits diagonally inside five commuting long-root $\mathrm {SL}(2)$ subgroups of $\mathrm {Sp}(14)$ . Since five is odd and each root is long, the preimage of H in $\widetilde {Sp}(14,F)$ does not split. Therefore the preimage of $M_{1}$ in $\widetilde {Sp}(14,F)$ does not split, giving the embedding $\widetilde {M}_{1}\hookrightarrow \widetilde {Sp}(14,F)$ .

If $(\pi ,V)$ is a smooth representation of G, then the Fourier–Jacobi functor with respect to the Heisenberg parabolic P sends $\pi $ to

$$ \begin{align*} \mathrm{FJ}(\pi)=\mathrm{Hom}_{N}(W_{\psi},V_{(Z,\psi)}). \end{align*} $$

The space $\mathrm {FJ}(\pi )$ is an $\widetilde {M_{1}}$ -module with the action defined by $[m\cdot f](w)=\pi (m)f(m^{-1}w)$ , where the action of $\widetilde {M}_{1}$ on $V_{(Z,\psi )}$ factors through $M_{1}$ . The Fourier–Jacobi functor does not depend on $\psi $ ([Reference Weissman30, Proposition 3.1]).

4.1 At most two nontrivial constituents

We begin by using the Fourier–Jacobi functor and a classical theta correspondence to show that $\Theta (\tau )$ has at most two nontrivial constituents. This is accomplished in Corollary 4.5.

Let $\mathcal {M}_{1}\subset \mathcal {M}$ be the commutator subgroup. Let $\mathcal {P}_{1}=\mathcal {M}_{1}\mathcal {N}$ .

Lemma 4.2. Let $P_{1}=M_{1}N$ . As an $\mathrm {Aut}(C)\times P_{1}$ -module,

$$ \begin{align*} \mathcal{V}_{(Z,\psi)} \cong \rho_{\psi}^{N}\otimes \omega_{\psi}, \end{align*} $$

where $\omega _{\psi }$ is the Weil representation of $O(C^{0})\times \widetilde {\mathrm {Sp}}(V_{6})$ as a dual pair in $\widetilde {\mathrm {Sp}}(C^{0}\otimes V_{6})$ . Under this isomorphism, $\mathscr {G}$ acts on the second factor, while the action of $M_{1}$ is on both factors. (We note that $M_{1}$ does not act on either factor individually. Rather $\widetilde {M}_{1}$ acts genuinely on both factors, thus the diagonal action factors through $M_{1}$ .)

Using Lemma 4.2 we show in the next proposition that the Fourier–Jacobi functor with respect to P applied to $\mathcal {V}$ is isomorphic to the Weil representation. This allows us to study $\Theta (\tau )$ using a classical $O(3)\times \mathrm {Sp}(6)$ theta correspondence.

Proof. By definition $\mathrm {FJ}(\mathcal {V})=\mathrm {Hom}_{N}(\rho ^{N}_{\psi },\mathcal {V}_{(Z,\psi )})$ . By Lemma 4.2, $\mathcal {V}_{Z,\psi }\cong \omega _{\psi }\otimes \rho ^{N}_{\psi }$ as $\mathscr {G}\times M_{1}N$ -modules. Thus as $\mathscr {G}\times \widetilde {M}_{1}$ -modules

$$ \begin{align*} \mathrm{FJ}(\mathcal{V})\cong \mathrm{Hom}_{N}(\rho^{N}_{\psi},\omega_{\psi}\otimes\rho^{N}_{\psi}). \end{align*} $$

Since $\rho ^{N}_{\psi }$ is a finitely generated N-module, $\mathrm {Hom}_{N}(\rho ^{N}_{\psi },\omega _{\psi }\otimes \rho ^{N}_{\psi })\cong \omega _{\psi }\otimes \mathrm {Hom}_{N}(\rho ^{N}_{\psi },\rho ^{N}_{\psi })$ . By Schur’s lemma $\mathrm {Hom}_{N}(\rho ^{N}_{\psi },\rho ^{N}_{\psi })\cong \mathbb {C}$ . Thus $\mathrm {FJ}(\mathcal {V})\cong \omega _{\psi }$ as $\mathscr {G}\times \widetilde {M}_{1}$ -modules.

Now we introduce some notation to discuss the $O(C^{0})\times \mathrm {Sp}(6,F)$ theta correspondence. Since $\mathscr {G}\times M_{1}\cong \mathrm {SO}(C^{0})\times \mathrm {Sp}(6,F)$ , we almost have a classical dual pair. The representation $\tau $ admits two extensions to the group $O(C^{0})\cong \mathrm {SO}(C^{0})\times \{\pm id_{C^{0}}\}$ determined by whether $-id_{C^{0}}$ acts by $\pm 1$ . We write $\tau ^{\pm }$ for the two extensions and $\Theta ^{\dagger }(\tau ^{\pm })$ for the big theta lift of $\tau ^{\pm }$ with respect to the action of $O(C^{0})\times \widetilde {\mathrm {Sp}}(6,F)$ on the Weil representation $\omega _{\psi }$ of $\widetilde {\mathrm {Sp}}(18,F)$ .

Proposition 4.4. Let $\tau \in \mathrm {Irr}(\mathscr {G})$ . There is a surjective $\widetilde {M}_{1}$ -module homomorphism

$$ \begin{align*} \Theta^{\dagger}(\tau^{+})\oplus \Theta^{\dagger}(\tau^{-})\twoheadrightarrow\mathrm{FJ}(\Theta(\tau)). \end{align*} $$

Proof. We apply the Fourier–Jacobi functor, which is exact, to the surjective map $\mathcal {V}\twoheadrightarrow \tau \otimes \Theta (\tau )$ to get a map of $\mathscr {G}\times \widetilde {M}$ -modules

(4.1) $$ \begin{align} \mathrm{FJ}(\mathcal{V})\twoheadrightarrow \tau\otimes \mathrm{FJ}(\Theta(\tau)). \end{align} $$

By Proposition 4.3, we know that $\mathrm {FJ}(\mathcal {V})\cong \omega _{\psi }$ as $\mathscr {G}\times \widetilde {M}_{1}$ -modules. Therefore, we have a surjective $O(C^{0})\times \widetilde {\mathrm {Sp}}(6,F)$ -module map

$$ \begin{align*} \omega_{\psi}\twoheadrightarrow (\tau^{+}\otimes \Theta^{\dagger}(\tau^{+}))\oplus (\tau^{-}\otimes \Theta^{\dagger}(\tau^{-})). \end{align*} $$

Upon restricting to $SO(C^{0})\times \widetilde {\mathrm {Sp}}(6,F)\cong \mathscr {G}\times \widetilde {M}_{1}$ we get a surjective homomorphism

$$ \begin{align*} \mathrm{FJ}(\mathcal{V})\cong \omega_{\psi}\twoheadrightarrow \tau\otimes (\Theta^{\dagger}(\tau^{+})\oplus \Theta^{\dagger}(\tau^{-})). \end{align*} $$

Moreover, this is the surjection onto the maximal $\tau $ -isotypic quotient of $\mathrm {FJ}(\mathcal {V})$ . Thus the map from line (4.1) factors through the maximal $\tau $ -isotypic quotient to give a surjection

$$ \begin{align*} \tau\otimes (\Theta^{\dagger}(\tau^{+})\oplus \Theta^{\dagger}(\tau^{-}))\twoheadrightarrow \tau\otimes \mathrm{FJ}(\Theta(\tau)). \end{align*} $$

By construction, this map factors over the tensor product and the result follows.

4.2 Unique nontrivial constituent

In this subsection, we show that $\Theta (\tau )$ has exactly one nontrivial constituent.

Using Propositions 3.7 and 3.11 we can compute twisted coinvariants of $\Theta (\tau )$ .

Proposition 4.6. Let $(1,0,c, d)\in \mathbb {W}_{F}$ be an element of rank $3$ such that $\mathrm {SO}(c,3) \cong {\mathrm {Aut}}(C).$ Let $\Psi ^{\pm }$ be the character of $\overline {N}/\overline {Z}$ corresponding to the element $(1,0,c,\pm d)\in \mathbb {W}_{F}\cong N/Z$ . Let $\tau \in \mathrm {Irr}(\mathscr {G})$ (not necessarily supercuspidal). Then as $\mathrm {SO}(c,3) \subset \mathrm {Stab}_M(\Psi ^{\pm })$ -modules

$$ \begin{align*} \Theta(\tau)_{(\overline{N},\Psi^{\pm})}\cong\widetilde{\tau}. \end{align*} $$

and $\Theta (\tau )$ must have a nontrivial constituent.

Furthermore, if $\rho _{1}$ and $\rho _{2}$ are distinct irreducible subquotients of $\Theta (\tau )$ , then

1. 1. $(\rho _{j})_{(N,\Psi ^{+})}\cong (\rho _{j})_{(N,\Psi ^{-})}$ as $\mathrm {SO}(c,3)$ -modules, $j=1,2$ ;

2. 2. for $\epsilon \in \{\pm \}$ , $(\rho _{1})_{(N,\Psi ^{\epsilon })}$ and $(\rho _{2})_{(N,\Psi ^{\epsilon })}$ cannot both be nonzero.

Proof. The first part is a simple consequence of Corollary 3.7, and Lemma 2.8.

Suppose that $\rho _{1},\rho _{2}$ are two distinct irreducible subquotients of $\Theta (\tau )$ . Note that the characters $\Psi ^{\pm }$ are M-conjugate, because $s_{-1}(1,0,c,d)=(1,0,c,-d)$ . Thus

$$ \begin{align*} (\rho_{j})_{(\overline{N},\Psi^{+})}\cong (\rho_{j})_{(\overline{N},\Psi^{-})}. \end{align*} $$

Finally $(\rho _{1})_{(\overline {N},\Psi ^{+})}$ and $(\rho _{2})_{(\overline {N},\Psi ^{+})}$ cannot both be nonzero because this would imply that the irreducible $\mathrm {Aut}(C)$ -module $\Theta (\tau )_{(\overline {N},\Psi ^{+})}\cong \widetilde {\tau }$ has length greater than or equal to 2.

The next lemma employs two Heisenberg parabolic subgroups in G. Let $\overline {P}=M\overline {N}$ and $\overline {P}^{\prime }=M^{\prime }\overline {N}^{\prime }$ be two Heisenberg parabolic subgroups. Let $\overline {Z}\subset \overline {N}$ and $\overline {Z}^{\prime }\subset \overline {N}^{\prime }$ be the centers of the Heisenberg groups. Furthermore, suppose that $\overline {Z}$ ( $\overline {Z}^{\prime }$ ) is the root subgroup associated to the $G_{2}$ relative root $2\alpha +3\beta $ ( $\alpha +3\beta $ ).

We also use the following notation. Let $\overline {N}^{\alpha }$ be the subgroup of $\overline {N}$ generated by the root subgroups of the roots $\{2\alpha +3\beta ,\alpha +3\beta ,\alpha +2\beta ,\alpha +\beta \}$ in the $G_{2}$ relative root system. Let $\overline {N}_{\alpha +\beta }=M^{\prime }\cap \overline {N}$ , which is the root subgroup of $\alpha +\beta $ . Let $L \subset M$ be the subgroup generated by elements $hs_{\mathrm {det}(h)}^{*}$ , where $h\in H_{F}$ . For a character $\Psi $ of $\overline {N}$ , we abuse notation and continue to write $\Psi $ for its restriction to $\overline {N}^{\alpha }$ and $\overline {N}_{\alpha +\beta }$ .

Lemma 4.7. Let $\sigma $ be a smooth representation of G. Let $\Psi $ be the character of $\overline {N}/\overline {Z}$ corresponding to the element $(1,0,c,d)\in W_{F}$ , where $\mathrm {SO}(3,c)\cong \mathscr {G}$ . Then as $\mathrm {Stab}_{L}((\overline {N}^{\alpha },\Psi ))=\mathrm {Stab}_{L}((\overline {N}_{\alpha +\beta },\Psi ))\cong O(3,c)$ -modules

$$ \begin{align*} \sigma_{(\overline{N}^{\alpha},\Psi)}\cong \mathrm{FJ}^{\prime}(\sigma)_{(\overline{N}_{\alpha+\beta},\Psi)}. \end{align*} $$

Proof. We begin with some preliminaries. Let $W^{+}$ be the subgroup of $\overline {N}^{\prime }$ generated by the root subgroups of the relative roots $\{\alpha +2\beta ,\alpha +3\beta , 2\alpha +3\beta \}$ in the $G_{2}$ relative root system. We extend the character $\psi $ to $W^{+}$ so that it is trivial on the $\alpha +2\beta $ and $2\alpha +3\beta $ root spaces, and continue to call this extended character $\psi $ . Note that this is the restriction of $\Psi $ to $W^{+}$ . Now we prove the lemma.

From Weissman [Reference Weissman30, Proposition 3.2], we have $\sigma _{\overline {Z}^{\prime },\psi }\cong \mathrm {Hom}_{\overline {N}^{\prime }}(\rho _{\psi }^{\overline {N}^{\prime }},\sigma _{(\overline {Z}^{\prime },\psi )})\otimes \rho _{\psi }^{\overline {N}^{\prime }}$ as $M^{\prime }_{1}\ltimes \overline {N}^{\prime }$ -modules. (Remember, $\widetilde {M}_{1}^{\prime }$ acts genuinely on each factor.) It suffices for us to restrict the action of $M_{1}^{\prime }$ to the subgroup $L\cong \mathrm {GL}(3,F)$ .

Since $W^{+}/Z^{\prime }$ is a maximal isotropic subspace of $N^{\prime }/Z^{\prime }$ , the $(W^{+},\psi )$ -coinvariants of $\rho _{\psi }^{\overline {N}^{\prime }}$ is a one-dimensional space. Thus as $\mathrm {Stab}_{L}((W^{+},\psi ))=L$ -modules

$$ \begin{align*} (\sigma_{(\overline{Z}^{\prime},\psi)})_{(W^{+},\psi)}\cong \mathrm{Hom}_{\overline{N}^{\prime}}(\rho_{\psi}^{\overline{N}^{\prime}},\sigma_{(\overline{Z}^{\prime},\psi)})=FJ^{\prime}(\sigma). \end{align*} $$

Applying $(\overline {N}_{\alpha +\beta },\Psi )$ -coinvariants and using transitivity of coinvariants we get an isomorphism of $\mathrm {Stab}_{L}((\overline {N}^{\alpha },\Psi ))=\mathrm {Stab}_{L}((\overline {N}_{\alpha +\beta },\Psi ))\cong O(3,c)$ -modules

$$ \begin{align*} \sigma_{(\overline{N}^{\alpha},\Psi)}\cong [(\sigma_{(\overline{Z}^{\prime},\psi)})_{(W^{+},\psi)}]_{(\overline{N}_{\alpha+\beta},\Psi)}\cong FJ^{\prime}(\sigma)_{(\overline{N}_{\alpha+\beta},\Psi)}.\\[-37pt] \end{align*} $$

Proof. By Proposition 4.6, $\Theta (\tau )$ has at least one nontrivial irreducible subquotient.

By Proposition 4.4 (applied using $P^{\prime }$ ) we know that there is an $\widetilde {M}_{1}^{\prime }$ -module surjection

$$ \begin{align*} \Theta^{\dagger}(\tau^{+})\oplus\Theta^{\dagger}(\tau^{-})\twoheadrightarrow FJ^{\prime}(\Theta(\tau)). \end{align*} $$

Since we are assuming that $\tau $ is supercuspidal it follows that $\Theta ^{\dagger }(\tau ^{\pm })$ is an irreducible $\widetilde {M}_{1}^{\prime }$ -module. Thus $\mathrm {FJ}^{\prime }(\Theta (\tau ))$ is completely reducible of length at most 2.

Suppose that $\Theta (\tau )$ has two distinct irreducible subquotients $\sigma ^{+},\sigma ^{-}$ different from the trivial representation. Since $\sigma ^{\pm }$ is not trivial $\mathrm {FJ}^{\prime }(\sigma ^{\pm })\neq 0$ . Then without loss of generality we may assume that we have $\widetilde {M}_{1}^{\prime }$ -module isomorphisms $\Theta ^{\dagger }(\tau ^{\pm })\cong \mathrm {FJ}^{\prime }(\sigma ^{\pm })$ .

We take $(\overline {N}_{\alpha +\beta },\Psi )$ -coinvariants and apply Lemma 4.7 to get $O(3,c)$ -module isomorphisms $\Theta ^{\dagger }(\tau ^{\pm })_{(\overline {N}_{\alpha +\beta },\Psi )}\cong (\sigma ^{\pm })_{(\overline {N}^{\alpha },\Psi )}$ .

Now by an analog of Proposition 4.6 in the classical case, we have $\Theta ^{\dagger }(\tau ^{\pm })_{(\overline {N}_{\alpha +\beta },\Psi )}\cong \widetilde {\tau }^{\pm }$ as $\mathrm {O}(3,c)$ -modules.

The natural $\mathrm {SO}(3,c)$ -module quotient maps $(\sigma ^{\pm })_{(\overline {N}^{\alpha },\Psi )}\rightarrow (\sigma ^{\pm })_{(\overline {N},\Psi ^{\pm })}$ define an isomorphism

$$ \begin{align*} \widetilde{\tau}\cong (\sigma^{\pm})_{(\overline{N}^{\alpha},\Psi)}\rightarrow (\sigma^{\pm})_{(\overline{N},\Psi^{+})}\oplus (\sigma^{\pm})_{(\overline{N},\Psi^{-})} \end{align*} $$

of $\mathrm {SO}(3,c)$ -modules. But by Proposition 4.6, $(\sigma ^{\epsilon })_{(\overline {N},\Psi ^{+})}\oplus (\sigma ^{\epsilon })_{(\overline {N},\Psi ^{-})}=0$ for at least one $\epsilon \in \{\pm \}$ . Thus $\widetilde {\tau }=0$ , a contradiction. Therefore, $\Theta (\tau )$ must have at most one nontrivial irreducible subquotient.

Finally, we rule out the existence of trivial irreducible subquotients of $\Theta (\tau )$ .

Proof. Suppose that the trivial representation $1$ is a subquotient of $\Theta (\tau )$ , then $\tau \otimes 1_{\overline N}$ is a subquotient of $\mathcal {V}_{\overline N}$ . Observe that $1_{\overline N}$ is the trivial representation of M. By Proposition 3.11, $\mathcal {V}_{\overline N}$ has a $\mathscr {G}\times M$ -module filtration with $\mathcal {V}_{\overline {\mathcal N}}$ as a quotient. From Theorem 3.1,

$$ \begin{align*} \mathcal{V}_{\overline{\mathcal{N}}}\cong (\mathcal{V}(\mathcal{M})\otimes |\mathrm{det}|^{3/32})\oplus |\mathrm{det}|^{6/32}, \end{align*} $$

where the center of M acts trivially on $\mathcal {V}(\mathcal {M})$ . Thus the center of M acts by two nontrivial characters on the two summands of $\mathcal {V}_{\overline {\mathcal {N}}}$ hence the trivial representation of M cannot be a subquotient of $\mathcal {V}_{\overline {\mathcal {N}}}$ . The bottom part of the filtration, which appears if C is split, is a principal series representation of $\mathscr {G}$ , and hence $\tau $ cannot be a subquotient there.

Now we prove our main theorem on the theta lift of super cuspidal representations.

Proof. (1) By Propositions 4.8 and 4.9 the representation $\Theta (\tau )$ is irreducible. (2) By the assumption, and using (1), we have a surjection $\Theta (\tau _{2})\twoheadrightarrow \Theta (\tau _{1})$ . Then, by Proposition 4.6,

$$ \begin{align*} \widetilde{\tau}_{2}\cong\Theta(\tau_{2})_{(N,\Psi^{\pm})}\twoheadrightarrow\Theta(\tau_{1})_{(N,\Psi^{\pm})}\cong\widetilde{\tau}_{1}.\\[-37pt] \end{align*} $$

5.1 Untwisted Jacquet modules

Our objective in this section is to describe the $\mathscr {T}\times G$ -module $r_{\overline {\mathscr {B}}}(C^{\infty }_{c}(\omega ))$ . This is accomplished in Proposition 5.5.

For $S\subset \overline {\mathcal {N}}$ we write

$$ \begin{align*} S^{\perp}=\{x\in \mathcal{N}|\psi(\langle x,s\rangle)=1, \text{ for all }s\in S\}. \end{align*} $$

Let $\mathcal {J}^0$ be the set of trace 0 elements in $\mathcal {J}$ . Under the identification $\mathcal N \cong \mathcal {J}$ , we have $\overline {\mathscr {U}}^{\perp }\cong \mathcal {J}^0$ . Let $\omega _{0}=\omega \cap \mathcal {J}^0$ .

Lemma 5.2. The restriction map $C^{\infty }_{c}(\omega )\rightarrow C^{\infty }_{c}(\omega _{0})$ induces an isomorphism $r_{\overline {\mathscr {B}}}(C^{\infty }_{c}(\omega ))\cong C^{\infty }_{c}(\omega _{0})$ of $(\mathscr {T}\times G)$ -modules, where the action on $C^{\infty }_{c}(\omega _{0})$ is given by

$$ \begin{align*} ((t,g)\cdot f)(x)=|t|^{6-\frac{1}{2}}f(g^{-1}\cdot x t), \hspace{1cm} (t,g)\in \mathscr{T}\times G. \end{align*} $$

If $x_0\in \omega _0$ , then, as $\mathscr {T}\times G$ -modules

$$\begin{align*}C^{\infty}_{c}(\omega_{0})\cong |-|^{\frac{11}{2}} \cdot \mathrm{ind}_{\mathrm{Stab}_{\mathscr{T}\times G}(x_{0})}^{\mathscr{T}\times G} (1) \end{align*}$$

Next, we want to describe the stabilizer of a point in $\omega _0$ . The highest weight of the action of G on $\mathcal J^0\subset \mathcal {J}$ is the fundamental weight $\varpi _{4}$ taking value $1$ on the simple coroot $\alpha _{4}^{\vee }$ and $0$ on the other simple coroots. (We are using the Bourbaki labeling for simple roots [Reference Bourbaki3, Plate VIII]. In particular, $\alpha _{4}$ is the short simple root of degree 1 in the Dynkin diagram.) The next lemma follows directly from definitions.

Lemma 5.4. Let $x_{0}\in \omega _{0}$ be a highest weight vector with respect to the Borel subgroup B. Let $Q\supseteq B$ be the maximal parabolic subgroup of G obtained by removing $\alpha _{4}$ from the $F_{4}$ Dynkin diagram. This yields a parabolic subgroup of type $B_{3}$ . Then

$$\begin{align*}\mathrm{Stab}_{\mathscr{T}\times G}(x_{0})= \{ (t,q)\in \mathscr{T}\times Q ~|~t=\varpi_{4}(q) \}. \end{align*}$$

By transitivity of induction, we have the $\mathscr {T}\times G$ -module isomorphism

$$ \begin{align*} \mathrm{ind}_{\mathrm{Stab}_{\mathscr{T}\times G}(x_{0})}^{\mathscr{T}\times G}(1) \cong \mathrm{Ind}_{\mathscr{T}\times Q}^{\mathscr{T}\times G}( \mathrm{ind}_{\mathrm{Stab}_{\mathscr{T}\times G}(x_{0})}^{\mathscr{T}\times Q}(1)) \cong \mathrm{Ind}_{\mathscr{T}\times Q}^{\mathscr{T}\times G} (C^{\infty}_{c}(F^{\times})). \end{align*} $$

In order to write the last module in terms of normalized parabolic induction, we need to replace $C^{\infty }_{c}(F^{\times })$ by $\delta _{Q}^{-1/2}\cdot C^{\infty }_{c}(F^{\times })$ . Recall that $\delta _{Q}^{1/2}(q)=|\varpi _{4}(q)|^{-11/2}$ . We also need to bring back the twist by $|t|^{11/2}$ . Observe that the two exponents are inverses of each other. Hence

(5.5) $$ \begin{align} r_{\overline{\mathscr{B}}}(C^{\infty}_{c}(\omega))\cong i_{\mathscr{T}\times Q}^{\mathscr{T}\times G}(C^{\infty}_{c}(F^{\times})), \end{align} $$

where the action of $\mathscr {T}\times Q$ on $C^{\infty }_{c}(F^{\times })$ is given by

(5.6) $$ \begin{align} ((t,q)\cdot f)(x)=f(\varpi_{4}(q^{-1})xt). \end{align} $$

Putting everything together we have the following description of $r_{\overline {\mathscr {B}}}(\mathcal {V})$ .

Proposition 5.5. As a representation of $\mathscr {T}\times G$ , the module $r_{\overline {\mathscr {B}}}(\mathcal {V})$ has a filtration with successive quotients

$$\begin{align*}|-|^{5/2}\cdot \mathcal V(\mathcal M) \oplus |-|^{11/2}, \end{align*}$$

$$\begin{align*}i_{\mathscr{T}\times Q}^{\mathscr{T}\times G}(C^{\infty}_{c} (F^{\times})), \end{align*}$$

where the action of $\mathscr {T}\times Q$ on $C^{\infty }_{c}(F^{\times })$ is given by equation (5.6).

5.2 Twisted Jacquet modules

In this subsection we compute the twisted Jacquet modules of $\mathcal {V}$ with respect to $(\overline {\mathscr {U}},\psi )$ . (Recall that $\overline {\mathscr {U}}\cong F$ , so $\psi $ defines a character of $\overline {\mathscr {U}}$ .) Note that $G=\mathrm {Stab}_{\mathcal {M}}((\overline {\mathscr {U}},\psi ))$ .