Robust testing in generalized linear models by sign flipping score contributions

Summary

Generalized linear models are often misspecified because of overdispersion, heteroscedasticity and ignored nuisance variables. Existing quasi‐likelihood methods for testing in misspecified models often do not provide satisfactory type I error rate control. We provide a novel semiparametric test, based on sign flipping individual score contributions. The parameter tested is allowed to be multi‐dimensional and even high dimensional. Our test is often robust against the mentioned forms of misspecification and provides better type I error control than its competitors. When nuisance parameters are estimated, our basic test becomes conservative. We show how to take nuisance estimation into account to obtain an asymptotically exact test. Our proposed test is asymptotically equivalent to its parametric counterpart.

1 Introduction

We consider the problem of testing hypotheses about parameters in potentially misspecified generalized linear models (GLMs). The types of misspecification that we consider include overdispersion and heteroscedasticity. When the model is misspecified, traditional parametric tests tend to lose their properties, e.g. because they estimate the Fisher information under incorrect assumptions. By a parametric test we mean a test which fully relies on an assumed parametric model (Pesarin, 2015) to compute the null distribution of the test statistic.

When a parametric model to be tested is potentially misspecified, the most obvious approach is to extend the model with more parameters, e.g. to add an overdispersion parameter. However, such approaches still require assumptions, e.g. that the overdispersion is constant. Hence a fully parametric approach is not always the best option.

Another well‐known approach to testing in possibly misspecified GLMs is to use a Wald‐type test, where a sandwich estimate of the variance of the coefficient estimate is used. The sandwich estimate corrects for the potentially misspecified variance. As long as the linear predictor and link are correct, such a test is asymptotically exact under mild assumptions. We call a test asymptotically exact if its rejection probability is asymptotically known under the null hypothesis. For small samples, however, sandwich estimates often perform poorly and the test can be very liberal (Boos, 1992; Freedman, 2006; Maas and Hox, 2004; Kauermann and Carroll, 2000).

Recent decades have seen an increase in the use of permutation approaches for various testing problems (Tusher et al., 2001; Pesarin, 2001; Chung and Romano, 2013; Pauly et al., 2015; Winkler et al., 2016; Hemerik and Goeman, 2018a; Ganong and Jäger, 2018). These methods are useful since they require few parametric assumptions. Especially when multiple hypotheses are tested, permutation methods are often powerful since they can take into account the dependence structure in the data (Westfall and Young, 1993; Hemerik and Goeman, 2018b; Hemerik et al., 2019). In the past, permutation methods have already been used to test in linear models (Winkler et al. (2014) and references therein). Rather than permutations, sometimes other transformations are used, such as rotations (Solari et al., 2014) and sign flipping of residuals (Winkler et al., 2014). The existing permutation tests for GLMs, however, are limited to models with identity link function.

Like some existing methods for testing in linear models, this paper presents a sign flipping approach. Our approach is new, however, since, rather than flipping residuals, we flip individual score contributions (note that the score, the derivative of the log‐likelihood, is a sum of n individual score contributions). Moreover, we allow testing in a wide range of models, not only regression models with identity link. Under mild assumptions, the only requirement for the test to be asymptotically exact is that the individual score contributions have mean 0. Consequently, if the link function is correct, our method is often robust against several types of model specification, such as arbitrary overdispersion, heteroscedasticity and, in some cases, ignored nuisance parameters.

The main reason for this robustness is that we do not need to estimate the variance of the score: the Fisher information. Rather, we perform a permutation‐type test based on the score contributions where, rather than permutation, we use sign flipping. Intuitively, the advantage of this approach over explicitly estimating the variance is as follows: if the score contributions are independent and perfectly symmetric around zero under the null, then our test is exact for small n, even if the score contributions have misspecified variances and shapes (Pesarin and Salmaso, 2010a). A parametric test, in contrast, is then usually not exact.

In case nuisance parameters are estimated, the individual score contributions become dependent and our basic sign flipping test is no longer asymptotically exact. To deal with this problem, we consider the effective score, which is less dependent on the nuisance estimate than is the basic score (Hall and Mathiason, 1990; Marohn, 2002). In this case we need slightly more assumptions: the variance misspecification is not always allowed to depend on the covariates. The resulting test is asymptotically exact.

The methods in this paper have been implemented in the R package flipscores (Hemerik et al., 2018), which is available in the Comprehensive R Archive Network.

In Section 2 we consider the scenario that no nuisance effects need to be estimated. In Section 3 we show how the estimation of nuisance effects can be taken into account. Section 4 provides tests of hypotheses about parameters of more than one dimension. Section 5 contains simulations and Section 5 an analysis of real data.

The programs that were used to analyse the data can be obtained from

https://rss.onlinelibrary.wiley.com/hub/journal/14679868/series-b-datasets.

2 Models with known nuisance parameters

Consider random variables ν₁,…,ν_n, which satisfy assumption 1 below. These will often be individual score contributions (see Section 3, Rao (1948) or Hall and Mathiason (1990), page 86), but the results in Section 2.1 hold for any random variables satisfying this assumption.

Assumption 1.The random variables ν_i, $i\in \mathbb{N}$, are independent of each other, have finite variances and satisfy the following condition. For every ε>0,

$$\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^n\mathbb{E}({\nu }_i^2{\mathbb{1}}_{\{{\nu }_i/\surd n>ϵ\}})=0.$$

Further, as n→∞, $s_n^2:=(1/n){\mathrm{\Sigma }}_{i=1}^n\text{var}({\nu }_i)\to s^2$ for some constant s²>0.

Throughout Section 2, we consider any null hypothesis H₀ which implies that $\mathbb{E}({\nu }_i)=0$ for all 1⩽i⩽n. If ν₁,…,ν_n are score contributions and H₀ is a point hypothesis, then, under mild assumptions, $\mathbb{E}({\nu }_i)=0$ is satisfied under H₀.

A key assumption throughout Section 2 is that the ν_i, $i\in \mathbb{N}$, are independent. As soon as nuisance parameters need to be estimated, however, score contributions become dependent. Section 3 is devoted to dealing with estimated nuisance.

2.1 Basic sign flipping test

Let α ∈ [0,1). For any $a\in \mathbb{R}$, let ⌈a⌉ be the smallest integer that is larger than or equal to a and let ⌊a⌋ be the largest integer that is at most a. Given values $T_1^n,\dots ,T_w^n\in \mathbb{R}$, we let $T_{(1)}^n⩽\dots ⩽T_{(w)}^n$ be the sorted values and write $T_{[1-\alpha ]}^n=T_{(\lceil (1-\alpha )w\rceil )}^n$.

Throughout this paper, w ∈ {2,3,…} denotes the number of random sign flipping transformations to be used. Define $g_1=(1,\dots ,1)\in {\mathbb{R}}^n$ and for every 2⩽j⩽w let g_j=(g_j1,…,g_jn) be independent and uniformly distributed on {−1,1}ⁿ. Throughout the rest of Section 2, for every 1⩽j⩽w, we let

$$T_j^n=n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\nu }_i.$$

We now state that the basic sign flipping test is asymptotically exact for the point null hypothesis H₀ that implies that $\mathbb{E}({\nu }_i)=0$, 1⩽i⩽n. All proofs are in the appendices.

Theorem 1.Suppose that assumption 1 holds. Consider the test that rejects H₀ if and only if $T_1^n>T_{[1-\alpha ]}^n$. Then, as n→∞, the probability of rejection of this test converges to ⌊αw⌋/w⩽α under H₀. Moreover, the statistics $T_1^n,\dots ,T_w^n$ are asymptotically normal and independent with mean 0 and common variance ${lim}_{n\to \infty }{s}_n^2$ under H₀.

We now provide an extension of theorem 1 to interval hypotheses. The proof is a straightforward adaptation of the proof of theorem 1.

Corollary 1. ((interval hypotheses))Suppose that assumption 1 holds. Consider a null hypothesis H^′ which implies that $\mathbb{E}({\nu }_i)⩽0$ for all 1⩽i⩽n. Then, for every ε>0, there is an $N\in \mathbb{N}$ such that, under H^′, for every n>N, $\mathbb{P}(T_1^n>T_{(\lceil (1-\alpha )w\rceil )}^n)$ is at most ⌊αw⌋/w+ε. Similarly if H^′ implies that $\mathbb{E}({\nu }_i)⩾0$ for all 1⩽i⩽n, then there is an $N\in \mathbb{N}$ such that, under H^′, for every n>N, $\mathbb{P}(T_1^n<T_{(\lfloor \alpha w+1\rfloor )}^n)$ is at most ⌊αw⌋/w+ε.

The following corollary extends theorem 1 to two‐sided tests. The proof is analogous to that of theorem 1.

Corollary 2. ((two‐sided test))Suppose that assumption 1 holds. Consider ${\alpha }_1,{\alpha }_2\in \{0/w,1/w,\dots ,(w-1)/w\}$. Under H₀, as n→∞,

$$\mathbb{P}\{(T_1^n<T_{({\alpha }_{1}w+1)}^n)\cup (T_1^n>T_{((1-{\alpha }_2)w)}^n)\}\to {\alpha }_1+{\alpha }_2.$$

Note that our test does not rely on an approximate symmetry assumption (as for example that of Canay et al. (2017) does). Indeed, even if the scores are very skewed, asymptotically the test of theorem 1 is exact. However, if the ν_i are symmetric, then even for small n the size is always at most α, as noted in the following proposition. A special case of this result has already been discussed in Fisher (1935), section 21, where every element of {1,−1}ⁿ is used once.

Proposition 1.Suppose that ν₁,…,ν_n are independent and continuous and that under H₀, for each 1⩽i⩽n, ν_i=^d−ν_i. Then the size of the test of theorem 1 is at most α for any $n\in \mathbb{N}$. Moreover, if g₂,…,g_w are uniformly drawn from {1,−1}ⁿ∖{(1,…,1)} without replacement (so that only g₁ takes the value (1,…,1)), then the probability of rejection under H₀ is exactly ⌊αw⌋/w. (Note that w cannot exceed 2ⁿ then.)

If the g_j are drawn with replacement or the ν_i are discrete, then under H₀ the probability of rejection of the test of proposition 1 is (slightly) smaller than ⌊αw⌋/w for finite n, because of the possibility of ties among the test statistics $T_j^n$, 1⩽j⩽w. Otherwise the rejection probability under H₀ is ⌊αw⌋/w.

When the rejection probability under H₀ is ⌊αw⌋/w, it can be advantageous to take w such that α is a multiple of 1/w, to exhaust the nominal level.

In theorem 1, we did not assume continuity of the observations ν_i. There, under the mild assumption 1, for n→∞, $\mathbb{P}(T_j^n=T_k^n)\to 0$ for any 1⩽j<k⩽w, regardless of the distribution of the ν_i. This allows the use of theorem 1 for discrete GLMs.

2.2 Robustness

As a main example we consider the exponential family, i.e. suppose that independent variables Y₁,…,Y_n have densities of the form

$$f(y_i;{\eta }_i)=\exp \{\frac{y_i{\eta }_i-b({\eta }_i)}{a_i}+c(y_i)\},$$

where ${\eta }_i=x_i\beta +{\mathbf{z}}_i\mathit{\gamma }$, $x_i,\beta \in \mathbb{R}$, ${\mathbf{z}}_i,\mathit{\gamma }\in {\mathbb{R}}^m$ for some $m\in \mathbb{N}$. Here β is the coefficient of interest and at present we assume that the other coefficients γ are known. The canonical link function g satisfies η_i=g(μ_i), where $g^{-1}({\eta }_i)={\mu }_i=\mathbb{E}(y_i)=b^{\prime }({\eta }_i)$ and a_i=var(y_i)/b^′′(η_i) (Agresti, 2015). For $H_0:\beta ={\beta }_0$, the score ${\mathrm{\Sigma }}_{i=1}^n{\nu }_i={\mathrm{\Sigma }}_{i=1}^n\partial log\{f(y_i;{\eta }_i)\}{/\partial \beta |}_{\beta ={\beta }_0}$ is

$$\sum _{i=1}^n\frac{x_i\{y_i-b^{\prime }({\eta }_i)\}}{a_i}{|}_{\beta ={\beta }_0}=\sum _{i=1}^n\frac{x_i\{y_i-\mathbb{E}(y_i)\}}{a_i}{|}_{\beta ={\beta }_0}.$$

For example, the Poisson model has g‐log‐link function, b(η_i)= exp (η_i), a_i=1 and c(y_i)=− log (y_i!). Hence $\mathbb{E}(y_i)=b^{\prime }({\eta }_i)=\exp ({\eta }_i)$. Thus the score function is

$$\sum _{i=1}^n{x}_i(y_i-{\mu }_i){|}_{\beta ={\beta }_0}=\sum _{i=1}^n{x}_i\{y_i-\exp (x_i{\beta }_0+{\mathbf{z}}_i\mathit{\gamma })\}.$$

For the normal distribution, a_i=σ², so the score is

$$\sum _{i=1}^n\frac{x_i(y_i-{\eta }_i)}{{\sigma }^2}{|}_{\beta ={\beta }_0}.$$

Apart from some mild assumptions, the main assumption that is made in theorem 1 is that $\mathbb{E}({\nu }_i)=0$, i=1,…,n. This is satisfied as soon as ${\mu }_i{|}_{\beta ={\beta }_0}$ is the true expected value of Y_i. Then the test is asymptotically exact even if the a_i are misspecified, i.e. if the variance or distributional shape of Y_i is misspecified. The a_i are even allowed to be misspecified by a factor which depends on the covariates, as long as assumption 1 holds.

As a concrete example, consider the normal model with identity link function, which assumes that var(Y₁)=…=var(Y_n). If the real distribution is heteroscedastic, then the test will still be exact for finite n, since the ν_i are symmetric. The parametric test, however, loses its properties, e.g. because the estimated variance does not have the assumed χ²‐distribution. In Section 5 it is illustrated that our approach can be much more robust against heteroscedasticity than a parametric test.

Another example is the situation where the model is Poisson, i.e. var(Y_i)=μ_i is assumed, but in reality var(Y_i)>μ_i, which is a form of overdispersion which occurs very often in practice. Then the parametric score test underestimates the Fisher information and is anticonservative. To take the overdispersion into account it could be explicitly estimated. However, if the overdispersion factor is not constant, but depends on the covariates, then again the parametric test loses its properties. Theorem 1, however, often still applies, so an asymptotically exact test is obtained.

Further, note that if $\mathbb{E}(Y_i)$ depends on a nuisance variable $Z_i^l$ which is latent and ignored, where $Z_i^l$ is independent of X_i, then the test may still be valid. The reason is that, marginally over $Z_i^l$, $\mathbb{E}(Y_i)$ may still be computed correctly (see, for example, Section 5.2). Such latent nuisance variables will increase the variance of Y_i, however, which can pose a problem for the classical parametric score test, which needs to compute the Fisher information. When the latent variable is not independent of X_i, this usually does pose a problem for our test (even as n→∞), since $\mathbb{E}(Y_i-{\mu }_i)$ becomes dependent on X_i under H₀.

3 Taking into account nuisance estimation

Consider independent and identically distributed (IID) pairs $({\mathbf{X}}_i,Y_i)$, i=1,…,n, where ${\mathbf{X}}_i$ is some covariate vector and $Y_i\in \mathbb{R}$ has distribution ${\mathbb{P}}_{\beta ,{\mathit{\gamma }}_0,{\mathbf{X}}_i}$, which depends on the parameter of interest $\beta \in \mathbb{R}$ and unknown nuisance parameter ${\mathit{\gamma }}_0$, which lies in a set $\mathbb{G}\subseteq {\mathbb{R}}^{k-1}$, where k is the total number of modelled parameters. We shall discuss the issues arising from estimating ${\mathit{\gamma }}_0$ and propose a solution, which enables us to obtain an asymptotically exact test based on score flipping. In this paper, the above model is the model that is considered by the user. It is the model that is used to compute the scores. We consider this model to be correct, unless explicitly stated otherwise, e.g. in Section 3.2. The parameter ${\mathit{\gamma }}_0$ is part of the model that is considered by the user, so it is always modelled and estimated. For example, ${\mathit{\gamma }}_0$ never represents ignored overdispersion.

We consider the null hypothesis $H_0:\beta ={\beta }_0\in \mathbb{R}$. Generalizations to interval hypotheses and two‐sided tests can be obtained as in corollaries 1 and 2. The case that the parameter of interest is multi‐dimensional is considered in Section 4.

Suppose that, for all $\mathit{\gamma }\in \mathbb{G}$, ${\mathbb{P}}_{\beta ,\mathit{\gamma },{\mathbf{X}}_i}$ has a density $f_{\beta ,\mathit{\gamma },{\mathbf{X}}_i}$ around ${\beta }_0$ with respect to some dominating measure. For 1⩽i⩽n write

$${\nu }_{\mathit{\gamma },i}=\frac{\partial }{\partial \beta }\log \{f_{\beta ,\mathit{\gamma },{\mathbf{X}}_i}(Y_i){|}_{\beta ={\beta }_0}\},$$

where we assume that the derivative exists. The value ν_i is the score for the ith observation. Under H₀, $\mathbb{E}({\nu }_{{\mathit{\gamma }}_0,i})=0$, i=1,…,n. The score for all n observations simultaneously is $n^{1/2}{S}_{\mathit{\gamma }}$, where

$$S_{\mathit{\gamma }}=n^{-1/2}\sum _{i=1}^n{\nu }_{\mathit{\gamma },i}.$$

Assume that $\widehat{\mathit{\gamma }}$ is a √n‐consistent estimate of ${\mathit{\gamma }}_0$, taking values in $\mathbb{G}$. For every 1⩽i⩽n, let

$${\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-1)}=\frac{\partial }{\partial \mathit{\gamma }}\log \{f_{{\beta }_0,\mathit{\gamma },{\mathbf{X}}_i}(Y_i)\}{|}_{\mathit{\gamma }=\widehat{\mathit{\gamma }}}\in {\mathbb{R}}^{k-1}$$

denote the (k−1)‐vector of score contributions for the nuisance parameters, which is assumed to exist. Let

$${\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}=n^{-1/2}\sum _{i=1}^n{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-1)}\in {\mathbb{R}}^{k-1}$$

be the vector of nuisance scores.

For 1⩽j⩽w, let the superscript j denote that g_j has been applied:

$$\begin{array}{cc}& S_{\widehat{\mathit{\gamma }}}^j=n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\nu }_{\widehat{\mathit{\gamma }},i},\hfill \\ & {\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1),j}=n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-1)}.\hfill \end{array}$$

3.1 Asymptotically exact test

When the nuisance parameter ${\mathit{\gamma }}_0$ is unknown, it needs to be estimated, which is typically done by maximizing the likelihood of the data under the null hypothesis. The distribution of $S_{\widehat{\mathit{\gamma }}}$ can be substantially different from that of $S_{{\mathit{\gamma }}_0}$: the score based on the true nuisance parameters. Indeed, under the null hypothesis, the asymptotic variance of $S_{\widehat{\mathit{\gamma }}}$ is not the Fisher information, but the effective Fisher information (Rippon and Rayner (2010), Rayner (1997), Hall and Mathiason (1990), Marohn (2002) and Cox and Hinkley (1979), section 9.3), which is also the asymptotic variance of the effective score, which is defined below. The effective information is smaller than the information, given that the score for the parameter of interest and the nuisance score are correlated. Intuitively, the reason is that the nuisance variable will be used to explain part of the apparent effect of the variable of interest, also asymptotically.

The estimation of ${\mathit{\gamma }}_0$ makes the summands ${\nu }_{\widehat{\mathit{\gamma }},1},\dots ,{\nu }_{\widehat{\mathit{\gamma }},n}$ underlying $S_{\widehat{\mathit{\gamma }}}$ correlated, in such a way that $\text{var}(S_{\widehat{\mathit{\gamma }}})<\text{var}(S_{{\mathit{\gamma }}_0})$ (if the score is correlated with the nuisance score). Note, however, that, after random flipping, the summands are not correlated anymore. This means that the variance of $S_{\widehat{\mathit{\gamma }}}$ is asymptotically smaller than the variance of $S_{\widehat{\mathit{\gamma }}}^j$, 2⩽j⩽w (see the proof of theorem 2 in Appendix B.3). Hence, using ${\nu }_{\widehat{\mathit{\gamma }},1},\dots ,{\nu }_{\widehat{\mathit{\gamma }},n}$ in the test of theorem 1 can lead to a conservative test, even as n→∞.

To make the test asymptotically exact again, we would like to adapt the individual scores such that they are less dependent on the random variation of $\widehat{\mathit{\gamma }}$. We do this by considering the so‐called effective score, which ‘is “less dependent” on the nuisance parameter than the usual score statistic’ (Marohn (2002), page 344).

The effective score $S_{\widehat{\mathit{\gamma }}}^{*}$ and the underlying summands ${\nu }_{\widehat{\mathit{\gamma }},i}^{*}$, i=1,…,n (which we assume have non‐zero variance for $\widehat{\mathit{\gamma }}={\mathit{\gamma }}_0$), are defined as

$$\begin{array}{cc}& S_{\widehat{\mathit{\gamma }}}^{*}=S_{\widehat{\mathit{\gamma }}}-{\widehat{\mathcal{I}}}_{12}^{\prime }{\widehat{\mathcal{I}}}_{22}^{-1}{\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)},\hfill \\ & {\nu }_{\widehat{\mathit{\gamma }},i}^{*}={\nu }_{\widehat{\mathit{\gamma }},i}-{\widehat{\mathcal{I}}}_{12}^{\prime }{\widehat{\mathcal{I}}}_{22}^{-1}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-1)},\hfill \end{array}$$

so that

$$S_{\widehat{\mathit{\gamma }}}^{*}=n^{-1/2}\sum _{i=1}^n{\nu }_{\widehat{\mathit{\gamma }},i}^{*}.$$

Here

$$\widehat{\mathcal{I}}=\left(\begin{array}{cc}{\widehat{\mathcal{I}}}_{11}& {\widehat{\mathcal{I}}}_{12}^{\prime }\\ {\widehat{\mathcal{I}}}_{12}& {\widehat{\mathcal{I}}}_{22}\end{array}\right),$$

with ${\widehat{\mathcal{I}}}_{11}\in \mathbb{R}$ and the (k−1)×(k−1) matrix ${\widehat{\mathcal{I}}}_{22}$, assumed invertible, is a consistent estimate of the population Fisher information $\mathcal{I}$, which is assumed to exist and is the variance of ${({\nu }_{{\mathit{\gamma }}_0,i},{{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}^{(k-1)}}^{\prime })}^{\prime }$ marginally over ${\mathbf{X}}_i$, under H₀. The matrix $\mathcal{I}$ is assumed to be continuous in the parameters. In GLMs, typically $\widehat{\mathcal{I}}=n^{-1}{\mathbf{X}}^{\prime }\widehat{\mathbf{W}}\mathbf{X}$, where X is the design matrix and $\widehat{\mathbf{W}}$ the estimated weight matrix (Agresti (2015), page 126). Further, for 1⩽j⩽w we write

$$S_{\widehat{\mathit{\gamma }}}^{*j}=S_{\widehat{\mathit{\gamma }}}^j-{\widehat{\mathcal{I}}}_{12}^{\prime }{\widehat{\mathcal{I}}}_{22}^{-1}{\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1),j}.$$

As discussed, $S_{\widehat{\mathit{\gamma }}}$ is not generally asymptotically equivalent to $S_{{\mathit{\gamma }}_0}$. The effective score $S_{{\mathit{\gamma }}_0}^{*}$ (based on $\widehat{\mathcal{I}}=\mathcal{I}$), however, is the residual from the projection of the score $S_{{\mathit{\gamma }}_0}$ on the space spanned by the nuisance scores. Hence $S_{{\mathit{\gamma }}_0}^{*}$ is uncorrelated with the nuisance scores ${\mathbf{S}}_{{\mathit{\gamma }}_0}^{(k-1)}$ (Marohn (2002), page 344). Correspondingly, as noted in the proof of theorem 2, under mild regularity assumptions $S_{\widehat{\mathit{\gamma }}}^{*}=S_{{\mathit{\gamma }}_0}^{*}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)$, i.e. asymptotically the effective score is not affected by the nuisance estimate.

If $\widehat{\mathit{\gamma }}$ is the maximum likelihood estimate under H₀, then ${\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}=\mathbf{0}$, so $S_{\widehat{\mathit{\gamma }}}^{*}=S_{\widehat{\mathit{\gamma }}}$. The summands ${\nu }_{\widehat{\mathit{\gamma }},i}^{*}$ and ${\nu }_{\widehat{\mathit{\gamma }},i}$ are different, however, and the key point is that $S_{\widehat{\mathit{\gamma }}}^{*}=S_{{\mathit{\gamma }}_0}^{*}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)$.

Like Marohn (2002), we assume that, if $\xi \in \mathbb{R}$ and $\beta ={\beta }^n={\beta }_0+n^{-1/2}\xi$, then

$$\begin{array}{cc}\hfill S_{\widehat{\mathit{\gamma }}}& =S_{{\mathit{\gamma }}_0}-{\mathcal{I}}_{12}^{\prime }\surd n(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)+o_{{\mathbb{P}}_{{\beta }^n,{\mathit{\gamma }}_0}}(1),\hfill \\ \hfill {\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}& ={\mathbf{S}}_{{\mathit{\gamma }}_0}^{(k-1)}-{\mathcal{I}}_{22}\surd n(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)+o_{{\mathbb{P}}_{{\beta }^n,{\mathit{\gamma }}_0}}(1),\hfill \end{array}$$

which is satisfied under mild assumptions such as continuous second‐order derivatives.

Theorem 2.Consider the test of theorem 1 with $T_j^n=S_{\widehat{\mathit{\gamma }}}^{*j}$, 1⩽j⩽w. As n→∞, under H₀ the probability of rejection converges to ⌊αw⌋/w⩽α.

The test of theorem 2 has a parametric counterpart, which uses that, under H₀, $S_{\widehat{\mathit{\gamma }}}^{*}$ is asymptotically normal with zero mean and known variance: the effective information (Marohn (2002), page 341). This test is asymptotically equivalent to the test of theorem 2, as the following proposition says.

Proposition 2.Let $\xi \in \mathbb{R}$ and suppose that the true parameter satisfies $\beta ={\beta }^n={\beta }_0+n^{-1/2}\xi$. As in theorem 2, let $T_j^n=S_{\widehat{\mathit{\gamma }}}^{*j}$, 1⩽j⩽w. Define ${\varphi }_{n,w}={\mathbb{1}}_{\{T_1^n>T_{[1-\alpha ]}^n\}}$ to be the test of theorem 2. Let ${\varphi }_n^{\prime }$ be the parametric test ${\mathbb{1}}_{\{T_1^n>{\sigma }_0\mathrm{\Phi }(1-\alpha )\}}$, where ${\sigma }_0^2\in \mathbb{R}$ is the effective Fisher information and Φ the cumulative distribution function of the standard normal distribution. Then ${lim}_{w\to \infty }\text{lim}{\text{inf}}_{n\to \infty }\mathbb{P}({\varphi }_{n,w}={\varphi }_n^{\prime })=1.$

3.2 Robustness

In Section 2.2 it was explained that the test of theorem 1 is often robust against misspecification of the variance of the score. The test of theorem 2 is also robust against certain forms of variance misspecification. An example is the case that $S_{\widehat{\mathit{\gamma }}}$ and ${\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}$ are misspecified by the same factor; see proposition 3. This happens in particular if the variance is misspecified by a factor which is independent of the covariates.

Proposition 3.Suppose that $\widehat{\mathcal{I}}=n^{-1}{\mathbf{X}}^{\prime }\widehat{\mathbf{W}}\mathbf{X}$, where X is an n×k design matrix with IID rows and $\widehat{\mathbf{W}}$ a weight matrix. Consider a misspecification factor c₁>0 and misspecified scores

$${\tilde{\nu }}_{\widehat{\mathit{\gamma }},i}=c_1{\nu }_{\widehat{\mathit{\gamma }},i},{\tilde{\mathit{\nu }}}_{\widehat{\mathit{\gamma }},i}^{(k-1)}=c_1{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-1)},i=1,\dots ,n.$$

Further, for c₂>0 consider the misspecified weight matrix $\tilde{\mathbf{W}}=c_2\widehat{\mathbf{W}}$. Let $\tilde{\mathcal{I}}=n^{-1}{\mathbf{X}}^{\prime }\tilde{\mathbf{W}}\mathbf{X}$ be the misspecified average Fisher information. Let ${\tilde{\nu }}_{\widehat{\mathit{\gamma }},i}^{*}={\tilde{\nu }}_{\widehat{\mathit{\gamma }},i}-{\tilde{\mathcal{I}}}_{12}^{\prime }{\tilde{\mathcal{I}}}_{22}^{-1}{\tilde{\mathit{\nu }}}_{\widehat{\mathit{\gamma }},i}^{(k-1)}$ be the misspecified effective scores, i=1,…,n. Consider the test of theorem 2, with $S_{\widehat{\mathit{\gamma }}}^{*j}$, j=1,…,w, replaced by the misspecified effective score

$${\tilde{S}}_{\widehat{\mathit{\gamma }}}^{*j}=n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\tilde{\nu }}_{\widehat{\mathit{\gamma }},i}^{*}.$$

Under H₀, as n→∞, the probability of rejection of this test converges to ⌊αw⌋/w⩽α.

Proposition 1 is useful, since it tells us that if in a GLM var(Y_i) is misspecified by a constant, such that $\widehat{\mathbf{W}}$ and the scores are misspecified by a constant, the resulting test is still asymptotically exact. In proposition 3 we assume that the misspecification factors of the weights and the scores are the same for all observations. This is satisfied for example when the model is binomial or Poisson, but the true distribution is respectively quasi‐binomial or quasi‐Poisson. Moreover, in practice the test can be very robust against heteroscedasticity (see Section 5). The variance misspecification is not generally allowed to depend on the covariates, since then $S_{\widehat{\mathit{\gamma }}}$ and ${\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}$ can be misspecified by different factors asymptotically. There are exceptions, however; see Sections 5 and 5.

When there are estimated nuisance parameters, we can sometimes nevertheless decide to use the test of theorem 1 with the basic scores ${\nu }_{\widehat{\mathit{\gamma }},i}$ plugged in (rather than using effective scores). Indeed, this test has been shown to be very robust to misspecification, as long as $\mathbb{E}({\nu }_{\widehat{\mathit{\gamma }},i})=0$, 1⩽i⩽n. It is asymptotically conservative if the score $S_{{\mathit{\gamma }}_0}$ is correlated with the nuisance scores ${\mathbf{S}}_{{\mathit{\gamma }}_0}^{(k-1)}$, i.e. when ${\mathcal{I}}_{12}\ne 0$. Hence, when using this test, it can be useful to redefine the covariates such that ${\mathcal{I}}_{12}=0$ (as in Cox and Reid (1987)). When $\widehat{\mathbf{W}}=b\mathbf{I}$, b>0, this means ensuring that the nuisance covariates are orthogonal to the covariate of interest. If the model is potentially misspecified, then the weights and hence ${\mathcal{I}}_{12}$ are not asymptotically known, but the user could substitute a best guess for the weights.

3.3 An example

As discussed, the test of theorem 2 is not generally asymptotically exact if the variance misspecification depends on the covariates. An important exception is the case where the model is

$$Y_i\sim N({\gamma }_0+\beta X_i,{\sigma }^2)i=1,\dots ,n,$$(1)

where γ₀ is the unknown intercept and $X_i\in \mathbb{R}$ . If the null hypothesis is $H_0:\beta ={\beta }_0$, then γ₀ is a nuisance parameter that needs to be estimated. (We do not need to know σ and can simply substitute 1 for it.) Hence, we compute the effective score. For 1⩽i⩽n,

$$\begin{array}{cc}& {\nu }_{\widehat{\gamma },i}=x_i(y_i-{\widehat{\mu }}_i)/{\sigma }^2,\hfill \\ & {\mathit{\nu }}_{\widehat{\gamma },i}^{(k-1)}=(y_i-{\widehat{\mu }}_i)/{\sigma }^2.\hfill \end{array}$$

We can consistently estimate ${\mathcal{I}}_{12}{\mathcal{I}}_{22}^{-1}$ by $\overline{x}=(1/n){\mathrm{\Sigma }}_{i=1}^n{x}_i$, so that the effective score contributions are

$${\nu }_{\widehat{\gamma },i}^{*}=(x_i-\overline{x})(y_i-{\widehat{\mu }}_i)/{\sigma }^2.$$

Thus, the effective score contributions are exactly the basic score contributions after centring x₁,…,x_n at 0. Similarly, if x₁,…,x_n are already centred, then ${\nu }_{\widehat{\gamma },i}$ and ${\nu }_{\widehat{\gamma },i}^{*}$ coincide, since then ${\widehat{\mathcal{I}}}_{12}=0$.

The test of theorem 2 is not always asymptotically exact if $S_{\widehat{\gamma }}$ and ${\mathbf{S}}_{\widehat{\gamma }}^{(k-1)}$ are misspecified by different factors. However, if ${\widehat{\mathcal{I}}}_{12}=0$, then this does not apply anymore. The test of theorem 2 then remains asymptotically exact and reduces to the test based on the basic score. For model (1), this means that, even if the misspecification of var(Y_i) depends on X_i, we obtain an asymptotically exact test.

A particular case where this principle applies is the generalized Behrens–Fisher problem, where the aim is to test equality of the means μ¹ and μ² of two populations (or to test whether μ¹⩽μ² or μ¹⩾μ²). In this problem, it is assumed only that two independent samples from these populations are available, without making other assumptions such as equal variances. It is well known that this problem has no exact solution with good power under normality (Pesarin and Salmaso, 2010a; Lehmann and Romano, 2005). Under mild assumptions, we obtain an asymptotically exact test for this problem. Pesarin and Salmaso (2010a) have already suggested sign flipping residuals to solve this problem. This is equivalent to flipping scores in our linear model (1) if $|x_1|=\dots =|x_n|$.

4 Multi‐dimensional parameter of interest

Until now we have considered hypotheses about a one‐dimensional parameter $\beta \in \mathbb{R}$. Here we extend our results to hypotheses about a multi‐dimensional parameter $\mathit{\beta }\in {\mathbb{R}}^d$, $d\in \mathbb{N}$. Our tests are defined even if d>n, but in the theoretical results that follow we consider d fixed and let n increase to ∞. The extension to multi‐dimensional β shares important characteristics with the test for a one‐dimensional parameter, such as robustness and asymptotic equivalence with the parametric score test.

4.1 Asymptotically exact test

Our tests below are related to the existing non‐parametric combination methodology (Pesarin, 2001; Pesarin and Salmaso, 2010a,b). This is a very general permutation‐based methodology that allows combining test statistics for many hypotheses into a single test of the intersection hypothesis. Non‐parametric combination methods can be extended to the score flipping framework. Our tests below could be considered a special case of such an extension of the non‐parametric combination methodology. This special case has certain power optimality properties, which are discussed below.

The parametric score test has a classical extension to a hypothesis on a multi‐dimensional parameter, $H_0:\mathit{\beta }={\mathit{\beta }}_0\in {\mathbb{R}}^d$ (Rao, 1948). We shall extend our test in an analogous way. We first assume that the nuisance ${\mathit{\gamma }}_0\in {\mathbb{R}}^{k-d}$ is known. Since $\mathit{\beta }\in {\mathbb{R}}^d$, the score is ${\mathbf{S}}_{{\mathit{\gamma }}_0}=n^{-1/2}{\mathrm{\Sigma }}_{i=1}^n{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}\in {\mathbb{R}}^d$, where

$${\mathit{\nu }}_{{\mathit{\gamma }}_0,i}=\frac{\partial }{\partial \mathit{\beta }}\log \{f_{\mathit{\beta },{\mathit{\gamma }}_0,{\mathbf{X}}_i}(Y_i)\}{|}_{\mathit{\beta }={\mathit{\beta }}_0}\in {\mathbb{R}}^d,$$

1⩽i⩽n, which are now d‐vectors. We assume that the derivatives exist. About the elements of ${\mathit{\nu }}_{\mathit{\gamma },i}$ (and the nuisance scores that are considered later) we make the assumptions which are analogous to the earlier assumptions about ${\nu }_{\mathit{\gamma },i}$.

Let ${\widehat{\mathcal{I}}}_{11}$ be a consistent estimate of ${\mathcal{I}}_{11}$: the d×d Fisher information for $\mathit{\beta }\in {\mathbb{R}}^d$. Rao's classical statistic for testing $H_0:\mathit{\beta }={\mathit{\beta }}_0\in {\mathbb{R}}^d$ is

$${\mathbf{S}}_{{\mathit{\gamma }}_0}^{\prime }{\widehat{\mathcal{I}}}_{11}^{-1}{\mathbf{S}}_{{\mathit{\gamma }}_0}=\left(n^{-1/2}\sum _{i=1}^n{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}^{\prime }\right){\widehat{\mathcal{I}}}_{11}^{-1}\left(n^{-1/2}\sum _{i=1}^n{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}\right),$$

which asymptotically has a ${\chi }_d^2$‐distribution under H₀.

Instead of requiring a matrix ${\widehat{\mathcal{I}}}^{-1}$ which converges to the inverse of the Fisher information, in our test that follows we allow replacement of the Fisher information by any random matrix $\widehat{\mathbf{V}}$ converging to some non‐zero matrix V, i.e. we do not require the Fisher information to be asymptotically known, just like in the one‐dimensional case. The matrix V can be any matrix of preference, including ${\mathcal{I}}_{11}^{-1}$ (if ${\mathcal{I}}_{11}$ is invertible), or we can take $\widehat{\mathbf{V}}=\mathbf{V}=\mathbf{I}$. We shall discuss various choices of V shortly.

Theorem 3.The result of theorem 1 still applies if for 1⩽j⩽w we define

$$T_j^n=\left(n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}^{\prime }\right)\widehat{\mathbf{V}}\left(n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}\right).$$

In the case that the nuisance parameter ${\mathit{\gamma }}_0$ is unknown and we have a √n‐consistent estimate $\widehat{\mathit{\gamma }}$, we can use the same test, but with effective scores instead of basic scores plugged in. See theorem 4. For multi‐dimensional β, the effective score contributions are

$${\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{*}={\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}-{\widehat{\mathcal{I}}}_{12}^{\prime }{\widehat{\mathcal{I}}}_{22}^{-1}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-d)}\in {\mathbb{R}}^d,$$

1⩽i⩽n, where

$${\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{(k-d)}=\frac{\partial }{\partial \mathit{\gamma }}\log \{f_{{\mathit{\beta }}_0,\mathit{\gamma },{\mathbf{X}}_i}(Y_i)\}{|}_{\mathit{\gamma }=\widehat{\mathit{\gamma }}}\in {\mathbb{R}}^{k-d}.$$

Here ${\widehat{\mathcal{I}}}_{12}$ and ${\widehat{\mathcal{I}}}_{22}$ are (k−d)×d and (k−d)×(k−d) matrices respectively.

Theorem 4. ((unknown nuisance))The result of theorem 1 still applies if for 1⩽j⩽w we define

$$T_j^n=\left(n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{{*}^{\prime }}\right)\widehat{\mathbf{V}}\left(n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{*}\right).$$

The test of theorem 4 is asymptotically equivalent to its parametric counterpart, as proposition 4 states. In particular, if we take $\widehat{\mathbf{V}}={({\widehat{\mathcal{I}}}^{*})}^{-1}$, where ${({\widehat{\mathcal{I}}}^{*})}^{-1}$ is a consistent estimate of the inverse of the effective Fisher information, then the test of theorem 4 is asymptotically equivalent to the parametric score test (Hall and Mathiason (1990), page 86).

Proposition 4. ((equivalence with parametric counterpart))Define $T_j^n$ as in theorem 4, 1⩽j⩽w. Let $\mathit{\xi }\in {\mathbb{R}}^d$ and suppose that the true value of the parameter of interest is $\mathit{\beta }={\mathit{\beta }}^n={\mathit{\beta }}_0+n^{-1/2}\mathit{\xi }$. Let ${\varphi }_{n,w}={\mathbb{1}}_{\{T_1^n>T_{[1-\alpha ]}^n\}}$. This is the test of theorem 4. Let ${\varphi }_n^{\prime }$ be the parametric test ${\mathbb{1}}_{\{T_1^n>q_{\alpha }\}}$, where q_α is the (1−α)‐quantile of the distribution to which $T_1^n$ converges as n→∞ under $\mathit{\beta }={\mathit{\beta }}_0$. (This is the ${\chi }_d^2$‐distribution if V is the inverse of the effective information matrix ${\mathcal{I}}^{*}$). Then ${lim}_{w\to \infty }\text{lim}{\text{inf}}_{n\to \infty }\mathbb{P}({\varphi }_{n,w}={\varphi }_n^{\prime })=1.$

We have seen that the test of theorem 1 is often robust against overdispersion and heteroscedasticity: as long as the score contributions have mean 0, the test is asymptotically exact, under very mild assumptions. Moreover, it is not required to estimate the Fisher information. The same applies to the multi‐dimensional extension in theorem 3.

The test that takes into account nuisance estimation (theorem 4) uses effective scores, so it does need to estimate the information. However, as in the one‐dimensional case, it can be seen that the test remains valid if the information matrix is asymptotically misspecified by a constant (as in proposition 3). Additional robustness is illustrated with simulations in Section 5.5.

4.2 Connection with the global test

The test of theorem 3 is related to the global test, which was developed in Goeman et al. (2004, 2006, 2011). We can combine the global test with the score flipping approach. In certain cases, the resulting test coincides with the test of theorem 3.

The global test is a parametric test of H₀. For the test to be defined, it is not required that d⩽n. For GLMs with canonical link function, the test statistic of the global test is

$${\mathbf{S}}_{{\mathit{\gamma }}_0}^{\prime }\mathbf{\Sigma }{\mathbf{S}}_{{\mathit{\gamma }}_0},$$(2)

with Σ a freely chosen positive (semi)definite d×d matrix (Goeman et al., 2006, 2011). The choice of Σ influences the power properties.

When $\widehat{\mathbf{V}}=\mathbf{\Sigma }$, statistic (2) coincides with the statistic of theorem 3. Thus, it immediately follows from our results that the global test can be combined with our sign flipping approach, leading to a test which becomes asymptotically exact as n→∞ and asymptotically equivalent to its parametric counterpart: the original global test (by proposition 4). Combining the global test with sign flipping is useful in the light of our robustness results. Moreover, the sign flipping variant can be combined with existing permutation‐based multiple‐testing methodology (Westfall and Young, 1993; Hemerik and Goeman, 2018b; Hemerik et al., 2019).

Goeman et al. (2006) provided results on the power properties of the global test as depending on the choice of Σ. Since the global test is asymptotically equivalent to its sign flipping counterpart, these results can be used as recommendations on the choice of $\widehat{\mathbf{V}}$ in theorem 3. In particular, according to Goeman et al. (2006), section 8, taking $\widehat{\mathbf{V}}=\mathbf{I}$ leads to good power if we expect that relatively much of the variance of Y is explained by the large variance principal components of the design matrix. If this is not so, taking $\widehat{\mathbf{V}}$ to be an estimate of the inverse of the Fisher information (if invertible) can provide better power. In general, the global test has optimal power on average (over β) in a neighbourhood of ${\mathit{\beta }}_0$ that depends on Σ (Goeman et al., 2006). Hence the same holds asymptotically for the test of theorem 3, for GLMs with canonical link.

5 Simulations

To compare the tests in this paper with each other and existing tests, we applied them to simulated data. In particular we considered scenarios where the model was misspecified. Simulations with a multi‐dimensional parameter of interest are in Section 5.5.

5.1 Overdispersion, heteroscedasticity and estimated nuisance

In Sections 5.1 and 5.2 the model assumed was Poisson, but in fact Y₁,…,Y_n were drawn from a negative binomial distribution.

The covariates $X,Z,Z^l\in \mathbb{R}$ were drawn from a multivariate normal distribution with zero mean and var(X)=var(Z)=var(Z^l)=1. (For non‐zero means, similar simulation results were obtained as below.) The response satisfied

$$\log \{\mathbb{E}(Y_i)\}=\log ({\mu }_i)={\eta }_i=0+\beta X_i+{\gamma }_0{Z}_i+{\gamma }_0^l{Z}_i^l.$$

The null hypothesis was $H_0:\beta =0$. In Section 5.1 we took ${\gamma }_0^l=0$. The coefficient γ₀ and the intercept 0 were nuisance parameters that were estimated by maximum likelihood under H₀. We took γ₀=1 and ρ(X_i,Z_i)=0.5, $\rho (Z_i^l,Z_i)=0$ and $\rho (Z_i^l,X_i)=0$. We took the dispersion parameter of the negative binomial distribution to be 1, so that $\text{var}(Y_i)={\mu }_i+{\mu }_i^2$.

The model assumed, however, was Poisson, i.e. var(Y_i)=μ_i was assumed. Thus the true variance was larger than the assumed variance and the variance misspecification factor depended on μ_i, i.e. on the covariate Z_i. The assumed log‐link function was correct and in Section 5.1 the linear predictor was correct as well.

In Fig. 1 the estimated rejection probabilites of four tests under $H_0:\beta =0$ are compared, based on 5000 repeated simulations. In all simulations the tests were two sided.

Figure 1

Estimated rejection probabilities for four tests under misspecified variance and estimated nuisance (the null hypothesis was $H_0:\beta =0$) (, parametric; , sandwich; , flip basic; , flip effective): (a) β=0, n=50; (b) β=0, n=200

One of the tests that was considered was the parametric score test. Since the model assumed was Poisson, the computed Fisher information was too small and the test was anticonservative.

We also applied a Wald test, where we used a sandwich estimate (Agresti (2015), page 280) of the variance of $\widehat{\beta }$, to correct for the misspecified variance function. We used the R package gee (Carey et al., 2019) for this (available on the Comprehensive R Archive Network), specifying blocks of size 1. As can be seen in Fig. 1, this test was quite anticonservative (especially for small α, e.g. α=0.01). This was in particular due to the estimation error of the sandwich (Boos, 1992; Freedman, 2006; Maas and Hox, 2004; Kauermann and Carroll, 2000).

Further, we applied the sign flipping test based on the basic scores ${\nu }_{\widehat{\mathit{\gamma }},i}$. Because of the estimation of γ₀, the variance of the score was shrunk and the test was conservative, as explained in Section 3.1. In the simulations under H₀ we took w=200. Taking w larger led to a very similar level (see also Marriott (1979)). In the power simulations we took w=1000.

Finally, we used the sign flipping test of theorem 2, which is based on the effective scores ${\nu }_{\widehat{\mathit{\gamma }},i}^{*}$. In Section 3.2 it has already been shown that this test is asymptotically exact under constant variance misspecification. Here, however, the variance misspecification factor was 1+μ_i (i.e. it depended on Z_i). Nevertheless the rejection probability under H₀ was approximately α. This illustrates that the test has some additional robustness, which we have not theoretically shown.

5.2 Ignored nuisance

The same simulations were performed as in Section 1, but with ${\mathit{\gamma }}_0^l=1$. Since ${\mathit{\gamma }}_0^l=0$ was assumed, $Z_i^l$ represented an ignored latent variable. Fig. 2 shows similar results to those in Fig. 1. The parametric test was even more anticonservative than in Section 5.1. The reason is that the introduction of $Z_i^l$ increased the variance Y_i, so the variance of the score was even more misspecified than in Section 5.1.

Figure 2

Estimated rejection probabilities for four tests under misspecified variance, estimated nuisance and ignored nuisance (the null hypothesis was $H_0:\beta =0$) (, parametric; , sandwich; , flip basic; , flip effective): (a) β=0,n=50; (b) β=0,n=200

The test of theorem 2 was still nearly exact for n=200, even though μ_i was misspecified. (Even marginally over $Z_i^l$, μ_i was misspecified. Possibly the estimation of the intercept corrected for the misspecification.)

A conclusion from the simulations of Sections 5.1 and 5.2 is that the sandwich‐based approach should not always be seen as the most reliable way of testing models with misspecified variance functions. Indeed, in our simulations the test of theorem 2 was substantially less anticonservative (while having similar power; see Section 5.3).

5.3 Power

For a meaningful power comparison of the four tests, we considered the scenario where the model assumed was correct, i.e. the data distribution was Poisson and ${\mathit{\gamma }}_0^l$ was 0: Fig. 3. The estimated probabilities are based on 2×10⁴ simulation loops.

Figure 3

Power comparison of four two‐sided tests under the correct model, with estimated nuisance (the null hypothesis was $H_0:\beta =0$) (, parametric; , sandwich; , flip basic; , flip effective): (a) β=0.2, n=50; (b) β=0.2, n=200

Since the model was correct, asymptotically there was no better choice than the parametric test. The sign flipping test of theorem 2 had very similar power. The basic sign flipping test was again conservative because of the estimation of ${\mathit{\gamma }}_0$. The sandwich‐based test had the most power but was anticonservative (the null behaviour is not shown).

5.4 Strong heteroscedasticity

When a Gaussian linear model is considered with $Y_i\sim N(\beta x_i,{\sigma }^2)$, x₁=…=x_n=1 and $H_0:\beta =0$, the score contributions are ν_i=X_i(Y_i−0)/σ²=Y_i/σ². Thus the test of theorem 1 simply flips the observations Y_i, 1⩽i⩽n. The parametric counterpart of this test is the one‐sample t‐test. The t‐test needs to estimate the nuisance parameter σ²; the sign flipping test does not (simply substitute σ=1).

We simulated strongly heteroscedastic data: we took $Y_i\sim N(\beta x_i,{\sigma }_i^2)$, with σ_i= exp (i), 1⩽i⩽n=10. Consequently the t‐statistic did not have the distribution assumed and under H₀ the rejection probability of the t‐test was far from the nominal level for most α: Fig. 4(a). The sign flipping test did not need to estimate the variance. In this setting the test has rejection probability ⌊αw⌋/w exactly if the transformations g₁,…,g_w are drawn without replacement, since the observations are symmetric; see proposition 1. (We drew g₁,…,g_w with replacement for convenience, but this gives almost the same test as drawing without replacement, because of the small probability of ties.)

Figure 4

Comparison of the one‐sample t‐test () and the sign flipping test () (the null hypothesis was H₀:μ=0): (a) μ=0, strong heteroscedasticity; (b) μ=0.5, correct model

For a meaningful power comparison, we considered the correct homoscedastic model with σ₁=…=σ₁₀=1. Fig. 4(b), based on 10⁵ repeated simulations, shows that the tests had virtually the same power.

5.5 Multi‐dimensional parameter of interest

We considered the same setting as in Section 5.2, except that β and the estimated nuisance parameter ${\mathit{\gamma }}_0=(0.5,0.2,0,0,0)$ were five dimensional (so ${\mathbf{X}}_i$, ${\mathbf{Z}}_i\in {\mathbb{R}}^5$). All corresponding covariates were correlated (ρ=0.5). There was an ignored nuisance covariate as before ( ${\mathit{\gamma }}_0^l=0.5$), which was uncorrelated with the other covariates. Thus there were in total 11 covariates. We took the overdispersion such that $\text{var}(Y_i)={\mu }_i+0.5{\mu }_i^2$, i.e. the overdispersion again depended on the covariates (heteroscedasticity).

Instead of the basic score test for a one‐dimensional parameter we now used the multi‐dimensional extension in theorem 3. Similarly, instead of the test of theorem 2 based on effective scores, we used the multi‐dimensional extension in theorem 4. We took $\widehat{\mathbf{V}}=\mathbf{V}$ to be the identity matrix.

In Sections 5.1 and 5.2 we compared our tests with a Wald test based on a sandwich estimate of $\text{var}(\widehat{\mathit{\beta }})$. Here we proceeded analogously, using a sandwich estimate of the 5×5 matrix $\text{var}(\widehat{\mathit{\beta }})$ in the multi‐dimensional Wald test. This test uses that ${\widehat{\mathit{\beta }}}^{\prime }\text{var}{(\widehat{\mathit{\beta }})}^{-1}\widehat{\mathit{\beta }}$ asymptotically has a ${\chi }_d^2$‐distribution under the null hypothesis $H_0:\mathit{\beta }=\mathbf{0}$.

The results under H₀ are shown in Fig. 5, where each plot is based on 10⁴ simulation loops. They are comparable with those in Section 5.2, except that the sandwich‐based method is now even more anticonservative. This is because $\text{var}(\widehat{\mathit{\beta }})$ is now a 5×5 matrix, which is difficult to estimate accurately. For n=50 and α=0.01, the rejection probability of the sandwich‐based method was 0.27 instead of the required 0.01.

Figure 5

Estimated rejection probabilities under the null hypothesis (the model was misspecified because of overdispersion, heteroscedasticity and ignored nuisance) (, parametric; , sandwich; , flip basic; , flip effective): (a) β=0, n=50; (b) β=0, n=200

For a meaningful power comparison of the four tests, we again considered the scenario where the model assumed was correct, i.e. the data distribution was Poisson and ${\mathit{\gamma }}_0^l$ was 0. See Fig. 6, where each plot is based on 10⁴ simulation loops. As usual, the sign flipping test based on basic scores had low power due to nuisance estimation. The power of the sign flipping test based on effective scores was comparable with that of the parametric score test. As in Section 5.3, the test based on a sandwich estimate was the most powerful, but this has limited meaning, since it was also quite anticonservative under the correct model (the plot is not shown).

Figure 6

Power comparison under the correct model (the null hypothesis was $H_0:\mathit{\beta }=\mathbf{0}$) (, parametric; , sandwich; , flip basic; , flip effective): (a) $\mathit{\beta }={(0.2,0,0,0,0)}^{\prime }$, n=50; (b) $\mathit{\beta }={(0.2,0,0,0,0)}^{\prime }$, n=200

To conclude, sign flipping provided much more reliable type I error control than the sandwich approach, while giving satisfactory power (comparable with that of the parametric test, under the correct model).

6 Data analysis

We analysed the data set warpbreaks. These data are used in the example code of the gee R package, available on the Comprehensive R Archive Network. The data set gives the number of warp breaks per loom, where a loom corresponds to a fixed length of yarn. There are 54 observations of three variables: the number of breaks, the type of wool (A or B) and the tension (low, medium or high). For each of the six possible combinations of wool and tension, there are nine observations. Using various methods, we tested whether the number of breaks depends on the type of wool.

We first considered a basic Poisson model with

$$\log ({\mu }_i)={\mathit{\gamma }}_1+\beta {\mathbb{1}}_{\{\mathrm{wool}=B\}}+{\mathit{\gamma }}_2{\mathbb{1}}_{\{\mathrm{tension}=M\}}+{\mathit{\gamma }}_3{\mathbb{1}}_{\{\mathrm{tension}=H\}}.$$

The ${\mathit{\gamma }}_i$, 1⩽i⩽3, were nuisance parameters that were estimated by using maximum likelihood. We first tested $H_0:\beta =0$ using the parametric score test, obtaining a p‐value of 6.29×10⁻⁵. (All the tests that were performed were two sided.)

However, the data were clearly overdispersed: for each combination of wool and tension, the empirical variance of the nine observations was substantially larger than the empirical mean. Thus the p‐value based on the parametric test had limited meaning. Fitting a quasi‐Poisson model, which assumes constant overdispersion, gave a p‐value of 0.059.

As in Section 5, we also applied a Wald test, where we used a sandwich estimate (Agresti (2015), page 280) of the variance of $\widehat{\beta }$, to correct for the misspecified variance function. This resulted in a p‐value of 0.048.

Further, we used the sign flipping test based on the basic scores ${\nu }_{\widehat{\mathit{\gamma }},i}$, i=1,…,54 (still using the basic Poisson model). We took w=10⁶. This resulted in a p‐value of 0.113. This test is quite robust to model misspecification, but we know that it tends to be conservative when the score is correlated with the nuisance scores, as was the case here.

Finally, we performed the test of theorem 2 based on the effective score. This test is asymptotically exact under the correct model and has been shown to be robust against several forms of variance misspecification. It provided a p‐value of 0.065.

On the basis of this evidence, when maintaining a confidence level of 0.05, it seems that we cannot reject H₀. Indeed, only the sandwich‐based test provided a p‐value below 0.05, but this test is often anticonservative, as discussed in Section 5.1.

7 Discussion

We have proposed a test which relies on the assumption that individual score distributions are independent and have mean 0 (in the case of a point hypothesis) under the null. If the score contributions are misspecified because of overdispersion, heteroscedasticity or ignored nuisance covariates, then the traditional parametric tests lose their properties. The sign flipping test is often robust to these types of misspecification and can still be asymptotically exact.

When nuisance parameters are estimated, the basic score contributions become dependent. If a nuisance score is correlated with the score of the parameter of interest, the estimation reduces the variance of the score, so the sign flipping test becomes conservative. As a solution we propose to use the effective score, which is asymptotically the part of the score that is orthogonal to the nuisance score. The effective score is asymptotically unaffected by the nuisance estimation, so we again obtain an asymptotically exact test. We have proved that this is still the case when the scores and the Fisher information are misspecified by a constant, and simulations illustrate additional robustness.

When the parameter of interest is multi‐dimensional, our test statistic involves a freely chosen matrix, which influences the power properties. If this matrix is taken to be the inverse of the effective Fisher information and the model assumed is correct, then our test is asymptotically equivalent to the parametric score test. Under the correct model, in certain situations our test is asymptotically equivalent to the global test (Goeman et al., 2006), which is popular for testing hypotheses about high dimensional parameters.

Acknowledgement

We thank two referees for comments that helped to improve the paper.

Appendix A: A lemma

Lemma 1.Suppose that, for n→∞, a vector ${\mathbf{T}}^n=(T_1^n,\dots ,T_w^n)$ converges in distribution to a vector T of IID continuous variables. Then $\mathbb{P}(T_1^n>T_{[1-\alpha ]}^n)\to \lfloor \alpha w\rfloor /w.$

Proof.Note that $\mathbb{P}(T_1^n>T_{[1-\alpha ]}^n)=\mathbb{P}({\mathbf{T}}^n\in A)$, where

$$A=\{(t_1,\dots ,t_w)\in {\mathbb{R}}^w:|\{2⩽j⩽w:t_j<t_1\}|⩾\lceil (1-\alpha )w\rceil \}.$$

Let ∂A be the boundary of A, i.e. the set of discontinuity points of ${\mathbb{1}}_A$. If t ∈ ∂A, then t_i=t_j for some 1⩽i<j⩽w. It follows that $\mathbb{P}(\mathbf{T}\in \partial A)=0$. Since ${\mathbb{1}}_A$ is continuous on (∂A)^c, it follows from the continuous mapping theorem (Van der Vaart (1998), theorem 2.3) that

$${\mathbb{1}}_A({\mathbf{T}}^n)\stackrel{\mathrm{d}}{\to }{\mathbb{1}}_A(\mathbf{T}).$$

The elements of T are IID draws from the same distribution. Hence it follows from the Monte Carlo testing principle (Lehmann and Romano, 2005) that, under H₀, $\mathbb{P}(\mathbf{T}\in A)=\lfloor \alpha w\rfloor /w$. Thus $\mathbb{P}({\mathbf{T}}^n\in A)\to \lfloor \alpha w\rfloor /w.$

Appendix B: Proofs of the results

B.1. Proof of theorem 1

Suppose that H₀ holds. We shall show that ${\mathbf{T}}^n=(T_1^n,\dots ,T_w^n)$ converges in distribution to a multivariate normal distribution with mean 0 and variance ${lim}_{n\to \infty }{s}_n^2\mathbf{I}$, where I is the w×w identity matrix. It then follows from lemma 1 that $\mathbb{P}(T_1^n>T_{[1-\alpha ]}^n)\to \lfloor \alpha w\rfloor /w$.

Under H₀, for each 1⩽j⩽w, $\mathbb{E}(T_j^n)=0$. For every 1⩽j⩽w, $\text{var}(T_j^n)=n^{-1}{\mathrm{\Sigma }}_{i=1}^n\text{var}({\nu }_i)=s_n^2$. Let ${\mathbf{Q}}_n$ be the covariance matrix of ${\mathbf{T}}^n$. ${\mathbf{Q}}_n$ has 0s off the diagonal. Indeed, for 1⩽j<k⩽w,

$$\text{cov}(T_j^n,T_k^n)=\text{cov}(n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\nu }_i,n^{-1/2}\sum _{i=1}^n{g}_{\mathit{ki}}{\nu }_i)=0,$$

since the g_ki, 2⩽k⩽w, are independent with mean 0. Hence ${\mathbf{Q}}_n$ converges to ${lim}_{n\to \infty }{s}_n\mathbf{I}$. Note that ${\mathbf{T}}^n$ is a sum of n vectors. By the multivariate Lindeberg–Feller central limit theorem (Van der Vaart, 1998) ${\mathbf{T}}^n$ converges in distribution to a multivariate normal distribution with mean vector 0 and covariance matrix ${lim}_{n\to \infty }{s}_n^2\mathbf{I}$.

We have shown that ${\mathbf{T}}^n$ converges in distribution to a vector T, say, of IID normal random variables. It now follows from lemma 1 that $\mathbb{P}(T_1^n>T_{[1-\alpha ]}^n)\to \lfloor \alpha w\rfloor /w.$

B.2. Proof of proposition 2

Note that

$$({\nu }_1,\dots ,{\nu }_n)\stackrel{\mathrm{d}}{=}(g_{j1}{\nu }_1,\dots ,g_{\mathit{jn}}{\nu }_n)$$

for every 1⩽j⩽w. This means that the test becomes a basic random‐transformation test and the results follow from the proof of theorem 2 in Hemerik and Goeman (2018b).

B.3. Proof of theorem 2

Suppose that H₀ holds. Note that

$$\begin{array}{cc}\hfill S_{\widehat{\mathit{\gamma }}}^{*}& =S_{\widehat{\mathit{\gamma }}}-{\widehat{\mathcal{I}}}_{12}^{\prime }{\widehat{\mathcal{I}}}_{22}^{-1}{\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}=S_{\widehat{\mathit{\gamma }}}-{\mathcal{I}}_{12}^{\prime }{\mathcal{I}}_{22}^{-1}{\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1)}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)=S_{{\mathit{\gamma }}_0}-{\mathcal{I}}_{12}^{\prime }\surd n(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)-{\mathcal{I}}_{12}^{\prime }{\mathcal{I}}_{22}^{-1}\{{\mathbf{S}}_{{\mathit{\gamma }}_0}^{(k-1)}\hfill \\ & -{\mathcal{I}}_{22}\surd n(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)\}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)=S_{{\mathit{\gamma }}_0}^{*}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1).\hfill \end{array}$$

Let 2⩽j⩽w and

$$\begin{array}{cc}& S_{\mathit{\gamma }}^{j+}=n^{-1/2}\sum _{i=1}^n{\mathbf{1}}_{\{g_{ji=1}\}}{\nu }_{\mathit{\gamma },i},\hfill \\ & S_{\mathit{\gamma }}^{j-}=n^{-1/2}\sum _{i=1}^n{\mathbf{1}}_{\{g_{ji=-1}\}}{\nu }_{\mathit{\gamma },i}.\hfill \end{array}$$

Note that

$$\begin{array}{cc}\hfill S_{\widehat{\mathit{\gamma }}}^j=S_{\widehat{\mathit{\gamma }}}^{j+}-S_{\widehat{\mathit{\gamma }}}^{j-}& =\{S_{{\mathit{\gamma }}_0}^{j+}-\frac{1}{2}\surd n{\mathcal{I}}_{12}^{\prime }(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)\}-\{S_{{\mathit{\gamma }}_0}^{j-}-\frac{1}{2}\surd n{\mathcal{I}}_{12}^{\prime }(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)\}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)\hfill \\ & =S_{{\mathit{\gamma }}_0}^{j+}-S_{\widehat{\mathit{\gamma }}}^{j-}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)=S_{{\mathit{\gamma }}_0}^j+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1).\hfill \end{array}$$

The intuitive reason why $S_{\widehat{\mathit{\gamma }}}^j=S_{{\mathit{\gamma }}_0}^j+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)$ is that the estimation of $\widehat{\mathit{\gamma }}$ does not cause the summands underlying $S_{\widehat{\mathit{\gamma }}}^j$ to be correlated. Similarly we find that ${\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1),j}={\mathbf{S}}_{{\mathit{\gamma }}_0}^{(k-1),j}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)$ and conclude that $S_{\widehat{\mathit{\gamma }}}^{*j}=S_{{\mathit{\gamma }}_0}^{*j}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1).$

Let ${\mathbf{T}}^n$ be as in the proof of theorem 1, with ν_i replaced by ${\nu }_{{\mathit{\gamma }}_0,i}^{*}.$ Suppose that H₀ holds and $\widehat{\mathcal{I}}=\mathcal{I}$, so that the summands underlying $T_j^n$ are independent. For every 1⩽i⩽n, $\mathbb{E}({\nu }_{{\mathit{\gamma }}_0,i}^{*})=0.$ The elements of ${\mathbf{T}}^n$ are uncorrelated and have common variance $\text{var}({\nu }_{{\mathit{\gamma }}_0,1}^{*}).$ By the multivariate central limit theorem (Van der Vaart, 1998; Greene, 2012), ${\mathbf{T}}^n$ converges in distribution to $N\{\mathbf{0},\text{var}({\nu }_{{\mathit{\gamma }}_0,1}^{*})\mathbf{I}\}$. We supposed that $\widehat{\mathcal{I}}=\mathcal{I}$ to use the central limit theorem, but the asymptotic distribution of ${\mathbf{T}}^n$ is the same if $\widehat{\mathcal{I}}$ is any consistent estimator of $\mathcal{I}$.

Let ${\widehat{\mathbf{T}}}^n$ be as in the proof of theorem 1, with ν_i replaced by ${\nu }_{\widehat{\mathit{\gamma }},i}^{*}.$ For every 1⩽j⩽w, $S_{\widehat{\mathit{\gamma }}}^{*j}=S_{{\mathit{\gamma }}_0}^{*j}+o_{{\mathbb{P}}_{{\beta }_0,{\mathit{\gamma }}_0}}(1)$. Thus ${\widehat{\mathbf{T}}}^n$ and ${\mathbf{T}}^n$ are asymptotically equivalent. The result now follows from lemma 1.

B.4. Proof of proposition 2

For 2⩽j⩽w consider

$$\begin{array}{cc}& S_{\widehat{\mathit{\gamma }}}^{*j+}=n^{-1/2}\sum _{i=1}^n{\mathbf{1}}_{\{g_{ji=1}\}}{\nu }_{\widehat{\mathit{\gamma }},i}^{*},\hfill \\ & S_{\widehat{\mathit{\gamma }}}^{*j-}=n^{-1/2}\sum _{i=1}^n{\mathbf{1}}_{\{g_{ji=-1}\}}{\nu }_{\widehat{\mathit{\gamma }},i}^{*}\hfill \end{array}$$

$$\begin{array}{cc}& S_{\widehat{\mathit{\gamma }}}^{(k-1),j+}=n^{-1/2}\sum _{i=1}^n{\mathbf{1}}_{\{g_{ji=1}\}}{\nu }_{\widehat{\mathit{\gamma }},i}^{(k-1)},\hfill \\ & S_{\widehat{\mathit{\gamma }}}^{(k-1),j-}=n^{-1/2}\sum _{i=1}^n{\mathbf{1}}_{\{g_{ji=-1}\}}{\nu }_{\widehat{\mathit{\gamma }},i}^{(k-1)}.\hfill \end{array}$$

We have

$$\begin{array}{cc}\hfill S_{\widehat{\mathit{\gamma }}}^{*j+}& =S_{\widehat{\mathit{\gamma }}}^{j+}-{\widehat{\mathcal{I}}}_{12}^{\prime }{\widehat{\mathcal{I}}}_{22}^{-1}{S}_{\widehat{\mathit{\gamma }}}^{(k-1),j+}=S_{{\mathit{\gamma }}_0}^{j+}-\frac{1}{2}\surd n{\mathcal{I}}_{12}^{\prime }(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)-{\mathcal{I}}_{12}^{\prime }{\mathcal{I}}_{22}^{-1}\{S_{{\mathit{\gamma }}_0}^{(k-1),j+}-\frac{1}{2}\surd n{\mathcal{I}}_{22}(\widehat{\mathit{\gamma }}-{\mathit{\gamma }}_0)\}+o_{{\mathbb{P}}_{{\beta }^n,{\mathit{\gamma }}_0}}(1)\hfill \\ & =S_{{\mathit{\gamma }}_0}^{*j+}+o_{{\mathbb{P}}_{{\beta }^n,{\mathit{\gamma }}_0}}(1)\hfill \end{array}$$

and analogously $S_{\widehat{\mathit{\gamma }}}^{*j-}=S_{{\mathit{\gamma }}_0}^{*j-}+o_{{\mathbb{P}}_{{\beta }^n,{\mathit{\gamma }}_0}}(1)$. By Marohn (2002), page 341, for 2⩽j⩽w, $S_{{\mathit{\gamma }}_0}^{*j+}$ and $S_{{\mathit{\gamma }}_0}^{*j-}$ have an asymptotic $N(\frac{1}{2}\xi {\sigma }_0^2,\frac{1}{2}{\sigma }_0^2)$ distribution. Since they are independent, it follows that $T_j^n=S_{\widehat{\mathit{\gamma }}}^{*j+}-S_{\widehat{\mathit{\gamma }}}^{*j-}$ has an asymptotic $N(0,{\sigma }_0^2)$ distribution, 2⩽j⩽w. With the multivariate central limit theorem we find that $(T_2^n,\dots ,T_w^n)$ converges in distribution to a vector of w−1 IID $N(0,{\sigma }_0^2)$ variables as n→∞.

Let ε, ε^′>0. Let $(T_1^{\prime },\dots ,T_w^{\prime })$ have the asymptotic distribution of $(T_1^n,\dots ,T_w^n)$. Let $(T_1^{″},\dots ,T_w^{″})$ be a vector of w IID $N(0,{\sigma }_0^2)$ variables. Apart from the first element, these two vectors have the same distribution. For w ∈ {2,3,…}, define $T_{[1-\alpha ]}^{[w]}$ like $T_{[1-\alpha ]}^n$, but based on the values $T_1^{\prime },\dots ,T_w^{\prime }$ instead of $T_1^n,\dots ,T_w^n$. Also define $T_{[1-\alpha ]}^{[[w]]}$ like $T_{[1-\alpha ]}^n$, but based on the values $T_1^{″},\dots ,T_w^{″}$. Note that, as w→∞, the empirical quantile $T_{[1-\alpha ]}^{[[w]]}$ converges in distribution to the constant σ₀Φ(1−α). Further note that, for w→∞, $T_{[1-\alpha ]}^{[[w]]}-T_{[1-\alpha ]}^{[w]}$ converges in distribution to 0. Thus there is a $W\in \mathbb{N}$ such that, for all w>W,

$$\mathbb{P}(|T_{[1-\alpha ]}^{[w]}-{\sigma }_0\mathrm{\Phi }(1-\alpha )|<{ϵ}^{\prime })>1-ϵ.$$(3)

Since the distribution of $(T_1^n,\dots ,T_w^n)$ converges to the distribution of $(T_1^{\prime },\dots ,T_w^{\prime })$ as n→∞,

$$T_{[1-\alpha ]}^n\stackrel{\mathrm{d}}{\to }{T}_{[1-\alpha ]}^{[w]}$$(4)

as n→∞. Since in the present proof w is not fixed, we shall write $T_{[1-\alpha ]}^n=T_{[1-\alpha ]}^{n,w}$. By results (3) and (4), for w>W, $\text{lim}{\text{inf}}_{n\to \infty }\mathbb{P}\{|T_{[1-\alpha ]}^{n,w}-{\sigma }_0\mathrm{\Phi }(1-\alpha )|<{ϵ}^{\prime }\}>1-ϵ$. Thus ${lim}_{w\to \infty }\text{lim}{\text{inf}}_{n\to \infty }\mathbb{P}\{|T_{[1-\alpha ]}^{n,w}-{\sigma }_0\mathrm{\Phi }(1-\alpha )|<{ϵ}^{\prime }\}=1$.

The distribution of $T_1^n$, which does not depend on w, converges to a continuous distribution as n→∞. It follows, that for every ε^′′>0, there is a W^′ such that there is an N such that, for all w>W^′ and n>N, $\mathbb{E}(|{\mathbb{1}}_{\{T_1^n>T_{[1-\alpha ]}^{n,w}\}}-{\mathbb{1}}_{\{T_1^n>{\sigma }_0\mathrm{\Phi }(1-\alpha )\}}|)<{ϵ}^{″}$. This means that ${lim}_{w\to \infty }\text{lim}{\text{inf}}_{n\to \infty }\mathbb{E}(|{\mathbb{1}}_{\{T_1^n>T_{[1-\alpha ]}^{n,w}\}}-{\mathbb{1}}_{\{T_1^n>{\sigma }_0\mathrm{\Phi }(1-\alpha )\}}|)=0$, as was to be shown.

B.5. Proof of proposition 3

For every 1⩽j⩽w we have

$${\tilde{S}}_{\widehat{\mathit{\gamma }}}^{*j}=c_1{S}_{\widehat{\mathit{\gamma }}}^j-c_2{\widehat{\mathcal{I}}}_{12}^{\prime }{c}_2^{-1}{\widehat{\mathcal{I}}}_{22}^{-1}{c}_1{\mathbf{S}}_{\widehat{\mathit{\gamma }}}^{(k-1),j}=c_1{S}_{\widehat{\mathit{\gamma }}}^{*j}.$$

Hence the test is identical to that of theorem 2, since that test is unchanged if all $T_j^n$, 1⩽j⩽w, are multiplied by the same constant.

B.6. Proof of theorem 3

Suppose that H₀ holds. Consider the d×j matrix

$${\left(n^{1/2}\sum _{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{{\mathit{\gamma }}_0,i}\right)}_{1⩽j⩽w}.$$(5)

It follows from the multivariate central limit theorem (Van der Vaart, 1998) that, as n→∞, this matrix converges in distribution to a matrix with identically distributed columns which are independent of each other. Note that, for every 1⩽j⩽w, $T_j^n$ is a function of the jth column of matrix (5). Thus, with the continuous mapping theorem (Van der Vaart (1998), theorem 2.3) it follows that $(T_1^n,\dots ,T_j^n)$ also converges in distribution to a vector with continuous IID elements. The result now follows from lemma 1.

B.7. Proof of theorem 4

Consider the case $\widehat{\mathit{\gamma }}={\mathit{\gamma }}_0$. As in the proof of theorem 3, under H₀, $(T_1^n,\dots ,T_w^n)$ converges in distribution to a vector of w IID variables. As in the proof of theorem 2, the same is true if we take $\widehat{\mathit{\gamma }}$ to be a different √n‐consistent estimator of ${\mathit{\gamma }}_0$. (Again, the reason is that the effective score based on $\widehat{\mathit{\gamma }}$ is asymptotically equivalent to the effective score based on ${\mathit{\gamma }}_0$.) The result now follows from lemma 1 again.

B.8. Proof of proposition 4

By Hall and Mathiason (1990), $n^{-1/2}{\mathrm{\Sigma }}_{i=1}^n{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{*}$ has an asymptotic $N(\mathbf{0},{\mathcal{I}}^{*})$ distribution under $\mathit{\beta }={\mathit{\beta }}_0$. Analogously to the one‐dimensional case at proposition 2, for 2⩽j⩽w, the vector $n^{-1/2}{\mathrm{\Sigma }}_{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{*}$ is asymptotically the difference of two mutually independent $N(\frac{1}{2}{\mathcal{I}}^{*}\mathit{\xi },\frac{1}{2}{\mathcal{I}}^{*})$ vectors (Hall and Mathiason, 1990), so it also has an asymptotic $N(\mathbf{0},{\mathcal{I}}^{*})$ distribution (under $\mathit{\beta }={\mathit{\beta }}^n$). As in the proof of theorem 3, by the multivariate central limit theorem, the d×(w−1) matrix ${(n^{-1/2}{\mathrm{\Sigma }}_{i=1}^n{g}_{\mathit{ji}}{\mathit{\nu }}_{\widehat{\mathit{\gamma }},i}^{*})}_{2⩽j⩽w}$ converges to a matrix with w−1 independent $N(\mathbf{0},{\mathcal{I}}^{*})$ columns as n→∞. Hence, by the continuous mapping theorem, as n→∞, $(T_2^n,\dots ,T_w^n)$ converges in distribution to a vector of w−1 IID variables (under $\mathit{\beta }={\mathit{\beta }}^n$), which follow the asymptotic distribution which $T_1^n$ has under $\mathit{\beta }={\mathit{\beta }}_0$.

The result now follows as at the end of the proof of proposition 2.