posts

p-uniform* is a three-parameter selection model

James E. Pustejovsky — Sun, 08 Mar 2026 00:00:00 GMT

Robbie van Aert and Marcel van Assen have a new article out on a method for estimating average effect size in meta-analyses subject to publication bias, called p-uniform*. The method has been around for a while (the first version of their paper is from 2018, van Aert & van Assen, 2018) and has been cited hundreds of times. The recent appearance of the peer-reviewed article prompted me to take a closer look.

The authors frame the p-uniform* method as a generalization of an earlier proposal of theirs, called p-uniform, which was limited in that it applies only to meta-analytic data where all effect size estimates are statistically significant at the conventional level. The new method improves upon p-uniform by using all of the data (including effect size estimates that are not statistically significant) and by allowing for heterogeneity in the distribution of effect size parameters from different samples. The authors contrast p-uniform* with the well-known three-parameter selection model (3PSM), which is a special case of the step-function selection model introduced by Hedges (1992). They note that both methods can be understood as selection models, in that they are based upon explicit assumptions about the process that determines whether an effect size estimate from a study is reported and available for meta-analysis. Further, they argue that the new p-uniform* method is preferable to the 3PSM because it does not involve estimation of a selection weight parameter, and thus is more parsimonious:

P-uniform* has the advantage over 3PSM that the weight in the selection model does not need to be estimated. These weights are known to be imprecisely estimated in more complex selection models than the selection model of 3PSM where multiple weights need to be estimated (Hedges & Vevea, 1996, 2005; Vevea & Woods, 2005). P-uniform* only assumes that these weights are the same for the statistically significant and the same for the nonsignificant primary studies’ effect sizes, without requiring estimation of the weights. This makes p-uniform* a more parsimonious model than 3PSM with similar statistical properties as shown in our simulation studies (van Aert & van Assen, 2026, p. 15).

In this post, I offer a different interpretation of p-uniform*. Rather than being a more parsimonious model than the 3PSM, I think it is useful to understand p-uniform* as a different estimator of the average effect size and heterogeneity parameters in the 3PSM. I will demonstrate that the maximum likelihood estimator of p-uniform* is exactly equivalent to the maximum likelihood estimator of 3PSM under the (admittedly artificial) condition that all effect sizes in the meta-analytic data have the same sampling variance. Outside of this condition, the methods are not exactly equivalent but will tend to produce very similar point estimates of average effect size and between-study heterogeneity because both are based on unbiased estimating equations. However, confidence intervals produced by each method will differ except when based on the same method of construction and when all effect sizes have the same sampling variance.

Notation

My discussion is necessarily going to get a bit technical. Consider a meta-analysis that includes independent effect size estimates, where and denote the observed effect size estimate and corresponding standard error from study . Let’s say that statistical significance is defined based on the level (which will usually be .05, so ). I will assume that the effect sizes are defined so that positive effects are what would be theoretically expected, and I will call an effect size estimate affirmative if it is statistically significant and in the expected direction. Let be an indicator for whether effect size estimate is affirmative, so , where is the standard normal quantile function. I will also use to denote the cumulative function and to denote the density function of the standard normal distribution.

The three-parameter selection model

Before considering how p-uniform* relates to the 3PSM, let me explain the latter model. The three-parameter selection model makes two sets of assumptions. First, it is assumed that if we could observe every effect size estimate ever produced, then those effect size estimates would follow a conventional random effects model with mean and between-study standard deviation . Second, it is assumed that the probability of observing a given estimate depends only on its sign and statistical significance. Specifically, the probability of observing a non-affirmative estimate differs from the probability of observing an affirmative estimate by a factor of . If , then there is selection in favor of affirmative (statistically significant) results, which produces upward bias in average effect size estimates from conventional meta-analytic methods. These assumptions are essentially identical to those of p-uniform*.¹

Under these assumptions, the distribution of an observed effect size estimate is what I call a “piece-wise normal” distribution, with density where . Note that depends on and but not on . From the density, the log-likelihood of the 3PSM given the observed effect size estimates is where is the number of non-affirmative effect size estimates observed. The maximum likelihood estimators , , are the values that maximize .

One way to compute maximum likelihood estimators is by profiling in one or more of the variables. For instance, we can consider the profile log-likelihood of and , which is the log-likelihood function for given values of and , but maximized over . With profiling, the maximum likelihood estimator of is treated as a function of and , as The profile log-likelihood is then and the maximum likelihood estimators and are the values that maximize , with . The end result is the same as maximizing the whole log-likelihood all at once—it’s just that in some cases, it’s easier to maximize in pieces (especially if there is a simple way to compute ).

For the three-parameter selection model, the maximum likelihood estimator of can also be defined as the solution to the score equation (the derivative of with respect to ) The quantity on the left is the observed number of affirmative effects, equal to . The quantity on the right is the expected number of affirmative results in the observed sample based on the three-parameter selection model. Equation 4 does not have a closed-form solution except in some simple special cases, but it can be solved readily with numerical algorithms.

p-uniform*

Now let me consider van Aert and van Assen’s proposed method, p-uniform*. The method is an estimator of and , but defined by a different objective function that involves truncated normal distributions (Equation 3 of van Aert & van Assen, 2026). For the affirmative effects, the objective function involves the density of each effect size estimate, given that it is affirmative; for the non-affirmative effects, it involves the density of each effect size estimate, given that it is not affirmative. Taking the log of their objective function (Equation 4 of van Aert & van Assen, 2026) gives The p-uniform estimators and are the quantities that maximize .

The objective function can be derived directly from the three-parameter selection model. The truncated densities are the conditional densities of the effect size estimates, given their affirmative or non-affirmative status. Thus, , whereas the 3PSM log-likelihood uses the unconditional density of the observed effect size estimates. The p-uniform* method is essentially the maximum likelihood estimator of and , conditional on whether each of the effects are affirmative (i.e., conditional on . Conditioning removes all of the information in the likelihood that relates to the selection parameter , so this parameter does not appear in .

Using conditional distributions for inference is warranted when the conditioning information is an ancillary statistic that carries no information about the model parameters (Ghosh et al., 2010). However, the statistical significance of the effect size estimates are not ancillary for the average effect size and heterogeneity parameters of the 3PSM. van Aert & van Assen (2026) offer no justification for basing inferences about these parameters on the conditional distribution. It is not obvious to me that confidence intervals based on the conditional distribution would be valid—even asymptotically as increases (except under conditions where the p-uniform* objective function coincides with the 3PSM log-likelihood).

Equal standard errors

The main simulations reported in van Aert & van Assen (2026) are based on a data-generating process where all the effect size estimates have equal standard errors, so . This is a fairly artificial scenario that does not arise frequently in real meta-analysis, as far as I know. However, it is nonetheless a useful case to consider because it makes the math much easier and cleaner.

In this scenario with equal standard errors, the estimators of and from p-uniform* are exactly equal to the corresponding maximum likelihood estimators from the 3PSM. To see this, observe that when all are equal, then all the ’s become constant, with for . The maximum likelihood estimator of (Equation 4) simplifies to The difference between the profile log-likelihood of the 3PSM (Equation 3) and the objective function of p-uniform* (Equation 5) can then be written as The difference between the objective functions of the 3PSM and p-uniform* is a constant that does not depend on the model parameters. Consequently, the maxima of the functions must be equal, and the profile likelihood confidence intervals for and must be identical.

Several implications follow from this. First, the equivalence of the p-uniform* and 3PSM estimators explains why the performance of the point estimators for is so similar (in terms of bias and root mean-squared error) in the main simulations of van Aert & van Assen (2026) Second, the equivalence of the estimators runs counter to van Aert and van Assen’s argument that p-uniform* is more parsimonious and has fewer limitations. If the estimators are the same, then clearly one cannot have advantages over the other. Third, van Aert & van Assen (2026) reported differences between p-uniform* and 3PSM in the performance of the confidence intervals for in the main simulations. However, given the equivalence of the objective functions, differences in the behavior of the confidence intervals must be attributable to differences in how they are constructed (profile likelihood confidence intervals for p-uniform* but Wald-type confidence intervals for the 3PSM), rather than differences between the models.

Varying standard errors

Except when the standard errors are all equal, the objective function of p-uniform* is not exactly identical to the profile log-likelihood of the 3PSM. However, objective function of p-uniform* produces unbiased estimating equations for and . To see this, observe that another way to express the p-uniform* estimator is as the solution to the system of equations that set the derivative of with respect to the parameters equal to zero. For p-uniform*, these derivatives are where .

My previous post gave expressions for the mean and variance of an observed effect size estimate under the step-function selection model. For the case of a single-step model, these results imply that Furthermore, Substituting these expressions into the expectations of Equation 7 leads to and Thus, the p-uniform* estimator is based on unbiased estimating equations for and .

How different is the p-uniform* estimator from the 3PSM maximum likelihood estimator in this case of unequal variances? The degree of potential discrepancy depends on how different the objective functions are. Allowing for unequal variances where is the probability of observing an affirmative result under the three-parameter selection model. The estimated selection parameter is defined by the constraint that . Letting and , Equation 8 can be written as with the constraint that , where and are the average probabilities among the non-affirmative and affirmative effects.

van Aert and van Assen argue that p-uniform* is advantageous because it does not entail estimation of the selection weight . However, one could also take this as a deficiency, insofar as the p-uniform* estimator does not yield information about the strength of selection—a substantively interesting parameter that quantifies the severity of publication bias. This deficiency could be remedied by borrowing the estimator of from the step-function model, as . This estimator can be computed by solving Equation 4 for with evaluated at and .

Like the 3PSM MLE, p-uniform* can be defined by a set of unbiased estimating equations. Although this property is not sufficient to ensure that p-uniform* estimator will work well, it does suggest that p-uniform* might have biases similar to those of the 3PSM MLE. Furthermore, sandwich variance estimators or non-parametric bootstrapping might be useful for assessing uncertainty in model parameter estimates generated with p-uniform* (as we examine for MLEs in Pustejovsky et al., 2025). However, it seems unlikely that p-uniform* would be more efficient than the MLE of the 3PSM under the model’s assumptions—maximum likelihood estimation is usually pretty hard to improve upon, at least for low-dimensional problems. Does p-uniform* have some robustness advantage that maximum likelihood lacks? In my view, there needs to be more thorough examination of the properties of p-uniform* (be it through theoretical analysis or simulation under varied, realistic data-generating processes) before it can be recommended for conducting meta-analyses of real data.

References

Ghosh, M., Reid, N., & Fraser, D. A. S. (2010). Ancillary statistics: A review. Statistica Sinica, 20(4), 1309–1332. http://www.jstor.org/stable/24309506

Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis. Statistical Science, 7(2), 246–255. https://doi.org/10.1214/ss/1177011364

Pustejovsky, J. E., Citkowicz, M., & Joshi, M. (2025). Estimation and Inference for Step-Function Selection Models in Meta-Analysis with Dependent Effects. https://doi.org/10.31222/osf.io/qg5x6_v1

van Aert, R. C. M., & van Assen, M. A. L. M. (2018, October 2). Correcting for publication bias in a meta-analysis with the p-uniform* method. https://doi.org/10.31222/osf.io/zqjr9

van Aert, R. C. M., & van Assen, M. A. L. M. (2026). Correcting for publication bias in a meta-analysis with the p-uniform* method. Psychonomic Bulletin & Review, 33(3), 102. https://doi.org/10.3758/s13423-025-02812-4

Footnotes

The p-uniform* method does not explicitly define the , but it is implicit in the assumption that the “weights are the same for the statistically significant and the same for the nonsignificant primary studies’ effect sizes.” Although the weights are constant conditional on whether the effects are affirmative, they are allowed to differ between affirmative and non-affirmative effect sizes, and is the ratio of the weights for non-affirmative effects to the weights for affirmative ones.↩︎

Notes on regression adjustment for selectively reported results in meta-analysis

James E. Pustejovsky — Sat, 17 Jan 2026 00:00:00 GMT

Several of the most prominent methods for investigating selective reporting bias in meta-analyis take the form of simple linear regressions, estimated by weighted least squares. Methods of this form include the Precision Effect Test (PET) and Precision Effect Estimator with Standard Error (PEESE) adjustments proposed by Stanley (2008) and Stanley & Doucouliagos (2014) as well as the endogenous kink (EK) meta-regression proposed by Bom & Rachinger (2019). These methods are widely used (PET and PEESE especially) and have been studied pretty extensively using Monte Carlo simulations (e.g., Bramley et al., 2021; Carter et al., 2019; Hong & Reed, 2021; McShane et al., 2016; Moreno et al., 2009; Stanley, 2017). By my read, the simulation evidence is mixed: these regression adjustments seem to be less biased than regular random effects models but they are pretty crude corrections for the bias created by selective reporting, and they don’t always work very well. In this post, I want to look at how the methods work analytically. I’ll derive expressions for their bias and accuracy and consider how one might use these expressions to say something about the conditions under which regression adjustment might be expected to perform reasonably well.

Regression adjustments for publication bias

For a meta-analysis of studies, let be an effect size estimates and be its standard error for .¹ Many bias adjustment methods for this context involve the regression where is a predictor that is related to the precision of the effect size estimate and is a residual error. Equation 1 is estimated by some form of weighted least squares with weights , which might be the inverses of the sampling variances, (i.e., fixed effects weights) or the inverses of the total residual variance , where is an estimate of between-study variance in the effect size parameters.

The intuition behind this sort of estimator is that selective reporting on the basis of statistical significance tends to produce larger biases for smaller primary studies (which have larger standard errors), leading to what looks like “small-study” effects. PET, PEESE, and EK adjust for the small-study effects using some form of linear regression, treating the estimate of the intercept in Equation 1 as the average effect size in a population of infinitely large studies (so that their standard errors are ).² The main difference between the methods is in the predictor used to extrapolate to this hypothetical population:

PET uses ,
PEESE uses , and
EK uses , where is the so-called “kink” of the meta-regression.³

For now, let’s work with the generic predictors . Basic results from linear regression tell us that the estimator of the intercept of a linear regression fitted by weighted least squares has the form where , , and . Denoting the full vector of standard errors as and observing that the predictors are all functions of the standard errors, the conditional expectation and conditional variance of can be written as and If the real regression is linear in , then . Substitute this into Equation 16 and you’ll find , so the regression estimator is indeed unbiased. But what if the real conditional expectation is not linear?

Data-generating processes

Regression adjustment methods are a bit hard to pin down because they’re not full generative models of the process by which the observed set of effect sizes arises. Instead, they’re just different forms of approximations, which don’t really commit to any particular model for how selective reporting occurs. To say anything about how well the regression adjustments work, I will need to look beyond these specific methods to other models for the selective reporting process.

Stanley & Doucouliagos (2014) motivated the PEESE adjustment through a Taylor series approximation to the bias of effect size estimates under a simple form of a step-function selection model, in which the real form of the bias is nonlinear in . There are a number of other models where the strength of selection depends on the statistical significance of the effect size estimate (e.g., the beta density model), as well as models that exhibit small-study effects produced by other selection processes (e.g., the Copas selection model). Here I’ll look at a step-function model and the Copas model, mostly because these are fairly tractable to analyze. Both models posit that effect size estimates follow a conventional random effects model prior to being put through the selection process. Denoting an effect size estimate prior to selection as , we assume where is the overall average effect size, is a mean-zero random effect with variance , which captures heterogeneity in the effect size parameters, and is a mean zero sampling error with known standard error . In the ideal, a selective reporting correction would recover an estimate of the mean of the effect size distribution .

Step function models

In the step function model, we assume that the generated effect size estimate is observed with some probability that depends on its -value for the one-sided hypothesis test of . Letting , the selective reporting process of the step-function model is then for some set of steps and selection weights .

In my previous post on the step-function model, I showed that the conditional expectation of the observed effect size estimates is where is a certain, rather complicated non-linear function of the model parameters given by where with and .⁴ Letting and , the conditional variance is

Substituting Equation 7 into Equation 16, the bias of a regression-adjustment estimator under the step-function selection model is Thus, the bias of a regression adjustment depends on how well the predictor approximates the non-linear function .

Substituting Equation 9 into Equation 4, an expression for the sampling variance of the regression adjustment estimator is Combining the bias and the variance in the usual way leads to an expression for RMSE: In general, the magnitude of the bias and RMSE will depend on:

the parameters of the true distribution of effects, and ,
the strength of selective reporting, measured by ,
the choice of predictors and weights , and
the distribution of values (i.e., the average and spread of the standard errors).

That’s a lot of moving pieces. However, the bias and RMSE are at least computable, and analytic expressions are much quicker to calculate than running a couple of thousand replications of a simulation.

Copas selection models

The Copas selection model posits a different selection process than the step-function model, in which the probability that a generated effect size estimate is reported is expressed in terms of an unobserved, normally distributed variable that is correlated with the sampling error of the effect size estimate (i.e., correlated with in Equation 5). See my previous post on the Copas model for a more detailed description, interactive illustration, and some spicy takes on why this model is pretty odd. The selection process here has three parameters, which I will denote as , , and (my previous post used and , but in this post I need these to define the regression equations). The parameter controls the overall fraction of generated effect size estimates that are observed, the parameter controls how strongly the probability of observation depends on the precision of the effect size estimate, and the parameter controls the degree to which the probability depends on the magnitude of the observed effect size estimate relative to the overall average effect. The probability that a generated effect size estimate is observed is given by In my previous post, I gave expressions for the mean and variance of observed effect size estimate conditional on . The conditional expectation is where . The conditional variance is Substituting into Equation 16, the bias of a regression-adjustment estimator under the Copas selection model is This expression is a bit simpler than the bias under the step function model. For one, the bias depends neither on nor on , and is multiplicative in the parameter. An implication is that, if you were running a simulation where the Copas model was the true data-generating process, the bias of the regression adjustment estimator should be invariant across and and should be strictly proportional to . Just as with the step-function selection model, the bias will depend on how well the predictor approximates the functional form of the bias curve , which will in turn depend on the distribution of the standard errors. In other words, the regression adjustments will perform better or worse, depending on the average and shape of the distribution across the included studies.

Why care?

I’ve just thrown down a lot of rather ugly looking formulas, and one might well ask why anyone should care about this mess of math. What are these formulas good for? To be honest, I worked through this stuff mostly for myself,⁵ but I do think there might be some useful implications to be drawn from this. For one, the expressions for bias and RMSE show how the performance of the regression adjustments depend on the model parameters, which is useful to know if you’re trying to determine how to set up a simulation study involving PET or PEESE, choose which parameters to vary, or fit a meta-model to the results.

Another, perhaps more pragmatic application would be to use the expressions to compare the bias or accuracy of different forms of regression adjustment. For a given application, should we use PET or PEESE? And should we weight by inverse sampling variances (fixed effects weights) or inverse total variances (random effects weights) or something else (like equal weighting)? We might be able to get some purchase on these questions by computing bias or RMSE for some hypothetical scenarios. Under either the step function model or the Copas model, the performance of a regression adjustment will depend on the distribution of sampling standard errors, . When conducting a meta-analysis, we will know this sample distribution but we won’t know the true values of , , or the parameters of the selection process. Still, we could evaluate the expressions above under some plausible hypothetical scenarios and see whether one regression adjustment is consistently better than the others. This will take some further fleshing out, which I will leave for another day.

References

Bom, P. R. D., & Rachinger, H. (2019). A kinked meta-regression model for publication bias correction. Research Synthesis Methods, 10(4), 497–514. https://doi.org/10.1002/jrsm.1352

Bramley, P., López-López, J. A., & Higgins, J. P. T. (2021). Examining how meta-analytic methods perform in the presence of bias: A simulation study. Research Synthesis Methods, 12(6), 816–830. https://doi.org/10.1002/jrsm.1516

Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2(2). https://doi.org/10.1177/2515245919847196

Hong, S., & Reed, W. R. (2021). Using Monte Carlo experiments to select meta-analytic estimators. Research Synthesis Methods, 12(2), 192–215. https://doi.org/10.1002/jrsm.1467

McShane, B. B., Böckenholt, U., & Hansen, K. T. (2016). Adjusting for publication bias in meta-analysis an evaluation of selection methods and some cautionary notes. Perspectives on Psychological Science, 11(5), 730–749. http://pps.sagepub.com/content/11/5/730.short

Moreno, S. G., Sutton, A. J., Ades, A., Stanley, T. D., Abrams, K. R., Peters, J. L., & Cooper, N. J. (2009). Assessment of regression-based methods to adjust for publication bias through a comprehensive simulation study. BMC Medical Research Methodology, 9(1). https://doi.org/10.1186/1471-2288-9-2

Stanley, T. D. (2008). Meta-regression methods for detecting and estimating empirical effects in the presence of publication selection*. Oxford Bulletin of Economics and Statistics, 70(1), 103–127. https://doi.org/10.1111/j.1468-0084.2007.00487.x

Stanley, T. D. (2017). Limitations of PET-PEESE and other meta-analysis methods. Social Psychological and Personality Science, 8(5), 581–591. https://doi.org/10.1177/1948550617693062

Stanley, T. D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5(1), 60–78.

Footnotes

I will treat the standard errors as known quantities that are independent of the effect size estimates, even though the standard errors are actually stochastic and, for some effect size metrics, correlated with the effect size estimates.↩︎
All of the adjustment methods will run into trouble if small study effects are the result of some mechanism other than selective reporting. For instance, if some primary study feature is systematically, positively associated with effect size and tends to be less common in larger studies, then it will act as a confounder, inducing correlation between effect size and standard error even in the absence of selective reporting.↩︎
With the EK estimator, the kink is an estimated quantity, defined by a preliminary estimator of and . This makes is quite tricky to analyze, but I’m going to gloss over that detail entirely (at least for the remainder of this post).↩︎
For a model with a single step at and a single selection parameter , ↩︎
Sometimes, I get a theoretical itch that I just can’t help but scratch.↩︎

Variance stabilization of Cohen’s d

James E. Pustejovsky — Wed, 23 Jul 2025 00:00:00 GMT

In a recent Psychological Methods paper, Haaf & Rouder (2023) suggest conducting meta-analysis of standardized mean differences after transforming them to point biserial correlations and then applying Fisher’s -transformation. They argue that this leads to a variance-stabilized effect size estimator, which has sampling variance that is constant across varying values of the true parameter . Bartoš et al. (2023) cites Haaf & Rouder (2023) as justification for conducting meta-analysis of standardized mean differences after applying this -to--to- transformation. All this surprised me a bit because the transformation was unfamiliar, although I’ve written about these sorts of effect size conversions before (cf. Pustejovsky, 2014). In other work (Pustejovsky & Rodgers, 2018), I’ve used the variance-stabilizing transformation given in Hedges & Olkin (1985), which is different than this approach. What gives? Is this -to--to- business really variance stabilizing?

The first step in the transformation proposed by Haaf & Rouder (2023) depends on the relative sample sizes of the two groups, which I will parameterize as . Letting , the transformation function is given by

The shape of the transformation is depicted in Figure 1.

Code

Interactive display of simulation results with Observable.js

James E. Pustejovsky — Sun, 19 Jan 2025 00:00:00 GMT

Monte Carlo simulations are an essential and ubiquitous feature of my work as a statistical methodologist. With my students and collaborators, I have conducted simulations for nearly all of my methodological publications (and many of my blog posts, for that matter). For research simulations, I usually use factorial designs involving several factors, each with multiple levels, to explore a big space of model parameters and study design features.¹ If you’ve got the infrastructure to do the computing, then it doesn’t take much further effort to run big multifactor simulations. What does become more challenging is interpreting and making sense of all the results.

For analyzing simulation results, I rely heavily on graphical representations of performance measures (such as bias, root mean-squared error, coverage, etc.). Because the simulation designs are multifactor, the graphs are usually small multiples plots (i.e., facet-grids in ggplot2 terminology). Small multiples plots can be used to represent variation in a performance measure (plotted on the vertical axis) across up to four different factors, if those factors are mapped to the horizontal axis and color (and/or stroke and/or shape), along with the horizontal and vertical facets. For designs with more than that many factors, one needs to find other ways to deal with remaining factors. That might mean creating a bunch of plots that vary across the remaining dimensions, or finding ways to marginalize across those dimensions. It can quickly become unwieldy.

For a long while now, I’ve been looking for easy ways to create more interactive visualizations of simulation results. Interactivity could be a big help with managing dimensionality, by letting a viewer iterate over different values of a parameter to see how a graph changes. Interactivity also seems useful and appealing as a way to focus in (even literally zoom in) on particular results. With the development of Quarto and support for tools like Observable.js, creating such interactive visualizations has evidently gotten much easier, even for someone (yours truly!) who knows next to nothing about web development.
In this post, I’ll give an example of an interactive visualization of simulation results to illustrate what’s possible. I’ll also comment on some alternative approaches that I’ve seen and note how they differ from the approach demonstrated here.

An example

In my previous post on bootstrap confidence intervals for correlation coefficients, I represented the results of a small simulation with graphs like this one:

This graph encodes four distinct factors:

sample size on the horizontal axis
correlation parameter in the column facet
degrees of freedom in the row facet
confidence interval type, represented with color.

This was a fairly small simulation (by my standards) of a relatively simple problem. But even here, some interactivity would be appealing to allow a viewer to zoom in on a panel. It would also be nice to be able to focus in on just one or two of the interval types or to easily remove an interval type that you don’t care about.

Below is my attempt at an interactive version of this graphic. I specified the interactive tools with Observable’s inputs library. I developed the figure with Observable’s plot library. I will not get into the details of the programming because I’m still very new to both of these tools.² Suffice it to say, I more or less followed the first example in the Quarto guide to Observable, then fiddled with the plotting code to make it look more like a ggplot2 graph.

A bigger simulation of bootstrap confidence interval variations

James E. Pustejovsky — Sat, 18 Jan 2025 00:00:00 GMT

In my previous post, I demonstrated some new brand-new additions to the simhelpers package: utilities for calculating bootstrap confidence intervals. That post walked through validating the functions against the results of other packages and showing how they can be used to extrapolate confidence interval coverage rates using fewer bootstraps than one would want to see for a real data analysis. In the course of illustrating the functions, I set up a small simulation study (with just one set of parameters) to show how the extrapolation technique works. In this post, I’m going to expand these simulations across a bigger set of conditions in a multi-factor simulation. This exercise is mostly an excuse to showcase some of the other useful features of simhelpers—especially the bundle_sim() function for creating simulation drivers and the evaluate_by_row() function for executing simulations across a grid of parameter values.

The goal of the simulation is to evaluate how the bootstrap CIs work for estimating the Pearson correlation from a bivariate data-generating process. I will compare the performance of four different confidence intervals for the correlation coefficient: 1) the Fisher-z interval, which is derived assuming bivariate normality of the measurements; 2) a percentile bootstrap CI; 3) a studentized bootstrap CI; and 4) Efron’s bias-corrected-and-accelerated bootstrap CI. For the bootstrap intervals, I will use my implementation of the Boos & Zhang (2000) extrapolation technique (which I described in much greater detail here).

Code

Bootstrap confidence interval variations

James E. Pustejovsky — Wed, 15 Jan 2025 00:00:00 GMT

I recently added some new utilities for calculating bootstrap confidence intervals to the simhelpers package. The functions are designed to make it a bit more convenient to implement Monte Carlo simulations of bootstrap CIs, including when using an extrapolation technique suggested by Boos & Zhang (2000), which I wrote about a while ago. With the latest update, the package now provides options for a bunch of different variants of bootstrap CIs, including:

the normal CI, which uses bootstrapping to estimate a standard error and bias-correction, then takes plus or minus a normal critical value times the SE;
the studentized CI, which uses the bootstrap distribution of a t-statistic rather than the point estimator;
the percentile CI, which takes percentiles of the bootstrap distribution as the end-points of the CI;
the so-called “basic” CI, which is similar to the percentile CI but pivots the bootstrap distribution around the point estimator;
a bias-corrected version of the percentile CI;
Efron’s (1987) bias-corrected-and-accelerated CI, which is supposed to provide more accurate coverage levels than alternative CIs for some classes of estimators.

These CI variants are also implemented in other packages. Most notably, all of them are implemented in boot (Canty & Ripley, 2024), the venerable R package companion to the Davison & Hinkley (1997) book on bootstrapping. Although very full-featured and widely used (it has 377 reverse dependencies!), the boot package does not offer a super-friendly user experience. Its output is a unwieldy, there are several quirks to its naming conventions, and its function for confidence interval calculations require the user to implement the bootstrap resampling calculations through boot::boot(), which is sometimes a bit awkward. Newer packages that provide some of the same methods include infer (Couch et al., 2021) and rsample (Frick et al., 2024). Both of these offer pipe-friendly workflows, but neither provides the full slate of CI variants. As with boot, these packages also lock in (or at least strongly nudge) the user to the package’s resampling tools. And none of the packages support a workflow for the Boos & Zhang (2000) extrapolation technique.

In this post, I’ll demonstrate the simhelpers implementation of these different CI variants with an example, compare the results to implementations in other packages, and then show how the simhelpers implementation can be used for the Boos & Zhang (2000) extrapolation technique.

Bootstrap confidence intervals for a mean

Let me demonstrate the confidence intervals using a simple example of estimating a correlation. For illustrative purposes, I’ll use the swiss dataset of various socio-demographic measures of provinces in Switzerland from the late 19th century. I’ll look at the correlation between level of education and a measure of fertility:

Code

A quirk of nlme::varIdent

James E. Pustejovsky — Sat, 28 Dec 2024 00:00:00 GMT

The nlme package is a venerable, now quite old package for fitting hierarchical models in R. Compared to the better-known lme4 package, nlme provides greater flexibility for specifying dependence structures at the finest-grain level of the model, such as allowing for heteroskedastic errors or correlated errors that follow an auto-regressive structure within each higher-level unit.¹ Due to its quite advanced age (the first entry in the changelog is from December 23, 1999!), it is perhaps not surprising that the package has a number of what one might call quirks—odd, counter-intuitive behaviors that are unlike the behavior of other core model-fitting functions in R. A few of my past posts have documented some of these quirks. Here’s another one.

¹ nlme is a minor obsession for me because Several packages that I’ve worked on provide formal enhancements to or are at least very closely connected to it. The clubSandwich package provides cluster-robust standard errors for fitted models, and it supports models fitted with nlme::lme() and nlme::gls(). The lmeInfo package provides analytic derivatives for models fitted with nlme::lme() and nlme::gls(). The scdhlm package for calculating effect sizes from single-case designs relies heavily on nlme::lme() and on lmeInfo::g_mlm().

varIdent()

In nlme’s syntax, varIdent() is used to specify a model in which the variance of the lowest-level error term differs depending on a categorical covariate. I’ll call this a “heteroskedastic error” model. Say that we have repeated measures (indexed by ) nested within units (indexed by ), so that is measurement on unit for and . Let denote the category of observation , and suppose that the last category serves as a reference level. Say we have some model with predictors and potentially also with random effects terms described by : where The coefficients capture how the variance of errors in category differ from the variance of errors in category . The question is, how can one control the reference level (i.e., choose which category is ) when fitting this model?

Let me give a few examples of models with this feature using Orthodont, one of the standard datasets included in the nlme package. Below I do a bit of tidying up on the factors in this dataset.

library(tidyverse)
library(nlme)
data("Orthodont")

Orthodont <- 
  Orthodont %>%
  as.data.frame() %>%
  mutate(
    age = as.integer(age),
    Subject = factor(Subject, levels = sort(levels(Subject)), ordered = FALSE),
    Sex_rev = fct_rev(Sex),
    Sex3 = factor(if_else(
      Subject %in% c("M11","M12","M13","M14","M15","M16"),
      "Non-binary", 
      Sex), levels = c("Female","Male","Non-binary")
    )
  )

Here is a simple example of a heteroskedastic error model, without any random effects or any other bells or whistles.

gls1 <- gls(
  distance ~ age + Sex, 
  weights = varIdent(form = ~ 1 | Sex),
  data = Orthodont
)
summary(gls1)

Generalized least squares fit by REML
  Model: distance ~ age + Sex 
  Data: Orthodont 
       AIC      BIC    logLik
  494.3251 507.5949 -242.1626

Variance function:
 Structure: Different standard deviations per stratum
 Formula: ~1 | Sex 
 Parameter estimates:
     Male    Female 
1.0000000 0.9378976 

Coefficients:
                Value Std.Error   t-value p-value
(Intercept) 17.811621 1.1119255 16.018718       0
age          0.650648 0.0975569  6.669421       0
SexFemale   -2.321023 0.4396048 -5.279794       0

 Correlation: 
          (Intr) age   
age       -0.965       
SexFemale -0.173  0.000

Standardized residuals:
        Min          Q1         Med          Q3         Max 
-2.58304984 -0.65117352 -0.02923653  0.51805238  2.30992386 

Residual standard error: 2.329342 
Degrees of freedom: 108 total; 105 residual

The reference level of the variance structure is Male.²

² For ease of exposition, let me make a little helper function that pulls out the reference level of the variance structure.

ref_level <- function(mod) {
  attr(mod$modelStruct$varStruct, "groupNames")[1]
}
ref_level(gls1)

[1] "Male"

Here’s another gls model, this time allowing the errors to be correlated within subject:

gls2 <- update(
  gls1, 
  correlation = corCompSymm(form = ~ 1 | Subject)
)
ref_level(gls2)

[1] "Female"

The reference category has now changed to Female. The same thing happens if I fit a heteroskedastic error model using lme():

lme1 <- lme(
  distance ~ age + Sex, 
  random = ~ 1 | Subject, 
  weights = varIdent(form = ~ 1 | Sex),
  data = Orthodont
)
ref_level(lme1)

[1] "Female"

It would be nice to be able to control the reference level.³

³ I’m not the only one seeking to do this. Someone on the R-SIG-mixed-models listserv had the same query in July of 2013, but none of the responses provide a solution.

What doesn’t work

The usual way that one would specify reference levels is to set a contrast for the factor:

contrasts(Orthodont$Sex) <- contr.treatment(2, base = 1)

But this does not seem to have any effect on the selected reference level of the variance structures:

update(gls1, data = Orthodont) |> ref_level()

[1] "Male"

update(gls2, data = Orthodont) |> ref_level()

[1] "Female"

update(lme1, data = Orthodont) |> ref_level()

[1] "Female"

Nor does using a factor with the levels in reverse order:

update(gls1, weights = varIdent(form = ~ 1 | Sex_rev)) |> ref_level()

[1] "Male"

update(gls2, weights = varIdent(form = ~ 1 | Sex_rev)) |> ref_level()

[1] "Female"

update(lme1, weights = varIdent(form = ~ 1 | Sex_rev)) |> ref_level()

[1] "Female"

Emma Knight suggested on R-SIG-mixed-models that the reference level is selected based on the category with the largest number of observations (which is Male). That could be what determines the reference level in gls1 (though it clearly can’t be so for gls2 or lme). I’ve created another factor called Sex3 where the largest category is Female. If this theory is true, then the reference level of gls1 should change to Female:

update(gls1, weights = varIdent(form = ~ 1 | Sex3)) |> ref_level()

[1] "Male"

Nothing changes with the other models either:

update(gls2, weights = varIdent(form = ~ 1 | Sex3)) |> ref_level()

[1] "Female"

update(lme1, weights = varIdent(form = ~ 1 | Sex3)) |> ref_level()

[1] "Female"

What about the order of the data? There is a bug report documenting that varIdent() depends on the sort order of the categorical grouping variable when used in a gls() call.⁴ Sorting the data in descending order of Sex3 does affect the reference level for gls1:

⁴ This StackOverflow question notes the same issue and links to a patch from Ben Bolker.

Orthodont_sort <- arrange(Orthodont, Sex3)
Orthodont_rev <- arrange(Orthodont, desc(Sex3))
head(Orthodont_sort$Sex, 4)

[1] Female Female Female Female
attr(,"contrasts")
       2
Male   0
Female 1
Levels: Male Female

head(Orthodont_rev$Sex3, 4)

[1] Non-binary Non-binary Non-binary Non-binary
Levels: Female Male Non-binary

update(
  gls1, 
  data = Orthodont_sort
) |> 
  ref_level()

[1] "Female"

update(
  gls1, 
  weights = varIdent(form = ~ 1 | Sex3),
  data = Orthodont_rev
) |> 
  ref_level()

[1] "Non-binary"

However, this doesn’t seem to affect gls2 or lme1 at all:

update(
  gls2, 
  data = Orthodont_sort
) |> 
  ref_level()

[1] "Female"

update(
  gls2, 
  weights = varIdent(form = ~ 1 | Sex3), 
  data = Orthodont_rev
) |> 
  ref_level()

[1] "Female"

update(
  lme1, 
  data = Orthodont_sort
) |> 
  ref_level()

[1] "Female"

update(
  lme1, 
  weights = varIdent(form = ~ 1 | Sex3), 
  data = Orthodont_rev
) |> 
  ref_level()

[1] "Female"

Very curious.

How to control the reference level

I posted about this issue to R-SIG-mixed-models with some of the above notes. Sebastian Meyer provided a very helpful response, noting that

lme() internally reorders the data by the grouping factor(s) before initialization, so the order of that factor (‘Subject’ in your example…) will indirectly determine the reference level of varIdent(), regardless of how your data or strata levels are ordered originally.

Similar re-ordering behavior occurs in gls() when the model includes a grouping structure, as it does in gls2 because of the corCompSymm() correlation structure. This insight provides a key for controlling the reference level: for a given categorical variable, the reference level will be taken as the level of the first observation in the first level of the grouping factor. Thus, for models with grouping structure, one can set the reference level to Male by re-ordering the levels of Subject so that a unit with Sex = 'Male' occurs first.

Orthodont$Subject2 <- fct_relevel(Orthodont$Subject, "M01")
table(Orthodont$Subject2) |> head()


M01 F01 F02 F03 F04 F05 
  4   4   4   4   4   4

update(
  gls2, 
  correlation = corCompSymm(form = ~ 1 | Subject2)
) |> 
  ref_level()

[1] "Male"

update(
  lme1, 
  random = ~ 1 | Subject2
) |> 
  ref_level()

[1] "Male"

For models without grouping structure, the reference level is indeed controlled by sort order, as if there were only a single group. To control the reference level, one would need to sort the data accordingly. For example, to make Female the reference level in gls1, the first observation in the data needs to have Sex = 'Female'. This is so in Orthodont_sort:

update(gls1, data = Orthodont_sort) |> ref_level()

[1] "Female"

What if the categorical variable varies within the grouping factor? For instance, say that we want to fit a model with heteroskedastic errors varying by age of the subject, which varies from 8 to 14 across the repeated measures of each participant. Here’s a basic heteroskedastic error gls():

gls3 <- gls(
  distance ~ age + Sex, 
  weights = varIdent(form = ~ 1 | age),
  data = Orthodont
)

And here is one with a correlation structure:

gls4 <- update(
  gls3, 
  correlation = corCompSymm(form = ~ 1 | Subject)
)

In both cases, the reference level is the youngest age:

c(ref_level(gls3), ref_level(gls4))

[1] "8" "8"

To set this to something else in the first model, just re-arrange the first few rows:

Orthodont2 <- Orthodont[c(4,1,2,3,5:108),]

update(
  gls3, 
  data = Orthodont2
) |> 
  ref_level()

[1] "14"

However, this doesn’t work for gls4⁵ because the first level of the grouping factor is not the first one to appear in the data.⁶ But we could sort the data by the grouping factor and then by age so that the desired reference level appears first:

⁵ See

update(gls4, data = Orthodont2) |> ref_level()

[1] "8"

⁶ The first level of Subject corresponds to rows 65 through 68:

which(levels(Orthodont2$Subject)[1] == Orthodont2$Subject)

[1] 65 66 67 68

Orthodont2_sort <- 
  Orthodont %>%
  arrange(Subject, desc(age))

head(Orthodont2_sort)

   distance age Subject    Sex Sex_rev   Sex3 Subject2
68     23.0  14     F01 Female  Female Female      F01
67     21.5  12     F01 Female  Female Female      F01
66     20.0  10     F01 Female  Female Female      F01
65     21.0   8     F01 Female  Female Female      F01
72     25.5  14     F02 Female  Female Female      F02
71     24.0  12     F02 Female  Female Female      F02

update(gls4, data = Orthodont2_sort) |> ref_level()

[1] "14"

It’s not elegant, nor particularly intuitive, but it seems to work.

Setting a `value`

Sebastian also suggested another technique for directly controlling the reference level of the variance structure:

It seems that only if there are more than two strata, the reference level for varIdent() can be chosen via a named initial parameter ‘value’, leaving out the desired reference level, but I haven’t tested if this works as intended with both gls() and lme(). It would then make sense to support such a choice also in the case of only two strata, so for a single parameter.

Setting the argument value of varIdent() specifies initialization values for the variance structure. Sebastian’s suggestion is to specify values for every level of the categorical variable except for the desired reference level. Let me give this a try with the Sex3 variable, which has three distinct levels.

gls_M <- update(
  gls1, 
  weights = varIdent(form = ~ 1 | Sex3)
)

gls_F <- update(
  gls1, 
  weights = varIdent(
    value = c(`Non-binary` = 0.01, Male = 0.01), 
    form = ~ 1 | Sex3
  )
)

gls_NB <- update(
  gls1, 
  weights = varIdent(
    value = c(Female = 0.01, Male = 0.01), 
    form = ~ 1 | Sex3
  )
)

coef(gls_M$modelStruct$varStruct, unconstrained = FALSE)

Non-binary     Female 
 0.7438755  0.8608531

coef(gls_F$modelStruct$varStruct, unconstrained = FALSE)

Non-binary       Male 
  0.864114   1.161639

coef(gls_NB$modelStruct$varStruct, unconstrained = FALSE)

  Female     Male 
1.157254 1.344311

Indeed, this works for models with variance structures but no correlation structures or random effects. Likewise for gls models that include grouping structure:

update(
  gls2, 
  weights = varIdent(form = ~ 1 | Sex3)
) |>
  ref_level()

[1] "Female"

update(
  gls2, 
  weights = varIdent(
    value = c(`Non-binary` = 0.01, Female = 0.01), 
    form = ~ 1 | Sex3
  )
) |>
  ref_level()

[1] "Male"

update(
  gls2, 
  weights = varIdent(
    value = c(Female = 0.01, Male = 0.01), 
    form = ~ 1 | Sex3
  )
) |> 
  ref_level()

[1] "Non-binary"

And for lme models with random effects:

update(lme1, weights = varIdent(form = ~ 1 | Sex3)) |>
  ref_level()

[1] "Female"

update(
  lme1, 
  weights = varIdent(
    value = c(`Non-binary` = 0.01, Female = 0.01), 
    form = ~ 1 | Sex3
  )
) |>
  ref_level()

[1] "Male"

update(
  lme1, 
  weights = varIdent(
    value = c(Female = 0.01, Male = 0.01), 
    form = ~ 1 | Sex3
  )
) |> 
  ref_level()

[1] "Non-binary"

However, as Sebastian noted, it does not work for categories with only two levels (so only one non-reference level):

update(
  gls1, 
  weights = varIdent(
    value = c(Male = 0.01), 
    form = ~ 1 | Sex
  )
) |>
  ref_level()

[1] "Male"

update(
  gls2, 
  weights = varIdent(
    value = c(Female = 0.01), 
    form = ~ 1 | Sex
  )
) |>
  ref_level()

[1] "Female"

update(
  lme1, 
  weights = varIdent(
    value = c(Female = 0.01), 
    form = ~ 1 | Sex
  )
) |>
  ref_level()

[1] "Female"

Thus, this is only a partial solution. I agree with Sebastian that it would be useful to support user-specified levels within the value argument, even for factors with just one non-reference level.

Colophon

sessioninfo::session_info()

─ Session info ───────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.4.1 (2024-06-14 ucrt)
 os       Windows 11 x64 (build 22631)
 system   x86_64, mingw32
 ui       RTerm
 language (EN)
 collate  English_United States.utf8
 ctype    English_United States.utf8
 tz       America/Chicago
 date     2024-12-28
 pandoc   3.2 @ C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools/ (via rmarkdown)

─ Packages ───────────────────────────────────────────────────────────────────
 ! package     * version date (UTC) lib source
 P cli           3.6.3   2024-06-21 [?] RSPM
 P colorspace    2.1-1   2024-07-26 [?] RSPM
 P digest        0.6.37  2024-08-19 [?] RSPM (R 4.4.0)
 P dplyr       * 1.1.4   2023-11-17 [?] RSPM
 P evaluate      0.23    2023-11-01 [?] RSPM (R 4.4.0)
 P fansi         1.0.6   2023-12-08 [?] RSPM
 P fastmap       1.1.1   2023-02-24 [?] RSPM (R 4.4.0)
 P forcats     * 1.0.0   2023-01-29 [?] RSPM
 P generics      0.1.3   2022-07-05 [?] RSPM
 P ggplot2     * 3.5.1   2024-04-23 [?] RSPM
 P glue          1.8.0   2024-09-30 [?] RSPM (R 4.4.0)
 P gtable        0.3.6   2024-10-25 [?] RSPM (R 4.4.0)
 P hms           1.1.3   2023-03-21 [?] RSPM
 P htmltools     0.5.7   2023-11-03 [?] RSPM (R 4.4.1)
 P htmlwidgets   1.6.4   2023-12-06 [?] RSPM
 P jsonlite      1.8.8   2023-12-04 [?] RSPM
 P knitr         1.47    2024-05-29 [?] RSPM (R 4.4.0)
 P lattice       0.22-5  2023-10-24 [?] RSPM (R 4.4.1)
 P lifecycle     1.0.4   2023-11-07 [?] RSPM
 P lubridate   * 1.9.3   2023-09-27 [?] RSPM
 P magrittr      2.0.3   2022-03-30 [?] RSPM
 P munsell       0.5.1   2024-04-01 [?] RSPM
 P nlme        * 3.1-166 2024-08-14 [?] RSPM
 P pillar        1.9.0   2023-03-22 [?] RSPM
 P pkgconfig     2.0.3   2019-09-22 [?] RSPM
 P purrr       * 1.0.2   2023-08-10 [?] RSPM
 P R6            2.5.1   2021-08-19 [?] RSPM
 P readr       * 2.1.5   2024-01-10 [?] RSPM
   renv          1.0.5   2024-02-29 [1] RSPM (R 4.4.0)
 P rlang         1.1.4   2024-06-04 [?] RSPM
 P rmarkdown     2.27    2024-05-17 [?] RSPM
 P rstudioapi    0.17.1  2024-10-22 [?] RSPM (R 4.4.0)
 P scales        1.3.0   2023-11-28 [?] RSPM
 P sessioninfo   1.2.2   2021-12-06 [?] RSPM (R 4.4.0)
 P stringi       1.8.4   2024-05-06 [?] RSPM
 P stringr     * 1.5.1   2023-11-14 [?] RSPM
 P tibble      * 3.2.1   2023-03-20 [?] RSPM
 P tidyr       * 1.3.1   2024-01-24 [?] RSPM
 P tidyselect    1.2.1   2024-03-11 [?] RSPM
 P tidyverse   * 2.0.0   2023-02-22 [?] RSPM
 P timechange    0.3.0   2024-01-18 [?] RSPM
 P tzdb          0.4.0   2023-05-12 [?] RSPM
 P utf8          1.2.4   2023-10-22 [?] RSPM
 P vctrs         0.6.5   2023-12-01 [?] RSPM
   withr         3.0.2   2024-10-28 [1] RSPM (R 4.4.0)
 P xfun          0.45    2024-06-16 [?] RSPM (R 4.4.0)
 P yaml          2.3.8   2023-12-11 [?] RSPM (R 4.4.0)

 [1] C:/Users/jamespustejovsky/Documents/jepusto-quarto/renv/library/R-4.4/x86_64-w64-mingw32
 [2] C:/Users/jamespustejovsky/AppData/Local/R/cache/R/renv/sandbox/R-4.4/x86_64-w64-mingw32/e0da0d43

 P ── Loaded and on-disk path mismatch.

──────────────────────────────────────────────────────────────────────────────

Beta-density selection models for meta-analysis

James E. Pustejovsky — Fri, 08 Nov 2024 00:00:00 GMT

I still have meta-analytic selection models on the brain. As part of an IES-funded project with colleagues from the American Institutes for Research, I’ve been working on developing methods for estimating selection models that can accommodate dependent effect sizes. We’re looking at two variations of p-value selection models: step-function selection models similar to those proposed by Hedges (1992) and Vevea & Hedges (1995) and beta-density models as developed in Citkowicz & Vevea (2017). Both models fall within the broader class of -value selection models, which make explicit assumptions about the probability of observing an effect size, given its statistical significance level and sign. I’ve already described how step-function models work (see this previous post) and, as a bit of a detour, I also took a stab at demystifying the Copas selection model. In this post, I’ll look at the beta-density model, highlight how it differs from step-function models, and do a deep dive into some of the tweaks to the model that we’ve implemented as part of the project.

The beta-density selection model

The beta-density selection model is another entry in the class of -value selection models. Like the step-function model, it consists of a set of assumptions about how effect size estimates are generated prior to selection (the evidence-generation process), and a set of assumptions about how selective reporting happens as a function of the effect size estimates (the selective reporting process). In both the beta-density and step-function models, the evidence-generating process is a random effects model: where denotes an effect size estimate prior to selective reporting and denotes its (known) standard error.¹

The only difference between the beta-density and step-function selection models is the functional form of the selective reporting process. Letting be an indicator equal to 1 if is reported and be the one-sided p-value corresponding to the effect size estimate, the selective reporting process describes the shape of . The step-function model uses a piece-wise constant function with level shifts at analyst-specified significance thresholds (see Figure 1). The beta-density model instead uses…wait for it…a beta density kernel function, which can take on a variety of smooth shapes over the unit interval (see Figure 2).

Figure 1: Two-step selection model with

Figure 2: Beta-density selection model with

In the original formulation of the beta density model, the selection function is given by where the parameters and must be strictly greater than zero, and corresponds to a constant probability of selection (i.e., no selective reporting bias). Because is a function of the effect size estimate and its standard error, the selection function can also be written as where is the cumulative standard normal distribution. In proposing the model, Citkowicz & Vevea (2017) argued that the beta-density function provides a parsimonious expression of more complex forms of selective reporting than can easily be captured by a step function. For instance, the beta density in Figure 2 is smoothly declining from through the psychologically salient thresholds and and beyond. In order to approximate such a smooth curve with a step function, one would have to use many thresholds and therefore many more than the two parameters of the beta density.

A truncated beta-density

Although using smoothly varying selection probabilities may seem appealing, the beta density also comes with an important limitation, highlighted in a commentary by Hedges (2017). For some parameter values, the beta density implies selection probabilities that differ by many orders of magnitude. These extreme differences in selection probability can imply implausible selection processes, in which hundreds of non-significant effect size estimates would need to go unreported to observe a sample of a few dozen findings. Extreme differences in selection probabilities make the model highly sensitive to the inclusion or exclusion of some effect size estimates because the influence of each estimate is driven by the inverse of its selection probability (Hedges, 2017).

As a means to mitigate these issues with the beta density, we consider a modification of the model where the function is truncated at -values larger or smaller than certain thresholds. For user-specified thresholds and , let . The truncated beta density is then Setting the truncation thresholds at psychologically salient levels such as and (which correspond to positive and negative effects that are statistically significant based on two-sided tests with ) gives something kind of like the step-function selection model, but with smoothly varying selection probabilities in the interior. Alternately, one could set the second threshold at (corresponding to an effect of zero) so that all negative effect size estimates have a constant probability of selection, but positive effect size estimates that are non-significant have smoothly varying selection probabilities up to the point where .

Distribution of observed effect sizes

Just as with the step-function selection model, the assumptions of the evidence-generating process (1) and the selection process (2) can be combined to obtain the distribution of observed effect sizes, with

Here is an interactive graph showing the distribution of the effects prior to selection (in grey) and the distribution of observed effect sizes (in blue) based on the truncated beta-density selection model. Initially, the truncation points are set at and and the selection parameters are set to and , but you can change these however you like. Below the effect size distribution are the moments of the distribution (computed numerically) and a graph of the truncated beta density that determines the selection probabilities.

Copas selection models for meta-analysis

James E. Pustejovsky — Sat, 27 Jul 2024 00:00:00 GMT

Recently, I’ve been studying and tinkering with meta-analytic models for selective reporting of findings from primary studies. I’ve mostly focused on step-function selection models (Hedges, 1992; Hedges & Vevea, 1996; Vevea & Hedges, 1995; Vevea & Woods, 2005), which assume that selective reporting of primary study results is driven by the -values of the effect size estimates (and specifically, where those -values fall relative to certain thresholds). This is not the only form of selection model, though. One prominent alternative model, described in a series of papers by John Copas and colleagues (Copas, 1999; Copas & Li, 1997; Copas & Shi, 2000, 2001), is based on distinctly different assumptions than the step-function model. The Copas selection model has received a fair amount of attention in the literature on meta-analysis of medical and epidemiological research, but relatively little attention from the social science side of things. In this post, I’ll give an overview of the Copas model and try to highlight some of the ways that it differs from step function selection models.

The Copas selection model

Similar to the step function selection model, the Copas model is composed of two components: an evidence-generating process that describes the distribution of effect size estimates prior to selection and a selective reporting process that describes how selective reporting happens as a function of the effect size estimates and associated quantities like standard errors and -values. The evidence-generating process is just a conventional random effects model. Letting denote an effect size estimate prior to selective reporting, denote the corresponding effect size parameter, and denote its (known) standard error, we assume that effect size estimate is drawn from a normal distribution with mean and standard deviation : We further assume that the effect size parameters are drawn from a normal distribution with overall average effect size and standard deviation : The full evidence-generating process can also be written as a random effects model with two sources of error: where and .

The second component is where the Copas model diverges from the step-function model. The Copas model posits that selection is driven by a latent index that is normally distributed and correlated with , with mean that is a function of : where and . With this specification, is reported (and thus available for inclusion in a meta-analysis) if and is otherwise unreported. Following the notation of my previous post, I’ll use to denote a binary indicator equal to 1 if is reported and otherwise equal to zero. With this notation, .

Copas & Shi (2001) calls the propensity for selection. Under the posited model, the selection propensity depends both on the precision of the study through (because the mean of depends on the inverse of ) and on the magnitude of the reported effect size estimate (because is correlated with ). The parameter controls the overall probability of selection, irrespective of the findings; controls how strongly selection depends on study precision; and controls how strongly selection depends on the magnitude of the effect size estimate itself.

The Copas model posits a stable correlation between the selection propensity index and the samplng error of the random effects model, . If there is heterogeneity in the effect size parameters and heterogeneity in the sampling variances, then the correlation between and will be lower than and will differ from study to study. With a little variance algebra, it can be seen that Using the correlation, we can write the selection propensity index in terms of a regression on : Thus, given the effect size estimate and its standard error , the probability that finding is reported is (Copas & Shi, 2001, Equation 4; Hedges & Vevea, 2005, Equation 9.6).

Distribution of observed effect sizes

The observed effect size estimates follow a distribution equivalent to that of . It’s possible to find an analytic expression for the density of the observed effect sizes. Using Bayes Theorem, the density of the observed ’s is where the first term in the numerator is given in (6), the second term in the numerator can be derived from (3), and the denominator can be derived from (4). Letting and substituting everything into the above gives This density is identical to that of the generalized skew-normal distribution (Arellano-Valle & Azzalini, 2006).

Here is an interactive graph showing the distribution of the effects prior to selection (in grey) and the distribution of observed effect sizes (in blue) based on the Copas selection model.¹ Initially, the selection parameters are set to , and , but you can change these however you like. Try wiggling up and down to see how the bias and shape of the density changes.

Step-function selection models for meta-analysis

James E. Pustejovsky — Tue, 09 Jul 2024 00:00:00 GMT

In a recent post I looked at the distribution of statistically significant effect sizes in studies that report multiple outcomes, when those studies are subject to a certain form of selective reporting. I considered a model where each effect size within the study is more likely to be reported when it is affirmative—or statistically significant and in the hypothesized direction—than when it is non-affirmative. Because studies with multiple effect sizes are a very common occurrence in social science meta-analysis, it’s interesting to think about how this form of selection leads to distortions of the results that are actually reported and available for meta-analysis.

In this post, I want to look a different scenario that is simpler in one respect but more complicated in another. Simpler, in that I’m going to ignore dependence issues and just think about the distribution of one effect size at a time. More complicated, in that I’m going to look at a more general model for how selective reporting occurs, which I’ll call the step-function selection model.

The step-function selection model

The step-function selection model was introduced by Hedges (1992) and has been further expanded, tweaked, and studied in a bunch of subsequent work. The model has two components: a set of assumptions about how effect size estimates are generated prior to selection (the evidence-generation process), and a set of assumptions about how selective reporting happens as a function of the effect size estimates (the selective reporting process).

In the original formulation, the evidence-generation process is a random effects model. Letting denote an effect size estimate prior to selective reporting and denote its (known) standard error, we assume that just as in the conventional random effects model. Here is the overall average effect size and is the standard deviation of the effect size parameter distribution.

For the second component, we assume that the selective-reporting process is fully determined by the statistical significance and sign of the effect size estimates. We can therefore formalize the selective reporting process in terms of the one-sided p-values of the effect size estimates. Assuming that the degrees of freedom are large enough to not worry about, the one-sided p-value for effect size is a transformation of the effect size and its standard error: In the step-function model, we assume that the probability that an effect size estimate is reported (and thus available for meta-analysis) depends on where this one-sided p-value lies relative to a pre-specified set of significance thresholds, . These thresholds define a set of intervals, each of which can have a different probability of selection. Let be a binary indicator equal to 1 if is reported and otherwise equal to zero. The selective reporting process is then Note that the selection probability for the lowest interval is fixed to 1 because we can’t estimate the absolute probability that an effect size estimate is reported. The remaining parameters of the selection process therefore each represent a ratio of the probability of reporting an effect size estimate falling in a given interval to the probability of reporting an effect size estimate falling in the lowest interval.

In practice, the analyst will need to specify the thresholds of the significance intervals in order to estimate the model. One common choice is to use only a single threshold at , which corresponds to a two-sided level of .05—Fisher’s vaunted criteria for when a result should be considered significant. This is the so-called “three-parameter” selection model, where the parameters of interest are the average effect size , the heterogeneity SD , and the relative selection probability . Other possible choices for thresholds might be:

A single threshold at , so that negatively-signed effect size estimates have a different selection probability than positively-signed estimates;
A two-threshold model with and (I like to call this a four-parameter selection model); or
A model with thresholds for significant, positive results at , for the sign of the estimate at , and for statistically significant results in the opposite of the expected direction at .

Many other choices are possible, of course.

Distribution of observed effect sizes

Equations (1) and (3) are sufficient to describe the distribution of effect sizes actually observed after selection. If we let and denote effect size estimates that are actually observed, then the distribution of is the same as that of . By Bayes Theorem, For the specific distributional assumptions of the step-function selection model, we can find an expression for the exact form of (4). In doing so, it will be useful to define a further random variable—call it —that is equal to the p-value interval into which effect size falls. For a given effect size with standard error , these intervals are equivalent to intervals on the scale of the outcome, with thresholds . Now, let’s define as and as the corresponding interval for the observed effect size . Note that (3) is equivalent to writing the relative selection probabilities as a function of : Also note that, prior to selection, the effect size estimate has marginal variance , so we can write , where is the standard normal density. We can then write the distribution of the observed effect size estimates as where with and where we take , , and (Hedges & Vevea, 2005). Note that , the probability that an effect size estimate with precision will be observed.

The observed effect size estimates follow what we might call a “piece-wise normal” distribution. For a given and given the interval into which the effect size falls, the effect size follows a truncated normal distribution. Formally, Furthermore, the distribution of is given by which will be useful for deriving moments of the distribution of .

Here is an interactive graph showing the distribution of the effects prior to selection (in grey) and the distribution of observed effect sizes (in blue) based on a four-parameter selection model with selection thresholds of and . Initially, the selection parameters are set to and , but you can change these however you like.

Simulating bootstrap confidence intervals

James E. Pustejovsky — Mon, 01 Jul 2024 00:00:00 GMT

Bootstrapping is a brilliant statistical idea. I realize that it’s become pretty routine, these days, and maybe has lost some of the magical veneer it once had, but I will fess up that I am still a fanboy. I teach my students to think of bootstrapping as your back-pocket tool for inference: if you’re not sure how to get a standard error or a confidence interval for some quantity that you’ve estimated, bootstrapping is probably a good place to start—and possibly good enough.

The core idea behind bootstrap inference is to use simulation to emulate the process of collecting a sample, and to take the variation across simulation results as a proxy for the sampling uncertainty in your real data. A sample of simulated results (which we might call booties) form the basis of bootstrap standard errors, confidence intervals, hypothesis tests, and the like. The simulation process can be as simple as re-sampling observations from your original data, in which case the original sample is a stand-in for the population and each re-sample is a replication of the process of collecting a dataset for analysis. Other simulation processes are possible too, which can account for more complex study designs or make stronger use of parametric modeling assumptions.

Across all its variations, bootstrapping involves using brute-force computation to do something that would otherwise take a bunch of math (like delta-method calculations). This same feature makes it a royal pain to study bootstrap techniques using Monte Carlo simulation. In a Monte Carlo simulation, one has to generate hypothetical data, apply an estimator to each simulated dataset, and then repeat the whole process times (for some large ) to see the sampling distribution of the estimator. But if the estimator itself requires doing a bunch of re-sampling (or some other form of simulation) just to get one confidence interval, then the computation involved in the whole process can be pretty demanding. In this post, I’ll look at a trick that can make the whole process a bit more efficient.

Boos & Zhang (2000) proposed a technique for approximating the power of a bootstrap test, while using only a finite, feasibly small number of replicates. Their work focused on hypothesis tests, but the approach applies directly to confidence interval simulations as well. The trick is to calculate the bootstrap test several times for each replication, each time with a different number of booties, and then extrapolate to .

Simulating a bootstrap

Suppose we’re doing a simulation to evaluate the properties of a some sort of bootstrap confidence interval, where the main performance characteristics of interest are the coverage rate and the average width of the intervals. The simulation process will need to be repeated times, and will need to be fairly large to control the Monte Carlo error in the simulation results (e.g., to ensure that Monte Carlo standard errors for coverage rates will be less than 1%, we’ll need to run at least 2500 replications). For each of those replications, we’ll need to do bootstrap replications (ahem, booties!) to obtain a confidence interval. On top of all that, we will also often want to repeat the whole process across a number of different conditions. For example, the first simulation in Joshi et al. (2022) involved a factorial design (176 total conditions), with replications per condition and booties per replication. It quickly adds up—or rather, it quickly multiplies out!

A challenge here is that the power of a bootstrap hypothesis test depends on (as discussed in Davidson & MacKinnon, 2000); by extension, so too does the coverage rate and width of a bootstrap confidence interval. If we use only a small number of booties, then the simulation results will under-state coverage relative to what we might obtain when applying the method in practice (where we might use something more like or ) But if we use a realistically large number of booties, then our simulations will take forever to finish…or we’ll have to cut down on the number of conditions include…or both…or maybe we’ll decide that the delta-method math isn’t so bad after all.

Extrapolating

Now imagine that we generate booties, but then take several simple random sub-samples from the booties, of size . Each of these is a valid bootstrap sample, but using a smaller number of booties. We can calculate a confidence interval for each one, so we end up with a total of confidence intervals. Let be an indicator for whether replication of the confidence interval with booties covers the true parameter, for and . We can extrapolate the coverage rate using a linear regression¹ of on : Then is the coverage rate when . The outcomes will be dependent because they’re based on the same set of booties. However, the replicates are independent, so we can just use cluster-robust variance estimators to get Monte Carlo standard errors for an estimate of . Essentially the same thing works for extrapolating the width of a confidence interval: just replace with , the width of the confidence interval based on the bootstrap subsample from replication .

In the context of hypothesis testing problems, Boos & Zhang (2000) actually proposed something a little bit more complicated than what I’ve described above. Instead of just taking a single sub-sample of each size , they note that that expected rejection rate for each size sub-sample can be computed using hypergeometric probabilities. The same thing works for computing confidence interval coverage, but I don’t see any way to make it work for confidence interval width. However, instead of computing just one sub-sample of size , we could compute several and then average across them to approximate the expected value of or given the full set of booties.

An example

Here’s an example of how this all works. For sake of simplicity (and compute time), I’m going to use an overly simple, not particularly well-motivated example of bootstrapping: using a percentile bootstrap for the population mean, where the estimator is a trimmed sample mean. As a data-generating process, I’ll use a shifted distribution with population mean and degrees of freedom : Or in R code:

Code

Correlations between z-transformed correlation coefficients

James E. Pustejovsky — Thu, 06 Jun 2024 00:00:00 GMT

For a little meta-analysis project that I’m working on with Jingru, we are dealing with a database of correlation coefficients, where some of the included studies report correlations for more than one instrument or sub-scale for one of the relevant variables. This leads to every meta-analytic methodologist’s favorite tongue-twister of distribution theory: inter-correlated correlation coefficients. Fortunately, Grandpa Ingram worked out the distribution theory for this stuff long ago (Olkin and Siotani, 1976; reviewed in Olkin and Finn, 1990).

Suppose that we have three variables, , , and , where and are different measures of the same construct. From a sample of size , we have correlation estimates and , both of which are relevant for understanding the underlying construct relation. These correlations are estimates of underlying population correlations and respectively, and the population correlation between and is . Typically, we would analyze the correlations after applying Fisher’s variance stabilizing and normalizing transformation. Denote the Fisher transformation as ), with derivative given by . Let , with the other transformed correlations defined similarly. The question is then: how strongly correlated are the sampling errors of the resulting effect size estimates—that is, what is ?

Olkin and Finn (1990) give the following expression for the covariance between two sample correlation coefficients that share a common variable: If correlations are converted to the Fisher-z scale, then we can use a delta method approximation to obtain an expression for the covariance between two sample z estimates that share a common variable. Using in place of , the covariance between the z-transformed correlations is approximately The corresponding correlation is thus

In the context of a meta-analysis, we might expect that the focal correlations will usually be very similar, if not exactly equal. For simplicity, let’s assume that they’re actually identical, . The sampling correlation then simplifies further to
To apply this formula, we need to specify values of and . The following table gives the resulting correlation between z-transformed sample correlations for a few values of and .

Code

Distribution of the number of significant effect sizes

James E. Pustejovsky — Thu, 28 Mar 2024 00:00:00 GMT

A while back, I posted the outline of a problem about the number of significant effect size estimates in a study that reports multiple outcomes. This problem interests me because it connects to the issue of selective reporting of study results, which creates problems for meta-analysis. Here, I’ll re-state the problem in slightly more general terms and then make some notes about what’s going on.

Consider a study that assesses some effect size across different outcomes. (We’ll be thinking about one study at a time here, so no need to index the study as we would in a meta-analysis problem.) Let denote the effect size estimate for outcome , let denote the sampling variance of the effect size estimate for outcome , and let denote the true effect size parameter for corresponding to outcome . Assume that the study outcomes follow a correlated-and-hierarchical effects model, in which where the study-level error , the effect-specific error , and the vector of sampling errors is multivariate normal with mean , known variances , and compound symmetric correlation structure .

Define as an indicator that is equal to one if is statistically significant at level based on a one-sided test, and otherwise equal to zero. (Equivalently, let be equal to one if the effect is statistically significant at level and in the theoretically expected direction.) Formally, where is the critical value from a standard normal distribution (e.g., , ). Let denote the total number of statistically significant effect sizes in the study. The question is: what is the distribution of .

Compound symmetry to the rescue

As I noted in the previous post, this set-up means that the effect size estimates have a compound symmetric distribution. We can make this a bit more explicit by writing the sampling errors in terms of the sum of a component that’s common acrosss outcomes and a component that’s specific to each outcome. Thus, let , where and . Let me also define as the conditional mean of the effects. It then follows that the effect size estimates are conditionally independent, given the common components: Furthermore, the conditional probability of a significant effect is and are mutually independent, conditional on . Therefore, the conditional distribution of is binomial, where What about the unconditional distribution?

To get rid of the , we need to integrate over its distribution, which leads to where is a binomial density with size and probability and is a normal density with mean and variance .

This distribution is what you might call a binomial-normal convolution or a random-intercept probit model (where the random intercept is ). As far as I know, the distribution cannot be evaluated analytically but instead must be calculated using some sort of numerical integration routine.

Just the moments, please

If all we care about is the expectation of , we don’t need to bother with all the conditioning business and can just look at the marginal distribution of the effect size estimates taken individually. Marginally, is normally distributed with mean and variance , so , where By the linearity of expectations,

We can also get an approximation for the variance of by working with its conditional distribution above. By the rule of variance decomposition, where is, as defined above, a function of and thus a random variable. Now, and we can get something close to using a first-order approximation: Thus, If the amount of common variation is small, so is near zero and is near zero, then the contribution of the second term will be small, and will act more or less like a binomial random variable with size and probability . On the other hand, if the amount of independent variation in the effect sizes is small, so is near zero and is near 1, then the term on the right will approach and will approach , or the variance of times a single Bernoulli variate. So you could say that has anywhere between and variate’s worth of information in it, depending on the degree of correlation between the effect size estimates.

Interactive distribution

Here is an interactive graph of the probability mass function of , with probability points calculated using Gaussian quadrature. Below the graph, I also report , the exact mean and variance of , and the first-order approximation to the variance (denoted $V_{approx}). When and , the approximate variance is not all that accurate because the first-order approximation to isn’t that good.

Approximating the distribution of cluster-robust Wald statistics

James E. Pustejovsky — Sun, 24 Mar 2024 00:00:00 GMT

In Tipton and Pustejovsky (2015), we examined several different small-sample approximations for cluster-robust Wald test statistics, which are like statistics but based on cluster-robust variance estimators. These statistics are, frankly, kind of weird and awkward to work with, and the approximations that we examined were far from perfect. In this post, I will look in detail at the robust Wald statistic for a simple but common scenario: a one-way ANOVA problem with clusters of dependent observations.

Meta-ANOVA

Consider a setup where clusters can be classified into one of categories, with each cluster of observations falling into a single category. Let denote the means of these categories. Suppose we have an estimator of those means and a corresponding cluster-robust variance estimator . Note that is diagonal because the estimators for each category are independent. Assume that the robust variance estimator is unbiased so for . Let .

Suppose that we want to test the null hypothesis that that means of the categories are all equal, . We can express this null using a contrast matrix , where . The null hypothesis is then . The corresponding cluster-robust Wald statistic is Under the null hypothesis, the distribution of will converge to a as the number of clusters in each category grows large. However, with a limited number of clusters in some of the categories, this approximate reference distribution is not very accurate and tests based on it can have wildly inflated type I error rates.

Small-sample approximation

In the paper, we considered several different ways of approximating the distribution of that work at smaller sample sizes. One class of approaches to approximating the sampling distribution of is to use a Hotelling’s distribution with degrees of freedom . Given the degrees of freedom, Hotelling’s is a multiple of an distribution: The question is then how to determine .

Several of the approaches that we considered are based on representing the statistic as where , , , and The various approaches we considered involve different ways of approximating the sampling distribution of .

Zhang’s approximation

One of the approximations involves finding degrees of freedom by following a strategy suggested by Zhang (2012, 2013). These degrees of freedom are given by where is the entry in row , column of . To find , we can compute the denominator using general formulas given in the paper. However, with a bit of analysis we can find a much simpler expression for the special case of one-way ANOVA.

A bit of math

Before going further, it’s useful to observe that is invariant to linear transformations of . In particular, an equivalent way to write the null hypothesis is as , where is the diagonal of the true sampling variances of categories 2 through , omitting the first category. Thus, let me redefine , and This transformation of the constraint matrix will make it possible to find a closed-form expression for .

Deriving Zhang’s degrees of freedom

Now, observe that where . From the Woodbury identity, where .

Fasi, Higham, and Liu (2023) provide formulas for roots of low-rank updates to scaled identity matrices. Their results provide a neat closed-form expression for . From their Equation (1.9), where . Further, we can write the matrix as with entries given by Because and is diagonal, we can write the entries of as And because the variance estimators for each category are independent, In prior work, we derived expressions for the Satterthwaite degrees of freedom for variances of average effect sizes, and the same formulas can be applied here with the category-specific . Let me write for the degrees of freedom corresponding to category . Then We can use this to obtain an expression for Zhang’s approximate degrees of freedom: Now, all we need to do is simplify… and, for , Thus, or, rearranging, It’s a surprisingly clean formula! Once these degrees of freedom are calculated, the degrees of freedom for the reference distribution would be and .

Other approximations

In the paper, we also considered two other degrees of freedom approximations, which involve not only the variances of but also the covariances between entries. In principle, one could follow similar algebra to get expressions for these other degrees of freedom as well. However, our simulations indicated that the other degrees of freedom approximations tend to be overly conservative and produce type-I error rates way below the nominal level (essentially, hardly ever rejecting the null) and less accurate than HTZ. So, there’s not much reason to work through them unless you find algebra enjoyable for its own sake.

What’s the right non-central distribution?

A further question about this cluster-robust Wald statistic is how to approximate its sampling distribution under specific alternative hypotheses. In other words, given a vector of means where the null does not hold, plus some information to determine and for , how could we approximate the distribution of ? We need something like a non-central Hotelling’s distribution…

Implementing Consul’s generalized Poisson distribution in Stan

James E. Pustejovsky — Wed, 06 Dec 2023 00:00:00 GMT

For a project I am working on, we are using Stan to fit generalized random effects location-scale models to a bunch of count data. In a previous post, I walked through our implementation of Efron’s (1986) double-Poisson distribution, which we are interested in using because it allows for both over- and under-dispersion relative to the Poisson distribution. Another distribution with these properties is the generalized Poisson distribution described by Consul and Jain (1973).

In this post, I’ll walk through my implementation of the GPO in Stan. The gamlss.dist package provides a full set of distributional functions for the generalized Poisson distribution, including a sampler, but the functions are configured to only allow for over-dispersion. Since I’m interested in allowing for under-dispersion as well, I’ll need to write my own functions. As in my previous post, I can validate my Stan functions against the functions from gamlss.dist (although only for over-dispersion scenarios).

Code

Implementing Efron’s double Poisson distribution in Stan

James E. Pustejovsky — Fri, 15 Sep 2023 00:00:00 GMT

For a project I am working on, we are using Stan to fit generalized random effects location-scale models to a bunch of count data. We’re interested in using the double-Poisson distribution, as described by Efron (1986). This is an interesting distribution because it admits for both over- and under-dispersion relative to the Poisson distribution, whereas most of the conventional alternatives such as the negative binomial distribution or Poisson-normal mixture distribution allow only for over-dispersion. The double-Poisson distribution is not implemented in Stan, so we’ve had to write our own distribution function. That’s fine and not particularly difficult. What’s a bit more of a challenge is writing Stan functions to generate random samples from the double-Poisson, so that we can generate posterior predictive checks.¹ In this post, I’ll walk through the implementation of the custom distribution functions needed to use the double-Poisson in Stan. The gamlss.dist package provides a full set of distributional functions for the double-Poisson distribution, including a sampler. Thus, I can validate my Stan functions against the functions from gamlss.dist.²

Code

Cluster-Bootstrapping a meta-analytic selection model

James E. Pustejovsky — Thu, 30 Mar 2023 00:00:00 GMT

Important

The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through grant R305D220026 to the American Institutes for Research. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.

Selective reporting

Selective reporting of study results is a big concern for meta-analysts. By selective reporting, we mean the phenomenon where affirmative findings—that is, statistically significant findings in the theoretically expected direction—are more likely to be reported and more likely to be available for a systematic review compared to non-affirmative findings. Selective reporting arises due to biases in the publication process, on the part of journals, editors, and reviewers, as well as strategic decisions on part of the authors (Rothstein et al., 2006; Sutton, 2009). Research synthesists worry about selective reporting because it can distort the evidence base available for meta-analysis, almost like a fun-house mirror distorts your appearance, leading to inflation of average effect size estimates and biased estimates of heterogeneity.

If you read the meta-analysis methods literature, you will find scores of tools available to investigate and adjust for the biases created by selective reporting. Well known and widely used methods include:

graphical representations like funnel plots and contour-enhanced funnel plots (Sterne et al., 2011; Sterne & Egger, 2001);
tests for selective reporting (or at least funnel plot asymmetry) like Egger’s regression (Egger et al., 1997) or Begg and Mazumdar’s rank correlation test (Begg & Mazumdar, 1994);
bias-adjustment methods like PET-PEESE (Stanley, 2008; Stanley & Doucouliagos, 2014), Trim-and-Fill (Duval & Tweedie, 2000), and selection models (Hedges & Vevea, 2005);
p-value diagnostics like p-curve (Simonsohn et al., 2014), p-uniform (van Aert et al., 2016; van Assen et al., 2015), and the test of excess significance (Ioannidis & Trikalinos, 2007); and
sensitivity analyses based on various forms of selection models (Copas & Shi, 2001; Mathur & VanderWeele, 2020; Vevea & Woods, 2005).

However, nearly all of the statistical methods here have the limitation that they are premised on observing independent effect sizes. That presents a problem for meta-analyses in education, psychology, and other social science fields, where it is very common to have meta-analyses involving dependent effect sizes.

Dependent effects occur in meta-analyses of group comparisons when primary studies report effects for multiple correlated measures of an outcome, at multiple points in time, or for multiple treatment groups compared to the same control group (Becker, 2000). Dependent effects are also common in meta-analyses of correlational effect sizes, where primary studies report more than one relevant correlation coefficient based on the same sample of participants. Methods such as multi-level meta-analysis (Van den Noortgate et al., 2013) and robust variance estimation (Hedges et al., 2010) are available to accommodate dependent effects when summarizing findings across studies or investigating moderators of effect size using meta-regression, but these techniques have yet to be extended to methods for testing or correcting bias due to selective reporting. Consequently, it’s pretty common to see research synthesis papers that use very sophisticated models for part of the analysis, but then use kludgey, awkward, or hacky approaches when it comes time to investigating selective reporting (Rodgers & Pustejovsky, 2020).

Along with a group of our colleagues from the American Institutes for Research, we are currently working on a project to develop better methods for investigating selective reporting issues in meta-analyses of dependent effect sizes. In this post, we will share an early peek under the hood at one little piece of what we’re studying, by sketching out what we think is a promising and pragmatic method for examining selective reporting while also accounting for effect size dependency. The method is to use a cluster-level bootstrap, which involves re-sampling clusters of observations (i.e., the set of multiple effect size estimates reported within a given primary study) to approximate the sampling distribution of an estimator (Boos, 2003; Cameron et al., 2008). To illustrate this technique, we will demonstrate how to bootstrap a Vevea-Hedges selection model.

Selection models comprise a large class of models that have two parts: a model describing the evidence-generation process and a model describing the process by which evidence is reported (Hedges & Vevea, 2005). Vevea-Hedges selection models (Hedges, 1992; Hedges & Vevea, 1996; Vevea & Hedges, 1995) involve a random effects meta-regression model for the evidence-generation process and a step function for the reporting process. With a step function, we assume that the probability that an effect size estimate is observed depends on the range in which its p-value falls. For instance, effects with might have some probability of being reported, effects with might have some other probability , and effects with might have some other probability .¹ Because the Vevea-Hedges model and other selection models separate the data-generation process into these two distinct stages, their parameters have clear interpretations and they can be used to generate bias-adjusted estimates of the distribution of effect sizes and to test for selective reporting issues. The only problem is that available implementations of selection models do not account for effect size dependency—but that’s where cluster bootstrapping could potentially help.

Disclaimer

To be clear, this post is based on work in progress. The cluster-bootstrap selection model that we’re going to demonstrate is an experimental and exploratory technique. We’re currently studying its properties and performance using Monte Carlo simulations, but we don’t have formal results to share yet. In the spirit of open and collaborative science, we wrote this post to demonstrate our approach to coding the method, in case others would like to experiment with the technique. Given that there are so few methods available for investigating selective reporting in meta-analyses with dependent effect sizes, we think this method is worth playing with and investigating further, and we would be happy to have others try it out as well. But, if you do so, please treat the results as tentative until we learn more about when the methods work well enough to trust the results.

An Example

For demonstrating this method, we will use data from a recent meta-analysis by Lehmann and colleagues (2018) that examined the effects of the color red on attractiveness judgments. The data is available via the metadat package (White et al., 2022). The dataset includes 81 effect sizes from 41 unique studies. You can browse the data for yourself here:

Code

Cohen’s $d_z$ makes me dizzy when considering measurement error

James E. Pustejovsky — Fri, 17 Feb 2023 00:00:00 GMT

Meta-analyses in education, psychology, and related fields rely heavily of Cohen’s , or the standardized mean difference effect size, for quantitatively describing the magnitude and direction of intervention effects. In these fields, Cohen’s is so pervasive that its use is nearly automatic, and analysts rarely question its utility or consider alternatives (response ratios, anyone? POMP?). Despite this state of affairs, working with Cohen’s is theoretically challenging because the standardized mean difference metric does not have a singular definition. Rather, its definition depends on the choice of the standardizing variance used in the denominator.¹

In this post, I’m going to mull over how measurement error and design decisions influence the metric definition of Cohen’s in basic within-group experimental designs. The distorting effects of measurement error has long been a source of concern within psychometric meta-analysis, a perspective associated with the work of Frank Schmidt and Jack Hunter, and measurement-error corrections are well developed and often applied in meta-analyses of correlations. Straight-forward measurement-error corrections have also been described for Cohen’s from between-group designs (see recent work by Brenton Wiernik and Jeff Dahlke). However, I have literally never seen a meta-analytic application that applied these corrections and I have thus far been unable to locate work on such corrections specifically for effect sizes in within-group designs.² So, time to muck about…

Effect size definitions in within-group designs

In basic between-group designs, the only variances in the model are the within-group variances, so the choice of standardizing variance is limited to a) the singular population variance, assuming it is homogeneous across groups, b) the variance of one group, or c) the average of the variances in each group. In most applications, homogeneity is assumed (often without much reflection), version (a) of Cohen’s is estimated, and the meta-analyst can go along their merry way. For sake of succinctness, I’ll call this effect size , where the indicates the usual version for basic between-group designs.

For within-group or repeated measures designs, the set of choices is more involved and includes a) standardizing by the across-participant variance in one condition (or both conditions, assuming homogeneity) or b) standardizing by the variance of the difference scores. The former approach is sometimes called ³, the latter is called ⁴. The metric uses the same standardizing variance as the from a basic between-group design, and so results from both types of designs are, in principle, on the same scale.

In the context of meta-analysis, the comparability of and is useful when working with a set of studies that include both types of designs. On the other hand, in meta-analyses that consist solely of within-group or repeated measures designs, comparability with may be less of a priority and one could consider using for synthesis. Purely on a pragmatic level, using might be attractive because the only pieces of information needed to calculate it are the total sample size and the statistic (or -value) from the comparison between conditions. In contrast, calculating also requires the between-participant standard deviations from one or both groups, which primary studies might not always report.

Going in to this exercise, I had the notion that measurement error would affect to a greater degree than because involves difference scores and difference scores get hit by measurement error twice. Does this intuition hold up? Let me try to formalize things a bit.

A within-group design with measurement error

Suppose we have a within-group design involving two conditions, where participants are assessed on trials under each condition. Let denote the outcome from trial for participant under condition , for , , and . A basic model for this set-up is where and are participant-specific errors in the true scores under each condition and the ’s are measurement errors. For simplicity, I will assume that:

the true-score variance is equal across conditions, with for ,
the true scores are correlated across conditions, , and
measurement errors are uncorrelated and have homogeneous variance across conditions, with .

Let denote the reliability (intra-class correlation) of a single observed score. Note that we can write in terms of the reliability and true-score variance as .

Under this model, there are several different standardized mean difference metrics that we could consider. Since measurement reliability might vary from study to study, it would make sense to define the metric in terms of true score variances alone, as or in terms of the variance of the difference in true scores, as However, we don’t directly observe the true scores, and we can’t estimate their variance unless we have information about score reliabilities. Thus, meta-analysts will usually need to calculate effect sizes in terms of observed scores that include measurement error.

Suppose that the analysis is conducted by taking the average of the trials for each participant under each condition, , and conducting the analysis using these mean scores. The variance of the mean scores is so we can define the observed-score standardized mean difference using raw score standardization as From this expression, we can see that measurement error attenuates the true-score effect size because the multiplier term is always going to be less than one.

Similarly, the variance of the observed difference scores is so we can define the observed-score standardized mean difference using change score standardization as Again, we can see that measurement error attenuates the true-score effect size because the multiplier term is always going to be less than one. However, unlike with , the attenuation factor here depends on in addition to and . This additional term in the correction factor is one indication that might be less desirable for meta-analysis. Correcting for the distortion from measurement error would require estimates of both the true-score correlation and the reliability of the scores, whereas correcting would require only the latter.

Meta-analysis

The relationships between the true-score effect sizes and the analogous observed score effect sizes starts to be a problem when we consider a meta-analysis of multiple primary studies. Primary studies will often use different instruments and procedures for measuring outcomes (necessitating the use of some standardized effect size), and those differences in instruments and procedures might come along with differences in score reliability as well as variation in the number of trials collected per condition (and plenty of other things, such as sample size, participant characteristics, etc.). Procedural heterogeneity like this creates two potential challenges for meta-analysis: bias in average effect sizes and extra heterogeneity in the distribution of effect sizes. Both could make findings from a meta-analysis more difficult to interpret, although I will argue that extra heterogeneity is more concerning than bias.

To illustrate, let’s now imagine that the parameters of the within-group study design, or , , , and are random variables, drawn from the distribution of parameters across a population of hypothetical studies.

A model for

Let’s first consider and assume that it follows a random effects model, with Let’s also assume that the remaining parameters , , and are independent of . These parameters determine the attenuation factor , which relates the observed-score effect size parameter to the true score effect size parameter.

The bias of is therefore Thus, under my very simplistic assumptions, a meta-analysis of observed score Cohen’s estimates will be biased (downward) for the overall average effect in the true-score distribution.

You might find that the downward bias in is undesirable. On the other hand, bias might not be as a big a problem as it first seems. If all of the observed-score effect sizes are biased to a degree that is unrelated to the true effects, then bias just stretches or compresses the scale of measurement, but doesn’t necessarily lead to interpretive problems. Imagine you have a ruler that is half an inch too short, and you’re trying to compare the heights of different objects. As long as you use the same ruler, then you will still be able to determine which objects are bigger and which are smaller, and by how much, even if the measurements are off in an absolute sense.

Apart from bias, however, variability in will also induce additional heterogeneity in the distribution of observed score effect sizes. This is a clear problem because it creates additional uncertainty, making it harder to draw inferences about the distribution of effects, predict new effect sizes, or identify substantively interesting moderators. To measure this additional heterogeneity and keep its consequences separate from the consequences for bias, I will look at the coefficient of variation in . Under the assumption that is independent of , Thus, the coefficient of variation for is amplified by a factor that depends on the squared coefficient of variation of . Under the same model, , where , and so and

To see what’s going on here, let’s consider some specific distributions for these measurement factors. First, let’s assume:

, so that and ;
, so that and ;
, so and ; and
, , and are mutually independent and independent of .

Below I simulate 50000 samples from these distributions and calculate and .

Code

Corrigendum to Pustejovsky and Tipton (2018), redux

James E. Pustejovsky — Mon, 07 Nov 2022 00:00:00 GMT

UPDATE, March 8, 2023

The correction to our paper has now been published at Journal of Business and Economic Statistics. It is available at https://doi.org/10.1080/07350015.2023.2174123.

In my 2018 paper with Beth Tipton, published in the Journal of Business and Economic Statistics, we considered how to do cluster-robust variance estimation in fixed effects models estimated by weighted (or unweighted) least squares. As explained in my previous post, we were recently alerted that Theorem 2 in the paper is incorrect as stated. It turns out, the conditions in the original version of the theorem are too general. A more limited version of the Theorem does actually hold, but only for models estimated using ordinary (unweighted) least squares, under a working model that assumes independent, homoskedastic errors. In this post, I’ll give the revised theorem, following the notation and setup of the previous post (so better read that first, or what follows won’t make much sense!).

Theorem 2, revised

Consider the model where is an vector of responses for cluster , is an matrix of focal predictors, is an matrix of additional covariates that vary across multiple clusters, and is an matrix encoding cluster-specific fixed effects, all for . Let be the set of predictors that vary across clusters and be the full set of predictors. Let be an absorbed version of the focal predictors and the covariates. The cluster-robust variance estimator for the coefficients of is where are the CR2 adjustment matrices.

If we assume a working model in which for and estimate the model by ordinary least squares, then the CR2 adjustment matrices have a fairly simple form: where is the symmetric square root of the Moore-Penrose inverse of . However, this form is computationally expensive because it involves the full set of predictors, , including the cluster-specific fixed effects . If the model is estimated after absorbing the cluster-specific fixed effects, then it would be convenient to use the adjustment matrices based on the absorbed predictors only, The original version of Theorem 2 asserted that , which is not actually the case. However, for ordinary least squares with the independent, homoskedastic working model, we can show that . Thus, it doesn’t matter whether we use or to calculate the cluster-robust variance estimator. We’ll get the same result either way, but is bit easier to compute.

Here’s a formal statement of Theorem 2:

Let and assume that have full rank . If and for , then , where and are as defined in Equation 3 and Equation 4, respectively.

Proof

We can prove this revised Theorem 2 by showing how can be constructed in terms of and . First, because for any , it follows that is idempotent, i.e.,

Next, denote the thin QR decomposition of as , where is semi-orthogonal and has the same rank as . Next, let and observe that this can be written as It can then be seen that It follows that because . Further, as well.

Now, let and observe that this can be written as because .

We then construct the full adjustment matrix as Showing that will suffice to verify that is the symmetric square root of the Moore-Penrose inverse of . Because is idempotent, , and , we have

From the representation of in Equation 5, it is clear that .

Corrigendum to Pustejovsky and Tipton (2018)

James E. Pustejovsky — Wed, 28 Sep 2022 00:00:00 GMT

In my 2018 paper with Beth Tipton, published in the Journal of Business and Economic Statistics, we considered how to do cluster-robust variance estimation in fixed effects models estimated by weighted (or unweighted) least squares. A careful reader, Dr. Michael Pfaffermayr, recently alerted us to a problem with Theorem 2 in the paper, which concerns a computational short cut for a certain cluster-robust variance estimator in models with cluster-specific fixed effects. The theorem is incorrect as stated, and we are currently working on issuing a correction for the published version of the paper. In the interim, this post details the problem with Theorem 2. I’ll first review the CR2 variance estimator, then describe the assertion of the theorem, and then provide a numerical counter-example demonstrating that the assertion is not correct as stated.

A fixed effects model

For data that can be grouped into clusters of observations, we considered the model where is an vector of responses for cluster , is an matrix of focal predictors, is an matrix of additional covariates that vary across multiple clusters, and is an matrix encoding cluster-specific fixed effects, all for . The cluster-specific fixed effects satisfy for . Interest centers on inference for the coefficients on the focal predictors .

We considered estimation of Model Equation 1 by weighted least squares (WLS), possibly under a working model for the distribution of . Let be a set of symmetric weight matrices used for WLS estimation. Sometimes, these weight matrices may be diagonal, consisting of sampling weights for each observation. Other times, the weight matrices may involve off-diagonal terms as well. Consider a working model where is a symmetric matrix that may be a function of a low-dimensional, estimable parameter. Based on this working model, the weight matrices might be taken as , where is an estimate of .

The CR2 variance estimator

In the paper, we provide a generalization of the bias-reduced linearization estimator introduced by McCaffrey et al. (2001) and Bell & McCaffrey (2002) that can be applied to Model Equation 1. The variance estimator is effectively a generalization of the HC2 correction for heteroskedasticity-robust standard errors, but that works for models with within-cluster dependence and cluster-specific fixed effects, and so we refer to it the “CR2” estimator.

In order to define the CR2 variance estimator and explain the issue with Theorem 2, I’ll need to lay down a bit more notation. Let be the total sample size. Let be the set of predictors that vary across clusters and be the full set of predictors. Let , , , , and denote the stacked versions of the cluster-specific matrices (i.e., , etc.). Let and . For a generic matrix , let and . Let be the matrix that selects the rows of cluster from the full set of observations, such that . These operators provide an easy way to define absorbed versions of the predictors. Specifically, let be the covariates after absorbing (i.e., partialling out) the cluster-specific effects, let be an absorbed version of the focal predictors and the covariates, and let be the focal predictors after absorbing the covariates and the cluster-specific fixed effects.

With this notation established, the CR2 variance estimator has the form where is the cluster-specific matrix of absorbed focal predictors, is the vector of weighted least squares residuals from cluster , and are a set of adjustment matrices that correct the bias of the residual cross-products. The adjustment matrices are calculated as follows. Let be the upper-right Cholesky factorization of and define the matrices for . The adjustment matrices are then calculated as where is the symmetric square root of the Moore-Penrose inverse of . Theorem 1 in the paper shows that, if the working model is correctly specified and some conditions on the rank of are satisfied, then the CR2 estimator is exactly unbiased for the sampling variance of the weighted least squares estimator of . Across multiple simulation studies, it’s been observed that the CR2 estimator also works well and outperforms alternative sandwich estimators even when the working model is not correctly specified.

Theorem 2

The adjustment matrices given in Equation 3 can be expensive to compute directly because the matrices involve computing a “residualized” version of the matrix involving the full set of predictors —including the cluster-specific fixed effects . Theorem 2 considered whether one can take a computational short cut by omitting the cluster-specific fixed effects from the calculation of the matrices. Specifically, define the modified matrices and Theorem 2 claims that if the weight matrices are inverse of the working model, such that for , then and hence . The implication is that the cluster-specific fixed effects can be ignored when calculating the adjustment matrices. However, the claimed equivalence does not actually hold.

Here is a simple numerical example that contradicts the assertion of Theorem 2. I first create a predictor matrix consisting of 4 clusters, a single focal predictor, and cluster-specific fixed effects.

Code

posts

p-uniform* is a three-parameter selection model

Notation

The three-parameter selection model

p-uniform*

Equal standard errors

Varying standard errors

References

Footnotes

Notes on regression adjustment for selectively reported results in meta-analysis

Regression adjustments for publication bias

Data-generating processes

Step function models

Copas selection models

Why care?

References

Footnotes

Variance stabilization of Cohen’s d

Interactive display of simulation results with Observable.js

An example

A bigger simulation of bootstrap confidence interval variations

Bootstrap confidence interval variations

Bootstrap confidence intervals for a mean

A quirk of nlme::varIdent

varIdent()

What doesn’t work

How to control the reference level

Setting a value

Colophon

Beta-density selection models for meta-analysis

The beta-density selection model

A truncated beta-density

Distribution of observed effect sizes

Copas selection models for meta-analysis

The Copas selection model

Distribution of observed effect sizes

Step-function selection models for meta-analysis

The step-function selection model

Distribution of observed effect sizes

Simulating bootstrap confidence intervals

Simulating a bootstrap

Extrapolating

An example

Correlations between z-transformed correlation coefficients

Distribution of the number of significant effect sizes

Compound symmetry to the rescue

Just the moments, please

Interactive distribution

Approximating the distribution of cluster-robust Wald statistics

Meta-ANOVA

Small-sample approximation

Zhang’s approximation

A bit of math

Deriving Zhang’s degrees of freedom

Other approximations

What’s the right non-central distribution?

Implementing Consul’s generalized Poisson distribution in Stan

Implementing Efron’s double Poisson distribution in Stan

Cluster-Bootstrapping a meta-analytic selection model

Selective reporting

Disclaimer

An Example

Cohen’s \(d_z\) makes me dizzy when considering measurement error

Effect size definitions in within-group designs

A within-group design with measurement error

Meta-analysis

A model for

Corrigendum to Pustejovsky and Tipton (2018), redux

Theorem 2, revised

Proof

Corrigendum to Pustejovsky and Tipton (2018)

A fixed effects model

The CR2 variance estimator

Theorem 2

Setting a `value`