Posts

Interactive display of simulation results with Observable.js

simulation
programming
Monte Carlo simulations are an essential and ubiquitous feature of my work as a statistical methodologist. With my students and collaborators, I have conducted simulations…
Jan 19, 2025
James E. Pustejovsky

A bigger simulation of bootstrap confidence interval variations

programming
Rstats
bootstrap
In my previous post, I demonstrated some new brand-new additions to the simhelpers package: utilities for calculating bootstrap confidence intervals. That post walked…
Jan 18, 2025
James E. Pustejovsky

Bootstrap confidence interval variations

programming
Rstats
bootstrap
I recently added some new utilities for calculating bootstrap confidence intervals to the simhelpers package. The functions are designed to make it a bit more convenient to…
Jan 15, 2025
James E. Pustejovsky

A quirk of nlme::varIdent

Rstats
programming
hierarchical models
nlme
Wherein I document how to control the reference level in models that use nlme::varIdent() for specifying a variance structure.
Dec 28, 2024
James E. Pustejovsky

Beta-density selection models for meta-analysis

effect size
distribution theory
selective reporting
I still have meta-analytic selection models on the brain. As part of an IES-funded project with colleagues from the American Institutes for Research, I’ve been working on…
Nov 8, 2024
James E. Pustejovsky

Copas selection models for meta-analysis

effect size
distribution theory
selective reporting
Recently, I’ve been studying and tinkering with meta-analytic models for selective reporting of findings from primary studies. I’ve mostly focused on step-function selection…
Jul 27, 2024
James E. Pustejovsky

Step-function selection models for meta-analysis

effect size
distribution theory
selective reporting
In a recent post I looked at the distribution of statistically significant effect sizes in studies that report multiple outcomes, when those studies are subject to a certain…
Jul 9, 2024
James E. Pustejovsky

Simulating bootstrap confidence intervals

Save your booties by extrapolating!
programming
Rstats
bootstrap
Bootstrapping is a brilliant statistical idea. I realize that it’s become pretty routine, these days, and maybe has lost some of the magical veneer it once had, but I will…
Jul 1, 2024
James E. Pustejovsky

Correlations between z-transformed correlation coefficients

effect size
correlation
distribution theory
meta-analysis
For a little meta-analysis project that I’m working on with Jingru, we are dealing with a database of correlation coefficients, where some of the included studies report…
Jun 6, 2024
James E. Pustejovsky

Distribution of the number of significant effect sizes

In a study reporting multiple outcomes
effect size
distribution theory
A while back, I posted the outline of a problem about the number of significant effect size estimates in a study that reports multiple outcomes. This problem interests me…
Mar 28, 2024
James E. Pustejovsky

Approximating the distribution of cluster-robust Wald statistics

robust variance estimation
distribution theory
In Tipton and Pustejovsky (2015), we examined several different small-sample approximations for cluster-robust Wald test statistics, which are like F statistics but…
Mar 24, 2024
James E. Pustejovsky

Implementing Consul’s generalized Poisson distribution in Stan

Bayes
simulation
distribution-theory
generalized linear model
programming
Rstats
For a project I am working on, we are using Stan to fit generalized random effects location-scale models to a bunch of count data. In a previous post, I walked through our…
Dec 6, 2023
James E. Pustejovsky

Implementing Efron’s double Poisson distribution in Stan

Bayes
simulation
distribution-theory
generalized linear model
programming
Rstats
For a project I am working on, we are using Stan to fit generalized random effects location-scale models to a bunch of count data. We’re interested in using the…
Sep 15, 2023
James E. Pustejovsky

Cluster-Bootstrapping a meta-analytic selection model

bootstrap
dependent effect sizes
meta-analysis
publication bias
programming
Rstats
In this post, we will sketch out what we think is a promising and pragmatic method for examining selective reporting while also accounting for effect size dependency. The method is to use a cluster-level bootstrap, which involves re-sampling clusters of observations to approximate the sampling distribution of an estimator. To illustrate this technique, we will demonstrate how to bootstrap a Vevea-Hedges selection model.
Mar 30, 2023
James E. Pustejovsky, Megha Joshi

Cohen’s dz makes me dizzy when considering measurement error

effect size
standardized mean difference
design-comparable SMD
measurement-error
Meta-analyses in education, psychology, and related fields rely heavily of Cohen’s d, or the standardized mean difference effect size, for quantitatively describing the magnitude and direction of intervention effects. In these fields, Cohen’s d is so pervasive that its use is nearly automatic, and analysts rarely question its utility or consider alternatives (response ratios, anyone? POMP?). Despite this state of affairs, working with Cohen’s d is theoretically challenging because the standardized mean difference metric does not have a singular definition. Rather, its definition depends on the choice of the standardizing variance used in the denominator.
Feb 17, 2023
James E. Pustejovsky

Corrigendum to Pustejovsky and Tipton (2018), redux

A revised version of Theorem 2
robust variance estimation
econometrics
matrix algebra
In my 2018 paper with Beth Tipton, published in the Journal of Business and Economic Statistics, we considered how to do cluster-robust variance estimation in fixed effects models estimated by weighted (or unweighted) least squares. We were recently alerted that Theorem 2 in the paper is incorrect as stated. It turns out, the conditions in the original version of the theorem are too general. A more limited version of the Theorem does actually hold, but only for models estimated using ordinary (unweighted) least squares, under a working model that assumes independent, homoskedastic errors. In this post, I’ll give the revised theorem, following the notation and setup of the previous post.
Nov 7, 2022
James E. Pustejovsky

Corrigendum to Pustejovsky and Tipton (2018)

Theorem 2 is incorrect as stated
robust variance estimation
econometrics
matrix algebra
In my 2018 paper with Beth Tipton, published in the Journal of Business and Economic Statistics, we considered how to do cluster-robust variance estimation in fixed effects models estimated by weighted (or unweighted) least squares. A careful reader recently alerted us to a problem with Theorem 2 in the paper, which concerns a computational short cut for a certain cluster-robust variance estimator in models with cluster-specific fixed effects. The theorem is incorrect as stated, and we are currently working on issuing a correction for the published version of the paper. In the interim, this post details the problem with Theorem 2. I’ll first review the CR2 variance estimator, then describe the assertion of the theorem, and then provide a numerical counter-example demonstrating that the assertion is not correct as stated.
Sep 28, 2022
James E. Pustejovsky

Variance component estimates in meta-analysis with mis-specified sampling correlation

meta-analysis
dependent effect sizes
distribution theory
hierarchical models
In a recent paper with Beth Tipton, we proposed new working models for meta-analyses involving dependent effect sizes. The central idea of our approach is to use a working…
Nov 28, 2021
James E. Pustejovsky

Implications of mean-variance relationships for standardized mean differences

standardized mean difference
response ratio
distribution theory
meta-analysis
A question came up on the R-SIG-meta-analysis listserv about whether it was reasonable to use the standardized mean difference metric for synthesizing studies where the outcomes are measured as proportions. I think this is an interesting question because, while the SMD could work perfectly fine as an effect size metric for proportions, there are also other alternatives that could be considered, such as odds ratios or response ratios or raw differences in proportions. Further, there are some situations where the SMD has disadvantages for synthesizing contrasts between proportions. Thus, it’s a situation where one has to make a choice about the effect size metric, and where the most common metric (the SMD) might not be the right answer. In this post, I want to provide a bit more detail regarding why I think mean-variance relationships in raw data can signal that the standardized mean differences might be less useful as an effect size metric compared to alternatives.
Nov 2, 2021
James E. Pustejovsky

Inverting partitioned matrices

matrix algebra
There’s lots of linear algebra out there that’s quite useful for statistics, but that I never learned in school or never had cause to study in depth. In the same spirit as my previous post on the Woodbury identity, I thought I would share my notes on another helpful bit of math about matrices. At some point in high school or college, you might have learned how to invert a small matrix by hand. It turns out that there’s a straight-forward generalization of this formula to matrices of arbitrary size, but that are partitioned into four pieces.
Oct 20, 2021
James E. Pustejovsky

Standardized mean differences in single-group, repeated measures designs

effect size
standardized mean difference
distribution theory
I received a question from a colleague about computing variances and covariances for standardized mean difference effect sizes from a design involving a single group…
Oct 6, 2021
James E. Pustejovsky

Finding the distribution of significant effect sizes

In a study reporting multiple outcomes
effect size
distribution theory
In basic meta-analysis, where each study contributes just a single effect size estimate, there has been a lot of work devoted to developing models for selective reporting.…
Apr 27, 2021
James E. Pustejovsky

The Woodbury identity

A life-hack for analyzing hierarchical models
hierarchical models
matrix algebra
As in many parts of life, statistics is full of little bits of knowledge that are useful if you happen to know them, but which hardly anybody ever bothers to mention. You…
Dec 4, 2020
James E. Pustejovsky

An ANCOVA puzzler

meta-analysis
effect size
standardized mean difference
Doing effect size calculations for meta-analysis is a good way to lose your faith in humanity—or at least your faith in researchers’ abilities to do anything like sensible…
Nov 24, 2020
James E. Pustejovsky

From Longhorn to Badger

It’s taken me a while to finally get around to updating my website with some personal news. I’ve moved from UT Austin to the UW Madison School of Education, where I am now…
Aug 28, 2020
James E. Pustejovsky
No matching items
Back to top