richarddmorey

The significance test is perhaps the most used statistical procedure in the world, though has never been without its detractors. This is the second of two posts exploring Neyman's frequentist arguments against the significance test; if you have not read Part 1, you should do so before continuing (["The frequentist case against the significance test, part 1"](http://bayesfactor.blogspot.co.uk/2015/03/the-frequentist-argument-against.html)).

<!--more-->

Neyman had two major arguments against the significance test:

1. **The significance test fails as an epistemic procedure.** There is no relationship between the $p$ value and rational belief. More broadly, the goal of statistical inference is tests with good error properties, not beliefs.

2. **The significance test fails as a test.** The lack of an alternative means that a significance test can yield arbitrary results.

richarddmorey

M = 10000    # Number of sims  
power = .5   # power


## Binomial (fixed N)
N1 = rpois(M,2) + 1   ## random choice for N
X1 = rbinom(M,N1,power)  ## Perform studies
mean(X1 / N1)
sd(X1 / N1)

richarddmorey

> sessionInfo()
R version 3.1.3 (2015-03-09)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.10.2 (Yosemite)

locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

richarddmorey

#####################
# Code for computing probit model Bayes factors for
# 2x2 contingency table data
# Code by Richard D. Morey, July, 2015
# For:
# "What Are the Odds? Modern Relevance and Bayes 
# Factor Solutions for MacAlister's Problem from the 1881 Educational Times."
# Authors: Tahira Jamil, Maarten Marsman, Alexander Ly, 
# Richard D. Morey, Eric-Jan Wagenmakers
#####################

richarddmorey

#####################
# Code for computing probit model Bayes factors for
# 2x2 contingency table data
# Data and plots
# Code by Richard D. Morey and EJ Wagenmakers, July, 2015
# For:
# "What Are the Odds? Modern Relevance and Bayes 
# Factor Solutions for MacAlister's Problem from the 1881 Educational Times."
# Authors: Tahira Jamil, Maarten Marsman, Alexander Ly, 
# Richard D. Morey, Eric-Jan Wagenmakers

richarddmorey

N1 = 10 # Participants in "original"; inaccurate
N2 = 200 # Participants in "replication"; accurate

M = 100000 # number of simulations

# CI setup
alpha = .05
zstar = qnorm(1-alpha/2)

richarddmorey

A 11 64 68 -6 .66 .220 .305 -28 .15 .140 .206 -32 .12
B 20 75 75 -1 .93 .179 .257 -30 .051 .133 .192 -31 .070
C 33 83 80 4 .70 .167 .219 -24 .12 .139 .176 -21 .21
D 31 87 86 1 .96 .151 .208 -27 .050 .132 .179 -27 .072
E 31 85 77 -5 .49 .181 .189 -4 .77 .154 .169 -9 .56
F 48 91 98 -7 .079 .164 .195 -16 .25 .149 .191 -22 .12
Entire 174 95 99 -4 .24 .152 .184 -17 .20 .144 .182 -21 .13

richarddmorey

url <- 'https://gist.githubusercontent.com/richarddmorey/862ca2681afd3cd85b3b/raw/8de4af8d99cbff8745f7b52fd9009410ff450bd5/Neyman-etal-1969-whitetop.txt'
library(RCurl)
df <- getURL(url, ssl.verifypeer=FALSE)
dat.all <- read.table(textConnection(df))

## Break into three data frames
dat.freq.wet = dat.all[,c(1:2,3:6)]
dat.rainfall.wet = dat.all[,c(1:2,7:10)]
dat.rainfall.all = dat.all[,c(1:2,11:14)]

richarddmorey

set.seed(6)

pdf(file="CI_Bayes_poster.pdf",width=8,height=4,ver="1.4")
layout(matrix(c(1,1,2,3),2,2))
par(mar=c(4,4.5,0,1),cex.lab=2,mgp=c(3,1,0))

ci.cols = c(rgb(1,0,0,1),rgb(0,0,0,1))

M = 30

richarddmorey

t = 2
N = 25
# Create vector of r values from one-fifth the default to five times
# the default; a *huge* range
r = exp(seq(-log(5),log(5),len=100)+log(sqrt(2)/2))
# Compute BF
bf = sapply(r, function(r) BayesFactor::ttest.tstat(t,N,rscale = r,simple=TRUE))

# Plots
par(mfrow=c(1,2),las=1)