Animate and inanimate words chosen as stimulus materials did not differ in word frequency (p > 0.05).

Controls and aphasics did not differ in age (p > 0.05).

Control and ecological validity in conflict

• Experimental control not possible in the brain & behavioural sciences in the same way as in the physical sciences:
  • truly artificial stimuli problematic (novel as types & tokens, etc.)
  • natural stimuli strongly confounded, with unclear causality and primacy
• Still, we try to match our stimulus, participant, etc. groups

(Sassenhagen & Alday, under review at B&L)

What's the problem?

Animate and inanimate words chosen as stimulus materials did not differ in word frequency (p > 0.05).

Controls and aphasics did not differ in age (p > 0.05).

Where do I start?

Philosophy: Accepting the null.

Statistics: Getting useless information.

Random sampling

• You just aren't doing it by any stretch of the imagination.
• You are actively trying to distort measures of both central location and spread.

Populations vs. samples

• Inferential statistics, including statistical testing, draw conclusions from the data present about the data absent.
• The absent data are things we don't care about:
  • The set of all animate vs. all inanimate nouns
  • The set of all possible patients vs. all possible controls
• Alternatively, we have a completely sampled population and there are no absent data.
• So just use descriptive statistics and make sure they match!

Pragmatics: Testing what you care about.

• Even if we could
  • accept the null and
  • pretend that we're sampling randomly
  • from a population we care about

• we're still answering a boring question:

  do these two populations differ systematically in the given feature?

• when we actually care about:

  is the variance observed in my manipulation (better or at least partially) explained by the differences in the given feature?

What to do, what to do

• Stop inferential tests for confound control.
• Try to match groups as closely as possible using purely descriptive statistics (reduce confounds and collinearity).
• If you can (and this is a should could!), explicitly model these confounds as a covariate
  • Painful with ANOVA / ANCOVA / other 1970s statistics
  • Not a problem with modern (explicit) regression techniques like mixed-effects models
  • Which you really should be using anyway for many BBS designs (cf. Clark 1973; Judd, Westfall, and Kenny 2012; Westfall, Kenny, and Judd 2014)

Fresh off the presses

(DOI: 10.1371/journal.pone.0152719)

Arrows show causal true relationships

(All model diagrams from John Myles White)

Highlighting shows conditioning (in modelling)

But what if I didn't measure the actual causal variable?

Conditional probabilities and modelling (is so hard for frequentists)

• Conditioning effectively "blocks" a given path
• Non-blocked paths allow for "spurious" correlations and false positives

Simulated for typical data

(DOI: 10.1371/journal.pone.0152719.g002)

But all of this follows directly from the GLM

Standard GLM applications have a "vertical" error/variance term.

Y = β₀ + β₁X₁ + ε

ε ∼ N(0, σ)

But doesn't subjective temperature influence summertime fun?

Subjective handwaving is easy ...

... objectivity is hard

Modelling the full structure helps

So what do we do in practice?

• Mind your covariates and latent variables!

As always ...

You can find my stuff online: palday.bitbucket.org

If you have no questions about this stuff ....

We can discuss

• The ASA's statement on p-values
• Issues with optional stopping, both frequentist and Bayesian
• Brexit
• Catalonia's quest for independence
• The rise of the right in Germany and the rest of Europe

We will not discuss

• Donald Trump
• Ted Cruz

I really don't understand it either.