araştırıyorum.learn
TREN
Learning paths / Critically assess an experiment
Learn step by step · Advanced

Critically assess an experiment

Assess claims by considering assignment, multiple tests and uncertainty together.

10 concepts29 minutes

01

Random assignment

2 min · Use chance to allocate experimental conditions.
02

A/B test

3 min · Compare two options fairly.
03

Effect size

3 min · Express how large a difference or relationship is.
04

Practical significance

3 min · Evaluate whether a finding matters for a real decision.
05

Statistical power

3 min · The probability of detecting a specified real effect.
06

Type II error

3 min · Fail to reject a null hypothesis that is false.
07

p-value

2 min · Assess incompatibility with a null hypothesis.
08

Multiple testing

3 min · Consider false alarms across a set of tests.
09

Bonferroni correction

3 min · A simple way to limit false positives across a test family.
10

Confidence interval

4 min · Show uncertainty, not just a single number.
Advanced

Think through a case

A team randomly assigns users to two interfaces. Without selecting a primary metric in advance, it tests 20 metrics and highlights one with p=0.04 as proof of success.

Answer three questions about the scenario below.

1 / 3

True or false?

Random assignment removes the false-alarm risk from running many tests.

Revisit a related concept ↗

2 / 3

True or false?

p=0.04 means the new interface has a 96% probability of being better.

Revisit a related concept ↗

3 / 3

True or false?

A decision should consider effect size, its confidence interval and implementation cost.

Revisit a related concept ↗
Practice what you learned

Put this learning path into practice.

Choose a guide connected to the concepts in this path. These optional activities do not count towards your reading progress.