How to Pre-Register Experiments: Locking In Your Analysis Plan

The case for pre-registration rests on a well-documented problem: when researchers have the flexibility to choose their analysis approach after seeing the data, they tend to find significant results more often than the true base rate of effects would predict. This is not fraud, it is the natural consequence of human cognitive biases combined with strong incentives to publish significant findings. Pre-registration constrains these researcher degrees of freedom by committing to a specific analytical approach in advance.

Deviations from the Pre-Registration

Pre-registration does not require following the plan rigidly regardless of circumstances. If data quality issues, unexpected equipment failures, or other unforeseen problems require changing the analysis approach, those changes should be transparently reported and justified. The key distinction is between pre-specified analyses (confirmatory) and post-hoc analyses (exploratory). Both can appear in the same paper, but they must be labeled appropriately. Pre-specified analyses carry the full weight of hypothesis testing, while exploratory analyses are presented as preliminary findings that require confirmation in future studies.

Registered Reports

Registered Reports take pre-registration one step further by embedding it in the publication process. In the Registered Reports format, researchers submit their introduction, hypotheses, methods, and analysis plan to a journal for peer review before collecting any data. Reviewers evaluate the importance of the research question and the rigor of the methodology without being influenced by the results. If the Stage 1 submission passes review, the journal issues an in-principle acceptance, guaranteeing publication regardless of whether the results are statistically significant.

After receiving in-principle acceptance, the researchers collect data and carry out the pre-specified analyses. The Stage 2 submission includes the results and discussion, and reviewers verify that the protocol was followed and that the conclusions are supported by the data. Deviations from the protocol are flagged and discussed. Over 300 journals across disciplines now accept Registered Reports, including journals in psychology, neuroscience, ecology, political science, and medicine.

Registered Reports address a fundamental problem with traditional publishing: the tendency to publish only significant results. By committing to publish before the results are known, journals eliminate the file-drawer problem (where null results are shelved) and reduce incentives for questionable research practices. Empirical comparisons show that Registered Reports produce a much higher proportion of null results than traditional publications, suggesting that the traditional literature overrepresents significant findings due to publication bias.

Finally, some researchers question whether pre-registration is worth the effort for purely exploratory studies with no confirmatory hypotheses. In truly exploratory contexts, a pre-registration can still document the research question, the data source, the variables of interest, and the general analytical strategy. This level of documentation, sometimes called a pre-analysis plan, provides transparency about the starting point of the exploration even when the specific analytical path cannot be predetermined. The key principle is that more transparency is always better than less, and any documentation of prior intentions helps readers evaluate the final conclusions.

Common Concerns About Pre-Registration

Some researchers worry that pre-registration stifles exploratory research. This concern rests on a misunderstanding: pre-registration does not prohibit exploration, it simply requires labeling. Researchers can conduct any exploratory analyses they wish, as long as those analyses are transparently reported as exploratory rather than confirmatory. The distinction matters because confirmatory analyses (testing pre-specified predictions) and exploratory analyses (discovering unexpected patterns) have different error rates and evidentiary standards.

Others argue that pre-registration is impractical for complex studies where the analysis depends on the structure of the data. Qualitative research, complex computational modeling, and machine learning pipelines may not fit neatly into a traditional pre-registration template. For these cases, flexible pre-registration formats allow researchers to specify their general analytical approach, the decisions that will be data-dependent, and the criteria they will use to evaluate their results. Even a partial pre-registration is more transparent than none at all.

A third concern is that pre-registration creates unnecessary bureaucratic overhead. In practice, writing a pre-registration forces researchers to think carefully about their design and analysis before investing time and resources in data collection. Many researchers report that the pre-registration process itself improves their studies by revealing ambiguities in the protocol, identifying unstated assumptions, and clarifying the criteria for success. The time spent on pre-registration is typically recovered through smoother data collection and more straightforward analysis.