Bayesian Statistics Explained: Updating Beliefs with Evidence

The Bayesian Framework

The core of Bayesian inference is Bayes' theorem applied to parameters and data. In words: the posterior (what you believe about the parameter after seeing data) equals the likelihood (how probable the data would be for each possible parameter value) times the prior (what you believed before seeing data), divided by the marginal likelihood (the overall probability of the data under all possible parameter values). The marginal likelihood acts as a normalizing constant that ensures the posterior is a valid probability distribution.

The prior distribution encodes what you know or believe about the parameter before collecting data. A researcher with substantial previous evidence might use an informative prior that concentrates probability in a narrow range. A researcher with no prior knowledge might use a non-informative (flat or weakly informative) prior that treats all parameter values as roughly equally plausible. As more data accumulates, the prior becomes less influential and the posterior is increasingly determined by the data itself.

The posterior distribution is the complete answer to the inferential question. Unlike frequentist methods that produce a point estimate and confidence interval, the Bayesian posterior shows the full probability distribution over possible parameter values. You can read off the most probable value, compute credible intervals (ranges containing a specified probability of the parameter), and directly calculate the probability that the parameter exceeds or falls below any threshold of interest.

How Bayesian Differs from Frequentist

Frequentist statistics asks: "What is the probability of this data, assuming the null hypothesis is true?" Bayesian statistics asks: "What is the probability of the hypothesis, given the observed data?" These are fundamentally different questions that lead to different interpretations of identical data.

A frequentist confidence interval means: "If I repeated this procedure many times, 95% of intervals would contain the true value." A Bayesian credible interval means: "Given the data and my prior, there is a 95% probability that the true value falls in this range." The Bayesian interpretation is what most people intuitively think confidence intervals mean, even though frequentist confidence intervals do not technically support that interpretation.

Frequentist p-values cannot tell you the probability that a hypothesis is true. Bayesian analysis directly computes posterior probabilities for hypotheses, including the probability that an effect exists, the probability that an effect exceeds a practically meaningful threshold, or the relative probabilities of two competing models (Bayes factors).

Priors: The Controversial Element

The use of priors is both the greatest strength and the most criticized aspect of Bayesian statistics. Critics argue that priors inject subjectivity into what should be an objective analysis. Proponents counter that all statistical analyses involve subjective choices (which test to use, how to code variables, what significance level to set) and that making prior assumptions explicit and formal is more honest than hiding them in design choices.

Non-informative priors (also called objective or reference priors) attempt to minimize the influence of the prior, letting the data speak for itself. Common choices include uniform distributions (all values equally likely) or Jeffreys priors (derived from the Fisher information). With non-informative priors and large samples, Bayesian and frequentist results typically agree closely.

Informative priors incorporate genuine prior knowledge, such as the results of previous studies, physical constraints (parameters that cannot be negative), or expert opinion. When prior information is accurate, informative priors improve precision. When multiple studies exist on a topic, Bayesian meta-analysis naturally synthesizes evidence by treating each study's results as data that updates the posterior from the previous study's posterior.

Sensitivity analysis runs the same analysis with different priors to check whether conclusions are robust. If the posterior changes dramatically with different reasonable priors, results are prior-dependent and should be interpreted cautiously. If conclusions remain stable across a range of priors, they are robust and primarily data-driven.

Bayes Factors

A Bayes factor quantifies the evidence provided by the data in favor of one hypothesis over another. A Bayes factor of 10 means the data are 10 times more likely under the alternative hypothesis than under the null, providing strong evidence for the alternative. A Bayes factor of 0.1 means the data are 10 times more likely under the null, providing strong evidence against the alternative. A Bayes factor near 1 means the data do not discriminate between hypotheses.

Conventional interpretation guidelines suggest that Bayes factors between 1 and 3 represent "anecdotal" evidence, 3 to 10 are "moderate," 10 to 30 are "strong," 30 to 100 are "very strong," and above 100 are "extreme" evidence. Unlike p-values, Bayes factors can provide evidence in favor of the null hypothesis, not just against it. This is a significant advantage in fields where establishing the absence of an effect is scientifically important, such as demonstrating that a drug has no harmful side effects or that two treatments produce equivalent outcomes.

A Concrete Example

Suppose a medical test for a rare disease has a 99% sensitivity (true positive rate) and a 95% specificity (true negative rate). The disease affects 1 in 1000 people. If you test positive, what is the probability you actually have the disease? Intuitively, people assume the answer is close to 99%, but Bayesian reasoning reveals otherwise.

The prior probability of having the disease is 0.001. The likelihood of testing positive given disease is 0.99. The likelihood of testing positive given no disease (false positive rate) is 0.05. Applying Bayes' theorem: the posterior probability of having the disease given a positive test is approximately 0.019, or about 2%. Despite the test seeming highly accurate, the overwhelming majority of positive results are false positives because the disease is so rare. This example illustrates why base rates (priors) matter enormously, and why ignoring them leads to dramatically wrong conclusions.

Practical Applications

Bayesian methods excel in several practical situations. Sequential analysis allows you to update beliefs continuously as data arrives, stopping when sufficient certainty is reached rather than committing to a fixed sample size in advance. This is especially valuable in clinical trials where ethical considerations demand stopping early if a treatment is clearly effective or harmful.

Complex hierarchical models with many parameters are more naturally expressed in the Bayesian framework, where priors on parameters at different levels of the hierarchy provide regularization (shrinkage toward the overall mean) that prevents overfitting. Meta-analysis, multilevel modeling, and spatial statistics all benefit from Bayesian approaches that share information across groups while allowing each group to have its own parameter estimates.

Small samples benefit from informative priors that bring additional information beyond what the limited data can provide. Where frequentist methods may produce very wide confidence intervals or unstable estimates, Bayesian methods with reasonable priors produce more stable estimates by combining the small dataset with prior knowledge.

Modern Bayesian computation relies on Markov Chain Monte Carlo (MCMC) algorithms, which simulate draws from the posterior distribution when analytical solutions are unavailable. Software like Stan, PyMC, JAGS, and the R package brms make Bayesian modeling accessible without requiring manual derivation of posterior distributions. These tools handle complex models, including hierarchical, nonlinear, and mixture models, that would be difficult or impossible to analyze with closed-form frequentist methods. Convergence diagnostics (R-hat statistics, trace plots, effective sample sizes) verify that the MCMC algorithm has adequately explored the posterior, providing confidence that the computational results are reliable.