Open Source vs Closed AI

What "Open" Actually Means in AI

Traditional open source software provides the complete source code under a license that permits use, modification, and redistribution. Anyone can inspect how the software works, fix bugs, add features, and share their modifications. Open source AI is more complicated because an AI model involves multiple components: the training data, the training code, the model architecture, the model weights (the learned parameters), and the inference code. Most AI models described as "open source" release only the model weights and inference code, keeping the training data and training code proprietary. Meta's Llama models, for example, release weights under a license that permits commercial use with restrictions, but the training data, data processing pipeline, and training infrastructure are not shared.

The Open Source Initiative (OSI) published a formal definition of "Open Source AI" in 2024, requiring that the release include sufficient information for a "skilled person to substantially recreate" the system, including training data descriptions, code for data processing and training, and model weights. By this standard, very few models qualify as truly open source. The distinction matters because many benefits attributed to open source AI, like the ability to audit training data for bias, reproduce results independently, or understand why the model behaves a certain way, require access to components that open-weight releases do not provide.

The spectrum of openness ranges from fully closed (proprietary model, API access only, no weights published) through open-weight (weights published, training data proprietary, restricted license) to fully open (all components published under a permissive license). Examples across this spectrum: GPT-4 is fully closed, accessible only through OpenAI's API. Llama 3 is open-weight with a commercial license but proprietary training data. OLMo, developed by AI2, publishes the weights, training data, training code, and evaluation code under permissive licenses, qualifying as genuinely open source. Each position on this spectrum involves different tradeoffs between the benefits of openness and the risks of unrestricted access.

The Case for Open AI

Transparency and scientific reproducibility are the strongest arguments for open AI. When model weights, training data, and training code are publicly available, independent researchers can verify claims about model performance, audit for bias, identify safety vulnerabilities, and build on the work rather than duplicating it. This is how science is supposed to work: findings are published, replicated, and built upon by the community. Closed AI development, where companies make claims about their models' capabilities and safety properties that cannot be independently verified, resembles proprietary pharmaceutical research before clinical trial transparency requirements. Independent verification of safety claims requires access to the system being evaluated.

Democratized access prevents the concentration of AI power in a small number of companies. If only a handful of corporations control the most capable AI systems, they become gatekeepers of a transformative technology. Open models allow startups, researchers, nonprofits, governments, and individuals in developing countries to build on state-of-the-art AI without depending on a few corporate providers. This matters for innovation (startups can build products without API costs and terms-of-service constraints), for sovereignty (countries can deploy AI systems they control rather than depending on foreign providers), and for equity (organizations that cannot afford enterprise API pricing can still access capable models).

Security research benefits from open access. When model weights are available, security researchers can probe for vulnerabilities, test defenses, and develop safety techniques. Bug bounty programs for software rely on open access to the code being tested. Similarly, AI safety research benefits from open access to the models being studied. Mechanistic interpretability research, which seeks to understand how models represent and process information internally, requires access to model weights. Red-teaming exercises are more thorough when researchers can examine the model's internals rather than only testing through an API.

Ecosystem development around open models creates public goods. Fine-tuning techniques, safety alignment methods, evaluation benchmarks, and deployment tools developed for open models benefit the entire AI community. The Hugging Face ecosystem, built primarily around open models, has produced thousands of fine-tuned models, datasets, and tools that advance the field in ways that would be impossible if all research depended on closed, proprietary systems. This collaborative development model has historically produced more robust, better-tested software than closed development, as demonstrated by the success of Linux, Apache, and other open source infrastructure.

The Case for Caution

Once model weights are released publicly, they cannot be recalled. If a model turns out to have dangerous capabilities that were not identified during safety evaluation, there is no mechanism to prevent its continued use. This is fundamentally different from closed models accessed through an API, where the provider can update safety filters, patch vulnerabilities, and monitor usage. The irreversibility of open release creates an asymmetry: the costs of under-caution (releasing something that turns out to be dangerous) are potentially severe and permanent, while the costs of over-caution (delaying release until safety is better understood) are temporary and primarily economic.

Open models can be fine-tuned to remove safety guardrails. The safety training (RLHF, Constitutional AI, etc.) that makes a model refuse harmful requests is a learned behavior that can be unlearned through fine-tuning on a small amount of data. Removing safety training from an open-weight model requires modest technical skill and commodity hardware. This means that safety-trained open models provide safety for casual users who use the model as released, but not for determined bad actors who are willing to spend a few hours fine-tuning. Several research groups have demonstrated that safety guardrails can be removed from open models with minimal effort and without degrading the model's general capabilities.

The marginal risk from open release depends on the model's capabilities. A model that can generate persuasive text provides modest incremental capability over what a motivated human could produce alone. A model that could provide detailed instructions for synthesizing dangerous biological agents, designing novel cyberweapons, or building weapons of mass destruction would present qualitatively different risks if released openly. The challenge is assessing these capabilities accurately before release and making release decisions calibrated to the actual risk level, which requires evaluation methods that are still being developed.

Concentration concerns run in both directions. Advocates for open AI warn about concentration of power in large companies. Advocates for careful release warn about concentration of powerful capabilities in the hands of anyone with a GPU. Neither concentration is desirable, and the optimal policy depends on which risk is greater at a given capability level. For current models, the concentration risk from closed development may outweigh the misuse risk from open release. For future, more capable models, the calculus may shift.

Regulatory Implications

The EU AI Act creates different obligations for open and closed models. General-purpose AI model providers must comply with transparency requirements, copyright obligations, and training data documentation. However, open-weight models released under permissive licenses are exempt from most provider obligations unless they present systemic risk (models trained with over 10^25 FLOPs). This exemption reflects the practical difficulty of imposing compliance obligations on decentralized open source development and the recognition that open release provides its own form of transparency.

U.S. policy has been more ambivalent. Executive Order 14110 required reporting of large training runs to the government regardless of whether the resulting model would be open or closed, but did not restrict open release. Some legislative proposals have sought to create liability for releasing models that are subsequently used to cause harm, which could effectively prohibit open release of capable models by making developers legally responsible for all downstream uses. The open source AI community has strongly opposed these proposals, arguing that they would concentrate AI development in large companies that can absorb legal risk while shutting out academic researchers, startups, and independent developers.

The emerging consensus in policy circles is that some form of capability-dependent governance is needed: models below a certain capability threshold should be freely releasable, while models above the threshold require additional safety evaluation and potentially restricted release. The challenge is defining the threshold in a way that is technically grounded, predictable for developers, and adaptable as capabilities evolve. Current proposals use compute thresholds (models trained above a certain FLOP count) as proxies, but compute is an imperfect predictor of capability, and efficiency improvements mean that the same capability can be achieved with less compute over time.