How AI risks can outpace safeguards

GS III – Science & Technology (Emerging Technologies) | Artificial Intelligence (AI)

• Anthropic Report: Anthropic blocked users from using its Claude AI model for harmful purposes within cybersecurity, surveillance and biotechnology from December 2025 to August 2026.
• Self-Regulation: Most AI companies, such as Google, Anthropic and OpenAI, are self-policing.
• Concerns: Self-policing raises concerns that AI companies may not be giving sufficient attention to safety or may develop appropriate safeguards only after deploying new capabilities.

Sensing Danger

• Previous Models: Anthropic stated that less stringent safety requirements applied to biological use for its previous Opus 4 and Sonnet 4.5 models, as they performed poorly against internal benchmarks.
• Controlling Access to Newer Models: Anthropic decided to control such access in newer models, such as Fable 5.
• Nonfunctional Filters: Anthropic also said in an August post that filters preventing discussions of biological weapons had been nonfunctional for nearly a year on more than 133 million exchanges.
• Case Studies: Users had already been using earlier models for biology research for weeks.
• Potential Misuse: Examples included a researcher seeking assistance in preparing a grant application for gain-of-function work with chikungunya virus, and developers involving Claude in a guided rocket programme, including a field test that used code written by Claude.
• Account Ban: Anthropic said it started disabling accounts as soon as the possibility of abuse became clear.

Limits of AI Safeguards

• Input & Output Classifiers: AI models have specific software components called input and output classifiers to verify user inputs and model outputs.
• Dual-Use Problem: The same knowledge can be used to both enable benefits and cause harm.
• Data Exfiltration: A user could leverage Claude to create harmful content, such as missile guidance code, and save the output offline before Anthropic realises the misuse and terminates the account.
• Safeguard Requirement: A safeguard is not just a checkpoint but also requires the ability to detect whether the information presented is harmful.
• Pattern Recognition: A single message may not reveal bad intention. Like identifying whether pieces are being joined to build a rifle or a wheelchair, the larger pattern may become clear only over time.
• Dual-Use Code: Code for a control loop can be used both in guided missiles and air-conditioners, refrigerators or toilet tanks.
• Accumulated Pattern: Input and output classifiers work better after a sufficient pattern of activity has accumulated.
• Safeguard Limitation: In certain cases, harm may remain possible even with safeguards in place because data can be exfiltrated before termination.

Export Controls

• Older Systems: Similar issues have existed for decades in other technical systems.
• Two Approaches: Older systems broadly achieved consensus on:
  • Limiting who could use particular technologies.
  • Monitoring how something is used over time.
• Trusted Access: This is similar to the trusted access programmes mentioned in the Anthropic report.
• Export Controls: Export-control regimes are an example of limiting who can access particular technologies rather than analysing every individual use.
• International Mechanisms: The Coordinating Committee for Multilateral Export Controls, Wassenaar Arrangement and Nuclear Suppliers Group employ mechanisms such as export licences, end-user requirements and restrictions on barred organisations and countries.
• Other Examples: Similar methods are employed in anti-money-laundering, regulation of ammonium nitrate purchases and controls around pseudoephedrine.
• Classifier Limitations: Simply giving more power to classifiers is not enough.

Conclusion

• The key issue is whether it is sufficient to block all access to dangerous or potentially dangerous AI models when there remains the possibility of “data exfiltration before termination.”
• Consequently, AI safety becomes a matter of understanding where self-policing, safeguards and access controls end.

UPSC Mains Practice Question

Q. “AI safety does not imply that no harm has been done.” Evaluate the types of self-policing techniques and AI classifiers that can ensure safety in AI systems.