NIST updates its adversarial machine learning taxonomy

What the primary source recorded

NIST's March 24, 2025 release announced NIST AI 100-2e2025, a voluntary taxonomy and terminology update for adversarial machine-learning attacks and mitigations across predictive and generative systems. It covers evasion, poisoning, privacy, and misuse categories and was developed with input from U.S. and U.K. AI security institutes, industry, and academia.

The interface and data contract behind the headline

A threat model should connect taxonomy terms to concrete assets, trust boundaries, attacker capabilities, controls, telemetry, and response owners. For a tool-using assistant, that means separately modeling prompt injection in retrieved content, unauthorized tool invocation, sensitive output disclosure, model extraction, and training-data integrity. A category is useful only when it guides a testable system decision.

Tradeoffs and failure modes

Teams can mistake a taxonomy for a deployed defense. A prompt filter may reduce one injection pattern while leaving tool authorization unchanged. Privacy risk can arise from logs, embeddings, or error messages even when the generated answer looks harmless. Mitigations also create tradeoffs: aggressive blocking can hide useful warnings or push users toward unsanctioned paths.

A bounded hypothetical implementation

Hypothetical example: a knowledge assistant reads partner documents through an MCP server. A hostile document instructs the model to export customer records. The server exposes only a read query with tenant filtering, the policy layer denies export, and telemetry records the attempted instruction and decision. The incident response process can then update fixtures and controls.

Validation should test the system, not the slogan

Maintain attack fixtures by category, include benign near-misses to measure false positives, and test the whole path from untrusted input to external effect. Inspect logs for sensitive content, run access-control tests independently of model behavior, and verify that controls still hold after prompt or model upgrades. Record residual risk when a class is not covered.

SI direction requires evidence and authority boundaries

SI language does not remove conventional security engineering. Systems become more capable through tools and data paths, so their attack surface is primarily architectural. NIST's terminology helps teams discuss that surface precisely without claiming a taxonomy is certification, complete protection, or proof of superior intelligence.

Attack categories become useful only in architecture

An attack taxonomy becomes actionable when each category is mapped to a reachable surface in the actual architecture. Identify what an adversary can control: training examples, retrieved documents, request text, uploaded assets, tool output or access to a model endpoint. Then state the effect the attacker is trying to produce and the component that could enable it. Avoid assigning every category to every system by default. A source-aware assistant and a model-training pipeline have different exposed inputs. The threat model should make those differences inspectable so testing effort follows plausible paths rather than the length of a checklist.

Data poisoning concerns require lineage and change control around the affected dataset. Keep source identity, ingestion history and review state available to the people maintaining the pipeline. An unexpected distribution change can be a data-quality defect without being evidence of an attack, so investigation should distinguish intent from observed behavior. The response may involve quarantining a batch, restoring a reviewed source revision or repeating an evaluation. Record why that action was taken. A model score alone cannot identify which upstream records changed or whether the pipeline admitted them under the expected rules.

Evasion and extraction questions need different experiments. One examines how crafted inputs affect outputs; another concerns information or behavior obtainable through access to the model interface. Define the allowed test environment and request budget before running either. Use approved synthetic data where private content is unnecessary to establish the finding. Capture the input conditions and observed result without inflating a bounded successful test into a claim that every related attack works. Likewise, a failed attempt should remain a result under those conditions, not a general assertion that the system has no vulnerability in that class.

Mitigation should be evaluated at the boundary it is intended to protect. An input filter may block a known fixture but damage legitimate tasks or leave another ingestion route uncovered. Test both the repaired case and nearby normal behavior. For retrieval systems, inspect admission, source permissions, citation handling and tool boundaries separately. Hypler can turn taxonomy categories into reproducible tests and specific engineering work items with named owners. The resulting evidence supports a security decision about the defined architecture; it does not turn the taxonomy itself into a certification or eliminate the need for continued review as dependencies and exposed surfaces change.