Three Risk Categories: Abuse, Misbehavior, External Attack
Anthropic's safety taxonomy: user abuse / model misbehavior / Prompt Injection
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Three Risk Categories: Abuse, Misbehavior, External Attack”?
Anthropic's safety taxonomy: user abuse / model misbehavior / Prompt Injection
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Every AI product faces a different threat model, but all risks fall into three categories. Understanding these three risk types is the first step to designing a secure architecture.
- Using the Agent to generate phishing emails
- Inducing the Agent to execute malicious code
- Using the Agent to access unauthorized data
- Bypassing safety limits through jailbreak attacks
- Overreach: user only asked to view a file, but the model modified it on its own
- Hallucination-driven action: executed a real operation based on fabricated information
- Permission overstep: model tries to access resources outside the current task
- Can't stop: Agent enters an infinite loop
- Prompt Injection: attack instructions embedded in web pages/documents
- Supply chain attack: malicious MCP server returns tampered data
- Data poisoning: backdoors planted in training data
- Indirect injection: instructions injected through emails/files that the Agent reads
MODEL LAYER Model-Layer Defense
- RLHF/Constitutional AI trains safe preferences
- Model learns to refuse dangerous requests
- Model proactively asks the user when uncertain
- Follows the principle of least privilege
ENVIRONMENT LAYER Environment-Layer Defense
- Sandbox isolation: code executes in a restricted environment
- Permission controls: authorization scoped to task granularity
- Approval mechanism: high-risk operations require human confirmation
- Network isolation: restricts the Agent's network access scope
New Attack Surface Introduced by Model Context Protocol
Classifier + Sandbox: High Autonomy, Low Risk
Judges whether each operation is safe
No damage even if the classifier misjudges
Where the risk boundary sits in “Security Classification Framework”
“Every AI product faces a different threat model, but all risks fall into three categories.” moves security beyond telling a model not to make mistakes. The real protection is ensuring that a mistaken judgment cannot become an irreversible result through permissions, data, or the environment.
Separate model suggestions from real authority
In the flow described by “Every AI product faces a different threat model, but all risks fall into three categories.”, check what the user may request, what the model may suggest, what the tool actually permits, and who can approve a write or send action. Web pages, documents, and tool results can carry untrusted instructions; looking like documentation does not grant them authority.
- Using the Agent to generate phishing emails
- Inducing the Agent to execute malicious code
- Using the Agent to access unauthorized data
Security includes failure and recovery
Use “Every AI product faces a different threat model, but all risks fall into three categories.” for a reverse exercise: add bad input, a missing credential, or an approval that never arrives. Confirm that the system refuses, pauses, and leaves a trace instead of executing to completion.
From “Security Classification Framework” to “MODEL LAYER Model-Layer Defense”
“Security Classification Framework” grounds the problem in “Every AI product faces a different threat model, but all risks fall into three categories. Understanding these three risk types is the first step to designing a secure architecture”. “MODEL LAYER Model-Layer Defense” then moves it toward “Training the model to naturally prefer safe behaviors—like cultivating an employee with good values. RLHF/Constitutional AI trains safe preferences Model learns to refuse dangerous requests Model proactively as…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
For security, separate what the model wants to do from what the system permits. Check data boundaries, tool permissions, human confirmation, and recovery after failure.
- “Security Classification Framework”: Every AI product faces a different threat model, but all risks fall into three categories. Understanding these three risk types is the first step to designing a secure architecture
- “MODEL LAYER Model-Layer Defense”: Training the model to naturally prefer safe behaviors—like cultivating an employee with good values. RLHF/Constitutional AI trains safe preferences Model learns to refuse dangerous requests Model proactively as…
- “The closing point”: Overreach: user only asked to view a file, but the model modified it on its own
The final “The closing point” brings the discussion to “Overreach: user only asked to view a file, but the model modified it on its own”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.