A model-based agent, given the authority to execute tools or interact with APIs, may perform an action that is harmful, incorrect, or not aligned with the user's true intent in response to a prompt. This can be caused by the model misinterpreting an ambiguous prompt or being manipulated by an adversary into misusing its delegated authority.
AI/ML / Gen AI / Threats / DEV
Unintended Action by a Model-Based Agent
CCC.GenAI.TH06
Related Capabilities
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.CP21 | Generate Content | Ability to generate a response given a foundation model, parameter values, and a prompt. |
Related Controls
| ID | Title | Description |
|---|---|---|
| CCC.GenAI.CN02 | Model Output Filtering and Sanitisation | Inspect and validate GenAI model output before passing it to users, applications or plugins in order to filter or sanitise insecure or unreliable output and prevent sensitive data leakage. |
| CCC.GenAI.CN06 | Least Privilege for Plugins | Restricts the permissions of any external tools the GenAI system can call to limit the potential damage if an agent is coerced to perform unintended actions or vulnerabilities in the tools are exploited. |
External Mappings
| Framework | ID | Remarks |
|---|---|---|
| FINOS-AIGF | AIR-OP-018 | Model Overreach / Expanded Use |
| SAIF | RA | Rogue Actions |
| OWASP-LLM-TOP10 | LLM06:2025 | Excessive Agency |
| MITRE-ATLAS | AML.T0065 | LLM Prompt Crafting |