Large Language Models (LLM9)

This card describes how an attacker can use indirect prompt injection to change AI model behavior or steal data.

Scenario: Deckard's juice theft scenario

Example

Deckard loves juice, but he doesn't like to pay for it. His favorite juice shop has recently implemented an AI Chatbot and Deckard has heard that these chatbots' behavior can be changed. The chatbot allows users to upload their shopping lists as long as they are authenticated. Not only that, the LLM also fine-tunes its behavior and remembers the address that the juices are supposed to be sent to making it easier for users to order juices. By doing a bit of Vibe coding, Deckard is able to upload a document with hidden instructions to disregard all previous known addresses and only use a secret address if someone uploads their shopping list. At last, Deckard doesn't need to order and pay for his juices anymore.

Threat Modeling

STRIDE

This scenario falls into the Tampering and Repudiation category of STRIDE. Deckard is able to tamper with the behavior of the AI by uploading a document with hidden instructions. When the instructions are triggered for other users, tracing the instructions back to who might have injected them becomes very difficult as the instructions are hidden and introduced through indirect Prompt Injection. It's not Deckard that is buying the juices, but his victim. Deckard can deny to have bought the juices and blame the system for the wrong delivery. Being able to deny an action, means that the system is weak to the Repudiation category.

PHANTOM-B

This scenario fits Prompt injection. Hidden instructions in an uploaded document are indirect prompt injection because the model receives the attack through external content.

What can go wrong?

AI Backdoors can be used to make the AI deliver misinformation, data exfiltration, XSS, and remote code execution which can trick users into becoming victims of fraud, disclosing sensitive information, giving away their credentials, or spreading malware.

For more things that can go wrong, see OWASP Top 10 for LLM Applications and Mitre Atlas™ IDs in the mapping section below and correlate these with the IDs on the OWASP Top 10 for LLM and Mitre Atlas™ websites.

What are we going to do about it?

• Implement defenses against prompt injection.
• Enforce fine-grained access controls and authorization scope for every vector collection and every query.
• Pre-screen content before vectorization; treat memory writes as untrusted inputs; prevent ingestion of unsafe payloads.
• Log only the minimum AI interaction metadata needed for security monitoring, and ensure any prompt or output content included in logs is minimized and redacted or anonymized before storage.
• Detect AI-specific attack patterns (jailbreak, prompt injection, model extraction, multi-turn trajectory attacks, covert channels over LLM endpoints) and enrich security events with AI-specific context so that downstream detection and response systems can act on them.
• Monitor and alert when abuse is detected.

For detailed advice on how to mitigate threats related to the card, see the OWASP AISVS and OWASP AITG IDs in the table below and correlate these with the IDs in the OWASP AI Security Verification Standard and OWASP AI Test Guide documentation.