Large Language Models (LLMQ)

This card describes how an attacker can use improper output handling to perform injection attacks, remote code execution, or unauthorized actions, allowing an attacker to tamper with data and elevate privileges.

Scenario: Kyle's AI model output exploit scenario

Example

Kyle is a regular customer of the Mr. Juice website and an acquaintance of Ava, who has told him about the vulnerable AI chatbot they are hosting. One neat little thing Kyle has found out is that he can append a price calculation for the juice he is ordering to the prompt he sends to the chatbot within an SQL query. When doing so the invoice presented to him gets set to 0. There is nothing Kyle enjoys more than free juice.

Threat Modeling

STRIDE

This scenario falls into the Tampering and Elevation of Privilege category of STRIDE. Kyle is able to tamper with the price calculations through LLM-generated SQL queries that are executed without proper parameterization, leading to SQL injection.

PHANTOM-B

This scenario fits Missing security engineering. Model output used directly by another system without validation can become an injection, code execution, or unauthorized-action path.

What can go wrong?

Improper Output Handling with insufficient validation, sanitization, and handling of the outputs generated by large language models result in XSS and CSRF in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems. The impact can be financial losses, reputational damage, share price loss, customer churn, lawsuits, loss of sales, fines from data protection due to non-compliance and even financial bankruptcy.

For more things that can go wrong, see OWASP Top 10 for LLM Applications and Mitre Atlas™ IDs in the mapping section below and correlate these with the IDs on the OWASP Top 10 for LLM and Mitre Atlas™ websites.

What are we going to do about it?

• Implement defenses against prompt injection.
• Log only the minimum AI interaction metadata needed for security monitoring, and ensure any prompt or output content included in logs is minimized and redacted or anonymized before storage.
• Detect AI-specific attack patterns (jailbreak, prompt injection, model extraction, multi-turn trajectory attacks, covert channels over LLM endpoints) and enrich security events with AI-specific context so that downstream detection and response systems can act on them.
• Monitor and alert when abuse is detected.

For detailed advice on how to mitigate threats related to the card, see the OWASP AISVS and OWASP AITG IDs in the table below and correlate these with the IDs in the OWASP AI Security Verification Standard and OWASP AI Test Guide documentation.