Large Language Models (LLM4)

This card describes how an attacker can use prompt injection to manipulate an LLM into leaking sensitive data.

Scenario: David's user prompt disclosure scenario

Example

David loves trading, and he loves cryptocurrency. Lately, he has become familiar with CryptoniteAI, the latest LLM platform for finding hot tips on cryptocurrency trading and investment. David wonders whether it is possible to know what other users on the platform are doing. After learning about prompt engineering and prompt injection, he constructs some clever prompts to get the LLM to reveal what other users are asking the AI. And, wouldn't you know, after some clever adjustments, he finally manages to get the LLM to reveal other users' emails, names, usernames, and what they are asking, almost in real time. Armed with emails, names, and information about which coins other users are interested in, he decides to trick his victims into taking part in his cryptocurrency pump schemes. Little do they know that David will be cashing in on the movement while they are left in the red.

Threat Modeling

STRIDE

This scenario falls into the Information Disclosure category of STRIDE. David is able to get the LLM to send him information about other users, revealing confidential, sensitive information that can be leveraged for fraud.

PHANTOM-B

This scenario fits Missing security engineering. Insufficient controls over the data available to the model and its output can allow sensitive information or system prompts to be disclosed.

What can go wrong?

LLMs, especially when embedded in applications, risk exposing sensitive information. This can result in data breaches, privacy violations, and intellectual property theft.

For more things that can go wrong, see OWASP Top 10 for LLM Applications and Mitre Atlas™ IDs in the mapping section below and correlate these with the IDs on the OWASP Top 10 for LLM and Mitre Atlas™ websites.

What are we going to do about it?

• Implement defenses against prompt injection.
• Implement controls to prevent cross-tenant information from leaking through the shared infrastructure components like inference caches and shared model state.
• Implement controls to prevent cross-tenant and cross-user information leakage during retrieval and prompt assembly.
• Ensure data from Model Context Protocol (MCP) sources are validated and verified to be trustworthy before use.
• Implement model-extraction defenses to protect the LLM against intellectual property theft.
• Continuously monitor and identify backdoored or poisoned input at inference time especially if external data sources are used.
• Log only the minimum AI interaction metadata needed for security monitoring, and ensure any prompt or output content included in logs is minimized and redacted or anonymized before storage; monitor and alert when abuse is detected.

For detailed advice on how to mitigate threats related to the card, see the OWASP AISVS and OWASP AITG IDs in the table below and correlate these with the IDs in the OWASP AI Security Verification Standard and OWASP AI Test Guide documentation.