Large Language Models (LLMA)

Players can discuss any type of large language model (LLM) attack they think might be possible against the assessment target. If possible, try to identify an attack that is fairly unique to the application's LLM design, tool integrations, or deployment model rather than one already covered by another card in this suit.

Scenario: Invent Your Own Large Language Model Threat

You have identified an attack that misuses inherent large language model functionality or a related design flaw. Inventing a large language model threat can lead to:

1. Prompt Injection: Attackers can craft input that poisons and exfiltrates data or elevates privileges.
2. Sensitive Information Disclosure: System prompts, reasoning traces, or unauthorized data sources can be exposed through adversarial interaction.
3. Supply Chain Attacks: Compromised AI supply chains can affect the integrity of training data, models, and deployment platforms. The result of which is biased outputs, security breaches, or system failures.
4. Data and Model Poisoning: Manipulating data during pre-training, fine-tuning, or embedding can introduce backdoors, biases. Impact can be degraded model performance, biased or toxic content, malware, data exfiltration, identity theft, service unavailability, and data corruption.
5. Improper Output Handling: Insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed on to other components and systems can lead to XSS and CSRF in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems.
6. Excessive Agency: Weak input validation or sandboxing for AI tools and plugins allows unintended code or system command execution.
7. System Prompt Leakage: System prompts or other instructions used to steer the behavior of the model can contain sensitive information that attackers may exploit to facilitate other attacks.
8. Vector and Embedding Weaknesses: Compromising vectors and embeddings when they are generated, stored, or retrieved can (intentional or unintentional) lead to biased or toxic content, malware, data exfiltration, identity theft, service unavailability, and data corruption.
9. Misinformation: Hallucination may be caused by data poisoning or gaps in the training data that may result in factual inaccuracies causing users to become misinformed. The impact can be security breaches, reputational damage, and legal liabilities.
10. Unbounded Consumption: Large Language Model (LLM) application allows users to conduct excessive and uncontrolled inferences. The high computational demands of LLMs, especially in cloud environments, make them vulnerable to resource exploitation and unauthorized usage.

Threat Modeling

STRIDE

The appropriate STRIDE category depends on the specific threat you create and the way the agent is misused.

PHANTOM-B

An invented LLM threat can fit any PHANTOM-B prompt.

What can go wrong?

Attacks on large language models can do irreparable damage to any type of asset. It may be possible to fix an AI model after its data has become poisoned, but it's extremely challenging, costly, and often requires complete retraining. A large language model can also be used as a springboard for other types of attacks on the system if an attacker is able to convince it to execute code or actions on behalf of the attacker. As an AI model also connects to vast amounts of data, given that access controls are insufficiently implemented, the model itself can become a tool for data exfiltration of sensitive information as well as getting stolen. The information can, in turn, be used to make users victims of fraud, identity theft, phishing, and malware attacks. The impact of such attacks can be financial losses, reputational damage, share price loss, customer churn, lawsuits, loss of sales, fines from data protection due to non-compliance and even financial bankruptcy.

For more things that can go wrong, read the OWASP Top 10 for LLM and Mitre Atlas™ documentation.

What are we going to do about it?

To mitigate against threats related to large language models, make sure you design, implement, and verify your system according to the requirements defined by OWASP AI Security Verification Standard and test your system using the OWASP AI Test Guide.