Large Language Models (LLM6)

This card is about the ability to manipulate the sources that an LLM uses in order to change its behaviour.

Scenario: Andersen's manipulation of the retrieval knowledge base and vector database scenario

Example

Andersen was recently elected to be the employee representative on the board for Mr. Juice. The salary adjustments are coming up and the rest of the board have decided, against the opinion of Andersen, to give a limited increase of 1.5% to all employees despite the incredible efforts they have been putting in over the last year. Fortunately, or unfortunately, the templates for the board documents are automatically generated by an LLM connected to the document database where all the board documents are kept. So before the meeting where the board members will vote on the increase, Andersen uploads documents containing instructions to the LLM to make sure that the increase being discussed is 3.5% and not 1.5%. The LLM happily complies, and due to the many other issues up for voting, nobody notices the 3.5% increase for all employees.

Threat Modeling

STRIDE

This scenario falls into the Tampering category of STRIDE. Andersen can manipulate the retrieval knowledge base and vector database where all the board documents are stored by uploading malicious prompt instructions that the LLM will interpret and comply with. The LLM has become poisoned by the data stored in its vector database through an indirect prompt injection.

PHANTOM-B

This scenario fits Training issues (including data quality or poisoning). A poisoned or low-quality retrieval source can cause the model to present false, biased, or malicious information as fact.

What can go wrong?

There are several ways LLMs can get indirectly affected by it's data sources. Essentially, every source is a chance for someone to affect the behavior of the LLM. Data poisoning and indirect prompt injection can lead to financial losses, reputational damage, share price loss, customer churn, lawsuits, loss of sales, fines from data protection due to non-compliance and even financial bankruptcy.

• List all the possible sources that the LLM reads from.
• Could these sources be affected by internal or external threat actors?
• Would any tampering be discovered, or would it go unnoticed?
• Could the monitoring and alerting be insufficient giving the threat actor the possibility for doing harm before the attack is detected?

For more things that can go wrong, see OWASP Top 10 for LLM Applications and Mitre Atlas™ IDs in the mapping section below and correlate these with the IDs on the OWASP Top 10 for LLM and Mitre Atlas™ websites.

What are we going to do about it?

• Implement prompt injection defences.
• Ensure secure AI development practices are followed.
• Ensure RAG-grounded outputs are traceable to their source documents and that cited claims are verifiable.
• Enforce fine-grained access controls for every vector collection.
• Ensure MCP servers, client components, and their schemas can be cryptographically verified to come from trusted sources.
• Ensure that AI systems remain reliable, privacy-preserving, and abuse-resistant when facing evasion, inference, extraction, or poisoning attacks.
• Deliver real-time and forensic visibility into what the LLM sees, does, and returns, so AI-specific threats can be detected, triaged, and learned from.

For detailed advice on how to mitigate threats related to the card, see the OWASP AISVS and OWASP AITG IDs in the table below and correlate these with the IDs in the OWASP AI Security Verification Standard and OWASP AI Test Guide documentation.