Table of Contents
November 10, 2023
|15m
Explore What is There in OWASP Top 10 for LLM with TheSecMaster
The emergence of large language model (LLMs) applications like ChatGPT has sparked a revolution in artificial intelligence. These models can generate remarkably human-like text and code, unlocking new possibilities. However, the excitement over LLMs has caused their integration and adoption to rapidly outpace security considerations. As organizations implement LLMs into client offerings and business operations, they expose themselves to serious vulnerabilities.
LLMs introduce new attack surfaces and risks compared to traditional software applications. Their core natural language processing capabilities can be exploited in ways developers are unaccustomed to. Attackers are already developing methods to manipulate LLMs through crafted inputs. Without proper safeguards, LLMs risk leaking sensitive data, enabling social engineering, and threatening backend infrastructure.
To address these concerns, OWASP recently released its OWASP Top 10 for Large Language Model Applications report. OWASP is an international nonprofit improving software security through community initiatives. Their Top 10 for LLM list identifies the most prevalent and dangerous vulnerabilities discovered in real-world LLM apps. It distills insights from nearly 500 cybersecurity experts worldwide into actionable guidance for securing LLMs.
This blog post will provide an overview of the key vulnerabilities highlighted in the OWASP Top 10 for LLM report. We’ll summarize examples for each vulnerability, explain how attackers can exploit them, and offer prevention tips. Understanding these risks is the first step toward properly securing your LLM apps. As LLMs become further entrenched across industries, mitigating these vulnerabilities grows increasingly crucial.
Introduction to OWASP and the Significance of LLM Security
For background, OWASP is a nonprofit organization focused on improving software security. Since 2001, OWASP has built an extensive community of cybersecurity experts who contribute their knowledge to develop free and open resources for the developer ecosystem.
Some of OWASP’s popular offerings include:
OWASP Top 10 – A list of the most critical web application security risks. This provides a great starting point for organizations to have security conversations and build a risk mitigation strategy.
OWASP Testing Guides – Detailed manuals and cheat sheets for security testing methods like DAST, SAST, IAST, and manual penetration testing.
OWASP Application Security Verification Standard – A framework of security requirements and controls that can be used to design, develop, and test secure applications.
OWASP Security Automation Tools – Open source tools like ZAP for security testing and automation.
When LLMs exploded in popularity after ChatGPT’s release, OWASP recognized the need for security guidance tailored to these models. The natural language nature of LLMs introduces new nuances and risks not covered by existing resources.
To fill this gap, OWASP assembled a team of around 500 cybersecurity experts from diverse backgrounds to compile the OWASP Top 10 list for LLMs. Contributions came from AI and security companies, cloud providers, hardware vendors, researchers, and others.
The Top 10 list identifies the most critical security vulnerabilities that can manifest in LLM-based applications. It provides a great starting point for development teams to have security conversations and institutes best practices to mitigate risks.
Now let’s explore what is there in OWASP Top 10 for LLM with TheSecMaster!
Summary of OWASP Top 10 for Large Language Model Applications
The OWASP Top 10 for LLM provides a list of the most prevalent and impactful vulnerabilities found in LLM applications. It aims to raise awareness among developers unfamiliar with LLM-specific risks.
The top 10 vulnerabilities are:
LLM01: Prompt Injection – An attacker manipulates an LLM via crafted inputs, causing unintended actions. This is done by overwriting system prompts or manipulating external inputs.
LLM02: Insecure Output Handling – Blindly accepting LLM output without scrutiny can enable attacks like XSS, CSRF, SSRF, and RCE on backend systems.
LLM03: Training Data Poisoning – Manipulating or poisoning the training data can introduce biases, inaccuracies or backdoors into the LLM.
LLM04: Model Denial of Service – Attackers can overload LLMs with carefully crafted inputs that are resource intensive to process. This results in service degradation.
LLM05: Supply Chain Vulnerabilities – Vulnerabilities in third party models, datasets, libraries or plugins used in building LLM apps can compromise security.
LLM06: Sensitive Information Disclosure – Lack of data sanitization can cause LLMs to reveal confidential or sensitive information through their outputs.
LLM07: Insecure Plugin Design – Poor input validation and insufficient access control in LLM plugins increases risk of exploits like code execution.
LLM08: Excessive Agency – Granting excessive permissions and autonomy to LLMs enables a broader range of unintended consequences from ambiguous outputs.
LLM09: Overreliance – Overdependence on LLM outputs without enough human validation can propagate misinformation, legal issues and other risks.
LLM10: Model Theft -Stealing proprietary LLM models can lead to IP theft, financial losses, and unauthorized access to sensitive data within the model.
LLM01: Prompt Injection
Prompt injection involves manipulating an LLM by crafting inputs that cause it to execute unintended and potentially malicious actions outside of the developer’s intent. Since LLMs process natural language, they cannot inherently distinguish between legitimate prompts and data that has been manipulated with malicious intent.
There are two main types of prompt injections:
Direct Injections: Also known as “jailbreaking”, this involves directly overwriting or replacing the LLM’s system prompt. By providing a custom prompt, attackers can interact with backend systems, data stores, and internal functions through the compromised LLM.
Indirect Injections: Here, prompts from external sources like websites or files are manipulated to contain malicious instructions. When these prompts are passed to the LLM during normal usage, the injected content hijacks the conversation context, tricking the LLM into undertaking unauthorized actions dictated by the attacker.
Example Attack Scenarios:
An attacker performs a direct prompt injection, instructing the LLM to ignore the developer’s prompts and instead query sensitive user data or internal functions. The compromised LLM now acts as an agent for the attacker.
A website contains indirect prompt injections. When summarized by an LLM tool, these instructions trick the LLM into soliciting sensitive info from the user and exfiltrating it via JavaScript back to the attacker.
An indirect prompt injection exploits a plugin linked to an e-commerce site. This causes the LLM to make unauthorized purchases on behalf of the victim user.
How to Prevent:
Enforce least privilege controls, only granting the LLM necessary access to backends via granular API permissions and tokens.
Implement human confirmation for any extensible functionality like plugins to prevent unauthorized actions.
Use syntax like ChatML to distinguish external content from user prompts.
Establish trust boundaries between the LLM, external sources, and plugins. Treat the LLM output as untrusted.
LLM02: Insecure Output Handling
This vulnerability occurs when downstream components blindly accept LLM-generated output without proper validation and scrutiny, similar to an insecure direct object reference vulnerability. Successful exploitation can lead to a wide range of impacts like XSS, CSRF, SSRF, privilege escalation, or even remote code execution.
Since LLM output can be manipulated via prompt engineering, it should not be implicitly trusted. Proper controls need to be in place to sanitize and validate any data passed from the LLM to downstream functions or external users.
Example Attack Scenarios:
An LLM chatbot passes user input directly to a command execution function without sanitization. This allows arbitrary code execution on the backend.
A summarizer tool’s LLM is tricked using prompt injection to exfiltrate sensitive user data by encoding it within the generated summary text returned to the user.
LLM generated SQL query text is passed to a database without validation, allowing deletion of tables.
How to Prevent:
Treat LLM output like any other untrusted user input, validating and sanitizing it before passing to backends.
Encode LLM output to users as per OWASP standards to prevent unintended code execution from JavaScript or Markdown.
Adhere to secure coding practices like OWASP ASVS for input validation, sanitization and output encoding.
Implement additional controls like rate limiting LLM functionality to reduce risk.
LLM03: Training Data Poisoning
This vulnerability involves manipulating the training data or fine-tuning process of an LLM to introduce biases, vulnerabilities or unethical behaviors. Since LLMs are trained on raw text data, poisoning the data can fundamentally compromise the model’s security and trustworthiness.
The impacts include biased or incorrect outputs, performance degradation, and reputational damage. Even if the problematic outputs are distrusted, risks remain from impaired model capabilities and loss of user trust.
Example Attack Scenarios
An attacker adds intentionally inaccurate or falsified documents to the training data corpus. This causes the victim LLM to be trained on bad data, reflected in its misleading outputs to end users.
The training data contains embedded biases causing the LLM to learn unethical associations that are reflected in its generative outputs. This leads to reputational damage for the LLM provider.
A competitor poisons public data sets that a victim LLM provider is using for training, intentionally corrupting the model to reduce its capabilities.
How to Prevent:
Carefully vet training data sources, especially crowd-sourced public data. Use trusted suppliers and data sets only.
Implement data sanitization pipelines to filter out poisoned content using techniques like statistical outlier detection.
Enable capabilities like federated learning and adversarial training to minimize the impact of poisoned data on models.
Perform extensive testing to detect abnormal model behavior that could indicate poisoning.
LLM04: Model Denial of Service
Attackers can abuse LLMs to consume excessive computational resources, leading to service degradation, high infrastructure costs, and potential context window manipulation. This vulnerability is amplified due to the resource intensive nature of LLMs.
By carefully crafting queries, attackers can create conditions leading to disproportionately high resource consumption beyond normal usage levels. This slows down the system, impairing responsiveness for legitimate users.
Example Attack Scenarios:
The attacker floods the LLM with variable length inputs approaching the context window limit. This aims to exploit inefficiencies in the LLM’s variable length input processing.
A malicious website causes excessive resource consumption during LLM based content summarization by inducing recursive context expansion.
The attacker sends a stream of continuously overflowing input exceeding the LLM’s context window capacity, consuming excessive resources.
How to Prevent:
Set strict input size limits based on the LLM’s context window to prevent resource exhaustion.
Enforce API rate limiting to restrict excessive requests per user.
Continuously monitor for abnormal resource consumption patterns that could indicate denial of service attempts.
Configure load balancing and auto-scaling to handle variable traffic bursts.
LLM05: Supply Chain Vulnerabilities
Vulnerabilities can arise at any stage of the LLM supply chain – including training datasets, pre-trained models, and plugins. This leads to issues like training data poisoning, introduction of biases, and exploitation of vulnerable model components.
With multiple third parties involved, adequate security reviews and partnership vetting is essential to avoid compromising the LLM’s security posture.
Example Attack Scenarios:
An attacker exploits a vulnerability in a Python package to compromise the LLM development environment and steal proprietary training data or model architectures.
A pre-trained model downloaded from an online model marketplace contains intentional poisoning to generate biased outputs that benefit the attacker.
An LLM plugin with vulnerabilities gives attackers a vector to bypass input sanitization and exploit the backend host system.
How to Prevent:
Perform extensive due diligence on all third-party suppliers – training data providers, pre-trained models, and plugins.
Maintain software bill of materials (SBOM) and patch components with known vulnerabilities.
Isolate and containerize third-party integrations to limit blast radius from potential compromise.
Use techniques like model watermarking and output auditing to detect poisoning attempts.
LLM06: Sensitive Information Disclosure
LLMs can inadvertently reveal confidential or sensitive information through their responses if proper controls are not in place. This leads to issues like unauthorized data access, intellectual property exposure, privacy violations, and compliance breaches.
The core problem arises from training models on data that has not been adequately sanitized, resulting in memorization of sensitive details. Lack of access controls on LLM responses further increases the risk of unauthorized exposure.
Example Attack Scenarios:
Incomplete filtering of sensitive information during training causes the LLM to memorize and later expose private user data in its responses.
An unsanitized resume processed by an LLM recruiting tool leads it to disclose details about the candidate that should remain private.
Insufficient access controls on LLM query responses allows unintended exposure of confidential business data to unauthorized users.
How to Prevent:
Implement robust data sanitization pipelines to scrub sensitive details before training.
Enforce strict access controls on training data based on the principle of least privilege.
Mask or embed sensitive data to prevent memorization while preserving utility.
Clearly communicate risks of potential information disclosure through Terms of Use and disclaimers.
LLM07: Insecure Plugin Design
LLM plugins with insufficient input validation and weak access control can be exploited by attackers to achieve objectives like code execution, data exfiltration and privilege escalation.
Since plugins interface between the LLM and external systems, vulnerabilities make them an attractive target. Their inherent lack of application layer controls due to context limitations also increases risks.
Example Attack Scenarios:
An LLM plugin allows arbitrary code execution due to lack of input sanitization, enabling an attacker to compromise the backend host system.
Inadequate authentication in a plugin allows escalation of privileges, allowing an attacker access to unauthorized data.
A vulnerable plugin gives attackers a vector to bypass filters and exploit additional systems that the LLM can access.
How to Prevent:
Enforce strict input validation in plugins and limit exposed functionality to only what is essential.
Implement least privilege access control principles in plugins to isolate them and limit blast radius.
Adhere to secure coding methodologies like OWASP ASVS when developing plugins.
Perform extensive security testing of plugins including SAST, DAST and IAST scans.
LLM08: Excessive Agency
Excessive agency refers to granting an LLM excessive permissions, functional scope or autonomy that enables it to undertake potentially damaging unintended actions.
The root cause lies in capabilities like plugins, data access, and downstream functions provided to the LLM beyond what is strictly necessary for its core functionality.
Example Attack Scenario:
A mailbox reader LLM plugin that only needs read access is erroneously given send message permissions as well. This allows an attacker to exploit the plugin to send unauthorized emails.
An LLM personal assistant can access and manipulate a broad range of user data due to insufficient scoping of data access permissions.
How to Prevent:
Carefully limit functionality and permissions granted to the LLM on a need-to-have basis.
Implement human confirmation requirements for sensitive operations performed by LLM systems.
Validate LLM plugin actions against security policies before calling downstream APIs.
Adopt a zero trust approach with LLMs and minimize their scope of influence.
LLM09: Overreliance
Overreliance refers to the excessive dependence on LLMs for decision making, content generation or other capabilities without sufficient oversight and validation.
This can lead to issues like propagation of misinformation, legal liability due to inappropriate content, and integration of LLM-suggested insecure code.
Example Attack Scenarios:
LLM generated news articles require numerous corrections due to hallucinated or factually inaccurate information.
Insecure code suggested by an LLM during software development introduces vulnerabilities when integrated into the application.
An LLM personal assistant provides inaccurate medical advice to a user, leading to harm.
How to Prevent:
Establish stringent review processes for LLM outputs with validation against reliable external sources.
Implement safeguards like visual warnings and content flags to alert users about potential LLM inaccuracies.
Maintain human oversight and approval workflows for high-risk LLM-generated outputs like news articles or code.
Adhere to stringent secure coding practices when leveraging LLMs for software development.
LLM10: Model Theft
Attackers may steal models to extract sensitive information, replicate capabilities, or stage further attacks using the stolen model against its owners.
Example Attack Scenarios:
An attacker exploits a vulnerability to gain access to proprietary model repositories, exfiltrating LLM models for competitive advantage.
A malicious insider leaks confidential model architecture details and training data.
The LLM’s public API is abused to stage extraction attacks, retrieving sufficient information to replicate model capabilities.
How to Prevent:
Implement robust access controls, encryption and monitoring to safeguard proprietary models.
Detect extraction attacks by analyzing API usage patterns and monitoring for suspicious activity.
Consider model watermarking to track provenance and enable identification if stolen.
Maintain rigorous security protocols across model development, training, and deployment pipelines.
Wrap Up
The recent explosion in the adoption of large language model applications like ChatGPT has opened up new capabilities but also introduced complex security risks that many developers are just beginning to grasp. OWASP’s Top 10 for LLM list provides a comprehensive overview of the most prevalent vulnerabilities that developers need to be aware of when building LLM applications.
As highlighted in this post, threats like prompt injections, training data poisoning, blind trust in LLM outputs, and excessive permissions granted to models can have serious security consequences if not addressed adequately. By understanding these risks and applying OWASP’s recommended prevention strategies, organizations can develop more robust and secure LLM apps.
OWASP Top 10 for LLM provides an invaluable starting point to build expertise in this emerging domain of AI security. But it is just the tip of the iceberg. As LLMs continue advancing rapidly, new and unforeseen risks will surely emerge.
Organizations leveraging these models need to prioritize continuous education on LLM security, implement rigorous development practices, perform extensive testing and audits, and monitor systems closely. Adopting a proactive security posture and zero trust mindset will be key to unraveling the multilayered risks introduced by large language models going forward.
We hope this post serves the purpose and becomes a good source of information for the list of security risks listed in OWASP Top 10 for LLM. Thanks for reading this post. Please share this post and help secure the digital world. Visit our website, thesecmaster.com, and our social media page on Facebook, LinkedIn, Twitter, Telegram, Tumblr, Medium, and Instagram and subscribe to receive updates like this.
You may also like these articles:
Arun KL
Arun KL is a cybersecurity professional with 15+ years of experience in IT infrastructure, cloud security, vulnerability management, Penetration Testing, security operations, and incident response. He is adept at designing and implementing robust security solutions to safeguard systems and data. Arun holds multiple industry certifications including CCNA, CCNA Security, RHCE, CEH, and AWS Security.
