Many companies are now experimenting with how LLM and other AI technologies can be used to optimize processes, create new workflows, and solve new use cases. In a fast-moving technology space, we often observe that security is a secondary concern to being “first to market”. By proactively incorporating security considerations into your application, you move from having “security by accident” to “security by design”. In this article we will outline some of the core risks to LLM and AI applications today, how you may increase the security level of your application, and how we at ReTest Security may be able to help you along the way.

Top risks of LLM and AI applications:

At its core Large Language Models (LLM) is a technology built to tell its users all the information it has learned. One of the core ways to ensure that an LLM application does not leak sensitive data is to never teach it that information. Furthermore, malicious training data could be used to bias the output or even to attack the application itself. Hence, having a solid understanding of the data used in the application is key to understanding the security of an LLM application.

Today there are many software-as-a-service (SaaS) and platform-as-a-service (PaaS) solutions that will allow your employees to quickly create a chatbot that has access to all your sensitive data storage. Granted that they have the needed permissions in your infrastructure, this could likely be created very quickly. If proper protections are in place for such tools, then they can be a valuable addition to an organization, but it is important to ensure proper access control on both which data it may access and who may be interacting with the chatbot. Setting up good protections against such (potential) “Shadow AI” chatbots is a prudent security measure. Furthermore, you may have risks of users using unauthorized and unsafe LLM/AI applications to aid their work, which may be storing and reselling sensitive information provided to them.

AI and LLM models are good at working with, and understanding, large amounts of data. To get high-quality output in many business contexts, it likely means you’d have to expose critical data to such models. There exist many different design patterns to give the application knowledge about sensitive data. It is important that your team selects one that allows for suitable data control given your context.

In “conventional applications” developers are used to validating narrowly defined data that, most of the time, comply with specific typing, formatting, and input rules. Furthermore, “conventional applications” are typically deterministic. This means that if you run an application multiple times with the same input you should expect the same output. The paradigm is switching to uncovering new challenges once you introduce technologies such as LLMs into an application. Most LLMs are nondeterministic, meaning that they have a level of randomness built into their text generation. The level of randomness in the generations can be tweaked by the developers, but in most cases, some degree of randomness is a desired feature. Furthermore, validating input and output is complicated by the fact that input is typically much more freely formatted when LLM applications are used. This makes it difficult to make static validation of input of the application. Even if specific keywords are filtered out, an attacker may get the LLM module to leak secrets by using different formatting, for example answering in base64 or by telling a poem where the first letter of the poem is a secret key. Only the creativity of the attacker is the boundary, which makes it a difficult task to verify the input and output of such models.

Most LLM models are naive, and an attacker with enough creativity may be able to trick it into telling secrets and/or performing unintended actions. It may know not to directly tell the secret, but you can likely trick it into disclosing the secret information without it noticing. Many attackers are dedicating a large amount of time to finding such tricks to get the models to perform unintended actions.

As LLM and AI adoption is still a relatively new field, there are not already established clear “best practices” for protecting these applications which have “stood the test of time”. Many such lists are ever-evolving, so it is important that you have employees who stay up to date with the newest security trends. One good place to start is by looking at the latest OWASP Top 10 for LLM applications 2025, which encompasses the following risks:

1. Prompt Injection: A user can break the limitations of the chatbot, misuse it, or gain access to data/functionality that was not intended.
2. Sensitive Data Disclosure: A chatbot is provided with too much access and/or critical data which should not be exposed to all its users, so it can start leaking confidential and critical information.
3. Supply Chain: Malicious actors can embed dangerous information into LLM models and freely distribute them. It’s important to trust the vendor of the model or make your own model.
4. Data and Model Poisoning: Malicious data that could have made its way into the training- or RAG-data could “poison” the model, such that it does not work or gives wrong/dangerous information.
5. Improper Output Handling: If output from an LLM is sent to a dangerous functionality or treated wrongly it could lead to many different issues. LLM output should be treated in the way that developers are used to securing against untrusted user input.
6. Excessive Agency: The LLM model is provided with too much access and is not limited to only being able to fulfill its desired task. Such models should be designed with “principle-of-least-privilege” in mind, as it should be assumed that the most thinkable worst-case scenario of the access provided to an LLM is possible for a malicious actor with enough time to cleverly craft a “prompt injection”.
7. System Prompt Leakage: The system prompt is the “source code of the LLM’s intended functionality”, as it defines what the LLM is allowed to do, who it is supposed to be, and how it should act. This is highly valuable information to anyone trying to abuse the chatbot, so it should be made close to impossible to leak this information.
8. Vector and Embedding: This relates to malicious input being used for “context searching” where the LLM will look up data provided by you when generating the response. Malicious input used to train the RAG could lead to wrong or dangerous output from the chatbot.
9. Misinformation: LLM-chatbots are typically non-deterministic and known for “hallucinating”, so it is not possible (or extremely difficult) to have it be 100% correct all the time. Relying too much on the output can lead to issues as well, along with needing to warn users of this fact.
10. Unbounded Consumption: LLM-chatbots are computationally and financially expensive to run currently. Hence, it is important to ensure that it is not possible to spam or misuse your chatbot. A newer term in cybersecurity is “denial-of-wallet”, which refers to an attack where a chatbot is spammed to build up a huge cloud-bill.

If you are interested in further reading on common AI/LLM risks, then we recommend starting by checking out the following resources:

• MITRE ATLAS
• OWASP AI Exchange
• MIT AI Risk Repository
• Google SAIF

How may you increase the security level of your application?

As discussed in the threat section, one of the core risks of LLM and AI applications is that the chance of leaking critical data is typically higher than in a conventional application. The reason is that for LLM and AI applications to function well for your use case, they likely need access to sensitive information. Furthermore, it is not a trivial task to validate the input and output of LLM/AI applications. In this section, we will outline some general recommendations that can be applied to increase the security level of your LLM/AI application.

• Follow “principle-of-least-privilege”, also for your data: As it is difficult to fully protect LLM/AI applications from misuse, then ensuring that you only give the application the data and functionality which is strictly necessary is important. While it may take more time to test and sort which data is needed, compared to just giving the application access to most of your data, it greatly reduces the risk of accidental data leakage. Furthermore, if you give the application access to functionality such as reading specific websites/domains, then ensuring that only these trusted sources may be visited greatly reduces the chance of indirect prompt injection by getting it to visit an attacker-controlled website.
• Deeply understand your data flows: We highly recommend that you conduct a threat modeling exercise for your application. If your time and budget do not allow for a full detailed threat modeling exercise, then at least spending time to create a high-level data flow diagram (DFD) using a whiteboard or FOSS tools like OWASP Threat Dragon is a great investment. This gives you a deep understanding of your attack surface. A common thing for LLM/AI applications is that the potential ways of affecting the input data of your application are much larger than you may initially expect. Especially, once you start utilizing methodologies such as retrieval augmented generation (RAG) to pull in data from many sources to increase the data quality and give your application your unique context.
• Try to validate input and output: Although it is a difficult task you should still try to validate both input and output (I/O) of your LLM/AI module. Similar to setting up a web application firewall (WAF) on your web application, attempting to validate I/O  in an LLM/AI application does not 100% guarantee security, but it greatly increases the effort required for an attacker to be able to successfully attack your application. We currently recommend the following 2-step approach: First, you identify any potential data that you do strictly not wish to have leaked, then you write static checks for if this data is in the output of the model and block it if the data appears in the output. Secondly, as writing static rules is not enough you need another LLM model to verify that the I/O is safe. We recommend that you use another LLM base model family than the one your core functionality is using, this is to reduce the risk of both models being vulnerable to the same prompt injection techniques due to a similar base dataset and technology. Using models that are fine-tuned for this specific use case, such as the Llama Guard models, can further increase the quality of your LLM-powered I/O validation.
• Ensure proper data controls, ex. Using APIs to fetch sensitive data: If you are adding data that should only be accessible to certain users, then consider providing this data via a “conventional API”. This way you can use the normal I/O validation and use the user’s session token to only provide access to the data to which they are intended to have access to. If you have more complex RAG setups, then ensure that you verify that proper data controls are set up.
• Treat LLM/AI output the same way you treat untrusted user input: At first, developers may have an implicit trust that LLM/AI output is safe. Unfortunately, this is typically not the case. When handling its output, you need to treat it with the same care as you handle untrusted user data, to prevent the output from being able to be used to attack your application.
• Reconsider if LLM/AI is the right tool for your use case: Building an application with such modules may be greatly beneficial in many use cases. However, it is worth considering if the added complexity of securing such systems is worth the potential productivity gain, especially if the reason to use it is just to live up to the hype of the technology and FOMO. For example, if your application is mostly handling and displaying smaller amounts of static data, then a “conventional application” may be sufficient in some cases.
• Consider your system prompt as public: Many of the most powerful “state-of-the-art” LLM models frequently have their system prompt leaked by experts. Over time models are becoming better at not disclosing the system prompt, and protecting against prompt injections, but currently, you should assume that your system prompt is purely for UX and that it is public to anyone who is using the application.
• Conduct a pentest of your LLM/AI module: You may have theoretically designed a secure application, however, a skilled attacker may still be able to find ways to attack and abuse the system. Getting a skilled assessment of your application done is highly recommended to highlight potential security issues and potential future improvements.
• Create a detailed threat model of your application: We already highlighted the importance of creating data flow diagrams for LLM applications. If you further expand the threat modeling to look at common risks and mitigations, for example using a model like STRIDE, then you will get a deeper insight into which risks you have, and the effort/cost needed to mitigate these risks. This places your decisionmakers with a solid dataset for deciding which risks may need to be accepted, and which must be mitigated or hardened against.
• Ensure that you have proper rate-limiting: Running LLM/AI modules can be computationally and financially expensive. Hence, you need to ensure that you have proper controls in place against unauthorized spam on your application. This could be by implementing rate limits, preventing bots via CAPTCHAs, and/or limiting the allowed use within a certain time-frame frame pr. user. Malicious actors may wish to perform a “denial of wallet” attack, where they will build up a huge cloud bill for you out of spite.

How can ReTest Security assist you in creating secure LLM and AI applications?

At ReTest Security we ground our approach to LLM pentesting in a solid threat modeling foundation. By focusing on ensuring that all stakeholders have a great understanding of the data flow in the application, we can identify and test for potential critical threats in your application. Building out a high-quality threat model at the early stages of your development process also helps to guide your development team in designing the application in a way where common risks may be easily mitigated. The outcome of the threat model also functions as a great way to communicate risks to the rest of your organization and is a great step towards compliance with EU AI Act article 9.

Building on top of our extensive knowledge of conventional application security, some of our consultants have further specialized in pentesting LLM/AI-specific issues. These include various ways of getting an LLM to perform unintended actions, data exfiltration, and business logic abuse to the benefit of someone trying to attack your application. Getting such a test performed on your application is a great step towards compliance with EU AI Act article 15 point 5, which highlights the importance of a secure application that cannot be easily manipulated.

Should you wish to hear more about how we may be able to help your team secure your LLM/AI application(s), then please book a meeting with us. We would be happy to introduce our approach to pentesting your specific application and/or guiding you toward a more secure application.