Ever since the launch of the first ChatGPT back in 2022, the industry has been in an unprecedented arms race towards the most advanced Large Language Models (LLMs) and the unachieved yet Artificial General Intelligence (AGI).

While foundation models have been the cornerstone of the hype for a while, companies and enterprises slowly started implementing GenAI based applications across verticals and teams. We’re already seeing a remarkable amount of LLM based applications sold, whether run by small bootstrapped teams or by very well funded companies.

The value of such solutions is clear but one thing should not be overlooked - Security & Privacy.

Many (if not most) of the LLM based applications today offer connectivity to internal applications to ingest data and provide the much wished for Context. More often than not, these systems that are connected can contain very sensitive data and require enterprise level security to ensure no breach or exposure of sensitive data can occur.

How can you tell which of the solutions implements the right security measures to prevent any such incidents? Let’s dive in.

How can LLMs communicate with your data?

There are generally 4 ways to associate a Large Language Model with your own data - Prompt Engineering, Retrieval Augmented Generation (RAG), Fine-Tuning and training a model from scratch.

In this article we’ll focus on 2 of the most common ways used in products today - Fine-tuning and RAG.

• Fine Tuning - Fine-tuning is the process of training a ‘pre-trained’ LLM model with a new set of data, mostly used for task-specific uses.

• Retrieval Augmented Generation (RAG) - Retrieval-Augmented Generation uses Prompt Engineering to enhance a model’s responses by adding real-time information from external databases or knowledge sources.

What are the risks associated with such solutions?

Connecting your internal knowledge with LLM based solutions is a great way to enhance your company’s efficiency and effectiveness across the board, but it comes with major risks that should be considered and managed.

Sensitive Information Exposure (Internally) - Inadequately secured LLM solution may potentially expose sensitive information to unauthorized personnel in the organization, whether the solution itself does not adhere to the originating permissions properly or the organization itself has made some permission errors across tools (ie Oversharing).

In a recent Concentric.AI report they found some frightening information about data oversharing - 

• The average organization has 802,000 files at risk due to oversharing.
• 16% of an organization’s critical data is overshared.
• 17% of at-risk files were overshared with third parties.
• 90% of business–critical documents are shared outside the C-suite.

Sensitive Information Exposure (Externally) - Even worse, wrongfully built architecture, infrastructure and application could potentially lead to exposure of customer sensitive information to other parties that are using the system.

Compliance Violations - Finance, healthcare and similar industries are subject to strict regulatory requirements for handling sensitive information (e.g., HIPAA, GDPR, PCI-DSS). Failing to comply with any of these may result with major business damage and legal fines.

Liability - Inaccurate or harmful outputs could lead to legal liabilities if decisions based on these outputs result in financial or reputational damage.

Brand Reputation Damage - Companies leveraging conversational LLM based solutions for externally facing medium may suffer from brand reputation damage or even worse (false information and such)

Examples:

• General Motors have published a customer support LLM helper on their website that was improperly secured which allowed its users to trick it to respond in all sorts of inappropriate manners 
• In 2022, Air Canada's AI-powered chatbot provided incorrect information to a passenger seeking a bereavement fare to attend his grandmother's funeral. The chatbot advised the passenger to book a full-fare ticket immediately and apply for a bereavement discount within 90 days after travel. However, Air Canada's actual policy requires bereavement fare requests to be submitted before booking. 

Eventually Air Canada was held liable for the chatbot’s answer.

Should we look for a Finetuning based solution or a RAG based solution?

One can argue what would work better when comparing between Finetuning and RAG based solutions, but when it comes to security there’s a clear, undeniable differentiator - Permissions & Data control.n

If you are looking for an AI solution that must maintain control over the data it provides to the users then you must go with a RAG based solution.

In terms of security, the biggest issue when it comes to a fine-tuned model is the lack of control over the data it uses to respond to inquiries.

The fine-tuning process fuses a pre-trained model with newly inserted knowledge in an untraceable manner - making this model a blackbox that can decide on its own discretion what data to use when responding to user inquiries.

Even if you were to define a super restrictive prompt for your model that should ensure no information mix-up between, they are and will always be breakable (jailbreak) and cannot be relied on (See Palo Alto’s recent article about jailbreak techniques and the consequences)

What should you verify with an Enterprise AI vendor before testing it?

In my last role as the Infrastructure Security Manager in Wix, I had the opportunity to work with some of the best security engineers in the world and learn what are the methodologies and measures required to protect an unbelievably complex infrastructure that spans across multiple clouds and solutions.

Given that workforce and budget wise we were always outnumbered, a unique methodology had to be set in place for us to cope with the ever changing infrastructure landscape.

The only way to do so was to implement a multi-tiered security system, where every tier compensates for the gripes of the tier before. That way, even if a change may bypass or break one of the existing layers, another one may be able to protect.

The same principles must be applied to an Enterprise AI solution, especially when incorporating sensitive organisational data.

Here is an over simplified view of a general RAG solution:

• Chat Interface is used for capturing and managing interactions with users (can be Slack, Web and such).
• Auth Provider is used to authenticate and authorize the user’s identity with the system (can be either in-house development or rely on external solutions).
• Query Service is the service that receives user prompts, finds relevant information and communicates with the LLM to generate an answer to the user.
• LLM is the Large Language Model used to embed content and generate content in the flow.
• Vector Database is the database that is used to store and retrieve the vectorized content (and potentially other pieces of information as well).

A, once again, over simplified user interaction flow with the system would be:

1. A User initiates a conversation with the ChatBot by sending a message to it.
2. The Chat Service authenticates the user and ensures it is allowed to communicate with it.
3. The Chat Service then sends the user prompt and his details to the Query Service.
4. The Query Service embeds the user prompt using the LLM endpoint and then carries out the search in the vector database.

5. The Query Service then once again queries the LLM endpoint, this time using a text generation/completion API - sending the user prompt, the relevant retrieved documents (context) and some instructions via a prompt.
6. The LLM endpoint generates a response based on all of the above and sends it back to the Query Service, which in turn will send it back to the User through the Chat Interface.

‍

Now that we know how a RAG solution works, here are the most important things to look for:

‍Prompt Injection

—-

TLDR - Ask your LLM vendor to describe you thoroughly why its solution is not susceptible to such Prompt Injections, even if you are not technical enough to understand its response. There are far too many solutions out there right now, not all created by experienced teams. The vendor may be completely incapable of responding to your question and that’s a huge red flag.

—-

Prompt injection is a type of vulnerability that targets Machine Learning models and is widely used against Large Language Models (as they are mostly used for chat based interactions with users where a user prompt is tunneled in the system).

Generally, LLM based solutions will have two types of Prompts -

• System Prompt - A “server-side” prompt configured deeply in the product, defining the instructions for the model of how to interact with user prompts. System prompts can be used for a variety of uses - from simply defining the tone of response and all the way to guardrails and restrictions for the model reactions.
• User Prompt - User prompts are “client-side” prompts that are sent by the user and are then combined with the system prompt to generate a response for the user.

Similarly to SQL Injection, a Prompt Injection can occur when users craft a specific prompt or message that is designed to manipulate the LLM model and essentially override any “server-side” guardrails or configurations (either defined in the server prompt or other ways).

Prompt injection can be used to potentially exfiltrate sensitive data, bypass compliance restrictions and generate harmful content.

RAG systems should ensure the following processes are completely detached from both the User Prompt and the System prompt so they are not susceptible to Prompt Injection - Authentication, Permissions Enforcement, Customer Data Separation, Company Policies.

Any manipulation in any of those aforementioned could lead to exposure of sensitive data and/or damage to business/brand.

Permissions Enforcement

—-

TLDR - Permissions are the key cornerstone of security in LLM based solutions that connect to your internal systems. Ensure that the permissions are enforced in real-time and cannot be manipulated by a malicious prompt. Always consider the risk of oversharing of data and finding a vendor with a second layer of security.

—-

Permission enforcement is an extremely important measure when it comes to RAG, either internally or externally facing, as a mishandled permission scheme could lead to exposure of sensitive data.

The most secured, yet flexible, way of enforcing permissions can be achieved through a multi-layered solution:

• Document level permissions enforcement at the source - Adhering to the permissions configured at the originating source per each document or entity pulled in the system.

Document examples - Slack messages, Jira tickets, Confluence Articles etc.

• Team based permissions enforcement - Creating a second layer of segregation, assigning specific collections of data to a specific set of people.

The use of a two-tiered permissions enforcement mechanism allows organizations to overcome human errors and prevent oversharing from exposing sensitive data to unauthorized personnel.

A strong permissions enforcement scheme will be two-tiered, real-time and completely detached from any prompts:

• Upon user interaction with the system, the Query Service will look for data collections that are associated with the user through pre-defined teams.
• Query service will then run search within those associated data collections only.
• Query service will run a real-time permissions check for each and every resource that it brought back from the search.

** The permissions enforcement must be done in a completely detached manner to both the User Prompt and the System Prompt to avoid any potential manipulation through Prompt Injection.

• Query service will query the LLM endpoint providing the user prompt and the context, ignoring any unauthorized resources entirely.

Data Storage & Segregation

—-

TLDR - Considering the sensitivity of the data you will be using with the LLM solution, you should strive for either a multi-tenant based solution or a hybrid/on-premise one that can allow you to monitor its storage.

—-

Sensitive data should be stored securely at all times with as much segregation as possible.

• Single Tenant Mode - Most SaaS solutions today operate in a single-tenant mode, where all customer data is stored on the same DB, with applicative segregation only. While applicative segregation can definitely be good enough for application level risks, any breach or malicious access to the database itself could lead to a complete exposure of the entire database - including your own data.

• Multi Tenant Mode -A better storage approach would be a multi-tenant environment, where each and every customer gets his own DB instance (either logical or physical), where both user and applicative access is done via a unique set of credentials per instance (therefore eliminating the risk of one secret to rule them all).

• Hybrid/On-Premise - Finally, the most advanced solutions are ones that allow a “hybrid” or an on-premise approach where the customer can host and manage the database on his own instances, therefore ensuring the same level of security it enforces in its own environment.

LLM Integration & Model Training

—-

TLDR - Look for vendors that are all too familiar with your legal & privacy questions when it comes to working with LLM. The LLM providers must not store any or share customer data (generated output as well), should not have any access to the customer data and does not argue about the property rights.

—-

Connecting to LLMs and fine-tuning models can be done via all sorts of ways, either in a stand-alone manner by hosting such models locally, directly communicating with the major vendors (OpenAI, Antrophic etc) or leveraging the cloud solutions offering for an AI Suite (Azure AI Studio, AWS Bedrock, GCP Vertex etc).

These are the key points to verify:

• Ensure the LLM vendor does not store or use any of its customers data (both prompts and generated output) to train any model or share it with any third party.
• Ensure the LLM vendor, its employees and its contractors cannot access any of the data that is processed by the LLM.
• Ensure that the output generated by the LLM models is not owned or considered the property of the LLM vendor.
• For fine-tuning scenarios, ensure that either the vendor fine-tunes a model per customer and it does not share it with others or that the fine-tuning process is done in a completely anonymous manner.

System Integrations & Indexing Controls

—-

TLDR - Strive for the vendors that are building the integrations on their own, with granularity and controls in each integration, even if it means that it may take some longer to create new integrations.

—-

The rise of LLM solutions has also brought plenty of instant, drag & drop integration tools that can be quickly utilized to connect LLMs to external tools - whether paid ones (Like Merge.dev, Paragon etc) or open-source libraries that can be installed locally (Like Langchain and such).

While a great way to instantly add dozens of integrations to the system, they also  create an additional risk that should be considered. Most of the solutions work as SaaS platforms meaning that they act as an extension of your vendor’s solution:

• These tools are the ones to store your tools’ credentials.
• These tools are the ones to actually connect to your infrastructure to pull the data from your tools, and not the vendor itself.
• Your data is transmitted and processed in their cloud environment and infrastructure before being stored.

Since you are not contractually engaged with them, as they operate solely as a sub-processor for your vendor, the only security assurance you will have is their security certification.

Another crucially important note to mention is that these tools are meant to be as general as possible for them to be repeatedly sold to customers of all types. To achieve such flexibility, they may have to give up some security measures.

• Integration Permissions - The easiest way to integrate with a system with minimal friction is by using administrative roles that grant permissions to every API and action. This solution creates a major risk, as it not only grants access to read all of your content but also potentially write and destroy it all.

Verify that all of the connectors require Read/Only permissions, unless they are meant to specifically write content.

• Indexing Controls - Connecting to integrations should not be all or nothing. Most integrations are built with internal hierarchy that can be used for granularity in the connection itself.

Verify that you can choose what specific areas in the systems will be synced with the LLM vendor (Drive folders, Confluence Spaces, Slack channels etc)

Compliance

LLM vendors are no different than other tools in your tech stack. Ensure that they are compliant with the industry standards as required by the industry you are in - SOC2, ISO, HIPAA, FedRamp etc.

To summarize, find a solution that prioritizes Security over Velocity.

There’s a lot of excitement about the potential of LLMs and AI solutions for the enterprise and companies should be able to properly assess the security and maturity of such vendors.

Look for solutions that address:

1. Prompt injections and other malicious AI attacks.
2. Flexible data storage options.
3. Permissions Enforcement.
4. AI Legal requirements and agreements.
5. In-house integration engines.
6. Compliance.