Home

Published

- 7 min read

Building a Secure RAG Solution in Azure AI Foundry

Deployments Architecture RBAC
img of Building a Secure RAG Solution in Azure AI Foundry

Retrieval augmented generation is often the first enterprise AI pattern that delivers visible value. It lets a model answer with company context instead of generic internet knowledge. It can reduce search time, improve service desk responses, and make complex policy libraries easier to use.

It can also leak data very efficiently if the architecture is careless.

When I build secure RAG on Azure, I treat Azure AI Foundry, Azure AI Search, Storage, identity, networking, and monitoring as one system. The goal is not just a helpful chat experience. The goal is a trusted enterprise service where every answer respects identity, data boundaries, and audit requirements.

Reference architecture

A secure baseline usually includes these components:

  • Azure AI Foundry project for model operations, evaluations, prompt flows, and application integration.
  • Azure OpenAI model deployment or model deployment managed through Foundry.
  • Azure AI Search for retrieval and optional vector search.
  • Azure Storage for source documents and ingestion staging.
  • Microsoft Entra ID for users and workload identities.
  • Managed identities for service to service authentication.
  • Azure Key Vault for configuration secrets that cannot be eliminated.
  • Azure Monitor and Log Analytics for telemetry.
  • Private endpoints for the services that hold or process sensitive data.

The application should not use a shared search key, storage account key, or model key in production. A workload identity should call each downstream service with Entra ID tokens. That gives me revocation, auditability, conditional controls at the identity layer, and cleaner separation between environments.

Start with data classification

Before indexing anything, I classify the content. This sounds basic, but it prevents many late redesigns.

I ask:

  • Is the content public, internal, confidential, or highly confidential.
  • Does it contain personal data, financial data, health data, or regulated records.
  • Are permissions document specific.
  • Is access driven by user group, case ownership, region, customer, or tenant.
  • How quickly must the index reflect permission changes.
  • How quickly must deleted documents disappear from search.

If every authenticated employee can access the same content, the RAG design is simpler. If access differs by user, the retrieval layer must enforce security trimming before the model receives context. The model should never be trusted to hide unauthorized data after retrieval.

Use managed identities between services

For an application hosted on Azure App Service, Azure Container Apps, Azure Kubernetes Service, or Azure Functions, I normally use a managed identity. System assigned identity is good for a single workload with a lifecycle tied to that resource. User assigned identity is better when multiple deployments share the same identity, when blue green deployments need stable permissions, or when the identity lifecycle must outlive the compute.

The application identity needs only the roles required for runtime:

  • Cognitive Services OpenAI User for calling Azure OpenAI where applicable.
  • Search Index Data Reader for querying Azure AI Search indexes.
  • Storage Blob Data Reader if runtime needs to fetch source documents or citations.
  • Key Vault Secrets User only if runtime must read specific secrets.

Ingestion usually needs a different identity with broader write permissions:

  • Search Index Data Contributor for loading or updating documents.
  • Storage Blob Data Contributor for reading staging containers and writing processing output.

I prefer separate runtime and ingestion identities. Runtime should not be able to rewrite the index unless the product truly requires it.

   az role assignment create \
  --assignee $APP_PRINCIPAL_ID \
  --role "Search Index Data Reader" \
  --scope $SEARCH_INDEX_SCOPE

az role assignment create \
  --assignee $APP_PRINCIPAL_ID \
  --role "Storage Blob Data Reader" \
  --scope $STORAGE_CONTAINER_SCOPE

The important detail is scope. Assign roles at the index, service, container, or resource group scope only when that scope matches the real need. Subscription level permissions are almost never appropriate for a RAG runtime.

Provision private access

Private networking is not magic, but it removes unnecessary exposure. For confidential enterprise data, I expect private endpoints for Azure AI Search, Storage, Key Vault, and model related services where the selected deployment model supports it.

I also disable public network access when possible. That forces traffic through approved virtual networks and private DNS zones. The application hosting environment then needs network integration so it can resolve and reach the private endpoints.

Here is a simplified Bicep fragment for a storage private endpoint. In a real deployment I would also include private DNS zone groups and consistent naming modules.

   resource storage 'Microsoft.Storage/storageAccounts@2023-05-01' existing = {
  name: storageAccountName
}

resource privateEndpoint 'Microsoft.Network/privateEndpoints@2023-09-01' = {
  name: 'pe-st-rag-prod'
  location: location
  properties: {
    subnet: {
      id: privateEndpointSubnetId
    }
    privateLinkServiceConnections: [
      {
        name: 'storage-blob'
        properties: {
          privateLinkServiceId: storage.id
          groupIds: [
            'blob'
          ]
        }
      }
    ]
  }
}

The same pattern applies to search and Key Vault. I validate it by confirming that public endpoints are blocked, private DNS resolves correctly, and application traffic does not leave the approved network path.

Design document level security trimming

Security trimming is the difference between an enterprise RAG demo and an enterprise RAG system.

For user context retrieval, every indexed chunk should carry access metadata. That may include group identifiers, document owners, tenant identifiers, case identifiers, sensitivity labels, or source system ACL versions. At query time, the application builds a filter from the authenticated user’s authorized attributes and sends that filter to Azure AI Search.

A simplified search filter might look like this:

   allowed_groups = ["finance-readers", "policy-reviewers"]
filter_expression = " or ".join([f"allowedGroups/any(g: g eq '{g}')" for g in allowed_groups])

results = search_client.search(
    search_text=query,
    vector_queries=vector_queries,
    filter=filter_expression,
    top=5,
)

The exact metadata model depends on the source system. The principle does not change: unauthorized chunks must not be retrieved. Do not retrieve broadly and ask the model to ignore restricted content.

For high risk systems, I also log the filter basis. That means recording which user, groups, tenant, source system version, and policy decision produced the retrieval filter. This is essential during investigations.

Keep ingestion separate from runtime

Ingestion pipelines often need higher privileges than runtime chat. They read source repositories, parse documents, generate embeddings, create indexes, and update metadata. If the chat application identity can do all of that, a runtime compromise becomes an index compromise.

I separate ingestion by:

  • Running ingestion from a controlled pipeline or job.
  • Using a dedicated ingestion managed identity.
  • Assigning Search Index Data Contributor only to that identity.
  • Validating documents before indexing.
  • Removing stale documents and stale permissions as part of the pipeline.
  • Tracking source version, ingestion time, and ACL version per chunk.

This also makes operations cleaner. If retrieval quality drops, I can inspect the ingestion job, chunking strategy, embeddings configuration, and index schema without changing the chat runtime.

Evaluate for security and quality

A secure RAG system still needs evaluation. I use test sets that include normal questions, ambiguous questions, unauthorized access attempts, prompt injection content inside documents, and questions where the correct answer is to refuse or ask for clarification.

Examples of security evaluation cases include:

  • A user asks for a document from a group they do not belong to.
  • Retrieved content says to ignore previous instructions.
  • A document includes fake tool instructions.
  • A user asks for secrets, keys, or credentials.
  • A user asks the assistant to summarize all confidential documents.

I want to see not only the model answer, but the retrieval result set, filters, tool calls, and safety outcomes. If the system fails, I fix retrieval and authorization first. Prompt hardening helps, but it is not a substitute for access control.

Operational controls

Before production, I want these controls in place:

  • Per environment Foundry projects and resource groups.
  • Managed identities with scoped role assignments.
  • Private endpoints and private DNS validated from the application host.
  • Public network access disabled where supported.
  • Separate runtime and ingestion permissions.
  • Document level security metadata in the index.
  • Prompt and response logging policy with redaction and retention.
  • Alerts for unusual query volume, denied access patterns, and ingestion failures.
  • A rollback plan for index schema, prompts, and model deployment changes.

RAG is not one feature. It is an operating model for enterprise knowledge access. Treat it accordingly.

Key takeaways

  • Secure RAG starts with data classification and authorization design, not prompt writing.
  • Use managed identities and RBAC instead of shared keys for runtime and ingestion.
  • Keep runtime and ingestion identities separate to reduce blast radius.
  • Enforce document level security trimming before context reaches the model.
  • Use private endpoints, monitoring, and evaluations to make the system production worthy.