Private AI Architecture: How to Deploy AI Agents Inside an Enterprise

Every request — whether it originates from a person or an upstream system — is authenticated against the enterprise's existing identity provider before it reaches an agent or a model. There is no separate credential store for the AI system to maintain and no shared API key standing in for a real identity. The user's enterprise identity is what the rest of the architecture authorises against: which documents can be retrieved, which actions can be taken, and which results require approval all trace back to who is actually making the request.

The gateway is the single entry point every authenticated request passes through on its way to a processing path. It resolves which of the four paths — RAG, OCR, Voice Assistant or AI Crawler — should handle the request, enforces rate limits and request-level policy, and is the one place in the architecture where every request can be observed regardless of which path it takes. Centralising routing and enforcement here — rather than duplicating security logic inside every individual path — is what keeps the security posture consistent as the product set grows.

Retrieval-augmented generation gives an agent access to the enterprise's own knowledge — documents, records, internal wikis — without retraining a model on it. The retrieval index lives inside the same boundary as the model, and retrieval itself is permission-aware: an agent can't surface a document the requesting identity couldn't already open through the source system. This is the path Conversational AI runs on.

Optical character recognition and document understanding applied to a scan, PDF or phone photo — pulling out the specific fields a task needs and validating them, rather than handing back a wall of unstructured text. Nothing uploaded here leaves the boundary to reach an external vision API. This is the path Image and Document OCR runs on.

Real-time speech understanding and response — transcribing what's said, deciding what to ask next, and speaking back, inside one continuous turn-taking loop rather than a fixed script. The follow-up questions are generated from how the conversation is actually going. This is the path Vaani runs on.

Crawls a target website and extracts the specific signals a task needs — comparing them against a set of criteria, or assembling them into a sourced profile with a confidence score per field. This is the path Sift and Data Intelligence both run on.

The model itself — the layer actually doing the reasoning behind every retrieval query, tool call and drafted response. Model selection is a discovery-stage decision: the smallest open-weight model that reliably clears the accuracy bar for the task, hosted and sized to real traffic rather than over-provisioned for headroom nobody uses. Inference happens entirely inside the deployment boundary, with no route out to a public model API — the model can be fine-tuned on the enterprise's own data without that data, or the resulting weights, ever leaving the environment.

Every request, retrieval, tool call and model decision is logged here, inside the same environment as everything else — not exported to a third-party observability platform by default. This layer covers four related jobs: audit logging a compliance team can query directly, ongoing evaluation of the system against real tasks, ongoing monitoring for quality or cost drift, and the incident trail needed if something does go wrong. An architecture without this layer wired in from day one is not something we consider production-ready, regardless of how the rest of it is built.