AI assistant on your company documents: architecture, GDPR and cost

Most companies that ask for an “AI chatbot” need an internal assistant: something employees can ask “what’s the notice period in our contract with supplier X?” or “which of our products are approved for outdoor use?” and get a correct answer with a link to the source. This guide is for IT and operations leads planning one. The short answer: build it on retrieval over your own documents and systems, enforce permissions when content is retrieved, keep the data in the EU, and test the answers against a set of real questions before anyone relies on them. A simple version over one document collection typically costs €20,000–30,000. Connecting it to your systems and taking it into production is what our fixed-price AI Production Launch is for.

This post covers internal assistants. A public chatbot on your website is a different project with different risks.

What is an internal AI assistant, and how is it different from a chatbot?

An internal AI assistant answers employees’ questions from your company’s own documents and data, with references, instead of from the model’s general knowledge. Almost all of them are built with retrieval-augmented generation (RAG): the system first searches your sources for the relevant passages, then asks the model to answer using only those.

It helps to be clear about which of three things you’re building:

Start with an assistant that only reads. Add actions later, with a person approving each one.

Which documents and systems can it answer from?

The best first sources are documents that people ask about all the time and that have a clear owner:

• Policies and handbooks: HR, IT, quality and safety.
• Contracts: supplier and customer agreements, renewal dates, liability terms and service levels.
• Product data: specifications, certifications and approvals from your PIM or ERP.
• Project documents, past tenders and offers.

These usually live in SharePoint or OneDrive, Google Drive, Confluence, a file server, or an ERP or CRM reached through its API. One distinction matters: data that changes by the hour, such as prices, stock levels or order status, should be looked up live through the system’s API when someone asks, not copied into the index, or the assistant will give yesterday’s answer with confidence. Start with one or two sources, not the whole intranet.

How does the architecture work?

Sources: SharePoint, contracts, PIM/ERP
   |  sync, parse, split into passages, keep permissions
   v
Search index: keyword + vector search, with access lists
   |  find the passages this user is allowed to see
   v
Assistant service  <-->  model API in an EU region
   |  answer with sources, or "not in the documents"
   v
Teams, intranet or web app  ->  logs, evaluation, cost monitoring

1. Ingestion: connectors sync documents and parse them, including tables and scanned pages (with OCR). Each passage keeps its metadata: source, owner, date, version and who may see it.
2. Search index: documents are split into passages and indexed for both keyword and vector (meaning-based) search. You need both, because article numbers, contract IDs and exact terms are easy to miss with vector search alone.
3. Retrieval: the system filters on the user’s access first, then ranks the remaining passages and keeps the best handful.
4. Answering: the model gets the question and those passages, with instructions to answer only from them, cite the source and say so when the answer isn’t there. The model runs through an EU-region endpoint.
5. Live lookups: for prices, stock or order status, the assistant calls the ERP or CRM at question time.
6. Interface: put it where people already work, such as Microsoft Teams, the intranet or a simple web app with single sign-on.
7. Logging and monitoring: questions, retrieved sources, answers, user feedback and cost per answer.

We used the same building blocks, RAG over business data and AI agents adapted to each user’s role, in the UpliftIQ decision intelligence platform.

How do you make sure people only see what they’re allowed to?

This is the part that most often goes wrong, and the part your security team will ask about first.

• Mirror permissions from the source systems, such as SharePoint groups or ERP roles, and tie them to your single sign-on, for example Microsoft Entra ID.
• Filter at retrieval. The model should never receive a passage the user couldn’t open in the source system. Instructions in the prompt are not access control.
• Sync permission changes as often as content changes, so someone who changes role loses access the same day.
• Leave the most sensitive material, such as salaries, health information or ongoing HR cases, out of the index unless there’s a clear need and a separate decision.
• Test with at least two roles. A permission leak blocks the release; it isn’t a bug to fix later.

What does GDPR mean for an assistant on company data?

This isn’t legal advice, but these are the points that come up in every project:

• Your documents contain personal data: names in contracts, employee details in HR files. Decide the purpose, check the legal basis, and leave out what the assistant doesn’t need.
• Host the application and run the model in an EU region, through an endpoint that doesn’t use your data for training. Every provider involved should be listed as a subprocessor in your data processing agreement.
• Questions often contain personal data too. Redact logs where you can, keep them for a short, defined period, and restrict who can read them.
• If your supplier’s engineers are outside the EU, the agreement has to cover that transfer. Ours work from Ukraine, so we sign a data processing agreement with the EU Standard Contractual Clauses before anyone on the team touches personal data. Our security and data protection page has the details.
• If the assistant handles sensitive data or a lot of employee data, carry out a data protection impact assessment (DPIA) before launch. The Swedish Authority for Privacy Protection (IMY) publishes guidance on when one is required.
• The EU AI Act treats AI used for decisions about employees, such as recruitment or performance evaluation, as high-risk. An assistant that answers policy questions isn’t in that category, but users should still know they are talking to an AI.

How do you know the answers are good enough?

Build an evaluation set before you build the assistant. Ask the people who answer these questions today, such as HR, the IT helpdesk or sales support, for 50–100 real questions from the last few months, each with the correct answer and the source document. Include questions whose answer isn’t in the documents, and questions the person asking isn’t allowed to have answered. Over time, grow the set to 200–1,000 questions using failures from production.

Re-run the set every time you change the prompt, the model, the retrieval settings or the sources. That’s how you find out a change made things worse before users do.

What does an internal AI assistant cost?

Running costs come on top. Model usage for an assistant with 250 weekly users is about $3,100 a year in raw tokens on a mid-tier model at current prices, and roughly double that once you add evaluation runs and test environments. Hosting in your own cloud account is usually tens to a few hundred euros a month. The cost that’s easiest to forget is people: someone has to own each source and keep it current. We break the numbers down in the real cost of AI integration.

Why do document assistants fail after launch?

• Stale content. Old versions of a policy stay in the index, and the assistant quotes them. Give every source an owner, mark documents with a valid-until date, and remove superseded versions.
• Guessing instead of refusing. An assistant that always produces an answer gives confident, wrong answers with citations. Make “not in the documents” an acceptable answer, and test for it.
• Permission shortcuts “just for the pilot” that never get removed.
• It lives outside the tools people use, so nobody opens it after the first week.
• Feedback that nobody reads. Thumbs-down answers are your best test cases; review them weekly.
• Scope creep. “AI for everything” before the first use case is trusted.

Should you build it, or use Copilot or ChatGPT Enterprise?

If your documents are in Microsoft 365 and general search and summaries are enough, try an off-the-shelf tool such as Microsoft 365 Copilot first. It’s faster and cheaper to start. Build your own when you need live data from the ERP or PIM, strict per-document permissions across several systems, answers tied to specific contract clauses, control over the model, hosting and logs, or the assistant as one step in a workflow rather than a separate chat window.

What to do next

Pick one group that gets the same questions over and over, such as HR, the IT helpdesk or sales support, and collect 50 real questions from the last month with the right answers. That’s your first evaluation set, and it’s the fastest way to see whether an assistant would help. When you want it built and in production, the AI Production Launch starts with a 5-day audit for €3,500, credited in full against the launch if you start within 60 days.