Key points
- “Training AI” on your organization usually does not mean building a new model. It means giving an existing system the right context, sources, tools and rules.
- There is a ladder of approaches, from instructions and retrieval to structured knowledge, agents, fine-tuning and, rarely, pre-training. Start at the lightest rung that works.
- Fine-tuning helps with consistent format, style, classification and domain language. It is a poor way to teach facts that change.
- Evaluation and feedback loops belong at every rung. Without them, you cannot tell whether the system is getting better or worse.
When leaders say they want to “train AI on our organization,” they usually mean something reasonable: they want an AI system that knows their policies, products, clients, terminology and ways of working, and gives answers that reflect them. The confusion is in the word “training.” To an engineer, AI model training often means adjusting a model’s internal parameters with large amounts of data. That is one way to shape a system, and frequently not the right one.
This guide lays out the realistic options, from lightest to heaviest, and when each is worth it. The core idea is simple:
The goal is not to make a model know everything. It is to give the right system reliable access to the right organizational truth.
The ladder: from lightest to heaviest
Each rung adds capability, and also cost, complexity and maintenance. Most useful organizational AI systems combine the first three or four rungs. Few need the fifth. Almost none need the sixth.
| Approach | What it does | Best for |
|---|---|---|
| 1. Instructions and context | Tells the model who it is serving, what rules apply and what good output looks like | Tone, format, boundaries, role-specific behaviour |
| 2. Retrieval | Finds relevant passages in trusted sources and supplies them with each request | Policies, procedures, product information, anything that changes |
| 3. Structured knowledge | Represents entities and relationships explicitly | Questions that span clients, contracts, people, products and systems |
| 4. Tools, actions and agents | Lets AI query systems and take defined actions within permissions | Live data, multi-step tasks, workflow execution |
| 5. Fine-tuning | Adjusts a model’s behaviour using curated examples | Consistent format, style, classification and domain language at scale |
| 6. Pre-training | Builds a foundation model from very large datasets | Rarely appropriate for an individual organization |
1. Instructions, system prompts and context
A system prompt is a standing set of instructions given to a model before any user request: its role, the audience, the rules it must follow, the format to use, and what to do when it is unsure. Combined with contextual data about the user and task, this is the fastest way to shape behaviour. It costs little, can be changed in minutes, and should always be the first thing you get right.
2. Retrieval over trusted sources
Retrieval-augmented generation, usually shortened to RAG, means the system searches your approved documents for passages relevant to a question and gives them to the model along with the question. The model answers from those passages, ideally with citations. Because nothing is baked into the model, updating a policy document updates the answers. Retrieval is the workhorse of organizational AI, but it is only as good as the sources behind it: outdated, duplicated or contradictory documents produce outdated, inconsistent answers.
3. Structured knowledge
Many important questions are about relationships: which contracts belong to this client, who owns this process, which products are affected by this regulation. Documents alone handle these poorly. Structured knowledge represents the things your organization cares about (entities such as clients, products, locations and people) and how they connect. This can be as simple as a well-maintained schema or as rich as a knowledge graph. It gives AI a reliable map rather than a pile of pages. Our guide on better organizational information explains this layer in more depth.
4. Tools, actions and agents
Some answers live in systems, not documents: an order status, a budget balance, a ticket history. Giving AI access to tools lets it look these up, and in some cases take action, such as drafting a record or routing a request. An agent is a system that plans and carries out several such steps toward a goal. The critical design questions are about permissions: what the system can read, what it can change, and where a person must approve before anything happens.
5. Fine-tuning
Fine-tuning continues training an existing model on a curated set of examples so that its default behaviour shifts. It is justified when you need highly consistent output format or style across a large volume of work, reliable classification into your own categories, or fluency with specialized domain language that instructions alone do not achieve. It is not a good way to inject frequently changing facts. Facts learned through fine-tuning are hard to update, hard to cite and hard to audit. Retrieval handles those far better. Not every AI system needs fine-tuning, and many never will.
6. Pre-training a foundation model
Building a foundation model from scratch requires enormous datasets, specialized expertise and significant compute. For an individual organization, it is rarely the right choice. The practical question is which existing model, open or commercial, best fits the workload, and how to shape it using the rungs above.
Evaluation and feedback at every rung
Whatever combination you choose, you need a way to know whether it works. Evaluation means testing the system against a set of realistic questions or tasks with known good answers, and checking results before and after every significant change. Useful evaluations include:
- Accuracy against trusted sources, including whether citations actually support the answer.
- Behaviour on questions the system should decline or escalate.
- Consistency across phrasing, users and languages, including French where relevant.
- Checks for bias or uneven quality across groups where outputs affect people.
Feedback loops close the gap between testing and real use. When users flag a wrong answer, someone should be responsible for tracing it: was the source wrong, was retrieval poor, or was the instruction unclear? Each answer points to a different fix.
Preparing and curating training data
Whether you are building retrieval, structured knowledge or a fine-tuning dataset, the quality of your AI training data sets the ceiling. Practical preparation involves:
- Identifying authoritative sources for each topic, and retiring or flagging superseded ones.
- Removing duplicates and resolving contradictions before they reach the system.
- Adding metadata: owner, effective date, audience, sensitivity and permissions.
- For fine-tuning, assembling representative examples of good output, reviewed by people who know the work.
- Keeping a record of what data went into which system, and when.
Privacy considerations
Using personal information in retrieval indexes, evaluation sets or fine-tuning datasets raises distinct questions. Information embedded in a fine-tuned model is difficult to remove or correct, which matters when individuals have rights over their data. In Canada, PIPEDA governs private-sector handling of personal information federally, and Alberta, British Columbia and Quebec have their own private-sector laws. Purpose, consent, retention and access all need thought.
A sensible default is to minimize personal information in anything used to shape a model, prefer retrieval with permissions over baking personal data into model weights, and document your reasoning. The Office of the Privacy Commissioner of Canada publishes relevant guidance, and specific requirements should be confirmed with counsel. See our responsible AI approach for how we build this into system design.
How Trained helps
Trained’s model and knowledge training work starts with the work you want AI to do, then chooses the lightest combination of instructions, retrieval, structured knowledge, tools, customization and evaluation that does it reliably. We recommend fine-tuning when it is justified and say so when it is not. If you are trying to make AI understand your organization, tell us what you are working on.