The pattern is now recognizable. A mid-market company appoints a Chief AI Officer or a VP of Digital Innovation, assembles a small team, and announces the creation of an AI Centre of Excellence. Twelve to eighteen months later, the CoE has produced several pilots, a vendor shortlist, and a slide deck on responsible AI principles — but line-of-business adoption is flat, the finance team is running its own ChatGPT Enterprise instance, and the operations director has signed a contract with an automation vendor without involving IT or legal. The CoE has become a bottleneck for the people who engage with it and irrelevant to the people who do not. This is not a technology failure. It is an operating model failure, and it is entirely preventable.

Why most AI CoEs are structured incorrectly from the start

The most common mistake organizations make when establishing an AI CoE is modelling it on an IT centre of excellence or a project management office. Both of those structures exist to enforce standards on work that is already well understood. AI adoption in a mid-market context involves work that is not well understood — the business problems are ambiguous, the technology options change quarterly, and the organizational capability to absorb automation is uneven across functions. A governance-first, project-office structure imposes process overhead before value has been demonstrated, which drives business units to find workarounds.

The second common mistake is treating the CoE as a delivery team. When a CoE owns and builds AI solutions directly, it becomes a bottleneck almost immediately. A team of six or eight people cannot serve the AI needs of a 500-person organization if they are responsible for scoping, building, deploying, and maintaining every solution. They also develop an incentive to own projects rather than enable others, which is the opposite of what a CoE should do at scale.

The governance model: what the CoE actually owns

A well-structured AI CoE owns four things: standards, intake, evaluation, and enablement. It does not own delivery. That distinction is foundational.

• Standards: The CoE defines which AI platforms and vendors are approved for enterprise use, what data governance requirements apply to AI systems (particularly those handling personal information subject to Canadian privacy law, including PIPEDA and provincial equivalents), what security review is required before deployment, and what model documentation and audit trail standards apply to systems that inform consequential decisions.
• Intake: Every AI initiative — whether it originates in IT, finance, operations, or HR — enters a single intake queue. The intake process is the primary mechanism for preventing shadow AI. It does not need to be slow; a well-designed intake process can return an initial assessment within five business days.
• Evaluation: The CoE evaluates vendor proposals, open-source tooling, and internally proposed use cases against a consistent framework that considers technical feasibility, data readiness, regulatory risk, and estimated business value. This prevents individual business units from making isolated vendor decisions based on sales presentations.
• Enablement: The CoE builds internal capability — training programs, prompt engineering guides, reusable components, and embedded support for business units executing approved AI projects. This is how the CoE multiplies its impact without becoming a delivery bottleneck.

What the CoE does not own: the P&L for AI initiatives, the implementation timeline, or the technology infrastructure. Those accountabilities stay with the business unit and with IT respectively. The CoE provides the governance layer and the expertise; the business unit provides the use case, the subject-matter knowledge, and the accountability for outcomes.

The three roles that determine whether the CoE works

In our experience working with mid-market organizations building out AI governance functions, the staffing model is where most CoEs make their most consequential early decisions — and most make at least one significant error. The three roles that matter most are not the ones that appear most prominently in job postings.

The AI Product Owner is responsible for the intake process, the use case pipeline, and the relationship between the CoE and the business units it serves. This person needs strong business acumen, the ability to assess feasibility without building anything, and the credibility to push back on business units when a proposed use case is not ready or not appropriate. A common mistake is hiring a technical person into this role. The AI Product Owner does not write code. They write business cases, facilitate prioritization decisions, and manage stakeholder expectations across multiple concurrent initiatives. In a 200–500 employee organization, this is typically a single senior individual. In a 1,000-plus employee organization, there may be two or three, each aligned to a cluster of business functions.

The AI Engineer provides the technical depth the CoE needs to evaluate vendor claims, assess integration complexity, define data requirements, and build reusable components. This is not a data scientist role and it is not a software developer role in the traditional sense — it sits at the intersection of both. The AI Engineer needs to understand LLM orchestration, API integration patterns, prompt design, retrieval-augmented generation, and enough data engineering to assess whether a proposed use case has the data foundation it requires. In mid-market deployments, organizations typically need one AI Engineer for every three to four concurrent active initiatives.

The Business Translator is the role most organizations do not hire for explicitly, which is why they struggle to move from pilot to production. The Business Translator is embedded in a business unit — they are not technically an AI engineer, but they understand enough about how AI systems work to bridge the gap between what the business unit needs and what the CoE and IT can deliver. They write requirements that an AI Engineer can actually implement. They train their colleagues on new tools. They surface friction points during rollout. In some organizations this role is filled by a particularly capable operations analyst or a business systems specialist who has developed AI literacy. In others, it is a deliberate hire. Either way, the CoE needs to invest in developing this capability across the organization through a formal enablement program.

The org chart: how the CoE fits into the enterprise structure

The reporting structure of the CoE signals its mandate and its authority. CoEs that report into IT are perceived as a technology function and struggle to engage business units on strategic questions. CoEs that report into a Chief Digital Officer or Chief Innovation Officer often lack the organizational authority to enforce standards. In our experience, the reporting structure that works best for mid-market organizations places the CoE under the Chief Operating Officer or, where a CAIO role exists, under that executive — with a formal dotted-line relationship to the CIO for technology standards and to the CFO for investment governance.

Regardless of reporting line, the CoE needs a formal steering committee with representation from Finance, Legal, IT, and at least two operating business units. The steering committee meets quarterly to review the use case pipeline, approve changes to the approved vendor list, and review any material risk findings from deployed AI systems. This is not optional governance overhead — it is the mechanism that gives the CoE legitimate cross-functional authority without requiring it to be housed in every function simultaneously.

The funding model: why centralized and federated both fail alone

Centralized funding — where the CoE controls all AI investment — creates a single point of rationing. Business units that cannot get CoE resources either wait or go around the process. Federated funding — where each business unit funds its own AI initiatives independently — produces fragmented tooling, duplicated vendor contracts, inconsistent data practices, and eventually a portfolio of ungoverned AI systems that the CoE discovers after the fact. Neither model works on its own at mid-market scale.

The hybrid model that works allocates funding in two pools. The first pool is a central CoE operating budget that covers the CoE team, the approved tooling stack, shared infrastructure (such as a centrally managed LLM API gateway), and the enablement program. This is typically in the range of 0.3 to 0.6 percent of annual revenue for a mid-market organization, though the right figure depends heavily on the breadth of the AI mandate. The second pool is a business-unit AI budget that each function controls, subject to the CoE’s intake and approval process. Business units fund their own initiatives and own the business case; the CoE provides the standards, the evaluation, and where necessary the engineering support.

The key design element in the hybrid model is the chargebacks and shared services structure. If the CoE provides AI Engineering support to a business unit initiative, that support is either absorbed into the central CoE budget (for high-priority or cross-cutting initiatives) or charged back to the business unit at a transparent internal rate (for discretionary or business-unit-specific work). This creates a rational incentive structure: the CoE has an incentive to be useful and efficient; the business unit has an incentive to engage the process rather than circumvent it, because the CoE’s engineering capacity is genuinely faster and cheaper than procuring equivalent capability externally.

The intake process: preventing shadow AI without creating a bureaucracy

A well-designed intake process has five stages and should complete the first three within two weeks of submission for a standard use case. The five stages are: submission, initial triage, feasibility assessment, prioritization, and initiation.

1. Submission: The business unit completes a structured intake form covering the business problem, the proposed solution concept (if any), the data involved, the expected business value, and the urgency. The form should take no more than 45 minutes to complete for a well-defined use case.
2. Initial triage (2–3 business days): The AI Product Owner reviews the submission, categorizes it by complexity and risk, and either routes it to full feasibility assessment or returns it with a recommendation to pursue a non-AI solution. Not every business problem requires AI, and saying so early is a credibility-building function, not a failure.
3. Feasibility assessment (5–7 business days for standard use cases): The AI Engineer assesses data readiness, integration requirements, and technical complexity. Legal and privacy are consulted on any use case involving personal data or consequential automated decisions. The output is a one-page feasibility brief with a recommended path forward, estimated resource requirements, and a risk rating.
4. Prioritization (steering committee or delegated authority): Approved use cases enter the priority queue. The steering committee reviews the queue quarterly; the AI Product Owner has delegated authority to approve low-complexity, low-risk use cases between steering committee meetings.
5. Initiation: An approved use case is assigned resources, a Business Translator is identified or developed within the business unit, and the initiative begins with a defined scope and success criteria.

The intake process must be visible. A shared dashboard showing all active, queued, and completed AI initiatives — accessible to anyone in the organization — is one of the most effective tools for reducing shadow AI. When business units can see that their peers are engaged with the CoE and making progress, the incentive to bypass the process diminishes.

Measuring CoE effectiveness: the metrics that matter

A CoE should not measure itself primarily on the number of AI projects completed. Projects are a lagging indicator. The leading indicators of a healthy CoE are: intake volume and growth rate (is the organization engaging with the process?); time-from-submission-to-decision (is the process fast enough to be useful?); business unit AI literacy scores (is the enablement program working?); ratio of CoE-supported to independently executed AI initiatives (is AI capability distributing across the organization?); and the number of approved vendor relationships versus total vendor relationships in use (is the approved vendor list comprehensive enough to be practical?).

Organizations we work with that have effective AI CoEs typically see a meaningful reduction in unsanctioned AI tool usage within the first year — not because the CoE is policing behaviour, but because the official process has become genuinely faster and more useful than finding a vendor independently.

Frequently asked questions

How large does an organization need to be before a formal AI CoE is justified?

In our experience, a formal CoE structure with dedicated staffing is most appropriate for organizations with more than 300 employees and more than five active or planned AI initiatives. Below that threshold, a lighter governance model — a cross-functional AI steering group with a designated AI Product Owner operating part-time — typically provides sufficient coordination without the overhead of a dedicated function. The critical threshold is not headcount; it is the point at which uncoordinated AI activity creates material risk or duplication.

What is the single most common reason AI CoEs fail?

The most common failure mode is a mandate that is too narrow and too project-focused. CoEs chartered to “deliver ten AI use cases in the first year” succeed at that objective and then have no reason to exist. CoEs chartered to “build the organization’s capacity to adopt AI responsibly and at scale” have a mandate that compounds over time. The charter is a governance document and it should be written accordingly, with clear success criteria that extend three to five years, not twelve months.

How do we handle AI tools that employees are already using without approval?

Start with an audit, not an enforcement action. Most unsanctioned AI usage in mid-market organizations is not malicious — it reflects employees trying to do their jobs more effectively in the absence of an official path. The audit will surface which tools are in use, what data is being shared with them, and which use cases are most common. Use that information to accelerate the approval of tools that meet your standards and to build a practical path for regularizing the most common use cases. Blanket prohibitions without alternatives drive usage underground; they do not eliminate it.

Should the CoE own the relationship with AI vendors?

The CoE should own the evaluation and approval process for vendors, and it should be the primary commercial relationship manager for enterprise AI platform contracts — tools like Azure OpenAI Service, Google Vertex AI, or similar foundational platforms. For business-unit-specific applications built on approved platforms, the business unit can own the vendor relationship, subject to CoE standards. The distinction matters because it prevents the CoE from becoming a procurement bottleneck while ensuring that foundational platform decisions remain coordinated.

How do we prevent the CoE from becoming a bottleneck as AI adoption scales?

The bottleneck risk is real and it typically emerges in the second year, when intake volume grows faster than CoE capacity. The mitigation is the enablement model: as the Business Translator network develops across the organization and as AI literacy improves, an increasing proportion of lower-complexity use cases should be executable by business units without direct CoE engineering support. The CoE shifts its focus toward higher-complexity, cross-functional, or higher-risk initiatives. This requires intentional design from the beginning — the CoE should be measuring and reporting on capability distribution from the first quarter of operation, not waiting until the bottleneck is already a problem.

Building an AI Centre of Excellence: The Operating Model That Works at Scale

Most senior operations directors and VPs of IT at mid-market organizations face the same underlying challenge: AI adoption is happening faster than governance, and the gap between what the business is doing and what the organization has sanctioned is widening. This post provides a concrete operating model — governance structure, team roles, funding design, and intake process — for organizations that are ready to build an AI Centre of Excellence that actually scales.