How to choose an AI agent development company

What to ask for in the first call when you choose an AI agent development company

To choose an AI agent development company, ask every shortlisted vendor for the same six things before you compare prices. Each is a written artefact or a test you can run yourself, so the answers can be checked rather than believed:

1. A one-paragraph definition of the task, with an out-of-scope list.
2. An evaluation plan: a test set built from your real cases and a pass mark agreed before building.
3. The permissions the agent needs, and which actions wait for a human approval.
4. A data-flow diagram that names the model providers and any other processors.
5. The contract terms for who owns the code, prompts, tool definitions and test set.
6. A pilot proposal on one measurable task, with a stop or continue rule.

A vendor that can produce these is doing the work you are paying for. A vendor that answers with a deck of logos is selling you a feeling. The rest of this guide explains what good answers look like, with a score sheet, questions to ask and a way to shape the pilot.

Do you need an agent at all?

“AI agent” covers very different products. Anthropic’s engineering guidance separates two kinds of system. In a workflow, language models and tools are orchestrated through predefined code paths. In an agent, the model dynamically directs its own process and tool use. The same guidance recommends finding the simplest solution possible and adding complexity only when needed, and notes that agentic systems often trade latency and cost for better task performance, with a risk of compounding errors.

That should shape your first conversation:

• Known steps (pull these fields, check these rules, draft this message): a workflow is likely more predictable and easier to test.
• Open-ended task with an unpredictable number of steps: an agent may be justified.
• One model call with retrieval and examples would do: you may not need a custom build.

A vendor that recommends the most autonomous design whatever your task is not weighing your interests. One that can tell you when not to build an agent is worth listening to. The cost factors article explains why more autonomy raises both build and run effort.

Which criteria should you score?

Score each shortlisted company against the same list.

Logos and case studies are cheap to produce and hard to verify. Weight what you can test yourself over what you are shown.

What should you ask every vendor?

Put the same questions to each company and compare the answers in writing.

Scope and design

• What will the agent do, in one paragraph, and what is out of scope?
• Would a workflow solve this? What would make you advise against an agent?
• Which of our systems does it read from, which does it write to, and which actions are irreversible?
• When it is unsure, does it stop, ask a person or guess?

Evaluation

• Who builds the test set, and what result counts as a pass?
• Will you hold back part of the set while tuning? Anthropic’s guidance on writing tools for agents describes using held-out test sets to avoid overfitting to the tuning examples.
• What do you measure besides accuracy? The same guidance lists runtime, total tool calls, token consumption and tool errors.
• Can we read full transcripts of failed runs, including tool inputs and outputs?

Guardrails and security

• How is prompt injection handled when the agent reads an email, web page or document?
• What identity does the agent use in our systems, and what is the narrowest permission set that works?
• Which actions need human approval, and what caps exist on steps, retries and spend per task?
• How is sensitive data treated in prompts, logs and traces?

Data and IP

• Where is our data processed, and which model providers and subprocessors are involved?
• Is our data used to train anything?
• Who owns the code, prompts, tool definitions, test set and any fine-tuned artefacts, and is that in the contract?
• Can we run the system without you, in our own cloud and model-provider accounts?

Delivery and operations

• Are the people we meet in sales the people who will build it?
• When the underlying model changes, what re-testing happens?
• What drives the monthly run cost?
• If we end the engagement, what do we receive and in what form?

Using a public checklist for security answers

The OWASP GenAI Security Project publishes the OWASP Top 10 for LLM Applications. On its page, checked on 2026-10-14, the 2025 edition lists: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. Hand the list to each vendor and ask them to explain, for your design, how the relevant entries are addressed. Excessive agency and unbounded consumption bear most directly on agents that act and loop.

Which red flags should slow you down?

None of these is an automatic disqualifier. Each is a reason to ask a follow-up.

• No evaluation plan. If success is “it looks good in the demo”, failures will surface in production.
• Promises of full autonomy. Unattended operation across sensitive systems with no approval steps ignores how errors compound.
• Guaranteed accuracy. Language models are probabilistic. A fixed accuracy figure without a test set and a definition is not a commitment.
• Vague security answers. If you ask about prompt injection or retention and get a brochure, move on.
• Lock-in by design. A runtime you cannot host, or an IP clause that gives you only a licence to use what you paid for.
• Framework worship. Anthropic’s guidance warns that frameworks can add layers of abstraction that obscure the underlying prompts and responses and make debugging harder. Ask what the framework is doing for you.
• A demo you cannot reproduce. Ask to run your own examples through it.
• No view of run cost. Anthropic’s pricing documentation counts tool definitions, tool calls and tool results as tokens. A vendor who cannot discuss per-run cost has not measured it.

How do you scope a pilot that produces evidence?

A pilot is the cheapest way to buy evidence. Keep it small and measurable.

1. Pick one task that is frequent enough to matter and bounded enough to test, with a person who does it today and can judge the output.
2. Write the pass mark first, plus the test set and what counts as a failure, before anyone builds.
3. Build the test set from real cases, including awkward ones. Anthropic’s guidance notes that strong test tasks may need many tool calls, not just simple lookups.
4. Limit permissions. Start read-only where possible and put approval in front of any write action.
5. Run in shadow mode. Let the agent propose and compare its proposals with what people did, before it acts.
6. Read the transcripts, including what the agent did not do.
7. Agree the next step in advance: criteria for extending, changing or stopping.
8. Keep it portable. Deliverables include the code, prompts, test set and run instructions.

For governance, NIST’s AI Risk Management Framework is a voluntary framework. NIST’s page, checked on 2026-10-14, says version 1.0 was released on 26 January 2023, that a Generative AI Profile (NIST-AI-600-1) followed on 26 July 2024, and that the framework is being revised. The page names Govern, Map, Measure and Manage as its elements. You do not have to adopt it formally to borrow the habit of writing down what could go wrong and who is responsible.

What should a proposal contain?

Before comparing prices, check that each proposal includes:

• a written task definition and out-of-scope list;
• the evaluation plan and pass criteria;
• an integration list with the permissions each needs;
• guardrails, including approvals and limits;
• data flow and subprocessors;
• IP and handover terms;
• run-cost drivers and how they will be monitored;
• what support continues after launch.

Proposals missing several of these describe different projects, so their prices cannot be compared. Agents that expose tools through the Model Context Protocol also deserve a question on how that layer is secured; what an MCP server is and MCP versus REST APIs explain the pattern. If the agent’s data comes from a warehouse, the data engineering buyer’s guide covers the scoping questions for that side.

Where Vexio fits

Vexio builds custom AI agents as engagements, and the checklist above is a fair test to apply to it too: a defined task, a test set agreed up front, narrow permissions, approval before risky actions, and a clear handover. The custom AI agent development page outlines the work, and you can request a demo or a call to run your shortlist questions past Vexio.

Sources

Checked on 2026-10-14.

• Anthropic Engineering, “Building effective agents” (published 2024-12-19), for workflows versus agents, simplicity, cost and error trade-offs, frameworks: https://www.anthropic.com/engineering/building-effective-agents
• Anthropic Engineering, “Writing effective tools for agents” (published 2025-09-11), for held-out test sets, metrics and realistic tasks: https://www.anthropic.com/engineering/writing-tools-for-agents
• OWASP GenAI Security Project, “OWASP Top 10 for LLM Applications 2025”: https://genai.owasp.org/llm-top-10/
• NIST, “AI Risk Management Framework”, for voluntary use, release dates and revision note: https://www.nist.gov/itl/ai-risk-management-framework
• Anthropic, Claude API pricing, for tool-use token accounting: https://platform.claude.com/docs/en/about-claude/pricing