Here is a situation that plays out in a lot of growing companies. An operations manager spends every Monday matching carrier invoices against delivery records. It takes her most of the morning. The company brings in a freelance developer to automate the job with AI, and within two weeks there is a demo: upload a PDF, the model reads it, and the numbers land in the right spreadsheet columns.
Then a carrier changes its invoice template. One invoice lists a fuel surcharge the system has never seen, so it quietly files the charge under "other." Nobody notices until the monthly close, when the totals are off. By then the manager is back to checking every invoice by hand, and now she checks the AI's work too.
The demo was never the problem. The developer built something that worked on the twenty invoices he tested. What the company needed was someone who would ask what happens on invoice number 2,000, when a field is missing, when two systems disagree, and when volume triples at quarter-end.
That gap is what this guide covers: how to scope the work, which roles to look for, how to test candidates on the things that break after launch, and what to budget for once the system is live.
Traditional automation follows fixed rules. If an invoice total is above $5,000, send it to the finance director. Tools like Zapier, Power Automate, and robotic process automation (RPA, which is software that clicks through screens the way a person would) are good at this. They break when the input stops looking the way the rules expect.
AI automation adds a step that can deal with messy input. It can read an email and work out what the customer wants, pull fields from a scanned form, or sort tickets by urgency. The AI part makes a judgment call. The rest of the workflow still needs ordinary software around it to check that judgment, route it, store it, and record what happened.
This is why hiring for it is harder than it looks. You are not only looking for someone who can connect to a language model. You need someone who can build the careful, unexciting plumbing around the model so the whole thing holds up on a bad day. Teams that build reliable AI automation pipelines tend to spend more of their time on input checks, fallbacks, and logging than on the model itself.
A quick way to tell the two approaches apart:
• Rule-based automation answers "if this happens, do that." It is predictable and cheap to run, but it cannot read a paragraph or a blurry photo.
• AI automation answers "what is this, and what should probably happen next?" It copes with variety, but it can be wrong in a confident-sounding way, so it needs checks around it.
• Most real workflows use both: the AI reads and sorts, and rules decide what may happen without sign-off.
The numbers behind the hiring rush
▪ McKinsey's State of AI survey (published November 2025, 1,993 respondents across 105 countries) found that 88% of organizations regularly use AI in at least one business function, up from 78% a year earlier. Only about one-third say they have begun scaling it across the company.
▪ The same survey found that 62% of organizations are at least experimenting with AI agents, yet in any single business function no more than 10% report scaling them.
▪ Gartner's May 2024 survey found that only 48% of AI projects make it into production, and the move from prototype to production takes about eight months on average.
▪ Gartner predicted in February 2025 that through 2026, organizations will abandon 60% of AI projects that are not supported by AI-ready data. In the related survey, 63% of organizations said they either lack the right data practices for AI or are unsure whether they have them.
▪ Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value, or weak risk controls.
▪ Stack Overflow's 2025 Developer Survey (more than 49,000 responses) found that 84% of developers use or plan to use AI tools, while 46% distrust the accuracy of AI output, up from 31% in 2024.
▪ Estimates for the MLOps market disagree. Grand View Research put it at about $2.19 billion in 2024, rising to $16.6 billion by 2030. Fortune Business Insights valued it at $2.98 billion in 2025, and IMARC Group put 2025 at $4.0 billion. Each firm defines the market differently, so read these as a direction of travel rather than an exact size.
The pattern across these sources is consistent. Getting AI to work in a demo is common. Getting it to keep working inside a real business process is where most projects stall, usually because of data, cost, and missing controls rather than the model. That should shape who you hire.
Most poor hires in this area start with a vague brief. "We want to use AI to automate operations" gives a candidate nothing to push back on. Spend a few days documenting one workflow first. You need a spreadsheet and access to the people who do the work, not technical skills.
1. Pick one workflow with real volume and real pain. Invoice matching, lead qualification, support ticket sorting, contract intake, and onboarding paperwork are common starting points.
2. Count it. How many items come in per day or per week? Peak volume matters more than the average, because that is when the system is under the most strain.
3. Collect 50 to 100 real examples, including the ugly ones. Blurry scans, half-filled forms, emails in two languages.
4. Write down the exceptions people already handle in their heads. Ask the person who does the job, "When do you stop and ask someone?" Those moments are your edge cases.
5. Decide what "good enough" means. Does the system need to be right 99% of the time, or is 90% acceptable if a person reviews everything marked as uncertain? This number shapes the design and cost.
6. Find the data owner. Someone has to approve access to the CRM, the ERP (enterprise resource planning software, the system that runs finance and inventory), or the shared inbox. If nobody can say yes, the project stalls.
Pro tip
Time the manual process for a full week before you talk to anyone. "It takes forever" is not a baseline. Something like "it takes 11 minutes per invoice, and one in seven needs a follow-up email" is. You will use that figure to judge proposals and, later, to decide whether the project paid off.
"AI automation developer" is not a title that means the same thing everywhere. Depending on the workflow, you may need one generalist or a small mix of people. The table below lays out the usual roles in simple terms.
Role
What they actually do
When you need them
AI automation engineer
Connects AI models to your tools (email, CRM, databases), writes the logic around them, and handles errors and retries
Almost always. This is usually the first hire
Machine learning engineer
Trains or adjusts models on your own data and measures how accurate they are
When ready-made models are not accurate enough on your specific documents or decisions
Data engineer
Cleans, moves, and organizes data so the AI receives consistent input
When your data sits in several systems that do not agree with each other
MLOps engineer
Sets up deployment, version tracking, monitoring, and rollback for models in production
Once more than one model is live, or when mistakes are expensive
Process analyst
Documents how the work is done today and where people make judgment calls
When the process is poorly documented or spread across several teams
QA or evaluation specialist
Builds test sets and checks whether each change makes the system better or worse
When accuracy directly affects money, compliance, or customers
For a small company automating one or two workflows, one strong automation engineer plus part-time help from a data person often covers it. Larger rollouts are where it makes sense to hire AI/ML developers with real depth in training and testing models, because general-purpose models tend to level off in accuracy on company-specific paperwork.
MLOps, short for machine learning operations, comes up in almost every conversation about this work. Think of it as the maintenance side of AI: keeping track of which model version is running, watching whether its accuracy slips over time, and being able to roll back to an earlier version when something goes wrong. Teams that skip it usually regret it when a model update quietly changes their results. If you do not have that skill in-house, MLOps services from an outside team can handle setup and monitoring while your own staff focus on the workflow.
There is no single right answer. Here is how the three common routes compare.
Factor
Freelancer
In-house hire
Development partner
How fast work starts
Usually quickest
Slowest, since recruiting and onboarding take time
Typically a few weeks after scoping
Best fit
One well-defined workflow with a clear finish line
Automation that is central to your product or operations for years
Several workflows, or work that needs several skills at once
Range of skills
One person's strengths
Grows with each hire
Engineers, data, MLOps, and QA under one contract
What happens to know-how
Leaves with the person unless documented
Stays in the company
Depends on the handover terms in the contract
Ongoing maintenance
Often not included
Built in
Can be added as a support agreement
Main risk
Single point of failure, and demos that fail on real data
High fixed cost and slow hiring
Lock-in and generic solutions if poorly scoped
A common pattern is to use AI development services from a partner to build the first version and set up monitoring, then hire one or two in-house engineers who take over once the system is stable. The part to get right is the handover. Put documentation, access to the code, and a walkthrough of the monitoring screens into the contract from the first day.
If you go with a partner, ask whether they build inside your accounts (your cloud setup, your code repositories) or theirs. Code and prompts that live only in a vendor's account are hard to take back later.
This is where most of your hiring effort should go. Nearly every candidate can show a demo. What separates them is how they handle the situations demos skip. Below are five areas to probe, each with a question you can ask and what a strong answer tends to include. Listen for specifics, and for the word "depends" followed by an actual explanation.
1. Data gaps
Ask them: "About a third of our customer records are missing a phone number or company size. What does your system do with an incomplete record?"
Real business data has holes. A blank form field, a rushed CRM entry, an invoice with no purchase order number. A weak system either crashes or, worse, lets the AI guess the missing value and carries on as if the guess were fact.
Strong answers usually include these points:
• Treating "unknown," "empty," and "zero" as three different things, because they mean different things to the steps that come after.
• Checking required fields before the AI step runs, and deciding field by field whether to skip it, look it up in another system, or send the item to a person.
• Never letting the model invent values for fields that feed money or legal decisions. If the model does fill a gap, the output is marked as a guess.
• Tracking how often gaps appear, since a sudden jump usually means a system upstream has changed.
2. Conflicting signals
Ask them: "A customer's email says they want to cancel, but our billing system shows they upgraded their plan yesterday. What should the automation do?"
Workflows pull from several sources, and those sources disagree more often than people expect. The AI reads an email as angry, while the account history shows a loyal customer who simply writes bluntly.
A good candidate will describe a clear order of trust, meaning which system is treated as correct for which piece of information. They will suggest that when sources clash on something with real consequences, the system stops and flags it instead of quietly picking one. Be wary of anyone who says the AI will "figure it out." Language models are very good at producing a confident answer from mixed evidence, and that is exactly the risk.
3. Real-time decisions
Ask them: "A customer is waiting in live chat while your system checks their order. How long can that step take, and what happens if the AI provider is slow?"
Some workflows can wait. Others cannot, like fraud checks at checkout. A call to a large language model can take several seconds, and outside AI providers sometimes slow down or return errors during busy periods.
Strong candidates talk about a time budget for each step, a timeout (a hard limit after which the system stops waiting), and a fallback such as a simpler rule, a saved answer, or a polite handoff to a person. Here is a simplified version of the logic a good engineer might sketch on a call:
result = call_model(ticket, timeout_seconds=4)
if result is None: # model too slow or unavailable
send_to("human_queue", reason="timeout")
elif result.confidence < 0.80: # model unsure
send_to("human_queue", reason="low_confidence")
elif result.category in HIGH_RISK: # refunds, cancellations, legal
send_to("human_review", reason="high_risk")
else:
apply_action(result)
log_decision(ticket.id, result, model_version="v3.2")
Nothing here is clever. Every path ends somewhere safe, and every decision is logged with the model version that made it. A careful engineer will also warn that a model's confidence scores are not always trustworthy, so they test them against your real examples before choosing the cut-off.
4. Exceptions and edge cases
Ask them: "How would you find the cases your system gets wrong before our customers find them?"
Edge cases are the unusual inputs nobody planned for: a refund request for an order that never shipped, a supplier invoice in a new currency, a contract with two signature blocks where the system expects one. Each one is rare, but across thousands of items a month, rare things happen daily.
Listen for these in the answer:
• A test set built from your real examples, including the ugly ones from your workflow map, that gets rerun every time a prompt or model changes.
• A written list of actions the system may never take on its own, such as issuing refunds above a set amount or deleting records.
• A review queue that feeds back into the test set, so every exception a person handles becomes a new test case.
• Honesty about failure. Someone who says their systems rarely make mistakes has probably never measured them.
5. Behavior under pressure and at scale
Ask them: "At month-end our volume triples in two days. What changes in your system?"
A pipeline that handles 200 items a day can behave very differently at 5,000. AI providers enforce rate limits (caps on how many requests you can send per minute), so calls start failing. Costs climb with every extra request. The human review queue fills faster than people can clear it, and a pile of items marked "waiting for review" quietly turns into delays for customers.
Good answers mention queues that hold work instead of dropping it, retries that wait a little longer after each failure instead of hammering a busy service, and alerts when the review backlog passes a set size. Some bring up sending easy items to a smaller, cheaper model and saving the larger one for hard cases. The best candidates ask back: who gets called if the system stops at 2 a.m.?
There is a slower kind of pressure too. Inputs change over time. A model that performed well at launch can slide without anyone noticing, a problem engineers call drift. Watching for it is ongoing work, and a main reason teams without a specialist pay for MLOps services after the build.
What a reliable pipeline looks like from the outside
You do not need to read code to judge whether a system is built well. When someone shows you finished work, look for these parts. Seeing all of them is a good sign the team knows how to build reliable AI automation pipelines and not just one-off scripts.
• An intake step that checks every item before the AI sees it. Is the file readable, are required fields present, is it a duplicate?
• The AI step itself, with its prompt and model version recorded so any result can be traced later.
• A rules layer after the AI that enforces business limits no matter what the model says.
• A review screen that shows a person what the AI decided and why the item was flagged.
• Logs for every decision, searchable by item, date, and model version.
• A dashboard with a handful of numbers: volume, share sent to review, error rate, processing time, and cost per item.
• A switch that turns off the AI step and falls back to the manual process within minutes.
That last item is skipped most often. Someone at your company should be able to pause the system without phoning its developer.
Quick check for any demo
Ask the person presenting, "How would I turn this off?" A calm, specific answer, such as "there's a setting, and items go back to the shared inbox," tells you more about their production experience than a polished demo ever will.
Run a paid trial instead of a whiteboard interview
Coding puzzles tell you little about this work. A short, paid trial on a slice of your real workflow tells you far more. Here is a four-week structure.
WEEK 1 Discovery and data review. The developer studies your examples, confirms system access, and writes a one-page plan covering expected edge cases.
WEEK 2 A first working version on a limited set of inputs. Logging and a review queue should be there from the start, not bolted on later.
WEEK 3 Shadow mode. The system processes real items while a person still does the actual work, and you compare their decisions side by side. It is the most honest test available, with no risk to customers.
WEEK 4 Review the results together: accuracy against your "good enough" target, how many items needed a person, where it failed, and what it would cost each month at your peak volume.
At the end, you have something far more useful than an interview score. You know how the person communicates and whether their estimates matched reality. Many providers of AI automation services now offer a pilot like this as a normal first step. A vendor who refuses any trial and wants a long commitment up front is telling you something.
Red flags worth taking seriously
These apply whether you are interviewing an individual or comparing firms that sell AI automation services.
• The proposal promises a specific accuracy figure before anyone has looked at your data.
• Nearly every answer is about which model to use, and very little is about data, testing, or failure.
• There is no plan for human review, or review is described as something you "can add later."
• They cannot explain past projects without jargon, or they turn vague when you ask what broke after launch.
• The price covers building the system but says nothing about maintenance, monitoring, or model updates.
• They want full access to your live systems in the first week without a clear reason.
• They call a chatbot or a simple rules script an "AI agent." Gartner has named this habit "agent washing" and estimated that only about 130 of the thousands of vendors claiming agentic AI offer the real thing.
The costs that show up after launch
Budgets usually focus on the build, and running costs catch people off guard. Exact amounts vary a lot by provider and volume, so ask every candidate to estimate each of these for your workflow:
• Model usage fees. Most hosted AI models charge by the amount of text processed, measured in tokens (roughly, pieces of words). Long documents cost more than short ones.
• Human review time. If one item in ten goes to a person, that is real staff time every week.
• Monitoring and logging tools, plus storage for audit records. Some teams fold this into a contract for MLOps services instead of running it themselves.
• Maintenance. Connected systems change and providers retire older model versions. Plan for ongoing engineering hours rather than a one-time fee.
• Re-testing. Every change to a model or prompt should be checked against your test set before it goes live.
• Compliance work if you handle personal or regulated data. Under the EU AI Act, the most serious violations can bring fines of up to €35 million or 7% of global annual turnover, so legal review belongs in the budget for some use cases.
Ask for a monthly running-cost estimate at peak volume, not the average. A good provider of AI development services will give you a range and explain what drives it. Teams that build reliable AI automation pipelines put maintenance into the budget from the first month.
Writing a job post that attracts the right people
Generic posts attract generic applicants. Describe the workflow, the volume, and the mess. Experienced people will recognize the problem straight away.
Sample opening for a job post
We process about 3,000 supplier invoices a month from 120 vendors, arriving as PDFs, scanned images, and emails. We want to automate data extraction and matching against purchase orders, with a human review step for uncertain cases. You will work closely with our finance team, who run the process today. In your application, tell us about a production system you built where the input data changed after launch, and what you did about it.
That final question filters out a surprising number of applicants. It is also a fair question for any firm pitching AI automation services, since the answer shows whether they have supported a system past launch.
If you hire AI/ML developers in a different time zone, agree early on overlap hours and on who responds when something breaks. A system that stalls overnight while everyone who understands it is asleep loses goodwill fast.
Key takeaways
✓ Map one workflow in detail, with real messy examples and a measured baseline, before contacting anyone.
✓ Match the role to the problem. Most teams start with an automation engineer and add data, ML, or MLOps skills as the system grows.
✓ Interview on data gaps, conflicting signals, time limits, edge cases, and peak load, since demos rarely show any of them.
✓ Use a paid four-week trial with a shadow-mode phase to test both the system and the working relationship.
✓ Budget for running costs at peak volume, and make sure someone in your company can switch the system off.
Where to start this week
McKinsey's figures show that most organizations already use AI somewhere, while only a small share have scaled it. The companies getting lasting results did the unexciting work: clean inputs, clear limits on what the system may do alone, review queues, logs, and someone watching the numbers every week.
So start small and specific. Pick the workflow, count it, gather the ugly examples, and decide what "good enough" means. Then look for people, in-house or outside, who get more interested when you show them the messy cases. Whether you hire AI/ML developers directly or work with a partner, the questions in this guide will help you tell apart someone who can build a demo from someone who can build a system your team still trusts a year from now.
Back to Blog.png)
