Choose an AI developer by what they can ship and keep running, not by what they can demo. Write down the job and the risks first. Then check for production skills: evaluations, guardrails, cost control and security. Then ask them

to show how they review AI-generated code, and pick the hiring model that fits your budget and your need for support after launch.

This guide walks through that process in the order we would do it. It is written for founders, CTOs and business owners who need working software, not a research project.

TL;DR

  • Define the problem, the data involved and the cost of a wrong answer before you talk to anyone.
  • Look for production skills: evaluations, guardrails, cost and latency tuning, and clear documentation of what the AI can and cannot do.
  • Test the process, not just the portfolio. Ask how they scope prompts, test and change AI-generated code.
  • Match the hiring model (employee, contractor, agency, review-then-build) to how long you need help and who will own the code.
  • Plan for support after launch. Most failures happen in the last mile, not the first demo.

Table of Contents

Start with the job, not the résumé

Most bad hires start with a vague brief like “we need AI.” A developer cannot be judged against a brief that vague, and neither can you.

Before you interview anyone, answer these questions in a page or less:

  • What decision or task is the software handling? Drafting emails is not the same as approving refunds.
  • What data does it touch? Customer records, payment data and health information carry different rules than public product descriptions.
  • What does a wrong answer cost? A typo in a draft is cheap. A wrong price quoted to a customer is not.
  • Who is affected? Staff, customers, partners, regulators.
  • What does “working” mean? A number you can measure, such as a share of tickets resolved without a human or a drop in processing time.

This is not busywork. A submission to NIST’s AI risk management process from MIT Lincoln Laboratory recommends settling privacy and data-access rules, possible bias, liability for consequential mistakes, security consequences and acceptable error thresholds before design begins. NIST’s own work on evaluation says organizations should build test and validation assessments tied to their objectives, so they can tell whether an AI system meets its goals while limiting negative effects.

If you want a worked example of turning that thinking into a plan, our AI workflow integration guide for decision-makers covers scoping a workflow before you hire for it. If acceptable error rates and data handling are new territory, a look at security and data handling practices for a customer-facing tool helps you frame the questions.

Write the acceptance test first

Give every candidate the same one-paragraph acceptance test: “We will consider this done when X percent of Y cases pass Z check, with no exposure of W data.” Developers who push back with sharper questions are usually the ones you want. Developers who agree to everything are telling you they have not thought about it.

What an AI developer actually needs to know

“AI developer” covers a wide range: someone who wires a chatbot into your site, someone who builds an agent that touches your billing system, someone who deploys a model on your own servers. The titles overlap, so judge skills against the job you wrote above.

For a production business application, here is a reasonable baseline:

  • Core languages. OpenAI’s guide to taking AI apps from concept to production says developers should be comfortable with Python or JavaScript and know prompt engineering, retrieval-augmented generation (RAG), fine-tuning, evaluations, guardrails, and cost and latency optimization.
  • Evaluations and guardrails. The same guide notes that production AI applications rely on evaluations and guardrails for quality, safety and reliability. If a candidate cannot explain how they would measure whether the system got worse after a change, they are building a demo.
  • Trade-off judgment. A competency map from the Business-Higher Education Forum describes AI developers as people who understand trade-offs among model capability, latency, cost and reliability and can document AI behavior, assumptions and limitations.
  • Ordinary engineering. Version control, tests, deployment pipelines, logging, monitoring. AI does not replace any of it. It makes each of them more important.

Notice what is not on the list: a particular model brand or a trendy framework. Models change every few months. Fundamentals do not.

Match depth to the project

Not every project needs a research-level specialist. The BHEF map itself specifies basic-to-medium proficiency for several competencies in AI design and integration and in AI operations. A strong software engineer with solid AI integration habits is often the right hire for a business application. Save the deep specialists for custom model work.

Your project What to prioritize
Add AI features to an existing app Strong software engineer, API integration experience, testing habits
Automate an internal workflow with agents Agent design, error handling, permissions, monitoring
Run a model on your own hardware for privacy Infrastructure, deployment, model serving, security
Rescue an AI-generated or vibe-coded app Code review, refactoring, security, CI/CD

How to test ability, past work and review habits

A portfolio shows what someone has built. It does not show how they work with AI tools, and that is now the bigger question. Anyone can generate a lot of code quickly. The skill is knowing what is wrong with it.

Ask for work that is live

Request links or walkthroughs of systems running in production. Ask what broke in the first month, how they found out, and what they changed. People who have run real systems answer with specifics: a rate limit they missed, a prompt that drifted, a bill that tripled. People who have only built demos answer in generalities.

Three production failure examples and responses

Run a short practical exercise

Give a small, paid task and watch the process. A hiring-assessment rubric from Karat suggests judging task completion and process quality, including how a candidate scopes AI prompts, evaluates and tests output, modifies AI-generated code and navigates the codebase. Apply that to a vendor too:

  1. Hand over a small piece of your codebase with a known flaw.
  2. Let them use any AI tools they like.
  3. Ask them to explain what the tool got wrong and how they confirmed the fix.

If the answer is “it looked right,” keep looking.

Questions worth asking

  • How do you test code that an AI assistant wrote?
  • What do you do when the model gives a different answer to the same input tomorrow?
  • How do you keep token costs from growing as usage grows?
  • What would you refuse to automate in our process, and why?
  • Who owns the code, prompts and evaluation sets at the end?

The last two matter more than they look. A developer who tells you what not to automate is protecting you. A contract that is silent on ownership is a problem you meet later, at a bad time.

Security, code quality and maintainability checks

This is where choices get expensive. A feature that works in a demo can still leak data, break on the second deployment, or be impossible for the next developer to read.

NIST’s Secure Software Development Framework (SSDF) tells organizations to integrate security practices throughout the software development life cycle and presents the framework as a common language for procurement and risk management. You can borrow that directly. Put these questions in your vendor evaluation:

  • Data handling. Where does your data go? Is it sent to a third-party model provider, stored, or used for training? Can a local or private deployment meet your compliance needs?
  • Secrets and access. How are API keys stored? What permissions does an AI agent get, and can it be limited to the minimum?
  • Dependencies. Can they produce a list of the components in the application? Our guide to an SBOM for applications explains what to include.
  • Testing and gates. Do tests run automatically before anything ships? See our notes on QA for AI applications and code quality metrics and CI gates.
  • Readability. Could a new engineer understand the repo in a week? Ask to see a sample README and a recent pull request.
  • Monitoring. How will you know when the AI starts giving worse answers or the bill spikes?

If you are a nontechnical founder who already has an app built with AI tools, you do not need to hire a builder first. You need an independent review. A review tells you whether to keep, repair or rebuild, which is a different and cheaper decision than starting over.

Hiring models compared

There is no single right answer. The right model depends on how long you need help and who will maintain the result.

Model Best for Watch out for
Full-time employee Ongoing product work, core to your business Long search, high fixed cost, one person’s skill set
Freelance or contractor Defined, short projects Handover, availability, knowledge walking out the door
Agency or consultancy Projects needing several skills, speed, ongoing support Needing clear ownership, scope and documentation terms
Team augmentation Adding senior capacity to your own team Needs your own management time
Review first, then build Existing code, vibe-coded apps, unclear scope Only useful if the review is independent and written down

If you are weighing outsourcing in general, our guide on how to outsource a software project successfully covers contracts and handover. For a longer view of staffing, see AI developer staffing at 12 months.

What it costs in the United States

Be skeptical of any blog that gives you one “average AI developer rate.” Pay varies with seniority, location, the model (salary versus contract versus project) and the work itself.

A reliable anchor comes from the government. The Bureau of Labor Statistics reports a median annual wage of $135,980 for software developers in May 2025. That is general software work, not AI specialization, so treat it as a floor for conversation. An employee also costs more than salary once you add benefits, equipment, recruiting time and management.

Contractors charge more per hour because they carry their own overhead and have no guaranteed work. Project pricing flips the question: you pay for an outcome, so scope and acceptance tests matter even more.

Here are our own published prices, so you can see what scoped work looks like. All are one-off unless noted, as listed on our pricing page and vibe code review page:

Service Price
ShipDoctor scan $59
Vibe Check $449
Senior Developer Review from $449
Infrastructure Assessment $2,750
Automated Monitoring / Logging / Alerts $2,750
Dockerizing Your Application $3,800
Continuous Integration (CI) Assessment $4,500
AI Assisted Engineering from $9,950
Ship Shape Service from $1,850 per month
Crew Service from $2,950 per month

Other work, such as AI agent creation, local model development and team augmentation, has no published price because it depends on scope. We scope those after a conversation.

The pattern to notice: small, fixed-price reviews let you find out whether a developer is a good fit before you commit tens of thousands of dollars.

What it costs in the United States

Telling whether a developer fits your budget and needs

A good fit shows up in how they handle the first two weeks, not in the sales call. Look for these signals:

  • They narrow the scope. Good developers offer a smaller first step with a clear result.
  • They give ranges with assumptions. “This depends on X; here is how we would find out” beats a flat number with no reasoning.
  • They talk about running costs. Token spend, hosting and monitoring are part of the budget, not a surprise.
  • They put things in writing. Scope, acceptance tests, ownership of code and prompts, and what happens after launch.
  • They plan for the last mile. Launch is the start of maintenance. Ask who fixes the bug at 9 p.m. next quarter.

Red flags are just as consistent: promises of full automation with no human review, refusal to show live work, no questions about your data, and pricing that never mentions testing.

Our perspective

We see the same pattern again and again. A founder or a team gets a working prototype fast, sometimes in a weekend with an AI coding tool, and then hits what we call the prototype trap. The demo works. Production does not. Logins fail under load, secrets sit in the repo, nobody can explain why the bill doubled, and there are no tests to say whether the next change breaks anything.

That is why we pair AI tooling with experienced human engineers. AI writes code quickly. Engineers decide whether the code is clean, secure and maintainable enough to run a business on. We think that combination is the right way to choose and use an AI developer: speed from the tools, judgment from the people.

Our advice is to start small. A review of what you already have, whether that is a vibe-coded app or a half-finished build, tells you what is solid and what needs repair. From there, you can decide between fixing it, extending it or rebuilding with a clear plan. We also keep working with clients after launch, because the projects that last are the ones that get monitored, patched and tuned.

We also care about cost and privacy. Careful design cuts token spend, and local AI models can keep sensitive data inside your own environment when compliance requires it. Pick a developer who talks about those trade-offs before you sign, not after.

Chad

Where to start if you already have code

If an AI tool built part of your product, you have a different starting point from a company hiring from scratch. The cheapest first step is an independent look at what exists.

We built our review services for that situation. The Vibe Check is a $449 written review of AI-generated or vibe-coded apps, and the ShipDoctor scan at $59 is a lighter first pass. If you want ongoing help with shipping and maintenance, our monthly Ship Shape and Crew services start at $1,850 and $2,950 per month. For a full picture of what we build and fix, from custom applications to agents and local models, see our services overview, or start with the vibe code review page.

Bowtie

If you want to talk through a project first, contact us through bowtie.co and tell us what you are trying to ship. For CTOs who want production-ready AI from a consultant, our AI software consultant page explains how we work.

A simple scorecard for your shortlist

Score each candidate from 1 to 5 on these lines, then compare totals and, more importantly, the weakest line:

  1. Scope clarity. Did they restate your problem and acceptance test accurately?
  2. Production evidence. Can they show live systems and what went wrong in them?
  3. Review process. Do they test and challenge AI-generated code?
  4. Security and data. Do they ask where data goes and how access is limited?
  5. Cost control. Did they discuss token, hosting and monitoring costs?
  6. Ownership and handover. Are code, prompts and documentation yours?
  7. Support after launch. Is there a named way to get fixes and updates?

A candidate with a 2 on security or ownership should not win on enthusiasm alone. Those are the lines that cause real damage later.

For the build itself, our piece on how an AI app can survive the six seams of deployment shows what a production plan should cover, and our continuous integration assessment is one way to check whether your delivery pipeline is ready.

FAQ

Should I hire an AI developer or an AI agency?

Hire an employee when AI work is core to your product and will keep going for years. Use an agency or consultancy when you need several skills, a faster start or support after launch. Many teams start with an agency review, then decide what to bring in-house.

How much does it cost to hire an AI developer in the US?

It depends on the model. The Bureau of Labor Statistics puts the median software developer wage at $135,980 a year as of May 2025, and that excludes benefits and AI specialization. Contract and project pricing varies with scope, so ask for a written range and its assumptions.

What should I ask an AI developer before hiring?

Ask how they test AI-generated code, what live systems they have shipped, how they control token costs, where your data goes, and who owns the code and prompts. Then give them a small paid task and watch how they work.

Can I fix an app my AI tool built, or do I need to start over?

Often you can fix it. An independent code review sorts the app into what is solid, what needs repair and what should be rebuilt. That review costs far less than a rebuild, and it gives you facts to decide with.

Do I need a local AI model?

Only if your data, compliance rules or costs call for it. Local models keep data in your own environment, which can help with privacy requirements. A good developer will ask about your data before recommending one.

Sources