Agentic coding: writing code got cheap, verifying it did not

The tools and their three modes

In the IDE, the agent edits inside your editor while you watch. In the terminal, a CLI agent reads the repository, runs commands and tests, and edits files. In the cloud, you assign a task and get a pull request back.

Two open formats limit lock-in: AGENTS.md, a Markdown file of instructions for agents launched in August 2025 and adopted by more than 60,000 open-source projects by December, and MCP for tools and data (see MCP explained).

Benchmarks: read the version and the harness

SWE-bench Verified, the human-validated subset OpenAI published in August 2024, was the number every coding launch quoted. In February 2026 OpenAI stopped reporting it: the state of the art had moved only from 74.9% to 80.9% in six months, an audit of 138 hard problems found flawed tests in 59.4% of them, and GPT-5.2 solved 31 tasks judged almost impossible, a sign of contamination. It is absent from the Claude Opus 5.5 and GPT-6 Astra launch tables.

Its successors are harder and change faster. SWE-bench Pro (Scale AI, September 2025) has 1,865 long-horizon tasks from 41 repositories, but its public board still lacks the mid-2026 models. Terminal-Bench went from 1.0 to 4.0 in 15 months; on the official 4.0 board, GPT-6 Astra in Codex leads at 58.2%, ahead of Claude Fable 5.1 in Claude Code at 57.9%, while the 66.4% Anthropic reports for Opus 5.5 (see our analysis) is not yet listed. In OpenAI's DeepSWE v1.1 table, five models from three vendors sit within 7 points.

So name the model and the harness, the software that runs the agent; compare only within one version; and test on your code. Anthropic recommends starting agent evaluations with 20 to 50 tasks drawn from real failures: for coding, closed issues from your own repositories. Method in our LLM evals guide.

What the productivity studies found

The most rigorous study is still METR's randomized trial of July 2025: 16 experienced open-source developers, 246 real issues of about two hours each, in mature repositories of over a million lines. With AI allowed, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, tasks took 19% longer. The developers had expected a 24% speedup and still believed afterwards they had been 20% faster.

The follow-up, run from August 2025 and published in February 2026, estimated time savings of 18% for returning developers and 4% for new recruits, but both confidence intervals include zero. METR calls the data unreliable: 30% to 50% of participants withheld tasks they did not want to do without AI, so the true gain is probably larger.

Surveys show the same gap. In DORA 2025 (nearly 5,000 respondents), 90% use AI at work and over 80% report higher productivity, yet 30% have little or no trust in AI-generated code, and adoption goes with higher throughput but lower delivery stability. In Stack Overflow's 2025 survey, only 14.1% used AI agents daily.

Security: AI code fails in familiar ways

In 2025 Veracode tested more than 100 models on Java, Python, C# and JavaScript tasks: 45% of samples introduced an OWASP Top 10 vulnerability, from 72% in Java to 38% in Python. Functional quality improved; security did not, whatever the model size.

• Dependencies. A model can suggest a package that does not exist, and an attacker can register that name first. Pin versions, install through an internal registry or allowlist, and make each new dependency a human decision (see AI supply-chain security).
• Secrets. An agent sees what its shell sees. Give it short-lived, narrowly scoped credentials, never production ones, and scan every pull request for secrets.
• Prompt injection. An issue, a web page or a README can carry instructions the agent follows, and OpenAI considers the problem unlikely ever to be fully solved (see prompt injection and agents).
• Sandboxing. Filesystem and network isolation cut permission prompts by 84% at Anthropic, which curbs rubber-stamping. Anthropic also saw about 0.8% of agent actions that appeared irreversible: rare, not zero.

Our recommendation: the same security gates for every pull request, human or agent, and an external web and API pentest before exposing a critical application.

What changes inside the team

Specs before prompts. Agents work best from a short spec: acceptance criteria, files in scope, what must not change. AWS built Kiro around this idea, and Anthropic's harness for long-running agents works one feature at a time, after seeing agents try everything at once, declare victory early and leave features untested.

Tests as contracts. An agent that writes both code and tests can make both wrong the same way. Tests that define behavior should be written or approved by a person, protected by code owners and never changed in the pull request they validate.

Review is the bottleneck. Keep pull requests small (one task, one pull request), ask the agent for evidence (test output, screenshots, what it skipped) and label agent work to measure it apart. Anthropic found that with experience, oversight shifts from approving each action to monitoring the work.

Checklist before switching on background agents

1. An AGENTS.md with build, test and lint commands and no-go areas.
2. Human-owned acceptance tests on critical paths.
3. CI gates: tests, types, static analysis, secret, dependency and license scanning.
4. A size limit per pull request; agents never merge their own work.
5. Short-lived, least-privilege agent credentials, separate from human accounts.
6. A network-restricted sandbox and a label on agent commits.

Governance, licensing and IP in Colombia and Europe

Policy first. DORA's AI Capabilities Model includes a clear AI policy: which tools, which repositories, which data, who can switch on background agents. Without one, people bring their own tools.

Code is data. Repositories hold trade secrets and sometimes personal data. Choose plans that exclude training on your code, keep production data away from agents and check where inference runs. In Colombia, a vendor processing personal data on your behalf needs a transmission contract, and international transfers fall under Article 26 of Law 1581 of 2012. In the EU the GDPR applies, and the amended Article 4 of the AI Act (in force since July 27, 2026) asks companies to support AI literacy among their staff.

Ownership and licenses. The US Copyright Office concluded in January 2025 that prompts alone do not make AI output copyrightable: protection requires human authorship. Keep evidence of human contribution (specs, reviews, commits), read your vendor's IP terms and scan licenses, since generated code may resemble existing open-source code.

Vendor risk. Tools change owners (Cursor now belongs to SpaceX) and access can change for political reasons: a US export-control directive made Anthropic withdraw Fable 5 for all users from June 12 to July 1, 2026. Keep instructions in open formats, tools on MCP and a tested second option.

An operating model and the metrics that matter

Roll out in stages. Start where specs are clear and verification is cheap: bugs with a reproducing test, dependency updates, test writing, documentation, mechanical migrations. Add background agents once gates and metrics exist, and keep architecture, sensitive code and data migrations under close human control. After 8 to 12 weeks, compare with the baseline and decide repository by repository.

Our rule of thumb when we build custom software: the agent writes, the tests and the reviewers decide. For agents beyond code, see AI agents in production.