Large Language Models for Business Automation by the Job

Opening answer (BLUF)

Pick the model by the job, not by the brand. For operations, that usually means four jobs: extraction into a schema, tool calling against your systems, long-context document review, and cheap classification or routing. Stanford HAI's 2026 AI Index found the top closed models clustered within about 25 Elo points on the Arena human-preference leaderboard as of March 2026, which is why a generic "smartest model" ranking is a weak buying guide. [2] Use large language models for business automation as a portfolio: small models for volume, schema-strict models for records, tool-fluent models for agents, and long-context models for the packet of files a person would otherwise read.

The jagged frontier is the real selection problem

Frontier quality is not plateauing, and it is also not even. The same 2026 Index reports 88% organizational adoption of AI, gold-medal performance on the International Mathematical Olympiad, and a top model that still reads analog clocks correctly only about half the time. [1] That is the jagged frontier: a model can look extraordinary on a math benchmark and still miss a date or a "do not auto-renew" clause.

Agent work shows the same pattern. On OSWorld, which tests agents on real computer tasks across operating systems, accuracy rose from roughly 12% to 66.3%, yet systems still fail about one in three attempts on structured agent benchmarks. [1][2] In tax, mortgage processing, corporate finance, and legal reasoning, Stanford reports top-model scores in the 60% to 90% range, with the top 15 models often separated by as little as 3 percentage points. [2] Those gaps are too small to crown a vendor, and too large to skip evaluation on your own documents.

NIST's July 2024 Generative AI Profile (NIST AI 600-1) is the U.S. companion guide for generative AI risk. It tells organizations to identify risks unique to these systems and to evaluate them in use, not only in demos. [3] That is the operating rule we use: score the job, then assign the model.

Extraction: constrain the output, then pick the model

Invoice lines, contract parties, claim fields, and CRM updates fail when the model writes fluent prose that a downstream system cannot ingest. The first control is not "use the largest model." It is a schema.

OpenAI's Structured Outputs feature constrains the model to a developer-supplied JSON Schema so required keys are present and invalid enum values are blocked. The official guide's canonical example is an operations job: extract event name, date, and participants from unstructured text. [5] OpenAI also draws a clean line: use Structured Outputs when the model's reply must match a schema, and use function calling when the model must reach tools or data in your system. [5][6]

Claude can take a custom tool with an `input_schema` and, with `strict: true`, is documented to match that schema exactly. [10] Gemini 3-series models can combine Structured Outputs with built-in tools such as search, URL context, code execution, and function calling, which helps when the extract must be grounded in a file or a live lookup before it is written to JSON. [15] Gemini 2.5 Pro is positioned on Vertex AI for dense documents such as legal contracts and medical records. [14] That is a fit when the extract needs the whole packet in one pass. It is not a reason to send every three-field form through a frontier model.

Score the job on a gold set of real records. Measure field-level precision and recall plus schema-valid rate. A 99% valid JSON rate with a 12% error on "effective date" is still a broken process. Use a mid-size model with schema enforcement for volume. Escalate to a reasoning or "thinking" model only when the field requires inference (ambiguous addenda, conflicting exhibits). OpenAI's August 2025 GPT-5 system card reported that `gpt-5-main` had a 26% smaller hallucination rate than GPT-4o on production-style factual checks, and that `gpt-5-thinking` had a 65% smaller rate than OpenAI o3. [7] Those are vendor measurements on OpenAI's own traffic, not a substitute for your gold set.

Tool calling: test the loop, not the demo

A tool call is a structured request from the model to your code: look up an order, post a journal, open a ticket, fetch a policy. All three families support native function calling. What matters is whether the model chooses the right tool, fills arguments that validate, knows when not to call anything, and survives a later turn when the first tool returns an error.

Berkeley's Function Calling Leaderboard (BFCL) is the independent eval we treat as a baseline for that loop. The V4 board, last updated 12 April 2026, measures simple, parallel, multi-turn, and agentic tool use rather than chat quality. [4] On that snapshot, Claude Opus 4.5 with native function calling led the published overall score at 77.47. [4] That is not a lifetime title. It is evidence that you should not assume the model with the highest Arena Elo is the one that will call `create_credit_memo` correctly.

Vendor docs fill in the product surface. Current Claude Opus and Sonnet models have 1 million token context windows; Haiku 4.5 is the fast tier at 200,000 tokens. [8] Claude 4 added parallel tool use and, at launch, extended thinking with tool use so the model can alternate reasoning and tools in one trajectory. [9] Client tools run in your application. Server tools such as web search run on Anthropic's infrastructure. Computer-use and browser-use tools exist for systems with no API. [10] OpenAI documents the same five-step loop and recommends `strict: true` so arguments adhere to the schema instead of being best effort. [6] The GPT-5 system card reports 0.99 on OpenAI's tool-calling prompt-injection eval for `gpt-5-thinking`, versus 0.80 for o3, which matters once tools can read email, tickets, or web pages. [7] Google's function-calling guide covers parallel calls, chained calls, and combining built-in tools such as Google Search with your own functions in one request. [11] Gemini 2.5 Pro is documented with a 1,048,576-token context window plus native function calling and structured output. [13]

Stanford's agent numbers still apply: even after a large jump on OSWorld, roughly one in three structured agent attempts fails. [1] NIST's profile flags human-AI configuration and information integrity as generative-AI risks. A tool-using model without allowlists, dry-run modes, and an audit log is not an operations control. [3] We design the loop first (which tools, which arguments, which human gate), then pick the model that wins on your tool set.

Long-context document review: window size is capacity, not competence

Review work is different from extraction. The input is a due-diligence folder, a claims file, a board packet, or a year's worth of SOPs. The output is a finding, a risk list, or a "what changed" memo that a person still owns.

Google documents a 1 million token context as standard for Gemini. Current 2.5 Pro lists 1,048,576 input tokens. Gemini 3-series tables use the same 1 million / 64,000 input-output split. [12][13][15] Current Claude Opus and Sonnet models are also documented at 1 million tokens. Haiku stays at 200,000. [8]

Two caveats belong in the runbook. Google's long-context guide still cites near-perfect retrieval (greater than 99%) from the 2024 Gemini 1.5 technical report for a single-needle setup, and the same page states that accuracy drops when you need many specific facts, not one. [12] A large window also does not erase NIST's warning that generative systems need explicit evaluation, not demo-level trust. [3] If the finding is "the indemnification cap is $2 million," you still need a citation back into the source PDF, plus a human sign-off on anything that moves money or liability.

When the packet is multimodal (scanned PDFs, slides, call recordings), Gemini's native image, audio, and video input is a documented advantage of that family. [13][14] When the packet is a long, text-heavy negotiation, a 1 million token Claude or Gemini pass can beat a brittle retrieval pipeline, provided you measure recall on held-out questions rather than trusting the window size.

Cheap classification and routing: spend intelligence where it pays

Most production tokens in an operations stack are not the hard cases. They are "invoice or spam," "which queue," "P1 or P3," "contains PII, yes or no." Those jobs should not hit a frontier reasoning model.

Each family now publishes a small, fast tier for that work. Anthropic lists Haiku 4.5 at $1 per million input tokens and $5 per million output tokens, versus $5/$25 for current Opus 5 and $2/$10 for Sonnet 5 (API list prices on the models overview, which change). [8] Google lists `gemini-3.1-flash-lite` at $0.25 per million input tokens and $1.50 per million output for text, with a 1 million token window, as the workhorse for high-volume tasks. [15] OpenAI's August 2025 GPT-5 system card describes a nano thinking model built for developers who need lower latency. [7]

A pattern that holds up:

1. Classify and route with the cheap model.
2. Extract with a schema-constrained mid-size model.
3. Call tools or review long documents with the model that won that eval.
4. Keep a human gate on irreversible actions.

Stanford's 88% adoption figure is a reminder that "we use AI" is no longer a differentiator. [1] Unit cost per classified ticket, per extracted field, and per reviewed packet is.

How we choose in production

We do not run a single-model bake-off and declare a winner for the company. We run four short evals on the client's own artifacts: field-level F1 plus schema-valid rate for extraction; BFCL-style exactness on the actual tools for calling; question-answering against a labeled packet for long-context review; accuracy at a latency and cost ceiling for routing. [4][6][10]

Vendor docs tell you what the API can do. Independent boards (Stanford HAI, BFCL) tell you the public baseline. [2][4] NIST tells you not to skip measurement because a demo looked clean. [3] The model that ships is the one that wins the job on your data. Whether the operations team sits in Charlotte or Philadelphia, the sequence does not change: name the job, label a gold set, pick the smallest model that meets the bar, and only then talk about agents.

Practical takeaways

• Stop asking which of GPT, Claude, or Gemini is "best." Ask which job you are buying: extract, call tools, review a long packet, or classify at volume.
• Constrain extraction with JSON Schema or strict tool schemas. Fluent summaries are not records. [5][10]
• Evaluate tool calling on your functions and multi-turn error paths. Berkeley BFCL is a public baseline, not a substitute for that test. [4]
• Treat a 1 million token window as capacity. Measure recall, especially when you need many facts, not one. [12]
• Route cheap classification to Haiku-class, Flash-Lite-class, or nano-class models. Spend frontier tokens on the cases those models escalate.
• Keep a human gate on money movement, customer commitments, and legal conclusions. Agent benchmarks still miss roughly one attempt in three. [1] Re-run the four evals when a vendor ships a new snapshot. Leaderboard clustering is tight, and names change faster than process design. [2]

How we can help

Have more questions or want to get in touch? Write us through our contact page, call (980) 322-4500, or email [email protected]. We help operations leaders design model-by-the-job stacks: gold-set evals, tool-calling loops against systems of record, and long-context review with human sign-off.

Citations

1. Stanford Institute for Human-Centered Artificial Intelligence, "The 2026 AI Index Report" (2026)
2. Stanford HAI, "Technical Performance | The 2026 AI Index Report" (2026)
3. National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)" (2024-07-26)
4. UC Berkeley Gorilla / BFCL, "Berkeley Function-Calling Leaderboard (BFCL) V4" (2026-04-12)
5. OpenAI, "Structured model outputs" (2026)
6. OpenAI, "Function calling" (2026)
7. OpenAI, via arXiv, "GPT-5 System Card" (2025-08-13)
8. Anthropic, "Models overview" (2026)
9. Anthropic, "Introducing Claude 4" (2025-05-22)
10. Anthropic, "Tool use with Claude" (2026)
11. Google AI for Developers, "Function calling with the Gemini API" (2026-08-17)
12. Google Cloud, "Long context" (2026-08-21)
13. Google AI for Developers, "Gemini 2.5 Pro" (2026-06-23)
14. Google Cloud, "Gemini 2.5 brings enhanced reasoning to enterprise use cases" (2025-04-09)
15. Google AI for Developers, "Gemini 3 developer guide" (2026-08-18)