The Reality of Small Language Models in Enterprise IT: Costs, Challenges, and Architecture

It is really a math exercise, Jason James said regarding enterprise adoption of small language models. Organizations running specialized, task-specific artificial intelligence models are finding that efficiency gains come with unexpected production costs, data quality pitfalls, and management overhead.

Enterprise Budget Shifts Toward Specialized Models

The pitch for small language models has become reflexive for organizations facing high expenses or slow speeds with general-purpose tools. By 2027, according to a Gartner prediction, companies will operate specialized, task-driven models at approximately three times the scale of broad, general-purpose large language models. Enterprise budgets are already reflecting this shift. Gartner projected in July that spending on domain-specific and specialized artificial intelligence models will grow 210% in 2026 alone, more than double the growth rate of foundation models.

CIOs running small language models in production describe a more complex operational reality. These models offer efficiency, but not always to the expected level, and entail costs that do not surface until after go-live. Ameet Ubhayaker, director of enterprise data analytics and AI at Protiviti, stated that enterprise adoption has moved from pilots toward mainstream architecture and growth is accelerating. He noted that he is not seeing wholesale replacement of large models, but an emerging tiered stack utilizing small models for routing, classification, and structured output, while frontier models remain reserved for ambiguous, multistep work.

Cost Calculations and Production Math at Retail Tech Vendors

At the retail tech vendor Aptos, former CIO Jason James experimented with local hardware rather than cloud deployment to test three or four specialized model prototypes. He evaluated them specifically to measure whether the cost and effort of running them on-premises beats what the company pays frontier models in token costs. What is the cost to implement? What is the cost to run? And what is the volume of use? If these are all significantly less to run locally, only then should it be considered. If it’s only slightly less, then why bother? James said.

While James recently left Aptos to take on the CIO role at Acumatica, the numbers trended in favor of small models enough for Aptos to continue the pilots. He emphasized that these models did not replace frontier tools, but provided more options while avoiding tokenization bankruptcy. That condition occurs when inadequately trained users burn through a year’s token budget in weeks due to poor model selection and prompting. Ubhayaker noted that narrow, repeatable, and verifiable tasks in environments with real data-boundary or compliance requirements represent where small language models shine. This does not mean SLMs are replacing frontier models, Ubhayaker said. The destination is a layered architecture, not wholesale migration.

Data Quality Challenges and Production Scale Failures

Eric Pace, who recently left an enterprise IT leadership role and now advises companies on artificial intelligence deployment, experienced production data failures firsthand. His team deployed a small, fine-tuned Llama model to curate marketing messaging, trained on historical data labeled good or bad. Testing looked strong, and the path to production was fast and inexpensive, but the data that appeared reliable in testing did not hold up at production scale. We eventually pulled it out, Pace said.

Pace still views small language models as a time-saver with manageable drift and versioning, but he now treats data quality as a materially bigger issue for fine-tuned small models than for general artificial intelligence use. Not only is there a need to ensure that the data sets represent what you want as inputs to the model, but you have to find the line, he said, adding that overfitting remains a real risk. Ubhayaker observed that companies often evaluate small models against public benchmarks instead of their own production traffic and data. Organizations should also optimize for cost per successful task, not simply cost per token, Ubhayaker said, pointing out that retries, human review, and escalation to larger models can quickly erase initial savings.

Fleet Management Overhead and Tracing Failures

Running several specialized models introduces distinct operational overhead. James explained that when a single workflow passes through more than one specialized model, troubleshooting and tracing a failure becomes harder because it requires working through multiple model owners to find where the problem originated. Every specialized model requires independent evaluation and monitoring to catch behavioral drift, according to Ubhayaker. A fleet of specialized models adds complexity around routing, upgrades, and lineage, while base model updates and downstream fine-tunes require retraining and re-benchmarking.

According to Ubhayaker, deciding whether a task belongs on a compact or robust language model ought to be evaluated on a case-by-case basis, with small language model utility depending heavily on the specific workload, the underlying data, and post-deployment ownership.