Which Part of Your Internal AI Tool Should You Still Own?

Richard Meadows reconsidered an AI tool when a search failed to find a note the team knew existed. The search was still answering queries. The problem sat earlier in the process: new material had stopped entering the collection the search drew on.

Meadows, head of content at Streamrise, described the failure in a written response to PSEEDR. The team used semantic search to find incident notes written by people and by AI coding agents, along with a library of research articles. An outside embedding API, paid per call, supplied the numerical representations the search used to compare meaning. On October 26 and 27, 2026, the prepaid credits ran out. The provider began returning HTTP 402 responses, and eleven files failed to re-index. Nobody noticed from a dashboard. The missing note exposed the problem.

The team replaced the API with an open embedding model running on a machine it already owned, and kept the search pipeline around that model. Where another company might outsource a troublesome internal component, Streamrise brought an external component in house.

Both directions can make sense. The useful question is which part of the work should move, what burden that move removes, and who will check that the whole service still does its job afterward. Six practitioners' written responses to PSEEDR show why the answer depends on the result a business must deliver, not simply on whether a model runs inside or outside the company.

A working component does not prove a working service

Meadows reported that the team re-indexed 889 notes and 270 articles with the local model, with zero errors on that run. The team judged search quality by impression on its own queries, without a formal benchmark, and Meadows asked that this part be treated as an estimate.

"The deciding factor was not cost per call, which was small," Meadows wrote. "It was that the failure mode of the outside product was silent: an exhausted balance looked exactly like an empty search result."

That word "silent" needs precision, because the provider had returned an explicit error. What stayed quiet was the consequence: some files never reached the searchable collection. Search kept answering queries against the material already indexed, so a responsive interface told the team nothing about the files that were missing. The account does not say how the integration handled the 402 responses. It does show that the failure never reached the people relying on the result until one of them went looking for a note.

With the local model, Meadows said, the failure mode becomes a service that is down, which the team's existing health checks already catch. The one new obligation is checking that the local model service is running, and the team wrote that check into the same note that describes the system, so the next person who sees empty results knows where to look first. That makes one particular failure easier for this team to recognize. It does not establish that local models are inherently more reliable, and a running service does not prove that every document was processed. A local service that stays up but drops documents would produce the same empty result.

Google's Site Reliability Engineering book, in its chapter on monitoring, separates checks of a system's internal state from tests of behavior as a user sees it. It also counts a successful response paired with the wrong content as an error, and notes that only end-to-end tests catch that class of failure. Checking that a process is alive and checking that a user got the right result answer different questions.

For this search tool, PSEEDR's proposed comparison therefore has two parts. First, check whether newly added material was actually processed. Tracking failed indexing jobs is the obvious start, though a job that never ran leaves no failure to record. Second, test whether a user can retrieve material that should be available, which a small, maintained set of known-item searches would do. Meadows did not report implementing either check.

Even those checks have limits. Finding a known note says little about ranking quality across other queries. A migration that changes the embedding model also needs a way to judge whether typical searches still return useful results. Representative queries with expected answers would give a firmer comparison than an impression of quality. Preparing and maintaining that comparison is work the team should include when deciding whether to switch.

Meadows' advice to others was to "ask how each option fails, not only what it costs." The case supports a narrower version: ask how each option's failure reaches the person who depends on the result. The migration removed the prepaid API dependency. It did not remove the team's responsibility to know whether its knowledge base was searchable.

Identify the work a supplier would actually take over

For Runbo Li, co-founder and CEO of Magic Hour, the pressure pointed in the opposite direction. The company's early AI video system combined open-source models, custom queuing logic and GPU orchestration, all managed by a two-person team. Each new open-source model meant re-integrating, re-optimizing and re-testing. By mid-2024, Li estimated, keeping that infrastructure working consumed roughly 30 to 40 percent of engineering cycles.

Growth made the burden harder to accept. Li said the user base rose from thousands to millions within months, and rebuilding the internal system for that demand would have paused product development for weeks. The team evaluated several third-party inference providers and moved certain workloads to them where it considered the model commoditized and reliability more important than customization.

It kept its template system and the orchestration logic that lets non-technical users get finished output without writing a prompt. In other words, a provider could operate the model while Magic Hour continued to determine how a customer used it to get a result.

"The key question wasn't 'is this cheaper?' It was 'does this free us to ship faster?'" Li wrote. The time estimate describes the team's own experience, not a measured comparison between providers. Li reported no commercial relationship with any inference provider and said the team now reassesses quarterly, on the view that something built internally six months ago might be better rented today, and vice versa.

Bharat Sharma, delivery manager for enterprise CX solutions at eSignly, described a similar division of work. The team had built a natural-language engine for regional legal risk assessment because, in Sharma's account, commercial models at the time lacked the required regulatory depth. By early 2024, senior engineers were spending more time on data labeling and infrastructure than on the legal logic that distinguished the product.

The team moved document classification and entity recognition to a specialized commercial platform while keeping its proprietary scoring algorithms. Sharma reported higher document processing volume afterward without additional engineering headcount, but the response did not define how volume was measured. The clearer evidence is the division itself: the supplier now decides how documents are classified and which entities are identified, and the internal team keeps the scoring logic that turns those inputs into a risk assessment.

That division also shows why retaining code does not guarantee unchanged behavior. If a replacement classifier assigns a document to a different category, or an entity recognizer supplies different inputs, the same scoring algorithm can produce a different result. This is a possible consequence of the arrangement, not a failure Sharma reported. Evaluating the combined output remains necessary even when the proprietary part of the code does not change.

Before making a similar move, a team needs to distinguish recurring upkeep from work that improves the system. The same book's chapter on eliminating toil describes toil as operational work that tends to be manual, repetitive, tactical and without enduring value, leaving the service in the same state afterward. It also notes that grungy work with lasting value is not toil. A difficult one-time change that permanently improves a service differs from repeatedly restoring it to the same state after each model release.

Applied to a sourcing decision, that distinction changes what should be counted. A supplier may remove repeated infrastructure repairs. It may leave integration, evaluation and customer-specific rules with the team. The relevant saving is the work that actually disappears, weighed against the work the new arrangement requires. Li's estimate is a reason for another team to examine its own burden, not a universal percentage at which a company should outsource.

Compare components, workflows and capabilities separately

Owning an internal platform can include buying its components. In an October 20, 2025 account, BNY described building Eliza, an internal AI platform that uses models and vendor tools from several providers and offers a menu of models to internal builders. BNY said the platform was purpose-built around its data, risk, legal and compliance standards and lets it integrate new models as they become available. This is the company's description of its own architecture, not evidence that other businesses should copy it.

It does make the question "build or buy?" more precise. Replacing a model changes one component. Replacing the workflow around it changes how work moves through the business. Retiring the capability means deciding that the business no longer needs that result, or will obtain it another way. These decisions remove different amounts of work and leave different responsibilities behind.

In Streamrise's case, the result was finding the knowledge the team had already written down. In Magic Hour's, it was helping a user produce a finished video. Neither result required every component to have the same owner. Starting from the result makes it easier to identify which parts can move without giving up what the business needs.

This distinction also prevents a misleading comparison. An inexpensive model service is not a substitute for a complete workflow if the team still has to build and maintain everything around it. Conversely, a complete outside product may take on work that the team has good reasons to retain. A proposal to "buy instead" needs to state which of those changes it actually means.

Low usage leaves the case for automation unproven

Kaleel Lawrence-Boxx, founder of Boxx Automations, had a different problem: an automation with little traffic to handle. The agency's lead-nurture system ran on a self-hosted n8n server. When someone requested a site audit, it pulled their first name and context into email templates, then kept following up until they booked a call or opted out. A Google Sheets CRM tracked each lead's progress.

Around September, Lawrence-Boxx considered replacing it with Claude plus Gmail and Google Sheets connectors, which would have left far less to maintain. "At my current volume the custom build was overkill," the response said. Almost nobody was requesting audits.

Lawrence-Boxx kept the system because, in the founder's assessment, the connector approach depends on being personally in the loop. Once volume rises, the founder wants every lead enrolled in a flow automatically, with personalized follow-up and a record of who responds to which content. Keeping the pipeline in the agency's own email stack also mattered. Lawrence-Boxx sells lead-nurture automation to small businesses and disclosed that commercial interest.

"I have no performance data yet," Lawrence-Boxx wrote. "Volume is too low to judge, so this is a bet on future scale, not a measured result."

The uncertainty is central to the choice. Low volume means the larger system has had no opportunity to show its benefit. It also means a simpler process that needs human attention may be adequate for now. At higher volume, that same human involvement could become the constraint. The current experience cannot settle which arrangement will work better then.

Keeping the system therefore rests on an anticipated requirement, not demonstrated savings. A useful reassessment would compare the maintenance burden against the human work the simpler option requires as real inquiries arrive. Lawrence-Boxx plans to retest the simpler setup once volume increases. That creates an opportunity to replace the forecast with evidence, rather than treating the original build as a permanent commitment.

Customer requirements can make an apparent substitute unsuitable

Dora Bloom, chief revenue officer at iotum, described a reason to retain ownership that does not depend on visible product features. The company built its AI meeting assistant in house in 2018, Bloom said, and still owns it across its product line, including the API side and healthcare work.

"The reason it stays owned is the business we are in rather than anything clever about the code," Bloom wrote. iotum sells white-label products: service providers put their own names on the product and sell it to their customers. In Bloom's account, every component iotum does not own becomes a dependency that those partners' customers are exposed to.

Bloom also cited a published commitment that iotum and its subsidiary brands will never use or sell customer information for AI training, and said that commitment has to hold for everything in the stack. PSEEDR did not locate the published text of that commitment during research, so it is reported here as Bloom's statement. Bloom described a standing commercial requirement rather than a dated comparison with a named supplier, and offered no measured saving.

If that data condition applies to the service, a replacement must satisfy it as well as perform the task. A meeting assistant that produces suitable output but handles customer information on unacceptable terms does not meet the same requirement. Internal ownership alone does not establish compliance either: the condition has to be checked across the actual service, including any dependency the team still buys.

This is the limit of treating differentiation as a list of visible features. How a service is permitted to handle data can determine whether a customer can use it at all. That requirement belongs in the comparison before price or maintenance savings can decide the outcome.

Keep an owner for the complete result

Victor Smushkevich, founder of Tested Media, a marketing and growth agency for service businesses, described a recurring pattern in automation work. A custom lead follow-up workflow or form-scoring script works on day one. Later, a model changes, a form field moves or the CRM updates its API, and someone has to repair the connection.

"The question I ask is who gets the call when it breaks," Smushkevich wrote. "If the answer is nobody, it belongs in an outside product with a vendor behind it." What stays in house, in that account, is anything that encodes how a business qualifies or routes its work. Once those rules are written down, the model underneath can be swapped without touching them. Smushkevich presented this as a pattern from agency work, not a dated case study.

A supplier can take responsibility for operating a component or a workflow. The business still needs someone to recognize a missing result, raise the issue with that supplier and decide what happens to affected customers in the meantime. Otherwise the gap remains, even if the code has a new owner. Streamrise's missing note illustrates the same gap from the other side: the embedding API reported an error, but the people using search did not know what it meant for the collection they were searching.

An effective reassessment should therefore end with a specific account of the change: the result the business needs, the component or workflow that will move, the work that will disappear, and the checks and decisions that stay with the team. It should also name the evidence that would justify reopening the choice, such as sustained maintenance pressure, a changed customer requirement, or enough real usage to test a forecast. Li's quarterly review and Lawrence-Boxx's planned retest are two forms of that commitment.

Keep ownership where the business needs control over its particular rules or customer commitments. Consider moving a component when another arrangement can meet those requirements and remove work that is holding the team back. In either case, keep a clear owner for checking the complete result. Changing who runs the model does not answer whether the user can find the note, finish the video or receive the promised service. The customer sees the result, not the model.