After Go-Live: Why AI Systems Quietly Lose Value

The easiest day of an AI project is the day it goes live. The integrations work, the test examples are answered correctly, the sponsor is happy, and the project is closed. The real problem usually starts after that: the system doesn't crash, throw errors, or raise alarms. It just becomes a little less accurate, a little less current, and a little less used every month. This silent loss of value is one of the least discussed reasons why technically successful projects fail to deliver business value.

We've covered the pilot phase in How Does an AI Pilot Project Create Value?, adoption in Why Don't Employees Use AI Tools?, and security in AI Safety: Control and Trust in Production. Here we look at one question none of them fully answers: after go-live, who knows, and based on what, that the system still works?

Why Is the Decay Silent?

Traditional software shows an error message when it breaks. An AI system that breaks produces a fluent, well-formatted, wrong answer. An assistant that finds an outdated policy text in its knowledge base repeats it with the same confidence as a current one. The retrieval layer can't tell: similarity scoring doesn't include how old a document is, so a stale paragraph can score just as high as a fresh one.

The result is a system nobody complains about but nobody trusts either. A user gets a wrong answer once, doesn't try a second time, and quietly goes back to the old way. The usage curve drops; the team reads it as “interest is fading,” when the cause is quality.

Four Sources of Lost Value

1. The Knowledge Base Goes Stale

Price lists, return terms, product features, and internal procedures change every month. When a knowledge base is loaded once and left alone, it starts giving outdated answers within a few months. In case analyses shared by practitioners, this is cited as the most common cause of the quality drop seen in the first months after launch. A knowledge base isn't an asset you build once; it's a product with a shelf life.

We described which source should serve as the reference in What Data Should a Chatbot Use?. A second question belongs with it: who updates that source, and how often? If the answer isn't written down from the start, updating it becomes nobody's job.

2. The Incoming Data Changes

The system was tested on a certain data distribution. Then a new supplier starts sending invoices in a different format, customers ask different questions during a campaign, a new product launches. In the literature this is called data drift: the model stays the same, but the inputs move away from the world it learned. Accuracy falls slowly, and it can take months to show up in the average metric.

3. The Environment Changes

From the outside the system looks the same, but the ground beneath it has shifted: the model provider updates a version and the same prompt gives a different answer, a field in the CRM is renamed, an API version is deprecated, permissions are reorganized. Integrations often don't fail with an error; they keep running on empty or incomplete data.

4. The Users Change

Users who give feedback in the first weeks stop over time. Corrections no longer flow back into the system, and people develop their own workarounds. This is one face of the adoption problem, but here it has an added consequence: the only signal that could improve the system disappears.

Who Owns the System After Handover?

The project team disbands, the sponsor moves to new priorities, the external team's contract ends. The system keeps running but has no owner. On the day something goes wrong, it's unclear who to call. In the software world this is called an orphaned application; with AI it gets noticed much later because of the silent decay.

The solution isn't complicated, but it has to be written down from the start: one name responsible for the business outcome, one name responsible for technical continuity. Responsibility has to be measurable, too. “The system should work well” isn't a goal; “accuracy of support-ticket summaries must not drop below the agreed threshold, and response time must not exceed this limit” is.

Maintenance Budget: A Line Item Separate From the Project Fee

Most companies budget an AI project as a one-time investment. A live system, however, needs monitoring, knowledge base upkeep, retesting, compatibility with model versions, and security patches. A rule of thumb often cited in the industry puts annual running costs at roughly 15 to 30 percent of the initial development cost. That's not a standard, just a rough reference; the point is to see that having no such line item is the most expensive option.

Surveys by analyst firms such as S&P Global pointed to a marked rise in abandoned AI initiatives in 2025. It's worth noting that the definition of “abandoned” and the methodology vary between sources; the trend is nonetheless consistent: building a system is easier than sustaining it.

A Minimal Operating Framework

You don't need a large MLOps platform. Five habits that work at SME scale make most silent decay visible.

Weekly sampling. Each week, the responsible person reads twenty randomly chosen outputs from the system and marks them right or wrong. Twenty records aren't statistics, they're an early warning; but when the share of wrong answers starts rising, you'll be the first to see it.

Document owners and freshness dates. Every source in the knowledge base needs an owner and a “last verified” date. When a policy changes, updating the document must be part of the process that changes the policy.

A fixed test set. A list of fifty to a hundred real examples with known correct answers. When the model version, prompt, or knowledge base changes, the same list is run again; if results drop, the change is rolled back.

Change triggers. Events such as a new product, a new price, a new supplier format, or a CRM field change go onto a checklist that assesses their impact on the system. That way, change stops being a surprise.

Usage and correction data. Record which answers users reject or correct by hand. That record is both an improvement list and a quality indicator.

When Should You Retire a System?

Not every system is maintained forever. If its output isn't used, the process it served has changed, or maintenance costs exceed the value it delivers, shutting it down deliberately is a healthy decision. What's harmful is the system that isn't shut down but nobody looks at: it keeps giving outdated answers, keeps accessing data, and keeps generating cost. Set the criterion up front; for example, a review takes place if the target metric stays below the threshold for two quarters in a row.

The success of an AI project is measured not on the day it goes live but by its condition a year later. That's why a good project plan starts not with the demo date but with the operating plan for the first three months.

If you'd like to review the quality of a live system together with us, or set up the operating plan for a new project from the start: tell us about your project.