A team ships its first production AI feature. It starts with an application and an LLM API, and the first version works well enough to launch. Then the team needs better retrieval, so it adds an embedding API and a vector database. Exact-match queries don't work as well as expected, so keyword search and a reranker follow. Keeping indexes in sync requires an ingestion pipeline. Rising model costs lead to a semantic cache, and debugging the growing system requires more orchestration and observability.
Each decision makes sense on its own. The problem is what happens when they accumulate.
The cost of AI infrastructure goes beyond what you pay for GPUs, APIs, and managed services. It includes the network calls on every request, the dependencies that can fail, the data that has to stay synchronized, and the engineering time spent maintaining systems that exist largely to connect other systems.
For production AI, we've started treating more capabilities as things that need to run as separate services. But embedding, indexing, and retrieval don't always need their own network endpoint, database, or deployment. In many applications, they can run inside the application itself.
That's the idea behind an embedded AI runtime: move the capabilities that don't need to be separate services into the application, reduce unnecessary network calls and dependencies, and let the cloud handle the work that benefits from being centralized.
You still run AI infrastructure, but every additional service has to justify the complexity it introduces.
How One AI Feature Becomes Nine Services
The first version of a retrieval-augmented generation (RAG) application is usually straightforward. An application sends a question to an LLM, which generates an answer using relevant company documentation. A prototype might even fit the documents directly into the prompt. Then the corpus grows beyond the context window, and the team needs a way to find relevant information without sending everything to the model.
An embedding API and a hosted vector database are a natural next step. But semantic search doesn't solve every retrieval problem. A user searching for a part number, error code, or exact identifier may get better results from keyword search, so the team adds a search cluster. Now there are two sets of results to combine, which leads to a reranker. Each addition improves one part of the experience, but also creates another component to configure, operate, and maintain.
The complexity continues to compound. Two indexes need to stay synchronized with the source of truth, so the team builds an ingestion pipeline with queues and retries. The LLM bill starts climbing, so a semantic cache goes in front of it. Agent workflows need consistent timeouts and retries across the growing number of calls, and an observability platform becomes necessary to work out which component caused a bad answer.
The architecture is now the sum of every problem the team solved along the way. A feature that began as an application calling an LLM has become a distributed system of specialized services, each with its own configuration, failure modes, and operating costs.
Vector databases have become a standard part of this architecture. One industry survey found that the share of respondents using them increased from 20% to nearly 64% in a single year. That growth reflects a real need for retrieval in AI applications. It doesn't mean every retrieval workload requires a separate database.
Needing a capability is different from needing a service. Production AI systems need retrieval, ranking, fresh data, and reliable model calls. Whether each capability belongs behind its own network endpoint is an architectural decision.
The Bill You Can See
The most obvious cost of a distributed AI stack is the monthly bill. A managed vector database, a search cluster, and a production cache can each cost hundreds of dollars a month at list prices. Add observability seats, orchestration plans, queues, and other supporting services, and even a relatively small production application can accumulate more than $1,000 in monthly infrastructure costs before usage-based charges.
Those prices vary by vendor, capacity, configuration, and workload, so teams won't all pay the same amount. Still, the baseline cost of operating an AI feature can grow considerably before accounting for the model itself.
Networking adds another layer. AWS publishes charges for data transferred between availability zones and for traffic routed through NAT gateways. At AWS list prices, cross-zone transfer costs $0.01 per GB in each direction, while NAT gateway data processing costs $0.045 per GB. These charges may look small in isolation, but they accumulate as traffic moves between services.
The architecture also determines how much data moves around the system. A query might leave the application to generate an embedding, travel to a vector database, trigger a keyword search, send results to a reranker, and then return to the application before reaching the LLM. Every step introduces another interaction between components.
An embedded AI runtime changes that path for retrieval. When embedding, indexing, and ranking happen inside the application, the retrieval operation can become a local function call rather than a sequence of network requests to separate services.
You still pay to run infrastructure. The runtime changes which infrastructure needs to run separately and which work can happen on hardware the application already uses.
Latency: Every Network Hop Has a Cost
In real-time AI systems, latency determines whether an interaction feels responsive or frustrating. Voice AI feels it most, because retrieval and model generation have to fit within a limited response budget.
Every remote service adds network time before the service can do its own work. AWS's published examples illustrate how much network topology matters: round-trip times can range from hundreds of microseconds within an availability zone to milliseconds across zones, with higher latency still across geographically distant infrastructure. These figures describe network time alone, not the processing time of the services involved.
A request rarely makes just one call. A retrieval step may need an embedding request, a vector search, a keyword search, and a reranking request. An agent may repeat similar operations several times during one interaction. Even if each dependency is usually fast, the cumulative time can become significant.
Tail latency makes the problem harder. The average response time doesn't tell you how long users wait when one of the services is slow. The research paper The Tail at Scale describes how requests that depend on multiple servers can become disproportionately affected by occasional slow responses. Its classic example shows that when a request fans out to 100 servers, each of which is slow 1% of the time, the probability of encountering at least one slow server is approximately 63%, assuming independent failures.
A production AI request usually involves fewer dependencies than that example, but the underlying principle still applies. The more remote calls a request depends on, the more opportunities there are for one slow component to affect the entire interaction.
Moving retrieval into the application removes network round trips between the application and separate retrieval services. It doesn't eliminate every source of latency: loading an index, computing embeddings, filtering results, and calling a hosted LLM still take time. But it removes unnecessary network dependencies from a path that may execute on every user interaction.
For applications with tight latency budgets, eliminating those hops can matter as much as optimizing the individual components.
Reliability: Every Hard Dependency Adds Another Failure Mode
Distributed systems also multiply the number of things that can go wrong.
AWS reliability guidance describes a basic principle: when a system depends on multiple components that must all be available, the combined availability is the product of the availabilities of those dependencies, assuming independent failures. Five hard dependencies that each achieve 99.9% availability produce combined availability of approximately 99.5%.
That works out to roughly 3.6 hours of downtime in a 30-day month, even before accounting for failures in the application itself. This is a simplified model, not a prediction for every architecture, because real dependencies may fail together or have different recovery mechanisms. But it illustrates the cost of adding hard dependencies to a critical request path.
Regional incidents can make the problem worse. If an application depends on a vector database, cache, and queue hosted in the same affected region, those services may become unavailable together. Each dependency also brings its own operational status, incident history, retry behavior, and service-level commitments.
An embedded runtime reduces the number of remote services required to answer a retrieval query. Once the relevant index is loaded, local retrieval can continue without making a network request to a separate search service for every query. If connectivity to the cloud is interrupted, search can continue against the loaded index, provided the application and its local resources remain available.
There are still failure modes to manage. The application needs enough memory, index versions need to be distributed correctly, and the LLM or other external dependencies may remain unavailable. The advantage is that fewer components have to be healthy for the retrieval portion of the request to succeed.
Data Movement: The Hidden Cost of Keeping Copies in Sync
A separate search service usually needs its own representation of the data. The application has a source of truth, while the retrieval system maintains an index optimized for searching it. That separation can be useful, but it also creates a synchronization problem.
Whenever the underlying content changes, the system has to propagate those changes into the index. That means ingestion pipelines, queues, retry policies, and processes for recovering from partial failures. If an update succeeds in one system but fails in another, the application can return an answer based on stale information without any individual service reporting an error.
The problem is well understood outside AI. Instacart has described consolidating search into its primary database after maintaining data across separate systems created synchronization challenges. Its account reported substantial improvements in write workload and storage and indexing costs. The exact savings depend on that architecture, but the broader lesson applies: maintaining duplicate representations of data creates work that a system has to pay for somewhere.
Vector search introduces its own version of this problem. In its account of building vector search, Notion described operating batch and streaming ingestion paths and the challenges of updating embeddings efficiently. Without careful handling of changes, even a small edit can trigger unnecessary reprocessing.
Teams can engineer these systems to work well at scale, but that takes engineering effort, and the complexity grows with the number of representations and pipelines that must remain consistent.
An embedded runtime changes the division of responsibility. The cloud can build, version, and distribute indexes, while the application loads an index version and performs retrieval locally. Updates still need to be processed and distributed, but queries no longer require a separate hosted retrieval service to be available on every request.
For workloads where reads greatly outnumber writes, that separation can be a useful tradeoff: centralize the work that benefits from centralization, and keep the latency-sensitive query path close to the application.
Developer Experience: More Services Mean More Systems to Understand
The operational cost of a distributed AI stack doesn't stop when the application works in production. It also affects how developers build, test, and debug it.
When a retrieval request returns a bad answer, the problem could be anywhere along the path. The document may not have been ingested. The vector index may be stale. Keyword and semantic search may return conflicting results. The reranker may put the wrong passage first, or a cache may return an answer generated from older information.
Observability tools help connect these events, but developers still need to understand each system and how they interact. Each service has its own configuration, credentials, logs, release schedule, and failure behavior.
The same burden shows up in local development. Running a feature can require API keys for several vendors, mocks for tests, and integration tests that depend on external services being available. A developer who wants to change one part of the application may have to start or configure multiple components before testing the change.
This isn't an argument against observability, orchestration, or managed services. Production AI systems still need monitoring, and complex workflows still need coordination. It's an argument for questioning whether every part of the retrieval path needs a separate deployment and operational boundary.
An embedded runtime brings retrieval logic into the application process, where it can be tested and debugged alongside the code that calls it. Fewer remote components can mean fewer integrations to configure and fewer service boundaries to investigate when something goes wrong.
Operational discipline still applies. The goal is to deliver the same capability with less operational work.
Why the AI Stack Got So Complicated
Part of the explanation is historical. Production AI developed rapidly during a period when hosted infrastructure was the easiest way to get a working application into users' hands. Managed vector databases, search platforms, and orchestration frameworks made it possible to assemble sophisticated systems without building every component from scratch.
Reference architectures and tutorials reinforced that approach. Once a particular architecture became common, teams could copy it, connect the services, and move on to the application itself.
That was a reasonable way to get started. Managed services offer real advantages: they can simplify deployment, centralize data, support multiple applications, and offload operational work. The problem is when the architecture of a first prototype becomes the default for every production workload, regardless of its actual requirements.
Some teams have started questioning that default. Anthropic's guide to building effective agents emphasizes that successful implementations can rely on relatively simple patterns rather than complex frameworks. The Claude Code team's experience with agentic search also illustrates that a dedicated index is not automatically the best choice for every retrieval problem.
Specialized infrastructure still has its place, but the architecture should follow the workload. A large shared dataset with constant writes and many independent consumers has different requirements from a relatively stable corpus that belongs to one application and needs low-latency reads.
So before asking which services to add, ask which capabilities need to be remote services in the first place.
The Same Feature Without Most of the Boxes
An embedded AI runtime puts the retrieval path back inside the application. Instead of making separate network calls for embedding, vector search, keyword search, and result merging, those operations can run within the same process. Hybrid ranking and metadata filtering can happen locally, and the application can query an index that has already been loaded into memory.
The cloud still has an important role. It can build indexes, store versions, and distribute updates to the applications that need them. A runtime can load a new index version atomically so that queries continue against a consistent version while an update is applied.
The architecture changes from a chain of remote services into a local retrieval operation supported by a centralized index distribution process.
| Conventional stack | Why it's added | Embedded AI runtime |
|---|---|---|
| Embedding API | Convert text or queries into vectors | Run supported embedding models in-process |
| Vector database | Retrieve semantically similar content | Query a locally loaded index |
| Keyword search cluster | Find exact terms, IDs, and codes | Combine keyword and vector search locally |
| Reranker and result merging | Combine multiple result lists | Use hybrid ranking; add a separate reranker when needed |
| Ingestion pipeline | Keep indexes current | Build and distribute versioned indexes, with update handling appropriate to the workload |
| Semantic cache | Avoid repeated LLM calls | Retain when useful; it solves a different problem |
| Orchestration | Coordinate calls, retries, and timeouts | Simplify where fewer remote calls are needed |
| Observability | Understand failures and performance | Retain, with fewer retrieval services to trace |
| LLM API | Generate responses | Remains hosted in the typical architecture |
Not every box disappears, and it shouldn't. An embedded runtime doesn't replace the LLM, automatically eliminate caching, or remove the need for monitoring. It changes the retrieval portion of the architecture by moving the capabilities that can run locally into the application.
This is the architecture Moss is built around.
In our published benchmark, a top-five retrieval query over a 100,000-document index on an M4 Pro laptop measured 3.1 ms at p50 and 5.4 ms at p99, including query embedding. These measurements cover query execution after the index has loaded, rather than the initial cost of loading the index. The result illustrates the latency possible when retrieval runs on the application's own hardware rather than depending on a remote search service for every query.
Because the query runs locally, Moss doesn't meter each retrieval query. And once the index is loaded, a loss of network connectivity doesn't prevent the runtime from answering search queries against that index, although other parts of the application may still require connectivity.
For production AI teams, the benefit goes beyond a faster benchmark. Teams can reduce the number of services in the critical path, simplify deployment, and make retrieval a capability the application can call directly.
When a Separate Service Is Still the Right Call
Embedded runtimes are not the right answer for every workload. Some infrastructure belongs in the cloud, and some systems need separate services.
Frontier LLMs require large amounts of compute and are typically accessed through hosted APIs. A dataset with tens of millions of documents, frequent writes, and many applications querying the same shared corpus may benefit from a centralized database. Client-server systems exist for good reasons, especially when multiple writers and consumers need a consistent shared view of data.
Memory is another constraint. An in-memory index competes with the application for resources, which matters on small devices and in edge runtimes with strict memory limits. Some organizations also need centralized controls for auditability, data residency, or governance, and those requirements may outweigh the performance benefits of local retrieval.
Long context doesn't make retrieval obsolete either. Research has found that language models can become less reliable at using relevant information as input contexts grow, so retrieval remains useful even as context windows expand. The architectural question is where retrieval should run.
Embedded runtimes fit best when a corpus belongs to a single tenant or application, reads greatly outnumber writes, and the latency budget is tight. In those situations, moving retrieval into the application can remove unnecessary dependencies without giving up the capabilities the application needs.
The right architecture is the one that fits the workload. The mistake is assuming that every capability must be a separate hosted service before evaluating whether it needs to be.
Fewer Services Should Be the Default
Production AI applications still need semantic search, keyword matching, ranking, fresh data, model calls, and observability. None of those capabilities becomes less important as AI systems move into production.
What should change is the assumption about where those capabilities run.
Some belong in centralized services. Others can run inside the application that needs them. The cloud can build and distribute indexes without sitting on every retrieval request, and an embedded AI runtime can handle the query path without requiring a separate network call for every step.
The distinction matters because infrastructure complexity compounds. A separate service creates a deployment boundary, a network dependency, an operational responsibility, and often a synchronization problem. Sometimes those costs are justified. Sometimes the service exists because it was the easiest component to add at the time.
Before adding the next service to an AI architecture, ask: Does this capability need to be a service, or could it be a function the application calls?
For production AI systems, that question can lead to fewer dependencies, simpler operations, and faster responses. Every capability stays, running where it makes the most sense.
