We Spent a Decade Making AI Feel Instant. Here's What We Learned.

Last fall, I was prototyping an AI agent. The RAG pipeline was solid: good embeddings, well-indexed corpus, decent retrieval quality. On paper, it worked. In practice, every user interaction followed the same pattern: the agent needed context, so it called out to a vector database over the network. That call took anywhere from 300ms to a couple of seconds depending on load and location. The agent was making two to three retrieval calls per turn. By the time it had the context it needed, the user had been waiting over a second before anything useful started happening.

In a chatbot, a second is annoying. In a voice agent, it's a dead pause that makes people hang up. In a copilot, it's long enough for the user to context-switch to another tab and lose their train of thought.

I'd spent years watching this exact problem kill otherwise good products. This post is about why I started Moss and why I became convinced retrieval needed a completely different architecture.

Speed as the Ceiling on Intelligence

Before Moss, I was an ML Lead at Grammarly, working on real-time writing assistance for 40 million daily active users. Before that, I built ML systems for Bing and Office at Microsoft.

At Grammarly, I led personalization for Grammarly Keyboard: making AI suggestions feel right on a mobile device where every millisecond counts. We ran models on-device, ranked suggestions in real-time, and optimized until the keyboard felt like it was reading your mind. That work drove 300% retention growth. The model was already good. What changed was that users could actually feel how good it was, because the suggestions arrived before they lost patience.

A principle came out of that: in any interactive AI system, perceived intelligence is bounded by perceived speed. I published research at ACL and NAACL, filed multiple patents in real-time ML. But the most useful thing I learned was watching real users and seeing exactly when latency killed the magic.

We solved this for writing suggestions at Grammarly. But conversational AI (voice agents, copilots, anything with back-and-forth rhythm) hit a wall that model optimization alone couldn't fix.

The Problem

During early prototypes, one thing became obvious. The retrieval quality wasn't the bottleneck anymore. Latency was.

Every conversational turn required multiple network round trips to retrieve context. The embeddings were good. The ranking was good. The models were good. But the user still waited.

That's when I realized the bottleneck wasn't semantic search itself. It was the assumption that retrieval always had to happen over the network.

Modern AI infrastructure had accepted that assumption as inevitable. I wasn't convinced it was.

It wasn't just a performance problem. Developers building AI agents were cobbling together vector databases, sync pipelines, caching layers, and embedding services, then spending more time on plumbing than on the product. What should be a single operation (find the relevant context and return it) had become a multi-service architecture problem.

Vector databases like Pinecone, Weaviate, and Qdrant are good at what they do. But they were designed for a world where the query originates from a server, crosses a network, hits a managed cluster, and returns. That works for offline analytics or server-side RAG. It breaks down for voice agents making multiple lookups per turn, copilots that need instant recall, or anything where the user is waiting.

The problem isn't that these databases are slow. The problem is a network hop baked into the architecture. No amount of optimization on the database side eliminates that.

The Question That Changed Everything

Back to that voice agent prototype. In the early days of Moss, we tried caching, connection pooling, pre-fetching, embedding compression. Shaved off 50ms here, 30ms there. But as long as retrieval meant "send a request over a network and wait," we were fighting physics.

Then a simple question reframed everything: What if retrieval didn't happen over the network at all? What if the index lived in the same process as the agent?

Taking that question seriously means retrieval stops being a service you query and becomes a runtime: indexing, synchronization, and local semantic search that ship with the agent and run wherever the agent runs, whether that's a server, the edge, a browser, or a device.

Making that work requires an index format compact enough to distribute to browsers and edge devices, a runtime fast enough for vector similarity search in single-digit milliseconds in WebAssembly, a sync layer that keeps distributed indexes fresh without rebuilding them, and all of it packaged as a single pip install or npm install.

That's what Moss set out to build.

Why Rust and WebAssembly

If the search runtime has to live inside the agent process (Node.js server, Python backend, browser tab, mobile app) you need something that compiles to every target, runs at near-native speed, and has a small memory footprint.

C++ has the performance but painful WebAssembly tooling, memory safety liabilities, and rough developer experience. JavaScript gives portability but not performance: vector math in JS is an order of magnitude slower than native.

Rust gave us native speed, memory safety without garbage collection, first-class WebAssembly compilation via wasm-pack, and a type system that catches entire categories of bugs at compile time. The Rust core is the single source of truth. From it, we cross-compile to WebAssembly for browsers and edge runtimes, and generate native Python and TypeScript bindings. Developers get an SDK that feels native to their language, but under the hood it's the same Rust engine everywhere.

The result: one codebase that runs identically in a Python process, a Node.js server, a browser tab, a Cloudflare Worker, or a React Native app. Same code, same performance, same API.

What Moss Does

Moss is a real-time retrieval runtime. We coined that term because nothing existing described what we were making. Not a database (we don't store your data). Not a RAG framework (we don't orchestrate your LLM pipeline). We're the layer that makes retrieval instant and local, wherever your agent runs.

You connect your data source once. Moss indexes it, creates a compact distributable artifact, and pushes it to wherever your agent lives. When your agent needs context, it does a local lookup in under 10 milliseconds. No network hop. No cold start. No external dependency.

pip install moss, point it at your data, and retrieval just works.

What's Next

Since then, Moss has grown from an idea into production infrastructure powering tens of millions of real-time AI interactions every month. Developers use Moss across cloud, edge, browser, and on-device environments where every millisecond matters.

But we're still at the beginning. Retrieval is becoming one of the fundamental primitives of real-time AI systems, and we're building the runtime that makes it effectively disappear.

This post is the first in a series. Coming up:

• The Retrieval Latency Tax: where the bottlenecks actually live in AI agent architectures
• Inside Moss's Architecture: how we built sub-10ms semantic search in Rust and WebAssembly
• Benchmarks: reproducible performance comparisons against cloud vector databases on real agent workloads

If retrieval latency is on your critical path, try Moss or come talk to us on Discord.

---

Sri Raghu Malireddi is the Founder & CEO of Moss. Previously ML Lead at Grammarly and Microsoft, where he built real-time ML systems serving millions of users. His research has been published at ACL and NAACL, and he is an inventor on multiple patents in real-time machine learning. LinkedIn · X