I spoke with Neil Movva, the co-founder and CEO of Sail Research.
We start with the mechanics of inference (throughput vs latency tradeoffs, memory bandwidth) then follow those constraints through chip design, distributed systems, and the economics of smaller and larger data centers.
Table of contents:
This transcript has been edited for clarity.
The inference market: throughput, latency, and background agents
Cat Tang: Today, I’m talking with Neil Movva, the co-founder and CEO of Sail Research. Sail provides inference and infrastructure for long horizon agents. Neil, it’s really great to have you. Thanks for coming on.
Neil Movva: Thanks for having me.
Cat Tang: I’d love to kick off the discussion by talking about how you see the shape of the inference market. So I think by now it’s clear inference is this huge and rapidly growing market, but what are the workloads we actually see? What is the distribution of work that is, for instance, focused on throughput versus needs very low latency?
Neil Movva: Yeah, absolutely. So I think we can agree that one of the use cases that really kicked off the inference market was coding. It has extreme PMF today. It’s clear that coding has changed forever at this point. And coding is what really drove a lot of the inference demand and structure of the providers you have today over the last year. Specifically, the coding market demands low latency. It’s a very interactive use case. There’s a user, programmer, who is in conversation with an agent.
And they’re typically talking to that agent every minute to five minutes. That’s the window of turns at this point. And that is the core loop that all the current CLI harnesses, like Claude Code and OpenCode and otherwise, are built around.
And I think that’s what has pushed the inference market to focus on speed. If you want to have an interactive experience with a high value use case like coding, you’re going to focus on speed.
You have Claude Code having fast mode. You have inference providers like Fireworks and Baseten and Together really focusing on speed, tokens per second, that is. And there’s always going to be more appetite for that kind of demand.
But in the last couple months to about a year now, I would say, we’ve seen a new type of use case, which is these agents that can run autonomously or in workflows for maybe much longer between human interactions. Instead of running in a loop with a human providing input every five minutes, the agent will run a workload for maybe hours, maybe days, and eventually even weeks. We see this with, for example, OpenAI chasing math problems with very long horizon agents. We see this with OpenAI’s top internal users spending thousands of dollars per day on tokens.
I find it hard to believe that those thousands of tokens are being spent where the human is in the loop on all those tokens. These are some form of background agent that’s being used here. And that’s what really excites me is this rise of background agents. There is a pretty tight bound on how many tokens one person can consume when they’re at the laptop talking to the agent. There is no bound on background agents that are running against hard problems for hours, days, and weeks.
Cat Tang: Yeah, I think this makes a lot of sense.
Inference fundamentals: prefill, decode, and the roofline curve
Cat Tang: I think what would be helpful is maybe zooming back just a little bit and talking about the tradeoff between throughput and latency as well. So I think this is somewhat intuitive if you have some compute background, but a breakdown would be really helpful and maybe just thinking about the roofline curve and why eventually you have to shift between being memory bound to being compute bound.
Neil Movva: Absolutely. So yes, we talked about a little bit about how current inference is pretty focused on these real time interactive use cases.
And what that really means is you were focusing on having the language model produce output tokens as quickly as possible. That is the kind of like core workload we’re really dancing around here.
But maybe we should zoom out again and ask about what a language model is. What is the computation involved there? There’s a transformer, which is of course like a large neural network, which has billions if not trillions of parameters at this point.
And you can take the inference, I guess workload and break it into two phases. There’s pre-fill, which is processing a large prompt that the user has put in or it could be the output of a tool call. It’s a lot of text that’s coming in thousands of tokens at once. That can be processed in parallel and we call that pre-fill. The term comes from basically you’re filling the KV cache essentially. And once you’ve filled those tokens, you never have to compute on them again. You’re just recalling them from memory.
Then there’s an output token phase where you’re producing output tokens one step at a time. And we call that decode. We are decoding the answer from what we’ve pre-filled in the past. And the decode step is I would go so far as to say a decode is the only fundamentally expensive thing in transformers today. It fundamentally stresses, it pushes the GPU into a regime that it’s not necessarily originally designed for. Because what happens during decode is you have to read all the parameters of the model, billions or trillions of parameters, at least once for every token you’re producing.
And so we end up being bottlenecked by the time it takes to load the weights, these trillions of parameters, into the GPU, perform the computation, and then cycle that iteratively. And the GPU is basically spending all of its time loading the weights. It’s not spending its time doing the compute itself. The compute is cheap compared to the weight loading.
And so that’s why we get this idea of these two phases. The GPU can be either compute bound where it’s doing a lot of work during pre-fill. All the tokens are sent in at once and it can amortize the loading of the weights across maybe thousands of tokens that it’s computing. So it becomes compute bound. And the other phase where we’re processing one token at a time per stream. And so that becomes the memory bound decode phase.
Cat Tang: And I guess why can’t the GPU, I guess, make the memory faster?
Neil Movva: Well, this is an interesting question. This is one of the longest standing challenges in computer architecture is we have always found ways to advance the speed of compute faster than we can advance the speed of memory. Always it’s easier to make compute-- well, as Moore’s law basically gave us more transistors per square millimeter. We were able to basically pack on more logic and do more work per cycle of compute.
But memory is an entirely different industry. In fact, you have different fabs, different companies specializing in memory. And the fundamental structure of DRAM of memory that we use for computers is so different that it becomes hard to just incrementally speed that up at this point.
Cat Tang: Yeah. And so what you end up with then is essentially a whole industry that is bottlenecked on memory bandwidth.
Neil Movva: Yeah. I mean, that’s a whole-- that’s a large discussion we can have. We can talk for hours about memory.
But yes, we end up having this ratio to your point about the roof line.
There’s a ratio between the amount of work compute-- we can-- or arithmetic, really-- that we can do per cycle on the logic die compared to the memory we can fetch from memory chips that are stacked alongside the logic die. And so we consider this ratio to be-- well, I call it the roof line ratio, which is basically to say, what is the minimum amount of work you must do per byte of data you fetch in order to be compute bound?
If it’s more expensive to fetch data than it is to compute it, how many operations must I do per byte of data that I fetch to be compute bound?
And that number for NVIDIA GPUs has been remarkably stable over the last few years and across their families of chips. It’s usually in the mid hundreds, so 400 to 600 or 800 or so, pretty tight band. And there have been other accelerators in the past that have gone well above 1,000. That’s a really compute focused accelerator. And there are CPUs, which are much more focused on fast-- or don’t really focus on compute in the flops sense, but more on branching and control flow that have a much lower roof line ratio, which is to say between 10 and 100 typically.
Chip architecture, sparsity, and mixture-of-experts models
Cat Tang: I’m curious if you’ve seen a shift on the hardware layer for this workload as well. So I think that the changes we’ve seen workload-wise ripple through. And so you have initially maybe changes in the way that you distribute the workload coming in, changes in the way you write kernels. And then eventually it flows down to the hardware level too, where today there’s a lot of ASICs that some focus on high throughput, some focus on low latency. I would be curious your thoughts on where the overall distribution of workload is falling.
Neil Movva: Yeah. So I think the ASIC market is very interesting to look at as well. All hardware architects want one thing to make a great chip. They want specialization. They want control or I guess like tight definition of the workload. And that allows them to build more structure into the chip that suits that workload, broadly speaking. And in deep learning, that you could argue has already been done somewhat by NVIDIA in the sense that they’ve added specialized hardware to the chip that is uniquely suitable for deep learning or modern deep language models.
And that hardware is the Tensor Core. The Tensor Core is a very important part of, I guess, the foundation of what makes the GPU great at matrix multiplication. It is not something that-- well, the GPU initially was thought of as-- it’s a parallel machine. And so any parallel problem should fit on this parallel machine.
And that is true. But the GPU historically was a vector machine. Vector being that there’s like one dimension of parallelism here. And then you can just stamp out more vector units. And that is a reasonably good way to compute matrix products. But you can do even better. You can build a two-dimensional structure on the GPU that for large matrix multiplies turns out to be even more effective at basically computing tiles of a matrix multiply with much higher arithmetic intensity and much more ratio of compute to memory movement.
And so NVIDIA started putting those into the chip about 10 years ago. And those have really grown on the chip. So we started with small systolic arrays, which are the structures we use for matrix computation. And we grew them over generation over generation to where they are now, which is maybe a fairly large size systolic arrays. And so there are some other architects of chips who saw this pattern and said, what if we do it all the way? What if we build a massive systolic array onto the chip?
Systolic arrays have an interesting property of essentially the data movement. I guess like number of wires and energy you need to move it onto the systolic array is proportional to the, I guess, perimeter of the array. So if you think of it like a square systolic array, it’s going to have data movement proportional to N for an N by N systolic array. But then the compute on that array is N squared. And so you have this natural ratio that the arithmetic intensity of the systolic array is increasing as you get to larger N.
And so there’s kind of this goal to make that as big as you possibly can.
That is one of the ways to get really high efficiency of compute on a chip. And so to zoom back out to ASICs, there are people who have been kind of trying to build very large systolic arrays. You could argue that the TPU was the first to go big on this. And nowadays we see newer chips like what Etched is doing, what MatX is doing that go for larger systolic arrays. And they get more efficiency that way.
Cat Tang: And just to make sure I’m understanding you correctly then. So if you had, for instance, a gigantic block of compute, you might actually be less efficient on the bandwidth side because you’re not allowing enough transference in between. And so if you instead have a split systolic array, then you might have much better communication on your memory bandwidth.
Neil Movva: Yeah, yeah. I should define these a little bit more. So I think that you want the arithmetic intensity, the ratio of compute per cycle to bandwidth moved per cycle.
You want that to be high. Fundamentally, memory is expensive, memory bandwidth is expensive. And compute, I wouldn’t say it’s cheap, but it is relatively cheaper. And so higher arithmetic intensity is like a design that you are going to get higher efficiency. You’re going to get better performance per watt, performance per area. Everything is going to be better at higher arithmetic intensity. But you don’t control the workload always. The chip architect does not design the shape of the matrix multiply that the model wants. And so the challenge for architects has always been to balance flexibility of the chip with the essentially efficiency that is inherent to a larger fixed blocks.
And this is most concretely felt in ASIC design for AI with respect to the size of the systolic array. And to your point about splitting the systolic array, that’s absolutely what people do. If you make the systolic array too big, essentially what I’m saying so far is that big systolic arrays are good, so why not just make it very big? Well, if you have a small matrix multiply that sits on a large systolic, you’re going to be wasting a lot of the area and compute is going to be idle. You’re going to be padding it with zeros and wasting some precious area.
Cat Tang: Yeah. I think a lot of this was dependent on what workloads end up looking like, right? And I would love to hear how you’ve seen workloads change even in the last year, but I’d also like to hear thoughts on what it’s going to look like in the future and how much of that market you’re interested in.
Neil Movva: Yeah. Yeah. Okay.
So there’s one theme that I’ve followed for many years now in all the machine learning, which is this idea that most, I guess, like patterns that you were trying to model in a deep model, in a deep language model or even traditional machine learning model, most of these patterns are very sparse. You want to have a lot of world knowledge, but on any given prediction, you are not going to leverage all the world knowledge that you’ve trained or compressed into a model as a high-level idea. So there’s some inherent sparsity to these problems.
How do we capture that sparsity? Well, it’s been a challenge. We started by building these super large models heavily overparametrized. For example, people were training hundreds of billions of parameters of dense models, meaning on every forward pass through the model, you’re activating hundreds of billions of parameters. All the parameters in the model are being used to make a prediction. But we know intuitively and also if you mechanically interpret the activations or the projection of these weights onto this input, you’ll find that most of the parameters are not involved in a single prediction.
And we’ve started to formalize that about two years ago with what we call a mixture of expert models, the idea being that you can take a very large model and have it for itself figure out which subset of its parameters are most relevant to every token for every prediction that is going through the network. And that is a really great way to capture structured sparsity within the model. And we’ve gotten pretty good. I’m not familiar with the tricks. I’m not a pre-training guy. But there are many tricks that pre-training people have figured out to have the model essentially dynamically select which subset of its parameters are going to be used in every forward pass or on every processing each token.
And so that’s been a major change in the-- well, for hardware. It means that the systolic arrays are not just going to get bigger and bigger infinitely. Now we do have to consider that maybe we can get more granular for each expert that we are mixing over. And so maybe the array shape is not going to be just monotonically increasing. But rather, we’re going to get to a point where we have, yes, trillions of parameters, but only a small subset of them active on every forward pass. We are there today.
And that’s what’s kind of been a challenge for people who build really large systolic arrays. It’s hard to fill them with small experts.
Proactive agents, sleep-time compute, and deep research
Cat Tang: Yeah, absolutely. So I think mixture of experts is one big change. And then maybe even higher level than that is the change to multi-agent type workloads.
Neil Movva: Yeah, yeah, right. There’s two levels to talk about here. There is the mechanics of the language model itself. Model architecture is very near and dear to my heart. We want to be maximally efficient when we build software and hardware for these language model structures. But at the same time, there’s this larger pattern change around what we talked about earlier, which is moving from interactive chatbot type use cases to more asynchronous long horizon agent use cases.
And that transition has only really been quite recent. Models had to be good enough to trust to run for hours on their own. And we’re just now seeing this massive renaissance in personal agents. It feels like we see one of these every year, but now they really work. And so they’re getting there on the cusp of or actively achieving product market fit. It’s really amazing to see. And I think that that market is really interesting because it’ll be most people’s first experience with really proactive agents.
That’s the trend I’m most excited about right now.
Cat Tang: Is that primarily the type of work that you’d be targeting? Because there’s, you know, just to say that a workload is multi-agent doesn’t necessarily mean it’s latency insensitive, right?
Neil Movva: Yes.
Cat Tang: In fact, like much of multi-agent work is actually very latency sensitive. Jalapeño is actually built for latency for this reason. And so what do you look for when differentiating workloads that are not latency sensitive versus ones that are?
Neil Movva: Yeah. So really the axis I’m looking at is more proactivity. What can the agent do without you asking for it? There’s this like glib statement we like to make, which is that the best latency of all is none. If you can proactively predict what the user wants and go after essentially doing sleep time compute to get that work done ahead of time based off what you know about your priors on that person and their use case, well, that’s the best way to surprise and delight a user.
And our goal is to make compute so abundant that that kind of proactive computation becomes essentially cheap enough that you can do it much more often.
And the reality is that most of these proactive, I guess, threads are not going to resolve or are not going to result in anything visible to the user. Most proactive threads will not result in a Navier-Stokes proof. They will not do anything useful, but we want to encourage that exploration. There is a lot more you can sample from a language model than just hoping that it gets the right answer in response to users demand. And so there are some early ideas here. I wouldn’t say that they’re super well fleshed out yet, but the idea of sleep time compute, the idea of going after anything that requires lots of computation and search in particular, we think are really productive things to mine.
Not there yet, though.
Cat Tang: Yeah. What would you say are Sail’s ideal workflows?
Neil Movva: So we love deep research, hinted at it, just now. We think that deep research is one of the problems that scales best with compute. Search is a verifiable problem in a lot of cases. It’s easier to verify that you have a solution that matches the predicate that the user gave you than it is to sample that on your first try.
And so we’ve done a bunch of work on benchmarks like BrowseComp to show that you can, through compute scaling, specifically parallel compute scaling, you can launch many subagents to cover a massive haystack and find really, really specific needles in that massive haystack all through parallel intelligence. And so we’re really excited about test time compute scaling over hundreds of agents working on the same kind of high-level problem being broken down into subagents and converging again.
GPU supply and a network of modular data centers
Cat Tang: Very cool. I also wanted to shift gears a little bit and talk about compute just generally, compute capacity. So I think one big question that everybody has is how are you getting more compute onto the platform just to even start off with? And what does that distribution look like today between not necessarily who your providers are, but what the shape of your providers are and how you end up working with those relationships when the market is just so, so supply-constricted?
Neil Movva: Yeah. Well, we very much think that inference as a category is about supply.
There is not really a trick there. You have to just be more diligent on supply and find some alternative ways to get supply. That is what’s going to make winners in the inference space.
I will start by talking about the obvious market, which is the market for NVIDIA chips. NVIDIA is still the best chip manufacturer in the space. They are absolutely the most scaled. They have the best software out of the box to serve language models and really all AI workloads at this point. And to challenge them requires substantial, I guess, you have to be very careful on what regime you want to challenge NVIDIA. They are extremely good at what they do.
That said, the problem with NVIDIA is that even as good as they are and as scaled as they are, they are still very finite on supply. We cannot really double the number of NVIDIA chips coming online in the next year or two or maybe even three. It is hard to imagine that we can outbid the most well capitalized players in the market for the finite NVIDIA supply. And I am referring to here, OpenAI and Anthropic. I think the hyperscalers and the big labs are going to continue to buy as much NVIDIA supply as they possibly can.
Now NVIDIA is strategic about this. I think they are careful not to let one or a few small monopolistic kind of providers soak up all the supply. They understand that their supply is actually kind of the lifeblood for startups and anyone who wants to enter the AI market really. And so I think they are doing their best to balance supply.
But at the same time, the reality is that Anthropic makes a lot more money per GPU hour than any open source provider does. They seem to be willing to pay for a Blackwell hour, B300 hour. They are willing to pay something like $15 to $20 per hour. And if you compare that to an open source, providers can make or provide. It seems like it is hard to make a market above $10. And so given all that, we have to find some way to explore alternatives that are under explored.
And for us, that has meant alternative NVIDIA chips. We are really excited about one of their chips called the RTX Pro Blackwell, which is a consumer chip that was kind of repurposed to serve our use cases. We are very good at using that chip. I think we have written some of the best kernels for what NVIDIA refers to internally as SM120. Slightly different chip, different memory technology. So we sidestep the HBM shortage there that uses GDDR instead of HBM. Simpler packaging, more abundant to the market.
It’s great. It’s a really good chip. But it has some quirks that are not well discussed or explored in the open source inference engines or even by NVIDIA themselves. So lots of room for us to do better on that kind of chip.
And of course, there’s alternative vendors who are doing a really good job of playing in the regimes that are not totally dominated by NVIDIA. And so again, there’s these like air cooled PCIe cards that can be really well done by AMD, Intel, maybe some others who are coming to market. Qualcomm has some ideas here. And they just sidestep the bottlenecks that NVIDIA would run into as well. And so most importantly, that’s memory today.
You cannot be using HBM and have a good time.
Cat Tang: Yeah. How do you think about tackling this problem from the business standpoint? So one component is picking chips, but I don’t know if you guys are really in the data centers choosing the chips. It’s more high level than that, right?
Neil Movva: Yeah. So today we are building our first data centers, but that is a long and arduous process. We have a lot of respect for the problem. So we are taking our time. We will have this done. We have some test machines already, and they will be online by the end of the year.
And we are serving on some of these test systems already. But to really scale this play will be well into next year. On the other hand, though, I think we think the data center design is going to evolve quite a bit as well. Currently, the optimal thing to do if you have access to NVIDIA chips is to put them into a large monolithic data center. Maybe you want to put 10,000 chips in the data center. That’s something like 20 megawatts. And you should try really hard to go find 20 contiguous megawatts in one facility that’s high reliability.
And you should rack those chips and collect profit. It is a very good thing to do if you have those chips and you have access to that kind of powered shell capacity.
If you don’t have that, well, maybe you should try to do something a little bit more scrappy. We like the idea of small modular data centers. We think that the future for inference is that we don’t need to have large contiguous supply. We are very happy with small data centers that are sub-megawatt in scale, maybe just a few hundred chips, maybe not even a hundred chips. And so if that’s the future of inference, maybe we should work with many small pools of compute instead of one large monolithic compute pool.
Of course, any distributed systems engineer who just heard what I said would be very stressed about the overhead of managing so many small pools of compute, they’re all going to fail at different times. They’re very hard to kind of, I guess, like just even manage from a Kubernetes or similar perspective all these small, small pools of compute. But we think that’s the kind of trade off that we should make. We should be more aggressive in that direction because it is the area that is least likely to be explored or least likely to be easy for other players to do well.
That is where we should build our IP.
Cat Tang: It does feel like there’s more pressure to build these modular data centers even today. I mean, I think it was SemiAnalysis that said by the end of 2028, they estimate that about 30% of data center capacity in the US is going to be for modular. I think Nvidia right now is actively piloting some modular sites between like 5 to 20 megawatts. It depends on what your definition of mini is. I don’t think they’re thinking about one megawatt just yet, but it does seem like an intuitive direction to push on. Why are people not pushing even harder?
If I’m trying to build a data center today, it’s first very hard to secure capital for that and a smaller pool of capital is easier to secure. It’s even harder, arguably, to find land and to find power. And so the more you scale it down, if somebody is able to solve the distributed systems question, which it seems like you’re tackling and eventually, I think anyone who’s serving compute will think about tackling, why is this not an even bigger market?
