The a16z Show · Thursday, August 6, 2026
Simon Mo, co-founder and CEO of Infrac, highlights the evolution of open-source AI models, particularly large language models (LLMs), into critical infrastructure. He explains that the computational intensity and speed requirements for serving LLMs on accelerators like GPUs necessitate specialized engineering.
“Serving large language models is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs and it is a computationally intensive process that will require a lot of engineering and ensuring that for each request, user can see the LLM's response quickly and efficiently.”
“So this typically means we need to handle differences in input distribution, how long each request is, output distribution, which is non-deterministic, and batching and scheduling a lot more in the core of the inference engine.”