Case Studies

Baseten's Series F: $13B or $1.3B? Inside the Inference Engineering Race

A Latent Space podcast description lists Baseten's Series F valuation as $13B, which is almost certainly a typo for $1.3B. The real story isn't the number—it's how crowded the inference engineering market has become.

Font size

Training a model is the easy part. Open weights for Llama 4, Flux, and dozens of others sit on HuggingFace for anyone to download. Running a 70B model under 50ms with single-digit-cent inference costs is a different business entirely.

A recent episode description for the Latent Space "Inference Engineering Masterclass" podcast lists Baseten's Series F valuation as $13B. That's suspiciously high. $1.3B is the range you'd expect for a top-tier AI infrastructure company at this stage. The discrepancy could be a typo, or it could be a deliberate attention grab.

Either way, the headline number isn't the point. The point is that inference engineering is now a more crowded race than training.

Sorting Out the Number

The podcast description lists the Series F valuation at $13B. For context:

  • Together AI's Series C at the end of 2024 came in around $1.25B.
  • Anyscale was valued near $2.5B after its 2022 Series C; nothing disclosed since.
  • CoreWeave's final pre-IPO round valued it around $19B—but that's NVIDIA H100-cluster scale.

Baseten sells inference PaaS, not GPU cloud rental. It's in the same tier as Together and Fireworks. A $13B valuation would put it on par with CoreWeave, which makes no sense for a five-year-old company focused on inference optimization.

The most likely correct figure is $1.3B—possibly confused between the Series F raise amount and the post-money valuation. The description says "valuation," but the two are easy to mix up.

Regardless of which number is accurate, the funding pace itself signals something: inference is where the money is going.

At the start of 2024, everyone was competing on training clusters and scaling laws. The bottleneck on the training side now is data, alignment, and post-training. The bottleneck on the inference side is something else entirely: latency, cost, and throughput.

Capital is flowing toward inference.

What Inference Engineering Actually Is: Don't Call It "Deployment"

Most people understand LLM productionization as: train the model, then deploy it. That's too coarse.

On the podcast, Baseten frames the inference process for both autoregressive and diffusion models as a full engineering problem spanning the entire inference stack:

  • Batching: single-request vs. batched requests, balancing latency against throughput.
  • Quantization: FP16 → INT8 → INT4, trading model size against precision loss.
  • KV Cache: caching attention key/value pairs during autoregressive inference to skip recomputation.
  • Continuous Batching: dynamically adjusting batch composition, adding new requests without waiting for all existing ones to finish.
  • Speculative Decoding: using a small model to draft, then verifying with a larger one.

Each item is its own optimization battleground. Together's spec infer, Fireworks' Attention Optimizer, vLLM's PagedAttention—all different weapons in the same war.

Take vLLM's PagedAttention as a concrete example. It manages KV cache the way operating systems manage virtual memory: through pages. For long-sequence requests, this drops memory waste from several multiples to near zero. This isn't a polish-level optimization—it's hard engineering that shows up directly in GPU utilization numbers.

# vLLM continuous batching example
from vllm import LLM, SamplingParams

llm = LLM(model="meta--70B-Instruct")
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)

# continuous batching handles request lifecycle automatically
outputs = llm.generate(["Explain KV cache in one sentence"], sampling_params)

When this code runs, vLLM automatically handles KV cache paging, continuous batching, and dynamic scheduling under the hood. Developers never touch these details—but companies like Baseten and Together have to master every layer.

Why Now: Training Bottlenecks Have Given Way to Inference Bottlenecks

In 2024, the race was over training clusters and scaling laws. GPU supply was the scarce resource. Today, H100 and H200 capacity is no longer the main constraint—if you pay, you wait a few months and get the cards.

The real constraint has moved to inference:

  • Who can run Llama 70B under 50ms?
  • Who can drive the GPU cost of a single Flux image down to a few cents?
  • Who can hold p99 latency steady under bursty traffic?

None of these problems are solved by buying more GPUs. They're solved by engineering.

Here's a counterintuitive fact: inference accounts for over 70% of marginal cost in many LLM applications. Training is a one-time expense. That makes the ROI of inference optimization more direct than training optimization.

Baseten's Moat: Not the Model, but Scheduling and Compilation

Against competitors like Together, Fireworks, and Anyscale, Baseten emphasizes the depth of its inference stack. On the podcast, Philip Kiely and Ali Taha kept returning to one word: autoscaling.

Inference autoscaling is far harder than traditional web-service autoscaling. LLM requests vary wildly in token count (10 vs. 2000 tokens), and per-request GPU time is unpredictable.

If you scale on traditional CPU utilization signals, LLM inference ends up either wildly under-provisioned or wildly over-provisioned.

Baseten's answer is a custom scheduler that makes real-time decisions based on estimated token counts, current model load, and GPU queue length. The scheduler isn't a massive codebase, but the tuning knowledge behind it represents months to years of accumulated experience.

Another moat is compilation optimization. Open-source frameworks like vLLM, TGI, and SGLang are written in PyTorch, which means they pay Python interpreter overhead and dynamic graph scheduling costs. Both Baseten and Fireworks build their own compilation layers, pushing hot paths down to CUDA kernels or Triton.

In a market full of lookalikes, technical depth is where wins and losses get decided.

What the Podcast Left Unsaid

The Masterclass format is Latent Space's long-form interview style. Philip Kiely and Ali Taha represented Baseten as engineering speakers. Masterclasses tend toward advocacy: speakers talk about what they got right and skip what they got wrong or never tried.

A few questions worth pushing on:

1. How much margin is there really in open-source model inference?

You can drive Llama 70B deployment costs down with vLLM. But pricing power is also compressed by the open ecosystem. If customers know you're running the same open-source model, you have little room to negotiate. Baseten's premium comes from "ease of use" and "SLAs"—but how much are those actually worth?

2. Do enterprise customers really pay extra for "faster"?

In demos, latency is life or death—dropping p99 from 200ms to 80ms might win the deal. Once a workload hits production, though, "faster" matters less. Nobody dies from waiting another 200ms. The window where customers pay a premium for speed may only be open during demos and POCs.

3. Are inference optimization returns diminishing?

vLLM's PagedAttention solved memory waste. Continuous batching solved GPU idle time. Where does the next order-of-magnitude improvement come from? Hardware (Blackwell, B200)? Model architecture (MoE, hybrid)? New algorithms (speculative decoding, tree attention)? If hardware is the main lever, the software moat may be shallower than it looks.

4. Why doesn't Baseten build models?

Together launched its own model fine-tuning service. Fireworks is doing fine-tuning too. Baseten sticks to infrastructure. Strategic discipline, or a missed opportunity?

None of these got airtime on the podcast. Worth following up.

The Bottom Line

The inference engineering market is more crowded than most people realize. Together, Fireworks, Anyscale, Replicate, Modal, Baseten—at least six major players. Add cloud vendors like AWS Bedrock and Google Vertex AI entering the space, and the question of how much room is left for independent PaaS players becomes a real one.

Then again: inference for open models is a durable, long-term need. As long as Llama, Flux, and similar open weights keep getting downloaded, inference infrastructure has a reason to exist. Baseten and Together are betting that inference workflows will get managed the same way databases did.

Practical advice for developers: if you're choosing an inference PaaS today, don't bet on a single vendor. Together pushes hardest on spec decoding. Fireworks has depth in Attention Optimizer. Baseten emphasizes autoscaling. Run your actual workload benchmarks. Compare p99 latency and per-token cost. Vendor podcasts are less useful than your own numbers.

Whether this judgment holds will become clear within 3-5 years.


Reference: Latent Space Podcast: Inference Engineering Masterclass with Baseten

Source: https://www.latent.space/p/inference-eng