Tools

TabPFN vs XGBoost: 14–0 Sweep and the End of Default Tuning

A training-free tabular model just beat tuned XGBoost across 14 datasets. The default-first choice for tabular ML may no longer be what it was.

Font size

When I first landed on this blog post from efraingaray.com, I assumed it was another benchmark-grabbing paper. After reading through it, the picture turned out to be more nuanced than either the hype or the dismissals suggest.

The 14–0 Scorecard

Here is what the benchmark actually reports:

  • TabPFN beat XGBoost on 14 public tabular datasets
  • TabICL also went 14/14
  • The XGBoost baseline was not raw: it used hyperparameter tuning (grid search plus cross-validation)

That last point matters. The XGBoost opponent here was running at near-peak performance, not some default-out-of-the-box configuration an intern might ship.

Source: efraingaray.com/en/blog/tabpfn-vs-xgboost/. The post hit the Hacker News front page in late August.

What "No Training" Actually Means

The classic ML workflow splits training from inference: fit on data, then predict on new samples. TabPFN and TabICL take a different path, the same in-context learning (ICL) route that powers modern LLMs:

  1. Feed training samples directly into the model as context (a prompt-like input)
  2. The model emits predictions without updating any weights
  3. The boundary between training and inference dissolves

TabPFN is a Transformer trained on synthetic prior data; TabICL is an ICL architecture purpose-built for tabular inputs. Both work on the same principle: the model has seen so many tables during pretraining that it can reason over a fresh dataset at inference time.

The API is almost suspiciously simple:

from tabpfn import TabPFNClassifier

clf = TabPFNClassifier()
clf.fit(X_train, y_train)  # internally just caches the data
y_pred = clf.predict(X_test)

Note what fit is not doing: no gradient updates. It stores the training set so the model can use it as context during predict. From the user's perspective, training cost is zero, paid entirely during pretraining.

Why XGBoost Lost This Round

XGBoost ruled tabular ML for a decade on the strength of two levers:

  1. Feature engineering, manual cross features, target encoding, binning
  2. Hyperparameter tuning, sweeping learning_rate, max_depth, subsample and friends

ICL models take both levers off the table:

  • Feature engineering gets absorbed at pretraining. TabPFN has already internalized the patterns that humans used to hand-craft.
  • Tuning gets replaced by zero-shot generalization. No learning rate to pick, no overfitting to guard against. The model has effectively "seen" the kinds of distributions it will face.

The analogy is a student who has already drilled on 10,000 practice exams. You don't need to tutor them on test day.

Underneath, the assumption shifted. XGBoost assumes train and test come from the same distribution and searches for optimal tree splits. TabPFN-style models assume every tabular task is drawn from some prior over tabular tasks. In many business settings, the latter is closer to reality: your table did not fall out of the sky; it shares structure with countless other tables in the world.

The Catch: Inference-Time Cost

One issue the blog and most write-ups underplay: inference cost.

Under ICL, every prediction requires passing the entire training set as context. For N=1,000 this is fine. For N=50,000 and beyond:

  • Latency scales linearly with training set size
  • GPU memory grows with context length
  • Batched inference means duplicating the context for each batch

A rough comparison:

ScenarioXGBoostTabPFN
Inference on 100k rowsMillisecondsSeconds to tens of seconds
Memory footprintTens of MBGrows with context, can hit several GB
Cold startMinutes of trainingReady immediately
Incremental learningJust refitReplay the full dataset

So the deployment envelope is fairly clear:

  • Small to mid-size datasets (<5k rows): TabPFN is almost a no-brainer
  • Mid-size (5k–50k rows): depends on whether the inference latency is acceptable
  • Large-scale with strict latency budgets: XGBoost/LightGBM still win

There is also an ecosystem gap. SHAP values, feature importance, partial dependence plots: on TabPFN-style models these either do not exist or are not mature. "Accurate model" is only a third of what production needs. Explainability and debuggability are the other two-thirds.

What This Really Signals

The headline is not "TabPFN beats XGBoost."

It is the first time in a decade that XGBoost is not the default answer for a meaningful slice of tabular problems. For ten years, every Kaggle competition and every business tabular project started with XGBoost (or LightGBM or CatBoost) not because it was always best, but because it was the safest choice. TabPFN-style models are now stealing that "safest default" crown, at least for small and mid-size data.

A few honest caveats worth flagging:

  1. Sample bias. Which 14 datasets? OpenML's mid-size catalog leans toward ICL-friendly problems with clean distributions and limited feature-engineering headroom. In heavily heterogeneous, feature-engineered domains (credit risk, recommendation), the result might not hold.
  2. Single source of evidence. Today this is one independent blog's experiment. The author ran an honest comparison, no cherry-picked XGBoost config, but the claim still needs cross-checking against the TabPFN and TabICL papers and third-party benchmarks like TabArena or PMLB.
  3. Engineering gap. A model that works in a paper is still far from production. SHAP, drift monitoring, A/B testing frameworks: XGBoost's ecosystem took ten years to assemble. The new models are starting from zero.

My read on what happens next: through 2025–2026, small and mid-size data teams will adopt TabPFN/TabICL heavily during the prototyping phase, skipping the "train a baseline first" step entirely. In production, especially pipelines handling millions of rows per day, XGBoost-family models will remain the default.

Signals worth watching: the TabPFN team's work on larger-dataset support (the v2 roadmap for 10k+ rows), and whether TabArena delivers the kind of unified benchmark that finally gives tabular ML its "ImageNet moment."


Sources:

Source: https://efraingaray.com/en/blog/tabpfn-vs-xgboost/