Careful Scaling in Academia

Science part: what it is and why we want it

There are many interesting lectures or notes on what a scaling ladder is and you may find details there [1][2]. Here I just want to briefly mention it — given a set of at least 4 exp logs with different compute budgets and from the same model family, you fit a curve

This power-law relationship between loss and compute is studied in neural scaling laws [6]. This curve can help do two things — 1. predict the larger scale perf that you cannot afford (or verify their perf if you can); 2. compare two model families’ scaling ladders to see which actually works better. This style of comparison gives some 不烧心 (peace of mind) before burning chips, cuz the training compute of any arch can be calculated in theory (or sometimes plus a small run for MFU estimation if you want the actual wall time as the x axis).

Inspired by Prof. Tatsunori Hashimoto’s Deep Learning Alchemy course at Stanford [3], I came up with another academia view on why we want it:

Scaling ladder as a structured prediction objective for PhD training

Let’s now think of this from an inverse view — say, instead of you training neural networks, actually the training experiment logs are training your brain.

Your exp logs give you the input (data, arch config, compute) and the label (model perf) and your PhD career is minimizing the objective like equation 2. However, model perf alone is not that learnable (i.e. it may not have a distribution) while data, arch config (hparams) may have tons of combinations to try.

Here, both the input and prediction target change. Instead of the free-form data config, the scaling ladder enforces a fixed TPP budget; on the arch config side, it should also follow constraints like fixed sparsity, fixed aspect ratio, etc. The prediction target, on the contrary, is reduced from O(N) scaling data points to 3 variables — A, α, and L_∞. This design augments the training data by strengthening correlations between data points, while reducing the complexity of the prediction. Too good. I can’t imagine what my panda would be like if he were trained with this kind of training data.

Bitter lessons to actually run a scaling ladder exp

To actually apply a scaling ladder in your research, you may have to instantiate two things: the x-axis and the y-axis, where the x-axis is usually FLOPs while the y-axis is quite tricky if you are not working on language modeling. My advisor is a vision guy and was a bit reluctant to switch to language but was convinced by my evidence — yea, never try to run a scaling ladder on 300-epoch ImageNet with classification acc as the y-axis. It’s miserable. The following part would presume that you are working on language modeling (or at least using NTP to train your visual models, or you already invented a perfect y-axis for your domain).

Configuration

Once you’ve settled on your y-axes, the next question is how to pick your data points. Common choice is distributing the model sizes roughly by 1x, 2x, 4x, 8x to make them land beautifully on a scaling curve (it’s natural for a power law). Second, make sure your model is not undertrained. Chinchilla suggests roughly 20 tokens per parameter as a compute-optimal rule of thumb for dense models under its training setup [7]. For MoE, the compute-optimal tokens per active parameter depend on model scale and the expert configuration [8]. But yea, as you are in academia, don’t choose a too large one. Nestor picked 50 TPP with the denominator as the active number of params [5].

During setting the model archs, you may have another concern that makes you 挠头: arch variants often change the actual FLOPs of the model. Say if you want to compare arch A and arch B where B has a slight modification i.e. on the attention part, sparse attention reduces the FLOPs. Although a scaling ladder does not require A and B to have matched FLOPs point by point, it's still worth thinking whether you should compensate for the drop by adding more dims / more layers / more heads, etc. My personal take is, yes, but still make sure your tuned arch has slightly smaller FLOPs than the baseline, i.e. 0.95x-0.99x. Under this setting, direct perf comparison on this data point would be meaningful - i.e. you can 不烧心地 promote B if it achieves on par or slightly better perf than A without running the full scaling ladder.

MFU

After configuring everything, you are going to launch your real training exps on GPUs. Now it’s agent time and you can just prompt “go f**king run the exps and maximize MFU.” But this alone usually doesn’t work well — an example is Nestor’s native kernels were beaten by Twitter folks easily [4]. Here are two lessons from my suffering:

Parallelism is scale-dependent. My exp logs documented the following fun stuff. In one 200M scale exp of Nestor, optimizer-state sharding made the optimizer update slower; while things flipped at 1B, where the sharding helped unlock a larger per-GPU microbatch. A general takeaway is, fancy sharding helps MFU on large models but for smaller ones, you have to tolerate the number and keep reading.

Pay attention to the actual computation per step. GPUs generally run large, regular matrix multiplications more efficiently than many small ones [9]. Splitting the same arithmetic into lots of tiny matmuls can underutilize the GPU and introduce launch bubbles between kernels. MoE and Nestor are two perfect examples here — let’s first forget about the load balancing issues, MoE slices the FFN matrices, and Nestor in addition slices the attention projections. This makes 3 equal-FLOPs models have an obvious MFU ranking: dense > MoE > Nestor, though advanced infra magic may help close the gap especially on larger scales. Beyond the compute itself, communication can bite you too. In an earlier multi-node run of Nestor, gradients were supposed to be synchronized only once per optimizer step. But after torch.compile wrapped the model, the helper function failed to recognize the DDP instance and silently skipped no_sync, so gradients were actually synchronized after every microbatch. Meanwhile, the network configuration funneled cross-node traffic through a single network port.

All the above experiences came from communicating with agents. Though Claude may have way stronger infra skills than me, I need to nudge it forward if I find the MFU value suspicious or the decomposed actual computation suspicious.

Bottleneck may not be chips' compute

Bottleneck may not be chips especially for vision / robot folks. it maybe data loading, simulator rendering, etc; for RL guys, it maybe the speed of inference engine, RL algorithms, distribution of # training enginers and # inference engines. My panda 信誓旦旦 spoke "I cant run my exps without 32 h200s cuz it would be too slow" but the actual hidden reason for slowing is not the number of chips - used to random shuffle every time for each data sample and face some filelock issues when reading video data.

Checklist before asking for chips

Should you keep working on this project?

Let's propose a metric:

Your goal should be optimizing this metric instead of optimizing impact alone while overlooking the burnt GPU hrs. For a rough compute-cost illustration, use the following H200-equivalent estimates for the experiments in the papers, assuming MFU 20%:

• ResNet: λ_ice × 210 H200 GPU hrs.
• Transformer: λ_ice × 70 H200 GPU hrs.

These are illustrative estimates, not measured totals for the original research projects.

What's your smallest exp GPU hr requirement?

Say your group has K GPUs with m active users. Your fair share is K/m. A reasonable smallest exp should take ≤ K/m × 1/4 GPUs and finish within one day, leaving you room for enough sweeping. From there, follow a strict promotion rule: escalate part of your successful exps to 1/2 × K/m GPUs → K/m GPUs → ask your advisor to use more than fair share → ask me or Yi for a temporary extra quota.

What's your current MFU number?

Is your bottleneck really the number of GPUs, or something else like what section 2 mentions?

Have you tuned your hparams to be optimal or applied μP [10]?

Have you run some smaller data points and shown your ladder beats the baseline?

Have you coordinated with group members to see if they have idle jobs that can be killed?

Acknowledgement & Conclusion

Thanks reviews and comments from Druv and friends.

References

1. Post by @purshow04.
2. Stanford CS336 — Lecture 9 slides.
3. Stanford CS312 — Deep Learning Alchemy.
4. Post by @deepaksinghml.
5. Chu, Pai, and Ma (2026). Nested Sparsity for Scalable Pretraining and Continual Learning.
6. Kaplan et al. (2020). Scaling Laws for Neural Language Models.
7. Hoffmann et al. (2022). Training Compute-Optimal Large Language Models.
8. Zhao et al. (2025). Towards a Comprehensive Scaling Law of Mixture-of-Experts. Appendix G: Compute-Optimality with Fixed Configurations.
9. NVIDIA. Matrix Multiplication Background. Section 2.3: Typical Tile Dimensions in cuBLAS and Performance.
10. Yang et al. (2022). Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer.