Breakthrough in Test-Time Compute

I came across two interesting papers recently on scaling laws in AI and wanted to share a summary. Here are the key takeaways:

Scaling LLM Test-Time Compute Two papers looked at how to scale up test-time compute for LLMs:

1. Simple strategies like weighted voting keep improving as you scale up test-time compute
2. There's a regime where recognizing good solutions becomes the bottleneck, not generating them
3. The ratio of test-time to training-time compute is increasing
4. Batch size 1 inference may become less important; parallel generations could become standard
5. Tree search with Process Reward Models is emerging as a legitimate strategy
6. We may see more compound systems with separate proposer and verifier modules

Finetuning Effects A study on finetuning 1-16B param LLMs found:

1. Model size matters more than finetuning dataset size
2. Pretraining dataset size matters more than finetuning dataset size
3. Finetuning dataset size matters way more than params added by PEFT methods
4. Power law curves fit the results well, but coefficients vary by method/task