Focuses on Cross-Entropy Loss scaling laws. I’m curious to better understand the scaling laws for RL. There’s the paper Horizon Reduction Makes RL Scalable.

Citation

(Kaplan et al., 2020)

Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361