Publications

Fresh off the Press

• OpenThoughts-Agent: Data Recipes for Agents
  Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, Atula Tejaswi, Ryan Marten, Charlie Ruan, Tyler Griggs, Alexander Shaw, Hritik Bansal, Kelly Buchanan, Artem Gazizov, Reinhard Heckel, Chinmay Hegde, Sankalp Jajee, Daanish Khazi, Emmanouil Koukoumidis, Xiangyi Li, Hange Liu, Shlok Natarajan, Harsh Raj, Nicholas Roberts, Ethan Shen, Nishad Singhi, Michael Siu, Ashima Suvarna, Hanwen Xing, Patrick Yubeaton, Robert Zhang, Leon Chen, Xiaokun Chen, Steven Dillmann, Saadia Gabriel, Xunyi Jiang, Anurag Kashyap, Boxuan Li, Yein Park, Minh Pham, Sujay Sanghavi, Lin Shi, Yixin Wang, Zhiwei Xu, Erica Zhang, Siyan Zhao, Wanjia Zhao, Jenia Jitsev, Alex Dimakis, Benjamin Feuer, Ludwig Schmidt.
  Preprint.
  

• Test-Time Scaling Makes Overtraining Compute-Optimal
  Nicholas Roberts, Sungjun Cho, Zhiqi Gao, Tzu-Heng Huang, Albert Wu, Gabriel Orlanski, Avi Trost, Kelly Buchanan, Aws Albarghouthi, Frederic Sala.
  Preprint.
  [arXiv]

• Breakeven complexity: A new perspective on evaluating neural partial differential equation solvers
  Yijing Zhang, Nicholas Roberts, Tanya Marwah, Mikhail Khodak.
  Preprint.
  

• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
  Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, Aws Albarghouthi.
  Preprint.
  

• Pareto Optimal Code Generation
  Gabriel Orlanski, Nicholas Roberts, Aws Albarghouthi, Frederic Sala.
  Preprint.
  [arXiv]

Conference Publications

• Procedural Generation of Algorithm Discovery Tasks in Machine Learning
  Alexander Goldie, Zilin Wang, Adrian Hayler, Deepak Nathani, Edan Toledo, Aleksandra Kalisz, Ken Thampiratwong, Michael Beukman, Alistair Letcher, Shashank Reddy Chirra, Clarisse Wibault, Theo Wolf, Charles O'Neill, Uljad Berdica, Nicholas Roberts, Saeed Rahmani, Roberta Raileanu, Shimon Whiteson, Jakob Foerster.
  ICML 2026.
  [arXiv]

• Pretrained Hybrids with MAD Skills
  Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, Frederic Sala.
  COLM 2025.
  [Paper] [arXiv]

• MLGym: A New Framework and Benchmark for Advancing AI Research Agents
  Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Foerster, Yoram Bachrach, William Yang Wang, Roberta Raileanu.
  COLM 2025.
  [Paper] [arXiv]

• Compute Optimal Scaling of Skills: Knowledge vs Reasoning
  Nicholas Roberts, Niladri Chatterji, Sharan Narang, Mike Lewis, Dieuwke Hupkes.
  ACL Findings 2025.
  [Paper] [arXiv]

• Stronger Than You Think: Benchmarking Weak Supervision on Realistic Tasks
  Tianyi Zhang*, Linrong Cai*, Jeffrey Li, Nicholas Roberts, Neel Guha, Frederic Sala.
  NeurIPS 2024.
  [Paper] [arXiv]

• Geometry-Aware Adaptation for Pretrained Models
  Nicholas Roberts, Xintong Li, Dyah Adila, Sonia Cromp, Tzu-Heng Huang, Jitian Zhao, Frederic Sala.
  NeurIPS 2023.
  [Paper] [arXiv]

• Skill-it! A data-driven skills framework for understanding and training language models
  Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, Christopher Ré.
  NeurIPS 2023 (spotlight).
  [Paper] [arXiv]

• Generative Modeling Helps Weak Supervision (and Vice Versa)
  Benedikt Boecking, Nicholas Roberts, Willie Neiswanger, Stefano Ermon, Frederic Sala, Artur Dubrawski.
  ICLR 2023.
  [Paper] [arXiv] [Code]

• AutoWS-Bench-101: Benchmarking Automated Weak Supervision with 100 Labels
  Nicholas Roberts*, Xintong Li*, Tzu-Heng Huang, Dyah Adila, Spencer Schoenberg, Cheng-Yu Liu, Lauren Pick, Haotian Ma, Aws Albarghouthi, Frederic Sala.
  NeurIPS 2022.
  [Paper] [arXiv] [Code] [Blog]

• NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
  Renbo Tu*, Nicholas Roberts*, Mikhail Khodak, Junhong Shen, Frederic Sala, Ameet Talwalkar.
  NeurIPS 2022.
  [Paper] [arXiv] [Code] [Website] [Blog]

• Lifting Weak Supervision To Structured Prediction
  Harit Vishwakarma, Nicholas Roberts, Frederic Sala.
  NeurIPS 2022.
  [Paper] [arXiv]

• Universalizing Weak Supervision
  Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, Frederic Sala.
  ICLR 2022.
  [Paper] [arXiv]

• Rethinking Neural Operations for Diverse Tasks
  Nicholas Roberts*, Mikhail Khodak*, Tri Dao, Liam Li, Christopher Ré, Ameet Talwalkar.
  NeurIPS 2021.
  [Paper] [arXiv] [Code] [Software Package] [Talk]
  
  Preliminary version: Searching for Convolutions and a More Ambitious NAS
  Nicholas Roberts*, Mikhail Khodak*, Tri Dao, Liam Li, Maria-Florina Balcan, Christopher Ré, Ameet Talwalkar.
  AAAI 2021 Workshop on Learning Network Architecture During Training (plenary talk).
  

• Learning from Discriminative Feature Feedback
  Sanjoy Dasgupta, Akansha Dey, Nicholas Roberts, Sivan Sabato.
  NeurIPS 2018.
  [Paper]

Journal Publications

• Tabby: Tabular Adaptation for Language Models
  Sonia Cromp, Satya Sai Srinath Namburi GNVV, Catherine Cao, Mohammed Alkhudhayri, Samuel Guo, Nicholas Roberts, Frederic Sala
  Transactions on Machine Learning Research (TMLR) 2025.
  [Paper]

• Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  Aarohi Srivastava, ..., Nicholas Roberts (276), ..., (442 authors).
  Transactions on Machine Learning Research (TMLR) 2023 (Finalist for Outstanding Certification).
  ICLR 2025.
  [arXiv] [Code]
  

• NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
  Kaustubh D. Dhole, ..., Nicholas Roberts (85), ..., (128 authors).
  Northern European Journal of Language Technology (NEJLT) 2023.
  [arXiv] [Code]
  

• AutoML Decathlon: Diverse Tasks, Modern Methods, and Efficiency at Scale
  Nicholas Roberts*, Samuel Guo*, Cong Xu*, Ameet Talwalkar, David Lander, Lvfang Tao, Linhang Cai, Shuaicheng Niu, Jianyu Heng, Hongyang Qin, Minwen Deng, Johannes Hog, Alexander Pfefferle, Sushil Ammanaghatta Shivakumar, Arjun Krishnakumar, Yubo Wang, Rhea Sukthanker, Frank Hutter, Euxhen Hasanaj, Tien-Dung Le, Mikhail Khodak, Yuriy Nevmyvaka, Kashif Rasul, Frederic Sala, Anderson Schneider, Junhong Shen, Evan Sparks
  PMLR NeurIPS 2022 Competition Track.
  [Paper] [Website] [Submission Site] [Code] [Blog]

• Small Molecule Accurate Recognition Technology (SMART) to Enhance Natural Products Research
  Chen Zhang*, Yerlan Idelbayev*, Nicholas Roberts, Yiwen Tao, Yashwanth Nannapaneni, Brendan M. Duggan, Jie Min, Eugene C. Lin, Erik C. Gerwick, Garrison W. Cottrell, William H. Gerwick.
  Scientific Reports 2017.
  [Paper]
  
  Poster: Small Molecule Accurate Recognition Technology (SMART): A Digital Frontier to Reshape Natural Product Research
  Chen Zhang*, Yerlan Idelbayev*, Nicholas Roberts (presenter), Yiwen Tao, Yashwanth Nannapaneni, Brendan M. Duggan, Jie Min, Eugene C. Lin, Erik C. Gerwick, Garrison W. Cottrell, William H. Gerwick.
  Best Spotlight Presentation Award: Applied Machine Learning Days 2018.
  

Workshop Publications and Preprints

• R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
  Albert Ge, Tzu-Heng Huang, John Cooper, Avi Trost, Ziyi Chu, Satya Sai Srinath Namburi GNVV, Ziyang Cai, Kendall Park, Nicholas Roberts, Frederic Sala.
  ICML 2025 DIG-BUGS Workshop.
  ICML 2025 DataWorld Workshop.
  [arXiv]

• MoRe Fine-Tuning with 10x Fewer Parameters
  Wenxuan Tan, Nicholas Roberts, Tzu-Heng Huang, Jitian Zhao, John Cooper, Samuel Guo, Chengyu Duan, Frederic Sala.
  ICML 2024 Efficient Systems for Foundation Models (ES-FoMo) Workshop.
  ICML 2024 Workshop on Foundation Models in the Wild.
  [arXiv]

• Understanding Neural Architecture Search by its Architecture Parameters
  Nicholas Roberts, Yingyu Liang, Frederic Sala.
  Midwest Machine Learning Symposium 2023.
  

• ScriptoriumWS: A Code Generation Assistant for Weak Supervision
  Tzu-Heng Huang, Harit Vishwakarma, Catherine Cao, Spencer Schoenberg, Nicholas Roberts, Frederic Sala.
  ICLR 2023 Deep Learning for Code Workshop.
  

• AutoML for Climate Change: A Call to Action
  Renbo Tu, Nicholas Roberts, Vishak Prasad, Sibasis Nayak, Paarth Jain, Frederic Sala, Ganesh Ramakrishnan, Ameet Talwalkar, Willie Neiswanger, Colin White.
  NeurIPS 2022 Tackling Climate Change with Machine Learning Workshop.
  [arXiv]

• Decoding and Diversity in Machine Translation
  Nicholas Roberts, Davis Liang, Graham Neubig, Zachary C. Lipton.
  NeurIPS 2020 Resistance AI Workshop.
  [arXiv]

• A Simple Setting for Understanding Neural Architecture Search with Weight-Sharing
  Mikhail Khodak, Liam Li, Nicholas Roberts, Maria-Florina Balcan, Ameet Talwalkar.
  ICML 2020 AutoML Workshop.
  [Paper]

• Weight-Sharing Beyond Neural Architecture Search: Efficient Feature Map Selection and Federated Hyperparameter Tuning
  Mikhail Khodak*, Liam Li*, Nicholas Roberts, Maria-Florina Balcan, Ameet Talwalkar.
  MLSys 2020 On-Device Intelligence Workshop.
  [Paper]

• Deep Connectomics Networks: Neural Network Architectures Inspired by Neuronal Networks
  Nicholas Roberts, Dian Ang Yap, Vinay U. Prabhu.
  NeurIPS 2019 Real Neurons and Hidden Units Workshop.
  [arXiv]

• Using Deep Siamese Neural Networks to Speed up Natural Products Research
  Nicholas Roberts, Poornav S. Purushothama, Vishal T. Vasudevan, Siddarth Ravichandran, Chen Zhang, William H. Gerwick, Garrison W. Cottrell.
  NeurIPS 2019 workshop on Machine Learning and the Physical Sciences.
  [Paper]

• Grassmannian Packings in Neural Networks: Learning with Maximal Subspace Packings for Diversity and Anti-Sparsity
  Dian Ang Yap, Nicholas Roberts, Vinay U. Prabhu.
  NeurIPS 2019 workshop on Bayesian Deep Learning.
  NeurIPS 2019 workshop on Information Theory and Machine Learning.
  [arXiv]

• Model Weight Theft With Just Noise Inputs: The Curious Case of the Petulant Attacker
  Nicholas Roberts, Vinay U. Prabhu, Matthew McAteer.
  ICML 2019 workshop on Security and Privacy of Machine Learning (spotlight).
  [arXiv]