Nicholas Roberts PhD Student Department of Computer Sciences
University of Wisconsin-Madison
Advisor: Frederic Sala

Academic Bio

I am a Ph.D. student in CS at University of Wisconsin–Madison where I am advised by Fred Sala.

This Fall, I am interning with the Llama Generative AI team at Meta in London, where I work with Dieuwke Hupkes on language model evaluation. This past Summer, I interned at Together AI in San Francisco with Tri Dao, where I worked on hybrid language models. In Summer 2023, I interned with the Physics of AGI research group at Microsoft Research led by Sébastien Bubeck, where I also worked on language models.

Before starting my Ph.D., I had the pleasure of working with Ameet Talwalkar and Zack Lipton during my M.S. in the Machine Learning Department at Carnegie Mellon University. As an undergraduate, I was extremely fortunate to work with both Sanjoy Dasgupta and Gary Cottrell at the University of California, San Diego. Prior to all of this, I was a community college student at Fresno City College, where I was lucky enough to learn calculus, linear algebra, and C++ from Greg Jamison.

In 2023, I was named an MLCommons Rising Star. I have also been awarded the Prove AI and UnifyID AI Fellowships in 2021 and 2019, respectively.

Research Interests

My research is motivated by the need to democratize machine learning and foundation models to handle the long tail of emerging ML tasks in the sciences and beyond. In order to use these models to solve high-impact problems in the sciences, my work aims to solve two main challenges:

  1. determine what additional data to provide them and understand how it interacts with pretraining data, and
  2. automate the process of adapting them to new problems.

To address these challenges, I am focused on the intersection of data-centric ML (which aims to solve 1) and automated machine learning (AutoML; which aims to solve 2), or more concisely data-centric AutoML. As a result of these motivating challenges, my work on developing the foundations of data-centric AutoML has a focus on diverse ML tasks that are far afield from standard ML domains. These often include problems related to solving PDEs, protein folding, climate modeling, and beyond.

Email: nick11roberts [at] cs [dot] wisc [dot] edu
Office: CS Dept. 5384, 1210 W Dayton St, Madison, WI 53706

Fresh off the Press

  • Pretrained Hybrids with MAD Skills
    Nicholas Roberts, Samuel Guo, Zhiqi Gao, Satya Sai Srinath Namburi GNVV, Sonia Cromp, Chengjun Wu, Chengyu Duan, Frederic Sala.
    ICML 2024 Long-Context Foundation Models (LCFM) Workshop.
    ICML 2024 Next Generation of Sequence Modeling Architectures (NGSM) Workshop.
    ICML 2024 Efficient Systems for Foundation Models (ES-FoMo) Workshop.
    ICML 2024 Workshop on Foundation Models in the Wild.
    [arXiv]

  • MoRe Fine-Tuning with 10x Fewer Parameters
    Wenxuan Tan, Nicholas Roberts, Tzu-Heng Huang, Jitian Zhao, John Cooper, Samuel Guo, Chengyu Duan, Frederic Sala.
    ICML 2024 Efficient Systems for Foundation Models (ES-FoMo) Workshop.
    ICML 2024 Workshop on Foundation Models in the Wild.

Conference Publications

Journal Publications

  • Small Molecule Accurate Recognition Technology (SMART) to Enhance Natural Products Research
    Chen Zhang*, Yerlan Idelbayev*, Nicholas Roberts, Yiwen Tao, Yashwanth Nannapaneni, Brendan M. Duggan, Jie Min, Eugene C. Lin, Erik C. Gerwick, Garrison W. Cottrell, William H. Gerwick.
    Scientific Reports 2017.
    [Paper]

    Poster: Small Molecule Accurate Recognition Technology (SMART): A Digital Frontier to Reshape Natural Product Research
    Chen Zhang*, Yerlan Idelbayev*, Nicholas Roberts (presenter), Yiwen Tao, Yashwanth Nannapaneni, Brendan M. Duggan, Jie Min, Eugene C. Lin, Erik C. Gerwick, Garrison W. Cottrell, William H. Gerwick.
    Best Spotlight Presentation Award: Applied Machine Learning Days 2018.

Workshop Publications and Preprints

  • AutoML for Climate Change: A Call to Action
    Renbo Tu, Nicholas Roberts, Vishak Prasad, Sibasis Nayak, Paarth Jain, Frederic Sala, Ganesh Ramakrishnan, Ameet Talwalkar, Willie Neiswanger, Colin White.
    NeurIPS 2022 Tackling Climate Change with Machine Learning Workshop.
    [arXiv]

University of Wisconsin–Madison

Ph.D. Computer Science
Mathematics minor
August 2021 - Present

Carnegie Mellon University

M.S. Machine Learning
August 2019 - May 2021
  • MSML Student Committee Leader
  • MSML Admissions Committee Member

University of California, San Diego

B.S. Computer Science
Mathematics minor
CSE Honors Program
September 2015 - March 2019
Magna Cum Laude
CSE Highest Distinction

Fresno City College

Computer Science
Leon S. Peters Honors Program
August 2013 - May 2015
  • Tutor for CIT 65 (Android Application Development)
  • Mathematics Tutor
  • Computer Science Tutor
  • President/Founder, Google Developer Group Fresno City College
  • Treasurer, Science and Engineering Club

Meta AI

research scientist intern
(London, UK)
  • Incoming (Fall 2024)

Together AI

research intern
(San Francisco, CA)
  • Incoming (Summer 2024)

Microsoft Research

research intern
(Redmond, WA)
  • Physics of AGI research group led by Sébastien Bubeck
  • Developed activation function search techniques for large-scale LLM pretraining
  • Developed learning curve extrapolation techniques to ablate architectural choices in transformers
  • Technologies used: Python, PyTorch, Hugging Face

Amazon AI

applied scientist intern
(Seattle, WA)
  • AWS Transcribe research group led by Katrin Kirchhoff
  • Researched and developed methods for hypothesis rescoring in ASR systems using neural language modeling
  • Identified areas for improvement in many existing ASR systems when recognizing rare or zero shot entities
  • Technologies used: Python, PyTorch, RWTH ASR, Kaldi, AWS

UnifyID

ai fellow
machine learner intern
(Redwood City, CA)
  • UnifyID research lab fearlessly led by Vinay Uday Prabhu
  • Researched various ways in which research from network neuroscience could be applied to deep learning
  • Developed a novel model extraction attack against deep learning models for computer vision using just noise inputs
  • Technologies used: Python, Keras, PyTorch, TensorFlow, MATLAB, AWS

Intuit

software engineering intern
(Mountain View, CA)
  • Intuit Technology Futures research group
  • Researched and implemented a novel deep learning model for controllable text generation as a service within Intuit
  • Developed a system for proposing alternative candidate sentences for Intuit content writers using deep learning
  • Investigated the use of dynamic topic models for customer support tickets to gain actionable insights over time
  • Technologies used: Python, PyTorch, TensorFlow, Gensim, Keras

Altum

applied scientist intern
(La Jolla, CA)
  • Developed language model to extract NLP features from text data regarding cryptocurrency trading
  • Investigated unsupervised learning techniques for extracting sentiment data in real time from online forums
  • Technologies used: Python, PyTorch

Teradata

software engineering intern
(San Diego, CA)
  • Developed open source Spark-Teradata connector forked from Databricks’ connector for AWS Redshift in Scala
  • Designed and implemented Teradata stored procedures in Java to mimic Redshift’s UNLOAD and COPY using S3
  • Improved training methodology and architecture of deep learning time series model used internally
  • Implemented system for updating the time series dataset and fine tuning the deep learning model
  • Technologies used: Scala, Java, Maven, Teradata SQL, AWS, Tensorflow, Flask

The Comeback Community

volunteer full stack developer
(remote/Fresno, CA)
  • Developed site in Go, gohtml, and CSS on Google App Engine
  • Mentored new developers in web development
  • Technologies used: Go, Google App Engine, gohtml, HTML5, CSS3, JavaScript

Skqrl

software engineering intern
(La Jolla, CA)
  • Developed web crawler to compile needfinding and product data using Scrapy and Selenium
  • Designed and implemented an extensible product search solution designed to handle future user search needs
  • Technologies used: Python, Scrapy, Selenium, Django, MySQL, JavaScript

ModSpot

software engineering intern
(remote)
  • Implemented new user account, edit profile, and login designs in Objective-C for iOS application
  • Refactored analytics code for gathering statistics on app usage, helping designers make more informed choices
  • Technologies used: Objective-C, Cocoa Touch, Flurry Analytics

Extracurricular interests:

  • Aspiring triathlete
  • Sailing
  • Pottery
  • Guitar
  • Interior design
  • A budding interest in plants
  • Longboard construction and woodworking

Extra-Extracurricular interests:

Credentials:







Photograph of me shredding, circa 2009:

Mixtape:

1Don’t forget to check out Poolsuite FM: the ultra-summer music player for the Macintosh Computer; transporting you to a virtual vacation where the sun never sets.