Daniel D. Johnson

Education

University of Toronto  |  Toronto, ON, Canada
PhD, Computer Science
Sept 2021 - Oct 2025
Dissertation: “Building Representations and Quantifying Uncertainty With Pairwise Consistency”
Advisors: David Duvenaud, Chris J. Maddison
Harvey Mudd College  |  Claremont, CA  |  GPA: 3.98
Bachelor of Science, Joint Program in Computer Science and Mathematics
Aug 2014 - May 2018
Graduated with High Distinction and Honors in Computer Science, Mathematics, and Humanities

Research Interests

  • Building a scientific understanding of language model behavior and the causal factors that influence it.
  • Understanding how model personas, character, and “drives” shape model behavior.
  • Discovering and systematically measuring novel behaviors in AI systems.
  • Ensuring AI systems behave safely and beneficially via empirical science.

Employment History

Research Scientist
Transluce  |  San Francisco, CA
Dec 2024 - present
Built infrastructure for studying and measuring AI system behaviors in open-ended domains, working toward a public science of model behavior. Co-led research into surfacing and quantifying model behavior differences in the domain of user mental health, using a consistent set of simulated users to track differences between models and over time, and estimated measurement validity via production data access partnerships with Anthropic and OpenAI. Developed techniques for eliciting rare model behaviors using importance sampling.
Research Scientist
Google DeepMind  |  Toronto, ON, Canada
Apr 2022 - Nov 2024
Conducted research on probabilistic machine learning and uncertainty quantification applied to software development and language models. Developed open-source libraries for extracting and visualizing language model activations (Penzai and Treescope) and for summarizing uncertainty in code completion systems (R-U-SURE).
Research Software Engineer
Google Research, Brain team  |  Toronto, ON, Canada
June 2021 - Apr 2022
Conducted research on probabilistic machine learning and representation learning. Prototyped methods for summarizing language model uncertainty for code completion and developer assistance tasks.
AI Resident
Google Research, Brain team  |  Montréal, QC, Canada
Oct 2019 - June 2021
Conducted research on machine learning for software static analysis, diffusion models in discrete state spaces, and programming language design.
Software Engineer  |  Perception
Cruise Automation  |  San Francisco, CA, USA
Full time: July 2018 - Sept 2019. Intern: May 2017 - Aug 2017
Developed machine learning models for self-driving cars.

Publications

Mental Health Behavior Report
Daniel D. Johnson*, Robert Friel*, Cassidy Laidlaw, Luke Hewitt, Nari Johnson, Conrad Stosz, Sarah Schwettmann
AI behavior evaluation and technical report (behaviors.transluce.org), 2026 [report]
Toward A Public Science of Model Behavior
Daniel D. Johnson, Sarah Schwettmann
Position essay (transluce.org), 2026 [essay]
Predictive Concept Decoders: Training Scalable End-to-End Interpretability Assistants
Vincent Huang, Dami Choi, Daniel D. Johnson, Sarah Schwettmann, Jacob Steinhardt
Preprint, 2025; in submission for NeurIPS 2026 [arXiv]
Surfacing Pathological Behaviors in Language Models
Neil Chowdhury, Sarah Schwettmann, Jacob Steinhardt, Daniel D. Johnson
Technical paper (transluce.org), 2025 [paper]
Eliciting Language Model Behaviors with Investigator Agents
Xiang Li, Neil Chowdhury, Daniel D. Johnson, Tatsunori Hashimoto, Percy Liang, Sarah Schwettmann, Jacob Steinhardt
ICML 2025 [arXiv]
Experts Don't Cheat: Learning What You Don't Know By Predicting Pairs
Daniel D. Johnson, Daniel Tarlow, David Duvenaud, Chris J. Maddison
ICML 2024 [arXiv]
Parallel Algebraic Effect Handlers
Ningning Xie*, Daniel D. Johnson*, Dougal Maclaurin, Adam Paszke
International Conference on Functional Programming 2024 [paper, code]
Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data
Daniel D. Johnson
ICML Workshop on Mechanistic Interpretability 2024 [arXiv, Penzai code, Treescope code]
R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents
Daniel D. Johnson, Daniel Tarlow, Christian Walder
ICML 2023 [arXiv, code]
A Density Estimation Perspective on Learning From Pairwise Human Preferences
Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, Yann Dauphin
Transactions on Machine Learning Research (2023) [arXiv]
Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions
Daniel D. Johnson, Ayoub El Hanchi, Chris J. Maddison
ICLR 2023 [arXiv]
Uncertain Simulators Don't Always Simulate Uncertain Agents
Daniel D. Johnson
Technical blog post, 2023 [post]
A Library for Representing Python Programs as Graphs for Machine Learning
David Bieber, Kensen Shi, Petros Maniatis, Charles Sutton, Vincent Hellendoorn, Daniel Johnson, Daniel Tarlow
arXiv preprint [arXiv]
Learning Generalized Gumbel-Max Causal Mechanisms
Guy Lorberbom*, Daniel D. Johnson*, Chris J. Maddison, Daniel Tarlow, Tamir Hazan
NeurIPS 2021 (spotlight) [arXiv, code]
Structured Denoising Diffusion Models In Discrete State-Spaces
Jacob Austin*, Daniel D. Johnson*, Jonathan Ho, Daniel Tarlow, Rianne van den Berg
NeurIPS 2021 [arXiv, code]
Getting to the Point. Index Sets And Parallelism-Preserving Autodiff for Pointful Array Programming
Adam Paszke, Daniel D. Johnson, David Duvenaud, Dimitrios Vytiniotis, Alexey Radul, Matthew Johnson, Jonathan Ragan-Kelley, Dougal Maclaurin
International Conference on Functional Programming 2021 [arXiv, code]
Beyond In-Place Corruption: Insertion And Deletion In Denoising Probabilistic Models
Daniel D. Johnson, Jacob Austin, Rianne van den Berg, Daniel Tarlow
2021 ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models [arXiv, code]
Learning Graph Structure With a Finite-State Automaton Layer
Daniel D. Johnson, Hugo Larochelle, Daniel Tarlow
NeurIPS 2020 (spotlight presentation); also presented at GRL+ 2020 [arXiv, talk, code]
Latent Gaussian Activity Propagation: Using Smoothness And Structure to Separate And Localize Sounds in Large Noisy Environments
Daniel D. Johnson, Daniel Gorelik, Ross E. Mawhorter, Kyle Suver, Weiqing Gu, Steven Xing, Cody Gabriel, Peter Sankhagowit
NeurIPS 2018 [pdf, poster]
Learning Graphical State Transitions
Daniel D. Johnson
ICLR 2017 (oral presentation) [pdf, talk, code, blogpost]
Geometric Realizations of the 3D Associahedron (multimedia exposition)
Satyan L. Devadoss, Daniel D. Johnson, Justin Lee, Jackson Warley
International Symposium on Computational Geometry 2017 [pdf, demo]
Learning to Create Jazz Melodies Using a Product of Experts
Daniel D. Johnson, Robert M. Keller, Nicholas Weintraut
International Conference on Computational Creativity 2017 [pdf, blogpost]
Generating Polyphonic Music With Tied-Parallel Networks
Daniel D. Johnson
EvoMusArt 2017 [pdf, code, blogpost]

Invited Talks

Experts Don’t Cheat: Learning What You Don’t Know By Predicting Pairs
ELLIS Robust LLMs Workshop – Keble College, Oxford, UK – July 2024

Honors and Awards

  • TMLR Expert Reviewer (2023, 2024)
  • NeurIPS Top 10% Reviewer (2020)
  • Computing Research Association Outstanding Undergraduate Researcher - Runner-up (2018)
  • Greever Clinic Award (Senior Capstone Project) (2018)
  • Barry Goldwater Scholar (2017)
  • Stavros Busenberg Prize for Outstanding Promise in Applied Mathematics (2017)
  • Robert James Prize for Outstanding Performance in Mathematics (2015)
  • Harvey S. Mudd Merit Award (2014-2018)
  • National Merit Scholar (2014)