2026
Block Parallelism for Efficient Distributed Long-Context Diffusion Language Model Training
Submitted to ICLR 2027
Machine learning systems + architecture
Parth Shroff
I work on efficient machine learning systems, spanning model architecture, distributed execution, and hardware-software co-design.
I completed an M.S. in Electrical & Computer Engineering (AI & Software/Hardware Systems) at Stanford, where my research spanned multimodal LLM inference and token compression, relational deep learning, and efficient distributed training and inference for diffusion language models. At UT Austin, I earned B.S. degrees in Computer Science, Electrical and Computer Engineering, and Mathematics as a Turing Scholar and Dean's Scholar, graduating with University Honors and CNS Distinction in Research.
My research spans efficient model training and inference, distributed and long-context training, diffusion language models, multimodal token compression, relational deep learning, quantization, K/V-cache optimization, GPU memory systems, computer architecture, and hardware-software co-design. I currently work at NVIDIA.
Recent
- Released our work on Block Parallelism for efficient distributed long-context diffusion language model training.
- Our hybrid block diffusion language model work was accepted to the COLM 2026 NonAR-LM Workshop.
- RelBench v2 appeared at the ICLR DATA-FM Workshop.
- Presented token sequence compression at the CVPR eLVM Workshop.
01
Publications
Selected papers. My name is shown in bold.
2026
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
COLM '26 NonAR-LM Workshop · Submitted to ICLR 2027
2026
RelBench v2: A Large-Scale Benchmark and Repository for Relational Data
ICLR 2026 DATA-FM Workshop
2025
Token Sequence Compression for Efficient Multimodal Computing
CVPR 2025 Workshop on Efficient Large Vision Models
2022
The Virtual Quantum Optics Laboratory
IEEE International Conference on Quantum Computing and Engineering · Best Paper
2022
Machine Learning Techniques for Improved Non-Ideal Non-Volatile Memory Modeling
Government Microcircuit Applications and Critical Technology Conference (GOMACTech)
Conference paper
02
Background
Education, selected experience, and teaching.
2024–2026
Stanford University
M.S. in Electrical and Computer Engineering
AI and Software/Hardware Systems · Research focus: Multimodal LLM inference · GPA: 4.0 / 4.0
Advisors: Jure Leskovec, Azalia Mirhoseini, Thierry Tambe, and Pavlo Molchanov
2020–2024
The University of Texas at Austin
B.S. in Computer Science · B.S. in Electrical and Computer Engineering · B.S. in Mathematics
Turing Scholars Honors Program · Dean's Scholars Honors Program · University Honors · CNS Distinction in Research
GPA: 4.0 / 4.0
Selected experience
Summer 2024
Summer 2025
Present
Summer 2025
Present
NVIDIA
GPU Design Architecture Intern
GPU MMU, cache, and memory-subsystem design for AI training and inference, graphics, and HPC workloads.
Fall 2025
AMD
Deep Learning Quantization Research Engineer Intern
Head-level low-precision K/V-cache compression for open-source mixture-of-experts models in Quark.
Summer 2023
Apple
CPU Design Verification Intern
Generated load/store and memory-subsystem test sequences for ARM-based Apple SoCs.
Summer 2022
Cisco
Machine Learning Engineer
Built machine-learning and automation tools for network-change validation and project cost prediction.
2019–2021
Applied Research Laboratories
Researcher
Developed the Virtual Quantum Optics Laboratory for quantum-computing and communication experiments.
03
Let's talk research.
I'm always interested in efficient model architectures, training and inference systems, and hardware-software co-design.
pshroff@cs.stanford.edu