Controllable and efficient diffusion models

Controllable and efficient diffusion models

Abstract: State-of-the-art diffusion models can generate visually compelling images and videos, but they require copious amounts of annotated training data and compute time, available. At the same time controlling the output to particular conditions, descriptions, scene and global context etc. without extensive retraining, becomes very important as visual generation applications become democratized. In this talk I will go over our group’s efforts to make diffusion models more controllable and efficient. First, Ι will discuss how to align video diffusion models more closely semantically with the input controls and also produce content that more accurately follows physics constraints using post-training. Then, I will introduce a method that guides diffusion models to generate fewer semantically implausible samples (hallucinations) at inference time. Finally, I will review our line of work that uses self-supervised representations as conditioning signals for diffusion models in digital histopathology for large image generation, to provide better, more direct control over the generated images. Such large image generation has led us to introduce an algorithm for test-time, constrained sampling that does not require any backpropagation and thus more efficiently produces results as good as previous optimization methods. Finally, I will talk about two ideas on optimizing the inference of large image and video diffusion models for efficiency: how to focus computation in important regions only by merging tokens efficiently while preserving quality and how to effectively perform mixed-resolution inference to decrease inference time by appropriately adapting RoPE positional embeddings.

About the speaker: Dimitris Samaras received the diploma degree in computer science and engineering from the University of Patras, in 1992, the MSc degree from Northeastern University, in 1994, and the PhD degree from the University of Pennsylvania, in 2001. He is a SUNY Empire Innovation Professor of Computer Science with Stony Brook University, where he directs the Computer Vision Lab. His research interests include human behavior analysis, generative models, illumination modeling and estimation for recognition and graphics, and biomedical image analysis. He has co-authored over 200 peer-reviewed research articles in Computer Vision, Machine Learning, Computer Graphics and Medical Imaging conferences and journals. He was Program Chair of CVPR 2022, General Chair of ACCV 2024 and ICCV 2025, and is frequent Area Chair in Computer Vision and Machine Learning conferences.

Join us on Zoom:

https://naverlabs.zoom.us/j/93663487587?pwd=GEtb99hnoCnB3A9xptSKPWLsxGWonb.1
Meeting ID: 936 6348 7587
Passcode: 606077