Style Analysis of Image Diffusion LoRAs in Weight Space

Abstract:
Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning of image diffusion models, with over 100K models trained to replicate artistic styles and shared within online communities. One way to analyze styles of LoRA models is to generate image samples and apply image-based features like CLIP. However, this generation requires additional computation and prompt selection. In this work, we propose analyzing styles directly in the LoRA weight space, bypassing these limitations while achieving better performance even on unseen LoRAs. We show that LoRA weights alone serve as effective feature representations for the adapted styles. We extensively evaluate across data domains (fine-art paintings, architectural photos) and generate images of diverse prompt settings.
These evaluations confirm that our approach preserves fine-grained styles and outperforms image-based methods in clustering and retrieval of unseen LoRAs.

Speaker:
Chenxi Liu is a postdoctoral researcher in the Dynamic Graphics Project (DGP) Lab at the University of Toronto, supervised by Professor Alec Jacobson. Chenxi's research focuses on computational methods for understanding and assisting visual creation, with recent projects analyzing style-adapted LoRA models, 2D neural fields with learned discontinuities, and sketch processing. Chenxi holds a Ph.D. from the University of British Columbia, where research under the supervision of Professor Alla Sheffer bridged the gap between rough freehand sketches and precise digital representations. Chenxi has interned at Adobe Research and Disney Research and was recognized as a 2022 Rising Star by ACM’s Community Group, WiGRAPH, and a recipient of the Faculty of Arts & Science Postdoctoral Fellowship at the University of Toronto.