DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations

Krishna Sri Ipsit Mantri1, Carola-Bibiane Schönlieb2, Bruno Ribeiro1, Chaim Baskin3, Moshe Eliasof2
1Purdue University, 2University of Cambridge, 3Ben-Gurion University of the Negev
CVPR 2025
Paper arXiv Code Data
Interpolation end reference image.

DiTASK achieves state-of-the-art performance on PASCAL MTL, using 75% fewer trainable parameters compared to other PEFT methods.

Abstract

Pre-trained Vision Transformers now serve as powerful tools for computer vision. Yet, efficiently adapting them for multiple tasks remains a challenge that arises from the need to modify the rich hidden representations encoded by the learned weight matrices, without inducing interference between tasks. Current parameter-efficient methods like LoRA, which apply low-rank updates, force tasks to compete within constrained subspaces, ultimately degrading performance. We introduce DiTASK, a novel Diffeomorphic Multi-Task Fine-Tuning approach that maintains pre-trained representations by preserving weight matrix singular vectors, while enabling task-specific adaptations through neural diffeomorphic transformations of the singular values. By following this approach, DiTASK enables both shared and task-specific feature modulations with minimal added parameters. Our theoretical analysis shows that DiTASK achieves full-rank updates during optimization, preserving the geometric structure of pre-trained features, and establishing a new paradigm for efficient multi-task learning (MTL). Our experiments on PASCAL MTL and NYUD show that DiTASK achieves state-of-the-art performance across four dense prediction tasks, using 75% fewer parameters than existing methods.

Method

Interpolation end reference image. Interpolation end reference image.

Results

Interpolation end reference image.

Qualitative Comparison

Semantic Segmentation

Depth Estimation

BibTeX


      @misc{mantri2025ditaskmultitaskfinetuningdiffeomorphic,
        title={DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformations}, 
        author={Krishna Sri Ipsit Mantri and Carola-Bibiane Schönlieb and Bruno Ribeiro and Chaim Baskin and Moshe Eliasof},
        year={2025},
        eprint={2502.06029},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2502.06029}, 
  }