  Although single-task medical image restoration (MedIR) has witnessed
remarkable success, the limited generalizability of these methods poses a
substantial obstacle to wider application. In this paper, we focus on the task
of all-in-one medical image restoration, aiming to address multiple distinct
MedIR tasks with a single universal model. Nonetheless, due to significant
differences between different MedIR tasks, training a universal model often
encounters task interference issues, where different tasks with shared
parameters may conflict with each other in the gradient update direction. This
task interference leads to deviation of the model update direction from the
optimal path, thereby affecting the model's performance. To tackle this issue,
we propose a task-adaptive routing strategy, allowing conflicting tasks to
select different network paths in spatial and channel dimensions, thereby
mitigating task interference. Experimental results demonstrate that our
proposed \textbf{A}ll-in-one \textbf{M}edical \textbf{I}mage
\textbf{R}estoration (\textbf{AMIR}) network achieves state-of-the-art
performance in three MedIR tasks: MRI super-resolution, CT denoising, and PET
synthesis, both in single-task and all-in-one settings. The code and data will
be available at
\href{https://github.com/Yaziwel/All-In-One-Medical-Image-Restoration-via-Task-Adaptive-Routing.git}{https://github.com/Yaziwel/AMIR}.
