  The Casimir effect, arising from vacuum quantum fluctuations, plays a
fundamental role in the development of modern quantum electrodynamics. In
parallel, the field of condensed matter has flourished through the discovery of
various materials exhibiting broken symmetries, often connected to topology and
characterized by magneto-electric coupling. Here, we calculate the Casimir
forces between materials with time-reversal symmetry and/or parity symmetry
breaking. Remarkably, we obtain a universal phase diagram governing the sign of
symmetry-breaking-induced Casimir forces, contributing to a comprehensive
understanding on the sign of Casimir force for linear optical materials. The
discovered phase diagram serves as a roadmap for searching repulsive Casimir
forces, a subject bearing both theoretical interest and practical significance.
