  We prove the transformation formula of Donaldson-Thomas (DT) invariants
counting two dimensional torsion sheaves on Calabi-Yau 3-folds under flops. The
error term is described by the Dedekind eta function and the Jacobi theta
function, and our result gives evidence of a 3-fold version of Vafa-Witten's
S-duality conjecture. As an application, we prove a blow-up formula of DT type
invariants on the total spaces of canonical line bundles on smooth projective
surfaces. It gives an analogue of the similar blow-up formula in the original
S-duality conjecture by Yoshioka, Li-Qin and G\"ottsche.
