  The main focus of this work is to study magnetic soft charges of the four
dimensional Maxwell theory. Imposing appropriate asymptotic falloff conditions,
we compute the electric and magnetic soft charges and their algebra both at
spatial and at null infinity. While the commutator of two electric or two
magnetic soft charges vanish, the electric and magnetic soft charges satisfy a
complex $U(1)$ current algebra. This current algebra through Sugawara
construction yields two $U(1)$ Kac-Moody algebras. We repeat the charge
analysis in the electric-magnetic duality-symmetric Maxwell theory and
construct the duality-symmetric phase space where the electric and magnetic
soft charges generate the respective boundary gauge transformations. We show
that the generator of the electric-magnetic duality and the electric and
magnetic soft charges form infinite copies of $iso(2)$ algebra. Moreover, we
study the algebra of charges associated with the global Poincar\'e symmetry of
the background Minkowski spacetime and the soft charges. We discuss physical
meaning and implication of our charges and their algebra.
