  There has been intense recent interest in the two-dimensional electron gases
(2DEGs) that form at the surfaces and interfaces of KTaO$_3$ (KTO), with the
discovery of superconductivity at temperatures significantly higher than those
of similar 2DEGs based on SrTiO$_3$ (STO). Like STO heterostructures, these KTO
2DEGs are formed by depositing an overlayer on top of appropriately prepared
KTO surfaces. Some of these overlayers are magnetic, and the resulting 2DEGs
show signatures of this magnetism, including hysteresis in the
magnetoresistance (MR). Here we show that KTO 2DEGs fabricated by depositing
AlO$_x$ on top of KTO also show hysteretic MR, indicative of long range
magnetic order, even though the samples nominally contain no intrinsic magnetic
elements. The hysteresis appears in both the transverse and longitudinal
resistance in magnetic fields both perpendicular to and in the plane of the
2DEG. The hysteretic MR has different characteristic fields and shapes for
surfaces of different crystal orientations, and vanishes above a few Kelvin.
Density functional theory (DFT) calculations indicate that the magnetism likely
arises from Ta$^{4+}$ local moments created in the presence of oxygen
vacancies.
