The castle of Ormož is situated in the south-eastern part of the Bronze Age settlement which was protected with an earthen rampart and a defensive ditch.
The Lords of Ptuj started building the castle complex in the last third of the 13th century. It remained their property until 1438 when the family of the Lords of Ptuj died out.
After it was burnt down, it passed in the hands of the family Szekely in 1490 that had it restored with the purpose to settle there. In 1532 the castle was damaged by the Turks; in 1540 it was restored as part of the town fortifications. After the attack of the Kruci in 1704, the castle was derelict. Between 1710 and 1740 it was restored in the Baroque style by the family Pethe. At the beginning of the 19th century, the castle was bought by Jože Pauer, an industrialist, who had it restored in the Classical style and built a wall around the south-eastern castle garden. The last owners of the castle were counts Wurmbrandt whose coat of arms is built-in above the main castle entrance.