Halfway through the XII century the Annibaldi built a powerful fortress of strategic importance for its closeness to the Via Appia and the Tyrrhenian Sea.
The courtyard of the fortress was at Piazza d'Armi and in its centre had been escavated a cistern in which the rain water would flow from the roofs and terraces of the buildings. Elegant mullioned windows reflected light on delicately painted walls in gothic style, some of which (recently restored) can still be admired. The Mantower (42 metres high) and the short maletower are the only ones left of the primitive Annibaldi fortress, to wich belong also the foundations of the Hall of the Barons.
A church in the courtyard was named San Pietro in Corte and was razed to the ground under the Borgias in 1500, with no respect for the buried remains of the Caetani.
Towards the end of the XIII century Pietro Caetani, nephew of Pope Bonifacio VIII, paid the Annibaldi the sum of 140 thousand gold florins for the fiefs of Sermoneta and Bassiano. In October 1297 the military nobleman Anagni Catenazio, solicitor of the Caetani, appeared at the entrance of the fortress of Sermoneta where the door was immediately opened. He touched it, closed it and reopened it taking ritual possession of the
The Caetani enlargened and renewed the fortress with many buildings. The most important one was the Hall of the Barons also know as the House of the Lords, 22 metres long and adjacent to the Mantower. Under the seigniority of Onorato 111 Caetani, Sermoneta reached its golden peak: many of its buildings belong to this period and were much admired by his contemporaries.
It is said that during the visit of the Austrian emperor Federico 111 in 1452, his soldiers were permitted to have a free hand in the wine-cellars where they "ended in wine up to their knees".
Nicola II, a son of Onorato Ill, was guard of the Conclave during the election of Alessandro VI Borgia. He died tragically in 1494 by poisoning, his death most probably commissioned by Cesare Borgia. Two years later his brother Giacomo V was called to Rome and falling in a trap was imprisoned in Castel Sant'Angelo, then tried and poisoned.
Sermoneta was besieged by Cesare Borgia and only surrendered after harsh fighting in which also the
women participated. From then onwards what was once a strategic fortress became an impregnable one:
the most powerful fortress of the Lazio region, second only to Castel Sant'Angelo.