Long and difficult wars against the Ottomans prompted the rebuilding and extension of the fort into a Wasserburg, a castle surrounded by high earthen banks with bastions and two water-filled moats. The works were carried out by Italian builders from the vicinity of the town of Como, led by the main imperial builder of fortifications on the Slavonian border, Domenico del Lallio.

During its eventful past, the Old Town had changed hands many times – owned by numerous distinguished noble families such as the Counts of Celje, Ivan Ungnad, Juraj Brandenburg and the Croatian Ban Tomo Erdödy and his heirs. The fort became the property of the City in 1925, when the Old Town became the home of the newly established city museum.

The Old Town today houses the oldest, Cultural and Historical Department of the museum, whose collections are exhibited in stylish display rooms. The most valuable are the collections of guild objects, furniture, historical portraits, weapons, paintings, watches, porcelain and glass, along with many other items depicting the daily life of Varaždin nobility and rich citizens. The collections are on display in about 40 galleries for visitors to discover and explore. A special focus needs to be given to the collections of objects that once belonged to two well-known people from Varaždin – the politician and writer Ivan Kukuljević Sakcinski and the linguist Vatroslav Jagić.