Penso castle was built here on the highest point in Vigo during the Medieval period. It was owned by the Archbishop of Santiago, who ruled the local area. The fortification was knocked down after an attack by Pedro Madruga and houses were built in its place. These were later demolished to build an artillery fort known as San Sebastian Castle.

Documentary sources describe Penso castle as "being built of granite and lime, with battlements, two or three floors covered with tiles and wood, and very well made both inside and out with surrounding battlements and barbican". Dynastic conflicts and power struggles during the 15th century led to its destruction. At the end of the century the castle was captured by Pedro Álvarez de Soutomaior, known as Pedro Madruga. After it had been recaptured, the Archbishop ordered it to be demolished.

Houses were later built on the site of the castle. It is the <<montezillo alto>> (hill) in the earliest known image of the town, dating to 1597. A chapel dedicated to Saint Sebastian, defender against the plague, was also built here. The construction of the chapel could be associated with a series of epidemics suffered by the local people during this period.

The houses were knocked down when the town walls were built in 1656 and an artillery fort was erected. You can see these remains today. The chapel, which the fortification took its name from, was outside the artillery fort. Initially the fort only contained a small garrison. However, in the 19th century pavilions and other military buildings were built.

On 28th March 1813, during the fiestas celebrating the Reconquest of the town, the gunpowder store blew up. This caused the collegiate church, which had been in a poor state of repair, to collapse. The fortification was used as an army barracks until 1964 when the buildings were handed over to the town. In 1972 the current municipal building was built, and a number of the fortification’s original walls were knocked down. The surviving sections of the artillery fort continue to preserve the memory of the place.