By the second half of the 11th Century, much of Wales had already fallen to the rule of the conquering Normans and in 1099, after a bitter and bloody battle, Gower also came under their control. To celebrate their victory, legend suggests that the Normans constructed an immense castle upon the site where Gower's army finally succumbed to their superior invasion force. The result was the most impressive fortification ever built on the peninsula.
Unfortunately, the site, with its still formidable remains, lays in private hands and has been denied public access. Tantalising glimpses of the castle, however, can be gleaned from the A4118 road running from Penmaen to Reynoldston and, from the linking road, running down to Oxwich Bay from Oxwich Towers. A footpath also crosses near the castle through the Penrice Estate.