Excavations around the old castle site have provided evidence for Saxon settlement, a finding supported by the name, 'the farm or settlement (wick) by the sedge'. The remains of the castle include the West curtain and a tower on the West and remains of another on the South, together with parts of the Tudor fireplace in the Great Hall.
Early hunting rights were held by the de Braose family, who sub tenanted to Robert le Sauvage, as an outlying part of Broadwater manor, now part of Worthing. It is very likely that a 'lodge' was built for overnight use. The Castle was licensed to be crenellated in 1258 and in 1262 had a park of 400 acres with wild horses; evidence remains for two concentric moats, an unusual design for an English castle.
About thirty yards from the outer moat is a natural spring, called the Nun's well, which is beautifully constructed of large blocks of hewn stone, alternatively called "St. Mary's Well". The religious connection is uncertain, but may relate to religious activity at nearby Knepp Castle.
The castle passed eventually into the hands of the dukes of Norfolk, as successors to the de Braose estates. As the decades passed the castle eventually fell into a bad state of disrepair. In 1602 Queen Elizabeth I leased Sedgwick to Sir John Caryll for 60 years; he is said to have deserted it, and demolished much of the castle, using the stone and masonry to build a new house called Sedgwick Lodge on the higher site of the present house.
The castle is a Scheduled Ancient Monument recorded in The Domesday Book. Currently the site is carefully monitored by English Heritage.