Walden Castle was built around 1141 and 1143 by Geoffrey de Mandeville, 1st Earl of Essex. He built the castle probably on the site of an earlier 11th century motte castle, to reinforce his power across the region. The castle consisted of a square Norman keep and a bailey wall. In 1143 Geoffrey was arrested by King Stephen and Walden Castle was confiscated, amongst other castles. After Geoffrey was released he launched a military campaign against the King. He died during that campaign in 1144, still not having reclaimed Walden Castle.
In 1157, Henry II, King of England, ordered Walden Castle to be slighted to render it indefensible. At that time the keep was probably still unfinished.
Later the manor of Walden passed to the De Bohun family. In 1347 they got the licence to crenellate but to what extent the castle was refortified or rebuilt is not clear.
What remains of Walden Castle at present is the flint and mortar core of the walls of the first two stories of the keep. The keep would originally have been 3 stories high.