The attribution of the castle as a royal foundation is based on a charter of Henry I dated 1101, granting the town and castle of Colchester to Eudo Dapifer "as my father had them and my brother and myself", Henry's father and brother being William I, "William the Conqueror", and William II, "William Rufus". The somewhat unreliable Colchester Chronicle, written in the late 13th century, credits Eudo with the construction of the castle and gives a commencement date of 1076. The design of the castle has been associated with Gundulf of Rochester purely on the basis of the similarities between Colchester and the White Tower at the Tower of London; however, both keeps also resemble the much earlier example at Château d'Ivry-la-Bataille in Upper Normandy.
At one and a half times the size of the ground plan of the White Tower, Colchester's keep of 152 by 112 feet (46 m × 34 m) has the largest area of any medieval tower built in Britain or in Europe. The enormous size of the keep was dictated by the decision to utilise the masonry base or podium of the Temple of Claudius, built between AD 49 and 60, which itself was the largest Roman temple in Britain. The site is on high ground at the western end of the walled town and at the time of the Norman Conquest, a Saxon chapel and other buildings which may have constituted a royal villa lay close by the ruins of the temple. The obvious motive for reusing this site was the ready made foundations and the availability of Roman building materials in an area without any naturally occurring stone. Another factor may have been that the Normans like to see themselves as imperial successors to the Romans, William being frequently compared by his biographer, William of Poitiers, to Julius Caesar and his barons to the Roman Senate. The Colchester Chronicle described the temple site as a palace built by the mythical Roman-era King Coel; either way, it was providing a provenance for the Norman occupiers as the inheritors of a heroic past. Siting the castle so close to the centre of the town makes Colchester the exception to the rule that Norman castles were built as a part of the town's external defences, with access to open countryside.
The initial preparation of the site involved the demolition of the surviving superstructure of the Roman temple, resulting in a layer of mortar rubble at the Norman ground level. The walls of the keep sit on narrow foundation trenches filled with rubble and mortar, and directly abut the edge of the Roman podium, except in the south where they are set back to avoid the original temple steps and to facilitate the digging of a well. The walls themselves are made of coursed rubble, including septaria and Roman brick robbed from nearby ruins. Ashlar dressings are of Barnack and other stone, as well as Roman tile and brick. A large apse projects from the southeast corner, resembling St John's Chapel in the White Tower; however there is no firm evidence that a similar chapel ever existed at Colchester. It has been speculated that an apse was added to the Temple of Claudius in the 4th century during a putative conversion to a Christian church and that the Normans followed this outline. The keep was divided internally by a wall running from north to south, a second dividing wall was added to the larger eastern section at a later date.