To try and answer this question, you have to factor in a number of variables.
The first is the technicality of the descent. If it consists of a sinuous series of switchbacks, such as Alpe d’Huez, or is steep and poorly surfaced, such as you’d encounter in the Lake District, a road bike will undoubtedly have the edge. The handling will be more responsive and, as there will be limited opportunity to use the extensions on the TT bike, its aero edge will be removed. However, if it’s non-technical, not very steep and you’ve got the cojones to stay down on your extensions, you’ll fly on your TT bike. It doesn’t take much time down on the extensions to claw back time losses due to inferior handling.
Next is the skill of the rider. If you’ve spent hours training on your TT bike, ridden technical roads and are 100% familiar with how it handles, you won’t be at any significant disadvantage descending on it. However, if you’re one of those triathletes who only rides their TT set-up on race day and on the turbo, you’ll be shipping seconds on every downhill.
Finally, there’s the bike’s set-up. If your TT bike is equipped with disc brakes, that’ll go some way to off-setting time losses due to handling. Satellite shifters on the bull-horns will also help significantly on descents, allowing far better shifting in and out of corners than extension shifters only.