So the panic stuff is the adrenaline kicking in - the fight or flight response your body is programmed to generate in an "emergency" situation - all you need to do is teach your body that triathlon isn't that kind of situation OR learn to use the adrenaline hit. I get a similar reaction but on a smaller scale.
So that's the easy bit - the why - now to work on the how. You say you arrive early, set-up relaxed and it hits as the hooter goes off - have you tried getting there late? If you turn up late your adrenaline will already be pumping so your body will be used to the effect before you get in the water so the "hit" should be less noticeable when the hooter goes off. Alternatively find a way to wind yourself up before the hooter goes off.
The adrenaline will help you compete if you learn to use it, but you need to get it started a few minutes before the hooter so that your body can adapt to it and the panic subsides and the fight mode kicks in.