Emergency services rushed out on Saturday evening after the man fell from a tender at Royal Clarence Marina.
The strong tide caused him to drift around 300 metres towards Millennium Bridge where he managed to cling on to a pile in the middle of the channel.
He was only spotted thanks to a passer-by who was walking their dog.
Gosport and Fareham Inshore Lifeboat Service (GAFIRS) Coxswain James Baggott said: ‘The man is lucky to be alive.
‘He was swept away from the tender and clinging to the pile and if the dog walker hadn’t raised the alarm the consequences could have been very different.’
The dog walker called 999 after his dog became distressed while crossing the bridge and wouldn’t stop barking.
The coastguard dispatched an MOD marine police unit, Hill Head Coastguard Rescue Team and GAFIRS to assist.
When the lifeboat arrived on scene, the police RIB was just pulling the man out of the water who was suffering from the effects of hypothermia.
GAFIRS lifeboat crews assisted the police and the coastguard rescue teams with first aid care until an ambulance arrived.
The man was checked over by paramedics and later released.
A spokesman for Hampshire Constabulary added: ‘We can confirm we were called to the concern for the welfare of a man at the marina near Weevil Lane in Gosport at 8.08pm on February 5.
‘Officers attended the scene, along with partner agencies, and a man was located in the water and was brought to shore.’
The incident was the seventh of the year for the independent lifeboat service, based at Stokes Bay in Gosport.
GAFIRS is not part of the RNLI and independently funded. Last year it had its busiest year for more than a decade attending 117 rescues in The Solent.
For more information on the service visit GAFIRS.org.uk.