Fareham Borough Council says the 12-car rank in Harper Way, alongside Market Quay, is becoming too busy, with taxis spilling on to the nearby roundabout and blocking buses leaving the bus station.
Letters have now been sent to drivers saying that if the problem is not sorted out the rank could be closed.
But some cabbies say it is the council's fault for licensing too many taxis in the borough.
One driver, who wanted to remain anonymous, said: 'There are too many cabs and nowhere near enough spaces.
'There are maybe 18 spaces on that rank if we squeeze in, and two others which can fit four cars each.
'So there are 26 spaces for 10 times as many cars. How can the borough expect that to be sufficient?
'They must either provide more spaces or introduce a system like the one in Portsmouth city, where you can only get a licence if another driver decides to sell his.'
Another cabbie, who didn't want to give his name, said: 'We know we have got to pay for our licences and we accept that, but we get nothing back from the council except for veiled threats.
'If they take away this rank, it will be mayhem and bedlam.'
But Fareham council leader Sean Woodward said cabbies needed to obey the laws of the road.
'We have had complaints from the bus company and members of the public,' he said.
'This is one of the biggest ranks in the county and they are still over-ranking. It is both dangerous and illegal and we will take action.
'As for the complaint from the drivers about too many licenses, we believe in fair competition.
'We could limit licenses but then a lot of them wouldn't have jobs, and they wouldn't thank us for that either.
'All they have to do is abide by the Highway Code and everyone will rub along happily.'
RANK FACTSThe taxi rank was opened for trade in July 2004 as part of the Market Quay development and immediately proved controversial.
The entire rank was filled instantly, forcing drivers to back up on to the nearby roundabout.
Drivers claimed that, as well as insufficient spaces, a cycle path placed right alongside the rank where passengers would be waiting was 'an accident waiting to happen'.
At the time Fareham council's head of transportation and engineering Andy Viccars hailed it as an improvement, but he admitted there would be some 'teething problems'.
>> Vote in our latest web poll.