Gosport's market, which attracts hundreds of people on Tuesdays and Saturdays, may move to a car park after complaints were made by traders.
The move would be a blow for the High Street, which has seen several shops close in recent months as shoppers are tempted away to Fareham and Gunwharf Quays.
Two coffee shops have complained to the borough council that they have the right to put out chairs in the High Street for free, but market organisers say they have the right to charge them fees.
The News has seen a letter from Mike Jeffery, development services manager at the council, which states 'market stalls always take precedent' and adds that the cafes don't have the right to put out tables and chairs on market days.
But Gosport council is to look into whether the market can charge shops, at a meeting of its community and environment board.
Should the board decide that the matter needs to be addressed when it next meets, it has the power to order the market operator not to ask shop owners for money to put out tables and chairs.
Graham Rush, a director at Hughmark International, which runs the market, said: 'If they keep complaining about this, then we might just move the market because it just seems to be causing us too much trouble.
'These coffee shop owners are giving us a lot of problems, when we aren't in the wrong. Most of the people who come down High Street on market days are there for the market and it would be a blow to the street.
'We would stay in Gosport, but look at moving to a car park, so we don't have all this trouble.'
Ivano Lucie, owner of Lu Chino's cafe, told the council market representatives said he had to pay on Tuesdays and Saturdays when the market was on. Costa Coffee has also made a complaint.
Leader of Gosport council Councillor Dave Smith said: 'We have had complaints from both Costa Coffee and Lu Chino's – it's something we will look into and we will get to the bottom of this.'