The council has proposed £1 hourly charging for up to six hours between 10am and 6pm.
Portchester Castle is under the care of English Heritage which owns and maintains more than 400 historic sites across the country.
The charges have been proposed due to the council facing significant financial pressures made worse by Covid-19. The plans caused uproar when they were proposed over the summer.
Councillor Sean Woodward, executive leader said: ‘We’ve been the only one for many years with no charging in coastal car parks.
‘Every single council in southern England has charged for many years in some or all of their coastal car parks, so why wouldn't Fareham?
‘We used to as a council receive the revenue support grant from central government - it used to be millions of pounds a year so that's one hole that needs to be filled.
‘Inflation is another one of the significant costs and the fact that council tax only provides 16 per cent of the £47m that we spend a year on services.
‘So the other 84 per cent needs to be found from fees and charges.
‘We have to find that money from somewhere and we don't get it from the government, so we have to get it from other places.
‘I've often been struck when I've parked at Portchester - I've been asked on a number of occasions by people where's the parking meter.’
The council has launched a public consultation to ask residents about their views on a proposed increase in council tax amounting to 1p per day.
David Bailey, English Heritage’s Head of Historic Properties for Hampshire & the Isle of Wight said: ‘The car parks at Portchester Castle are managed and maintained by Fareham Borough Council.
‘We’ve spoken to the council throughout and understand the need to introduce the charge. Free parking will still be available in the village centre, a short walk from the castle.’