Mr Hunt had been due to go to the drinks and canapes evening at the Lysses House Hotel with Tory activists last night until the event’s open invitation began to circulate on social media.
Some junior doctors, who say Mr Hunt has refused to speak to them directly, said they would go to talk to him about the contract, which has sparked a series of strikes.
He did however visit Fareham Community Hospital but there were claims Mr Hunt changed the time to avoid protesters.
Raymond Hale, chairman of the Fareham and Gosport Diabetes UK Group, was due to meet Mr Hunt but was not told about the earlier time.
He said: ‘We were very disappointed that we didn’t get to tell Mr Hunt our comments and suggestions in person. He has let us down.
‘But we are grateful to the clinical commissioning group staying behind to meet us.’
Jon Woods, Portsmouth Trades Council president, went to the Lysses to check Mr Hunt had definitely cancelled. He has been taking part in the protests with junior doctors in the area.
He said: ‘It is disgusting that people at the hospital who were due to meet Mr Hunt didn’t get the chance because he was running scared.
‘We are seeing this as a victory that he had to cancel the event because of us protesters.’
One junior doctor who lives in the area said that she was disappointed that the event had been cancelled, as she had been invited and wanted to put a question to Mr Hunt.
She said: ‘It’s an upsetting time for all of us. The worst thing is the dishonesty and the way he has handled the situation – from our point of view it’s as if he doesn’t care.’
Mr Hunt has announced that he will impose a new employment contract on NHS junior doctors without their consent.
The visit to the hospital, in Sarisbury, was organised by Fareham MP Suella Fernandes. She invited Mr Hunt after raising the issue of securing more facilities on the site with him in the House of Commons in October.
She said: ‘I’m very pleased the secretary of state was able to find time to visit the hospital and see for himself the excellent facilities it has.
‘It gave me the chance to explain to him in more detail my campaign to secure addition services on site – something which I have been raising with him for some time.’