The Grade-II listed Manor House, Bedhampton, sits on land referred to in the Domesday Book.
Scores of residents objected when earlier this year Chris Flint applied to build the detached two-storey home for his family-of-five.
The manor sits in a Conservation Area, with residents objecting to the development, which would be built in the manor’s garden.
John Prime, from the Edward Gardens Owners Association, said the proposed home would be ‘undoubtedly obtrusive and overbearing’.
He said: ‘It is on same level as the neighbouring properties and dominates by nature of its size as a four-bedroom house and large triple garage (which is not replicated in any of the other six houses) on the proposed site.’
Maureen Thorneycroft, of Nursery Road, feared the home would set a precedent.
Miss Thorneycroft said: ‘It is not in keeping with Bedhampton Conservation Area nor the listed Manor House. If this application goes through it will be a precedent for other new houses to be built in the large gardens nearby.
‘Havant is gradually losing its charm that attracted me to move here in 1975.’
Mr Flint said it was a ‘unique opportunity’ for his family. He said: ‘We do appreciate that the new house is making a notable change to the existing setting, but we feel that this can be accommodated with a property that seeks to retain a setting for the Manor House whilst enhancing the local area and retaining a sense of separation within a private area.’
Philip Turner, from Hampshire Buildings Preservation Trust, said the proposal would do ‘irreplaceable harm to the building’s relationship with surroundings’.
Havant’s council rejected the application.