Adam Macleod, 90, has voted for the Gosport MP over the past three decades.
But he says he is now disillusioned with the man who tried to claim £1,645 for a duck house.
He has written to Tory leader David Cameron questioning why Sir Peter only has to pay back £10,000, despite claiming £30,000 for gardening expenses for his former Titchfield mansion. The claims included 28 tons of manure.
Mr Macleod, of Staplers Reach, Rowner, said: 'I know Mr Viggers had a large garden but surely £30,000 on gardening, grass cutting and landscaping over two years clearly breaks the rules.
'I can't understand how it would even cost that much; maybe he was paying someone to move that duck island of his around.'
Sir Peter, who has held the Gosport seat for 35 years, has been ordered to stand down at the next General Election by his party leader in the wake of his duck house claim.
If he hangs on to his seat until the General Election, he will pocket a £32,000 pay-off on top of his £1.1m pension.
The amount Sir Peter had to pay back was decided by the Tory party's scrutiny panel, which was set up to examine Conservative MPs' claims under the second home allowance.
When asked why Sir Peter hadn't been ordered to pay back more money, a Conservative Party spokesman said: 'We looked at all our MPs' claims for the last four years to determine what should be paid back.
'At the time, David Cameron said it wasn't a perfect process, but the scrutiny panel decided what it thought was right to be paid back.'
Sir Peter was not available for comment.
>> Vote in our latest web poll.