Summer signing Alex Przespoloweski and substitute Billy Connor got the goals as Shaun Wilkinson' s side recorded a 2-0 victory.
Wilkinson, who was not at the game because of an ongoing stadium ban, praised his players for remaining composed against a 'physical' Tadley outfit.
The hosts had Jordan Goater dismissed a minute before for break after picking up a second yellow, while Andrew Charsley received his marching orders a minute from time.
For Baffins, it was a first triumph of the season following their opening-day league draw at Fareham.
'We wanted to show him a bit of love' - Portsmouth-based Spurs fans' terrific gesture to ensure Dane Scarlett feels at home on the south-coast
Ronan Curtis' Portsmouth future spelt out by trusted source well in the know - with advice on where to play Irishman added, too
League One latest: Portsmouth linked duo set for Oxford United and Lincoln moves, Southampton prospect joins Blues rivals as Ipswich land former Blackpool and Derby veteran
Portsmouth's Miniesta bidding to live up to big reputation - and those comparisons with ex-Barcelona and Spain star
Denver Hume: Portsmouth's expensive forgotten man and why former Sunderland defender really could be one for the future
And Wilkison is hopeful his team can now build on the victory after pocketing £1,125 in prize money for the FA Cup win.
He said: ‘It was good to get through, especially in our first game away from home, it was win at all costs, really.
‘Obviously last season we went out at the first round so it was just pleasing to get through.
‘You always want to have a little run in the cup - first and foremost you want to win football matches - that breeds a bit of confidence.
‘It was good to get the first win of the season and in the cup competitions you want to go as far as you can.
‘Obviously I wasn’t there, but from what I hear we had loads of possession the other team were really physical.
‘They tried to intimidate us but we’ve got characters in our squad this season who won’t let that happen.
‘I think we were knocking on the door for a very long time, didn’t take our chances, but once we finally got the goal it opened up for us and we got the deserved second at the end.’
Goater collected a second booking on the stroke of half time as Baffins had a man advantage for all of the second period.
Przepolewski finally made the breakthrough just after the hour mark to hand the visitors the advantage.
Substitute Connor then struck a minute from time before Tadley finished with nine men following Charsley's dismissal.