A horrified Ronan Curtis had his head in his hands after gifting relentless Bristol Rovers a leveller.
Then, 10 minutes later, the Irishman was the most relieved man at the Memorial Stadium as he steered Pompey to a dramatic 2-1 victory.
Joy then for the Blues as they registered a fourth-successive win.
It is the first time they have claimed maximum points from their opening four games since 1980, under Frank Burrows.
This was a gutsy performance, with keeper Craig MacGillivray conjuring up a man-of-the-match display to keep Rovers at bay.
Yet despite lengthy periods of digging in and heroically fending off the hosts, Kenny Jackett’s men had held the lead from 32 minutes.
That arrived courtesy of a fierce left-footed shot from skipper Gareth Evans, who also struck against Oxford United at the weekend.
Admittedly, it was against the run of play, not that Pompey cared.
Then, on 76 minutes, Curtis attempted to head clear a corner from the right and after making contact was adjudged by referee Graham Salisbury to have handled.
Substitute Chris Lines stepped up to level – only to receive a red card three minutes later for a challenge on substitute Brett Pitman.
It injected added belief into the visitors and, four minutes from time, Curtis popped up to poke a left-foot shot which crept into the net for a late victory.
As it turned out, a last-gasp victory for Peterborough at Charlton kept Jackett’s men from top spot.
Regardless, that is four victories on the spin now for this ever-impressive Blues side.
Jackett retained the same starting line-up which defeated Oxford 4-1 so impressively on Saturday.
He also elected to keep the substitutes’ bench unaltered, despite Nathan Thompson being declared fit following injury.
That meant Pitman once again on the bench after just 45 minutes of first-team football in League One this season.
In addition, there was no room for Danny Rose and Louis Dennis, despite being fit.
Meanwhile, League Two title-winner Kyle Bennett lined-up against his former club, following his departure for the Pirates in January.
It was Pompey who kicked the game off, although Bennett produced the first moment of note.
The ex-Blues midfielder carved his way through the middle and attempted to slide a pass through to Tom Nichols but Jack Whatmough managed to cut it out.
On four minutes, Curtis had a golden opportunity to break the deadlock.
Jamal Lowe floated a ball in from the right which found the Irishman ghosting in at the far post, yet his angled shot was blocked and comfortably cleared.
The first corner of the game was won by the Blues on nine minutes. Evans delivered it from the right and Oli Hawkins’ header was punched out by Rovers keeper Adam Smith.
Liam Sercombe should have netted on 28 minutes when Nichols squared from inside the box but he sliced a first-time shot over the bar from close range.
But on 32 minutes Evans put Pompey in the lead with a devastating first-time left-foot shot which crashed into the roof of the net.
Hawkins drew two defenders to him under a ball into the box and – when it cleared his head – it fell to Curtis, who nudged a headed pass into Evans’ path and the skipper did the rest in emphatic fashion.
MacGillivray was called into action on the stroke of half-time when he pushed Sercombe’s fierce 30-yard right-foot drive around the post.
There were no substitutions at the break, although minutes into the second half Matt Clarke’s head had to be bandaged after blood was drawn when winning an aerial challenge with Sercombe.
On 51 minutes, MacGillivray flung himself to his left to keep out Nichols’ curling effort, with Sercombe harmlessly putting the follow-up into the sidenetting.
The goal, though, was coming and the hosts made a double substitution on 69 minutes with Lines and Alex Jakubiak coming on for Ed Upson and Stefan Payne.
On 76 minutes, the match was all-square after Lines delivered a corner from the right and the ball struck the hand of Curtis as he headed it clear.
Lines stepped up to confidently slot the penalty into the far corner to make it 1-1.
Then, barely three minutes later, Rovers’ goalscorer was dismissed after a reckless challenge on Pitman, setting up a fascinating finale.
The action continued, with a misjudgement from Whatmough allowing Jakubiak to outpace him and fire in a shot superbly tipped away by MacGillivray.
Then, on 86 minutes, up popped Curtis to squeeze home a left-footed effort which was deflected and bounced into the vacant net to secure a 2-1 victory.