Nahki Wells’ leveller 16 minutes from time cancelled out an own goal from Joel Lynch in the 1-1 draw at Fratton Park.
Kenny Jackett was aiming to guide the Blues into the fifth round for the first time since reaching the final in 2010.
And the upset looked on 18 minutes into a fine second-half display for the home side, when Lynch turned home Lee Brown’s cross to spark delirious scenes.
Pompey were undone from a corner nine minutes later to ensure a replay is added to their busy schedule.
They will look to a miss from Brett Pitman four minutes after the restart as a big moment in front of a crowd of 19,378, but can take heart particularly from their display after the break.
Pompey made three changes from the side who lost at Oxford with Louis Dennis, Dion Donohue and Brett Pitman all starting.
It was a high-tempo opening with plenty of energy from both sides on a sticky-looking Fratton surface.
There was little in the way of chances, however, until the 26th minute when Darnell Furlong crossed for Aramide Oteh, but his header hit Jack Whatmough to take the sting off the ball and Craig MacGillivray gathered.
Two minutes later an even better chance arrived for Joel Lynch from Luke Freeman’s free-kick, but his header was straight at MacGillivray.
It took until the 41st minute for Pompey threaten, despite getting into some decent crossing positions.
Brett Pitman had the ball in the net when finishing inside the six-yard box but the ref ruled the goal out after spotting a push.
The Blues ended the half strongly as Dennis hit a dipping 25-yard effort which the back-peddling Joe Lumley did well to tip over for a corner.
Pompey should have been in front four minutes after the restart when Dennis clipped a lovely ball into Pitman.
The striker had time to take a touch and put the ball away, but elected to take the ball first time and fired just over the top.
Pitman then couldn’t direct his header on target from Evans’ cross after 56 minutes.
But that didn’t matter in the 63rd minute as the Blues took the lead with Lee Brown delivering a lethal cross from the left, which which Lynch steered past his own keeper.
The lead lasted nine minutes before Nahki Wells levelled from Freeman’s corner, after MacGillivray saved from point-blank range but the Bermudian lifted the loose ball into the roof of the net.
Moments later a huge melee ensued which saw Lynch, Ronan Curtis and Brown booked as tempers boiled over.
Dennis curled a shot over late on but there was no late drama to ensure a return at Loftus Road.