Ellis Harrison’s stoppage-time header delivered Pompey a smash-and-grab maiden League One away win of the season at Doncaster.
The Blues were put under constant pressure at the Keepmoat Stadium but someone managed to snatch all three points.
Gareth Evans opened the scoring for the visitors on the hour-mark, only for Reece James to equalise in the 82nd minute.
But it was Harrison, introduced as a half-time substitute, who cropped up to deliver Kenny Jackett’s men the win.
Pompey made three changes from last weekend’s defeat of Bolton.
Ronan Curtis was axed from the Blues’ squad altogether and was replaced by Gareth Evans in a left-midfield role.
Fit-again Brown came in for Brandon Haunstrup, while Marcus Harness started on the right wing in place of Ryan Williams.
The Blues were given a scare in the third minute when Alfie May slid a ball across the face of goal but no Donny player could get on the end of it.
On six minutes, John Marquis – facing the side he joined Pompey from in the summer – had a shot on the spin but his effort lacked purchase.
Jon Taylor would have been ruing his miss in the eighth minute for the hosts. Reece James pulled a ball back for the winger but he blazed his effort over the bar.
And three minutes later it was May who struck the woodwork when his fierce right-foot shot cannoned off the bar.
Jackett’s side continued to be put under incessant pressure. In the 18th minute, a through ball split the visitors’ defence but keeper Craig MacGillivray was off his line to thwart Taylor's effort.
Home-manager Darren Moore would have been in disbelief his men didn’t break the deadlock with 11 minutes of the first half remaining.
James Coppinger was pulling all the strings for Rovers and his cross found Kieran Sadlier in acres of space – but somehow missed from inside the six-yard box.
Harrison was introduced for Marquis at half-time, although the home side continued on the front foot.
On 48 minutes, MacGillivray made a fine save to his left to keep out Whiteman’s fizzing effort.
The Donny favourite had another effort on goal that deflected off Sean Raggett five minutes later but MacGillivray adjusted well to gather.
And despite being put under the pump, it was Pompey who broke the deadlock against the run of play
The visitors broke from a Doncaster corner, with Ben Close sliding in Evans, who finished with aplomb past Dieng.
The Blues then nearly grabbed a second on 69 minutes when Close’s shot that was heading for the top corner was tipped over by the home stopper.
Moore’s men again started to get on top as they searched for an equaliser.
For a second time in the game, they struck the woodwork when Whiteman’s effort bounced off the bar with 15 minutes remaining. Moments later, Brad Halliday also went close but his shot just went over.
Pompey could only hang on for so long, though, and Doncaster levelled with eight minutes remaining.
May’s cross could only be parried out by MacGillivray and there was James to finish the rebound.
Rovers almost stole all three points a minute of normal time remaining but Whitman’s bending shot went just the wrong side of the target.
There was to be another goal in the game – and it was Pompey who'd celebrate it in stoppage-time.
James Bolton’s cross found Harrison and he did brilliantly to steer his header over Dieng.
It certainly came against the run of play – but the Blues didn’t mind at all.
After a stuttering start to the season, Pompey collected three precious points as they moved up to 16th in the table.
Pompey: MacGillivray, Bolton, Burgess, Raggett, Brown, Harness (Williams 82), Naylor, Close, Evans, Pitman, Marquis (Harrison 46).
Subs not used: Bass, Walkes, Downing, Haunstrup, McCrorie.
Doncaster: Dieng, Hilliday, Anderson, Daniels, James, Whiteman, Sheaf, May, Coppinger, Taylor, Sadlier.
Subs not used: Lawlor, Wright, Gomes, Amos, Longbottom, Kiwomya, Greaves.