Guy Whittingham saluted the quality of David Connolly after the veteran struck twice to make it successive Pompey wins.
The striker pounced in the second half to put Bury to the sword in a 2-0 success and made it four goals in his past three Blues appearances.
Pompey had to survive a poor first half in which Craig Fagan’s awful miss from close range gave them a real let-off.
Then it was down to Connolly to become the match-winner - and lift Pompey off the bottom of League One.
Whittingham said: ‘At half-time we had to remind them what we were asking them to do before we went out into the first half.
‘We allowed Bury to get at us rather than us getting at them so we told them we weren’t happy with that.
‘Then it was down to David.
‘The first one was good work by Jed (Wallace) who got the cross into the box and David showed all his experience there the way he calmly pushed the ball into the net. It was a great finish.
‘The second one was the calmness, not lashing at it straight away, but picking his spot.
‘He makes a difference because it could have been a 0-0, but he has given us the win and we are pleased we got him.’