The sub keeper came to the rescue after a shambolic first-half performance at Broadhall Way almost saw Kenny Jackett’s side knocked out by the side who finished bottom of League Two last season - and only stayed in the league because of Macclesfield’s points deductions.
It finished 3-3 after normal time with the Blues conceding three first-half goals, with defending which would have shamed a parks football side.
Two woeful goals in two minutes gave Jackett’s men a mountain to climb, before Ronan Curtis pulled one back in the 13th minute.
Some shaky keeping from Alex Bass, who was withdrawn at the break, and more statuesque defending allowed Scott Cuthbert to put his side back in front.
A Gareth Evans penalty on the stroke of half-time and emphatic John Marquis finish levelled it with the game going to penalties - where MacGillivray shone.
The Scot’s saves from Danny Newton, Elliot Osbourne and Ben Coker spared Pompey’s blushes, but that failed to cover up for the woeful defensive play with big early questions over the back four.
Pompey’s defending was woeful from the outset with Stevenage taking advantage of the gifts.
Elliott List was the recipient for the first as Sean Raggett completely missed an angled ball in and the former Gillingham man rifled home the finish.
The defending got even worse seconds later, as Raggett again failed to clear and Charlie Carter burst through the orange shirts and rifled in a finish.
Bass then got it wrong as he came out of his box but Pompey rode their luck on that occasion before a moment of quality from Curtis reduced the arrears after 21 minutes, as he cut in from the left and rifled in a finish.
Bass then failed to deal with a corner and a crowd of Blues defenders abysmally failed to clear, as Scott Cuthbert took advantage of the parks defending.
But a Ben Coker handball on the stroke of half-time gave Gareth Evans the chance to slot away a penalty - and give the game a very fortunate scoreline for Pompey.
After his shaky first half Bass was withdrawn at the break for Craig MacGillivray and the visitors were level six minutes after the restart, as Marquis confidently advanced past Luke Prosser and dispatched an emphatic finish from the edge of the box.
After the first-half shambles, Pompey had a grip on the game after the break and were the side looking for the winner.
It went to penalties, though, and with Marquis, Curtis and Brown scoring MacGillivray did the rest.