But the striker reckons playing the Boxing Day fixture behind closed doors – should it go ahead – will make the Robins' task easier.
Pitman spent three years at Fratton Park after arriving from Ipswich for an undisclosed fee in 2017.
He scored 42 times in 99 appearances for the Blues – with 25 of his goals coming in his maiden campaign.
Predicted Portsmouth XI v Cheltenham: changes expected to side that beat Cardiff so handsomely - but attacking duo could keep places and defender set to return
League One latest: Oxford land former Pompey striker target from Swansea as Sheffield Wednesday bemoan exciting duo's move to Brentford and former Peterborough boss criticises club
Portsmouth boss Danny Cowley hails Spurs loanee Dane Scarlett as move already shows signs of putting Miguel Azeez's short stint from Arsenal in shade
Portsmouth loan strategy revealed following duo's arrival from Ipswich and Blackpool respectively
Portsmouth boss Danny Cowley spells out why Blues can rival Sheffield Wednesday, Ipswich and Derby for promotion as he prepares for welcome return to Cheltenham
However, in his final season, the Jersey-born ace was surplus to requirements for the most part.
Pitman’s release was met by disappointment among sections of Pompey fans before he joined Swindon on a two-year deal.
The 31-year-old’s now preparing to be back at PO4, although the game is in doubt after three Blues players tested positive for coronavirus.
But if the clash is given the green light, it will take place behind closed doors due to Portsmouth being placed in tier four restrictions.
And that'll make it bitter-sweet for Pitman.
He told the BBC Wiltshire: ‘The manager has been around long enough to know what I'm capable of and what I can bring to the team.
‘Portsmouth is another game for us. It's obviously a club I've just come from and something I’m looking forward to.
‘I’m gutted no fans will be in the ground because Portsmouth fans are fantastic.
‘It is what it is and everyone has just got to get on with it.
‘It will (mean a lot going back to Pompey) but they are a good side in great form.
‘It's going to be a tough game. If there are 19,000 fans there then it's a very, very tough game but if it’s behind closed doors then maybe it does make it slightly easier.
‘It’s still going to be a tough game because they're a good side at this level - and so they should be with the budget they've got.’
Swindon languish in the relegation zone after being promoted as League Two champions last season.
They snatched a 2-2 draw with Charlton on Saturday, with Pitman bagging a late equaliser when coming off the bench.
The former Bristol City man’s now hoping to use that result as a springboard for John Sheridan’s side to climb up the table.
He added: ‘Hopefully, we can use this point as a platform for something better.
‘That’s what we’re aiming for, but really actions speak louder than words, so we’ll see.
‘We have got a little bit of a break now before quite a few games in a short space of time, so it’s going to be a busy period.
‘But we’ve got good players, we just need to try and cut out as many of those individual errors as we can and make the game a little bit more simple, if you like.’