The left-back faces his former club when they travel to Fratton Park tomorrow night (7.45pm).
Brown moved to the Gas in 2011 after being released by QPR.
There, the Farnborough-born talent had a whirlwind seven-year spell at Rovers.
He made more than 300 appearances for the Memorial Stadium outfit, experiencing relegation to the National League, as well as helping them to back-to-back promotions.
Brown opted to allow his Gas contract to expire at the end of last season, though, and subsequently joined the Blues.
The charismatic defender admitted the need for a new challenge was why he departed Rovers.
And with Kenny Jackett’s side pushing for promotion from League One this term, he feels a switch to the south coast is paying off.
When asked if he had any regrets leaving Rovers, Brown said: ‘No, no. I needed a fresh challenge and that’s what I got here.
‘At the moment it’s paying off.
‘I’m at a fantastic football club, we’re third in League One and have a great opportunity for automatic promotion.
‘At the minute it is going well.
‘I felt ready to leave. There were a number of factors why I left and I think it was time after seven years.
‘We won back-to-back promotions, consolidated in League One, finished in the top half of League One and I was ready to leave.
‘I left on a high. I never wanted to leave on a negative and that’s the way it could have went.
‘I’d never want to say Bristol Rovers could never go any higher, that would be disrespectful.
‘But I wanted a knew challenge. There aren’t many people who stay at seven years, especially nowadays and here was the perfect fit.
‘I had a number of offers from other clubs, especially when it was announced.
‘But this is where I came in the end.’
Brown has made 34 appearances for Pompey this campaign.