The Blues suffered a 1-0 defeat at Luton Town at Kenilworth Road on Saturday.
Kenny Jackett’s men fell behind in first-half stoppage time when James Collins’ volley broke the deadlock.
Pompey had their chances to get back on level terms in the second period but were unable to beat Marek Stetch.
The Luton goalkeeper stood defiant in the hosts’ goal, making smart saves to deny Matt Clarke and Ben Close.
Denver Hume: Portsmouth's expensive forgotten man and why former Sunderland defender really could be one for the future
'Real driving force... lion-hearted display... classy presence' - Neil Allen's Portsmouth ratings from 3-0 Carabao Cup win at Cardiff
'We wanted to show him a bit of love' - Portsmouth-based Spurs fans' terrific gesture to ensure Dane Scarlett feels at home on the south-coast
League One latest: Portsmouth linked duo set for Oxford United and Lincoln moves, Southampton prospect joins Blues rivals as Ipswich land former Blackpool and Derby veteran
'We’ve got a real problem': Portsmouth boss reveals headache after career first at Cardiff
Haunstrup – who replaced Dion Donohue in a left-back role – revealed he was gutted Pompey exited the Cup.
He said: ‘It is the one cup that has all of the lower-league teams in it and everyone who enters wants to win the competition.
‘The FA Cup is such a big trophy with such a great history.
‘You see a lot of teams in the FA Cup wanting to prove they can play at a higher level – like we do.
‘We went to Luton wanting to put a good display in.
‘I feel like we did that, especially in the second half.
‘However, we just couldn’t find the goals we needed to progress into the next round.’
Haunstrup returned Jackett’s starting line-up for the first time since September’s loss to Rotherham.
He revealed he felt the affects of being out of the team – but believes it would have been different if the goals had have went in.
‘You always feel the last 20 minutes of games, especially when you’ve been out of the team,’Haunstrup added.
‘However, it probably would have went the other way if we’d have got a few goals. It would have given us a confidence boost and the momentum to have a good chance of winning the tie.’