And he insisted Danny Cowley desperately needs a Michael Doyle-type figure in his ranks to stop his side from crumbling when the chips are down.
The Fratton Park outfit suffered their fifth defeat of the League One season after they lost heavily at the New York Stadium.
Marcus Harness’ 49th-minute strike provided a glimmer of hope as he got Pompey back on level terms after Michael Smith’s 29th-minute opener for the hosts.
Portsmouth transfer news: Blues receive bids for former Luton and Millwall midfielder
Transfer gossip: Portsmouth eye raids for ex-Chelsea man and Charlton Athletic's former Wigan Athletic midfielder
Done deals: all the latest League One transfers including Ipswich Town, Oxford United, Bristol Rovers, Charlton Athletic and other Portsmouth rivals' business
Former Portsmouth pair help Ipswich flex muscle as Norwich City man is slammed for conduct in Peterborough's Plymouth Argyle reverse
Danny Cowley makes Portsmouth team selection admission following Cheltenham win - and gives honest assessment of current squad amid early-season promotion hopes
Yet any ambition of a memorable away day soon turn to despair as the Millers scored three goals in eight second-half minutes to put the game to bed.
A second effort from Smith (55 minutes), coupled with goals from Richie Wood (60) and Ben Wiles (63), affectively ended the match as a contest and left Cowley nursing his wounds at the final whistle.
And Whittingham, who was a pundit for BBC Solent in south Yorkshire, did the same – suggesting Pompey’s squad lacks the strength to come good when the going gets tough.
Referring to Cowley’s interview with Solent, when the Blues boss chose his words carefully when sharing his thoughts on the game, Whittingham said: ‘It's hard not to say it and he (Cowley) just said it in a way that he didn't want to have too much of a go at the players and I can understand that - you have to be delicate.‘But, ultimately, resilient, determined, gritty – not enough is what he said.
‘And it boils down to them (the players) not being tough enough as a group of players.
‘There's nobody on that pitch like (Michael) Doyle.
‘When things were going wrong he'd get about people and tell them how to do their job - there's nobody doing that.
‘When a goal goes in they just turn and walk away.
‘There's nobody shouting at anybody.’
Three of the four goals Pompey conceded against Rotherham came from crosses into the box.
Cowley admitted it was a threat he and his players were wary of heading into the game.
Yet Whittingham was astounded by the ease with which the hosts were able to get balls into the box for the likes of Smith to capitalise on.
He added: ‘The only goal that was a real mistake today from an individual's point of view was (Gavin) Bazunu.
‘He doesn’t make those mistakes but he did today.
‘But (the others) they were from crosses that Danny Cowley said they new about before the game.
‘So if you know about it, why not try and stop them? Why are the players not determined enough to go and stop crosses coming in?
‘I remember one of my first games at Fratton Park under Alan Ball - he had a right pop at me for not making it hard enough for a player to get a cross in.
‘They didn’t score from it but he absolutely lambasted me in the changing room and that really woke me up to how important that is.
‘And today it was just too simple for Rotherham to get their crosses into the box.
‘They got plenty of numbers into the box and ultimately those goals they scored have come from that - apart from the mistake from Bazunu.
‘He's been too kind to the players, which a manager will do on occasions.
‘I can't remember which game it was, but Lee Brown came it and said: "You know, we've had a real hard look at ourselves, words have been spoken”.
‘For me, players can't go on saying that, they've got to sustain their ability to be tough.
‘They just can't do it for one game.’