The goal was a gift as young substitute goalkeeper Lewis Watch allowed a harmless James White shot through his hands and into the net.
Fellow strugglers Whitehawk and Truro City also won condemning Borough to relegation.
Borough coach Danny Thompson was devastated.
‘It was the last kick of the game which destroyed us,’ he said Thompson.
‘But for that we would at least have gone into our final game against Whitehawk still having a chance to stay up.
‘It is heartbreaking to get so close to doing that.
‘We are all gutted but it just sums up our season.
‘After all that has happened at the club this season, in reality we should have been dead and buried months ago.
‘At one point we only picked up one point from 20 games.
‘The fact we got so close after that is unbelievable.’
Borough had problems before the start with experienced goalkeeper Nick Jordan carrying an injury.
He only lasted until half-time when he had to be replaced.
‘We had talked Nick into playing in the first place but by half-time he could hardly walk,’ said Thompson.
The visitors were also forced into an early change when Ben Harding limped off after just six minutes.
After a goalless first half Borough fell behind when Joe Oastler headed past Watch into his own goal.
Borough though hit back with two Tony Lee goals but it wasn’t enough to as the late goal saw them drop out of the division.
‘We got ourselves back into the game with two great Tony Lee goals,’ said Thompson.
‘After that we looked to have things under control.’
Borough will host Whitehawk on Saturday in their final game.
Fans will hope for some positive takeover news.
There is also speculation about the future of manager Alex Pike who failed in his bid for the vacant Weymouth job.