Network Rail said the ordnance was discovered at a construction site between Netley and Sholing, Hampshire.
It released an image showing the bomb in the bucket of an excavator.
Network Rail said an explosive ordnance disposal team and officers from Hampshire police and British Transport Police are at the scene.
The discovery means all lines between Netley and Sholing are blocked, causing disruption to Great Western Railway, South Western Railway and Southern services.
The find comes after four people were injured, including one seriously, when a Second World War bomb exploded on a railway construction site near the main station in Munich, Germany, on Wednesday.