New Lawn after bricks, help

Evening everyone, newbie here, Im after some advice please,I bought a house with a bricked back garden, Ive now decided to return it to how it was by relawning it, the garden was previously grass but the previous owner had the garden bricked over, now ive removed all the bricks and the few inches of sand underneath the bricks, the surface is now compacted ballast, now the question is can I simply soil over the compacted ballast or do I need to remove all the ballast and add new soil, I originally wanted to remove the ballast but it looks like theres about 6 inches of it and it will take forever to remove, so is it possible to simply soil over the ballast and get a new lawn growing, many thanks in advance for any replies.