camellia in dodgy soil?

Hi there,
I'm trying to turn the pathetic patchy lawn that is my front garden into something decent. I'm so far at the stage of digging it up and improving the soil. The problem, as I've discovered, is that below the topsoil level (which is a good 3-5'' deep so not a disaster) it's basically pockets of clay, building sand, normal-ish soil and - oh, the delight! - rubble. Out of a 5x5x1 foot hole that I've so far dug up, I pulled out a bucket full of bricks, stones, and pieces of metal. A neighbour said our road used to be a builder's yard
the good news is, because of the amount of stones in the ground, the soil seems to drain well. My hole didn't get waterlogged for the whole of this miserable rainy week. The odd thing tho, having dug up a fair bit, I only came accross SIX individual worms. Whatever that's about. But I'm hoping once I've chucked lots and lots of humus into it, and planted a greater variety of plants, worms might feel more at home.
I was planning to plant a camellia, among other things. The soil pH is 7, according to my cheapy B&Q pH reader. So far as I'm aware, 7 is about as alcaline as camellias can tolerate. My question is, because the soil isn't well mixed and I'm only digging about a foot deep, would a camellia be ok with that? I have actually come across a few pieces of chalk which weren't big, but how much of a problem would it be for a camellia if one of its thousand roots ended up drilling into a bit of limestone?
Or could someone perhaps suggest other compact evergreen plants for semi-shade in a north-west facing garden? (if they flower as well I'd be over the moon!:happydance
Many thanks!

 

oh, by compact I meant ideally under 3 metres tall. 1.5-2 m would be fab!