Have a guess what this stuff is I just dug up

I should update this thread finally as the mystery substance was identified.

When I got my fire in our dining room replaced we had the hearth ripped out and it was made from the exact same stuff. I'd never made the connection between the two as it was under tiles. Anyway the engineer who ripped it out said it was plaster. It was all spread on the chimney breast too, to make it look like artex. Only this stuff came out with a wall paper scraper underneath it. It was easier than striping wall paper.

 

Glad it wasn't asbestos ...

 

I agree with Pete, but only if it isn't asbestos. Don't go digging that out if you think that it might be the case. If you think you might have uncovered some spray it with water, if possible with some PVA in the water, as that will bind the particles together and stop them flying about, but wear a mask and gloves anyway.