Gravel Garden once upon a time

Many years ago, my father turned my front garden into a gravel garden, with some big rocks, rockery plants, dwarf tulips etc. Over the years, I have tended to ignore it, as I don't use the front door, and now I have retired it is time I sorted it out. Somehow it has acquired a Daylilly, some Astilbe ( not doing well) many dwarf clematis corms, very invasive Geranium, but most of all it is overrun with Euphorbia. It faces east, and gets full sun until about 1.30, after which shade sets in.
I spent 3 hours pulling stuff out today, and really don't know what to do with it. It's dry, the original gravel has disappeared, and measures about 6feet by 20 feet.
Start again with rocks and gravel? In which case, what plants would have a chance of success there, given a bit of tlc? Dig it all out and do something different?I have attached ( I hope) a pic of it before and after my weeding.
Any suggestions gratefully received! Thankyou

 

If that's Mares/Horsetail you've got there Ariadae, you've got your work cut out, it's very difficult to kill. Don't attempt to dig it out, you will need to use glyphosate on it. I'd do the same with the Euphorbia if you want rid.

I'd be inclined to clear the whole garden and start again.

 

I think it may be a sedum, one of the yellow flowered varieties. Much easier to deal with than mares tail whose roots go so deep the sometimes come up in australia!