Should I move the Japanese holly?

I have a Japanese holly in a very shady location - north facing garden next to a low wall, overshadowed at times by the geraniums that pop up every year. I bought it because the tag said it was good for shade but in the two years I've had it, it has lost a lot of leaves and doesn't seem very happy. I'm thinking about digging it up and moving it but I don't know if it would be sufficient to dig up the geraniums instead.

My idea was to have a mini hedge of evergreens along the wall but the holly and its friend the Euronymous (sp?) have not grown well there.