Raised bed nightmare help needed!

I agree with Pete - it is not structural. The fence is a well constructed self supporting structure. So rip it out and and go with your ideal. The only affect that ripping it out would have is on your decking but as I'm the forums resident decking hater I would have that on my wood burner anyway Seriously though, you could retain the wall next to the decking.

The problem is that you don't say what area you're in? It will make a big difference to what you may get away with in terms of tropical looking plants.