Pruning a potted conker tree

Hi there, this is my first time on this forum.

Some background information; I have a small Conker tree (Aesculus hippocastanum) growing in a medium-sized pot. The tree is ~10 years old and 6 foot tall. It has not flowered yet. It is doing wonderfully and continues to grow at a rate of about 1.5 feet per year. This however is becoming a slight problem as it is now dominating my tiny garden and starting to block access. As the constrains of the pot have not deterred its growth, I need to reduce its height or at least stop it from growing any taller somehow.

So my question is, will it be able to take some pruning and cutting back, or will doing so likely kill it? I do not wish to plant it in the ground at this time as I would like to be able to take it with me when I move.