why is my gingko not growing?

i have lovely gingko bought four years ago --- it is in a pot, first year its stem grew 3 inches but since then it has done nothing to grow upwards. It has lovely big leaves, they are green and look happy, i also gave it composted soil at the start of the year and still it has not grown upwards.
What am i doing wrong?
Should i plant it into a deeper pot so its roots can grow downwards / not give it rich soil.....? Suggestions appreciated
karaman

 

I tend to think it needs a bigger pot.
As a hardy tree it would probably be best potted on into bigger pots each year until it is big enough to plant out.

 

Thanks, willco. karaman