If colonialism has made those countries poor, then they should have gotten rich once they were no longer part of a colonial empire. At the same time countries which had large colonial empires should have gotten poor, when loosing their empire.

Of you look closer you will see that these countries often still exert significant control, in particular economically over their “former” colonies.