The list is smaller than you think:
A new study has found that at various times the British have invaded almost 90 per cent of the countries around the globe.
The analysis of the histories of the almost 200 countries in the world found only 22 which have never experienced an invasion by the British…
…Only a comparatively small proportion of the total in Mr Laycock’s list of invaded states actually formed an official part of the empire.
The remainder have been included because the British were found to have achieved some sort of military presence in the territory – however transitory – either through force, the threat of force, negotiation or payment.
That’s partially why, no matter where you are, English (or some hybrid form) can be heard.