Currently, when I tell someone my phone number, I recall my phone number in Japanese and translate it into English. It is very hard to recall it in English directly. I noticed that numbers are more strongly connected to my Japanese language. Same thing happens when I think of arithmetic and multiplication table.
To condition my mind with English numbers, I started to create my new habit to count numbers in English. For example, I memorized my phone number in English, I count numbers in English when I do my daily push-ups both ascending and descending orders, I memorize all numbers in English from now on etc.
Things seem to be getting crazier and that's totally OK with me!