I think most people say the phrase with a good intention. It's lovable. Much less hostile than "yeah course" that we threw at Gerrard just because he was from Liverpool.
I am a foreigner in this country, and my family all have funny accents. I've dealt with xenophobia and laughing at things I say, but this is not it in my opinion. It's a warm catchphrase, at least in my own intentions and views.
But if you are not comfortable with it I get it. I just wouldn't be that harsh on it.