I think the reason is 4 fold,
First, because for a long time it was associated with the BNP. and most moderate people found them abhorrent.
Secondly, It appeared that if you raised concern, people often just called you a racist,
Third is as Tony mentioned, times are tough
Fourth, with the rise of UKIP people may think something might be done about it.
These are in no particular priority.