These days, anti-immigration is not always equal to racists' opinion.
For example, I take up the case happened in Kanazawa, Ishikawa, Japan. Some of local inhabitants protested against construction of a mosque. At a glance, it seems to be resulted from prejudice against Islam. Though it must be true that some residents had prejudice, an inhabitant said "If Moslems come here without inhabitants' understanding, more people will have prejudice against Islam."