The problem lies between freedom of thought and moral.
When it does Jewish people harm, it's the problem of moral. On the other hand, those who like Nazi-related cultures may feel disappointment, if the disputing cafe is forced to close.
But, when we take it into consideration that Nazi is under a taboo in some countries, the former problem seems to be more serious than the latter one.
