It's important for us to study histories of our destinations before visiting there. But, when it comes to droping in at facilities we found by accident, it wound be impossible to do it.
Though I don't know when they decided to visit the shrine, they are likely to have been there without knowing the history of Yasukuni shrine. It means they wouldn't have political meaning for visiting there.
So, it would be better to advise them not to go there. But, it won't be better to blame them for visiting.