In his character profile it says he stands out "for obvious reasons", which I've interpreted to mean his Scottish-ness, his size, his loud accent, and his army gear. In Wellington Wells, conformity is the only acceptable social standard so he is de facto hated for these reasons. When it comes to the army guys, however, I think they hate him because of what he did while he was in the army; I'm not 100% sure of what that was.