Husband probably reminds horse of someone who roughed horse up. It definitely happens. Years ago a 'breaker' who had worked with one of the horses at a barn where I was, showed up. That horse took one look out the front of her stall and tried to climb the back wall of her stall to get away from him. And I've heard lots of other accounts of a similar nature over the years. They remember how people smell and look.