The very thing that attracts you to someone can end up putting you off.