People look back on those days through a thick veil of nostalgia, but life was hard if you were anything other than a rich, powerful white male