One of the unfair realities of the professional speaking business is that celebrities get a pass when it comes time to rate the speaker after the speech.  And ratings are incredibly important to speakers because they are so essential for good buzz, repeat business, and long-term success.

I’ve seen my fair share of celebrities speak, and, as a group, they tend to be underprepared and overconfident.  Of course, there are exceptions; there are great celebrity speakers.  But all too often what happens is that the celebrity believes his or her own hype, thinks for some reason that he or she doesn’t have to prepare, and end up delivering a disorganized stream of consciousness talk all about the moment with precious little in it that’s momentous.

So it’s a dreadful speech, if we’re being honest.  But that doesn’t matter when it comes to the ratings because the audience in general is thrilled to have seen and heard the celebrity.  What they’re really rating is their satisfaction at being able to tell their friends for quite a long time afterward that they were in the same room as Celebrity Jones.

There’s a further issue with that rating moment as well.  When I was a graduate student, I worked as a research assistant for the great E. D. Hirsch, Jr., who was a longstanding advocate for classical education, standards, and teaching everyone up to a basic level of competence.  He wrote a book, Cultural Literacy, back in the ‘80s that is still in print because it touched a nerve for people who were afraid that getting too loosey-goosey about educational standards would condemn everyone to mediocrity.

He also did research on interpretive and testing validity, because he wanted to know what you could measure objectively in the subjective world of the arts and literature (unlike the objective world of the hard sciences).  That’s what I worked on with him as a lowly RA.

What he found, amongst other things, was that you could socialize people to rate an experience (like reading an essay or hearing a speech) with validity if you did two things.  First, you discussed beforehand in detail what the criteria were for your rating and got a lot of clarity about that.

Second, you only allowed for three ratings – excellent, good, poor.  One, two or three.  More than that, and you lose any chance at cohesive judgment.  You just get essentially a random response with no validity.

Apply that to the usual way speakers are rated and you see why I’m always skeptical about the results that my clients and every other public speaker who’s ever been rated anywhere obsess about.  There’s no agreed-upon criteria in advance, and the usual system is a five-point scale.

No validity.  Add to that the celebrity factor, and what you’re getting is nonsense.

If a set of criteria could be agreed upon in advance, criteria that did not include, “How thrilled am I just to be in the same room as this person?” and if the rating system could be kept to three buckets, then we could begin to develop a rigorous, objectively useful, global rating system for speakers everywhere.

Imagine that.