First, the models:

• A new model of American Idol, the main prediction methodology for finals rounds (i.e. what 90% of people care about)
• The semi-final model (when Dialidol was functional) and the post-Dialidol semi-final model
• The reasoning behind the Top 3 Tracker, used to project the Top 3 as the year went on

I am also particularly proud of the Statistical Snapshot of American Radio (methodology here, see all posts here), a large data-mining effort to quantify what songs are predominantly played on radio by format, time-period. I built some fairly sophisticated data visualization tools for these, also on Github, along with all the data.

On more Idol related issues:

• Brief description of the Twitter Sentiment methodology, code here.
• The distinct advantage of singing last, quantifying how the “pimp spot” is a very good place to be
• How (the also now-defunct) VoteForTheWorst was funny but made bad choices about who to vote for (this post ruffled some feathers)
• A quantitative look at how great Dialidol was (pre-2012)
• An expansive, 3 part series about how screen time during the audition rounds was a way for the producers to affect the contest (part 1, part 2, part 3)
• The save rule, and how it was mostly used to great effect
• Also, a link to one of my most cited posts, Americans love white guys with guitars

It wouldn’t be a website in the 2010s without some listicles:

• The most totally baffling runs in Idol history, a recounting of amazing runs of luck awful singers had
• On the flip side, the contestants who were seriously robbed

Finally, I spent many hours compiling video retrospectives of the first three seasons. You can watch them all here.

Yesterday I read the story of how Idol began from the man at Fox who was mostly responsible for it. He goes by the moniker Masked Scheduler on Twitter, though for some time it’s known that he is Preston Beckman, former scheduler at Fox and NBC before that. The story is worth a read. Beckman, for instance, had to come up with the idea of a results show that was separate from the performance show due to the fact that the US has many time zones, but Britain has only one:

Since Idol was a competition show based on viewer voting they suddenly realized that the US has four time zones (five if you include Hawaii) and that Idol could not replicate the British formula of voting and then returning to the air later in the evening to report the results. Mike and I had to explain the difference [to the producers] between network and affiliate time and all the issues involved with that. We played around with different solutions such as excluding the West Coast from participating in the voting. Finally, I threw out the idea of a second show that would air the night following the voting.

At its peak, Fox had The Death Star as 3 full hours of primetime in its weekly schedule. I don’t envy the person who tries to fill it.

What most interested me, though, was the explanation of what I would call the end of the golden era, when Fox changed the format and brought in a fourth judge

For me Season 8 was the pivotal year for American Idol. The ratings were starting to decline so there was a feeling among the powers that be that we needed to shake things up…and we did. We eliminated the boy/girl round of 24 and went back to a final 36 where groups of 12 contestants performed over three weeks. We added a fourth judge in Kara Dioguardi thus telling the world Paula Abdul’s days were numbered. After years of resistance on our part we added the judge’s save to the show. We negotiated with the producers as to how deep into the show the save could be used and we made sure that they stuck to it. Finally, the auditions, which were the highest rated part of the show, were taking on a nasty tone and the ratings were reflecting it.

I think this was a mistake for many reasons. For one thing, the ecosystem that had come up around Idol depended on many things, and discussions of Paula’s lucidity were part of the fun. The producers ought to have done anything they could to keep the original judging panel for as long as possible. Dioguardi rarely added anything to the discussion. For instance, I once went through and tabulated how the judges rated the performances for each show on a scale from 1 (being bad) to 3 (being great). Looking only at season 9, see the table below

Paula gave the highest reviews overall (no surprise there), and a small standard deviation. But Kara, while overall giving more negative reviews, had a standard deviation lower than Simon or Randy. So why have her there? Paula at least was interesting. And don’t get me started on Ellen Degeneres, who gave an overall average 2.317 and standard deviation 0.779. Ellen was a combination of reviews that were as uniformly positive as Paula’s but without any other redeeming qualities (such as forgetting how many times the person had sung).

Changing judges was trivial, though, compared to possibly the biggest mistake Idol ever made: combining men and women into the same semifinal rounds. As I’ve said in this space many times (and is borne out by the data), women face a significant disadvantage overall, and tended to be eliminated at a high rate in semifinals. The superstars and semi-stars that have come out of Idol have all been women (save for Chris Daughtry). Jordin Sparks and Carry Underwood both took time to mature, and the semi-final round rules kept them in and gave them that opportunity. By eliminating this, they shifted the contest to one that was advantageous to one subgroup of people: people who were white, people who played guitars, and people who were male.

The judges’ save similarly took out part of the fun of the show, because it muted the effect of VoteForTheWorst, a site dedicated to keeping terrible singers in the contest for as long as possible.  The judges’ save would never have allowed  Gina Glocksen to be eliminated while Sanjaya stayed in Season 6.

I also missed the theme nights, which forced the contestants to perform out of their comfort zone. Almost nothing brought me the kind of glee I got from seeing a supposed hard-rock guy have to pick an Andrew Lloyd Webber tune and perform it hilariously on stage.

For his part, Beckman ascribes the main part of the decline to be due to a critical mistake:

Was there a moment when this all could have ended differently? Yes, it was when we delayed the launch of X Factor by a year after we planned the transition of Cowell from Idol to X Factor. That gave NBC the opening to introduce The Voice that was more akin to X Factor than to American Idol. This also kept Cowell off the air for over a year. That was not the plan. Once Idol was not THE singing competition but one of three well….

I think this analysis is probably at least partly true. But allowing Cowell to leave at all seems to be the main issue. The X-Factor was never popular, and was canceled after 3 seasons, but in the meantime the audience had two shows being run on Fox (though not concurrently). I don’t think that diversifying in a market that was likely already at saturation was a very good idea.

What seems clear to me is that the show could still do some good, but needs to go away for probably a decade. Idol for years has been drawing at least partially from cast-offs from other shows (like The Voice), and has seen a disturbing trend of recruiting contestants who already have released records and have a fan-base (Jax comes to mind as someone who had an organizational effort on Twitter from day 1). The thing that the producers of Idol perhaps forgot was that good singing was always only part of the equation: those of us who relished the show loved the portions where there was bad singing, we loved the stupid disco theme, we loved the judges bickering, and we even liked the injustice of the Worsters spoilers.

The ironic thing is that I can point out very memorable moments from even recent years (particularly season 12), but the totality of it is severely nerfed from Idol’s heyday. Every reality show eventually declines and dies, and Idol had a good long run. But it’s long past time to end it. Maybe if things lie fallow for a number of years, there will be a new generation of undiscovered singers who need such a show. As it stands, I think that field is infertile. Season 12 might have been a fitting end, if I had to choose.

Entertainment Weekly is out with the report this morning:

Fox is officially closing the curtain on American Idol after its upcoming 15th season, the network announced Monday.

American Idol XV will have the same judges as the last two cycles—Jennifer Lopez, Keith Urban and Harry Connick, Jr.— along with the show’s host, Ryan Seacrest, who has been with the series from the beginning.

This shouldn’t be a shock to anyone. American Idol’s ratings have cratered due to a number of reasons. First is the competition from other shows that do the same thing, some would say better [NB: I have never watched The Voice or America’s Got Talent]. Second is the waning of the early 21st century reality show ecosystem, as cable and network dramas have eliminated the empty rerun season that Idol was originally envisioned to fill. The show has also become much too expensive to run: it requires a full-time auditorium, the salaries of three star judges, recording studio time, plus the audition tour. Fox reported before this year started that American Idol was no longer turning a profit.

I have to say, I’m puzzled as to what Fox thought would change this year. If the show was already unprofitable, was this the team to turn it around? Could anyone turn it around at this point?

Fox has also signaled that the final season will be a nostalgia tour, saying “the final season will pay tribute to the show’s previous years.” That’s an admission in and of itself: the previous years were just better. The judges were harsh and not constantly fawning over the contestants. They steered the competition, rather than letting a pure democracy rule. But that all ended the day Simon Cowell left the show on an ill-fated attempt to bring the X-Factor to the US, a show that flopped spectacularly. The judges panel never recovered.

It remains to be seen whether the other shows of this ilk will go by the wayside as well. Many commenters at TvbytheNumbers predict that all of the singing reality-shows are on their way out, and maybe they’re right. It is possible that at some point you exhaust the nation of all its talented, undiscovered artists. This is particularly true when there are a number of new avenues for distributing demos, such as Youtube, which, I will remind the reader, didn’t even exist before about the 4th season of American Idol. Indeed, several singers in recent years were from other shows, and cut Idol contestants have appeared on the other shows as well. It’s all beginning to look like bottom-of-the-barrel.

So that’s it. The Death Star goes out not with a bang, but with a nostalgic whimper. We have one more year of Seacrest et al, and then finis. I myself am hoping for a return of Brian Dunkleman before the end.

Yeah, no idea. On a historical basis, there’s no way to say what will happen on Tuesday. The people all had a well-rated best performance (Back Home for Nick, My Immortal for Jax, and Earned It for Clark), though Jax was the highest-rated.

There have been some irregularities on the Votefair poll (the runner, Mr. Fobes, says that a Canadian was spamming his site with votes). That being said, Votefair isn’t particularly predictive at this point anyway. But for what it’s worth, Nick is the lowest rated on that service by a comfortable margin. The values printed above are the values from today, normalized.

If you want my subjective opinion, Clark and Jax nudge out Nick for a finale.

Of the three remaining contestants, Twitter has the most negative view of Clark, with Nick in close second:

Jax is well ahead of the others on Twitter. Note that Bing Predicts (which has access to the most actual data by a mile) said Jax was in danger tonight. They have not seen fit to present the reasoning behind that.

Rayvon is heavily favored to go home. More than 7 points outside the margin of error. It would be a shocker if he weren’t eliminated.

Nick has sort of risen to the front of the pack, but not by much. Nobody touches Clark on Votefair (but then again, neither did anyone get very close to Angie Miller, third place in S13 … that index is not particularly predictive at this point).

Rayvon has, for the first time since the Top 11, the highest negative sentiment on Twitter: