My Jeopardy! Strategy Advice To IBM - Part 1

 Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

RULES, FORMAT, UNKNOWNS

I got the impression that the final rules and format of the Jeopardy! Challenge have not been determined, or maybe that information is known but no one was authorized to share it with me.  I’ll assume the former.  The problem is a lot more difficult if you don’t know what you’re preparing for.  Will the public get to see a one-day televised match between Watson and two humans, or will Watson be in a multi-day tournament, like the standard Tournament of Champions, or even the huge Ultimate Tournament of Champions?  If there is a tournament, who are the other players and pairings, and does Watson have to win his way into the finals, or will he be given a bye?  Will all 60 clues on the board be revealed, or will some clues be left unused, depending on the time available?  If the latter, is there a fixed precise amount of playing time for each round, which Watson can keep track of, or does it depend on the whim or gut feel of the show’s director?  What is the intended difficulty level of the questions?  Will they be written at the same level as the regular episodes, or will they be more difficult than usual?  Will good players know 60% of the answers or 90% of the answers?  All of these factors and many others could have a significant impact on how the game is played and which strategies will be most effective.  At this point, Watson does not appear to have a specific goal, other than to play Jeopardy! well.  Watson seemed content to settle for a tie in some cases when it would have been no more risky to go for a win.  This will certainly annoy the Jeopardy! purists, and if that strategy is used in a tournament, it could result in Watson’s elimination from the next round.  There are also some tournament situations where the goal is not just to win, but to attain the highest possible score.  Depending on how the Jeopardy! Challenge is structured, the specific goal will affect the strategy.

My Jeopardy! Strategy Advice To IBM - Introduction

On September 17, 2010, I spent the day at the IBM Watson Research Center in Yorktown Heights, New York.  I met most of the research team for the Jeopardy! Challenge and discussed several aspects of game play optimization with the strategy team.  I spent most of that day observing six sparring matches between Watson and each possible combination of two human opponents, from a group of four former contestants IBM invited in for the day.  Each of them had been successful on Jeopardy! and competed in the Tournament of Champions.  I took a lot of notes that day and spent a couple weeks thinking about the challenges involved as I wrote a report with my recommendations to IBM.  On October 8, 2010, I sent my report to IBM.

I was told that my report was well received and that it prompted a lot of discussion within the Watson team.  We had some follow up phone calls and e-mails to discuss specific strategy topics, especially after it was revealed that the Jeopardy! Challenge would consist of two matches in the format of a tournament final, with Watson facing Brad Rutter and Ken Jennings.  In early January 2011, I was invited back to the Watson Research Lab to play eight practice matches against the final version of Watson, as a final tune up.  I was also in the live audience that watched the taping of Watson's televised matches against Ken and Brad.

On Saturday, July 9, 2011, I will join Jeopardy! legends Bob Harris and Ken Jennings on a panel discussion at the Trivia Championships of North America at the MGM Hotel in Las Vegas ( http://tcona.com/ ).  One of the main topics will be Jeopardy! strategy and how human contestants on the show can learn from the research done by the IBM Watson team.  To provide background and prompt further discussion, I am using this blog to publicly release my initial advice to IBM, based on what I knew and observed on my initial visit last fall.  Since then, I have learned that IBM's researchers implemented many of my ideas in a far more sophisticated way than the basic concepts I suggested in my report.  More than three years of work had already been done before I was asked to share my ideas, so most of my input independently confirmed what was already being done.  However I was pleased to see that one of my ideas for Daily Double wagering was added to the final version of Watson, and some of my other ideas would be too subtle to know for sure whether they were used in Watson or not.

The next several posts in this blog will be the text of the report I sent to IBM on October 8, 2010.  It is divided into separate parts for each topic.

Is There A Lawyer In The House?

The buzz has been building up for weeks.  The big event is almost here.  Today is the day before the much anticipated men-versus-machine Jeopardy! episodes.  Will IBM's feisty upstart Watson computer defeat two legendary Jeopardy! juggernauts, Ken Jennings and Brad Rutter?  Or will Brad handle Watson as easily as he dispatched Ken in the Ultimate Tournament of Champions?  Or will Ken reclaim his title as the show's biggest money winner?  In just three days, the world will finally know.  Actually, a few of us already know, because we were there when it happened.  But we agreed not to talk about it.

I've spent countless hours reading and watching a wide range of traditional media and internet coverage of this event.  Some of it is very insightful and informative, especially the many IBM videos at http://www-03.ibm.com/innovation/us/watson/ and the links Watson has posted on his Facebook and Twitter pages.  (Ever since I competed against Watson, I've mostly thought of the big guy as "him" rather than "it.")  I even signed up for a Twitter account, just so I can keep up with what the three contestants are saying.  On the other hand, I have encountered hundreds of articles and internet posts from opinionated people who really don't know what they're talking about.  I guess that's to be expected, since most people aren't experts in either artificial intelligence or Jeopardy!, let alone both.

My own experiences with Ken, Brad, IBM, and Jeopardy! give me a unique perspective that I think is worth sharing.  That's where this blog comes in.  Over the next several days, I plan to post some comments and observations about watching Watson play last summer, meeting the development team, suggesting new and improved strategies, playing against Watson, and watching Brad and Ken play against Watson.  Don't expect to read any dirt or gossip.  I have nothing but respect and admiration for everyone involved, and I am grateful for the opportunity I was given to contribute to this project.

There's just one little complication.  I had to sign three different non-disclosure agreements that limit what I am allowed to say.  I certainly have no problem with that.  My long history as both an IBM employee and a game show contestant has proven that I'm good at keeping secrets.  Fortunately, there's very little that remains confidential.  In its publicity campaign, IBM has been systematically revealing most of the inside information that I've been protecting.  As far as I know, the only thing left that I can't talk about is the actual match with Rutter, Watson, and Jennings, until the third episode airs on Wednesday.  I am not a lawyer, but that is my interpretation of the document below.

With that introduction, I conclude my initial blog entry.  I hope to enhance your enjoyment of the Jeopardy! IBM Challenge.  I welcome your comments and questions.

The IBM Non-Disclosure Agreement