QRANKs For The Memory

I received an unexpected honor today.  I was one of the multiple choice responses on the daily QRANK online trivia quiz.  QRANK is a popular game for Facebook, iPhone and iPad.  It usually asks a broad range of general knowledge questions, but today was a Jeopardy! theme quiz, in honor of Alex Trebek's 71st birthday.

Okay, so my name wasn't the correct answer, but at least they spelled it right. I wonder why the other three people are referred to as "Who," but I got the special phrasing, "What is Ed Toutant?" Maybe QRANK is confusing me with Watson.  I get that a lot.

Thanks to Judy for telling me about this and to Myron for sending me this screen capture. If you haven't tried QRANK yet, check it out at
http://www.qrankthegame.com/

This was a 400 point QRANK question on July 22, 2011.  I wonder how many people got it wrong.

My Jeopardy! Strategy Advice To IBM - Part 9

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

IBM IMAGE, OBJECTIVES, AND PUBLIC RELATIONS

It will be important to manage public perceptions and expectations.  If Watson wins, many people will think, “So what?”  They don’t realize how complicated it is to play Jeopardy!  They probably think it’s harder to play chess, and IBM already solved that challenge years ago.  Most people can watch Jeopardy! at home and answer some of the questions, so it seems pretty easy.  Far fewer people think they can play chess well, so they may assume that chess is a harder task.  The presentation that David Ferrucci gave before the matches opened a lot of eyes and minds.  It clearly explained the many processes that must be mastered in order to play a good game of Jeopardy!, and how those abilities can be extended in the future to solve more critical problems in the real world.  IBM will need to widely publicize those points before the Jeopardy! Challenge broadcast, to help the public appreciate Watson’s accomplishments and have a positive view of the benefits that his technology can bring to their lives.

Based on what I saw, the outcome of the Jeopardy! Challenge is hard to predict.  Watson plays a strong game, but he has shown that he can be beaten, at his current level of performance.  IBM needs to be prepared for a range of possible outcomes.  What is the message (or PR spin) if Watson loses badly, as he did against Alan Bailey?  (Yes, I know that it was really a close match, but to the public, a final score of $60,800 to $1,000 sounds like a rout.)  What is the message if Watson loses a close match, or ties, or wins a close match?  And what if Watson totally humiliates his human opponents?  Is that the best possible outcome, or would it make Watson seem too much like a villain?

The general public will be rooting against Watson.  It’s inevitable that humans will cheer for their own kind.  People naturally like an underdog, or someone who has to overcome difficult obstacles, or who has a pleasant personality.  It may be possible for Watson to attract a lot of supporters if he can be somewhat anthropomorphized, imbued with some of those same human characteristics.  There are many ways to set the stage in the weeks leading up to the Jeopardy! broadcast, and even during the show itself.  I loved it when Watson expressed himself the way a real contestant would, saying things like “please,” “let’s finish the category,” and “let’s make it a true Daily Double.”  I think the public will appreciate that, and I believe there are opportunities to expand his repertoire of human-like phrases.  Watson would be even more impressive if he appeared to have a personality and a sense of humor.  I think it would be hilarious if he ever had a chance to say, “I’ve always wanted to say this, Alex, let’s make it a true Daily Double.”  If the show follows the usual format, there will be a break during the first round in which Alex chats with the contestants.  I’m not sure if that would work for Watson, but maybe he could use the opportunity to show a short video of how he became so smart and all of the obstacles he had to overcome along the way.  I don’t know who Watson’s opponents will be, but I assume that one of them will be Ken Jennings.  Ken is a good guy, but he is not universally loved by the general public.  If Ken can be portrayed as an unbeatable superhuman Juggernaut, Watson may pick up a lot of the underdog support.  Some people would just like to see Ken lose, regardless of the opponent.

I think it would be helpful if Watson had more of an instantly recognizable image, which could become an important brand or trademark for the IBM company.  I’m sorry if this hurts anyone’s feelings, but the hologram that was used to represent Watson did nothing at all for me.  I didn’t hate it, but it didn’t leave any lasting impression.  It didn’t attract the eye or grab much attention.  I don’t remember it and I don’t think I’d recognize it if I saw it again today.  Technical nerds and the general public relate more to a fun logo than to an abstract shape.  Since the goal is to show that Watson can think like a human, I think Watson’s icon or avatar should incorporate some human qualities.  With the right image, it’s not far-fetched to think that Watson could become a cult hero, symbolizing a new era in which computers work collaboratively with their human colleagues on everyday tasks.  He needs a friendly, likeable image that matches that role, an image that techies will want to wear on t-shirts along with their Google doodles, Linux penguins, and Apple logos.  It’s all about perception.  If a great iconic image for Watson catches on with the public, it would make all of IBM seem like a much cooler company to work for and to buy products and services from.

My Jeopardy! Strategy Advice To IBM - Part 8

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

MISCELLANEOUS OBSERVATIONS

It appears to me that Watson’s biggest disadvantage is his lack of any speech recognition or audio sensing.  There are many occasions in a typical Jeopardy! match when it is essential to know what is being said.  The most common example is during a rebound opportunity.  Since Watson doesn’t know what his opponent’s answer was, he might buzz in and give the same wrong answer.  Occasionally there are clues or categories that are multiple choice, usually with three different choices.  If the category is Executive, Legislative, or Judicial and they ask which branch a particular government official serves in, Watson will look stupid if he can’t pick up the free money after his opponents both answer incorrectly.  There are also many times when Alex Trebek pauses to think or to get a ruling on an answer that might be acceptable.  That’s a huge hint to the other players that the answer is something close to what the contestant said, but Watson won’t be able to capitalize on those opportunities with his current design.

I observed one match in which Watson had a big lead going into Final Jeopardy! and didn’t even attempt to answer the question.  I thought that was poor form and not very sporting, even though it didn’t change the outcome of the match.  The whole point of the Jeopardy! Challenge is to see how well Watson can play, so the public is going to want to see his best effort, even when his confidence is low.

In two of the six matches I saw, Watson led going into FJ, but did not have a lock game.  Alan Bailey was in second place both times.  In both cases, Watson’s FJ wager would have conceded a tie to Alan if he was right and went all in on his bet.  This is a very unorthodox strategy that seems ill-advised.  Alan learned from experience and bet it all in the last match when he saw that Watson is programmed to play for a tie in non-lock games.  The IBM strategists apparently think that a tie is as good as a win.  I don’t think the public will see it that way.  A tie will be seen as a moral victory for the human.  Someone on the IBM team told me that playing to allow a tie slightly improves Watson’s chances of not losing outright, but I have not seen a plausible hypothetical case where that would be true.  In fact, if the word leaks out (which seems likely) that Watson offers ties, contestants will make unconventional FJ wagers to take advantage of that weakness and Watson will win less often than he would with a more traditional wager.

I assume the IBM team must be familiar with the J! Archive.  It includes a FJ wagering calculator for the conventional strategy that most good contestants agree with and try to follow.

http://www.j-archive.com/wageringcalculator.php

The site also has a glossary of unofficial terms and strategies, summarizing the consensus thinking of a lot of past contestants and serious analysts of the game. 

http://www.j-archive.com/help.php

I believe the most important of these is the Two-thirds Rule, which explains how to bet FJ when you are within two-thirds of the leader’s total.  Almost every week, contestants lose games they should have won, simply because they did not know this betting tip.  It happened again today (10/07/10), when the second place contestant should have won on a triple stumper, but she lost by betting too much.  There are also some worthwhile post-game discussions every day on the official Jeopardy! message board.  Very little escapes the attention of those devoted and generally knowledgeable fans, and any unusual incidents or strategic errors are usually thoroughly analyzed. 

http://boards.sonypictures.com/boards/forumdisplay.php?f=11

It occurred to me that Watson could test the value of different strategy variations by partitioning himself into three separate Watsons, each running a different strategy and then playing many simulated Jeopardy matches against different versions of himself.  Reset his memory each time so he doesn’t remember seeing the clues before.  If one version significantly outperforms the others after hundreds of simulations, that would tend to prove which approach works best.

Many of the tips I have described here could be used not only by Watson, but also by human contestants.  But for a human, the game progresses so fast that it seems like a blur.  Any plan to use a wide array of incremental tactics would become impossible for a human to implement without distracting from the main game.  One of Watson’s strengths is that he can be programmed to take advantage of many small cumulative factors that may add up to a meaningful edge.  Unlike humans, Watson doesn’t get distracted.  Of course, any changes should be thoroughly tested prior to the Jeopardy! Challenge, to assure that no new bugs have been introduced.

My Jeopardy! Strategy Advice To IBM - Part 7

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

HOW TO BEAT WATSON

When I visited IBM, I was asked what strategy I would use to compete against Watson, or how I would play differently against Watson.  From what I have seen, Watson is already good enough to beat elite contestants most of the time.  I assume he will continue to improve between now and the televised match(es).  I don’t think contestants like Brad Rutter or Ken Jennings are a whole lot better than some of the opponents Watson has already faced, so I would expect nearly as good results against them.  If I got to play against Watson, I would start with the assumption that Watson is heavily favored, and be prepared to take greater risks than usual.  I would expect Watson to beat me to the buzzer almost at will, so my best hope is to score points in categories where Watson is weak and won’t try to buzz.  I would try to identify which categories on the board are most likely to cause him trouble, especially the humorous subjects involving word play, like Daffynitions or Before and After. 

Watson obviously knows the strategic importance of Daily Doubles, used either offensively or defensively.  I would prefer to wait and find them later in the game when they are worth more, but it appears that Watson likes to find them early to prevent his opponents from getting them first.  So I know that I can’t afford to just play passively from the top to the bottom of the columns.  I noticed that Watson would often select the top clue in a column, essentially tasting it to see if he likes it, then go straight to the fourth clue, apparently searching for a Daily Double.  He gets a sense of what the category is all about before going to the high dollar value clues.  I would try to thwart Watson’s strategy by starting at the bottom and working up.  I think that could help me grab some of the high dollar clues while Watson is still learning what the category is really about.  I would hunt for Daily Doubles, partly in self-defense, because if I don’t find them, he will, and he tends to bet aggressively on them. 

Watson doesn’t have to worry about emotions and frustrations, but his opponents do.  I noticed a lot of frustration in at least one of the contestants in the practice matches.  That is understandable, but very unproductive.  I would try to prepare myself to expect the worst and not let it bother me when Watson keeps beating me to the buzzer.  I’d just keep plugging away and hope for a lucky break.  If my best laid plans were failing miserably and I found myself falling behind, I’d switch into desperation mode and start taking bigger chances and making more aggressive bets.  Being a great player is the best strategy, but sometimes great players can be beaten by a stroke of dumb luck. 

If Watson has a commanding lead, I would expect him to focus on defense and play very conservatively, not risking any big mistakes that could put his opponents back into contention.  If I were trailing by a large margin, I would hope that Watson’s conservative play might allow me more opportunities to buzz in than if he were playing more aggressively.

My Jeopardy! Strategy Advice To IBM - Part 6

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

BUZZER PERFORMANCE

A lot of Jeopardy! players will tell you that the game winner is determined by the buzzer, since most of the players know most of the answers.  There is no doubt that the buzzer is a huge part of the game.  Watson’s buzzer appeared to perform well, but I was left with a lot of unanswered questions about it. 

I assume, but was never explicitly told, that Watson receives an electronic signal to indicate that his buzzer is now active, and that it arrives at the exact time that the light is switched on.  In theory, Watson could be given a handicap by delaying his activation signal to give humans a better chance of buzzing in, but I assume that is not being done.  Since the signal travels at roughly the speed of light, the only real delay in activating Watson’s buzzer is the time it takes for the solenoid to depress the buzzer button.  I would be very interested in knowing how long that motion takes.  Is it just a few milliseconds, or more like a typical human reaction of 200 milliseconds? 

In a buzzer race between Watson and a human opponent, the only way the human can win is if he has nearly perfect timing and anticipates the lights to buzz in during the brief window when Watson’s solenoid is still being activated.  The shorter that window is, the less chance there is of a human beating Watson to the buzzer. 

My very first job at IBM was to develop an electromagnetic print hammer for a daisy wheel printer.  I did a lot of theoretical calculations, exotic alloy selection, instrumentation and testing to produce the fastest possible actuator for that application.  The print hammer could accelerate a steel projectile from rest to maximum velocity and travel about 5 millimeters in 2 milliseconds.  That seems pretty similar to what Watson is trying to do to actuate his buzzer.  Is Watson’s buzzer activation time in that same range?  If not, I think it could be, with an optimized design.  As long as it’s less than about 10 milliseconds, I wouldn’t worry much about it, but if it’s a lot longer than that, it would be an opportunity for improvement.

I noticed that Watson sometimes hit his buzzer only once, and sometimes hit it multiple times in rapid succession.  I asked a couple people why that was, but never got a clear answer.  If Watson is always reacting to the buzzer activation signal, I can’t think of any situation where more than one buzz would be necessary.  Am I missing something here?

My Jeopardy! Strategy Advice To IBM - Part 5

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

GAMESMANSHIP AND INCREMENTAL STRATEGIES

There are several subtle acceptable variations of game play in a typical Jeopardy! match that are commonly used, sometimes using technicalities and loopholes in the rules to gain a competitive advantage, and sometimes just trying to speed up the game.  These little things can sometimes make a difference in the outcome of a game.

Use the full time allowed to respond

This was what surprised me the most about how Watson played.  Watson usually responded immediately after he was prompted by the host, even though contestants are allowed up to five seconds to begin their answer.  I was surprised that those five extra seconds (okay, let’s say four seconds, just to be safe) were not being used to keep crunching data and improve  the accuracy of the responses.  This was especially noticeable on the Daily Double questions, since Watson did not intentionally buzz in already thinking that he knew the answer.  Alex Trebek allows far more time to answer a Daily Double.  It is common to have a contestant think for six or seven seconds before Alex gently prompts “I need an answer.”  Not using the full time available to respond seems like a lost opportunity.  Ken Jennings was famous for buzzing in, confident that the answer was somewhere in his head, and then taking a few seconds to extract it.

Related to the above, on Final Jeopardy! clues, you have 30 seconds to respond, so maybe that’s more time than Watson would ever need.  Still, I think Watson could get a great head start as soon as he sees the FJ category.  There is typically about five minutes of break time while contestants make their wagers and prepare for FJ.  During that time, it seems that Watson could be narrowing his focus to concentrate his search on the most relevant material.  For example, if the FJ category is Baseball, Watson could partition his database so he won’t need to search areas like African Explorers or Nobel Prizes (though you can never be sure about possible crossovers between subjects).  If that approach is effective, it should work nearly as well during the first and second rounds of Jeopardy!, with ample time to pre-sort some data on each subject before the clues are selected.  Maybe Watson already does that.  I don’t know.

Faster or slower game pace

I don’t know if Jeopardy! is committed to revealing all 60 clues on the board, or if they will leave some clues unused if time runs out.  If some clues are left unseen, that will help whoever is leading and hurt whoever is trailing.  There are many ways to slow the pace of the game, if that is desired.  Usually the best contestants play faster and complete the board, but one recent contestant on the show, Roger Craig, who broke Ken Jennings’ single day win record with $77,000, tended to pause and hesitate a lot.  I don’t know if he did that intentionally, but he left a lot of unused clues on the board, including 9 unused on the 09/17/10 episode. 

Board bounce when selecting clues

Some viewers find the practice annoying, but there is an increasing tendency for contestants to bounce around the board and not select clues sequentially from top to bottom.  Sometimes they are hunting for Daily Doubles, but a few contestants intentionally try to keep their opponents off balance by continuously changing categories.  The theory is that the person selecting has a little more time to think of the answer because he knows which category he will select, whereas the opponents will need a little more time to adjust.  It’s not clear how effective the bounce is against regular opponents, but it seems obvious that it would be more disconcerting against humans than against a computer, so it might be an effective strategy for Watson to use.

Shorten or paraphrase the category name when selecting clues

It is common for contestants to select the next clue by saying, “Same category for $1200, Alex” or, “Let’s finish the category, please.”  This can serve two purposes.  It might simply be an attempt to speed up the game, or it might be a conscious effort to avoid reminding the opponents of the specific category name.  Contestants often buzz in confidently with a response, only to find that it doesn’t fit the category name.  Some of the smartest (and most devious) players will intentionally leave out a key word when they select a clue, hoping their opponents will forget.  This sometimes works.  It seems a little sleazy, but it’s allowed, and sometimes even encouraged by the producers, to avoid wasting the several seconds needed to enunciate a complicated category name.  For example, a player might say “Stallone Movies” to select “Stallone Movie Sequels,” or say “Nursery Rhymes” to select “Nursery Rhyme Towns.”

Watson’s temporary initial buzzer advantage

The best Jeopardy! players do not wait for the lights to come on and then try to react and press their buzzers.  (I wish I had known that when I played in 1989.)  Good players try to anticipate the “go light,” by studying Alex Trebek’s speaking pace and trying to buzz in after the last syllable, when an unseen human activates the lights.  It typically takes several clues before players feel in sync with the light.  This is when Watson has his maximum buzzer advantage, because he is fast enough to react to the light signal and activate his buzzer, and does not need to anticipate the light (I assume).   Therefore, Watson should exploit his temporary increased buzzer advantage at the beginning of the game by selecting the clues with the highest value.  If Watson has a temporary advantage while the opponents are still working on their timing, there’s no sense in wasting it on $200 questions.

Don’t include a person’s first name in your response

This is very basic strategy, Jeopardy 101.  Never provide more information than what is required, because part of it might be wrong.  I was surprised to see that Watson usually (but not always) answered with both a first and a last name.  This seems pointlessly risky, since Watson might ruin an acceptable answer by adding an unneeded first name.  No one knows this better than Ken Jennings.  The only reason anyone knows who he is today is because he applied this rule in his very first game.  His FJ clue was: “THE 2000 OLYMPICS: She’s the first female track & field athlete to win medals in 5 different events at a single Olympics.”   Ken had a vague idea, but wasn’t sure of the first name, so he wrote: “Who is Jones?”  If Ken had guessed wrong about Marion Jones’s first name, none of us would have ever heard of him and this Jeopardy! Challenge probably wouldn’t be happening.

Try betting irregular dollar amounts

One of the most challenging parts of Jeopardy! for many players is the need to do quick math in their head under pressure, especially when making a bet.  It is always easier for humans to do math that involves only round numbers.  Unlike humans, Watson can’t get flustered and forget to carry the one during addition.  So Watson should exploit his inherent math superiority by never using a round number on a Daily Double wager.  For example, instead of betting $8,000, try betting $7,837.  This may give viewers the impression  that Watson’s thinking is very precise, but the real motivation is to make the math more difficult for his opponents when they have to make a wager.  Important Jeopardy! tournament games have been lost due to math errors.  That is how Brad Rutter beat Michael Rooney in the UTOC.  Michael told me he would have won that match if he had just done the FJ math right, and Brad displayed very weak math ability when he competed on Grand Slam.

My Jeopardy! Strategy Advice To IBM - Part 4

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010


DYNAMIC STRATEGY ADJUSTMENTS
I believe that in order to maximize the chance of winning, any contestant (human or computer) must at all times have a sense of where he stands and how likely he is to win, so he can adjust his tactics to fit the game situation.  When a contestant knows he has very little chance of winning, his best strategy is to be aggressive, take huge risks, and hope for some good luck.  When a contestant knows he is the prohibitive favorite, his best strategy is to play conservatively and to eliminate all risks that could produce an unlucky loss.  With enough self-knowledge, possibly augmented with knowledge of the opponents, it is possible for each contestant to continuously update their projected chances of winning the match, then make strategy adjustments to fit their situation.  For human players, their game status rating will be very imprecise – probably no more than a few levels, such as 1) I’m kicking butt, 2) I’m holding my own, and 3) I’m so screwed.  For Watson, his game status rating (let’s call it GSR) could be based on a much more complicated algorithm that takes many factors into account.  Watson’s GSR might be expressed as a percent confidence that he will win the game.  After each clue, Watson could recalculate his GSR and decide whether to make strategy adjustments.  If it’s late in the game and Watson’s GSR is 95%, he might want to play conservatively, making smaller bets, not buzzing on questions if he’s not sure, and maybe even stalling for time.  If Watson’s GSR is 20%, he probably needs to start taking more risks, hunting for Daily Doubles, and selecting the highest value clues. 

Most Jeopardy! contestants have a gut feel for their own GSR even before the game starts.  They might know from watching at home how well their knowledge compares to typical contestants.  So they arrive at the Sony studios hopeful but realistic about their chances, some more confident than others.  Much of the outcome depends on the categories and whom they are playing against.   Defending champions probably feel more confident than challengers, but it’s mostly speculative until the game gets underway.  Watson can be more certain about his GSR before the game starts.  He has proven that he can beat some of the best players, most of the time.  I assume that Watson will know who his opponents are long before the match is played, so he can spend a lot of preparation time studying their strengths and weaknesses.  He knows that he is fast on the buzzer and won’t be nervous under the lights.  So Watson can have a pregame GSR that reflects his confidence rating going into the match, just like the sports betting books in Las Vegas do.  Maybe Watson learns that he will be playing against Ken Jennings and Brad Rutter, so he churns the data and projects himself the 43% favorite, with Rutter at 35% and Jennings at 22%.  Then when the game board is revealed and Watson sees there’s a category called Prince Song Lyrics, he adjusts himself up to a 45% favorite.  Then after each clue, Watson would recalculate his GSR, making corresponding strategy adjustments as needed.

I think it would be very entertaining to display Watson’s dynamic GSR projections throughout the game, so viewers could gain insight into his thoughts and strategy.  This is very similar to the popular televised poker tournaments, where each player’s hole cards are revealed, along with their continuously updated probability of winning the hand.  It would be interesting to watch the momentum changes and the impact of bold moves on the projected outcome.

In conjunction with the GSR probabilities of winning, Watson could generate a continuously updated projected final score for each contestant.  I’m not sure if it would have strategic value or just be used for entertainment purposes, but Jeopardy fans love statistics, so it would be another way to impress them.

My Jeopardy! Strategy Advice To IBM - Part 3

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

WATSON’S KNOWLEDGE OF HIS OPPONENTS

If IBM knows who Watson’s potential opponents will be, it may be possible to study each player’s performance history and other biographical details, to assess their strengths and weaknesses and develop a customized strategy for each opponent.  A lot of information can be found online about the legendary Jeopardy! players.  I assume IBM is very familiar with the amazing online database at the J! Archive, which can be found at http://www.j-archive.com/.  Every game that Ken Jennings and most other top contestants have ever played is meticulously transcribed there.  Watson can prepare for his upcoming matches by analyzing each contestant’s knowledge base and style of play.  With that information, Watson can play the game slightly differently against each player.  If Watson determines that Ken Jennings is especially good at wordplay categories and especially weak in sports, he could hunt for Daily Doubles in Ken’s strongest categories to prevent Ken from cashing in on those, then switch to sports questions after taking the DD out of the game.  Similarly, if Watson knows that Ken grew up in South Korea, and that Brad Rutter grew up in Pennsylvania (and both of those details are true, by the way), then Watson could try to prevent them from scoring big in categories about “The Korean War” or “The Keystone State.”  Watson has the potential to know much more about his opponents than they know about him, so that could be used to gain an advantage.

Ken Jennings shares many details about his personal life online.  You can get a pretty accurate picture of the guy by reading his daily blog posts from the past four years.  It would be easy to compile a list of the places he has been, books he has read, his hobbies, interests and other insights into what he knows and cares about.  He even blogged recently about how TV quiz shows never write questions based on what a specific contestant would or would not know.  He admitted that his personal Achilles heel would be questions about Fashion.

http://ken-jennings.com/blog/?p=2157

One subject that I’m sure Ken excels in is Movies.  If Ken were to play against Watson in a special all-movies theme episode, I’d bet the house on Ken.  Here is an old list Ken posted online, showing all the films he had seen in his life, as of several years ago.  Unfortunately, he hasn’t kept it updated, but it shows how dedicated he is to learning all there is to know about films.

http://jenningsbaby.8m.com/

There are several other web sites that have statistical data about some players’ strengths and weaknesses, since most of them participate for fun in a variety of different quizzing formats.  For example, there is a spreadsheet with category results for each participant in the annual World Quizzing Championships, which is an intense written quiz that started in Europe and has attracted many top American game show contestants.  Check it out here on this Belgian site: http://www.iqa.be/.  If you look at the full results for 2010, you’ll see that Jerome Vered finished in 27^th place, out of 1175 players worldwide.  Brad Rutter finished in 140^th place, and I finished in 153^rd place.  A quick look suggests that Brad is somewhat weak in Sports and Science, and relatively strong in Culture and Lifestyle.  Jerome’s pattern is similar, with higher scores overall, and a notable strength in History.  Ken Jennings did not register for the World Quiz, but he took it unofficially at home and reported his results on his blog: http://ken-jennings.com/messageboards/viewtopic.php?p=57985#57985

Ken’s score of 118 would have put him around 60^th place.  He showed weakness in the Sports and World categories, and relative strength in Culture and Entertainment.

Ken, Jerome, and Brad have taken the World quiz and similar written tests of overall knowledge several times, and their results are consistent – Jerome knows more than Ken, and Ken knows more than Brad.  And yet, when they competed against each other in the finals of the Jeopardy! Ultimate TOC, the results were just the opposite.  Brad won convincingly all three days, and Ken played consistently better than Jerome.  A lot of people assume that Brad was just quicker on the buzzer, and there may be some truth in that, but I believe the main reason why Brad won was that Ken’s and Jerome’s knowledge bases overlapped each other a lot, whereas Brad knew more things that neither Ken nor Jerome knew.  When there is a three-way competition, the two people who know the most might lose to the person whose knowledge is more unusual.  The same principle holds in the Watson matches.  When I spoke to the IBM strategy team, they acknowledged that Watson has a much better chance to win against two human opponents than it would against one, because Watson will get most of the questions that are computer friendly, and the two humans will have to split the questions that are human friendly.  Since Watson fares best when his two opponents are most similar to each other, he may be at a slight advantage if his two opponents are of the same sex, or the same age range.  That would tend to make their knowledge bases overlap the most, allowing Watson to answer a greater share of the questions.

My Jeopardy! Strategy Advice To IBM - Part 2

Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

WATSON’S SELF-KNOWLEDGE

All good Jeopardy! contestants must have a broad range of knowledge, but one of the most important requirements is to know what you know, and to know what you don’t know.  Without awareness and quantification of one’s own strengths and weaknesses, no strategy will be very effective.  With a general understanding of the probable outcome of different choices, one can begin to play the game in a way that will maximize the expected result.  Back when I was trying to become a contestant, I transcribed two years of Jeopardy! episodes from my VCR and kept my personal accuracy statistics for each topic, each clue value, each Daily Double, each Final Jeopardy!, and the location of each Daily Double.  I also kept statistics for all contestants on the show, to see how I compared.  By the time I got the call, I had a good awareness of my probable chances on each clue.  I tried to bet accordingly and choose the most favorable clues, depending on the status of the game at the time.  I assume that Watson must have done this already, to an even greater extent than I was able to do.  There are thousands of Jeopardy! game transcripts available in the unofficial archives that could be used for that purpose.

What I found from tracking my own statistics was that the questions certainly got more difficult toward the bottom of the board, though not nearly in proportion to the dollar values of the clues.  I found that the clues in the fourth and fifth rows were almost equally difficult.  (This was 20+ years ago, so I don’t know if that is still true.)  It is therefore possible to generate an expected value for each clue on the board, which can be (over)simplified as: clue value x % accuracy x % chance of being first on the buzzer.  Each expected value could be further adjusted based on the clue’s subject and the probability of it being a Daily Double.  When this is done, it becomes obvious that the game is won or lost in the bottom half of the board, since that’s where most of the points are.

The clues are written with the intention of being increasingly difficult as you move to the bottom of the board.  I think that is true for all human contestants, since the clues are written from a human perspective and the clue values are assigned based on what humans would know.  I can’t say for sure, but I suspect that it is not as true for Watson, since he uses different thought processes to generate his responses.  In many cases, Watson might find a $2000 clue just as easy as a $400 clue.  For example, if Watson’s database includes the lyrics to all songs by Prince, identifying an obscure song is no harder than identifying a big hit.  If my theory is correct, and Watson does not notice the intended difficulty level of the question as much as humans do, that would make the high value clues even more important to Watson than they are to human contestants.

My Jeopardy! Strategy Advice To IBM - Part 1

 Ed Toutant Report on Watson Jeopardy! Visit – September 17, 2010

RULES, FORMAT, UNKNOWNS

I got the impression that the final rules and format of the Jeopardy! Challenge have not been determined, or maybe that information is known but no one was authorized to share it with me.  I’ll assume the former.  The problem is a lot more difficult if you don’t know what you’re preparing for.  Will the public get to see a one-day televised match between Watson and two humans, or will Watson be in a multi-day tournament, like the standard Tournament of Champions, or even the huge Ultimate Tournament of Champions?  If there is a tournament, who are the other players and pairings, and does Watson have to win his way into the finals, or will he be given a bye?  Will all 60 clues on the board be revealed, or will some clues be left unused, depending on the time available?  If the latter, is there a fixed precise amount of playing time for each round, which Watson can keep track of, or does it depend on the whim or gut feel of the show’s director?  What is the intended difficulty level of the questions?  Will they be written at the same level as the regular episodes, or will they be more difficult than usual?  Will good players know 60% of the answers or 90% of the answers?  All of these factors and many others could have a significant impact on how the game is played and which strategies will be most effective.  At this point, Watson does not appear to have a specific goal, other than to play Jeopardy! well.  Watson seemed content to settle for a tie in some cases when it would have been no more risky to go for a win.  This will certainly annoy the Jeopardy! purists, and if that strategy is used in a tournament, it could result in Watson’s elimination from the next round.  There are also some tournament situations where the goal is not just to win, but to attain the highest possible score.  Depending on how the Jeopardy! Challenge is structured, the specific goal will affect the strategy.

My Jeopardy! Strategy Advice To IBM - Introduction

On September 17, 2010, I spent the day at the IBM Watson Research Center in Yorktown Heights, New York.  I met most of the research team for the Jeopardy! Challenge and discussed several aspects of game play optimization with the strategy team.  I spent most of that day observing six sparring matches between Watson and each possible combination of two human opponents, from a group of four former contestants IBM invited in for the day.  Each of them had been successful on Jeopardy! and competed in the Tournament of Champions.  I took a lot of notes that day and spent a couple weeks thinking about the challenges involved as I wrote a report with my recommendations to IBM.  On October 8, 2010, I sent my report to IBM.

I was told that my report was well received and that it prompted a lot of discussion within the Watson team.  We had some follow up phone calls and e-mails to discuss specific strategy topics, especially after it was revealed that the Jeopardy! Challenge would consist of two matches in the format of a tournament final, with Watson facing Brad Rutter and Ken Jennings.  In early January 2011, I was invited back to the Watson Research Lab to play eight practice matches against the final version of Watson, as a final tune up.  I was also in the live audience that watched the taping of Watson's televised matches against Ken and Brad.

On Saturday, July 9, 2011, I will join Jeopardy! legends Bob Harris and Ken Jennings on a panel discussion at the Trivia Championships of North America at the MGM Hotel in Las Vegas ( http://tcona.com/ ).  One of the main topics will be Jeopardy! strategy and how human contestants on the show can learn from the research done by the IBM Watson team.  To provide background and prompt further discussion, I am using this blog to publicly release my initial advice to IBM, based on what I knew and observed on my initial visit last fall.  Since then, I have learned that IBM's researchers implemented many of my ideas in a far more sophisticated way than the basic concepts I suggested in my report.  More than three years of work had already been done before I was asked to share my ideas, so most of my input independently confirmed what was already being done.  However I was pleased to see that one of my ideas for Daily Double wagering was added to the final version of Watson, and some of my other ideas would be too subtle to know for sure whether they were used in Watson or not.

The next several posts in this blog will be the text of the report I sent to IBM on October 8, 2010.  It is divided into separate parts for each topic.