An Interview with Stephen Baker, Author of “Final Jeopardy!”

Tonight is the first of a three-episode special edition of Jeopardy! featuring two games between two of the show’s biggest winners and IBM’s Watson.  The long-awawited match will see the machine take on Brad Rutter and Ken Jennings for a million dollar top prize.  If you’re interested in learning more about the process of how Watson was put together and the stress behind it, I recommend the book, “Final Jeopardy: Man vs. Machine and the Quest to Know Everything,” by Stephen Baker.  Click here to find the book on Amazon.

The book follows the process of building and fine tuning Watson, the pressures for IBM, and the Jeopardy! match itself.  I read the book and it’s incredibly entertaining and makes you look at the program in a very different way.  I recently had the chance to interview Mr. Weber about the book.  He discusses the way Watson works, the fears of Sony and IBM, and where IBM can take this next.

AD: I don’t think people fully understand the giant technological feat Watson is until they read the book or learn more about it.  How big was this?

SB: One thing I noticed when I was watching the PBS NOVA thing…was that they were referring to it as “he”.  It’s just a tool.  I think people are worried about it.  They think it has human qualities.  I think that when people…attack Watson it’s because part of the promotion of it is that it’s sort of like a person.  And it really isn’t.  But it couldn’t promote its technology without giving it some of these human qualities and getting it on a stage like Jeopardy!.

AD: What interested you in technology, and the Watson project specifically?

SB: I was really eager to find a project that told a story…where the story would have a beginning, middle, and end.  I was always interested in the story of a development of a product.  This one was doubly good because it adds with a championship match, which makes it almost like a sports story.  I’ve always been a sports fan so this has been a dream come true for me.

Final Jeopardy by Stephen Baker

AD: How long of a process was it to write the book?

SB: A year.  Research and writing took a year.  I had to do it on a tight deadline because we wanted the book to come out as soon as the match was over.

AD: What’s the process like for Watson to find an answer?

SB: It’s like an enormous industrial process because it lacks common sense and it doesn’t know what it knows like we do.  It has to pursue every possible angle to find the answer.  Many of the angles it pursues are utterly ridiculous.  But, because it doesn’t know anything, it has to do that…so it’s an enormously wasteful process.  But because they have so much computing power they can afford to waste a lot.

AD: That was one thing I found interesting reading a lot.  The New York Post did a story recently where they had a computer expert who threw a bunch of Jeopardy style questions into search engines and it came out that Google was 6% higher in accuracy than human contestants.  I think that does a little bit of injustice to Watson.  It’s more than just a search engine.

SB: Google only points you to where you, with your magnificent brain, can find the answer.  Finding the answer and deciding upon it and building confidence on it is a really hard technological challenge.  That was one of the two key areas where Watson represents a significant advance.  The other is just an understanding of English.

AD: In your book you wrote that Sony wasn’t sold on the idea from the start.  What were some of Sony’s fears?

SB: The fears…and there were fears on both sides…but Sony’s fear was IBM would develop a machine that’s so dominant that it would beat humans every time and shut out the humans, and create an excruciatingly boring TV show.  Not to mention kind of depressing from our point of view.  They were worried that Watson would get too fast and too smart.

If they had waited another two years, if IBM developed the project for another two years, because Watson, as they continue to work on it, will only get faster and better.  [David] Ferrucci himself said, as he laid out the plans for IBM at the very beginning, that if they gave him five to seven years Watson would be unbeatable.

AD: It’s getting closer to that apparently.  I followed some link on your Twitter page and it had a slide show.  One of the slides was that when Watson was first being tested it took an extremely long time to find an answer.  Where do you see the improvement?

SB: There are two areas of improvement.  One is its ability to answer the clues.  Even when it was getting very smart in the beginning it took two hours.  The improvement to three seconds wasn’t just a factor of it getting a thousand times smarter.  What happened is that they changed the hardware platform and they distributed the work that one computer was doing to nearly three thousand.  That is what made Watson much much faster.

AD: To go back, what were IBM’s fears?

SB: Ferrucci, the chief researcher David Ferrucci, had trouble sleeping the past two years.  His nightmare was that he and his team

IBM's Watson Computing System

could build a fabulous machine, but the results of one game are left largely to chance.  If Watson runs into a category it’s confused by, if it blows a Final Jeopardy, if it doesn’t get a Daily Double, it’s going to lose…the same is true for Ken Jennings and Brad Rutter.

It’s a lot like sports.  You’d rather have Albert Pujols up at bat than Nate McLouth in a tough situation.  But in certain days a bad hitter goes 3 for 4 and Albert goes 0 for 4.  You never know, and that’s the same as Jeopardy.

AD: What are some weak areas for Watson?

SB: It’s slow on clues that are very short.  Alex Trebek reads it in less than a second and humans immediately know what the answer is.  Watson takes more time to process it.  It has trouble with clues that require various levels of analysis.

There’s one type of clue where if it’s like, “”______Deadly Sins” times “Jesus’s ______ Disciples” equals…” , and you’d say seven times twelve is 84.  So the people have to know what its talking about in two different areas and then they have to understand it well enough to know it’s expecting a mathematical calculation.  That would be the easiest part for Watson but it has to know those two areas to go into it.

AD: I’ve talked to people that have went on Jeopardy!.  They do well with the clues, obviously, or they wouldn’t be there.  But many struggle with betting.  Was it a struggle to teach proper betting methods to a machine like Watson?

SB: It was much less of a struggle to develop a good betting strategy for Watson.  They put it through millions of simulated games to come up with the statistically ideal bet for each situation.  The problem is that, while statically it makes sense for Watson to be much bolder and bet bigger, if the showdown is only two games, as it is in this case, do you want it to take those chances?  Even if it has a 60% higher chance of winning the game it also has a lot of risk.  It was difficult to come to the decision of how risky Watson should be.

AD: When I first heard about the idea I figured it would do well but given how Jeopardy! is set up I wasn’t sure.  Watching and reading about it more is surprise.  Does the success of Watson, and its accuracy, surprise you?

SB: Yes.  Even now after working on it for more than a year Watson continues to surprise me when it understands really obscure questions.  It blows my mind.  Sometimes it comes up with ridiculous and hilarious answers which is also fun, but its accuracy really surprises me.

AD: You played against Watson, right?

SB: I played once.

AD: How was that?

SB: It was really fun.  It was very discouraging when you know an answer and try to buzz in as fast as you can and Watson beats you to it.  That’s frustrating.  The funny thing was if I had bet more on Final Jeopardy I would have beaten it.  I was feeling pretty good about myself but later one of the IBM people told me the version of Watson I was playing was an old one pulled off his laptop so that took the wind out of my sales.

AD: Reports I read said that Watson’s memory size is relatively small for everything it does.  Is that right?

Watson test match, courtesy Engadget

SB: The data that it can access to search through to find the answer is about 75 gigabytes.

AD: That’s less than a lot of iPods.  Does that mean eventually, as size decrease when technology gets better, is this something that could be launched commercially for anyone to use, or is this not on the horizon for them?

SB: The first thing on the horizon is selling it to businesses so they can use this platform to develop specialized question and answering systems, whether its drug research, finance, law or, whatever it would be.  These people can afford to spend millions on a machine.

There was talk of turning Watson into a Google like service but that cannot happen in the short term.  Watson takes three seconds to respond to one question.  In the same time Google may respond to a million questions.  Watson can’t scale.  It’s too expensive, too much electricity, too much computing power to provide that service for consumers.  But that could change and they can find ways to make it cheaper and more focused for certain applications for consumers.

AD: What’s the next step after Jeopardy!?  If it’s deemed a success, where will we see Watson next?

SB: I think it would be the question and answering machine in business.  What they have their highest hopes for, initially, is in medical diagnostics.  Where a machine like Watson could look at a number of symptoms a person is reporting and go through thousands and thousands of research papers and case studies and find a correlation among those symptoms to come up with a list of possible illnesses.  That’s where they think Watson can be valuable.  It wouldn’t be running the diagnostics system…humans would do that.  But Watson would be a very brainy assistant.

You can purchase “Final Jeopardy” on Amazon by clicking here.