Google's Innovation Pathway: Some Classical Ingredients

Everyone acknowledges the role of "serendipity" in the progress of science and technology, where the term means "making desirable discoveries by accident."  Picture it as Google co-founder Sergy Brin enjoying zero-gravity at left - a major breakthrough is often something you bump into while happily floating somewhere else.

Google has marched from a search algorithim developed by graduate students at Stanford to a search engine and into a dominant position in web advertizing and onward to a global information empire now encompassing the past as well as the future -- with a diplomacy wing being incorporated into Google Ideas.  In spite of Google's ubiquity in the global information cloud, was its success in fact an accident? 
The question arises in Charles Petersen's review in the New York Review of Books of business writer Ken Auletta's history of Google.  Brin and his graduate program friend Larry Page began to work with Page's desire to understand every page of every site as the target of a set of links that arrived there from anywhere else in the web.  They developed a method of mapping these backlinks which they called Backrub, and in a story that has been well-told before, particularly by Wired's John Batelle, they developed the PageRank algorithm that created a search engine that was better than any then available.

Page really did grasp something that has become a fairly common idea since: you will get less noise in your result if you supplement a search for a word or a word string with an evaluation of the other sites that have linked to it.  He saw the web as a structure of citations, and mapping the citations introduced the judgements of hundreds, thousands, or millions of other people about the value of that site's use of the word or word string. Terms like "the wisdom of crowds" or "crowdsourcing" capture the idea of collective intelligence that Brin and Page used as the central principle of their search algorithm.

Except in fact there was no search algorithm when Page was thinking about graphing the web. In the 2009 Commencement speech he gave at his alma mater the University of Michigan, he said,

I had one of those [vivid] dreams when I was 23. When I suddenly woke up, I was thinking: what if we could download the whole web, and just keep the links and... I grabbed a pen and started writing! Sometimes it is important to wake up and stop dreaming. I spent the middle of that night scribbling out the details and convincing myself it would work. Soon after, I told my advisor, Terry Winograd, it would take a couple of weeks to download the web -- he nodded knowingly, fully aware it would take much longer but wise enough to not tell me. The optimism of youth is often underrated! Amazingly, I had no thought of building a search engine. The idea wasn't even on the radar. But, much later we happened upon a better way of ranking webpages to make a really great search engine, and Google was born. When a really great dream shows up, grab it!

Page is identifying some classic ingredients of creative breakthroughs:

• disconnection from specific goals: the "dream" is pure curiosity, unharnessed by any aspect of reality.
• absence of a sense of useful application
• discontinuity with the "mature" idea

Page's advisor Terry Winograd gave Brin and Page lots of rope to run with, Stanford gave them enormous bandwidth, and they came up with various tricks and wheedlings to obtain lots of cheap equipment. Thus a further classical element is

• quasi-free resources - no strict cost-accounting

 These are features that are more easily sustained in academic than in commercial environments. Highly creative commercial environments like the Eastman Kodak of Kenneth Mees, wartime Lockheed, monopoly-era Bell Labs, and perhaps like Google today, set up university-style environments with free space for playing around, daydreaming, and working obsessively on compelling ideas that look entirely marginal to the main business. 

A later conceptual leap connected the mathematics of worldwide links to web searches. And then a further major breakthrough consisted of Brin and Page's response to their attempt to sell their algorithm to Yahoo! so they could go back to the important work of finishing their PhDs.  Yahoo! didn't want their search engine, for a reason Auletta explains:

[The Yahoo founders] were impressed with [Google’s] search engine. Very impressed, actually; their concern was that it was too good…. The more relevant the results of a search were, the fewer [pages] users would experience before leaving Yahoo. Instead of ten pages, they might see just a couple, and that would deflate the number of page views Yahoo sold advertisers.

Brin and Page are sent away, and the crucial next step is their reaction to this defeat: they "disdained" the reasoning behind it (Petersen), and continued on their own path.  That is to say that they disdained the core business model of the commercial Internet of the time: rather than following what the market seemed to say, they rejected it.

It is easy in retrospect to see this as an act of enormous courage or genius. Let's posit that it was both, and then define what genius and courage meant in such a context: a kind of perversity and youthful defiance ("screw you, Yahoo!"), and the confidence or simple stubbornness or persisting in spite of a lack of alternatives against the industry's conventional wisdom.   Brin and page had in fact to reject the direct relevance of the business model as such to the development of an important idea: they "decided to continuing improving Google's search algorithms," on the tacit assumption that they would not be helped by following the market at least in the short run.

Petersen picks up the next phase:

It wasn’t until Google grew desperate for funding during the dot-com bust of the early 2000s that Page, Brin, and their colleagues began to see that the technical advantage they had gained over other search engines might translate into an economic advantage as well. Aside from page views, one of the few easily measured statistics on the early Web was “click-throughs,” the number of times visitors to a site found an ad displayed enticing enough to click on it, and then be taken to the advertiser’s own website, where the product or service in question might be purchased or used. Most websites, including those of other search engines, found that they could earn more by charging small amounts for each time an ad was seen (page views) rather than charging a larger amount for the far less frequent instances when a visitor clicked on an ad (click-throughs) and then visited the advertiser’s own website.

 Google changed this. A conventional interpretation might be that financial necessity is the mother of invention. But Brin and Page had been desperate for money at most phases of the development, so financial necessity was nothing new.  Instead, it's worth emphasizing again that some combination of their experience and their personal outlooks allowed them to accept that market potential is over the horizon w hen development was most important.  This meant that in spite of the pressure at the time, they didn't take short cuts. This allowed a more recessive but important element of the innovation process to emerge. For the innovation to have its most important impact, the innovation has to change the business model that at the time controls the market.  This is finally what "click-throughs" did.

None of this is to whitewash either the innovation process as it must be practiced under Silicon Valley rules, or to give a free pass to Google's current practices and impact.  Petersen is especially concerned about privacy issues, and I am especially concerned about the free circulation of information under Google globalization.  By making Internet advertizing massively more effective, Google made the commercialization of the Internet more pervasive, far more so than during the Netscape Era in which the founders started their doctorates.  But even here Google offers a counterintuitive lesson, and it is built into the advertizing model itself.

With Google’s search engine advertising, the value of the ads comes less from their quantity (the number of times the ads appear, for which the company receives nothing) than from their quality (the number of times the ads match what users are looking for and attract purposeful clicks, for which Google receives fees).

Averse as I am to sponsored links, Petersen's description here is right.  There's an open source idealism lurking in the business model Google finally hit on, one that makes unfake the often repeated assertion of the Google elders that "Our mission is to organize the world's information and make it universally accessible and useful." The idealism doesn't fit into its corporate embodiment, and yet the idealism is what make the company so successful, and made it's innovation possible.

A summary of the innovation elements would run like this:

1. the dream, disconnected from specific goals; pure curiosity
2. long absence of a sense of useful application; discontinuity with the "mature" idea
3. quasi-free resources - no strict cost-accounting during long "precommercial" phase
4. frame jumping (from graphs to web searches)
5. rejecting the dominant business model
6. changing the business model to fit the innovation, not fitting the innovation to the dominant model
7. pursuit of maximum social value, not just (internalizable) market value

All of these come with enough gaps to deter just about anyone - anyone except people that have overpowering interests untethered from markets of the time.