Google and the evolution of search – human evaluators

For many years, Google, on its Explanation of Our Search Results page, claimed that “a site’s ranking in Google’s search results is automatically determined by computer algorithms using thousands of factors to calculate a page’s relevance to a given query.”

Then in May of 2007, that statement changed: “A site’s ranking in Google’s search results relies heavily on computer algorithms using thousands of factors to calculate a page’s relevance to a given query.”

A slight adjustment in wording, but an important comment on the supremacy of the algorithm that Google had touted for years. Google had finally acknowledged that its search results were no longer solely and automatically determined by the company’s vaunted algorithms. Now they simply “relied heavily” on them. Why the sudden change?

Google claims it was arbitrary, unrelated to any sudden philosophical shifts within the company. But it seems far too specific an adjustment to chalk up to a random brand-management edit. We are, after all, talking about the company’s official explanation of its search results. And indeed, sources say the language was changed to account for the continual calibration of the algorithm, which these days is done with a bit of human help.

Google, for example, employs a vast team of human search “Quality Raters” (You’ll find a copy of an old training manual here). Spread out around the world, these evaluators, mostly college students, review search returns against established criteria–testing different algorithms and see which works “best” in predicting the quality of a site (though not directly judging the quality of any individual site itself).

They’re aided by Google’s own registered users, who can now, when logged into their Google accounts, promote and delete sites from their own search returns according to their preferences. These data too are used to tweak and further optimize the algorithm. So Google’s objective evaluation and ranking of Web sites is to some extent defined by subjective reasoning of a collective human intelligence. And so it must be if Google is to continue returning search results that we perceive to be the “best” answers to our search queries.

In interviews serialized over the next three days, key Google engineers with central roles in managing the company’s search engine discuss resources and techniques they use to optimize the system for users world-wide. The series kicks off below with Engineering director Scott Huffman, who oversees the company’s search evaluation team. Senior Google software engineer Matt Cutts appears tomorrow. And Google Fellow Amit Singhal wraps up the series on Friday.

Renaming Web 2.0

At the Web 2.0 Expo in San Francisco, internet guru Tim O’Reilly threw out the possibility that perhaps the name should be changed.

He said he and his friend John Battelle of Federated Media had been playing around with an alternative which was Web 2.0 + World = Web Squared.

When I asked Mr O’Reilly if he loved or hated the name Web 2.0 that he popularised, he let out a big sigh and said “Awww does it have to be one or the other?”

Eventually he admitted “I love it and I hate it. It’s a term that has been very effective and very successful in getting across an idea. I spent a long time talking about that idea around the turn of the Millenium, talking about building the internet operating system. It didn’t catch on and all of a sudden we had this new term Web 2.0 and everyone got it so how could you not love that?”

In the end he said “I have mixed feelings about it. I am delighted with its effectiveness, it did what I wanted it to do. To catalyse the industry after the dotcom bust that things weren’t over and that something mattered about the companies that had survived. They knew something that the others didn’t. And I think that continues to be true.

“The companies that are succeeding today understand better than others what it means to be building software in the age of the internet.”

As to really getting behind Web Squared, Mr O’Reilly said “It was just one of these idle thoughts where you go dub dub dub and then you go one more w and that gets you to web squared, right?”

My unscientific research on the expo floor found more people hating than loving the Web 2.0 title.

Paul Thompson said “Keep it. It hasn’t been around for very long and you need a few years to build an identity. If you replace it with Web Squared, people will go what happened to Web 2.0?”

Mark Kirthcart thought “it’s sounding a little dated and overused.”

Sindee Thomson’s view was “Web 3.0 will be here soon.” For her, Web Squared was a total no no. “I hate it. It reminds me of mathematics and I was never good at my sums. I think it should be Web Cubed.”

Brooklynn Morris was a big fan. “I think Web 2.0 is a great title but I think people don’t like titles in general especially when it gets in the way of free concepts.”

Kevin Marshall said he thought people were “tired of Web 2.0 because of all the hype around it. Web Squared however, I don’t think is any better.”

Alistair Mitchell suggested that instead of Web Squared it should be “Web Shared because the web today is all about sharing – sharing the content of your life through things like Flickr, Facebook, where you live, where you are and how you work.”

Taomas Rio said “Web 2.0 is too techy. Sure the core of people who come here know what it means but the internet is always evolving so why do you need versions or numbers to categorise it?”

As for Web Squared, Taomas was aghast. “Oh no that’s web weird!”

Any better suggestions?

Microsoft Research: A look at the intriguing social desktop prototype

Late last week, Microsoft Research shared a couple of things about Social Desktop, a prototype of which they are debuting at TechFest 2009 in a couple of days (along with dozens of other things). From the looks of it, this will be a much talked about product even if it stays in proof-of-concept phase for now.

And if they decide to open it up even just a little, this could be a major breakthrough in tearing down the virtual wall between the desktop and the web, a trend we’ve been noticing for years.

The service would essentially be capable of providing you with a secure unique ID for all the files and folders on your desktop, enabling users to share, comment on, tag and search files like photos and videos via a dedicated web page powered by .NET. Think of this as social URLs that link to files which could easily be pushed to third-party services like Twitter or Digg but also Microsoft’s own Windows Live Messenger without the need for you to copy, move or upload anything. Furthermore, social interaction around the files would be visible from inside the Windows desktop OS, blurring the line between the desktop and the web even more.

You can have a URL drill into a subportion of a document or a PowerPoint deck, or data can come from a Web service or a database. Social Desktop is a local service that maps the user’s local data into a .NET service bus service, enabling local data to be accessible through firewalls. Social Desktop also provides a Web-service view over the same data, with inherent RSS event streams for any container. New data sources can be mapped into the URL hierarchy, enabling a distributed view to be built. There are simple sharing paradigms that enable URLs to be shared temporarily or permanently.

Social Desktop runs on Silverlight and leverages both the Windows OS and Windows Azure, the software giant’s very own cloud services platform which Microsoft announced in October 2008. TechFlash reviewed the service as well last week, and asked the project leads how Social Desktop differs from Live Mesh. The response came from Lili Cheng, who manages Microsoft Research’s Creative Systems Group: “In the Mesh model, you can almost imagine your PC being pushed to the cloud,” she explained. “In this, you can almost imagine the Web being embedded inside your desktop.”

I don’t know about you, but to me this all sounds very promising and I’m curious if using Social Desktop would change my file sharing habits. Even with the plethora of free, simple and fast online backup and sharing services around, there’s still a trust barrier not easily overcome by startups who need to market their services extensively on an inherently low budget to reach any kind of scale. Besides, Social Desktop even relieves you from the not-so-cumbersome task of moving a file to the cloud in order to store or share it, so that makes for one hell of a substantial benefit compared to other services where you’d be required to register and do a series of actions before that happens.

Unfortunately, a Microsoft spokesperson told NetworkWorld that Social Desktop at this point is merely a research prototype which will not be a feature in Windows 7, nor will it be available for public use.

But I still want to get my hands on Windows 7 Beta (it makes use of the new operating system’s file-preview functions) right now even if just to test this application once (and if) they release it.

Celebrate the launch of twiggit

twiggit is an automated service that lets your friends on twitter know what articles you digg. Every so often the service checks for the last article that you voted for on digg, and updates your twitter status to reflect this. There are a number of options include the ability to only tweet the articles you submit rather than digg, pause the service at anytime, change the frequency of when to check digg and completly remove your twiggit account.

The site can be seen at http://twiggit.org/

Innovation: It’s all in how you see it

“Innovation” has been thrown around so often in technology circles that to some, it’s a four-letter word.

At one tech company, innovation can mean bringing a dazzling new product to store shelves. At another, it can translate to a tiny new button on a Web site. That’s why, executives say, the word itself has been overused and devalued.

Still, new cutting-edge products mean everything to a successful tech company.

Executives from eBay, Hewlett-Packard, Microsoft, and others were here at SDForum’s first Corporate Innovation and Research Fair on Friday to talk about their techniques for staying creative. Each company has its own style, with some strategies that overlap. But they all acknowledged it’s not easy to innovate, especially considering that large corporate cultures can be a curse to fresh ideas.

Max Mancini, eBay’s senior director of Platform and Disruptive Innovation, went so far as to say that Silicon Valley venture capitalists wouldn’t make so much money on start-up investments if tech companies were better at developing new products.

“Venture capital firms thrive on inefficiencies in large organizations,” said Mancini, who spoke at the gathering held at the Computer History Museum.

His counterpart at HP added to the idea by saying that demands from Wall Street and senior management can stifle innovation. “If you’re a larger company, there’s high probability you have creative people (in your organization). But creative people get impatient,” said Rich Friedrich, director of HP’s Enterprise Systems and Software Lab.

That means that these companies either must invest billions in research and development units, or bake in policies to ensure that people dream up new products. Google, of course, asks engineers to spend 20 percent of their time on pet projects. Microsoft, in contrast, employs more than 800 researchers in labs around the world.

A bottom-up style
Roy Levin, Microsoft’s director of research in Silicon Valley, said that one reason the labs have proven helpful to Microsoft, including bringing products like Windows Media to consumers, is their bottom-up style. The labs’ researchers pick projects themselves and collaborate with each other. They’re also not beholden to profit-and-loss goals or managers, he said.

“Every time you introduce (managerial) hierarchy, you introduce barriers to collaboration; and collaboration is key,” Levin said.

But once a technology is ready, transferring it to a product group or bringing it to market can be highly difficult, he said. That’s why so-called technology transfers are “a contact sport,” he said. Researchers must travel a lot to get new ideas and prototypes in front of the right people, Levin said.

eBay’s Mancini said that the auction company does two big things to promote creativity. The first is operating a technology platform that mirrors the eBay framework so that its engineers can experiment with new tools. That way, developers can test products outside of the company’s rigid software development process, he said.

The other method is to invite third-party developers into the fold through application programming interfaces. He said that in the last year developers have created an estimated 12,000 applications for eBay, producing as many as 60 percent of the listings on the site. “That’s innovation we probably couldn’t afford,” he said.

“Innovation is about the ecosystem, either removing barriers internally or allowing third parties to help meet the needs of your customers in ways you can’t afford to do (or have the time to do),” Mancini said.

Similarly, HP’s Friedrich said that one of his company’s strategies is to partner with outsiders on projects. “All of the innovative people don’t work for your company,” he said.

HP, for example, teamed up with DreamWorks years ago to work on technology for life-like animation and “cloud” services that were used to produce the movie Shrek. Last week, HP also teamed up with Intel and Yahoo to create six large-scale computing centers that would allow outsiders to test technology.

Cloud services are one of several areas of research for HP, which invests about $3.6 billion annually in R&D, Friedrich said. It’s also looking at projects in sustainability and managing data. On a broader level, HP is trying to shift the company from a hardware maker to a software company; and it’s doing that largely through acquisitions.

Oracle’s Marie-Anne Neimat, vice president of development for embedded databases, also pointed to acquisitions as a way to evolve, beyond Oracle’s multibillion dollar annual investment in R&D.

“It’s new blood,” she said.

Finally, some technology companies have turned into venture capitalists, too.

Ike Nassi, SAP’s executive vice president of research for the Americas and China, said it recently started a venture capital incubator. It solicits ideas from internal employees and external start-ups; and if it’s a good idea, SAP will help form a new business unit, fold the start-up into an existing product line, or spin it out as a new company, he said.

“If you have an interesting idea and don’t want to go the VC route, we provide seed funding,” Nassi said.

That’s similar to other technology companies. Intel, Google, Motorola, Amazon, and Comcast run venture capital units either formally or informally.

What about the word innovation?

“It’s completely devalued,” Nassi said. “The thing we need to look at is managing risk–whether placing an investment on this versus that, and what’s the payoff of that investment.”