User: “Who starred in the film Casablanca?”
Search Engine: “Humphrey Bogart and Ingrid Bergman.”
Imagine asking a more nuanced question, such as “What do Americans think of offshore drilling?” A search engine will be able to respond with a report indicating trends in opinion based on what has been posted to the Web.
Search engines may eventually be used to conduct polling and even help sort fact from fiction, said Weiyi Meng, a professor of computer science at Binghamton University. He’s helping to make such futuristic possibilities a reality, both through his research and as president of a company called Webscalers.
The way Meng sees it, big search engines such as Google and Yahoo are fundamentally flawed.
You see, the Web has two parts: The surface Web and the deep Web. The surface Web is made up of perhaps 60 billion pages. The deep Web, at some 900 billion pages, is about 15 times larger.
Google, which relies on a “crawler” to examine pages and catalog them for future searches, can search about 20 billion pages, just a small fraction of the entire Web. Web crawlers follow links to reach pages and often miss content that isn’t linked to any other page or is in some other way “hidden.”
Meng, along with colleagues at the University of Illinois at Chicago and the University of Louisiana at Lafayette, has helped pioneer large-scale metasearch-engine technology that harnesses the power of numerous small search engines to come up with results that are more accurate and more complete.
“Most of the pages on the deep Web aren’t directly ‘crawlable.’ We want to connect to small search engines and reach the deep Web,” he said. “That’s the idea. Many people have the misconception that Google can search everything, and if it’s not there it doesn’t exist. But we should be able to retrieve many times more than what Google can search.”
Not only can a metasearch engine probe deeper, it can also offer the latest information.
“In principle,” Meng noted, “small guys are much better able to maintain the freshness of their data. Google has a program to ‘crawl’ all over the world. Depending on when the crawler has last visited your server, there’s a delay of days or weeks before a new page will show up in that search. We can get fresher results.”
The concept is not new. In fact, the first metasearch engine was built in 1994.
“The big difference between our technology and the ones pursued by other people is that most of the other technologies do the metasearching on top of a small number of general purpose search engines, such as Yahoo, Google or MSN,” Meng explained. “We have a completely different perspective. We want to build large-scale metasearch engines on top of many small search engines.”
page 1 | page 2