History of Web Search Engines.

The goal of all search engines is to find and organize distributed data found on the Internet. Before search engines were developed, the Internet was a collection of File Transfer Protocol (FTP) sites in which users would navigate to find specific shared files. As the central list of web servers joining the Internet grew, and the World Wide Web became the interface of choice for accessing the Internet, the need for finding and organizing the distributed data files on FTP web servers grew. Search engines began due to this need to more easily navigate the web servers and files on the Internet.

The first search engine was developed as a school project by Alan Emtage, a student at McGill University in Montreal. Back in 1990, Alan created Archie, an index (or archives) of computer files stored on anonymous FTP web sites in a given network of computers (“Archie” rather than “Archives” fit name length parameters – thus it became the name of the first search engine). In 1991, Mark McCahill, a student at the University of Minnesota, effectively used a hypertext paradigm to create Gopher, which also searched for plain text references in files.

Archie and Gopher’s searchable database of websites did not have natural language keyword capabilities used in modern search engines. Rather, in 1993 the graphical Mosaic web browser improved upon Gopher’s primarily text-based interface. About the same time, Matthew Gray developed Wandex, the first search engine in the form that we know search engines today. Wandex’s technology was the first to crawl the web indexing and searching the catalog of indexed pages on the web. Another significant development in search engines came in 1994 when WebCrawler’s search engine began indexing the full text of web sites instead of just web page titles.

While both web directories and search engines gained popularity in  the 1990s, search engines developed a life of their own becoming the preferred method of Internet search. For example, the major search engines found in use today originated in development between 1993 and 1998.

Excite – Born in 1993

Excite was born in February 1993 as a university project called Architext involving six undergraduate students at Stanford seeking to use statistical analysis of word relationships to improve relevancy of searches on the Internet. This school project eventually led to Excite’s commercial release as a crawling search engine at the end of 1995. With solid growth in 1996, Excite purchased WebCrawler and Magellan. Toward the end of the 1990s, Excite partnered with MSN and Netscape providing search services. In 1999, Excite was sold to broadband provider @Home.com (later becoming Excite@Home) as part of a $6.7 billion merger after its traffic started to decline with the release of Google in 1998. With significant debts, Excite@Home filed for bankruptcy in October 2001 and sold its high-speed network to AT&T for $307 million. A month later, InfoSpace made a $10 million bid to buy Excite@Home’s assets including domain names and trademarks from bankruptcy court. 

Infospace’s offer was accepted and they subsequently powered the Excite web site and sold portal components to iWon. InfoSpace’s Dogpile crawler replaced Excite’s making Dogpile and Excite’s search results the same. Both Excite and Dogpile are also powered by LookSmart’s directory, except that Dogpile includes a number of other InfoSpace directories. Also as part of the deal, InfoSpace acquired rights to WebCrawler. Ask Jeeves (now Ask.com) purchased the Excite.com portal in 2004. Now, Excite offers search results through a metasearch tool combining results from pay-per-click and natural search tools.

Yahoo! - Born in 1994

David Filo and Jerry Yang started Yahoo! in 1994. Originally it was a highly regarded directory of sites that were cataloged by human editors. This directory provided an extensive listing of websites supported by a network of regional directories. In 2001, Yahoo! started charging a fee for inclusion in its directory listing. Yahoo!’s action helped control the number of sites listed and helped cover costs with additional revenue.

Initially, Yahoo! used secondary search engine services to support its directory. Partnerships have included agreements with Inktomi and Google. In October 2002, Yahoo shifted to crawler-based listing for its search results. In 2004, Yahoo! purchased Overture’s pay-per-click service, which had only months earlier purchased AltaVista and AlltheWeb, and Inktomi’s search database. With these acquisitions, Yahoo! combined these tools to create its own search index. Today, Overture is renamed Yahoo! Search Marketing and provides paid search advertising revenue. The Yahoo! Directory remains one of the top indexes powering search listings.

WebCrawler – Born in 1994

WebCrawler was the first search engine to provide full text search. In 1994, Brian Pinkerton, a Computer Science and Engineering student at the University of Washington, used his spare time to create WebCrawler. With WebCrawler, Brian generated a list of the Top 25 websites on March 15, 1994. Only a month later on April 20, 1994, Brian announced the release of WebCrawler live on the web with a database of 4000 websites. On June 11, 1994, Brian posted to the Usenet group comp.infosystems.announce that the WebCrawler Index was available for searching. By November 14, 1994, WebCrawler served its one millionth query. By the end of the year, WebCrawler signed two sponsors, DealerNet and Starwave providing needed capital to finance WebCrawler. A little less than a year later, WebCrawler was fully operating on advertising revenue.

A young America Online, without access to the web, acquired WebCrawler on June 1, 1995. On September 4, 1995, Spidey was created as WebCrawler’s mascot. On April 1, 1997 (no fooling), WebCrawler was sold by AOL to Excite. Initially, WebCrawler was going to run by its own dedicated team within Excite, but eventually the two were merged together on the same back end. In 2001, Excite went bankrupt and was purchased by InfoSpace. As part of the agreement, InfoSpace acquired WebCrawler. Today, InfoSpace runs WebCrawler as a meta-search tool blending results from Google, Yahoo!, Live Search (formerly MSN Search), Ask, About.com, MIVA, LookSmart, and others. As for Spidey, he is now purple.

Lycos – Born in 1994

Lycos was one of the earliest search engines developed in 1994 at Carnegie Mellon University by Dr. Michael Mauldin and a team of researchers. The Lycos name came from the Latin term “lycosidea” referencing wolf spiders that hunt and actively stalk prey. The company was founded on $2 million in venture capital funding from CMGI. The company was headed by Bob Davis, who concentrated on building Lycos into an advertising-supported web portal. The company went public in April of the next year with little money. With phenomenal growth in its catalog, Lycos had the largest index at the end of 1996 with 60 million documents. In 1997, Lycos Pro was launched with a new search algorithm and continued to grow. By 1999, Lycos would emerge from a crowded pack to become the most-visited web portal. Over the next few years, Lycos would become one of the most profitable Internet businesses and acquired nearly two dozen high profile Internet brands.

For example, in February 1998, Lycos acquired Tripod Inc., a website where people built their own web pages. That summer, language search was introduced and the search results pages were redesigned. At this same time, WhoWhere Inc.’s directory services and Mail City’s email services were acquired for $133 million in stock. Toward the end of 1998, Lycos acquired Wired Digital (owner of HotBot) for $83 million.

By 1999, Lycos was one of the most visited search tools on the web  and would continue to be involved in new projects and acquisitions.  In February that same year, Lycos became USA/Lycos Interactive Networks when USA Networks bought a 61.5% ownership in the company for $18 billion. As a result, Lycos later announced a project to create a search tool to query information on USATODAY.com’s news site. In April 1999, Lycos joined the Open Directory Project run by Netscape. In June 1999, Lycos joined with Intelliseek to provide a directory of over 7,400 databases previously not on the web. In September 1999, Lycos acquired Quote.com, an investment information site and launched the Lycos Zone, an educational website for kids with content filtering. Finally, in December 1999, Lycos invested in FAST search technology which began powering Lycos advanced search technology.

In May 2000, Terra Networks, an Internet arm of the Spanish telecommunications giant Telef?nica, purchased Lycos for $5.4 billion, forming a new company Terra Lycos. This takeover marked a 3,000 times return on the initial venture capital investment and 20 times the initial public offering value. Lycos remained the name of the US franchise brand. Overseas, the company was known as Terra Networks, and founder Bob Davis left the company.

Lycos suffered from the dotcom crash in 2001. In late 2001, Lycos abandoned its own crawler and began serving results exclusively from FAST. In August 2004, Terra sold Lycos to Daum Communications Corporation for $95.4 million. This low number was less than 2% of Terra’s initial investment. By October 2004, the deal was finalized and the company name returned to Lycos.

With new ownership, Lycos refocused on becoming a community destination for broadband entertainment rather than a search portal. In July 2006, Wired News, which had been a part of Lycos since the acquisition of Wired Digital in 1998, was sold. The Lycos Finance division, known for Quote.com and Raging Bull.com, and it’s online dating site, Matchmaker.com, were also sold. Lycos also regained ownership of the Lycos trademark from Carnegie Mellon University allowing it to become Lycos, Inc.

Since 2006, Lycos has announced the introduction of Lycos Phone, Lycos Mail, and Lycos MIX. These services and tools combine IM video chat, mp3 player, and unlimited file size sending and receiving via email, video watch and chat, and social media applications. Lycos remains a top 25 Internet destination, is the 13th largest online property worldwide and remains a top 5 Internet portal behind Yahoo!, MSN, AOL, and MySpace.

Infoseek – Born in 1994

Infoseek, also known as the “big yellow,” was founded by Steve Kirsch in 1994. At the start in January 1994, InfoSeek was a pay-for-use service. The fee service was dropped in August 1994 and Infoseek was re-presented as Infoseek Search in February 1995.

Infoseek’s position in the search engine world was accelerated in 1995 by a deal with Netscape in which it became the default Netscape search engine. In June 1996, Infoseek went public and by September the next year served 7.3 million visitors per month.

Infoseek uniquely featured a very complex system of search modifiers including Boolean modifiers. In November 1996, InfoSeek introduced Ultrasmart/Ultraseek and redesigned its website the next year to include channel or directory information. By March of 1998, Infoseek included a search page with advanced search techniques. Infoseek also uniquely offered a free web hosting service without advertising and had no limit on the amount of storage space for users.

In the summer of 1998, 43% of Infoseek was bought by Disney. From this point forward Infoseek was part of the Disney Corporation’s vast media business. The deal included Infoseek acquisition of Starwave Corporation, including ESPN.com and ABCNews.com. Infoseek’s technology was then merged with Starwave’s to form the Go.com network. In July 1999, Disney had a 72% interest in Go.com and Disney’s media interest included Disney.com, Family.com, ABC.com, ABCNews.com, and ESPN.com.

Aside from its relationship with Disney, Infoseek continued to change. In September 1998, Infoseek began offering a combination of manually-reviewed and traditional web-search results. The next month, Infoseek took over Excite as the default search engine on Microsoft’s WebTV and Infoseek Express, a free software website, was launched. In 1999, Li Yanhong, an Infoseek engineer, moved to Beijing, China and co-founded the search engine Baidu.

In the summer of 2000, Infoseek’s Ultraseek Server software technology was sold to Inktomi and renamed Inktomi Enterprise Search. In December 2002 (prior to Yahoo!’s purchase of Inktomi) Ultraseek was sold to a competitor Verity Inc. Verity re-established the Ultraseek brand name and development of the product until it was acquired by Autonomy PLC in December 2005. Autonomy continues developing and marketing Ultraseek’s site search.

In February 2001, many Infoseek employees tried to collectively buy out Infoseek from Disney when Disney laid off the entire Infoseek staff. Today, the Infoseek.com domain forwards to Go.com and the brand name is unused in North America. Only in Japan (as Infoseek Japan) and Australia (Infoseek Australia) is the Infoseek name used.

AltaVista – Born in 1995

AltaVista, meaning “a view from above,” developed out of research by scientists at Digital Equipment Corporation’s (DEC) Western Research Laboratory in Palo Alto, California during the spring of 1995. They were trying to showcase their computer database system called the Alpha 8400 TurboLaser that was faster than its competitors. Scientists developed a search tool to crawl, store and quickly index every word of all HTML web pages on the Internet. This new search tool was powerful. For example, in August 1995, it conducted its first full-scale crawl of the web bringing back about 10 million pages.

The two key scientists involved with AltaVista’s development were  Louis Monier and Michael Burrows. Louis wrote the crawler (called Scooter) and Michael wrote the indexer. After testing the new search engine with 10,000 DEC employees, AltaVista opened to the public on December 15, 1995 at altavista.digital.com. Initially, the back-end processing machines received 13 million queries per day. With its release, AltaVista became the first searchable full-text database on the World Wide Web with a simple interface. Over 300,000 visitors used the search tool on the first day and had 19 million hits by the end of 1996, and 80 million per day at the end of 1997.

Ironically in 1996, AltaVista started exclusively providing search results for Yahoo! who would one day become AltaVista’s owner. Before this time, DEC was acquired by Compaq for $9.6 billion at the start of 1998. Despite success as a modest search interface, in 1999 Compaq released AltaVista as a web portal once again ironically hoping to compete with Yahoo! It may be reasoned that AltaVista’s portal strategy was a cause of its eventual decline as well as the rise of Google. By 2002, efforts returned to sharpening the quality of the simple search interface.

In June 1999, CMGI, an Internet investment company with 20% ownership in Lycos, agreed to acquire 83 percent of AltaVista. CMGI planned a public offering of AltaVista in April 2000, but cancelled the IPO as the Internet bubble collapsed.

Near the end of 2002, AltaVista became the first Internet search engine to offer image, audio, and video search as part of a new range of multimedia functionalities. Additionally, AltaVista gained innovative recognition for its release of Babel Fish, the web’s first-ever multi-lingual Internet search. Babel Fish could translate words, phrases, or entire website to and from English, Spanish, French, German, Portuguese, Italian, and Russian. With its advanced multimedia search capabilities and language translation and recognition services, AltaVista was likely the Internet’s most technologically advanced search tool at the time.

In February 2003, Overture purchased AltaVista for $140 million. This price was a fraction of AltaVista’s valuation of $2.3 billion three years earlier. Unfortunately, AltaVista’s likelihood to continue its innovative heritage with Overture Services, Inc. was quickly killed when Yahoo! purchased Overture at the end of 2003. With this purchase, AltaVista became Yahoo! using the same search index and user interface. In May 2008, AltaVista’s Babel Fish translation service was also branded under its parent company as Yahoo! Babel Fish.

Inktomi – Born in 1996

Inktomi was founded in February 1996 by Eric Brewer, an assistant professor of computer science at the University of California Berkeley, and Paul Gauthier, a graduate student. They were involved in a government research project funded by the US government’s Advanced Research Projects Agency on parallel computing between personal computers and workstations making them function like a supercomputer. In the process they founded a company based upon their success developing a search tool.

The company was named after a mythical Lakota Indian spider known for cunning rather than brute force. After formation with Dave Perterschmidt as CEO, Brewer as the chief scientist, and Gauthier as the chief technology officer, the company established its first customer with HotWired who introduced the search engine HotBot in 1996. HotBot included Inktomi’s Audience 1 software that customized web pages and advertisements according to the user’s browser. HotBot eventually evolved into newer versions including 5.0 released in 1998 using a Windows NT rather than Unix platform. The interface and server infrastructure was changed to increase usability and offer new features.

Inktomi was not just a search provider; its mission was to build scalable software applications at the core of the Internet. As such, Inktomi’s efforts drove many of the search tool developments and functionalities of known search tools and also management tools of  Internet traffic and capabilities in general. For example in 1997, Inktomi beta tested web traffic servers that managed network data flow eliminating bottlenecks and redundant Internet and corporate Intranet traffic and bandwidth. The servers used caching to create localized repositories of information.

This moved information closer to the user rather than relying each time on the Internet’s backbone. At the time, the amount of redundant traffic on the Internet was between 40 and 80 percent. Accordingly, a large market existed for such technology with Internet service providers, network providers, and business enterprises. A large customer who took interest in Inktomi’s traffic server technologies was Microsoft in 1997. Microsoft agreed to use Inktomi’s traffic server and search engine technology in the Microsoft Network starting in 1998.

Starting in 1998, Inktomi signed many customer deals. America Online and Digex Inc. became licensees of Inktomi’s Traffic Server and began using it in their own networks later that year. Inktomi also worked out deals with Digital Equipment Corporation and Intel to port Traffic Server with their Unix and Windows NT platforms. Also in May of 1998, Yahoo! decided to use Inktomi’s search engine technology as their preferred choice. All of these deals influenced Inktomi’s initial public offering in June 1998. The price of the share the first day doubled from $18 to $36 and a month later was trading at $90. In November 1998 it had climbed to $130 a share.

Some other significant events in 1998 included the release of Traffic Server 2.0 with a streaming media cache, more protocols, and other performance and support functions. That same year, Inktomi acquired C2B Technologies for about $90 million, helping development of shopping search abilities and services for customers like Yahoo! and New Media. The shopping engine debuted in the spring of 1999. About the same time, Inktomi acquired ImpulseBuy.net for $110 million, providing a database and more capabilities for merchants. As its ecommerce shopping services expanded in 1999, Inktomi upgraded and released Traffic Server 3.0. Traffic Server 3.0 included more support for operating systems including Windows NT and new application programming interfaces allowing third party providers to provide value-added services. As a result, with Traffic Server 3.0’s release, Inktomi announced partnerships with six service partners.

In August 1999, a secondary stock offering was made raising $300 million. About the same time, Inktomi purchased WebSpective Software for $106 million. Towards the end of the year, Inktomi released Traffic Server E5000 and Traffic Server E200. These traffic servers helped corporate networks manage data for users without having to use the entire network and server infrastructure. At the end of 1999, America Online dropped Excite for Inktomi’s database to power its search engine. Likewise, MSN dropped AltaVista, whom they had moved to using earlier in the year, for Inktomi.

In 2000, Inktomi created alliances with several partners to enter the  wireless Internet infrastructure market. In June 2000, Inktomi acquired Ultraseek from Infoseek for $344.7 million. Inktomi announced creation of a 500 million record search engine database called GEN3. In August 2000, Inktomi created an alliance with America Online and Adero Inc. to form the Content Bridge distribution network. The Content Bridge would allow web producers and hosts of information and ecommerce, to pay to have their content pushed to caching servers of a large network of Internet hosting delivery providers. Right before the Content Bridge was to begin operation in January 2001, Adero backed out and sold its interest to Inktomi for $23.5 million.

In another big business deal in 2000, Inktomi acquired FastForward  Networks for $1.3 billion. FastForward Networks dropped its name and became a part of Inktomi’s media division. FastForward Network was a software developer for Internet broadcasts providing support for thousands of simultaneous Internet broadcasts. With this acquisition, Inktomi was able to release a product suite called Media Distribution Network that handled distribution of steaming media in a network. The Media Distribution Network suite complimented well their Content Delivery Suite for managing and distributing static content.

In 2001, Inktomi introduced its Search Everywhere solution integrating Inktomi’s various search products. Inktomi also commenced enhancing its search engine software with enterprise-level XML (Extensible Markup Language) and more comprehensive search results including relevance, classification and rankings. Inktomi added a distributed crawling architecture scanning the web more frequently with content blending from separate databases. In February 2001, Inktomi released Traffic Server 4.0 extending the platform to Linux operating systems with increased processing power and performance. In May 2001, Inktomi introduced a pay-for-placement program called Index Connect in which participants could submit meta information about multimedia and other files enabling them to appear in search results.

About the same time, in order to lessen the strain of its ecommerce business, Inktomi sold off its ecommerce division with its shopping  search engine and customer base to e-centives, an online marketing firm. Meanwhile, Inktomi expanded its content distribution services. In July 2001, Inktomi acquired eScene Networks with its streaming media sales. From this acquisition was developed the Inktomi Media Publisher with capabilities for business to catalog, index and publish their multimedia content.

Despite successful acquisitions and product releases, Inktomi sustained financial losses during 2001. With the dotcom bubble bust came economic strains to the Internet business sector. As a result, Inktomi cut back its workforce. Inktomi’s stock values decreased, causing a significant loss for the year.

In March 2003, Inktomi was purchased by Yahoo! for $235 million.   Inktomi continued to provide results to Yahoo! rival MSN Search. Meanwhile, Google continued to provide results to Yahoo! In February 2004, Yahoo! replaced Google with a search engine based on Inktomi’s technologies.

Ask Jeeves (now Ask) – Born in 1997

Ask was developed in 1996 by Garret Gruener and David Warthen and launched in 1997 as Ask Jeeves. In 2006, the “Jeeves” name was removed; revamping its image after Ask Jeeves was purchased in 2005 by Barry Diller’s InterActiveCorp (IAC). Originally as Ask Jeeves, human editors listed the prominent sites along with paid listings and results pulled from partner sites. Following acquisition of Direct Hit in 2000 and Teoma in 2001, Ask commenced developing its own search technology. With financial growth, Ask has acquired other companies including Excite and iWon.

Today, with emphasis on paid inclusion listings, Ask struggles for market share against Google, Yahoo!, and MSN Search.

Google – Born in 1997

Google was founded in 1998 as another school project at Stanford University in California. In January 1996, Stanford PhD students Larry Page and Sergey Brin began researching the concept of a search engine based on relevancy ranking. Page and Brin believed that search engines should analyze and rank websites based on the number of times search terms appeared on web pages. Likewise, Page and Brin developed a search engine nicknamed “BackRub.” BackRub checked the number and quality of links coming back to websites in order to estimate the value of a website. Brin and Page’s research eventually led them to develop the trademarked PageRank™ link analysis algorithm that Google’s search engine would use to assign a numerical weighting to hyperlinked document elements.

In 2000, Google replaced Inktomi as the provider of search results to Yahoo! and later AOL and Netscape. Even though Yahoo! broke away from Google in 2004, its market share has continued to grow to account for about 70 percent of all web searches. Google’s market share has steadily increased over the years.

MSN Search (now Windows Live) – Born in 1998

MSN Search was a service offered as part of Microsoft’s network of web services. The Microsoft Network debuted as an online service and Internet service provider in August 1995. During the 1990s, Microsoft launched Internet Explorer as a bundled part of their operating system and software products. MSN Search first launched in 1998 displaying search results from Inktomi and later blending results with Looksmart. For a short time in 1999, AltaVista search results were used instead of Inktomi. Since 2004, MSN Search began using its own built-in search results. Since this time, MSNBot has continually crawled the web. Today, image search is powered by Picsearch. MSN Search was renamed Windows Live in 2006.

Overture – Born in 1998

Overture is considered the pioneer of paid search. It was launched by Bill Gross in 1998 as GoTo. Overture was overpowered by Google when AOL selected Google as an ad partner. Overture bought AltaVista and AllTheWeb with hopes of gaining leverage against Google, but they ultimately were purchased by Yahoo! in 2003.

AllTheWeb – Born in 1999

AllTheWeb was launched in 1999 showcasing FAST’s Web Search Division search technologies. It is sometimes referred to as FAST or FAST Search. In April 2003, ALLTheWeb was bought by Overture for $70 million in 2003 and rolled into Yahoo! Search after Yahoo! purchased Overture.