The history of the search engine: from index cards to the AI chatbot

One of the greatest improvements brought about by the internet is the ability to find answers to almost anything almost immediately thanks to the search engine. How did we evolve from the days of index cards to AI-powered chatbots?




Search engines are essential to our day-to-day lives. No matter whether we are looking for the nearest pharmacy, searching for a new coffee machine to buy, or planning our next holidays. Everything starts with a search nowadays.

So, how did people find answers before the internet? That is a golden question. Even those of us old enough to remember a life pre-web struggle to recall how we did our homework, checked for correct spellings, or even resisted the urge to ask those questions we wouldn’t dare ask aloud without the relative safety of a search engine like Google.

And yet humans somehow managed to exist before the World Wide Web was created in 1989. Since then, search has improved exponentially to the point where a personal chatbot is now able to help you with your most routine tasks. But how did we reach this point? Let’s start with the basics.

What is a search engine and how does it work?

By now, search engines are a well-known concept. Programs like Google, Yahoo, Baidu, AOL, Duckduckgo are used by millions of people every day to search for and identify elements in endless databases. But how do they work?

  • First, search engine bots crawl all pages explicitly made available online. This includes downloading the content and following the links on the pages in order to index the searchable content.
  • Second, they index all pages, and perform a thorough analysis of titles, headings, content, and technical aspects.
  • Third, after a user types a query, the search engine uses complex algorithms to identify relevant pages and rank them hierarchically showing on top of the most relevant sites.

Search before the internet

Searching was far more laborious and in many cases would not even have even taken place before the creation of the search engine.

Index cards were first popularized by Carl Linnaeus to classify more than 12,000 species of plants and animals. In the years following his idea, libraries began to rely on them to index their collections.

Eventually, libraries settled on the Dewey Decimal System, which organized all books by subject, author and title – still in place across libraries today.

The first search engine

With the invention of the internet came the first example of what we are familiar with today. But it was not Google, Yahoo or even Ask Jeeves which were the first to introduce a whole new concept to us.

Archie was written in 1990 by Alan Emtage and indexed all the file lists of as many public FTP servers as possible to allow users to find and download publicly available files.

While it was not on a par with the search technology available today, it was indeed better than the alternative – word of mouth.

The web directory

For a while, it was regarded as the internet’s most important search engine, but that label did not fit the early versions of Yahoo. In fact, it was considered a web directory that relied on humans to summarize websites with short descriptions and to organize them into categories.

Created in 1994, Yahoo became so popular that publishers would delay posting their websites to ensure they would be included. Despite the advancements in search, the Yahoo Directory did manage to survive until 2014 when it was closed for good.

The first web crawler

1994 also saw the first web crawler released – appropriately titled, WebCrawler. It was the first to index entire pages and became so popular that at one point it could not be used during the day.

A search engine that Google arguably owes a lot to, Altavista was a pioneer in many of the online search techniques which we are still using today.

Notably, in 1995 Altavista became the first search engine to incorporate natural language technology. Among other achievements it also provided the first searchable full-text database of the web, allowed the multi-language search, and even translated pages.

Altavista’s move away from streamlined search towards a more complex web portal ultimately led to its demise as users flocked to the up-and-coming Google.


Which finally takes us to the granddaddy of them all. Google’s success can be attributed to many areas, but its most significant selling point is its famous algorithm which was able to yield within fractions of a second more relevant search results than its competitors.

In 1996 when Larry Page and Sergey Brin launched BackRub – Google’s precursor – they realized that their algorithm knew which webpage was the best for a topic based on accumulated links and, more importantly, citations from the most authoritative websites. It was this focus on the relevancy of a website that made Google so popular.

The history of search engines

Semantic search engines:

While Google was able to provide the world with answers to searches instantly, companies were still struggling to do the same on their own websites.

The Inbenta Semantic Search Engine was first created in 2010 and was able to understand searcher intent and the contextual meaning behind customer’s searches rather than rely on keywords. Much of this capability was due to Inbenta’s patented natural language processing which significantly improved companies’ self-service rates.


The concept of computers that could understand our voice had been around for 50 years or so before Apple’s Siri and Google both brought it into the mainstream.

Google added “personalized recognition” to Voice Search on its Android phones in 2010 as well as its Chrome browser in 2011. Its English Voice Search now incorporates 230 billion words from actual queries.

A Stanford Research Institute spin-off was sold to Apple in 2010 and led to Siri and its cloud-based processing. Ironically, its first offering was far more potent than the version embedded on our iPhones today – it was more intuitive, connected to the web, and could detect meaning from sentences more effectively.

The artificial intelligent chatbot:

Chatbots have existed since Eliza was billed in 1966 as the world’s first ‘chatterbot’ capable of communicating with humans as a psychotherapist would.

Only now have virtual agents started to make their mark in the search world by providing customers with information across all forms of social media as well as on company websites.

Many of them are powered by artificial intelligence and natural language processing which has provided users with a more personal experience when searching – think of a shop assistant minus the need to step out of your house.

The future of search engines: one chatbot to rule them all:

What is the next step in the search world? Chatbots are now starting to combine natural language processing with machine learning. This combination leads to agents that can provide high self-service rates and improve as it gathers more data.

Not only will bots become more accurate but we will soon be able to carry out all our searches as well as any transactions within a single conversation. Regardless of whether it is ordering a pizza, comparing the best energy prices or keeping up to date with the latest in the NBA it will all soon be handled within the same digital space.

The developers behind the search engine Ask Jeeves might have had a point when they decided to make a butler the face of their company. Search technology is doing all it can to adapt to us. It will continue to do so in ways that we cannot even comprehend.

Inbenta is a leader in natural language processing and artificial intelligence for customer support, e-commerce, and conversational chatbots providing an easy-to-deploy solution that improves customer satisfaction, reduces support costs, and increases revenue.

Interested in finding out more? Our team of experts is available to show you how Inbenta can benefit your company.


Check out our similar articles