Basics of Search Engine

What is a search engine?

The purpose of a search engine is to extract requested information from the huge database of resources available on the internet. Search engines become an important day to day tool for finding the required information without knowing where exactly it is stored. Internet usage has been tremendously increased in recent days with the easy to use search engines like Google. How does a search engine work? Functioning of a search engine depends on the type and basically there are two types of search engines widely used. Crawler based search engines Human powered directories Hybrid search engines Working of crawler based search engines There are four basic steps which every crawler based search engines follow before displaying any sites in the search results. Crawling Indexing Calculating Relevancy Retrieving the Result  All crawler based search engines need relevant time for crawling and indexing the new content which may vary from a day to week.   Search engines crawl the whole Web to fetch the web pages available. Crawling is performed by a software, called a crawler or a spider (Googlebot for Google and Bingbot for Bing). The crawling frequency depends on the search engine and it may take a few days between crawls. This is the reason sometimes you can see your old or deleted page contents showing in the search results. New content will get updated in the search results once the search engines crawls your site again.   Indexing is next step after Crawling which is a  process of identifying the words and expressions that best describe the page and assigning the page to particular keywords. Sometimes crawler may not understand the meaning of your page right but if you optimize your page, it will be easier for the crawlers to classify your pages properly and your site will get higher ranking in the search results.  Search engine compares the search string in the search request with the indexed pages in the database. Since it is likely that more than one page contains the search string, search engine starts calculating the relevancy of each of the pages in its index with the search string. There are various algorithms to calculate relevancy. Each of these algorithms has different relative weights for common factors like keyword density, links, or meta tags. That is why different search engines give different search results pages for the same search string. What is more, it is a known fact that all major search engines periodically change their algorithms and if you want to keep at the top, you also need to adapt your pages to the latest changes. This is one reason to devote permanent efforts to SEO, if you like to be at the top. The last step in search engines' activity is retrieving the results. Basically, it is simply displaying them in the browser –  the endless pages of search results that are sorted from the most relevant to the least relevant sites. Human powered directories  Human powered directories also referred as open directory system depends on human based activities for listings. The process of indexing will follow like below: Site owner submits a short description of the site to the directory along with category it is to be listed. Submitted site is then manually reviewed and added in the appropriate category or rejected for listing. Keywords entered in a search box will be matched with the description of the sites. This means the changes made to the content of a web pages are not taken into consideration as it is only the description that matters. A good site with good content is more likely to be reviewed for free compared to a site with poor content. Hybrid Search Engines use both types of crawler based and manual indexing for listings the sites in the search results.