Today, the person who has more information on his hand is more powerful. To have versatile knowledge we need to have versatile information with us and for that we need to have a strong search engine. We have many search engines available around us. For some specific search criteria one search engine can give better search results than others. Based on the search criteria it needs to identify which search engine has better results. So, a comparison among these search engines can be demanded. In our report to find this comparison we have used both qualitative and quantitative method. Our analysis has made a clear distinction about the performance of these search engines which have been measured based on some search criteria. The ultimate result has proved that Google is pioneered for searching for most of the search criteria while others are a bit lagging behind. Again most of the search engines produce better results for Boolean and keyword based searching.
Keywords: Search Engine, Keyword Search, Metadata Search, Precision, Recall
Table of Contents
Introduction 1
Structure of the Paper 1
Background 1
Description of the Research Area 2
Problem Definition 2
Research Questions 2
Goal 2
Objectives 2
Method 3
Method Discussion 3
Limitations 4
Extended Background 4
Search engines and basic search techniques 5
Truncation (Wildcards) 6
Boolean search approach 6
Keyword based search 6
Metadata search 6
Language search 7
Precision Vs Recall 7
Data collection 8
Prepare Search Queries 8
Calculate precision and recall 9
Analysis 11
Discussion 14
Conclusion 15
Introduction
Structure of the Paper
This paper has been structured according to the following manner.
Setting query criteria
Search query results and find their relevancy in different search engines
Observing the precision and recall of the search results with the different factors
Making comparison among different search engines based on above observations
Background
We have many search engines available. The popular search engines are Google, Yahoo, Windows live search, AOL, Ask, AllTheWeb, and AltaVista etc. Searching information in the web using these search engines is a common practice among human in the modern web based society. Now-a-days, we cannot live without searching information as it is the key of power and knowledge. It is not possible to acquire versatile knowledge for a single person. But according to the necessity human can ask for the knowledge while they are connected with internet and have a search engine in front of them. So, information mining or searching in the web is a regular practice. Human have their own preference for choosing search engines. So far various reports show that Google is the most favorite and used search engine worldwide [1, 2]. Other general purpose search engines are not being used as widely as Google. As an example based on US Internet usage 71.9% volume of search ratio belongs to Google[3]. Other search engines have obviously lower search ratio. All of these search engines are keyword based search engines[3]. They search documents matching with the given keywords in queries. They have different algorithms for searching and indexing the resultant pages or documents. That is why getting the most relevant search results for the same queries differ from search engine to search engine. The searching queries may be formed in multiple ways for the same topic. Thus even the results may vary within the same search engine for these differently formed queries. A clear observation of the relevancy of search results for these queries can be considered for tabling the comparisons of these search engines. The differently formed queries can be categorized according to the keyword based search, Meta data search, language based search, searching with wildcards etc.
A clear finding out of the strength of these search engines need to be addressed. So, our research area is about defining comparisons among different keyword based search engines.
Description of the Research Area
So far many researches have been conducted on finding various measurements regarding search engines. Mostly evaluation of the search engines have been done based on finding the precision and recall of the search results. Besides performance of search engines based on queries regarding metadata (e.g. image search), different languages, semantic search etc. has been measured [4-6].
It needs to define a transparent comparison among these search engines to identify their performance level. Depending on the relevancy of the retrieved documents for the differently formed query set the comparisons can be captured. The approach will be an experimental effort. For this experiment the most general purposed and keyword based search engines like Google, Yahoo, Bing (Windows Live Search) and Ask will be used.
Problem Definition
There are many online free search engines available throughout the world. They are ranked by their number of hits. Number of hits or human preference does not indicate the performance level. Depending on the performance issue a comparison within these search engines should be made.
Research Questions
Which search engines retrieves comparatively most relevant documents for the first several hits? Analyzing and answering this question we will measure the proficiency of the search engines and thus this question will be indicated as our research question.
Goal
The goal of this study is to make comparison among various search engines.
Objectives
The objectives of this paper refer to the ways that indicate how the goal of this research has been achieved. The following indications identify the means of achieving the desired goal.
Literature study and review in terms of query searching via search engines
Defining the query categories covering different search criteria
Evaluating the query search results
Observing the evaluation of query search results
Method
The method of this research is the combination of both qualitative and quantitative approach. Since our goal is to compare the search engines on the basis of some key search criteria, we have used inductive reasoning logic. Observing the consequences from the search results that are based on the set search criteria, we will infer the existing antecedents. From these antecedents we will generalize the observations that will draw the comparison among the search engines.
Method Discussion
The qualitative approach of this research convey the literature study and review for articulating the proper reasoning in terms of indicating comparison criteria among different search engines. The quantitative part of this paper is for evaluating the different query results. On the basis of these query results the comparison will also be indicated. This quantitative analysis will draw the performance of different search engines indicating the precision and recall of the query results. This quantitative approach can be discussed according to the following steps.
Firstly, the queries will be classified for performance measurement of the search engines.
Secondly, the relevant documents for the set queries will be retrieved from the search engines.
Thirdly, the precision and recall for the retrieved documents will be measured by the following formulas:
Formula 1: Precision Finding Formula[7]
Formula 2: Recall Finding Formula[7]
Fourthly, the precision–recall curves will be drawn to evaluate the performance of different search engines.
Finally, analyzing these curves for various search engines under different classification criteria the comparisons will be depicted as a generalized format.
Finding the existing antecedents from the precision-recall curve for various search criteria the comparison will be generalized and thus the inductive reasoning will also be maintained.
Limitations
The number of retrieved documents in first several pages for a specific query may vary for the time span due to their frequency of hits and search engines’ internal algorithms. So, a variation of analyzed results may occur time to time.
Extended Background
In this chapter we will discuss about the theoretical background of this research. The important keys and terminologies related to the topic for understanding it will be described here based on some prior studies by different authors or relevant sources.
Search engines and basic search techniques
Search engine is the engine for information retrial from a wide collection of sources. According to the Webopedia definitions search engine is “a program that searches documents for specified keyword and returns a list of the documents where the keywords were found”[8]. The top most search engines ranked by U.S. searches for August 2009 are Google, Yahoo, MSN, AOL, ASK etc.
Figure 1: Top ten search engines for August, 2009 [2]
The frequency of searching by Google is the highest and has a quite handsome difference than others. Although these differences show the level of human interaction with the specific search engine, it cannot conclude their performance level.
Search engines search information from web using hyper text transfer protocol (http) and return us the most relevant documents they find in the World Wide Web according to their strength. There are many techniques to get better search results from web. But all of them are not appropriate for all search engines [9].
“The ability to support advanced search options of native database interfaces, utilize Boolean, wildcard and proximity operators, and allow field searching are also significant functions”[10].
The variation of the search results can be observed using these functions. In this paper we have considered the following search techniques for building up search queries.
Wild cards
Boolean search approach
Keyword based search
Besides we will evaluate search engines with considering queries with different language and metadata respectively.
Query with Bangla language
Query for image searching (Metadata searching)
Truncation (Wildcards)
Truncation can be used for searching the documents related with truncated matched words. The asterisk (*) symbol is generally used for truncation and it is called wildcards. As example- if we truncate a word like “matern*”, it will bring all the documents where this pattern can be found in words (e.g.: maternal, maternity etc.). Wildcards can give more relevant results enhancing the effectiveness of search mechanism [11]. We can include some conditions for getting extended collection of information when we are roaming around the search queries.
“The ability to express typed conditions with wildcards is very useful to the users and it is necessary not to restrict the expressive powers of users’ queries against the object views of Web data sources”[12].
This is why we considered wildcards for forming queries and checking through selected search engines.
Boolean search approach
Presently the search queries that we use for information retrieval is mostly based on natural language form. But queries can also be formed using Boolean operators like AND, OR and NOT. The keywords joining with the AND operator will be retrieved in the resultant documents. The documents containing any of the keywords will be retrieved that are joined with OR operator. NOT operator will discard those documents where the NOT adjacent keyword will not be present.
Keyword based search
Keyword is the major or explanatory word of a query. Most of the search engines do the searching based on the keywords. Keywords are the most important key of searching information. As many keywords we can include in our searching, as much possibility it grows to find the most relevant documents.
Metadata search
Metadata refers the data that gives information about the data. So, metadata gives the data about data. Image, video, audio etc. are such type of data that are not self explanatory. We cannot search them according to their content. As a result, to identify their content we need to provide information about their contents. These data are basically the file objects. Files have their additional attributes like file type, created date, owner, title etc. Providing the information of these attributes we can search these data or files by the search engines. To get the most relevant data it requires specifying this information as accurately as can be done.
So searching these file objects to evaluate the search engines is another approach of this paper.
Language search
We have huge collection of information in the web in different languages. The search performance of the search engines can also be evaluated considering the queries in different language format. As an experiment in this paper we have considered the Bangla language.
Precision Vs Recall
We will find out the precision and recall from the query results. Precision is the fraction of retrieved documents that are relevant whereas recall is the fraction of relevant document that have been retrieved[13]. The query results will be plotted for getting the precision-recall curve that will guide us to find out comparatively better collection of retrieved documents by various search engines. The following figure shows the quality of precision-recall curves for retrieved documents.
Figure 2: Precision-Recall Curve’s Quality
The curve with red color has bad retrieval of documents. The blue curve has average retrieval and the green curve has a very good retrieval of documents. Theoretically the dotted curve has the best retrieval as it returns all the available documents where every one of them is relevant. But, in reality it is not possible.
“If an algorithm always retrieves all documents in a document base, it has one hundred percent recall. However, it presumably has lower precision … Precision at a certain point of recall indicates how much garbage readers have to wade through until they know they have found at least half of the interesting documents.[14]”
The characteristics of our analyzed curves will be measured according to this quality oriented precision-recall curves.
Data collection
Data collection has been conducted by browsing through different search engines for some specific search criteria. The first retrieved 30 documents for each query and each search engine have been enlisted as the preliminary experimental data. Manually checking each document the relevancy has been found. Based on the relevant documents and total retrieved documents the precision and recall has been calculated.
Prepare Search Queries
The preliminary and most important task of this data collection part is to set the search criteria and their sample queries. We have discussed earlier that there are different search techniques available for finding documents and according to these techniques we set our search criteria on six categories. These search criteria are as follows.
C1: Query with formal communicative words (including stop words )
C2: Query with Boolean search
C3: Query with based on keyword search
C4: Query with other language (Bengali)
C5: Query with wildcards
C6: Query focused on metadata search
For these six search criteria or categories (C1…C6) we have defined six queries (Q1…Q6) that belong to these categories. The defined queries are as follows.
C1
Q1: What are the maternal health problems in the rural area of Bangladesh?
The, in, of -> stop Words
C2
Q2: Maternal AND (Health OR Problems) AND (Rural AND Bangladesh)
AND, OR -> Boolean Operators
C3
Q3: Maternal health problem rural Bangladesh
All the words are keywords
C4
Q4: ??????? ?????? ????
Bengali words substitute to “risk during maternal period”
C5
Q5: matern* health problem in Bangladesh
*= wildcards; matern* should retrieve all the words containing ‘matern’ (e.g.: maternity, maternal etc.)
C6
Q6: Location of KTH Kista Campus Sweden
These query is for image search module of search engine (e.g.: http://images.google.com/ )
Table 1: search criteria and their adjacent sample queries
Calculate precision and recall
The set queries have been experimented for retrieving documents through the target search engines. The precision has been calculated with the regular difference of each 5 documents. As example, within top 5 documents we have found the number of relevant documents. Again, for next five documents that means within 10 documents we have found the total number of relevant documents. In this manner we have searched up to 30 retrieved documents.
So, for 30 documents with regular distant of 5 documents 6 precision points can be gained as follows.
Precision points = Nord top5/5, Nord top10/10, Nord top15/15, Nord top20/20, Nord top25/25, Nord top30/30
[Nordtop[n] = Number of relevant documents for top n retrieved documents]
All the relevant documents cannot be retrieved by a specific search engine. We can get some common retrieved and relevant documents for several search engines. For finding out the recall the set of relevant documents will be the summation of all the relevant documents found by the selected search engines. Since for each search engine we have considered the first 30 documents as experiment, we have found all the relevant documents within the first 30 documents of each search engine.
So, for each search engine we can again get 6 recall points for the regular distant of 5 documents as follows.
Recall points= Nordtop5/Rdoc, Nordtop10/Rdoc, Nordtop15/Rdoc, Nordtop20/Rdoc, Nordtop25/Rdoc, Nordtop30/Rdoc
[Nordtop[n] = Number of relevant documents for top n retrieved documents
Rdoc = Number of total relevant documents]
Below we have enlisted the detail precision (P) and recall (R) value for the search engines Google, Yahoo, Bing and Ask with respect to the set queries (Q1…Q6).
Q1
Q2
Q3
Q4
Q5
Q6
P
R
P
R
P
R
P
R
P
R
P
R
Top 5
5/5
5/54
4/5
4/67
5/5
5/70
4/5
4/14
4/5
4/47
4/5
4/16
Top 10
8/10
8/54
9/10
9/67
9/10
9/70
7/10
7/14
5/10
5/47
7/10
7/16
Top 15
12/15
12/54
14/15
14/67
14/15
14/70
9/15
9/14
9/15
9/47
9/15
9/16
Top 20
17/20
17/54
18/20
18/67
19/20
19/70
9/20
9/14
9/20
9/47
10/20
10/16
Top 25
19/25
19/54
23/25
23/67
22/25
22/70
9/25
9/14
10/25
10/47
13/25
13/16
Top 30
22/30
22/54
27/30
27/67
26/30
26/70
11/30
11/14
12/30
12/47
16/30
16/16
Table 2: Found precision and recall in Google for the queries (Q1...Q6)
Yahoo
Q1
Q2
Q3
Q4
Q5
Q6
P
R
P
R
P
R
P
R
P
R
P
R
Top 5
3/5
3/54
4/5
4/67
3/5
3/70
1/5
1/14
5/5
5/47
Top 10
6/10
6/54
9/10
9/67
6/10
6/70
4/10
4/14
10/10
10/47
Top 15
8/15
8/54
12/15
12/67
8/15
8/70
12/15
12/47
Top 20
11/20
11/54
17/20
17/67
11/20
11/70
13/20
13/47
Top 25
16/25
16/54
21/25
21/67
15/25
15/70
18/25
18/47
Top 30
18/30
18/54
26/30
26/67
17/30
17/70
20/30
20/47
Table 3: Found precision and recall in Yahoo for the queries (Q1...Q6)
Bing
Q1
Q2
Q3
Q4
Q5
Q6
P
R
P
R
P
R
P
R
P
R
P
R
Top 5
3/5
3/54
3/5
3/67
3/5
3/70
1/14
3/5
3/47
0/2
Top 10
4/10
4/54
8/10
8/67
8/10
8/70
4/10
4/47
Top 15
5/15
5/54
12/15
12/67
11/15
11/70
4/15
4/47
Top 20
10/20
10/54
17/20
17/67
16/20
16/70
4/20
4/47
Top 25
14/25
14/54
22/25
22/67
19/25
19/70
6/25
6/47
Top 30
16/30
16/54
25/30
25/67
22/30
22/70
8/30
8/47
Table 4: Found precision and recall in Bing for the queries (Q1...Q6)
Ask
Q1
Q2
Q3
Q4
Q5
Q6
P
R
P
R
P
R
P
R
P
R
P
R
Top 5
1/5
1/54
4/5
4/67
5/5
5/70
3/5
3/14
2/5
2/47
Top 10
5/10
5/54
6/10
6/67
8/10
8/70
5/10
5/14
4/10
4/47
Top 15
8/15
8/54
9/15
9/67
13/15
13/70
7/15
7/14
8/15
8/47
Top 20
9/20
9/54
9/20
9/67
17/20
17/70
7/20
7/14
10/20
10/47
Top 25
11/25
11/54
9/25
9/67
20/25
20/70
7/25
7/14
10/25
10/47
Top 30
14/30
14/54
10/30
10/67
24/30
24/70
8/30
8/14
10/30
10/47
Table 5: Found precision and recall in Ask for the queries (Q1...Q6)
Analysis
Figure 3: Precision-Recall Curves for query q1-q5 (experimented on Google)
For query one (q1) we have fair amount of retrieval. Out of first 30 documents we got 22 relevant documents. So, we have high precision. But out of total 54 relevant documents we had only 22 relevant documents that ensure a lower recall.
While searching with Boolean expression (q2) we have got a very good amount of retrieval. Out of first 30 documents we got 27 relevant documents that ensure a high precision. But the number of total found relevant documents are 67 that imply a lower recall.
Keyword based search (q3) also resulted a good precision with lower recall.
Query four (q4) was about searching with Bengali language. For this special search Google returned a very bad precision with high recall. Out of total 14 relevant documents it returned 11 of them whereas it only returned 11 relevant documents from 30 retrieved documents.
Searching with wildcards (q5) retrieved a wide collection of documents. Both the precision and recall are lower for this case. It returned only 12 relevant documents out of 47 relevant documents.
Figure 4: Precision-Recall Curves for query q1-q3, q5 (experimented on Yahoo)
Here we have plotted precision-recall curves for query q1, q2, q3 and q5. Query with Bengali language returned a limited amount of documents. It returned only 9 documents where only three were relevant.
According to the graph Yahoo has returned higher recall for q1. It produced comparatively lower precision.
For Boolean search (q2) Yahoo has returned higher precision and medium recall. Out of 67 relevant documents it retrieved 26 of them.
Yahoo has very bad recall for keyword based search (q3). Out of 70 documents it retrieved 17 of them.
For wildcard searching (q5) Yahoo has average precision-recall quality curve. Out of 47 relevant documents it retrieved 20 relevant documents.
Figure 5: Precision-Recall Curves for query q1-q3, q5 (experimented on Bing)
Again, for the search engine Bing we have got four curves. Query with Bengali language did not return a single document.
Bing has produced lower recall for q1. Out of 54 relevant documents it retrieved only 16 of them. However q2 has higher recall and precision. For q3 we have medium precision but lower recall. Out of 70 relevant documents we have only 22 of them.
Query with wildcards (q5) for Bing has returned a very poor precision recall curves.
Figure 6: Precision-Recall Curves for query q1-q5 (experimented on ASK)
For search engine ASK we have got five curves. Query (q4) with Bengali language returned only 30 documents and among these documents we have only 8 relevant documents. So, it has lower precision. Since we have in total 14 relevant documents for q4, it results a very high recall rate.
According to the figure the search result for q1 has a very lower precision and its recall is also lower because out of 54 relevant documents it only retrieved 14 of them.
Boolean search (q2) for ASK has resulted poor precision-recall curves. Only 10 relevant documents were retrieved out of 67 relevant documents.
For keyword based (q3) search the resultant precision-recall curves is fair. It has higher precision and medium recall.
Searching with wildcards (q5) for search engine ASK has returned a very lower precision as well as lower recall since we have got in total 47 relevant documents where only 10 of them were retrieved.
Discussion
Generally for a query, a search engine retrieves many documents relating with the given terms in the query and most of the time we check first couple of pages to get our desired answers. Many relevant documents can be found in the deeper pages of search results. Since it is not possible for a human to go through all the pages, we picked first 30 documents from each search engine and thus we accumulated our set of relevant documents from total of 120 documents. Since we may have many relevant documents in the deeper of the resultant pages, the analyzed recall will not give us accurate points to be plotted. Rather it may give us a generic idea about the resultant curves.
The search results on the set search criteria evaluate the considered search engines. For evaluating these search engines we have considered a scoring system. The scoring system defines the quality of the precision of the curves. According to the system the ‘scoring rage’ will define the range of relevant documents and the ‘scoring quality’ will define the quality of precision.
Scoring Range (no of relevant documents)
Scoring Quality (for precision)
Very few retrieved documents
Undefined
<10
Very Poor
10-14
Poor
12-19
Fair
20-24
Good
25-30
Very good
Table 6: Scoring System for determining precision quality
Now According to this scoring system the quality of precision for the search results and the corresponding search engines can be found in the following table.
Search Criteria (Query)
Natural Language Search (Q1)
Good
Fair
Fair
Poor
Boolean Search (Q2)
Very Good
Very Good
Very Good
Poor
Keyword Based search (Q3)
Very Good
Fair
Good
Good
Search with Bengali Language (Q4)
Poor
Undefined
Undefined
Very Poor
Search with Truncation (wildcards) (Q5)
Poor
Good
Very Poor
Poor
Metadata Search (Q6)
Fair
Undefined
Undefined
Undefined
Table 7: performance determination for various search engines
The precision-recall curves determine that for search criteria one Google has the best quality curve.
For Boolean search Google also produce the best precision recall curve. For this case, Yahoo and Bing also produce very good quality curves.
Again Google is pioneered for keyword based searching. It produces very good precision-recall curve respect to other pioneered ASK and Bing.
For query with Bengali language we couldn’t plot the precision and recall points for Yahoo and Bing as they could not find enough documents. Yahoo in total retrieved 9 documents and only 2 of them were relevant. Again, Bing only retrieved 2 documents and one of them was relevant. For this special case Ask had very lower precision. Although Google produces poor precision, it retrieved lots of documents.
While searching with truncation approach (using wildcards) Yahoo got better results than Google and other search engines. Although for this case the quality of precision-recall curves is not good for all of the search engines, Yahoo got slight better results than others.
The last search criteria were on metadata search. The probability of getting an object (image, video) on searching increases depending on the information provided against it. For this searching Google produced fair precision-recall curve. All other search engines could not find any relevant items.
The overall observation depicts that the Boolean search works best for information searching. All selected search engines produced good results for Boolean search except Ask. Then the keyword based search also works fine for information retrieval. It is applicable for all search engines, since search engines generally search documents on basis of keywords.
During searching we have also observed the following compatibility issues for targeted search engines. The tick symbol (v) denotes that the observed phenomenon is supported by the search engines. The cross symbol (X) denotes that it is not supported by the search engines.
Auto Suggestion
Spelling Suggestion (tried: Animol)
Stemming
computation
Truncation
Compute*
Yahoo
X
Bing
X
X
ASK
X
X
Table 8: General phenomenon supported by the search engines
Conclusion
Our analysis showed the precision and recall of the query results. These have been made based on the relevancy of the search results. Defining the quality of curves we have got the performance of search engines for various search criteria. The results show that Google has better performance for all the search criteria except truncation. For truncation Yahoo produces comparatively better results. Besides comparatively all the search engines have good performance regarding Boolean and keyword based searching.
Every search engine has their own mechanism of indexing the web pages so that they can be optimized for searching. If these mechanisms change, search results can be varied. Again better hits can give a page better preference of being shown within first few results. So, the relevancy for the specific query and search engine can also be varied time to time. Again, due to less hits a relevant page can go down of the search results.
This is Preview only. If you need the solution of this assignment, please send us email with the complete assignment title: ProfessorKamranA@gmail.com
