Do It Yourself Vertical Search Engine

There are a number of companies, big and small, trying to build a better search engine.  Whether it be vertical search engines, or more general search engines with better algorithms than Yahoo & Google, they all face a big hurdle to success.  Simply stated, the hurdle is the near zero marginal cost of doing a search.   And the more companies I see the less comfortable I get that any will cross that hurdle.

The promise of vertical search is a good one.  If the search engine focused on one sector, such as shopping, blogs, electronics or health, they don't have to crawl the entire web, and focus on much fewer sites and hone their algorithms to deliver more relevant results faster.  As a result, the company will attract more relevant ads, therefore both cost per click and click through rates will be higher than others.  Eventually, the company will be runaway success, reach out and conquer other verticals, and then eventually take over from Google.

Almost all of these companies, when fundraising, take a keyword, like "digital cameras"or "san francisco" and compare their highly relevant search results with the generic results you get from Google.  It looks great.  However, what's sorely missed is that when comparing search engines, one should not compare results for a given keyword, but compare the overall cost of finding the information that's desired.  If you type the wrong keyword, you may have to refine your search, lengthen the phrase, or reach out to page 2-3-4 of the search results to find what you need.  Eventually you find what you think is what you want.  It may take 1-2 minutes longer than in vertical search engine, but I claim that that cost is not a big problem for any consumer.  Let's test that hypothesis.  How many times have you searched something in Google and got frustrated concluding "I just can't find what I need!".  I would guess that that is never.

When a user tries a number of keywords or lengthens his/her phrase, what they are effectively doing is adding relevance manually.  That is what the vertical search engines try to do for you, but when the cost for the consumer doing it themselves is nearly zero what do they bring to the table?  If "soccer" is too generic, try "weekend soccer" or "weekend soccer san francisco", eventually you bring the relevance you are looking for and you do it happily without complaining.  So what's the real value proposition for the added relevance from a vertical search engine and how does that compare to the cost of acquiring the users and getting them to your web page routinely?  That's the hard question I have not yet seen the answer for.

As long as the cost of doing another search on Google is zero, I see a difficult time for vertical search engines to keep the relevance edge over time.  I would love to be proven wrong.

Comments

[Balaji]

Baris,

Nice post. Can you please shed some light on image search or media file search ? Do you consider them to be a vertical search or different ?

Coming back to other vertical search engines that you'd mentioned, I guess those companies are not just focussed on grouping all relevant information in one place. They are grouping it properly. For example, if I am planning for a trip to Lake tahoe, I would really want to goto a site where I can get everything that is required for the trip. I think that's what is focussed on all vertical search engines.

If I have to do that all by myself by searching on Google, then it's very likely that I'll miss something or ponder around for information.

Same is true with Health search. WebMD is still widely used for health related search and I heard Kosmix is getting traction too in that area.

[zheng]

Yahoo has launched the Search Engine Builder a few weeks ago. http://builder.search.yahoo.com/.

You build a vertical search engine limiting search to a set of sites of your choices. They even allow your ads on it.

[baris]

Balaji,
I don't consider image searh or media file search vertical search yet, because they are very primitive and not much more than text search on file names or tags.

I think Cem got it right here (http://www.typepad.com/t/trackback/5923224)
Vertial search has to do more than better search, be a mashup or bring in new content. Then it's not just search.

[Nitink]

Baris:

This is a great post - you make some good points. I agree
with you that if the only advantage offered by
Vertical Search Engines (VSEs) is to provide quicker and
more relevant results by limiting the scope of their
crawl/search to only a subset of the web, then that is
not enough to draw users away from Google, Yahoo, MSN et al.

But vertical search engines, by virtue of focusing on a
specific domain, can use their deep understanding of that
domain to help users solve important problems in
that domain - find a job, buy an iPod, research and book
a vacation. They can encourage communities in their area of
interest, integrate crawled data with proprietary content
and offer domain-specific comparison and configuration tools
based on structured metadata. I discuss this in detail in
this blog post on my blog. Examples of companies that are
already doing this are Shopping.com, SimplyHired and
Healthline.

What do you think? Is the addition of customized,
integrated content and services into crawled search results
enough to draw users away from the major search engines
for specific tasks?

[SolutionPoint]

I suppose more than search technology it the understanding of consumer search behavior that is more important to create a successful engine like Google. Technolgy that can appeal to masses has to be simple and marketed that way. Google excels in it plus it became a global brand synonymous with search + it has the first-mover advantage also. What you guys think? If you get time do visit my SEO Blog and put your comments.

[SEO Services Provider India]

Search Engine Optimization optimizes the content of website and build link popularity so that the website can achieve high ranking and visibility in several search engines.Seo Services India can not only provide SEO Services but also guide for maintaining the top position in Leading Search Engines for keywords.
Site : http://ww.ecodesolution.com

[Searchboth]

Searchboth.net is the first site to place google and yahoo side by side on one split screen. The web site takes the user's query and creates a browser window with two frames, with the results from Yahoo! on one side and those from google on the other. It has completely end up the hassles involved while searching different search engines at the same time.

[Search Engine Optimization Tips and Tutorial]

Good article; I never heard of vertical search engine before.