White Paper: The Long Tail of Search

download here (pdf, 2.5 MB) :

White Paper: The Long Tail of Search – Introduction and Recommendations by QueroMedia


The phrase ´The Long Tail´ was first coined by Chris Anderson in an October 2004 Wired magazine article to describe certain business and economic models such as Amazon.com and Netflix . This landmark article covered how the media and entertainment industries will succeed, not by pushing only mass market hits that are popular with many, but by also tapping into the “Long Tail” of interest of a few in less popular books, songs, movies and more.

Upon entering a bookstore, the first shelf you reach is often the bestsellers´ section. Because storage and distribution costs are high, traditional retailers carry only the most popular products. The Long Tail states that the large amount of misses is equally important, and perhaps even more profitable than the total number of hits. It is therefore suitable as a business model for companies who aim at the internet as their primary market. Shelf space is virtually unlimited on the internet.

The purpose of this paper is to look at the principles of the Long Tail in a search marketing context. I will explore the theory with a case study in search engine marketing, but this requires some understanding of the assumptions of the model. ´The Long Tail´ is a real buzz word, but as is mostly the case with buzzes, its actual meaning is often unclear. Therefore I will start by giving a short theoretic introduction.

The Long Tail

If you ask 100 people what percentage of the top 10.000 titles on Amazon they think will sell at least once a month, chances are the average of all guesses will be close to 20%. The reason for that is we have been trained to think that way! The Pareto principle (also known as the 80-20 rule) states that, for many phenomena, 80% of the consequences stem from 20% of the causes. This rule of thumb was named after the Italian economist Vilfredo Pareto, who observed that 80% of Italy´s income was earned by 20% of the Italian population. The assumption is that most of the results in any situation are determined by a small number of causes.

But in this case, the Pareto principle does not apply. The right answer, according to Anderson (2004), is 99%. There is demand for nearly every one of those top 10.000 titles. An important proposition of the Tail theory is that unlimited choice will create unlimited demand.

Let´s take books as an example. The average Barns & Noble carries 130.000 titles, whereas Amazon can deliver almost any book or magazine ever printed. But if we take a closer look at Amazon´s sales statistics, surprisingly enough, more than half of the total sales are made outside of the top 130.000 titles. It seems there is another market of books that are not being distributed by the average bookstore. It is even reasonable to believe that the market for this share is even bigger than the market for the more popular titles. In other words, the potential book market may be twice as big as it appears to be.

The beauty of it all is that if we dig below the bestsellers, it seems like the demand curve keeps going on for ever, even if it is just a few buyers per year. It literally goes beyond the moment when a local bookstore goes to zero, because it can´t carry any more books, or the few interested readers never enter the store. This is the Long Tail.

The Long Tail is a long-known feature of statistical distributions, in which a high-frequency or high-amplitude population is followed by a low-frequency or low-amplitude population which gradually ´tails off´. In many cases the infrequent or low-amplitude events can comprise the majority of the graph. The term is derived from the XY graph that is created when charting popularity to inventory. Along the horizontal axis it has the full inventory of items or services available for purchase. The vertical axis shows the number of units sold.

Supply and demand issues

We are stuck in a hit-driven mindset. We naturally assume that if a product is not a bestseller, it won´t provide a good enough return on investment. A key factor here is the cost of inventory storage and distribution. When inventory storage and distribution costs are insignificant, it becomes economically viable to sell relatively unpopular products. However when storage and distribution costs are high, only the most popular goods can be sold.

Traditional warehouses are restricted by the limitations of the physical world, so the need to find local audiences for their product can become an issue. An average record store has to sell at least a certain number of copies of a CD per year to make it worth carrying. A traditional warehouse can only attract customers from a certain radius around its location, perhaps a couple of kilometers for a record store, so the pressure to find a good area of distribution is high. When physical space is an issue, an audience too thinly spread is the same as no audience at all.

Fact is that a bricks and mortar warehouse like eg. Free Record Shop has only limited shelf space. The cost for this storage space is calculated in the form of overhead . In order to maximize its profits, Free Record Shop stocks only the most popular titles to ensure that no shelf space is wasted. Retailers will carry only products that can generate sufficient demand.

On the other hand, if we look at an online shop such as the iTunes Music Store, the situation is completely different. The online iTunes Music Store sells media files that are accessed through Apple´s iTunes application, that can be played on an iPod. Because iTunes only sells digital files, the inventory storage (gigabytes on a server park) and distribution cost (bandwidth) is significantly lower. Those economics of storage and distribution enable Apple to a profitable use of the Long Tail. This means that iTunes is able to offer a wider range of music than a traditional bricks and mortar warehouse, and they are able to set the price to digital costs, not physical ones.

The basic principle behind the Long Tail is that mass product or content aggregators can make money selling or renting content that appeals to a very small niche. While any niche represents only a small market opportunity, the sum of the revenues from all these niches can be very large. Limited shelf space and distribution bottlenecks are no longer an issue.

But that alone does not suffice to keep the Tail together. Demand must follow the newly created supply, or else the Tail will fall apart. The enormous expansion in choice is transformed into an economic and cultural force only by the sales and the participation of all those people in the new niches. The Long Tail is irrelevant if the niches can´t be filled by people who want them.

In the next chapter we will take a closer look at these forces that drive the Long Tail.

On the origin of Long Tails

A major economic trigger is indispensable for the origin of Long Tails: the reduction of costs to get to niche markets. The reason why these costs will drop varies from one market to another, but will almost certainly have something to do with one or more of the next tree forces.

1. democratization of means of production

Democratization of means of production can best be illustrated with the example of the personal computer. PC´s allow you to create, edit and distribute movies, compose music, write books, teach you how to be a professional photographer, etc. The digitization of our age lowers the threshold for creative minds to realize their goals. Not every product of the masses is a gem, but every now and then something valuable is created. This is why the universe of available content is increasing more rapidly than ever. Just think of MySpace.com: a social networking website offering an interactive, user-submitted network of friends where you can post your own music and videos internationally. Opportunities like these make the Tail grow even larger.

2. democratization of distribution

The second force consists of the lowered consumption costs due to the democratization of the distribution of products. The personal computer made everyone a potential producer or editor. The internet makes sure that everyone can be a distributor on a global scale. Just think of the power of eBay: an online auction and shopping website where people and businesses buy and sell goods and services worldwide. The internet makes the liquidity of the market in the Tail increase. Consequence: the Tail thickens.

3. connecting supply and demand

On the demand side, tools such as search engines, recommender software (“Customers who bought this also bought€¦”) and sampling tools allow customers to find products outside of their geographic area. This is the difference between push and pull, between broadcast and personalized taste.

These thriving forces offer a series of new opportunities in the growing Long Tail. Democratized means of production lead to a swift increase in the number of producers. Super efficient digital economies cause the rise of new markets and new marketplaces. The possibility to use the diffused intelligence of millions of consumers to match people with their products of preference, will eventually lead to new recommendation and marketing methods.

Long Tail aggregators are companies and services that collect a vast amount of goods and make sure they are available for all audiences, preferably in one place. Popular examples of Long Tail aggregators are Amazon, eBay and iTunes.

Aggregators can be split into three retail models: physical, hybrid and pure digital retailers. The first is the traditional bricks and mortar store. Hybrid retailers are companies that sell physical goods online, such as Amazon. With the help of digital catalogs they can offer an unlimited selection along with search, reviews, and recommendations. Pure digital retailers, such as iTunes, offer the additional savings of delivering their digital goods online at virtually no or a marginal cost.

Long Tail filters are tools that help you find what is right for you in the massive variety of the Tail. A Long Tail without good filters is just noise. The most frequent examples are recommendations and search. In current ´Long Tail´ markets, filters lead people from a familiar world (hits) to an unknown world (niches) by way of a comfortable route that is customized for their preferences. This is exactly what Google does on the search engine results page: it filters the web and lists only the pages that match your query.

So if it is Google´s objective to select the most relevant websites that match your query from the massive amount of data that is available on the web, wouldn´t it be wise to tap into that selection process and make maximal use of it?

That is exactly what we as a search marketing company are trying to achieve for our customers every day. In the next chapter I will elaborate on the Long Tail of Search by analyzing its properties and describing what the implications are in order to perform successful search marketing campaigns.

Search´s Long Tail

In brief, search marketing (or search engine marketing) encompasses three aspects. Search engine optimization (SEO) is the process of increasing the online visibility by optimizing the website and making it ´search engine friendly´. Search engine advertising (SEA) includes the creation, management and optimization of pay-per-click advertising campaigns. Pay-per-click or PPC means that the advertiser has to pay when someone clicks on his ad. Conversion optimization (CO) exists out of two aspects: usability and web analytics.

Both SEO and SEA rely heavily on the specific query people type in the search box, and more specific on the top converting keywords. Based on that query, a search engine like Google will display organic search results and sponsored listings in a certain ranking. It is in your best interest to rank as high as possible on this search engine results page.

Because search engines like Google act as a Tail filter on the massive amount of information the internet offers, it is important to reflect on the possibilities to make maximal use of it. QueroMedia uses a nearly unlimited amount of search terms in its campaigns, instead of focusing only on a few general queries which will presumably generate a large amount of traffic.

Let me clarify this with an example: a shoe salesman will target his search engine activities on “shoes” and “running shoes”, two general keywords which will drive a lot of traffic, but will also be very expensive because there is a lot of competition for that term. We as search marketing professionals use a different approach: we believe that it would be wise to use comprehensive keyword lists in our campaigns. Our collection of keyword will next to “shoes” and “running shoes” also include “white running shoe”, “nike tennis shoes”, “puma running shoe Antwerp”, “black leather spike heel boots” etc. Our keyword approach supports the use of “Tail terms” – relatively low volume, low-cost phrases composed of two or more keywords

Our working-method is based in part on the Google and Yahoo! keyword suggestion tools. These Keyword Tools suggest potential keywords for ad campaigns and report their statistics, including search performance and seasonal trends. With the Yahoo! Keyword Selector Tool, it is feasible to get a list of 100 related search queries that were executed on the Yahoo! Search engine in a certain month. Search marketers use this basic tool to determine which keywords are likely to drive most traffic.

For this test, I entered the keyword “shoe” in the Yahoo! Keyword Selector Tool. The results displayed in the graph are the top 100 queries related to shoes that were executed on the Yahoo! search box in February 2007. As you can see, there is a large number of queries that happen far less often than the “leading” terms like “shoes” or “running shoes” at the head of the list. Most queries form the Long Tail that is illustrated behind the head.

But what is the point of including a comprehensive keyword collection in your search marketing campaigns, when the most popular keywords on their own drive a lot (maybe enough) of traffic to your website? As far as website optimizing (SEO) is concerned, the more keywords you can get your website to rank on, the better. Moreover, according to D. Sullivan (2005), traffic generated by keywords in the Tail is reported to convert better than less general terms. According to R. Sullivan (2005), the bottom 20% of searches generates 60% of the sales. These keywords are more focused and specific, and thus more likely to convert. For example, a search for a generic term like “lawyer” is less likely to convert than “Antwerp divorce lawyer”. Most sites will have the majority of their business (sales and leads) generated from these specific terms that are rarely tracked.

I tested this theory on the account of one of our clients, an online shop that sells all kinds of sport shoes all across Europe. An initial report from this Google AdWords-campaign showed that the top 5 terms brought in around 50% of the site´s overall referrals. These top 5 terms include brand names such as Nike and Adidas. However, when looking at sales generated by search terms, 30% of conversions were from top 5 keywords. Conversely, 70 % of the conversions were from search terms outside of the top 5. That is the proof the Long Tail in Search is profitable.

Please note that these results were measured on one of the most popular search engines in the world. Next to the ´Major Three´, Google, Yahoo! & Windows Live, dozens of secondary search engines help people to find whatever they may be searching for online. In terms of the Long Tail, these specialized search engines filter out noise better than major search engines.

Secondary search engines can be divided into three categories: “Tier II” engines (eg. Looksmart), vertical-oriented engines such as Business.com, and shopping engines like Froogle. The benefits of secondary search engines include more volume, particularly on words too broad for major engines, lower minimum cost-per-clicks, less competition and more hands-on customer service. Many advertisers fail to include secondary engines in their ad spend, but they prove to be a worthy asset to a search marketing campaign.

Conclusion & Recommendations

The theory of the Long Tail offers an interesting point of view on the emerging digital market. According to the model, the media and entertainment industries will succeed not by pushing only mass market hits that are popular among many, but by also tapping into the “Long Tail” of interest among a few in less popular books, songs, movies and more. The basic principle behind the Long Tail is that mass product or content aggregators can make money selling or renting content that appeals to a very small public.

A key factor is the cost of inventory storage and distribution. Where inventory storage and distribution costs are insignificant, it becomes economically viable to sell relatively unpopular products. However when storage and distribution costs are high, only the most popular goods can be sold.

The reduction of costs to get to the niche markets lies at the base of the Long Tail. The personal computer made everyone a potential producer or editor. The internet makes sure that everyone can be a distributor on a global scale. Thanks to tools such as search engines and recommendation software, all what is available is easily accessible.

Viewed in the context of search engine marketing, the Long Tail can be thought of as a series of search practices that tap into this new paradigm of retail behavior.

Here are a few pointers that will help you create and manage targeted, cost-effective search marketing campaigns, making use of the Long Tail.

– For search engine optimization purposes, it is important that you create a sufficient amount of relevant content on your website. Essentially, the Long Tail is the hundreds to thousands of keywords and key phrases that a site is found for, yet rarely noticed or exploited. Try to tap into the vast amount of subjects and keywords related to your website, but be careful not to add superfluous content. Robots of search engines are fierce but fair. Why not create an additional website/blog to attract the Long Tail of search?

– The same holds for search engine advertising campaigns. Remember to build comprehensive keyword lists. Remember to include typos. (Chihuahua ” Chiwawa) Don´t forget to look at the end of the Tail, because there keywords are more focused and specific, and thus more likely to convert. Long Tail search is less competitive and it narrows the market.

Consider to include secondary search engines in your search advertising spend, because they may offer interesting opportunities to boost conversions.

Last but not least, track the sales your site is generating by keyword. If you want to keep it simple, you can subscribe to hittail.com, an online tool that is focused solely on exploring the Long Tail of your website. Do you really want to turn your website inside out? A free tool such as Google Analytics tells you everything you want to know about how your visitors found you and how they interact with your site.

Tom Michiels, team leader search engine advertising

download here (pdf, 2.5 MB) :

White Paper: The Long Tail of Search – Introduction and Recommendations by QueroMedia