Long Tail distribution.

< List of probability distributions

A long tail distribution represents data that doesn’t have a sharp drop off but instead tapers off more gradually. Unlike a normal distribution where most of the data falls around the mean, data in a long tail distribution usually includes many outliers or extreme values. In this blog post, we’ll dive deeper into what a long tail distribution is, how it differs from other types of distributions, and why it’s relevant in today’s data-driven world.

Long tail distribution definition

Loosely speaking, a long tail distribution has many values stretching to one side of a peak. Long tail distributions are a subset of heavy-tailed distributions, where many values fall into the extremes; heavy-tailed distributions may have long and thin or short and fat tails.

To understand a long tail distribution better, let’s first look at some examples. Long tail distributions can be found in a variety of settings, from sales figures to word usage in a natural language. For instance, the sale of books on Amazon demonstrates a long tail distribution where a small number of best-selling books account for the majority of sales, while the rest of the books sell in smaller numbers.

Statistics wise, a long tail distribution typically has a power-law shape, represented in the above graph. This means that the frequency of occurrence of an event is inversely proportional to its rank. In other words, the more common an event is, the less likely its rank will be high. Conversely, the less common an event is, the more likely its rank will be high.

There is a more precise definition, given by Foss et al. [1]

An ultimately positive function f is defined according to a limit. If the ratio of x + y and x converge to 1 as x-values get much larger, then the distribution is long-tailed:

We can also say that if f₁, f₂, … , f_n are all long tailed distributions, then any distribution that can be expressed as a product or as a linear combination of f_x= f₁ · f₂, … , · f_n is also long tailed. That means that any function

is long tailed, where c₁, c₂, … , c_n are positive constants.

Long tail distributions in real life

So why is understanding the long tail distribution relevant in today’s world? As we’ve become more data-driven, we’re creating and collecting vast amounts of data. The long tail distribution plays a crucial role in understanding this data. For example, in a business setting, it’s essential to understand the long tail distribution of sales data to determine not only which products are the bestsellers but also identify untapped markets that may make a significant contribution to overall revenue.

One industry that’s particularly benefited from understanding the long tail distribution is the entertainment industry. With the advent of streaming services such as Netflix and Spotify, companies can extract more value from content libraries by analyzing user preferences and recommending niche content based on long tail distribution insights. This, in turn, means that individuals can discover new content that suits their tastes better.

So how can you identify a long tail distribution? A common way to do this is by plotting the data on a log-log scale, which makes the long tail distribution more visible. However, this may not paint a clear picture if your data has noise in the tail. Other possibilities [2]:

1. A basic histogram (which may be a poor choice for noisy data)
2. Histogram on logarithmic scales
3. Histogram with logarithmic bin sizes
4. A cumulative frequency distribution,
5. A rank-frequency plot.

References

[1] Foss et. al (2013). Heavy Tailed and Long Tailed. Retrieved December 26 2013 from: http://www.springer.com/cda/content/document/cda_downloaddocument/9781461471004-c1.pdf?SGWID=0-0-45-1395304-p175250259

[2] Newman, M. Power laws, Pareto distributions and Zipf’s law