Sunday, March 31, 2013

The inevitability of the unlikely

Do you remember anything from that statistics class you took years ago? If you do, it's probably the "bell curve", aka the "normal distribution". That's a good thing to remember, because the bell curve is a really useful tool for understanding some aspects in the real world. If you were really paying attention, you remember all that stuff about mean and standard deviation, and maybe even skew and central limit theorem.

Attack of the 50 ft woman!

Like me, I'm sure you're concerned about the possibility of a gigantic swimsuit clad supermodel rampaging through our cities. After all, it's happened twice in the movies, once in 1953, and again in 1993. Just how likely is this scenario? Let's look at the height of women, over the age of 20. The statistics are based on the U.S. National Health Statistics Report - October 22, 2008.

As you can see the average height of adult females is around 64 inches, or about 5'4". The laws of statistics tell us that 99.7 percent of women will be between 4'3" and 6'4". Well, what do you suppose the odds are of a woman being 20 feet tall? See if you can find it on the graph - that's about 243 inches. (50 ft didn't fit)

That's right - the chances are infinitesimally small (around 1.0E-399). So, what are the chances of being attacked by a 50 foot woman - zero actually. There are real biological reasons that a human can't grow to be 50 feet tall. She would collapse and suffocate under her own weight. Maybe the 50 foot woman could live in the ocean, eating krill and sticking her nose out of the water to breathe. Water might support her colossal mass like it does for whales, but she could not survive on land. In any case, she wouldn't be much of a menace. 

Like many natural systems systems constrained by biological and physical limits, human height matches the bell curve pretty well

Why Volkswagen was the world's most valuable company: for a few minutes.

Some things don't fit well under a bell curve. There are lots of examples, including executive compensation, attendance of rock concerts, and lottery payouts. What these examples have in common is that a few rare events have an unprecedented impact.

A good example is the stock price of the German auto company Volkswagen. In early 2008, the company stock was trading for around €120 a share. The executives at rival German auto maker Porsche, decided to  take control of Volkswagen. They started by buying all the VW stock they could get their hands on. They also bought stock options to guarantee a reasonable price for shares once the news of the takeover became public. Porsche acquired 43% of all VW stock, and had options for another 32% - controlling a total of 75%. Another 20% was not for sale - leaving only 5% of VW shares on the market.

Stock options are complicated, but allow me to simplify the problem: Options are a type of contract promising to sell stocks in the future, for an agreed price. The seller doesn't have to own any shares when they sell the option - it's just a promise. Some of these sellers are betting that they can buy the stocks in the future more cheaply than they can now. They can make a lot of money if prices go down - or lose a lot if they go up.

Here's how it all went crazy: Porsche called up the people who owed them options, and said  "Gib mir meine Bestände!" ("Give me my stocks, now!" )
 The option sellers had to rush out and buy 32% of all VW stock - oops! Only 5% of the stock was for sale. This is called a "short squeeze".

Price history of Volkswagen shares (VOW). Hint: This is not a bell curve.
The buyers were desperate; sellers raised prices - the price shot up to over €1000 a share for a few minutes. Guess who had the most stock to sell? Why Porsche, of course. Porsche sold €1000 shares to the option sellers, so the option sellers could sell it back to them for €120 a share. Not a bad deal for Porsche Automobil Holding SE. (Ironically, some options traders had to sell their Porsche and buy a Volkswagen.)

Life in Mediocristan and Extremistan

The height of women, and the price of VW stock are examples of what economist Nassim Nicholas Taleb  humorously refers to as Mediocristan and Extremistan. These two fictional countries represent the predictable world of things like height, and the insanely unpredictable world of stock prices.

Map of the "Stans"

In Mediocrastan, everything fits under a bell curve. In Mediocrastan rare events are rare - and when they do occur, they can be explained. Yao Defen of China is seven-feet-eight-inches tall. Despite being the tallest woman in the world she belongs in Mediocristan. She she is a mere 90 inches on our graph above (+5 sigma).

Residents of Extremistan include Bill Gates and the price of VW stock on October 23, 2008, and most lottery grand prize winners. Bill Gates is has a net worth of $63 Billion, as compared to the mean U.S. net worth of $556,300*. If Bill Gates's height were proportional to his wealth, he would stand at a height of about 100 miles.

"In Extremistan, rare events are common"

How can this be? Any single rare event is, by definition, rare. But when you take all the possible rare events that can occur, chances of one of them occurring are high. For example, the odds of being struck by lightning are 1 in 280,000. The odds of being killed in an air-crash are 1 in 335,000. Both are pretty long odds. But, when you also add in the possibility of being eaten by a shark, hit by a meteor, bit by a black mamba, eaten by piranhas, run over by a bulldozer, and a thousand other rare events, the odds start adding up.

You probably knew someone who was killed by a rare event - maybe several. The odds of that event were tiny, but given all the possible rare events, and all the people you have known, the chances are pretty high.

Useful information:
  • Rare events are common, you just never know which event!
  • Options trading is riskier than you may think
  • The bell curve does not apply to everything
  • Bill Gates is 100 miles tall, financially speaking
Congratulations you're one of those rare people who can read a whole blog post about statistics. Many of the ideas in this post are misrepresentations of The Black Swan: The Impact of the Highly Improbable, by Nassim Nicholas Taleb

*$556,300 is the average U.S. net worth, including Bill Gates and Warren Buffet. Median is $120,300.

The Internet has Ruined Jokes

Years ago people told each other jokes. In my family, it would work something like this: my uncle in New York would hear a joke - he’ll call my dad and tell him - my dad would tell my mom, and I would overhear it - I would tell it to my friends. Everybody got to hear a joke, everybody got to tell a joke. It was a win-win all around.

Today most jokes are delivered wholesale by the electronic media, and they aren’t as funny. Maybe you see them on, or somebody spams you with the joke, or maybe you heard Garrison Keillor tell it on NPR. Most computer transmitted jokes aren’t even told – they're just forwarded, or re-posted on Facebook.

It’s time to revive the art of telling jokes. If we’re going to make this work, we need some guidelines.

  1.     Jokes may not be emailed or posted on social media.
  2.     Jokes must be told in person or live over voice media.*
  3.     One (1) point will be awarded for telling a joke.
  4.     Two (2) points are awarded for telling a funny† joke.
  5.     Three (3) points are awarded for making up and telling an original‡ joke.
  6.     Five (5) points are awarded for making up and telling a funny original joke.
  7.     A pun, however funny, may be awarded no more than two (2) points.
  1.     Forgetting the punch line – loss of one (1) point
  2.     Repeating the punch line – loss of one (1) point for each subsequent repetition.
  3.     Repeating any joke told by Jeff Foxworthy– loss of one (1) point.
  4.     Telling a dirty joke that isn’t funny – loss of three (3) points.
Please join me in the revival of this once great American tradition.

*Telephone, VOIP, or two-way radio. SMS and live chat are excluded. (Rule under review 3/30/2013)
†The joke will be considered funny if the audience exhibits genuine mirth.
‡Wholly original content or substantially modified punch-line.

Tuesday, March 19, 2013

The Dunning Kruger Effect, or why you shouldn't listen to bloggers

This should make you paranoid - but don’t worry, if it does, you’re fine. If it doesn't make you paranoid, it should. Confused? That’s a good thing!

In 1999 two Cornell University psychologists named David Dunning and Justin Kruger conducted 
a simple study comparing what students thought they knew, and what they actually knew. The results provide some deep insights into human behavior, and a cautionary tale to those willing to engage in some self-criticism.
The Study
Students were tested on humor, logic, and grammar. They were given an exam, and then asked to evaluate their own performance. Actual test scores were compared with the self-evaluation. Logic and grammar are easy to test objectively, but you may wonder about testing a subjective subject like humor. The study came up with a pretty good metric; the student’s answers were compared with a panel of experts - professional comedians.

 The worse a person was at the test; the more they overestimated their abilities. The absolute worst performers rated themselves nearly on par with the best performers. The worst performers also rated themselves above all but the very top performers.

It Gets Worse!
In a follow up study students were asked to grade the work of their peers on the test. Not surprisingly, the worse they scored, the worse were at judging the skills of others. In other words, they didn’t have the expertise to recognize real expertise in others.

Dunning and Kruger provide some insightful analysis of the results. Read the paper 
HERE. You probably don’t have the time to read it yourself, so here’s what I got out of it:
Four Key Points
1. Incompetent people think they are competent
2. Very competent people doubt their own abilities.
3. Incompetent people can’t recognize competence in others.
4. You have been, and will be, one of those people sometimes.
"Does this remind you someone you know?"

Actually, most people immediately recognize the Dunning Kruger effect from abundant incidents in their own life: a terrible boss, a pedantic teacher, an ignorant coworker who thinks he knows everything. We’ve all seen it in small children – they read a book on dinosaurs and proudly announce that they know everything about dinosaurs. Of course it’s OK in kids because we know their experience is so limited. If kids knew how much they had to learn, they would never try anything new. It’s a great coping mechanism for someone just starting out. A friend of mine once said “Kids are a lot like people”. When we see the same thing in adults, it is no longer cute.
"Could it apply to me?" 

Paradoxically, most of us never see ourselves as the incompetent one.  Of course, you and I have never fallen prey to the Dunning Kruger effect.

A few years ago, I was brought into a new project. I immediately looked around and saw that “everyone was doing it wrong!”  Everyone was making a big fuss about what was really very simple. Not so many years ago, I would have confidently explained to everyone within earshot how to fix everything.  Luckily, I kept my mouth shut. After a few months, I started to see the complexity of the project. Four years later I’m getting a sense of all the things I don’t know. Think how much I won’t know after ten years! It makes me appreciate the patient people who took the time to wean me from my ignorance.
"That explains why incompetents rate themselves too high. Why do competent people rate themselves too low?"

The explanation given by Dunning and Kruger is the “
False Consensus Effect”.  People tend to overestimate how much other people agree with them. They know enough to recognize the limits of their own knowledge. Competent people assume that everyone else is competent like them, making them average. The good news here is that when good performers are asked to rate the work of others; they often recognize this incorrect assumption.
How to Use This Information

This study is packed with insights. Here are some of the actionable ones:
  • The first, and obvious thing is don’t believe people’s self-assessment of their abilities. If they are incompetent, they don’t know it. If they are well above average, they may not know that either.
  • Recognize that the effect applies to you. Be humble. Test your assumptions before make a fool of yourself. 
  • Recognize that incompetent people don’t know they are incompetent. Just showing them the facts won’t convince them, because they won’t recognize the facts. You’re going to have to educate them before they can understand. 
Beware of Bloggers

Some bloggers are legitimate experts, but most aren’t. Sometimes that’s OK. Bloggers are usually more like journalists than experts. Journalists fill the role of publicizing and popularizing the work of others. This is important work, but it shouldn’t be mistaken for genuine expertise.

For complex and difficult issues, journalists (and especially bloggers) often get the facts wrong. Most real journalists are held by their editors and publishers to minimum standards.  Unfortunately, editors aren’t experts, and publishers are usually more concerned about issues like profitability. Bloggers don't have any standards at all.

If you’ve read this far, ask yourself about the author of this article. I’m not a PhD in psychology and I’ve never done any original research in the field. I can only hope that my esteemed readers will take me to task if I get the facts wrong.

Monday, March 18, 2013


[Disclaimer: This is re-post from another blog, now defunct. Unfortunately, I lost some rather excellent comments when I saved this copy.]


Here is an idea that 99 percent of people declare impossible. Ninety-nine percent of the rest can’t figure out how it’s done. The idea is known as DDWFTTW, or "Directy Downwind Faster Than The Wind".

It starts with a simple, but seemingly stupid question:
“Can a wind powered vehicle go downwind faster than the wind?” 
Being a logical person who has taken high school physics, the answer is simple. Impossible! It violates the laws of conservation of energy – i.e. it is a perpetual motion machine. Once the vehicle is going the same speed as the wind, the wind can no longer push it – so it obviously can’t work.

Back in 2001, a scientist named Rick Cavallaro answered the question "Yes", and published his speculation about such a vehicle on The Internets. He drew a complex diagram to illustrate his point.

Furious debates and name-calling ensued, (no, not the republican presidential race.) Prominent physicists, professors, and engineers all agreed it was impossible.  When the vehicle was finally built and it worked as designed, there was a brief hushed silence, quickly followed by embarrassed shouts of “Hoax!” 

Here it is working

How does it work? I can’t say I understand it well, but the important point is that it’s not at all like a sailboat. The wind pushes against the surfaces of the vehicle, which starts the wheels rolling, which turn gears, which turn the propeller. The propeller pushes against the wind; spinning opposite to the way a windmill spins. The wheels and propeller work together to push the vehicle. It doesn't violate the laws of physics because the pushing surface of the propeller is not going faster than the wind. 

Here is a good illustration of the principle, appropriately narrated by a German version of Mister Rogers.

Mister Rogers explains the physics of DDWFTTW

If you are interested, Wired Magazine does a great job of telling the story 

© 2008 Raoul Rubin