The rage-to-conclude bias sees patterns in data lacking such patterns.
The bias leads to premature, simplistic, and false inferences about causality.
Good statistical analysis seeks to calm down the rage to conclude,
to align the reality of the evidence with the inferences made from that evidence.
The calming down process often involves statistical analysis sorting out
the "random" from the "nonrandom," the "significant" and the dreaded
"not significant," and "explained" from "unexplained" variance.

Alas this statistical language is swamp of confounding puns and double-talk
that play on the differences between technical and everyday language.
Perversely, these puns also play to the rage to conclude, as in "explained
variance" derived from a cherry-picked, over-specified model on a small
data set--which is very unlike actually explaining something.

In everyday work, the integrity of statistical practice is too often compromised.
Researchers easily finesse standard statistical validation practices by posing
ridiculous alternative hypotheses, by searching millions of fitted models and
publishing one, and by little tilts in data analysis (and 3 or 4 little tilts will
generate a publishable finding).

The conventional practices for insuring analytical integrity, as implemented in
day-to-day research, have not headed off a vast literature of false findings.
See John P. A. Ioannidis, "Why Most Published Research Findings Are False,"
and the stunning recent report that only 6 of 53 "landmark studies" in clinical
oncology could be replicated (one researcher turned out to have done an
experiment 6 times, it "worked" once, and only the one-off was published).

Can our data displays be designed to reduce biases in viewing data--
such as the rage to conclude, cherry-picking, seeing the nonrandom in
the random, recency bias, confirmation bias? How can visual inferences
from data be improved to separate the real from the random, the
explained from the unexplained? How can the inferential integrity of
graphics viewing, as in exploratory data analysis, be improved?

Here are some practical ways to improve the quality of inferences
made from data displays.

All statistical displays should be accompanied by a unique documentation box

For both consumers and a producers of data displays, a fundamental task is to assess
the credibility and integrity of graphical displays. The first-line quality control
mechanism for analytical integrity is documentation of the data sources and the
process of data analysis.

The data documentation box accompanying the display should provide links to the
data matrix used in the graphic display, the location of those data in the full data set,
names of analysts responsible for choosing and contructing the graphic, and whether
the project's research design specified in advance the graphic (or was published graphic
an objet trouve, a found object, such as Marcel Duchamp's famous conceptual
artwork Fountain,1917?).

Some journals require documentation of the data analysis process. For example, from
the British Journal of Sports Medicine:

Contributors: All authors had full access to the data in the study. JLV takes responsibility
for the integrity of the data and the accuracy of the data analysis. He is guarantor.
JLV and GNH designed the study. GNH, DWD, NO, JLV and LJC acquired the data.
JLV, EAHW, LJC and GNH performed the analysis and interpreted the data.
JLV drafted the manuscript, which was critically revised for intellectual content
by all co-authors.

Note the assignment of personal responsibility, a guarantor of data integrity/accuracy
of analysis. (The study's finding was that watching television a lot was correlated
with a substantially shorter lifespan. However intuitively pleasing, that finding
was hopelessly compromised by use of nonexperimental data that confounded
multiple causes and effects.)

A documentation box makes cherry-picking of evidence more obvious, and may
thereby serve as a check on cherry-picking by researchers. A documentation box
also allows reviewers of the study to assess the robustness of the results against
the plausible alternative hypothesis that the findings were generated by data-selection
rather than by a fair reading of the overall data. Published data displays too often
serve as research sanctification trophies, serving up pitches to the gullible, based
unstable over-selected results.

All computer programs for producing statistical displays should routinely provide
a documentation box, just as they now provide a plotting space, labels, and scales
of measurement. Why haven't those programs done so years ago?

Diluting Perceptual Cluster/Streak Bias:
Informal, Inline, Interocular Trauma Tests

When people look at random number tables, they sees all kinds of clusters
and streaks (in a completely random set of data). Similarly, when people are
asked generate a random series of bits, they generate too few long streaks
(such as 6 identical bits a row), because their model of what is random
greatly underestimates the amount of streakiness in truly random data.

Sports and election reporters are notorious for their
streak/cluster/momentum/turning-point/trendspotting
narrative over-reach. xkcd did this wonderful critique:

To dilute streak-guessing, randomize on time over the same data,
and compare random streaks with the observed data.
Below, the top sparkline shows the season's win-loss sequence
(the little horizontal line = home games, no line = road games).
Weighting by overall record of wins/losses and home/road effects
yields ten random sparklines. Hard to see the difference between
real and random.

The 10 random sparkline sequences can be regenerated again and
again by, oddly enough, clicking on "Regenerate random seasons."

The test of the 10 randomized sparklines vs. the actual data is an
"Interocular Trauma Test" because the comparison hits the analyst right
between the eyes. This little randomization check-up, which can be repeated
again and again, is seen by the analyst at the very moment of making
inferences based on a statistical graphic of observed data. (Thanks to Adam Schwartz for his excellent work on randomized sparklines. ET)

This is looking a bit like bootstrap calculation. For the real and amazing
bootstrap, applied to data graphics and contour lines, see Persi Diaconis
and Bradley Efron, "Computer Intensive Methods in Statistics."
For recent developments on identifying spurious random structures
in statistical graphics, see the lovely paper by Hadley Wickham, Dianne Cook,
Heike Hofmann, Andreas Buja, "Graphical inference for infovis."

Edmond Murphy in 1964 wrote about dubious inferences about 2 underlying
causal mechanisms based on staring at bimodal distributions (from Edward Tufte,
The Visual Display of Quantitative Information, p. 169:

Sparklines can help reduce recency bias,
encourage being approximately right rather than exactly wrong,
and provide high-resolution context

From Edward Tufte, Beautiful Evidence, pages 50-51.

Sparklines have obvious applications for financial and economic data—
by tracking and comparing changes over time, by showing overall trend
along with local detail. Embedded in a data table, this sparkline depicts
an exchange rate (dollar cost of one euro) for every day for one year:

Colors help link the sparkline with the numbers: red = the oldest and
newest rates in the series; blue = yearly low and high for daily exchange
rates. Extending this graphic table is straightforward; here, the price of
the euro versus 3 other currencies for 65 months and for 12 months:

Daily sparkline data can be standardized and scaled in all sorts of ways
depending on the content: by the range of the price, inflation-adjusted
price, percent change, percent change off of a market baseline. Thus
multiple sparklines can describe the same noun, just as multiple columns
of numbers report various measures of performance. These sparklines
reveal the details of the most recent 12 months in the context of a
65-month daily sequence (shown in the fractal-like structure below).
   Consuming a horizontal length of only 14 letterspaces, each sparkline
in the big table above provides a look at the price and the changes in
price for every day for years, and the overall time pattern. This financial
table reports 24 numbers accurate to 5 significant digits; the accompanying
sparklines show about 14,000 numbers readable from 1 to 2 significant digits.
The idea is to be approximately right rather than exactly wrong. 1
   By showing recent change in relation to many past changes, sparklines
provide a context for nuanced analysis—and, one hopes, better decisions.
Moreover, the year-long daily history reduces recency bias, the persistent
and widespread over-weighting of recent events in making decisions.
Tables sometimes reinforce recency bias by showing only current levels
or recent changes; sparklines improve the attention span of tables.
   Tables of numbers attain maximum densities of only 300 characters per
square inch or 50 characters per square centimeter. In contrast, graphical
displays have far greater resolutions; a cartographer notes "the resolving
power of the eye enables it to differentiate to 0.1 mm where provoked to
do so." 2  Distinctions at 0.1 mm mean 250 per linear inch, which implies
60,000 per square inch or 10,000 per square centimeter, which is plenty.

1  On being "approximately right rather than exactly wrong,"
see John W. Tukey, "The Technical Tools of Statistics,"
American Statistician, 19 (1965), 23-28.

2  D. P. Bickmore, "The Relevance of Cartography," in J. C. Davis
and M. J. McCullagh, eds., Display and Analysis of Spatial Data
(London, 1975), 331.

Here is a conventional financial table comparing various return rates of
10 popular mutual funds: 1

This is a common display in data analysis: a list of nouns (mutual funds,
for example) along with some numbers (assets, changes) that accompany
the nouns. The analyst's job is to look over the data matrix and then decide
whether or not to go crazy—or at least to make a decision (buy, sell, hold)
about the noun based on the data. But along with the summary clumps
of tabular data, let us also look at the day-to-day path of prices and their
changes for the entire last year. Here is the sparkline table: 2



Astonishing and disconcerting, the finely detailed similarities of these
daily sparkline histories are not all that surprising, after the fact anyway.
Several funds use market index-tracking or other copycat strategies, and
all the funds are driven daily by the same amalgam of external forces
(news, fads, economic policies, panics, bubbles). Of the 10 funds, only
the unfortunately named PIMCO, the sole bond fund in the table, diverges
from the common pattern of the 9 stock funds, as seen by comparing
PIMCO's sparkline with the stacked pile of 9 other sparklines below.
   In newspaper financial tables, down the deep columns of numbers,
sparklines can be added to tables set at 8 lines per inch (as in our example
above). This yields about 160 sparklines per column, or 400,000 additional
daily graphical prices and their changes per 5-column financial page. Readers
can scan the sparkline tables, making simultaneous multiple comparisons,
searching for nonrandom patterns in the random walks of prices.

1  "Favorite Funds," The New York Times, August 10, 2003, p. 3-1.

2  In our redesigned table, the typeface Gill Sans does quite well
compared to the Helvetica in the original Times table. Smaller than
the Helvetica, the Gill Sans appears sturdier and more readable, in
part because of the increased white space that results from its
smaller x-height and reduced size. The data area (without column
labels) for our sparkline table is only 21% larger than the original's
data area, and yet the sparklines provide an approximate look at
5,000 more numbers.

-- Edward Tufte