All 4 books by Edward Tufte now in
paperback editions, $100 for all 4
Visual Display of Quantitative Information
Paper/printing = original clothbound books.
Only available through ET's Graphics Press:
catalog + shopping cart
Edward Tufte e-books
Immediate download to any computer
connected to the internet:
La représentation de l'information
quantitative 200 pages $12
La Representación Visual de Información
Cuantitativa 200 páginas $12
Visual and Statistical Thinking $2
The Cognitive Style of Powerpoint $2
Seeing Around + Feynman Diagrams $2
Data Analysis for Politics and Policy $2
catalog + shopping cart
Edward Tufte one-day course,
Presenting Data and Information
Bethesda, September 28
Baltimore, September 29
Arlington, October 1, 2
Indianapolis, November 9
Columbus, November 10
Cleveland, November 12
San Francisco, December 14, 15, 16
PowerPoint Does Rocket Science--and Better Techniques for Technical Reports
Jeff Bezos, Amazon CEO, abandoned PPT in 2012. His presentation replacement = ET method for doing meetings:
-- Edward Tufte
Return to Flight Task Group biographies
Here are the biographies of those on the Return to Flight Task Group who saw the NASA engineering by PowerPoint and denounced it in their final report (quoted extensively in the last 2 pages of my essay).
As I wrote in the essay above, "Both the Columbia Accident Accident Investigation Board (2003) and the Return to Flight Task Group (2005) were filled with smart experienced people with spectacular credentials. These review boards examined what is probably the best evidence available on PP for technical work: hundreds of PP decks from a high-IQ
government agency thorough practiced in PP. Both review boards concluded that (1) PP is an inappropriate tool for engineering reports, presentations, documentation; and (2) the technical report is superior to PP. Matched up against alternative tools, PowerPoint loses."
Here are the biographies of the NASA PowerPoint critics:
-- Edward Tufte
Elizabeth Lane Lawley, a professor visiting Microsoft, comments on "the culture of the
Her experience at Microsoft is comparable to that of the NASA Return to Flight Task
Group with regard to the persistent disutility of using PP decks to replace technical
-- Edward Tufte
Cognitive Style of PowerPoint 2nd edition now published
Now available: 2nd edition of The Cognitive Style of PowerPoint as 32-page essay or ebook. This essay is also a chapter in Beautiful Evidence.
For more information and to order paper copies: click here
To order ebook: click here
-- Edward Tufte
ET helps NASA with Probability Risk Assessments (PRA), for upcoming launch
Here is a link to William Harwood's excellent account of shuttle risks in the upcoming flight, scheduled for this Saturday. This link provides context for my comments that follow. http://www.spaceflightnow.com/shuttle/sts121/060629preview/part1.html
About 18 months ago in Houston I reviewed the shuttle Probability Risk Assessment (PRA) material for NASA. PRA works with a list of possible threats, estimates their probablilities and expected losses, and then seeks to assist decision-making for shuttle risk-reduction.
After the PRA group presented their results, I had two major suggestions:
(1) They should prepare a detailed summary matrix (on, of course, 11" by 17" paper), ordering the
risks and providing, in a comments column, relevant background for each estimate. Let
that intense matrix, backed up by similar more-detailed 11" by 17" arrays of risk
estimates, be the main presentation device and analytical tool for making decisions. This
was designed to replace their chippy and twiddly PP slides, which made a hash of their good technical work and made it difficult to assess the overall risk context.
(2) The PRA assessments did not take into account a major risk factor in both
the Challenger and Columbia accidents: on-ground intellectual failures in engineering
analysis. In the case of the Challenger, the analytic process on the
day before the accident was seriously deficient, in the sense that--in hindsight to be
sure--the Challenger would not have been launched on that very cold day (which
compromised the O-rings and caused the accident) if smarter engineering analysis and
better decision-making had taken place. In the case of the Columbia, better analysis and
decision-making during the flight might have yielded rescue efforts to try to
save the crew, which was endangered by damage to the Columbia suffered at launch. I suggested to the PRA group that on-ground analytic problems
contributed to something like 1.3 of the 2.0 accidents in the 113 flights. But there was no
risk assessment of such in the PRA; that is, about 65% of the directly observed empirical risk in the 113
flights was not accounted for by the PRA model. The shuttle itself was considerably less risky
than what was happening on the ground in decision-making about the shuttle.
At the meeting, I also handed out Richard Feynman's famous discussion of shuttle risks, which Feynman prepared as a part of the Challenger investigation in 1987.
The analysis for the upcoming launch of the Discovery in July 2006, as the link above indicates, was an intense evaluation of risks and trade-offs.
On the basis of reading some of the public documentation (and no direct knowledge) for the upcoming flight in
the last few weeks, I think that NASA has made a reasonable and well-informed decision for the upcoming
flight. It was also a contested decision. I would vote for the launch. The on-ground factors that contributed to
1.3 shuttle losses appear to be mitigated by the thorough analysis for this flight. The current risk number is a cloudy 1 in 100, which is
risky but has been acceptable in the past. The cloudy contributions to risk are the recent changes in the foam, which turns Discovery into something of an experiment.
In the Discovery discussions, a telling distinction was made between "programmatic risks" and "crew risks." The programmatic risk is very high right now no matter what happens. Having flown once in 3+ years, the shuttle program might well collapse if unable to fly soon (within a year or so), or if there is another accident even if the crew escaped unharmed. This rescue scenario is itself troublesome, since the rescue launch must quickly take into account what caused the need for the Discovery crew-rescue in the first place.
-- Edward Tufte
POWERPOINT FOR DISCOVERY FLIGHT READINESS REVIEW: THE FOAM SLIDES, OR "MAKE THAT CALL NOW, THAT'S 1-800-. . . . "
Here's the PP deck for "STS-121, Flight Readiness Review, External Tank Project (ET-119):"
This pdf file should be up in a separate window to read in parallel with the comments below:
These slides summarize the results of the enormous amount of resources (probably >$1 billion, some estimates are much higher) devoted to the external tank foam problem.
The slides do not display a sense of engineering intelligence or discipline. In the main report, there is a persistent habit of
dequantification and a general absence of units of measurement. The back-up slides are more quantified and at a higher intellectual level. Several of the slides look like they were produced by a designer lacking in scientific training.
The key overview slide (page 3) is a very good idea but a presentation mess. The good idea is to have an intense and fairly detailed summary early in the presentation. But PP's lightweight resolution and lousy design tools compromise the summary slide. Students of PP design might, however, appreciate the 5 sets of orange drop-shadows, 4 wavy-purple color fields, 3 unintentionally 3D blue time-lines, 2 overactive grids, and floating-off-in-space bullets in the highlight box (with an arbitrary change from dots to dashes midstream in the box). All this stuff on one over-produced but importantslide.
In real science, every photograph has a scale of measurement built right in to the photograph. This low-resolution display method makes it impossible to do so. (Even the shuttle close-out photos, just about the most documentary type of photographs one can imagine, have no scales of measurement and no rulers in the pictures.)
The bullet lists tend somewhat to be base-touching grunts, which show effects without causes, actions without actors, verbs without subjects, and nouns without predicates. The branding with 3 logos on every slide (the title slide has 4 logos) is unprofessional, pitchy,
turfy. Are we doing engineering analysis or marketing here? Some 20%
of the space of every slide (already a a very low resolution display method) is devoted to branding and to the boxed-in awkward and repetitive slide titles. It is as if each and every slide has to remind the viewer what the presentation is about. So the top 20% of every slide is something to skip, perhaps putting some viewers in the mode of skipping and sliding through the rest of the slide. It is as if the top of every slide announces "nothing important here, you've seen it all many times before."
In several slides, the visually most active materials are the cross-hatched exploding 3D arrows linking the external tank to the magnified areas. Why are the arrows pointing anyway? It's just a simple linking line. The idea here of close contextualized
imaging of the problem areas is a very good one, but the badly-drawn giant blue arrows are silly, and result in making the dequantified images of the foam problem areas too small.
The typography is poor, with odd hierarchies (underlined bold italic in parentheses at one point). Is "O2" the proper way for NASA contractors and NASA to write the oxygen molecule (even wikipedia
uses a subscript)? Does the slide designer know how to write a subscript in PP?
The overlapping statistical graphics on page BU-2 are presented as
decoration, not evidence.
The report is 33 slides long; yet about 10 slide-equivalents are essentially content-free (compulsive repetitive branding, twiddly hierarchical organization, empty space, assorted title pages, and so on). This PP fluffed-up material here and quite a bit more could easily be placed in a technical report on 4 pages of an 11" by 17" piece of paper (folded in half), an exercise left to the student.
The tone and style of the presentation seem alienated from professional engineering. It looks like the slides were prepared by a PP designer, assisted now and then by an engineer. Or maybe it is just the PP pitch style diluting the content. At an FRR?!
I hope the actual engineering for the shuttle is a lot better than the evidence for the engineering shown in this presentation.
How much does a problematic presentational style signal poor engineering? Is it just PP or a PP designer weakening the quality of evidence? Or are there deeper intellectual failures? The
dequantification, the failure to follow professional engineering conventions, the infomercial tone are worrisome. There is no sign of engineering discipline here, except in the back-up slides. Thus the effect of the presentation is to suggest that there just might be some problems with foam engineering and analytical quality. A danger of problematic presentational styles, such as NASA PP, is not only that they enable sloppiness but also that they can place the truth in disrepute.
It is also a shame that all that expensive engineering work winds up being represented in this manner at a Flight Readiness Review.
-- Edward Tufte
Response to PowerPoint Does Rocket Science (and the upcoming Discovery flight)
Did the news conference present the PP slides, or did they a different medium to convey the details?
-- Allan T. Grohe Jr. (email)
They usually give a brief talk and then answer questions in a straightforward and intelligent manner, accompanied by occasional physical props, such as the broken-off piece of foam or model of the external tank. They did not use PP in the 8 to 10 press conferences I've viewed. You can see the press conferences and the launch by going to nasa tv at
Apparently the PP Flight Readiness Review for the foam (reviewed above) was something of a leak; the other FRRs at the meeting are not going to be made available. Keith Cowing, who runs NASA Watch, sent me an email saying that I might be interested in the foam FRR that he had posted at his website. You can see more on this at
I think the press conferences are excellent, assisted by a well-informed space press. After the flight, the head of NASA Michael Griffin was asked at the press conference if he felt "vindicated" by his decision to launch. He said not at all, if anything, it was vindication for the scientific method--that is, looking at the evidence and the numbers at hand. What a wonderful
thing for the Director of NASA to say. This contrasts to the PP cognitive style, which often seems to encourage presenters to pitch rather than present evidence.
-- Edward Tufte
Below, a link to a good account of the Discovery inspections by John Schwartz of the New York Times on the problem of
distinguishing useful evidence from additional evidence, a problem that also occurs with
newly developed exquisitely sensitive measurements (for example, PSA tests and the
monitoring of contaminants of drinking water).
John Schwartz, New Scrutiny for Every Speck on the Shuttle, New York Times, 11 July 2006.
These issues can lead to quite subtle consequences, as my Yale colleague Alvan R. Feinstein suggested in many studies, including this one in the Archives of Internal Medicine: ". . . many breast cancers found by mammography screening have excellent prognosis not just because of early detection, but also because many of the cancers are relatively benign, requiring minimal therapy."
Sandra Y. Moody-Ayers, MD; Carolyn K. Wells, MPH; Alvan R. Feinstein, MD, MS, "Benign" Tumors and "Early Detection" in Mammography-Screened Patients of a Natural Cohort With Breast Cancer, Arch Intern Med. 2000;160:1109-1115.
(Thanks to Niels Olson for the NYT permalink above.)
-- Edward Tufte
From Nature, 13 July 2006, still more on PP, this from Martin Kemp, an Oxford art historian:
-- Edward Tufte
In your workshops, you describe how to replace PowerPoint presentations with 11 x 17
sized reports, and provide many good arguments for why "engineering by PowerPoint"
doesn't work very well.
This is a website that might interest you and your audience. It describes a process for
creating A3 sized technical reports, and using them to make better decisions.
Dr. Durward Sobek of Montana State University spent six months in Japan as a grad
student, interviewing and observing Toyota engineers to uncover the reasons why Toyota
was able to develop cars much more quickly than other auto makers and also maintain
high standards for reliability.
Toyota uses these A3 reports extensively in their engineering processes. They believe that
the discipline required to accurately capture a problem on a single sheet forces the author
to express the issue with both clarity and conciseness. They emphasize using visual
models to express ideas rather than a lot of text, and value the ability to have all of the
pertinent information within a single field of vision. The engineers are also required to
bring their supporting documentation, so that the team can dive into the details when
Since then, Dr. Sobek has taught many engineers how to use A3 reports to make better
technical decisions. I can tell you from my personal experience with this technique that it
is amazingly powerful. By using one of these reports, we solved a technical problem
within a single meeting that we had literally wrestled with for years through engineering
by PowerPoint. By forcing us to make
our knowledge about the problem visible in a systematic way, the tool helped us come to a
deeper understanding that led to the solution.
-- Katherine Radeka (email)
The "A3 Process" described above begins with a good idea and then dilutes it into a
Business Methodology Fad. BMFs are characterized by a germ of a good idea, but also
by over-reaching, over-simplifying, excessive focus on a single idea, pitchy and
over-simplified examples, and pretentious names ("The Toyota Method," "The Long Tail,",
"The Genghis Khan Guide to Mastering the Universe," "The Takahari Guide to Infinite
Profits," and so on).
In the Beautiful Evidence chapter on corrupt techniques in evidence presentations,
the section on over-reaching concludes with this: "When a precise, narrowly focused
technical idea becomes metaphor and sprawls globally, its credibility must be earned
afresh locally by means of specific evidence demonstrating the relevance and explanatory
power of the idea in its new application." (p. 151)
The A3 method, which at its heart is a good idea, requires some down-in-the-trenches
detailed and complex examples. And it should avoid bullet lists in describing the method.
-- Edward Tufte
A well designed single page technical report from Science:
-- Edward Tufte
PPT and Military Intelligence
Central Command Charts Sharp Movement of the Civil Conflict in Iraq Toward Chaos
By MICHAEL R. GORDON, The New York Times
Published: November 1, 2006
See the NYT news story here
Here are some preliminary comments on the slide "Iraq: I&W of Civil Conflict."
It appears that "I&W" means "Indications and Warnings." Replacing the acronym in the slide title does pep up meaning to outsiders: "Iraq: Indications and Warnings of Civil Conflict," but maybe it wouldn't fit on the slide.
Only this single slide was leaked (by the military? by DoD?), and so maybe some of the analytical
are better handled on accompanying slides. Maybe.
Doing competent political analysis, epidemiology, nation-building, and war planning (all of
which they're trying to do) in a chaotic situation is impossible, and not much good social
science and epidemiology can be expected in chaos and from a military entangled in Iraq. In real-time chaotic situations, the data-collection is going to be sloppy because people have more important things to do. (Recall, for example, the gross errors in counts of 9/11 deaths, as the count went from 6,000 to 2,800 in a few weeks.) And what's taking place is in profoundly different cultures and in different languages from those of the non-local military in Iraq. But sloppy data does not justify analytical sloppiness in reporting. In fact, sloppy data requires greater analytical precision of thought.
The slide reports performance data--a list of phrases, with each phrase accompanied by a measure of performance. This is what the tables in the sports section, mutual fund page, and weather page of newspapers do very well. Those designs are much better for reporting performance data than the slide format here. In sports and stock market tables, each phrase is accompanied by multiple measures of performance, often over varying time-periods. All that won't fit on the slide; this suggests that we should use better reporting method than PP, instead of abbreviating the evidence to fit the slide. As the millions of readers of sports tables each day demonstrate, people can easily manage large tables of information. Thus those being briefed in the military should ask: Why are our presentations operating at 2% of the data richness of routine tables found in the sports section? Let the viewers read and explore through a range of material; different eyes will search for different things in the evidence. The metaphor should be the cognitive style of the sports section (or weather or financial newspaper pages) not the cognitive style of PowerPoint.
There is no cloud of uncertainty or error history associated with the editorializing color. At times, such color codings suggest an excess of certainty.
The Iraq slide above provides some relevant but thin and overly short-run time-comparisons: 2 arrows on the left showing
since last week," and the "Index of Civil Conflict (Assessed)", which sort of compares "Pre-
Samarra" with "Last week" and "Current". And there's a potent time-comparison in words: ". . . violence at all-time high,
To get more time comparisons on the 14 "Reads" and
"Additional Indicators," 14 sparkline time-line histories for the last year (week by week, if
available) would be useful as a overall but detailed summary. This would reduce the
snapshot tone of the 14 reads and indicators. In our thread Sparklines: theory and practice, there are (at the top of the thread) data tables with sparklines that report daily and longterm financial data; one such table shows 14,000 numbers, many of them accurate to only 2 digits (not much for financial data) under the philosophy of "Try to be approximately right rather than exactly wrong." The short-run weekly jitters and non-reports need to be smoothed out to see (and compare with)the long-run trends. Weekly data cooperate with the notorious recency bias, whereby way too much weight is given to the most recent piece of data, just because it is recent. These weekly reports should be in the context of longer run information to reduce the chances that analysis will be dancing around only with today's news.
The list style, surely one damn thing after another here, is merely descriptive and thus
preliminary to policy analysis. That analysis might have been done on the other
slides or maybe this report is merely meant as a scorecard. If it is a scorecard, it is grossly impoverished compared to sports, weather, and financial tables.
The current fashion (it, too, shall pass) in government is the stoplight style (green, yellow, red), which tends to dequantify data. With categories of this sort, there's always a concern with how the breaks among categories are chosen and with the meanings of the categories. It will often be better to provide some evidence or numbers, and then a separate editorial-judgmental color about the number.
The slide contains odd uses of the color-words: for example, a green dot indicating "routine" next to the exciting phrase "unorganized spontaneous mass civil conflict". Shouldn't "routine unorganized spontaneous mass civil conflict" be red-critical? After Hiroshima, would Nagasaki get the routine green dot for nothing different than what happened three days earlier? It looks like weekly wiggles get too much attention, and longrun levels of seriousness too little attention on this slide, as chaos becomes routine week by week and bit by bit. Monthly rather then the sketchy weekly reports might be better for policy analysis. Or at least provide a monthly aggregations over a period of many months (even the entire war) in a scorecard along with the weekly incidents.
The leaking of the slide makes a point about the differences between the government's secret analysis
and the public reports by the
Administration, a common theme of the insider books on Iraq policy-making (most recently Colin Powell's book). At some
time, "reality must take precedence over public relations," as Richard Feynman remarked
about the shuttle Challenger accident.
A good many comments by our contributors are on-point but are not taken into account
Note the measurements, definitions, and comparisons to standards in
the customer scorecard in the "Report of the South Central Connecticut Regional
Water Authority" (above).
Finally, the over-riding metaphor of all this--"the war on terror"--is a big conceptual problem. Once it's a war, then it almost necessarily invokes large-scale military action and searching for a locale (some place, any place, Iraq) for large-scale military action. But terrorists are more like the Mafia or gangs than they are like armies. Perhaps a better metaphor is that terrorism is largely a law-enforcement problem (requiring focused and clandestine local action, informants, endless detective work, detailed knowledge of the local languages and cultures).
Among the grand truths about human behavior, surely the principles of "the unintended consequences of purposive social action" and "it's more complicated than that" are among the top five. Sometimes unintended consequences are largely virtuous or benign (as in market allocation of goods and services if externalities are mitigated) and sometimes the unintended consequences are appalling. That's because it's more complicated than that.
-- Edward Tufte
PowerPoint does blackholes?
John Wheeler, eminent physicist, died in 2008 at the age of 96. To commemorate his contributions to physics, astronomy, cosmology, quantum mechanics and more, the April 2009 issue of Physics Today contains several wonderful articles about Wheeler.
The article, "John Wheeler, relativity, and quantum information", by his former-students Misner, Thorne and Zurek (all prominent physicists themselves, now) contains this photograph taken by Kip Thorne. The caption reads, "John Wheeler lecturing at a conference in Cambridge, UK, in 1971. Wheeler's style was to cover the blackboard with inspirational colored-chalk diagrams and phrases before the lecture, then work his way through them, one by one."
The blackboard looks like a precursor to today's PowerPoint presentation, but not poor PowerPoint full of bullet grunts. Instead, a sequence of diagrams and key phrases to guide the audience, and the speaker, through the material. Judging from the photo, this must have been quite the lecture...
-- Peter (email)
The picture of Professor Wheeler prompted me to finally
stitch together pictures of two blackboards I saw at Baylor
School of Medicine in 2005. To this day I have no idea who
drew them, but it is one of the Baylor Biochemistry professors.
UPDATE 14 May 2009:
Juan Ruiz-Hau was kind enough to prepare these excellently corrected versions from the originals.
-- Niels Olson (email)
One fax = 2,500 PowerPoint slides
Here's one of my exhibits in The Drawing Room's FAX show:
-- Edward Tufte
Steve Balmer, ex-CEO of Microsoft: "No more decks"
There was an interview with Steve Balmer, Microsoft CEO, in the Sunday New York Times. When asked what it's like to be in a meeting run by Steve Balmer he says that he decided that what he calls the "long and winding road" meeting style of a few years ago at Microsoft isn't productive. He says that for most meetings, he now gets the materials in advance and he reads them. For the meeting he comes in and says "I've got the following four questions. Please don't present the deck."
-- Sam Perry (email)
PowerPoint and scientific fraud
July 14, 2009, Wall Street Journal Blog
"The oversight [of scientific data] is now vastly diminished. Even within the laboratory environment, many students and
post-docs and scientists are not showing raw data anymore. They're showing PowerPoint presentations. That gives the
individual, if they're so inclined, the ability to manipulate data right up-front. Unless a mentor is vigilant, there's a real
--John Dahlberg, director, Division of Investigative Oversight in the Office of Research Integrity, Dept. Health and
-- Prem Thomas (email)
I took Presenting Data and Information a few years ago in Boston while at another company, but ET's influence has stuck with me.
In my first year at my new employer, I can confidently credit the course with my win of an Individual Achievement Award in recognition of my technical analysis briefings of a systems performance problem which I resolved, as well as my recent win of my company's 2009 Information Systems Excellence Award in the Innovation category.
My "Quick Start Guide" for the roll-out of a crucial new capability in the network was designed and built by the book (the Tufte Book) - on an 11x17 folded sheet, multiple columns with embedded illustrations, plus full bleed thanks to the commendable graphic design and printing services staff - and it was a resounding, dazzling success. It turned heads all the way up the chain of command, and was received with delight by the users.
I was enthusiastically told by an IS department leader that nothing of the kind had ever been seen before in any other rollout in IS, and it represented just the kind of thing that our department should be doing more of. I and my colleagues stood ready on rollout day for questions and problems, but heard only the faint hum of heightened productivity.
While I fortunately didn't have to cope with the mechanics of the layout in Quark Express, I was also fortunate to be working with a designer with whom I could be specific and detailed in my vision of the final document, as I referred to my PD&I notes and materials.
My users found it helpful that I prepared two PDF versions of the document for the eRoom document archive - one as the original two-page 11x17 format, and the other a four-page letter-size version to permit people without access to a large-format printer to produce a copy of the document, single- or double-sided.
-- Michael Pelletier (email)
A quantum leap perhaps? "PowerPoint makes us stupid," Gen. James N. Mattis of the Marine Corps, the Joint Forces commander, said this month at a military conference in North Carolina."
From a NY Times article on the danger of PowerPoint in the U.S. military.
Accessed online on 4/26/2010 and to be in print on 4/27/2010.
Regards to all as I await Professor Tufte's return to Dallas for another lecture,
-- Ron Hekier (email)
"One of the first things [Steve] Jobs did during the product review process was ban PowerPoints. `I hate the way people
use slide presentations instead of thinking,' Jobs later recalled. `People would confront a problem by creating a
presentation. I wanted them to engage, to has things out at the table, rather than show a bunch of slides. People who
know what they're talking about don't need PowerPoint.'"
-- Walter Isaacson, Steve Jobs (New York: Simon & Schuster, 2011), 337.
-- Curt (email)