Information Design in Surveys

In the February 2002 issue of the Harvard Business Review there is an excellent aricle on the design of workplace surveys:

“Getting the Truth into Workplace Surveys,” by Palmer Morrel-Samuels.

The article is not available on-line, but you can look at the absract and order a copy here.

I heartily recommend it to anyone involved in the creation of surveys. Here is a brief excerpt to show you what it is like:

Guideline 4: Keep sections of the survey unlabeled and uninterrupted by page breaks.

Boxes, topic labels, and other innocuous-looking details on surveys can skew responses subtly and even substantially. The reason is relatively straightforward: As extensive research shows, respondents tend to respond similarly to questions they think relate to each other. Several years ago, we were asked to revise an employee quesionnaire for a large parcel-delivery service based in Europe. The survey contained approximately 120 questions divided into 25 sections, with each section having its own label (“benefits,” “communication,” and so on) and set off in its own box. When we looked at the results, we spotted some unlikely correlations between average scores for certain sections and corresponding performance measures. For example, teamwork seemed to be negatively correlated with on-time delivery.

A statistical test revealed the source of the problem. Quesions in some sections spanned two pages and therefore appeared in two separate boxes. Consequently, respondents treated the material in each box as if it addressed a separate topic. We solved the problem by simply removing the boxes, labels, and page breaks that interrupted some secions. The changes in formatting encouraged respondents to consider each question on its own merits; although the changes were subtle, they had a profound impact on the survey results.