Congress Presses Probe Into NSF Prediction Of Scientist Shortage

By Jeffrey Mervis

Date: October 28, 1991

A House panel questions methodology, motivation behind agency's warning that a huge shortfall threatens the work force

WASHINGTON--A congressional committee is investigating whether political considerations influenced the National Science Foundation's prediction that the United States faces a cumulative shortage of some 675,000 college-educated scientists and engineers over the next two decades.

The NSF prediction, based on studies conducted since 1985 by agency policy analyst Peter House and others, bolstered fears prevalent in the public and private sectors that a dearth of scientifically and technically adept workers will weaken the nation's future ability to compete in the global marketplace. The NSF numbers have been used to buttress arguments that the federal government needs to spend more on science education and research.

Last fall, then-National Science Foundation assistant director Karl Willenbrock created an advisory panel to examine how the agency carried out its survey-collection and policy-analysis functions. At its first meeting, in January, the panel heard from Janet Norwood, outgoing director of the Bureau of Labor Statistics, on the importance of keeping data collection separate from policy analysis. Then it heard a presentation from Peter House, head of the Division of Policy Research Analysis, on the supply of scientists. A transcript of that meeting, plus accounts from participants, reveal that the panel was critical of House's methods. In an exchange between House and Edward Tufte, a Yale University mathematician widely known for his work on the graphic depiction of statistics, House was explaining two curves, with differing slopes, that showed changes over time in the number of bachelor's degrees awarded by United States colleges, when Tufte broke in. "You're trying to compare changes," Tufte said, "but one base is 200,000 and the other is 800,000. You can't do that. If you were to put those on a common scale, you wouldn't have this [dramatic change] in slope." "Well, it would flatten out one of them," House replied. "Exactly my point," said Tufte. "I'm happy you got the sign right, but statistics is more than that. You've got to get the quantity right, too." A bit later, during a discussion of salary data that was used by House in assessing demand for Ph.D. scientists, an unidentified member of the panel made this point: "I know that may be all you can get, but the question is: Is all you can get good enough for the National Science Foundation? I know something about the data in this area, and I suspect those are just plain not good enough to deal with." --J.M.

In recent months, however, House's analysis has been criticized by statisticians and labor economists as simplistic and based on faulty assumptions. This past July, those criticisms came to the attention of Rep. Howard Wolpe (D-Mich.), chairman of the oversight and investigations subcommittee of the House Science, Space, and Technology Committee, which launched an investigation of the NSF prediction.

While the subcommittee is not attempting with its probe to judge the scientific methodology behind the analysis, it is interested in two key points: Did the analysis undergo sufficient review before it was disseminated? And was it politically motivated?

At stake, says Wolpe, is the continued credibility of the foundation as a reliable source of scientific information. A loss of that credibility, he believes, could seriously weaken support for the agency in congressional battles over funding.

The 675,000-scientist shortfall that House and his colleagues arrived at, says Wolpe, "may even be an accurate number. But we don't have any reason at this juncture to have any confidence in that number.

"What's more troubling is that I want the NSF to remain a uniquely credible and respected institution. The whole field of academic science does not need, at this juncture of growing public cynicism about all of our national institutions, the kind of skepticism that is the product of sloppy work."

NSF officials are expressing displeasure with the attention the agency is getting over the matter. Several have complained privately that they don't understand what Wolpe is trying to accomplish with his investigation. At the same time, Raymond Bye, NSF's chief lobbyist as well as director of its office of public affairs, denies that the foundation has ever tried to inflate its budget requests with bogus numbers.

"This wasn't a scheme hatched in a smoke-filled room, I can assure you," says Bye about the House analysis. "Whenever we used it, we tried to put all the appropriate parameters on it. And we never tied it to program considerations, like asking for X million dollars more because of a looming shortage of scientists. We were just trying to highlight an important problem that the country is facing, in line with our responsibility to monitor the future manpower requirements in science."

Not everyone accepts that explanation, however. Presidential science adviser Allan Bromley, for one, doubts the magnitude of the supposed shortage. Asked if it played a role in helping to build support for a larger NSF budget, Bromley responds, "Yes, definitely. But that should be no surprise. A smart administrator uses whatever is at hand to make his case."

Bromley worries about the impact of the forecast on the career decisions of college-age students and, in turn, whether those decisions will create imbalances in the number of scientists in various disciplines. "The smart ones see these numbers and go where there are projected shortages," he says. "And that could lead to the type of oscillations in the system that I think are dangerous." Alan Fechter, head of the Office of Scientific and Engineering Personnel at the National Academy of Sciences, says that NSF officials did not work hard enough to qualify the House numbers as tentative. "Policy analysts have to worry constantly about their credibility," says Fechter. "And the best way to do that is to be proactive. We can't take an attitude of caveat emptor when it comes to our data. We have a duty as professionals to tell people what the numbers mean, and what they don't mean."

As head of the Division of Policy Research and Analysis (PRA), House began his analysis in 1985 during a time when former NSF director Erich Bloch was seeking to, first, gain a presidential commitment to double the NSF budget, and then steer such an increase through Congress. At Bloch's request, House looked into the problem of the future supply of scientists. And he was disturbed by what he found.

House based his analysis on the demographic fact that the size of the U.S. college-age population had peaked in the early 1980s and was expected to drop sharply through most of the 1990s. He assumed that the percentage of students graduating with science and engineering-related degrees--historically between 4 percent and 5 percent--would remain steady into the next century. And he made the number of science graduates in the period 1984-86, a record-high level, a surrogate for future demand. Based on those assumptions, he calculated that the U.S. would produce 675,000 fewer B.S. graduates trained as scientists and engineers than it needed by 2006.

The number quickly became accepted wisdom on the lecture circuit, at scientific conferences, and in political debates. It appears in a law passed last fall-- The Excellence in Mathematics, Science, and Engineering Education Act of 1990--as a key justification for increased federal spending, although in the text of the bill, the shortage is predicted to appear by the year 2000. It's also gone out to the public. For example, a 13-part series airing this fall on National Public Radio on the dearth of women and minorities in science cites the number, without mentioning any time frame, in an opening segment entitled "The Shortage of Scientists."

Why did the 675,000 figure become so popular? Policymakers wrestling with a complex issue find it hard to resist a single, easy-to-understand number, says Charlotte Kuh, director of the Graduate Record Examination of the Educational Testing Service (ETS). "It's simple," says Kuh, who is a member of the advisory board that oversees NSF's statistical programs, about the number. "And I know how tempting that can be for people to grab onto."

Wolpe's subcommittee staff is sifting through cases of documents it received last month in response to its August 13 request to NSF director Walter Massey for all material relating to the foundation's analysis of a projected shortage. Wolpe says he doesn't know what the investigation will turn up, but that the data are sufficiently important to warrant a close look by his panel. "This is one number that has had unique political distribution and impact," says Wolpe. "There has been a generalized concern [in Congress] about the state of math and science education and the potential concern about American competitiveness that underlies this.

And the discussion of this number occurred in a research vacuum--there had been no other efforts that were widely disseminated that were trying to establish a number to this degree of concreteness."

But that number, first put forth in a 1987 internal NSF document, is now under attack. Statisticians have questioned the assumptions that underpin the analysis, as well as the choice of factors used (The Scientist, April 29, 1991, page 1; and May 13, 1991, page 1). Others are worried that House's conclusion goes beyond the existing data. And many labor economists don't believe that the supply of scientists can be determined independent of the market demand for their services; they question the value of any prediction that tries to separate the two.

Although PRA is located within the Scientific, Technological, and International Affairs (STIA) directorate, House worked closely with Bloch's office in generating his estimate of a potential shortage. By last fall, however, as concern about the number began to percolate within the scientific community, former STIA director Karl Willenbrock formed an advisory committee to look into the matter.

"My belief is that all those projections [of supply and demand] have so many variables that they are very dangerous," says Willenbrock, who left STIA last month, after two years, to become a visiting professor at Carnegie-Mellon University. "Rather than issuing one number, I prefer the technique used by BLS [the Bureau of Labor Statistics], in which they put out a range of numbers and make perfectly clear what the basis for each number is."

The advisory committee, chaired by Judith Liebman, vice chancellor and dean of the graduate college of the University of Illinois, held its first meeting in January 1991, at which House was questioned by several members on both his approach to the work and its underlying assumptions (see accompanying story).

House says that he has gone to great lengths not to overreach himself in interpreting the available data. He told the panel that his staff has resisted repeated attempts to break down the numbers by scientific field, as well as to come up with an independent assessment of demand. "We've gotten into a lot of trouble for not doing both those things," House told the panel, "and the reason is that there was a strong feeling that the numbers were so weak that we shouldn't do it."

Even so, many in the statistical community think that House went too far. Liebman says that his presentation of much of the data "was not up to the standards in the field." The National Academy's Fechter says that "the PRA analysis went well beyond what many responsible researchers would accept" in estimating demand. Alfred Blumstein, dean of the school of urban affairs at Carnegie-Mellon and a member of the STIA advisory board, notes that "demography is the easy part. The real issue is demand, and that can change quickly, depending on job opportunities."

House has described the shortage in different ways. The period over which the shortage is supposed to accrue, starting in 1986, has varied by as much as a decade--from 2000 to 2010--in several House papers. And a 1990 book published by NSF that includes a discussion of the issue begins with a disclaimer that the book's contents "do not represent the official policy of the Foundation, nor necessarily the judgment of any particular staff analyst."

Some observers believe that the foundation erred in disseminating the information so widely while, at the same time, insisting that the projections were not official NSF policy. To Wolpe, it's a case of trying to have one's cake and eat it, too.

"I think that the caveats that they put on were inadequate," says Wolpe. "They never noted that these were only discussion figures. They never identified the absence of any systematic methodology. It's true that this number was included in documents marked `working draft,' [but] they proceeded to take these working drafts and distribute them very widely. And they certainly had to know that this number became the principal finding of legislation."

ETS's Kuh says the dispute highlights an important difference between academic publications and government documents. "If this paper had come out of the [John F.] Kennedy School [at Harvard University], it would have been attributed to the author, not to the institution," she explains. "But within the federal government, nothing comes out if it doesn't have the imprimatur of the agency itself. This is an exception."

Bye says that the various drafts of the PRA paper "were intended to be a think piece that provides a wealth of information to the community." He says that the foundation "does not have an official position" on the precise size of a potential shortage of scientists, and he adds, "I don't think that we ever cited a particular number" in testimony before Congress.

At the same time, he says, "we never issued a press release disavowing it" when somebody quoted the number in a public forum. And he acknowledges that the number "probably was useful" in ongoing congressional debate over increased NSF support. House himself has stated that the number has been a politically influential commodity. In a forthcoming book entitled The Practice of Public Policy Analysis, House and coauthor Roger Shull, deputy director of PRA, take credit for the recent growth in the NSF budget.

"The shortfall argument helped to justify President Reagan's proposal to double the NSF budget over five years," they write, referring to a 1987 pledge that President Bush has also endorsed. "The education and human resources component of NSF's budget was increased to support new math and science curricula, and the basic research budget was increased to accelerate the production of natural science and engineering Ph.D.'s."

The two authors also engage in a certain amount of self-congratulation. "The `675,000' number became famous," they write, "and was used in countless speeches and magazine articles. There were a small number of critics who believed that participation would rapidly increase automatically, or that other market mechanisms would obviate the shortfall problem. But most analysts saw the projection as inherently reasonable." For Wolpe and others, that judgment won't come until the congressional inquiry is completed.

The Scientist 6[14]:0, Oct. 28, 1991

Original article [external link]

Back to Table of Contents