On Thursday, July 17, four science experts served as witnesses at the U.S. Senate Committee on Commerce, Science, and Transportation hearing, “The Federal Research Portfolio: Capitalizing on Investments in R&D.” The hearing considered the federal government’s role in research and development (R&D), and the nation’s STEM education and outreach initiatives.

Attendees in the Capitol hearing room were Mariette DiChristina, editor in chief and senior vice president of Scientific American; Vinton G. Cerf, computer scientist, Google’s Internet Evangelist and one of the fathers of the Internet; Neal F. Lane, former director of the White House Office of Science and Technology Policy; and Stephen E. Fienberg, professor of statistics and social science at Carnegie Mellon University.

Recognizing the need for long-term investments in science and technology, Congress passed the America COMPETES Acts of 2007 and 2010 to significantly increase federal R&D budgets, to promote STEM (science, technology, engineering and mathematics) education and to support the innovation necessary for economic growth.

Below is the full text of the written testimony by Stephen E. Fienberg.

Good afternoon, Mr. Chairman and members of the Committee. My name is Stephen Fienberg. I am Maurice Falk University Professor of Statistics and Social Science at Carnegie Mellon University with appointments in the Department of Statistics, the Heinz College, and the Department of Machine Learning, and I served as a member of the Committee on Assessing the Value of Research in Advancing National Goals of the National Research Council. The National Research Council is the operating arm of the National Academy of Sciences, National Academy of Engineering, and the Institute of Medicine of the National Academies, chartered by Congress in 1863 to advise the government on matters of science and technology. The Committee was established with funding from the National Science Foundation pursuant to Section 521 of the America COMPETES Act of 2010. Today I will share with your Committee some of the highlights of our report, Furthering America’s Research Enterprise, and I append to my remarks a list of members of the study committee and the Table of Contents of the report.

The context
The benefits of the federal investment in scientific research are manifest and have enabled the United States to achieve unprecedented prosperity, security, and quality of life. But the nation now faces increased global competition for new technologies and other innovations, in the face of growing economic exigencies. Congress wants to enhance the benefits of scientific research for the U.S. economy and other purposes and to keep the nation at the forefront of global competition for new technologies and other innovations.

How can that be done effectively and efficiently? In particular, how can we increase the returns on current federal investments in scientific research? In seeking answers to those questions, Congress asked the Academies to study measures of the impacts of research on society, especially those that could serve to increase the translation of research into commercial products and services. Also of interest was the use of such measures for purposes of accountability. The purview of the study was all federally supported research.

The Committee’s Findings
I. Current measures are inadequate.

While some measures of research outputs and benefits are useful for specific purposes, the committee found that current measures are inadequate to guide national decisions about what research investments will expand the benefits of science.

The problem is that metrics used to assess any one aspect of the research system in isolation, without a strong understanding of the larger picture, may prove misleading. The benefits of research investments tend to arrive unpredictably, vary widely in eventual value, and require substantial additional investment (as well as investment in other fields of science) to realize their economic payoff through innovation. With few exceptions, approaches to measure the impacts and quality of research programs cannot depict the diffuse, interconnected and highly non-linear pathways that lead from research to technologies and other innovations. The widespread adoption of the innovation is a process that itself requires investment and substantial know-how.

Existing metrics give some indication of how well the system is performing, but the ultimate impacts, the emergent phenomena that truly matter to society such as an abundant supply of natural gas enabled by fracking technology, communications and commerce enabled by Google and the Internet, and medical advances enabled by genomics depend on a number of critical components, and the relationships among them, in the complex systems of research and innovation. These components often are intangible, including opportunities and relationships that are not captured by most data collection programs and cannot be measured by currently available methods.

II. Reaping further benefits

The committee concluded that the American research enterprise is indeed capable of producing increased benefits for U.S. society, as well as for the global community. To reap those benefits, however, we first need to understand what has made the American research enterprise so successful: what drives it and why has it been so productive.

Our research enterprise has been so successful because it has evolved as a complex, dynamic system with many of the characteristics of American free enterprise. It is decentralized. It is pluralistic, driven by a diverse array of researchers, companies, institutions, and funding agencies. It is competitive, requiring researchers and organizations to compete for funding, for talent, for positions, for publications, and for other rewards. It is meritocratic, bestowing more significant rewards on those with highly competitive ideas and abilities through a built-in quality control system of peer review. And finally, it is entrepreneurial: it allows for risk taking, for facing the prospect of failure head on to reap potentially great rewards.

Just as business thrives in free enterprise for its products and services, so too does our extraordinarily productive research enterprise for its ideas and discoveries. As our assessment progressed it became clear to us that increasing the benefits from the federal investment in research depends far less on federal promotion of the commercialization of research discoveries or on trying to predict the scientific fields that are most likely to lead to commercial products and services, than on federal policies that promote the conditions for the research enterprise to thrive. We identified three crucial pillars of the research enterprise:

1. A talented and interconnected workforce. The importance of talent cannot be overstated, both as input and as output. Talent benefits not only from public investments in traditional education and research training in science and engineering but also from highly skilled immigrants; partnerships; supportive research environments; and worldwide networks through which researchers connect with others, develop professional relationships and share ideas and scientific resources.

2. Adequate and dependable resources. Stable and predictable federal funding encourages talented students to pursue scientific careers, keeps established researchers engaged over a career, and attracts and retains foreign talent. It also supports a diversity of institutions that both fund and conduct research, as well as essential scientific infrastructure–the tools necessary for conducting research. Flexibility and stability in funding, along with a culture that tolerates failure, may inspire researchers to pursue riskier and more innovative research with a greater chance of failure but also a greater likelihood of transformative impact. These resources are increasingly important to future U.S. competitiveness, given the rising investments in research by other countries, particularly China and other Asian nations.

3. World-class basic research in all major areas of science. Basic research, in which investigators pursue their ideas primarily for increased understanding and not necessarily toward a technological goal, often provides the foundation of discovery and knowledge for future economically significant innovations. Federal investments in basic research contribute to the growth of a trained research workforce, support the scientific infrastructure to conduct research, and enable U.S. researchers and would-be innovators to exploit the world-wide networks of researchers, who open access to a vast stock of knowledge and technological approaches. Absent a strong pool of scientists and engineers familiar with basic research at the cutting edge, scientific research and its products are unlikely to be developed and applied in ways that create value for society.

World-class basic research in all major areas of science is important because truly transformative scientific discoveries increasingly depend on research in a variety of fields. Moreover, in today's highly connected world, a discovery made somewhere is soon known everywhere. The competitive advantage may go not to the nation in which the discovery was made but to the nation that can use it more effectively to develop new technologies and other innovations by relying on a broad foundation of knowledge, talent, and capacity derived from basic research in a diversity of scientific fields. Finally, a world-class basic research enterprise attracts scholars 7 from around the world who further enhance excellence in research and create a selfreinforcing cycle.

The development of Google is a good example of why a diversity of basic research is important. Google owes its remarkable success in part to its algorithm for ranking Web pages. The 1997 patent application for the algorithm, which acknowledged support from the National Science Foundation (NSF), drew heavily on multiple discoveries spanning nearly 45 years of social and information sciences research— discoveries made possible by funding from four federal science agencies and protected by a handful of seemingly unrelated patents awarded to a university (Carnegie Mellon), corporations (Lucent, Libertech, AT&T, Matsushita), and industrial laboratories (AT&T Bell Labs). Critical to the development of the algorithm was decades-old research on methods to determine social status, and social network research from the 1970s. The development of the Google algorithm illustrates the importance of seemingly unrelated social science research; the convergence of research at universities, corporations, and industrial laboratories; and the unpredictable benefits of federally funded research. Moreover, the economic model for Google advertising utilizes a variant of the Vickrey auction, first described in a 1961 theoretical economics paper and later developed by many others with NSF support. Other internet-based companies have followed suit.

New as well as existing measures could be used to assess each of the three pillars. Such measures might include, for example, indicators of human and knowledge capital, indicators of the flow of knowledge in specific fields of science, indicators with which to track the flow of foreign research talent, portfolio analyses of federal research investments by field of science, international benchmarking of research performance, and 8 measures of research reproducibility. Another recent National Research Council report, Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy, identified many measures for assessing the performance of policies intended to strengthen the three pillars of the research system.

The levels, composition, and efficiency of federally funded research need to be adjusted to meet today’s circumstances and we need better metrics to inform policy decisions about research. But the United States lacks an institutionalized capability for systematically evaluating the nation’s research enterprise as a whole, assessing its performance, and developing policy options for federally funded research. An organization charged with such a responsibility would increase the demand for policy relevant data of high quality. Although NSF’s National Center for Science and Engineering Statistics produces valuable data (e.g., Science and Engineering Indicators) that could be used in policy analysis, NSF’s role differs from that of federal policy analysis agencies or statistics agencies such as the Bureau of Economic Analysis or the Economic Research Service that conduct policy analysis.

One U.S. data collection program—STAR METRICS (Science and Technology for America’s Reinvestment: Measuring the Effect of Research on Innovation, Competitiveness and Science)—is designed to collect a number of measures of the impacts of federally funded research. This data program is a joint effort of multiple science agencies (the White House Office of Science and Technology Policy, NIH, NSF, the Department of Energy, and the Environmental Protection Agency) and research institutions. While STAR METRICS aims to document the outcomes and public benefits 9 of national investments in science and engineering research for employment, knowledge generation, and health, our assessment is that it is not ready for prime time use.

STAR METRICS could potentially be of great value in assessing the value of research if efforts were made to (1) broaden coverage by enrolling additional institutions, (2) deepen coverage by expanding the data elements reported, (3) link the data to other national and international datasets, (4) establish the quality of the data, and, most importantly, (5) ensure broad, easy access for researchers. Such expanded data and access need to be coupled with modern analytical tools, such as complex network modeling and analysis. Our report provides a simple illustrative example, but with better data, such tools might reveal important interactions among components of the research enterprise using an expanded and restructured STAR METRICS program.

Enhancing America’s research enterprise requires a better understanding not just of the three pillars of talent, resources, and basic research, but also of the relationships and interactions among them. For example, resources for basic research also provide for talent through the training of a research workforce and, by engaging undergraduate students in research, as we do at my university, Carnegie Mellon.

Let me use my Department of Statistics as an illustration. My faculty colleagues and I have a diversity of research grants and contracts that employ and train our Ph.D. students. But this federal and international research support also creates a research environment that allows us to engage and train many undergraduates and master’s students, who go on to advance their research skills at other research universities in statistics and many quantitatively-related disciplines. And this pattern is replicated across 10 the university, fostered in part by the interdisciplinary activities of my colleagues. These students represent the future of our scientific workforce.

Other measures, which can help to make the research enterprise more efficient and which can provide information to guide the allocation of research funds arise in evaluations. We address in our report the evaluation of research funding programs, of peer review, and the effects of different funding programs, such as the NIH Pioneer Awards, on research performance. Unfortunately, most attempts at evaluation do not address the fundamental question: What would have happened but for the research funding program? At a higher level, evaluation efforts rarely address questions such as: what alternate allocation of resources between programs might promote a healthier research enterprise? If evaluations are conducted at all, they are often added after the fact. Evaluation needs to be built into research funding programs from the outset to help avoid the unmeasurable biases associated with ad hoc retrospective evaluation. Moreover, few evaluation studies or approaches adopt randomized controlled field experiments that control for biases and input differences. We need to address these evaluation challenges.

Measures of research activities, outputs, and technology transfer are important, but we need to improve both the measures and the underlying data. Greater benefit will come from measures to guide the pillars of the research enterprise—talent, resources, and basic research. If we cultivate talent, provide adequate and dependable resources, and invest in a diversity of basic research, fresh discoveries will continue to power our economy and to enrich our lives in unpredictable and unimaginable ways.