Evaluation of the quality of research and development can itself be considered a form of research ("meta-research," perhaps), but the methods used are in an early state of development. While many parameters could be used to measure R&D effectiveness, most are confounded with other factors. The trade data in Fig. 2 shows Japan's strength in the U.S. market for high technology products, but such successes also require good management, ample capital, aggressive marketing and many other resources. Indeed a company or nation does not even have to perform the original R&D to be successful in production; Japan, Korea, and Taiwan achieved their early successes with imported technologies. Likewise, assessing research by counts of patents, technical papers, or citations is flawed. While Fig. 3 shows that the Japanese are granted as many of U.S. patents from foreign sources as the rest of the world combined, company incentives for engineers to file patent applications may give the appearance of productivity without the reality. University publish-or-perish policies may lead to quantity over quality in publications and citations to them. Another problem is that many foreign technology assessment programs are conducted by a staff of policy analysts. Since a small, fixed staff cannot be expert in all the relevant disciplines, it is reduced to surveying the general opinions of others or to evaluating input resources to the research process. Since such investments are not always productive, it is really necessary to assess the output benefits.
The most direct way to isolate the success of R&D efforts is to organize a peer review team of experts in the same field to inspect the research results and make a judgment of their quality. This is the process that International Technology Research Institute (ITRI) has used in over 40 studies of foreign technologies since 1989. This approach has proven to be effective because foreign scientists have been generous in sharing their work with American delegations.
The initial focus of ITRI was on Japan under the name, Japanese Technology Evaluation Center (JTEC). The 18 JTEC studies completed studies completed between 1989 and 1994 (shown in Table 1) include a very substantial fraction of the publicly available assessments of Japanese technology funded by the U.S. Government during that period. With Japan-related WTEC studies undertaken since then (see Table 2), the results form a unique database documenting the rapid advance of Japanese R&D capabilities from 1984 to the present. In 1990 the scope was expanded worldwide through change to the name "World" Technology Evaluation Center (WTEC).
Much of the strength of the WTEC effort comes from the quality of its panelists. They include individuals who have served as Chief of Naval Research, Chief Scientist of the Air Force, Under Secretary of Commerce for Technology; Associate Administrator of NASA; vice presidents or provosts of Rensselaer, Ohio State, and the University of California System; and many distinguished academic, government, and industrial engineers and scientists.
Reports on each assessment are available from the National Technical Information Service. The JTEC/WTEC Program Summaries (Holdridge 1995, 1997) provide executive summaries, and many of the full reports are available on the Web at http://itri.loyola.edu. A more detailed summary of WTEC methods and findings was presented at the previous JISTC conference in Newcastle (Holdridge and DeHaemer 1995).
Shelton has analyzed the studies through 1994 to make the case for the success of Japanese technology policy. While the individual study findings vary, WTEC panels have found a consistent pattern in Japanese R&D:
After World War II, Japan perfected and exploited technologies largely imported from abroad. By 1984 it had become clear that Japan intended to develop its own technologies. The Japanese still emphasize applied research with relatively little lab work in basic research. Thus their basic-research component of R&D is an international technology-transfer process at which they excel. During 1984-1994, while continuing to import technologies, Japanese R&D advanced rapidly relative to the U.S. in many fields and came to equal or surpass U.S. capabilities in some. The Japanese now lead in R&D for many kinds of electronic devices, and in important aspects of telecommunications. U.S. R&D still leads in R&D for computer science and biotechnology, but the Japanese gained during 1984-1994. The Japanese place greater emphasis on materials R&D in most fields studied and are gaining relative to the U.S. In other fields results are mixed, but there were several (construction, nuclear power, and mechatronics) where Japanese advances relative to the U.S. were found. In addition to factors like patient low-cost capital, vertical integration, and protection of their domestic market; Japanese government technology policies, including encouragement of heavy R&D investment, have contributed to this advance. (Shelton 1994)