APPENDIX C

QUESTIONNAIRE

The questionnaire shown below was sent in advance to all the organizations visited in this study, except for Nikkei AI. Section A focuses on industrial applications, and hence is relevant to the twelve industrial sites only. Section B contains questions about advanced research and was potentially relevant to all the recipients. The questions in Sections C and E were directed specifically to ICOT personnel. Section D was prepared specifically for the EDR project.

The questionnaire is quite long and detailed (particularly Section A), and we were not sure if any of our hosts would take the time to prepare written answers to the questions or merely use them as talking points at the meetings. We were surprised and delighted at the level of effort expended by most of the organizations in preparing written answers. In particular, Section A was completed by all twelve industrial sites. Section B was addressed by a few industrial sites, and served as a guide for discussion at the universities and national projects. ICOT did not prepare written responses to Section C and E, but the topics were discussed during the visit; the same situation applies to the EDR project.

A. INDUSTRIAL APPLICATIONS

The following questions are for organizations that have developed and are using expert systems.

  1. Approximately how many expert systems has your company developed, or is developing?
    1. for internal use?
    2. for external use by customers?
  2. Please categorize the above according to development status (give approximate percentages):
    1. in routine operation
    2. now in field test
    3. in prototype stage
    4. prototype still in development
  3. Please state approximately how many applications you have in each of the following task areas:
    1. design
    2. advisory, or "help" systems
    3. process control
    4. planning and/or scheduling
    5. diagnosis and/or troubleshooting
    6. regulatory compliance
    7. management or business integration aids
    8. areas of manufacturing not included in any of the above categories
    9. sales
    10. finance
    11. software development and management
    12. intelligent interfaces to existing software systems
  4. What is the current most successful expert system that your company developed for its own use or for a customer? How do you measure its success?
  5. For each task area where you have one or more applications, select a representative system (including the most successful described above), and provide the following information:
    1. what is the specific task?
    2. how large is the knowledge base (number of frames, number of rules, lines of code, as appropriate)?
    3. was a commercial development tool or framework used to build the application? If so, which one? If not, was an internally developed tool used?
    4. is the application a stand-alone system, or is it integrated with conventional data processing systems?
    5. what programming language was used (if any)?
    6. what kind of hardware platform (personal computer, workstation, minicomputer, mainframe, client-server system, distributed or parallel system)?
    7. what was the size and composition of the team that designed and built the system?
    8. how long was the development time?
    9. what was the development cost?
    10. who developed the system (domain experts, professional programmers, professional knowledge engineers)?
    11. who are the users?
    12. who maintains the system, and how large is the maintenance team?
    13. what kind of payback has been realized from this application (e.g., cost reduction, quality improvement, speed of problem solving or decision making)?
    14. in economic terms, what has been the return per year?
    15. what was the payback time, i.e., the number of months or years to pay back development costs?
    16. does this application cooperate with or depend on other expert systems (i.e., do they contribute to each other's operation in some way) and if so is there a synergy between the applications?
    17. what problems were encountered, if any, in integrating the system into the corporate environment, and how were these problems addressed?
  6. Has your company implemented any applications that require multiple sources of expertise? If so, was the expertise combined in a single system or through the coordination of multiple systems?
  7. Do you have any applications written partly using AI tools and partly using standard high level languages (COBOL, FORTRAN, PL/1, etc.)?
  8. How fully integrated with your operating system are your AI tools? E.g., can you call any access methods, procedural higher level language routines, library subroutines from your AI tool (and vice versa)?
  9. What training programs for expert system development work are in place in your company?
  10. Has your company developed a methodology for developing expert systems? If so, what is the methodology?
  11. In your company, how are expert systems
    1. selected (i.e., what criteria are used)?
    2. developed (i.e., by data processing group, by a knowledge engineering group, by outside contractor)?
    3. inserted into the company's operational activity?
    4. maintained?
    5. integrated with conventional data processing activities?
  12. Are there systematic review and redesign cycles for expert system projects?
  13. What percentage of projects that are started get as far as completing a prototype system?
  14. What percentage of projects get from the prototype stage to an operational system?
  15. Have there been any expert system projects that were unsuccessful? If so, what were the reasons for the lack of success? Was the problem caught early or late in the development cycle?
  16. Given your existing AI tools, how large a project would your organization undertake (e.g., in terms of number of rules or number of person-years devoted to the project)?
  17. What new expert system applications are planned by your company?
  18. What will be the progress of expert systems in your company? For example, will process control applications use in-the-loop control?
  19. What are the main problems with present technology? What kinds of applications that are hard to build now would you like to see made easier? Do you see any need for advanced techniques (e.g., model-based reasoning, machine learning) in current or planned expert system applications? Do you see resistance to the use of those technologies?
  20. What other advanced computing technologies does your company use (not necessarily expert systems)?

B. ADVANCED KNOWLEDGE-BASED SYSTEMS RESEARCH

  1. Are any efforts being made (besides EDR) to build the technology for large knowledge bases? Are there experiments in large knowledge bases? Is there a software or communication infrastructure that has been developed?
  2. What is the research being done experimentally or theoretically on model-based reasoning? Modeling of devices and reasoning about their behavior? Are any languages or systems being developed to assist with this work?
  3. What are the major projects in machine learning (using knowledge-based techniques, not neural networks)?
  4. Have you developed any new techniques to handle problems of reasoning with uncertain knowledge?
  5. If you think of advanced knowledge-based systems research as involved in "inventing the second generation" of knowledge-based systems, then what are the most important dimensions of this "second generation?" That is, what is it most important for the researchers to invent?
  6. What do you think are the important features that should be included in second generation development tools or frameworks?
  7. What is the interest among computer science and engineering students in knowledge-based systems research? How many Ph.D. students do you have working with you, or are working on knowledge-based systems in your department?
  8. Are you working on any advanced truth maintenance systems?
  9. Are you applying knowledge-based systems to natural language understanding? To problems of education, at any level?
  10. Are you doing any work in case-based reasoning research? What efforts in Japan in CBR are most worth studying?
  11. Are you working on general problems of knowledge representation, for example by inventing new concepts of KR, or new languages for KR, or new systems for KR?
  12. What are your sources of support for the research? From Ministry of Education? From other government agencies? Does any support come from companies?

C. ICOT

  1. What do you consider to be the major technical successes on the Personal Sequential Inference (PSI) machine? How successful was the effort to commercialize the PSI machine?
  2. What is the current work of the various ICOT laboratories?
  3. What performance is expected from the Parallel Inference Machine (PIM) to be completed in 1992?
  4. Briefly described, what is the current PIM architecture -- that is,
    1. what are the activities at a node?
    2. how is the communication between nodes handled?
    3. what are the high-level parallel algorithms that organize the logical problem solving?
  5. What is the current state of the operating system for the PIM?
  6. What is the current state of ICOT's PROLOG work?
  7. What applications are being done at ICOT? Are there expert system applications? What organizations outside of ICOT are using ICOT-developed technology to build expert system applications? What is the best application done with ICOT-developed technology?
  8. What are the major advances that have been made by ICOT in the Natural Language Understanding area?
  9. Can you quantify the role that ICOT has played in spreading knowledge of AI and logic programming among Japanese company engineers and managers?
  10. Assuming that there will be a second period of government funding for ICOT, what are the plans? How long? How many people at ICOT? What research will be undertaken? What level of government funding has been promised?
  11. Of all the things that ICOT has done in ten years, what stands out as the most satisfying to the Director and the staff? What was the most disappointing experience of the ten-year period?
  12. Several Japanese companies (and a few American companies) also explored novel architectures for LISP. Was there any relationship between this work and the work at ICOT? If so, how did the efforts influence one another?
  13. PROLOG and other logic-programming formalisms have not captured the same attention in the U.S. AI community as they have in Japan. Are you satisfied with your choice of logic-programming as the basic framework? Do you think that you should have adopted more of the functional programming style of the LISP-like languages?
  14. While PSI and PIM were being developed, the RISC revolution took place. To what extent did this affect your work? Would you have adopted more of a RISC-like architecture if your work had started later?

D. ELECTRONIC DICTIONARY RESEARCH PROJECT

  1. What is the current state of the EDR project? In particular, what is the current state of completion of the semantic dictionary?
  2. Please give us some current details about implementation. What language is being used to handle representation? What set of concept primitives are you using?
  3. As the knowledge base gets larger and larger, what problems of scale-up are you encountering, and how are you handling them?
  4. Are the dictionaries (including semantic dictionary) in trial use in any of the companies? How is the use done?
  5. Are you going to use ICOT's PIM for running the dictionaries for natural language understanding projects?
  6. How big is the current staff of EDR? What are the plans for EDR? Will it grow? Shrink? What new projects will it undertake?

E. REAL WORLD COMPUTING PROGRAM

  1. What research efforts will RWCP undertake in the area of AI or, more specifically, in the area of knowledge-based systems?
  2. For these AI-related or KBS-related projects, how much government funding is planned? Over what length of time? What companies are involved? Will there be a project institute? What is ETL's role in RWCP?
  3. Will these AI-related research projects focus on particular application domains (perhaps as testbeds), and if so what are they?
  4. What role will parallel computing play in these AI-related projects? What kind of parallel computing is envisioned?

[Appendix B][Top of Report][Send Your Comments][WTEC Welcome Page][Appendix D]

Published: May 1993; WTEC Hyper-Librarian