2001-2006
A
CCOMPLISHMENT
R
EPORT
APPENDIX1
ARSN
ATIONALP
ROGRAMA
SSESSMENTPurpose and Goals: National Program Assessment is the final step of the ARS National Program cycle. The National Program’s performance is assessed by an external panel of knowledgeable customers and stakeholders against commitments (research goals, objectives, products, outcomes) identified in the National Program Action Plan created at the beginning of the 5-year National Program cycle. The National Program Action Plan defines the research to be done by ARS in response to identified needs supplied by users of ARS research. This input is gathered from customers and stakeholders at
workshops, from regulatory agencies, and from Congress (legislation). Performance is evaluated based on the quality of the research leading to actual impact or progress toward anticipated benefits to end- users, scientific communities, and/or broader society.
The assessment provides feedback to customers, partners, and stakeholders focusing on accomplishments relative to promises made, ARS managers as an indicator of National Program progress, ARS scientists to help them focus and direct their efforts on the potential goals for the next program cycle, and to OMB regarding performance and impact of the research programs within the Agency. The assessment process is managed by the Associate Administrator for National Programs. ARS National Program Cycle: The 5-year process
depicted conceptualizes how ARS conducts its business of research. The National Program cycle is shown aligned with the OMB Research and Development Investment Criteria. The three words around the outside—Relevance, Quality (Prospective and Retrospective), and
Performance—represent what a research organization must provide to be successful. The descriptors on the inside—Program Planning and Priority Setting, Scientific Merit Peer Review, Project Implementation and Coordination, and Program Assessment —
prescribe the actions that the Agency undertakes carefully, thoroughly, and with outside review to assure ourselves
and others outside the Agency that our research is of the highest quality.
Process: Program assessments are carried out by External Assessment Review Panels (generally five to six anonymous members and a non-anonymous chair). Panel chairs are selected by the Associate Administrator, and panel membership is determined by the panel chair. Panels are composed of accomplished scientists diverse in expertise, job assignment, and geographic location and who have minimal conflicts of interest.* The charge to the panel is to assess the 5-year performance and impact level of the program. They are asked to consider the overall National Program and determine if the
___________________________________________
*COI clarification:
• Does the reviewer receive current financial support (e.g. research and/or development grants or contracts, procurement contracts, consulting contracts, other grant support, etc.) from the program being evaluated that could be directly affected by the panel’s report (e.g. possible termination of current arrangements or loss of reasonably anticipated future funding)?
• Does the reviewer receive substantial current non-financial support (e.g. equipment, facilities, industry partnerships, research assistants and other research personnel, etc.) from the program being evaluated that could be directly affected by the panel’s report?
• Does the reviewer have any other current financial interest (e.g., patent rights, interests in partnerships and commercial ventures, etc.) obtained from or through the program being evaluated that could be directly affected by the panel’s report?
Relevance Relevance Performance Performance Quality Quality (Prospective) (Prospective) Quality Quality (Retrospective) (Retrospective)
OMB Research & Development Investment Criteria
Program Planning & Priority Setting
Project Implementation Scientific Merit Peer Review Program Assessment
ARS National Program Cycle
Program Coordination
National Program has met its customer/stakeholder needs or solved their problems since the last
customer workshop. The panel is provided a National Program Accomplishment Report developed by the National Program Team from data provided by the scientists who have research directly related to the National Program being reviewed. The Accomplishment Report is a synthesis of National Program accomplishments, not the specific accomplishments of individual research projects. The Accomplishment Report could provide by reference to the Action Plan:
• Why the research was done.
• What the targets of the research (goals, objectives, products, and/or outcomes) were. • What difference the research made.
• What the broad scientific accomplishment of the components was.
• What documentation evidence is available to support the reported findings (e.g., key technical publications and technology evidence).
• A brief description of the resources available to the entire National Program.
The panel also receives a copy of the National Program Action Plan and a set of specific criteria to provide clarity to the term impact. The following is a list of possible assessment criteria:
• Did the research contribute to the development and/or implementation of regulations? • Were other government and/or industry programs influenced by the research?
• Did the research influence/impact other researchers conducting research in the same or related scientific fields?
• Did the research advance knowledge?
• Were major agricultural problems ameliorated, mitigated, or solved?
• Has the research resulted in technology that has been patented or licensed, and if so, has it led to commercialization?
• Has the research yielded health, social or economic advantages for consumers?
• Have new or improved scientific methods or technologies been developed by ARS and adopted by others (e.g., customers, stakeholders, consumers, and/or other scientists)?
Not all criteria would apply to all National Programs or all program components.
The National Program Team and the National Program Assessment Coordinator provide panel chairs with an orientation, which includes an overview of the National Program assessment related to the ARS’ National Program cycle, the assessment goals and process, background on the National Program, and instructions on building their panel.
Product: The product of the panel review is a written report on the quality of accomplishments of each component of the National Program. The panel is asked to provide recommendations that ARS
managers can use in making decisions about the management of the next National Program cycle. The panel is asked to assess the quality of the accomplishments of each component as High Quality
(significant benefit, influence, and/or progress toward one or more of the assessment criteria), Medium Quality (moderate benefit), or Low Quality (little or no benefit). Additionally, panel chairs are expected to attend the next National Program Customer/Stakeholder Workshop to present the panel results.
Suggested Evaluation Definitions:
High Quality: “Significant benefit, influence, and/or progress toward” one or more of the
assessment criteria noted above, e.g., significant benefit to the development and/or implementation of soil management/conservation practices.”
Note to panel: Research that is of high quality would be considered to be leading edge research and to have had or expected to have a substantial impact.
Medium Quality: “Moderate benefit, influence and/or progress toward” one or more of the assessment criteria noted above, e.g., moderate progress has been made toward the development and/or implementation of regulations/policy.
Note to panel: Additional research and more time may be needed for this research to move from a status of potential impact to that of realized impact. Alternatively, research with this rating may be on a relevant topic, but the quality of the research is not providing the most pertinent information. Low Quality: “Little or no benefit, influence, and/or progress toward” one or more of the
assessment criteria noted above, e.g., little or no influence on the development and/or implementation of regulations/policy.
Note to Panel: Research with this rating may be good, but the topic is not relevant or the research is neither of high quality nor on a relevant topic.