A REPORT ON IMPROVING AND DEVELOPING
AN EVALUATION PROGRAM
THE RESEARCH DIVISION
THE NATIONAL ENDOWMENT FOR THE ARTS
PURCHASE ORDER NO. C80-441
*ILE COK, PLEASE «^*% iVERSITY ASSOCIATES, INC.
MAY 20, 1980
National Endowment for the Arts
Washington, D.C. 20506
University Associates, Inc.
475 L'Enfant Plaza, West ■ Suite 2100 ■ Washington, D.C. 20024 ■ Telephone 202/554-4710
A REPORT ON IMPROVING AND DEVELOPING
AN EVALUATION PROGRAM
THE RESEARCH DIVISION
THE NATIONAL ENDOWMENT FOR THE ARTS
PURCHASE ORDER NO. C80-441
UNIVERSITY ASSOCIATES, INC.
MAY 20, 1980
In February, 1980, the Evaluation Unit within the
Policy and Planning organization line was discontinued as
a distinct entity and the responsibilities were assigned
to the Research Division.
In March, 1980, Harold Horowitz, Director, Research
Division, inquired about my interest in working with him
during the planning period as he developed a systematic
evaluation study program. Since evaluation study approaches
are a long-term interest of mine in work along this line
for the U.S. Office of Education and earlier, the National
Science Foundation, I enthusiastically accepted. We agreed
that in 25 days I would review completed and on-going
evaluation studies and make recommendations for improvement.
I have read stacks of reports, documents and guides.
Possibly, I became involved in substance and content in
greater depth than necessary. Program directors and other
NEA officials freely discussed evaluation problems with me,
and I'm sure when they saw my interest, they willingly
extended their discussions to include the excitment of their
activities. Also, I have held discussions with individuals
outside the NEA staff who are knowledgeable about evaluation
in the arts. Mainly, in this project I have sought ideas
from others, talked about my experiences and in this report
have included my biases. The Research staff have generously
given me as much time as I needed. Bill Potter's help has
been invaluable, as has been the assistance of Candice
Parrish and Maryann Gerard. Thanks to everybody concerned.
Early in my work I became convinced that the assignment
could be expanded and that more would be gained than by
concentrating all of my time and attention on completed and
on-going studies. I discussed with Mr. Horowitz my desire
to range farther and consider some factors in launching a
viable system of evaluation studies. Section 1 covers that
aspect of my thinking. The purpose is to state the many areas
that should be included in a comprehensive evaluation program.
The details for the parts are not presented. Consideration
has not been given to scaling the overall plan to mesh with
resources likely to be available. Section 2 deals with
on-going and completed studies. The discussions that I have
held with Harold and will hold in the near future will be a
main contribution. This report is directed to him for his
Denzel D. Smith
University Associates, Inc.
It is not the purpose of this report to reinvent the
term evaluation, enumerate technical strategies, or debate
the effectiveness of particular evaluation methodologies.
Evaluation of some kind happens, as it always has, for any
program. We attach different values to the ranges of
information available for decision-making situations. Some
leaders feel that the realities of political imperatives
deny the uses of impact evaluation, for example. However,
the timing seems right to seek an improved vantage point
for understanding the extent to which the Endowment is
meeting stated goals.
Similar statements are expressed regularly concerning
the planning function. Yet, the commitment to effective
planning for the Endowment has been made. To many people
neatness of the exercise in planning is not considered to be
very important in the decision making process. To them the
exercise simply provides better copy for presentations, for
example, budget presentations. Also, precisely how information
on program effectiveness and impact can be used to facilitate
management decisions is not well communicated and often not
believed when communicated.
Digitized by the Internet Archive
in 2012 with funding from
Boston Library Consortium Member Libraries
The Endowment is committed to extensive planning — short
range, middle range and long range. I have discerned, however,
that it appears less committed to development of evaluation
in a systematic manner as a means of better understanding
progress toward achieving goals.
Structure and Substance
What, then, should be the structure and substance of an
organizational unit with responsibility for providing
important evaluation information.'
First, all levels within an agency should accept
responsibilities for demonstrating progress toward meeting
their objectives and the agency's goals. This is a coopera-
tive venture. As program planning begins, attention should
be given to stating objectives; the main strategies should
be named and some indicators identified that show progress.
A key factor is that evaluation requirements should be
identified during early planning phases.
The evaluation staff of the Research Division should have
the competence to provide technical assistance to program leaders
in the form of training when requested and advice on how achieve-
ment might be considered. The function should be cast at the
program level in terms of facilitating effectiveness of a
program; and at the agency level facilitating support toward
goal attainment. The supportive service would not duplicate
other services or assume responsibilities of other offices
in preparing program objectives or identifying measures of
The Planning and Budget Division, of course, has primary
responsibilities in developing and implementing the planning
function for budget considerations. The Office of Program
Coordination has the important role in planning at the
program level. At the Council level, it is verbalized that
planning is the responsibility of all -- the Council, panels,
program directors. With this in mind, in order for a network
of meaningful information to flow, input on how progress is
to be measured must be provided. Unless evaluation considera-
tions are taken into account at all stages in planning by
these groups, two results can be expected. First, planning
will become a narrow channel providing a script for budget
planners. Second, evaluation studies will continue to be
descriptive studies without a purpose or program application.
Staff specializing in evaluation approaches should be involved.
How the competence should be developed or where personnel is
stationed should not cause administrative irritation. Certainly
the Research Division must develop technical competence in
Second, postponing the identification of measures and indi-
cators showing objective and goal attainement tends to minimize the
role of evaluation in the planning/management evaluation scheme.
Improved planning capability and increased competence in
research and evaluation are evident under the Office of the
Deputy Chairman for Policy and Planning. Equally visible is
an increased interest in planning among program leaders and
the Office of Program Coordination under the Deputy Chairman
for Programs. These developments lead to the expectation
that the role of evaluation will receive more prominence
than it previously has attracted — not less prominence.
Third, the substance of evaluation programs is dependent
upon a good network that identifies issues. Top management,
the Council, the panels, and the program directors could be
more effective in seeking evaluation capabilities that are
problem solving oriented at the policy level. Research and
evaluation studies should be expected to provide assistance
in understanding the work of the Endowment. To date,
expectations seem especially low as far as evaluation studies
are concerned. For a variety of reasons, the Research
Division has been relegated to being an entity of its own.
The role of evaluation should not be considered as a mere
add-on. The research and evaluation programs should be
in the mainstream and the function oriented toward issues
and problems in goal attainment for the Endowment.
Fourth, the Endowment may not have identified and
communicated its data requirements. There are information
sources within the programs and at the general administrative
level. Some program directors feel that too many data
collections are incomplete, contain inaccuracies, and are not
current. Others believe their needs are met by supporting
non-agency data gathering groups. Some program leaders
develop their own capabilities in data collection. Internally,
there are data processing developments to capitalize on
application information. And, of course, there are the usual
fiscal and budget and grant administration data collection
operations. There are contracts for developing the format
for gathering data, e.g. , the National Information Systems
Project. Doubtless, there are other programs underway. Any
program, especially research and evaluation programs, is
dependent upon reliable statistical information. That, however,
is not the most important consideration. The basic question
is to determine what information is required to meet the
needs of the Endowment.
In 19 77 the Research Division accepted a very good
final report on a feasibility study for an economic data
program on the condition of the arts and cultural organizations.
Recommendations from this report have been followed in research
studies. But, it does not appear that the report was used to
define data requirements for the Endowment. To clarify and
* Grant No. RQO-22-3N, October 31, 1977 by Graduate School of
Public Administration, New York University.
integrate the thinking about information requirements, ' it is
suggested that a working group be formed under the direction
of the Deputy Chairman/Policy and Planning with representation
from all data users. The Research Division could provide
the coordination and assist in identifying linkages among the
many data requirements. It is necessary to emphasize that
evaluation supportive of Endowment policies will be suspect
until these data acquisition and data use areas are better
Fifth, in implementing evaluation studies in a systematic
fashion, there should be less separation between research
and evaluation studies. For example, a program of research
can be responsive to program needs as well as emphasizing the
impact analysis requirements for Endowment policy planning
and decision making. Also, the designs for evaluation
studies can be of high quality in terms of technical
considerations and problem solution.
The term evaluation is disturbing to many managers and
leaders for reasons stated so often. that they are not repeated
here. Emphasis should be placed on analysis of data and
interpretation of information. The requirement is that
researchers and evaluation specialists work with leaders in
defining issues and purposes that require solution. The
results need to be important to members with responsibility
for policy issues as well as to program leaders.
Sixth, program leaders conduct self -evaluation and
internal evaluation studies differently with varying degrees
of quality standards. Some programs will have a stated
impact requirement, for example, Office of Partnerships,
State Programs. Some programs may not have explicit evalua-
tion statements. Nevertheless, self-evaluation and internal
program analysis studies at the program level should be
encouraged. This approach tends to be less threatening and
is useful in improving program effectiveness.
Seventh, a plan for intensive program review should be
implemented to supplement the continuing data collection and
evaluation efforts of each program.
Such a plan would provide an in-depth review of a few
programs each year based on available resources. Innovative
arrangements could be devised using, for example, cross
cutting themes such as fellowships, touring, etc. A schedule
indicating a review every 3 years, for example, would encourage
program staff to be aware of and prepare for such a review.
In this way the network communicating results on issues
enhances chances for timely use of such results in policy
Eighth, provision should be made for discrete, once
only evaluation studies. This is necessary to meet crises.
It is unrealistic to believe such needs will not occur. Even
with a first-rate systematic evaluation approach "crises"
responses will be requested. However, a research and evalua-
tion unit should not be forced to expend many resources on
What does the above say about the development of a
systematic program in evaluation?
1. Blend the research program and the evaluation requirements
so that the purpose of these activities can be communicated
in terms of assistance in solving Endowment policy issues
2. Data requirements for the Endowment should be clarified
and defined. The Research Division, as a member of an
Endowment wide working group, should provide a coordinating
3. In support of all groups with responsibilities in planning
at the program level, develop a capability for technical
assistance in the phase of planning dealing with objectives
and identifying measures of objective attainment. This
would be a complementary role that would contribute to
improved working relations .
4. Seek the opportunity to provide technical and professional
assistance to groups responsible for short and long range
planning from the standpoint of laying the groundwork for
impact studies to be conducted at later dates. In this
way bridging the gap between meeting program objectives
and Endowment goals may have a better opportunity for
5. Develop a systematic multi-year program of evaluation
research that is directed toward Endowment policy and
6. Develop a scheme for periodic intensive external evalua-
tion of each program on a set schedule For example,
resources might allow intensive study of five programs
per year, to be repeated every three years. Innovative
arrangements could be devised using cross cutting themes,
e.g., fellowships, touring, etc.
7. Reserve resources for a few special, one-of-a-kind evalua-
tions each year based on important pressure requirements.
A requirement in the work order for consulting services
is to review completed as well as on-going evaluation studies
to assess procedures and circumstances affecting successful
completion of the projects. These studies have been reviewed
and reactions have been communicated in discussions with
members of the Research Division staff. Also, discussions
about the uses of the results of evaluation studies have been
held with many program directors, other NEA officials and
several individuals outside NEA who are knowledgeable about
The negative aspects noted in reviewing completed evalua-
tion studies can be listed, but that approach has not been
productive in this situation simply because the conditions
are different now than previously, and the capabilities
within the Research Division and throughout the Endowment are
quite improved in relation to the period that the Evaluation
Unit functioned as a separate entity. It would be interesting
to search out usable evaluation information and impact find-
ings and trace their uses. Tracing uses is an interesting
technique but is very time consuming and was not pursued.
Informal discussions on uses of findings of completed
evaluation studies were not productive either.
The Evaluation Report
1. The final reports are too long and difficult to understand.
This statement is a common one, and, expected. The
reasons for the lengthy presentations in reports are many.
Sometimes the sponsor desires a wide distribution of a
report and often encourages long programmatic descriptions.
Evaluation reports should be directed toward a stated
user with well defined purposes and the contractor should
not be in doubt about this.
2. Evaluation reports too often describe activities. The
analysis, assessment and interpretation become secondary.
Program staff need good descriptions of project activities
to meet a variety of requirements. But, evaluation
studies may not be the best source of description of
program activities. Too, there may be confusion in
terminology in some situations. Simply describing all
activity is not assessing its effectiveness, its quality,
or its impact. There are two considerations here. (A) More
precise language should be written in the solicitations.
Questions listed in solicitations too often encourage
description. (B) The language of evaluation is not common,
not standard among the program staff and the researchers.
The Research Division should provide a common vocabulary
as a means of improving internal communication. This
seems like an opportune time to establish a common technical
language in research and evaluation.
3. The format for evaluation reports may be confusing to
a contractor and may be dulling the interest of a
targeted reader. For example, in several reports the
executive summary is just another report and the importance
of outstanding impacts becomes less distinguishable. It
would be better to direct the contractor to deal with
significant findings. The Research Division should
assume responsibility for preparing the action document
when the report is forwarded to a target administrator.
In this way the issues can be restated, the policy
problem that had been the basis for the evaluation study
can be laid out and the action that should be taken can
be stated by the competent professional within NEA. All
of the above should be performed within the time frame
previously established by the administrator responsible
for action on the issue.
These paragraphs are directed toward improvement in
solicitations and do not enumerate the many good elements in
the format in use.
1. The most important improvement should occur prior to
preparing and issuing the solicitation. The policy
issues that suggest an evaluation study should be
documented. The present solicitations state questions
to be answered but the statement of issues and the
problems to be solved may not be evident. What policy
issue is being discussed?
The name of individual or group, e.g., a Congressman
wanting an evaluation of a program may be known to the
staff but this may not be documented. Statements on
issues, a statement of problems to be solved, who wants
the evaluation, and what action is required are items
to be accepted and approved. Agreement on these topics
would help to understand the needs of top management,
i.e., is the evaluation study an expedience dealing with
funding or is it intended to clarify the role of the
2. The substantive content of the solicitation would profit
from more critical analysis. The suggested questions to
be answered in a solicitation should not be a long
shopping list. The staff should discriminate between
information required and why and information that would
be "nice to know. "
The Design of Evaluation Studies
There are few comments concerning the design of completed
and on-going evaluation studies. The methods of sampling and
the approaches used vary in quality, as expected. As the NEA
staff gain more experience and acquire more technical competency
in evaluation methods, data analyses and interpretation will
improve. Even so, the turning point in using the results of
evaluation studies will be the astuteness in the design for
evaluation studies. There is no reason to believe that
questionnaire surveys will be discontinued but there will
not be many questions that ask a respondent simply whether or
not the grant funds received were useful. Also, for many
years NEA will depend upon secondary sources for data for use
in analysis but the responsible NEA staff will have a
better understanding of the accuracy and comprehensiveness of
the data collections available. Too, the quality of evalua-
tion proposals should improve as there are several research
groups around the nation with good track records in evaluation
research and eager for business. Be more agressive in selection
Monitoring a Contractor
The important work of the staff is done up front — the
pre-solicitation definitions, the staff decisions on acceptable
possible methodologies, the clarity and tightness of the
solicitation, and the definiteness of design during contract
negotiations. Assistance to the contractor in implementing
the study continues, of course, but after the up-front work
is completed, it is better to stay out of the way, to keep
informed and helpful so that the need for intervention is
unlikely. The research staff is competent and the need for
further discussion is unnecessary. However, because evaluation
results are feared by some and there are those who feel that
evaluation studies have little value, a review of the philosophy
and strategies in monitoring is suggested. For example, it
isn't useful to complain about the use of a data source after
the study is completed. It doesn't do much good to ask a
contractor for monthly statements of progress if the material
is not used except for administrative protection.
Finally, the evaluation studies to date may have been
used in more ways than have been suggested. For example,
information may be used for general purposes in preparing
program reports. The fault is that the results of evalua-
tion studies have had limited acceptance by the Endowment
staff and have been given a place of low value as aids .in
solving problems. Now, it can be hoped the thrust will be