(navigation image)
Home American Libraries | Canadian Libraries | Universal Library | Community Texts | Project Gutenberg | Children's Library | Biodiversity Heritage Library | Additional Collections
Search: Advanced Search
Anonymous User (login or join us)
Upload
See other formats

Full text of "A Review with recommendations of the North Carolina teacher performance appraisal instrument"

A REVIEW WITH RECOMMENDATIONS OF THE 

NORTH CAROLINA TEACHER PERFORMANCE 

APPRAISAL INSTRUMENT 

Submitted to; 

THE EDUCATION SUBCOMMITTEE 

OF THE JOINT LEGISLATIVE COMMISSION ON 

GOVERNMENT OPERATIONS 

North Carolina General Assembly 

State Legislative Building 

Raleigh, NC 27611 

Submitted by: 

Richard M. Brandt (Chair) 

Daniel L. Duke 

Russell L. French 

Edward F. Iwanicki 

May 21, 1988 



i 



i 



Route 1, Box 619 
Crozet, VA 22932 
May 21, 1988 



The Education Subcommittee of the 

Joint Legislative Commission on 

Government Operations 
North Carolina General Assembly 
State Legislative Building 
Raleigh, NC 27611 

Gentlemen: 

It is a pleasure to submit this report of our panel 
activities, findings, and recommendations regarding the 
performance appraisal instriiments . We trust that we have 
responded to your charge appropriately. We hope that our 
findings and recommendations will prove useful in the 
months ahead. 

We very much appreciate the courtesies extended to us 
in the conduct of our study by you, the members of the 
Subcommittee; by Joan Rose, contract administrator; by the 
several members of the Department of Public Instruction 
staff with whom we met; and by local school district 
personnel during my prepanel visits. Without splendid 
cooperation and assistance from all these fine people, our 
task would have been much more difficult. 



Sincerely yours, 
Richard M. Brandt 



RNB:bKs 



EXECUTIVE SUMMARY 

Our study and review of the TPAI leads us to one 
overriding conclusion. It is a quality instrument, one 
that is highly suited to its purposes. The 
recommendations we list below represent suggestions for 
improving it still further, especially to enhance its 
effectiveness as a basic tool for making career ladder 
decisions and for improving teaching. In no way do these 
recommendations imply fundamental deficiencies that would 
call for its elimination as the state teacher appraisal 
instrument. We consider our recommendations suggestions 
for fine-tuning an already good system of performance 
appraisal. 

Likewise, our limited commentary on the PPAI and other 
specialty measures does not suggest they are inadequate, 
only that we had less to examine because of their recent 
development and lack of widescale use. On the surface, 
they look promising and, with the few exceptions we have 
noted, consistent with the TPAI in purpose and procedure. 

We summarize our recommendations in the order in which 
they are presented in the text. 
CRITERIA 

1. Rearrange some functions and teaching practices to a 
total of twelve, ten of which relate directly to 
teaching and are observable in the classroom. 
Specifically; 

i 



o Reverse the order of functions 6 and 7. 

o Rename new function 6 "Communication and 

Interaction in the Classroom." Include function 
6 as one of those to be observed in the 
classroom, and list relevant teaching practices 
as follows: 

6.1 Teacher speaks fluently and precisely 
(currently 3.3) 

^'^ IrtZ^^ presents the lesson or instructional 
activity using concepts and language 
understandable to the students (currently 

^'' Tcu^^i'"^^''^^^^ effectively with students 
(currently part of 7.2) 

6.4 Teacher treats all students in a fair and 
eguitable manner (currently 7.1) 

Divide current function 3 (Instructional 
Presentation) into five specific functions as 
follows: 

o Initiating Instruction 

o Motivating Students 

o Managing Routines and Transitions 

o Presenting Accurate and Appropriate Content 

o Providing Closure 

Teaching practices for these latter functions 

would be taken from current function 3 (excluding 

3.3 and 3.4) and from such instruments as the 

Connecticut Competency Instrument. 

Expand the literature search for effective 



11 



teaching practices to include coaching, modeling, 

cooperative and mastery learning models of 

teaching in addition to process-product research 

on direct instruction 

o Encourage local districts to identify practices 

that reflect local needs for the two functions 

that are less observable in the classroom and 

relate less directly to instruction, i.e.: 

o Facilitating Instruction (currently F6) 

o Performing Non-Instructional Duties 
(currently F8) 

FODI AND POST-OBSERVATION CONFERENCE 

2. In addition to scripting, consider collecting other 
kinds of observational data on a pilot basis such as 
student participation and on-task behavior, teaching 
patterns, and other classroom action. While we do not 
recommend changing scripting procedures at this time, 
other kinds of data might well be needed in the future 
to assist in strengthening the objectivity and power 
of the TPAI to differentiate between good and superior 
teachers. 

FODI 

3. Provide teachers with copies of all descriptive data 
(FODIs and any other descriptive instruments used) in 
advance of the post-observation conference. 



Ill 



FODA 

4. Provide space on the FODA form for the evaluatee to 
respond to observations and quality assessments made 
by the evaluator. 

TRAINING 

5. Provide evaluators with specific training in 
conferencing skills and using the FODI and FODA in 
post-observation conferences. 

SCALE STEPS 

6. Maintain the six-point rating scale. 
RATING PROCEDURE 

7. Follow a step-by-step procedure in making summative 
judgments: 

a. Is performance at least at standard? 

b. If so, is it at or somewhere above standard? 

c. If above, is it merely above standard (4) or well 
above (5) or superior (6)? 

PDF 

3. Revise the Professional Development Plan form somewhat 
to reflect more highly individualized, professional 
development beyond basic skill competency, especially 
for above standard teachers. 

PORTFOLIO 

9. Develop a form and procedure for recording and 
analyzing portfolio contents systematically in 
connection with the non-classroom functions. Such 



forms and procedures should be tailored, however, to 
local district priorities. 
INTERVIEW 

10. Develop a structured interview format for collecting 
data from principals, special groups (e.g. school 
psychologists, counselors, etc.) and teachers in a 
systematic, consistent fashion about functions not 
readily observed in the classroom. 

CERTIFICATION OF EVALUATORS 

11. Require all evaluators to be certified for this role 
by passing a performance-based test of their 
competency to use the TPAI accurately and 
consistently. Evaluators should be required to 
demonstrate their competence on a regular basis 
against criterion measures. They should not be 
permitted to function as evaluators unless they are 
certified and remain certified as evaluators. 

CAREER LADDER DECISIONS 

12. Restrict the decision-making use of the TPAI to 
certification and career levels I and II decisions. 
Do not attempt to use it as the primary measure for 
career level III decisions. 

SUMMATIVE RATINGS BY CONSENSUS 

13. Require the joint participation of OEs, principals and 
other evaluators who are involved in the data 
collection process in the final summative judgment. 



Training in achieving data-based consensus is also 
recommended. 
CONSECUTIVE OBSERVATIONS 

14. Experiment with observing teachers up to four 
consecutive days teaching the same class for possible 
inclusion of such a requirement for career level II 
applicants. 

JOINT OBSERVATIONS/ANALYSES 

15. Create panels of administrators and teachers to review 
and rate samples of FODIs and FODAs for comparability. 
Conduct joint observations, gather data, and share 
analyses to enhance the consistency of instrument use 
and interpretation. 

RELIABILITY STUDIES 

15. Conduct various studies of rating reliability at both 
the state and district levels to assess the 
consistency across raters and the stability of teacher 
patterns from one time to another. 

VALIDITY STUDIES 

17. Conduct validity studies within districts of (a) 

achievement gain scores of students taught by teachers 
rated at standard (#3) versus those taught by teachers 
rated 5 and 6; and (b) student gains in pilot 
districts versus those in comparable non-pilot 
districts. 



VI 



UNIVERSITY RESEARCH ASSISTANCE 

18. Request assistance from university faculty and provide 
graduate student financial support for the conduct of 
TPAI reliability and validity studies in the pilot 
effort. 

WORD PROCESSORS 

19. Provide evaluators with word processors and 
appropriate training as needed to assist them in 
writing FODIs and FODAs. 

INCREASE OF OEs 

20. Reduce the ratio of teachers to OEs to 48 to 1 in 
order to increase the amount of assistance available 
to teachers for remediating deficiencies and improving 
their instruction. 

GROWTH BEYOND COMPETENCY 

21. Focus career level III criteria on leadership 
functions and growth beyond competency in basic 
skills. Consider a more extensive range of criteria 
and measures. 

PPAI/OTHER SPECIALTIES 

22. Review the several instruments for principals and 
other specialty personnel for parallel coverage and 
structure. Improve the consistency in format and 
comprehensiveness wherever possible. 

PRINCIPALS AS EVALUATORS 

23. Assign greater weight to the performance of principals 



Vll 



as teacher evaluators than to other functions in the 

PPAI as one way to improve the TPAS . 
EXTRA RESOURCES FOR TPAI REVISION 
24. Provide extra assistance for the Department of Public 

Instruction to help implement the 2 3 recommendations 
above during summer 1988. 



Vlll 



TABLE OF CONTENTS 



Page 



EXECUTIVE SUMMARY i 

INTRODUCTION 1 

PURPOSES OF THE TEACHER PERFORMANCE 

APPRAISAL INSTRUMENT 4 

TPAI CRITERIA: TEACHING FUNCTIONS AND 

PRACTICES 7 

PROCEDURES AND INSTRUMENTS 17 

Formative Observation Data Instrument (FODI) . . 22 

Formative Observation Data Analysis (FODA) . . . 24 

Teacher Performance Appraisal Instrument 

and Rating Scale 25 

Professional Development Plan (PDF) 27 

Need for Additional Instruments 28 

Training of Evaluators 30 

Use for Summative Purposes 34 

Use for Formative Purposes 4 

APPRAISAL OF PRINCIPALS AND OTHERS 45 

A FINAL RECOMMENDATION 49 

REFERENCES 50 



IX 



INTRODUCTION 

A panel of four teacher evaluation specialists met in 
Raleigh from May 8-12, 1988 to review the North Carolina 
Teacher Performance Appraisal Instrument (TPAI) and the 
Principal Performance Appraisal Instrument (PPAI) . 
Members of the panel were Drs. Richard M. Brandt 
(chairman) , Daniel Duke, Russell French, and Edward 
Iwanicki. 

The charge by the Education Subcommittee of the Joint 
Legislative Commission on Government Operations included: 
o evaluating the TPAI and PPAI 
o giving an opinion on the fairness and objectivity 

of these instruments 
o giving an opinion whether the currently adopted 
requirements established for and the distinction 
between career levels for teachers and principals 
are appropriate 
Before and during these five days, panel members 
studied several dozen documents about the TPAI and PPAI 
including legislation, manuals describing the instruments 
and procedures for their use, training materials, studies 
that show the distribution of summative ratings across 
career development pilot districts, surveys of teacher and 
principal reactions, memoranda and directives from the 
Department of Public Instruction, local district forms and 



reports, and notes made by the chairman while conferring 
with superintendents, career development program (CDP) 
coordinators, outside evaluators (OEs) , principals, and 
teachers during visits to several CDP pilot districts. 
Between May 9 and May 12, the panel met with the 
subcommittee on two occasions, the first to clarify the 
charge and the second to provide an overview of findings. 
The panel also net twice with DPI staff for approximately 
two hours each to ask questions about the instruments and 
the many documents we were studying. 

Needless to say, the performance appraisal system is 
complicated. It involves all of the certified school 
personnel in North Carolina. Many features must be 
reviewed and many factors considered if one is to provide 
a thorough and sound evaluation of such a comprehensive 
system. 

Before reporting our findings and recommendations, we 
wish to make three general observations: 

1. We commend the North Carolina legislature and the 
Education Subcommittee in particular for recognizing the 
need for a strong teacher evaluation system and for 
providing the leadership to see it established. Strong 
leadership and continued attention will be necessary to 
make sure it functions well and truly serves to improve 
the education of North Carolina's young people. 

2. We wholeheartedly support the incremental approach 



the Subcommittee has taken to pilot test the TPAS and CDP 
carefully before going statewide with them. Implementing 
the career ladder in pilot districts over a four-year 
period provides a unique opportunity to fine-tune the 
instruments and procedures as they are being developed and 
field tested. What the TPAI has become as a result of 
careful, thorough development over a several year period 
is a quality instrument, we all agree. What we will 
recommend should strengthen it even further and help in 
that fine-tuning process. 

3. We also want to commend the educators with whom 
we have talked at both the state and district levels. They 
have been gracious with their time, candid, and most 
helpful in providing the numerous materials we requested 
and in answering our many questions. We respect them for 
their great dedication and effort to implement these 
programs well. It is no easy task, and they should be 
congratulated for their many accomplishments so far. 



PURPOSES OF THE TEACHER PERFORMANCE APPRAISAL INSTRUMENT 

The TPAI is being used currently for the following 
purposes : 

1. Certification - To insure that new teachers have 
the skills necessary to continue in the teaching 
profession. By the end of their first two years 
in the profession, beginning teachers must 
demonstrate at least "at standard" performance 
with respect to the first five functions of the 
TPAI in order to be granted a renewable 
continuing certificate . 

2. Quality Assurance - To assure the public that 
only competent professionals are allowed to 
remain in the teaching profession. The TPAI is 
used to evaluate all teachers. Tenured teachers 
are required to demonstrate at least "at 
standard" performance on all eight functions of 
the TPAI. 

3. Career Development - The North Carolina Career 
Development Plan has been designed "to attract 
and retain the best people in teaching. ..." 
The TPAI is the "cornerstone" of this 
performance-based plan. Teachers must 
demonstrate at least "above standard" performance 
on all functions of the TPAI before they can 
advance to a higher career status level. 



4. Professional Growth and Development - To identify 
aspects of teaching performance which need to be 
strengthened or enhanced, and to monitor teacher 
development in this regard. For many teachers, 
especially beginning teachers and experienced 
teachers encountering difficulty, the TPAI 
provides rich information for use in designing a 
Professional Development Plan (PDP) to foster 
growth with respect to particular teaching 
practices. 

Is the North Carolina Teacher Performance Assessment 
Instrxunent adequate for these purposes? 

The TPAI is adequate for the purposes of certification 

and quality assurance . The first five functions of the 

TPAI are keyed to the research on effective instruction. 

Since the instructional practices subsumed under these 

functions are expected of the competent teacher, they 

provide an adequate basis for making certification 

decisions. Although the last three functions of the TPAI 

are less directly related to instruction, they do include 

additional teaching practices essential to teaching 

competency within a particular school context. Thus, the 

eight major functions of the TPAI provide a good basis for 

assessing teacher performance for the purpose of quality 

assurance , especially if the school system has delineated 



clearly its expectations for teachers with respect to 

functions 6-8. 

With respect to the final two purposes, career 
development and professional growth and development , the 
TPAS seems to be functioning reasonably well considering 
the CDP is only three years old. It needs to be 
strengthened and expanded in some areas, however, if these 
purposes are to be fully achieved. The remaining sections 
of this report will focus on how the TPAS can be 
strengthened to meet these purposes better. 



TPAI CRITERIA: TEACHING FUNCTIONS AND PRACTICES 

Are the major functions and inherent teaching practices 
which comprise the TPAI adequate? 

Although the major functions and teaching practices 
comprising the TPAI are adequate in many respects, they 
could be improved. Variations in rating consistency, 
especially on functions 6, 7, and 8, suggest the need for 
greater clarity. Also, a need to expand the functions and 
overall coverage of the instrument has been expressed by 
many teachers and administrators. 

Our first recommendation is that some reordering take 
place with respect to functions 6-8. Specifically, it is 
suggested that function 7 (Communicating within the 
Educational Environment) becomes function 6, and function 
6 (Facilitating Instruction) becomes function 7. 
Furthermore, the new function 6 should be renamed 
"Communication and Interaction in the Classroom". To 
accomplish this change, the current practice, "Teacher 
interacts effectively with students, coworkers, parents, 
and community", would be divided into the separate 
practices and placed in separate functions as noted below: 

6. Teacher interacts effectively with students. 

7.6 Teacher communicates effectively with coworkers, 
parents, and community. 
Finally, two practices from function 3 (3.3 & 3.4) should 
also be added to the new function 6, resulting in the 

7 



following new function: 

6. Communicates and Interacts in the Classroom 

6.1 Teacher speaks fluently and precisely. 

6.2 Teacher presents the lesson or instructional 
activity using concepts and language 
understandable to the students. 

6.3 Teacher interacts effectively with students. 

6.4 Teacher treats all students in a fair and 
equitable manner. 

As a result of this reordering, the first six functions of 
the TPAI would be related directly to instruction and 
observable in the classroom. All six would be used to 
make certification decisions. The final two functions 
would be related less directly to instruction and would 
not be directly observable in the classroom. 

Second the initial six functions of the TPAI dealing 
with instruction need to be reviewed to determine whether 
they clearly communicate to the teacher those critical 
aspects of effective teaching identified in the 
literature. During this process, it is recommended that 
function 3 (Instructional Presentation) be subdivided into 
a series of more specific functions as indicated below. 
Each of these new functions would include a set of 
specific teaching practices drawn in large part from 
current function 3. Additional teaching practices for 
this function could be drawn from the Connecticut 

8 



Competency Instrument. 

o Initiation of Instruction 
o Motivation of Students 

o Management of Routines and Transitions 
o Presentation of Accurate and Appropriate Content 
o Provision of Closure 

If this recommendation is adopted, the resultant 
version of the TPAI would consist of ten rather than six 
functions dealing with instruction which could be assessed 
through classroom observation. These functions would 
include the following: 

o Management of Instructional Time 
o Management of Student Behavior 
o Initiation of Instruction 
o Motivation of Students 

o Presentation of Accurate and Appropriate Content 
o Management of Routines and Transitions 
o Instructional Monitoring of Student Performance 
o Instructional Feedback 

o Communication and Interaction in the Classroom 
o Provision of Closure 
This expanded number of functions dealing with instruction 
would strengthen the validity and reliability of the TPAI. 
Validity would be enhanced since ten functions sample more 
comprehensively the dimensions of effective teaching 
presented in the literature (Good and Brophy, 1987; 



Wittrock, 1986) . Reliability would be enhanced since the 

TPAI could now measure more aspects of effective teaching 

performance in a highly focused manner. 

It is foreseen that the recommendation to include a 

function for Presenting Accurate and Appropriate Content 

in the TPAI will be questioned, since an argument has been 

developed for not evaluating content knowledge through the 

appraisal process (see letter of 2/15/88 to Principals and 

Teachers from R.D. Boyd). We do not intend to imply a 

deficit model, as suggested in the above letter, but 

rather, to include a dimension of instruction which is 

essential to the teacher appraisal process. To omit a 

reference to this function could prevent a principal from 

dealing with inadequate treatment of content by a teacher. 

In appraising teaching performance with respect to this 

function, the evaluator would focus on at least the 

following practices: 

o Teacher presents content without error or 
misinformation . 

o Teacher presents content which is developmental ly 
appropriate for the class 

In addition to expanding the number of functions 

related directly to instruction, the reliability and 

validity of the TPAI could be strengthened through a 

review of the teaching practices included in each 

function. With respect to validity, it is important that 

these practices reflect the current literature on 



10 



effective teaching. While the practices included in the 
TPAI are consistent with the process-product research, the 
direct instruction model, and the six-step lesson plan, it 
would be helpful to consider other well-researched and 
effective models of teaching, such as the coaching, 
modeling, cooperative, and mastery learning models, when 
identifying teaching practices to be included within each 
function. While parsimony is always an important 
consideration when selecting teaching practices for a 
particular function, it should not preclude adding 
practices which enhance the validity of the TPAI. We also 
recommend adding to Function 2 the following practice — 
"teacher reinforces appropriate behavior." Allowing this 
practice to become 2.6 would increase the validity of the 
Management of Student Behavior function. 

The reliability of the TPAI is enhanced by the extent 
to which the teaching practices for each function are 
stated clearly and thus are interpreted in the same 
manner. For example, the Management of Student Behavior 
Function includes the practice — "Teacher stops 
inappropriate behavior promptly and consistently, yet 
maintains the dignity of the student." This practice 
could be interpreted differently by various observers. 
Some might expect the teacher to actually stop the 
behavior, while others might expect the teacher to deal 
with the problems appropriately, which could include 

11 



ignoring the behavior. The differing interpretations of 
this practice would contribute to the unreliability of 
rating teachers on this function. 

The intent here is not to imply that there are 
substantial gaps between the literature on effective 
teaching and the practices included in the TPAI, nor to 
create the impression that there is considerable ambiguity 
in the manner in which these practices are described. For 
the most part there is good match between the literature 
on effective teaching and the practices in the TPAI. 
Also, these practices tend to be described clearly. The 
point being made here is that the TPAI is not a static 
instrument. It must be reviewed periodically to insure 
that the practices included are consistent with the 
current knowledge base regarding effective teaching, and 
that these practices are being interpreted in a similar 
manner by the parties using the TPAI and affected by its 
outcomes. The need for such review provides a prime 
opportunity to involve teachers and principals in any 
revisions of the appraisal instrument. A committee 
structure could be organized whereby teachers and 
administrators are actively involved in the process of 
updating the TPAI. Through this approach, the quality of 
the TPAI would be sharpened while building ownership among 
school practitioners in the revised appraisal process. 

To this point, the focus has been on those aspects of 

12 



the TPAI directly related to instruction. As a result of 
the revisions suggested, two TPAI functions remain which 
are related less directly to instruction. These are 
o Facilitating Instruction 
o Performing Non-Instructional Duties 
It is clear that there is considerable ambiguity as to how 
teachers should be evaluated with respect to these 
functions. While it is important for the DPI to provide 
some guidance in this regard, it also is critical that 
school districts take leadership in identifying those 
practices their teachers are expected to pursue with 
respect to these functions. To a large extent LEAs are at 
liberty to customize these two functions to their local 
needs. In doing so, practices may be included which a.) 
add a dimension of ingenuity and creativity to the 
appraisal process, and b.) begin to address the affective 
component of the educational process. To date these areas 
have not been addressed sufficiently through the appraisal 
process. 

Those who undertake the further development of these 
functions at either the local or state level should 
realize that only evaluation criteria derived from one of 
two sources will stand up to appeal or legal challenge: 
(a) research findings or (b) consensus agreement of those 
subject to them. Since there is little research to 
support many practices which could be included under these 

13 



two functions, evidence should be obtained on either a 
local or statewide basis that most teachers agree with the 
inclusion of these practices (French and Malo, 1987). 

This section has focused on a discussion of 
considerations in reordering and expanding the functions 
currently included in the TPAI . The direction advocated 
would strengthen the reliability and validity of the TPAI 
for naking certification and career status I and II 
decisions. Later in this report a series of alternatives 
for making career status III decisions will be addressed. 
One of these alternatives is simply to use the TPAI. If 
the TPAI is to be used in this manner, additional 
functions would need to be added. Such functions must be 
consistent with an expanded view of teaching applicable at 
the career status III level. If this view encompasses the 
role of teacher as instructional leader, then the 
instructional leadership functions of the PPAI might be 
adapted to the TPAI for use in making career status III 
decisions. More specifically, the functions to be added 
to the TPAI might include the following: 

Develops a comprehensive instructional plan 
Implements the comprehensive instructional plan 

fnSt^i:??^ ^?^ delivery of the comprehensive 
instructional plan 

° fn^^""^^?^ ^^f ^"'P^''^ °^ ^^^ comprehensive 
instructional plan 

The practices included in the PPAI for each of these 



14 



functions would also need to be reviewed and adopted for 
use in the expanded version of the TPAI . 

In summary, the functions and inherent practices that 
currently comprise the TPAI tend to be supported by the 
literature on effective teaching. This is true 
particularly for the first five functions which are 
related directly to instruction. To enhance the 
reliability and validity of the TPAI in making 
certification and career status I and II decisions it is 
recommended that a.) the functions of this instrument be 
reordered, b.) some functions be expanded, and c.) the 
practices within each function be reviewed in light of the 
most recent literature on models of teaching and effective 
instruction. If these recommendations are followed, the 
revised version of the TPAI would assess the functions 
noted below. 

Functions which can be assessed through classroom 
observation: 

*- Management of Instructional Time 

Initiation of Instruction 

Presentation of Accurate and Appropriate Content 

Motivation of Students 

Communication and Interaction in the Classroom 

Management of Routines and Transitions 
*- Management of Student Behavior 
*- Instructional Monitoring of Student Performance 

15 



*- Instructional Feedback 
Provision of Closure 

Functions which would be assessed primarily in other 
ways and with considerable local specification of 
criteria: 

* Facilitating Instruction 

* Performing Non-Instructional Duties 



♦Functions included in the current TPAI 



16 



PROCEDURES AND INSTRUMENTS 

Are the procedures and instruments used for 
collecting, analyzing and interpreting data sufficient a.) 
for the development of fair and equitable personnel 
decisions and b.) for the improvement of instruction? 

To respond to the opening question, four sub- 
questions must be addressed: 

1. Are instruments used appropriate to the purposes 
of the system and the established criteria? Are 
they fair and equitable to all persons to whom 
they are applied? 

2. Are those administering the system (those 
observing and evaluating) adequately trained? 

3. Is the use of the system for summative purposes 
well coordinated and consistently conducted from 
one time and place to another? 

4. Are the instruments being used effectively in 
formative ways to assist teachers and improve 
instruction? 

Each of these four sub-questions will be treated 

separately. 

Are the instruments used appropriate to the purposes 
of the system and the established criteria? Are they fair 
and equitable to all persons to whom they are applied? 

The purposes to which TPAS results are applied include 

continuing certification, adequacy of continuing 

performance and determination of level of teaching 

performance as defined for career ladder placement. The 

eight functions and 38 related practices presently defined 

17 



within the TPAS (or any revised version thereof) 
constitute the criteria upon which instrumentation and 
data collection procedures must focus. At present, the 
actual instruments contained within the TPAS include the 
Teacher Performance Appraisal Instrument (TPAI) , rating. 
scale descriptors necessary to its use, and the 
Professional Development Plan Form (PDP-1) . These 
instruments are supported by two others created to 
systematize the collection of classroom observation data 
(the Formative Observation Data Instrument) (FODI) and 
their application to the evaluation criteria (Formative 
Observation Data Analysis) (FODA) . 

Furthermore, the 1985 legislation establishing the 
School Career Development Pilot Program specifies that 
multiple sources of information will be used to evaluate 
candidates for career ladder placement. It further 
specifies that candidates for career status II will submit 
for analysis a portfolio of work-related documents for 
analysis and that an interview with the candidate can be 
used as yet another source of data. 

It is clear that the instruments used as a basis for 
assessing performance for certification purposes (FODI, 
FODA, TPAI) are appropriate to that purpose. The first 
five teaching functions to be evaluated within the TPAS 
are strongly supported by effective teaching research and 



18 



are observable in the classroom setting.* The practices 
identified within those functions represent basic 
pedagogical skills which teachers should be able to 
demonstrate in their classrooms as a condition of 
licensure. At this time, it is these first five functions 
and practices which are evaluated and used as a basis for 
recommending the issuance of a continuing certificate in 
North Carolina. The combination of the FODI, FODA and 
TPAI provides an appropriate means of gathering and 
analyzing teacher performance data pertinent to this 
decision for most, if not all, teachers. However, the 
validity and reliability of the decision process can be 
improved by modifying TPAS criteria as recommended. 

Obviously, the annual performance evaluations required 
of all teachers in North Carolina are perceived to be the 
primary means of a.) assuring that the quality of a 
teacher's performance remains constant at or above 
standard as defined within the structure of the state- 
approved evaluation system (TPAS) and b.) providing 
professional development assistance to both those who need 
improvement and those who desire to continually improve 
their teaching performance. Again, the instruments 
primarily used for these purposes are the FODI, FODA and 



*This statement will be even more true if 
modifications to the criteria suggested earlier in this 
report are made. 



19 



TPAI which are appropriate to assessment of performance 
of observable, instructional functions (such as current 
TPAS functions one through five) . However, these 
instruments alone are not sufficient to measure non- 
instructional, non-observable skills such as those 
embodied in the present TPAS functions 6, 7 and 8, 
although they do contribute some pertinent information. 
At the present time, there is little evidence of either 
state-developed or locally-developed instruments which 
have been adopted statewide to supplement the FODI , FODA 
and TPAI. There is evidence to suggest that some local 
school districts have developed and are using such 
instruments, but there has been no systematic statewide 
attempt to address this problem. 

The School Career Development Pilot Program has 
created a new set of teacher evaluation issues to be 
addressed, and logically, the TPAS is being modified and 
adapted to address some of those issues. Since the 
legislation and funding undergirding the Career 
Development Pilot Program is only three years old, both 
the State Department of Public Instruction and the sixteen 
local districts involved in pilot programs have had to 
confront and implement numerous policy and technical 
decisions in a very short amount of time. There has not 
yet been enough time to create all that needs to be 
created, to identify all the "bugs" in evaluation 

20 



instruments and procedures or to fine-tune and systematize 
the evaluation system. The efforts of all those who have 
been involved in program development and implementation 
are to be highly commended and applauded. 

As stated earlier in this document, some of the 
purposes defined for the evaluation system to be used in 
Career Development Program decisions are to 

o encourage differentiation of all teachers and 
administrators (differentiate among levels of 
performance — as currently defined career status I 
and II) 

o encourage recognition of high quality teachers 
(award career status II) 

o improve the quality of classroom instruction 
It does not appear that the instruments currently 
incorporated in the TPAS are fully adequate for achieving 
all of these purposes. However, this statement should not 
be interpreted to mean that the present TPAS structure and 
instruments should be discarded. Modification and 
supplementation, not discard, are needed, and those 
persons in the Department of Public Instruction who are 
charged with development and implementation of the program 
are already aware of and have clearly stated a number of 
those needs. 

As noted previously, the current instruments (FODI, 
FODA, TPAI) are not alone sufficient to assess performance 

21 



in all functions of the TPAS. Functions 6, 7 and 8, in 
either their present or modified form, require the 
collection and rating of information from evaluators 
either in the form of work products or oral descriptions 
of activities (data sources acknowledged in the 1985 
legislation) . 

Examination of the current instruments in TPAS (FODI, 
FODA, TPAI and the related rating scale PDP-1) reveals 
that there are refinements which could be made in each 
instrument to enhance its capacity to a.) differentiate 
between career status I and II performance and b.) improve 
the quality of classroom instruction (the teacher's growth 
toward mastery of the desired teaching functions and 
practices) . In the context of these analyses of the 
evaluation task and present and "missing" instruments, 
several recommendations are offered. 
Formative Observation Data Instrument (FODI) 

The Formative Observation Data Instrument provides 
descriptive information about the teacher's classroom 
performance. Since it is an instrument which requires the 
observer to simply record what he/she actually sees and 
hears without rendering judgment, it appears to have the 
capacity to treat all who are observed objectively, fairly 
and equitably. However, other forms of descriptive data 
could be used to supplement these scripts in the effort to 
provide an evaluatee specific information which will 

22 



enable him/her to improve performance and at the same time 
sharpen the ability of evaluators to discern and document 
differences in performance. For example, mapping of 
student and teacher movement in the classroom or making 
charts/scans of student on-task and off-task behavior are 
powerful tools for pinpointing specific teacher strengths 
and weaknesses related to certain of the TPAS criteria. 
Obviously, an observer cannot record several kinds of 
information simultaneously, particularly when scripting is 
used. But during a 50-minute observation, a pattern of 
different types of data gathering can be easily instituted 
without losing major teacher or student behaviors. For 
instance, one night employ a sequence of descriptive data- 
gathering activities such as the following: 

Script classroom events (FODI) - 10 minutes 
Map teacher/student movement - 3 minutes 
Scan student off-task behavior - 2 minutes 
Script classroom events (FODI) - 10 minutes 
Map teacher/student movement - 3 minutes 
Scan student off-task behavior - 2 minutes 
We would recommend that consideration be given to 
supplementing the script data collected with the FODI, but 
we are not prepared to say exactly what additional 
instrumentation or procedures should be used or even that 
they absolutely should be used. There are a number of 
technical, procedural and policy issues which will need to 

23 



be addressed in such considerations. 

In keeping with some of the recommended changes in 
TPAS criteria presented earlier in this document, we would 
recommend that teachers be provided copies of all 
descriptive data (FODI's and any other descriptive 
instruments used) in advance of the post-observation 
conference with an observer. This procedure would allow 
the evaluatee to do some self-evaluation and should make 
the post-observation conference more productive, 
especially in regard to professional development planning. 
Formative Observation Data Analysis (FODA) 

Although the Formative Observation Data Analysis 
instrument uses the word formative in its title, it 
embodies both formative and summative dimensions. One 
purpose of the instrument clearly is to summarize and 
synthesize the raw data contained in the FODI . However, 
users also are instructed to "use statements which 
accurately reflect the quality of performance documented 
by your raw data." In purely formative evaluation, the 
evaluator and evaluatee generally complete some kind of 
data analysis and professional development plan mutually 
during the post-observation conference. The FODA seems 
designed to fulfill this function and to serve as the 
middle step (hinge) between observation data and summative 
evaluation (performance judgments by evaluators) . Since 
both functions must be served in fulfilling the multiple 

24 



uses of the TPAI and its related instruments, there is 
probably no good alternative to the tensions inherent in 
this instrument. However, both evaluators and evaluatees 
should be aware of those tensions and every care should be 
taken to address both the formative and summative 
dimensions of the instrument as professionally as 
possible. 

The FODA contains no space for an evaluatee to respond 
to the observations and quality assessments made by an 
evaluator. While the evaluatee is invited to respond on a 
separate sheet of paper, providing space on the form would 
signal greater interest in teacher input. 

The dual purposes of the FODA require that every 
evaluator understand the instrument and be well trained in 
completing and using it. The Department of Public 
Instruction has made available both initial and "booster" 
training in FODA completion and several local school 
districts have further emphasized this component of the 
evaluation process. However, there does not appear to be 
sufficient training for evaluators (principals and 
observer evaluators) in conferencing and in using the FODI 
and FODA in the conference setting. 
Teacher Performance Appraisal Instrument and Rating Scale 

The Teacher Performance Appraisal Instrument is a 
summative instrument in which data generated from other 
instruments and sources are aggregated and translated into 

25 



a set of performance ratings. The primary vehicle for 
this translation is the six-point rating scale. 

The panel is aware that there have been requests that 
the current six-point rating scale be reduced to three 
points. This reduction should not be made because it 
would lessen greatly the capacity of the TPAI to 
differentiate among levels of performance - an important 
intended outcome of the Pilot Career Development Program. 
The use of a three-point scale would reduce the instrument 
to one which is capable of depicting only minimal 
competence (satisfactory or unsatisfactory performance) . 

The intervals between points on the rating scale are 
unequal, that is, the distance between critical points 2 
and 3 is greater than the distance between points 3-4, 4- 
5, 5-6. This feature was consciously designed into the 
scale. However, it may be contributing somewhat to lack 
of reliability among raters. Experimentation with a 
multi-stage decision-making process has been conducted, 
and the panel recommends that this process be implemented 
statewide. In essence, this recommendation means that an 
evaluator will first decide whether or not the teacher's 
overall performance on a function is at standard 
(adequate) or below standard (inadequate). If performance 
is judged to be at least at standard, the next decision 
will be whether the performance is at standard, or 
somewhere above standard. If the performance is somewhere 

26 



above standard, the evaluator must next decide whether the 
rating should be a 4 (above standard) or a 5 or 6. In 
reality, the standards currently operable in determining 
career status through level II (2 ratings at least 5, 5 
ratings at least 4, 1 rating at least 3) dictate that 
evaluators must be especially sensitive to the 
distinctions between 2 and 3, 3 and 4, and 4 and 5. If 
the standards are changed, or if career status III is 
added to the Career Development Program with higher 
performance standards required, our recommended procedure 
for use of the scale will also need to be adjusted. 

Some attention should be given to further 
clarification and elaboration of rating scale definitions 
and particularly to refining and extending the training of 
evaluators in recognizing distinctions in performance 
data. Both activities will help to improve rater 
reliability. 
Professional Development Plan fPDP) 

The Professional Development Plan form is the major 
instrument for targeting and directing professional growth 
activities. The form itself is well designed. However, 
it should be recognized that the plan targets growth only 
within the context of the TPAS standards; i.e., growth 
toward competence. While this type of growth is needed 
and especially important in the formative evaluation of 
individuals seeking continuing certification and in 



27 



continuing evaluations to assure quality of basic 
performance, teachers who are already well above standard 
or superior in their performance of basic teaching skills 
need to plan professional development beyond competence. 
The Professional Development Plan form could be slightly 
modified for use in Career Development evaluations to 
reflect this perspective. 
Need for Additional Instruments 

As indicated in previous statements, at least one 
additional instrument is needed — a form and procedure for 
recording and analyzing portfolio contents which can be 
used statewide to bring some consistency to the analysis 
of these required materials. Once this instrument is 
developed or adopted from among instruments already 
existent in local school districts, evaluators will need 
to be trained in its use. 

A second instrument should be considered — an interview 
format. Such instruments will need to be developed for 
use in the evaluations of principals and special groups 
(e.g., counselors, school psychologists, etc.), and a 
teacher interview would be most useful in collecting and 
interpreting data pertinent to assessment of those 
functions and practices which go beyond the classroom. If 
a teacher interview is constructed, it is recommended that 
it contain both a structured section (standard questions 
to be asked of all teachers) and an unstructured section 

28 



(a section in which a teacher can provide information 
pertinent to evaluation criteria which he/she feels to be 
important) . A time for unstructured input will help to 
involve teachers more fully in the evaluation process and 
overcome their concerns that they have little opportunity 
to provide data which they consider to be important. 
Obviously, evaluators will need to be trained to conduct 
both parts of the interview, record pertinent data and 
interpret them. 

Some examples of items or instruments which could be 
adopted to use for either the portfolio analysis or an 
interview protocol have been developed in the Pilot Career 
Development Program districts. For example, Harnett 
County has a portfolio analysis form and a questionnaire 
focused on Functions 6, 7 and 8 which could be used or 
modified for use in an interview setting. Teaching 
practices which lend themselves to data gathering through 
the interview process are 6.1, 6.2, 6.5, and 7.2. 
Portfolio contents should provide information pertinent to 
current practices 6.1, 6.3, 6.4 and 8.1-8.3 if appropriate 
documents are required for portfolio inclusion. 

Near the beginning of this section, we stated that the 
instruments now in use appear to be appropriate to most, 
if not all, teachers to whom they are being applied. In 
Tennessee's Career Ladder Program and in evaluation 
programs in several school districts (e.g., Warren, Ohio; 

29 



Youngstown, Ohio; Dundee, Illinois) , it has been found 
that some instruments (and even a few criteria) used to 
measure the performance of teachers in typical classrooin 
settings do not lend themselves to certain special 
education settings where teachers work primarily one-on- 
one with severely handicapped students, and they do not 
yield sufficient, appropriate data upon which to base 
evaluative decisions. It should be noted that the same 
problem does not exist in resource classes or other 
settings where the student population is composed of 
mildly handicapped learners. These findings in other 
settings lead us to recommend that careful monitoring of 
the evaluation criteria, procedures and instruments in 
these particular unique settings be conducted, and that 
appropriate changes be made if evaluation data and 
evaluator perceptions suggest that they are needed. 
Training of Evaluators 

Are those administering the system (those observing 
and evaluating) adequately trained? 

In applying the Teacher Performance Appraisal System 

statewide for the several purposes previously discussed, 

there are essentially four groups of people who are 

evaluating teachers: principals, mentor teachers, 

observer/evaluators and "others" who may serve on support 

teams in some school districts. At present, there are 

four types of training which impacted upon either the 

formative or summative dimensions of the Teacher 

30 



Performance Appraisal System: Effective Teaching 
Training, Teacher Performance Appraisal Training, 
Professional Development Plans and Mentor/Support Team 
Training. All of these training programs operate on the 
turnkey principle; i.e., the Department of Public 
Instruction trains trainers for local school districts who 
in turn train appropriate others. 

The Department of Public Instruction with the 
assistance of numerous educators from local school 
districts and higher education institutions has done an 
excellent job in developing these training programs and 
related materials and making the training available, 
especially when one considers they have had to "fly the 
plane while still building it." However, the review panel 
has found some problems which need to be addressed if 
evaluation processes and decisions are to be highly valid 
and reliable: 

1. Although the vast majority of teachers and 
principals in the state have experienced 
Effective Teaching Training or its equivalent, it 
is possible that there are evaluators within the 
four identified groups who have not been through 
this experience or mastered this knowledge which 
is essential to understanding the TPAS functions 
and practices and to coaching teachers for 
improvement in these performance areas. 

31 



2. Not all r.entors and others who evaluate (and 

perhaps even principals and assistant principals) 
have been required to take the Teacher 
Performance Appraisal Training and/or demonstrate 
competence in teacher appraisal. 

3. It is possible that there are persons evaluating 
who have not had the training in Professional 
Development Planning, 

4. All who evaluate must be prepared to give 
feedback, coach and support evaluators, but may 
not have had the Mentor/Support Team Training. 

5. A review of training materials in the various 
training programs indicates that only the 
Mentor/Support Team Training emphasizes the 
development of conferencing skills, yet every 
evaluator must be effective in conferencing if 
formative evaluation (evaluation for improvement) 
is truly to take place. 

It cannot be assumed that any evaluator— principal, 
central office administrator, higher education faculty 
member or experienced teacher-has the skills necessary to 
carry out the many evaluation-related activities. 
However, it is possible that individuals have gained those 
skills in settings other than the training programs 
developed and implemented by the Department of Public 
instruction. Therefore, the review panel recommends that 

32 



a process be developed for certifying these competencies 
of all persons who evaluate teachers in a.) the research 
on effective teaching, b.) teacher appraisal using the 
TPAS structure and instruments, professional development 
planning, and effective conferencing. Mentors should also 
be required to demonstrate competence in the additional 
knowledge and skills required of a mentor. Persons who do 
not meet the certification requirements should not be 
allowed to evaluate until they take appropriate training 
and can meet them. The implementation of this 
recommendation would do a great deal to improve evaluator 
reliability which at present is suspect in many cases 
based upon the data we have reviewed. It would also 
greatly improve the feedback, conferencing, and 
professional development planning processes which have 
received criticism from teachers. 

If consensus among evaluators is to be widely used as 
a procedure for arriving at TPAI ratings and identifying 
teacher strengths and needs for improvement as we 
recommend elsewhere, an additional type of training for 
evaluators (all groups) will be needed — training in 
achieving data-based consensus. It is very easy in a 
consensus process for one person's personality to dominate 
and for ultimate decisions to be based on feelings or 
impressions rather than the data collected by the 
consensus participants. Several models for consensus 

33 



training for evaluation purposes already exist. At least 
one nearby program which has been reasonably successful is 
that developed by French, Malo and Chalky for the 
Tennessee Career Ladder Program. 
Use for Summative Purposes 

Is the use of the TPAI sufficiently well coordinated 
and conducted with adequate consistency from one time and 
place to another to serve the summative purposes for which 
It is, in part, intended? 

The system it is designed to serve is statewide in its 
purposes, i.e. teacher certification and career ladder 
placement. It is part of the total reform effort of the 
legislature. Many of the main features are actually 
prescribed in the legislation itself. Of necessity, then, 
much of the major direction and responsibility for running 
this program does and probably must come from Raleigh and 
the DPI in particular. 

The system for which the instruments are being used is 
still evolving, with career ladder II decisions having 
been made for the first time only this past year; career 
level III criteria and accompanying instrumentation, have 
not yet been prescribed. Even in the pilot districts, the 
tempo of career development activity has been stepped up 
considerably over what it will eventually be, if the 
program is permanently installed. We recognize that all 
procedures are not yet fully developed or operationally 
consistent across and between districts. The various DPI 
reports, newsletters and memoranda, and coordinator and 

34 



steering committee meetings are serving important 
coordinating and continuing development functions as the 
system evolves. The DPI staff, along with district 
administrators, coordinators, and evaluators are to be 
commended for the several high level efforts being made to 
continue to study, develop, and refine procedures. 

Recognizing considerable effort to coordinate all this 
activity, we also note a considerable amount of 
inconsistency across and between the pilot districts in 
how the instrument is used to gather and analyze data and 
in how final summative ratings are made. Disparities are 
evident from one district to another, and one school to 
another, in the percentages of teachers at or below 
standard performance as well as percentages of teachers 
qualifying for career level I and career level II. While 
such differences may reflect real differences in the 
overall quality of teaching performance from one place to 
another, we suspect they probably indicate differences 
from one evaluator to another in a.) the interpretation of 
scale descriptors, b.) the use of recorded information, 
and c.) the sources of data being considered. This latter 
is particularly likely for current functions 6, 7, and 8 
which are less observable in the classroom. We believe 
there is a direct correlation between the amount and 
quality of the training of evaluators and the quality of 
their evaluations. 



35 



Substantial differences are noted also in a.) who is 
involved in achieving sununative ratings, b.) how observer 
evaluators perform their functions, c.) when teachers are 
observed during the year, and d.) how the appeals process 
functions. 

The amount of variation in all these procedures, while 
substantial in some instances, is understandable given the 
early developmental stage of the program. Furthermore, 
the coordination efforts are not only tending to reduce 
the discrepancies where they exist (e.g. most districts 
now require consensus judgment by OEs and principals in 
making summative ratings) but there is healthy development 
of specific operational procedures at the local district 
levels which tend to feed back into the overall plan and 
help refine the state system. An appropriate balance 
between state and local direction is important to the 
success of such an effort. 

To improve the procedures for collecting and analyzing 
information for making fair and equitable personnel 
decisions, we urge consideration of the following 
recommendations : 

1. Restrict the decision-making use of the 

instrument to determination of continuing 
certification, career ladder I and career ladder 
II decisions. If career ladder III criteria, 
when developed, should require a distinction be 

36 



made on the quality of teaching between level II 
and level III teachers, we do not believe this 
instrument, even with refinements, would be 
adequate. Other sources of data would be needed. 
If level III differs from level II primarily 
because of other criteria, then other 
instrumentation will be needed also. 

2. Intensify training requirements for all 
evaluators including frequent tests of 
evaluators' ability to use the instruments 
accurately. All individuals who participate as 
evaluators should be certified for that role by 
passing a performance-based test of their ability 
to evaluate accurately against a standard. 
Holding such a certificate should be prerequisite 
to serving as a summative evaluator. 

3. Require joint participation of OEs, principals 
and other evaluators who are involved in the data 
collection process in the final summative 
judgment. Most of the districts already seem to 
be doing this. The use of multiple judges 
strengthens the validity of the rating process. 
Also OEs as a group are the most highly skilled 
evaluators since this is their primary function. 
Their judgment should be at least as great an 
influence on the final decision as that of the 

37 



principals. 

Experiment next year with observing teachers 
three or four days in a row rather than their 
teaching of single lessons at different times of 
the year. We believe that a more complete 
description and analysis of total teaching 
performance might be gained by seeing how 
instruction is conducted on consecutive days. 
The continuity of instruction from one day to 
another could be seen as well as how reteaching 
is handled for those pupils who fail to learn on 
a first occasion. Teachers would have greater 
opportunity than at present to demonstrate a 
variety of lesson formats and show how creative 
and flexible they can be in adapting generic 
skills to a variety of ways and settings. If the 
career ladder becomes permanently installed, one 
requirement for level II might be to be observed 
teaching the same class for four consecutive days 
during at least one of the two years before 
promotion. We believe this would provide a more 
rigorous and valid performance test than the 
current schedule of unrelated single lessons. 
Create panels of administrators and teachers to 
review and rate examples of FODIs and FODAs for 
comparability. Encourage increasing exchange of 

38 



principals and other evaluators across school and 
district lines to conduct joint observations, 
gather data, and share analyses. Some districts 
have been doing this on an informal basis. The 
more that is done, the more likely it is that 
consistency of instrument interpretation and use 
will increase. 
6. Encourage research study of rating reliability 
and validity in local districts as well as 
statewide as important formative activities for 
evaluators. OEs and principal evaluators should 
be independently reading and rating the same 
data. Comparisons should be made between data 
collected at one time on a teacher and that 
collected at another to see how much stability 
there is in ratings of the same persons over time 
and setting. 

Validity studies of two types are needed: 
(a) Achievement gain scores of students taught by 
teachers rated "at standard" should be compared 
with those of students taught by teachers rated 5 
and 6. (b) Through studies of the state test 
data base, student gains in the pilot districts 
should be compared with those in comparable non- 
pilot districts. 
7. To provide greater research capability, we urge 

39 



involvement of university faculty and graduate 

students in the study of evaluator consistency 

and practice in the pilot effort. There is a 

wealth of information available for conducting 

such studies and too few resources in-house to 

get it done. Modest stipends for graduate 

dissertation research could provide a wealth of 

useful, valuable information regarding the 

system. 

8. Provide evaluators with word processors and 

appropriate training as needed to assist them in 

writing FODIs and FODAs. The amount of time 

spent rewriting FODIs and preparing FODAs could 

be reduced substantially if evaluators had such 

equipment and were well trained in its use. The 

quality of these records should improve as well. 

Use for Formative Purposes 

Are the instruments being used effectively in 
formative ways to assist teachers and improve instruction? 

Formative assistance of a formal nature that is tied 

to the TPAI typically takes the form of OE-led workshops 

for teachers and other staff development activities 

focused on one or more of the functions. Those teachers 

who have participated in these workshops are typically 

less anxious and more supportive of the evaluation process 

than other teachers. The more valuable activities seem to 

be those that are well planned and conducted and involve 

40 



participants in performing observations of actual teaching 
or videotaped lessons. 

Regarding individual formative help, we heard or saw 
many teacher reports which indicated that the use of the 
TPAI had sharpened skills and improved teaching. The 
emphasis on specific teaching functions and the 
specificity of feedback, while often stressful, was also 
quite helpful. One critical problem, however, was the 
need for much greater follow-up assistance by OEs, 
especially for teachers demonstrating skill deficiency. 
Evaluators — both OEs and principals — were just too busy, 
to provide the extra help needed. When a teacher is not 
up to standard or is having difficulty improving 
performance, OEs typically notify principals of the 
presence of difficulties but have little time themselves 
to provide individualized help and follow-up assistance. 

There is ample evidence that the current ratio of 96 
teachers per observer-evaluator is too high if the 
observer-evaluators are to provide the follow-through 
necessary to ensure professional improvement. At an 
average of two observations a day, with related FODI, 
FODA, and conference obligations, OEs do not have enough 
time to conduct two observations of each teacher much less 
provide follow-up assistance. We recommend a maximum 
caseload per observer-evaluator of 48 teachers. 

Increasing the number of OEs would also permit greater 

41 



differentiation of evaluation assignments than is now 
possible to take advantage of their teaching 
specializations by grade-level and subject matter. Such 
differentiation would help in evaluating our proposed 
content function in the TPAI . 

Attention also needs to be directed in some districts 
toward specific instructional strategies and resources 
available to improve performance on specific functions. 
Other types of observational data such as checklists and 
low inference coding systems (see Good and Brophy, 1987; 
McNergney and Carrier, 1981; Brandt, 1981) could be used 
to highlight differences in lesson videotapes on such 
variables as time on tasks, reinforcement patterns, and 
student participation rates. 

Professional development is not a simple phenomenon. 
Teachers grow in at least three different ways. They grow 
toward competence, they amplify on their strengths (areas 
of competence) , and they grow beyond competence, 
developing unique talents and perspectives. The TPAI, 
PPAI, and PDP are suited to the first two types of 
professional development, but probably not the third. 

Why, for example, is the TPAI unlikely to promote 
growth beyond competence? The TPAI currently is based on 
a set of essential functions that must be mastered by all 
competent teachers. These functions acknowledge the 
similarities that cut across teachers of all subjects and 

42 



age-groups. Growth beyond competence, however, must take 
account of each teacher's unique circumstances, including 
his/her prior experiences, strengths, subject matter area, 
and students. The TPAI is not well-equipped to handle 
these areas of individual difference. 

In the context of the Career Development Program, the 
TPAI, PPAI and PDP seem to be most appropriate for 
decisions regarding career levels I and II. To the extent 
that career level III involves growth beyond competence, 
it is unlikely to be well-served by these instruments. 

While it was not our charge to comment on career level 
III, we feel that four alternative strategies could be 
employed. 

First, the same set of teaching functions that 
constitute the TPAI (or PPAI) could be used, but the level 
of expected performance for each function could be raised 
substantially above career level II expectations. It is 
our judgment, however, that it would be difficult to 
operationalize and observe substantially greater 
performance in many of the existing functions. There are 
practical limits, for instance, to on-task behavior and 
effective classroom management. 

A second strategy would be to develop a set of master 
teacher functions for career level III that totally differ 
from those used in the TPAI. These functions might look a 
lot like the leadership functions that now constitute the 

43 



PPAI. 

A third strategy would permit each aspiring career 
level III teacher to negotiate unique professional 
development goals, based on such factors as their 
assignment, experience, students, and local setting. 
Accomplishment of these goals, based on external review 
and assessment, would result in movement from career level 
II to career level III. The Professional Development 
Plan, if it could be de-coupled from the current or 
proposed set of basic teaching functions, might be well- 
suited to this third strategy. 

A fourth strategy would involve some combination of 
the preceding strategies. 

A critical question that must be addressed before a 
plan for career level III can be developed is "To what 
extent must all candidates for career level III complete 
identical or comparable activities?" Our feeling is that 
the more veteran teachers are required to conform to 
similar standards, the less likely they will be to engage 
in meaningful professional development beyond competence. 



44 



APPRAISAL OF PRINCIPALS AND OTHERS 

In addition to the TPAI , the review panel was asked to 
evaluate the PPAI . Our assessment in this regard is much 
more tentative, given the lack of supporting documentation 
and field data and the early stage of the instrument's 
development. We feel that our reactions can best be 
expressed as a series of questions: 

1. For what purposes is the PPAI intended? 

We assume that the PPAI and TPAI are intended to serve the 
same purposes. 

2. Why are the performance functions for the PPAI 
more comprehensive than those for the TPAI? 

The functions for the TPAI are not intended to encompass 
all dimensions of the teacher's job. For example, the 
teacher's handling of curriculum content and affective 
characteristics were not covered originally. In the case 
of the PPAI, however, an effort apparently was made to 
include all functions of the principal. This fact may 
have implications for PPAI-based decisions involving 
principals below career level II. 

3. Upon what data sources are those using the PPAI 
to rely? 

We could find few guidelines or suggestions concerning 
data sources to be tapped by those completing the PPAI. 
Is it admissible, for example, to utilize student 
achievement data and teacher input? What role are 

45 



observations of principals to play? Are the FODI and FODA 
to be employed following observations? 

The review panel feels that the list of PPAI functions 
is reasonably well -documented and comprehensive. The PPAI 
functions are representative of our current knowledge of 
the job expectations of principals, including 
instructional leadership. Our uncertainty rests with how 
the PPAI is to be used for particular purposes. 

One further issue must be noted. If the principal's 
role in teacher evaluation is crucial enough to merit the 
attention it has received in North Carolina and elsewhere, 
it makes sense to accord the teacher evaluation function 
in the PPAI greater weight than other functions. 
Principals invariably have more to do than time available 
(Duke, 1987) . By stressing the importance of teacher 
evaluation through the principal evaluation process, those 
who supervise principals will be able to ensure that the 
intent of the TPAS will be realized. 

Regarding the evaluation of other certificated 
personnel, the review panel feels uneasy about making 
evaluative statements about the job descriptions and 
performance appraisal instruments for the following 
personnel: 

o Assistant Principals 

o School Counselors 

o Media Coordinators 

46 



o School Psychologists 

o School Social Workers 

o Speech-Language Specialists 

o Coordinators 

o Directors 

o General Supervisors 

o Observer/Evaluators 

Our uneasiness is based on lack of technical expertise 
in r.any of the areas listed above and lack of supporting 
materials and field data on the use of appraisal 
instruments for these roles. We can offer, however, a few 
general observations — 

o Parallel structure ; 

We are unclear why certain functions, such as 
"keeps own professional competence current," are 
used for certain roles, but not others. 
Presumably, functions of this kind are 
appropriate for all roles. In addition, there is 
a need to make uniform the wording of similar 
functions. For example, assistant principals are 
expected to "keep own professional competence 
current," counselors to "take part in 
professional development activities to improve 
knowledge and skills," and general supervisors to 
"upgrade own professional competence." 



47 



Technical Expertise; 

It is not clear whether the persons evaluating 
these personnel always possess expertise in the 
particular area of responsibility. Will 
provisions be made, particularly in the case of 
smaller units, for training and/or use of outside 
evaluators? 
Selective evaluation ; 

In the guidelines for the evaluation of school 
psychologists, counselors, and social workers, 
provision is made for supervisor/ evaluators and 
the individual to be evaluated to "mutually 
define the job functions and competencies to be 
evaluated at the beginning of each year." 
Furthermore, functions and/or competencies "may 
be modified to meet the specified role of an 
individual. . . in the local education agency." 
We wonder why similar latitude and discretion are 
not accorded other personnel, including teachers 
and principals. 



48 



A FINAL RECOMMENDATION 

If the recomnendations we have made are to be acted on 
this summer, the Department of Public Instruction will 
need two kinds of extra assistance: (1) Additional 
personnel should be hired for the summer months to rewrite 
materials, forms and instruments. Some of the district 
leadership familiar with the TPAS, i.e. OEs and 
coordinators especially, ought to be very capable for this 
task. (2) Experts in structured interview and portfolio 
design ought to be added to the staff temporarily to 
provide technical assistance on the new instrumentation. 

The primary reason for this addition of personnel is 
the heavy work load of the DPI staff responsible for this 
program. They are already working overtime providing 
coordinated direction of the program. The changes we are 
recommending, while not altering the fundamental character 
of the instruments, mean a great deal of extra work in the 
short term to accomplish a smooth transformation to the 
new materials and directives that will be needed. 



49 



REFERENCES 

Brandt, Richard M. Studying Behavior in Natural Setting s 

Washington: University Press of America, 1981. 
Duke, Daniel L. School L eadership and Instructional 

Improvement. New York: Random House, 1987. 
French, Russell and Malo, George, "Developing performance 
evaluation systems for career ladders, Thresholds . 
Vol. 8, No. 1, February 1987, 37-40. 
Good, Thomas L. and Brophy, Jere E. Looking in 

Classrooms. (Fourth Edition) . New York: Harper & 
Row, 1987. 
McNergney, Robert F. and Carrier, Carol A. Teacher 

Development. New York: Macmillan, 1981. 
Wittrock, Merlin C. (Ed.) Handbook of Reserrh on 

Teaching (Third Edition). New York: Macmillan, 1986. 



50