#.4

i6

TEACHERS COLLEGE BULLETIN

judges in distribution x assigned to x values above (to the right of) the true location of y. Then, if judges vary just as widely on one composition as on another, we should expect to find 75 per cent of the judges in distribution surface y rating y as below the true value of composition x on the base line. The sta- tistical name for the difference between the true value or median (50 per cent) point of a distribution and the point on the base which has 25 per cent of the distribution on one side and 75 per cent on the other is the Median Deviation (M.D. or P.E.). The second assumption may therefore be restated to read, " The Median Deviation of a distribution of judgments on one composition is equal to the M.D. of a distribution of judgments on another composition."

-5-4-3-2-10123456 Fig. 2. Assumed equal variability of judgments on all compositions

The statement may now be made, which might have been made at the very beginning of the discussion for those who are familiar with statistical terms, that the Median Deviation is the unit of difference employed in the Hillegas Scale. The sample composition which on the scale has a value of 2.6 was judged as better than 2.6 by 50 per cent and as poorer than 2.6 by the other 50 per cent of Hillegas' 202 judges. Twenty-five per cent of these judges would have judged it as poorer than a sample whose actual value was 1.6, and on the other extreme 25 per cent of these judges would have judged it better than a sample whose real value was 3.6. We may assume that if Dr. Hillegas' judges had included in their list the composition Re St used in this study, 50 per cent of them would have rated it as better than 8.0 and the other 50 per cent would have rated it as poorer than 8.0, for the median of 139 judgments places this sample Re St at 8.0. If the M.D. of judgments on one

W- Im to M. tm

yS0^- !4:lit

\^ ^ ■v^y V

^<K. • v/>V''-''(wn v» ■v.v. . ];%t !•■/■:>■ ),-',■ ji-. 'w» ;<;.■ \v: ■

« *

%^h 'W"''^, "^ ^ ^ W -ft'. P' #

■ € '4 ■•^■4^ M^ '€ ■€ ^€ ■€ .4 .'4 ■€ :4 -'^ % '% '^ *k ^]

.^ -^s^ -i^ .«T .^^ <^ -Si ^■■^ <-^ ■•'^ 1^ ^ ii^^ .^ 'Wf ....; ^i^jl ;^ .4

■% ^k ^'"^'

SUPPLEMENTING THE HILLEGAS SCALE

17

composition is equal to the M.D. of judgments on another, we may further assume that 25 per cent of Dr. Hille^^as' judges would have rated Re St as poorer than 7.0 and another 25 per cent of them would have rated it as better than 9.0. It is quite surprising, therefore, to find that only about 10 per cent of the 139 judges used in this study rated Re St as low as 7.0 and only about 10 per cent rated it as high as 9.0.

If the base line of the two curves in Fig. 3 is allowed to repre- sent M.D. units on the Hillegas Scale, the low rather flat curve encloses the surface representing the general distribution of judgments to be expected (according to Dr. Hillegas) for any

Re St

G©neral dla- tri'bution & acale

Ke St a&ale -> — HeSl scale

Fig. 3. Small variability of judgments on one composition

sample of composition, while the narrow high curve encloses the surface representing the actual distribution of judgments on composition Re St. whose median value was found to be 8.0. If the M.D. of the Re St distribution were used as the unit of a scale, in place of the M.D. of an assumed distribution common to all compositions, note how much shorter these scale units would be and how many more units would therefore be included between any two different points on the scale. Fig. 3 also shows the M.D. units that would result from using as the basis for a scale the distribution of judgments on composition He Si, whose

4. %''^^

n '^

1 #.^

i:^':-?

'} 'M- '^*i

;|iV, ", .^1 "...^l m i^^

■m% -iii ^^i «*•■

f.-*-- '■*■;■*■■ *:-r*; *-w*. «. »• «-\r*^ *»;..■*■ j^-*"- jr- JE^ .,-,~Jt-. i^;- *- *. ' .1.; ^

e p- fe p, p" fe; fr^ fe. S fe Jf p fr- r ^

■ 1^ %-^ te- te^ & ' fe. H: W fe. fe: te .fe- ^■5--

5 . % W

« '■■^'- v#- p-' f*''. 0'

fe^^ ft

1^' &i'^-' |s^-'W #. p l§ i' ^ ft |i^

i. |f'. J^ 1^, fv. ||%- p [^: 1^^ {^., ii^ p il'

"••■ p f Ip 1^ ||- ^- 1* l^ 1^ it #

.- ,^-'^' p-' 1^- F' p^ P- .^^' ^^ jr ff ^ ■ {# if

'i; fe- p" |i5 1^^' Tir^ ]#■ 'f^ 1.

It. (1^

HIS', , , ^ -^ \*i m, i*. j<l^ tv^

Vl;,: 1*. ' '■ > fffi liiii ■ -.I'li ''&.• liht. ^Vi^.v '%■,:;:,. |

¥' it

, u^ . . . .. .

i# 'fi (#• M-

(%■■ E-

■ -1^ p i^-" P' \^-.''^^' v*^'" 1^ .^' It-' it- ;^.- ;t te" ii< 1- ir' (f o

W f^ ' ^ (i p^ I»^ 1# !#;■ I^v p |i p'' {i; |i \§- &'^ (!^/ ft. (It i

:%'5%.^ 1% 1# It: ^^ p' p p' 1^' 1%^ it*: ii.-; t^ fi^ t-. It' *., Ift

,- :.., . ^ ...- ,i.;' W^ |t^ lll-"-lk 'il' t|. ^|>^ (t itf !s? ^^ It. -^i

:|^' I*' ■i^' # P^' ii^: #" |fei' W ¥■■ ¥ ¥ fr-' 5i' !^' i^- IP- (i ]#■' ir-" : >^ It. ^' It- -^^ 5*i:^ ^i: -m -'p. '^ If 'i?^- \^' ||; II . I*' ;^. l^- W ^^ ^^'^ ■■? 'f^,"' jm' .fi' •tiife- ■ iW .j^^ jPF-' p, ■-[f^: fto^ ^ 10,^ ^:p- W^ W' iP' W' P' IP-' "S '

. -:*.,.. v.. .,,. ^;:.,. ^«.. .^ ^ 1^, }^^ ,j^ ^, ||^ ^,; ^^.,, ^, p ^ .^|, :||,.. ^|^ .;^

■11^^, *■' 11^ :!■

p. jm -m p ^p^ m^ i^ ^^' Si:' p ]§■ '^ p

y' ^^' W -W -^^ ^^ . '* t^r:' 1%' W ^P'' ^■' 1^'

4.€ •■#"" ■^■; -m. W' l$u iff ■ fl"' i¥^ %'■' f^^ i#- il^

1^^ #.. ir ,, k. m W i^- :^'' ^^^" P 1^

^. ^-t— ^|.i m m 'Ir' 'i^', %': p 'i^ ii' »: 1%

<0i ^^' ^ p.' p. '^: It^ ■{# ;^^ 'if:- [f-^ !l. 1 •^■■■■■^•' ;% T?:' %■ ^/'^r l^r ^C^ ^*-' ■'^- '^- ' '"-"■

:.#:' j.^' :|*^^' ^-- '^

*,i 'i-t/' ii^^

;:'^'^ 1.'^:

W -k W

,%:■ #'^- . i^; ;;*|!

|&; t^^.' 4h' '^' M^- ^^x- v§\, ■'h '''■ "^''■" '' I.' «^:^ ^^ m^ '^t -■■ ■ %^ ....

h it' *'fe -^^^ <^* % %'' 'f^^' -^W !^^ i^}' ffe'' -ji*' |W 'p|.' ^''*'

^; fi;^ ai^.^ 1^ «r,' .^ -.- ,p,: '^p.- ffv' 1^^,. p' (|,,-' ?!;■; ]|^,.: [^

i'it/ t¥. fe *l^- %^' ■%• iifer ^^i;-' 1^' 1Jfev |t/ ■%: '%■ %:' (1;/ ii ^%. I*, i»t#:' *!?;■' ^t- '■•■■' jl^^'' t# il^- d'^' ^" •'

i^,^. lit' *4>*i. *iJi .^f- %:■''%' 'li*.^' |s^' ^i^. f#^ ft.- ,«- i-- 14 !#■ %, 'u ^%-' !%■ 'I^f" ^fc' ^^f" # 1^'- 1^/' >?' il^- 1

i;ii. i^t. 4^, 'fc i#^^ :.fe-^ '-^^'m m: m. # "U^. ^>. ^^' .if ij

fe' ^'- ^'' i^^:-

^i^. i^ "" ^ ■*■

:,^ 4' ' W if'

l#-: %%.. %:' lii^ 1,^^ ■ # fi'- .'^'' :f^v ^^,M %■ -^'-^^ '%■ i« % ¥' i^; ;i^ a I* fc >fi;-ifi&:<!^4r -f^^^ l^v 'If" 'W p' il?.:. ii^" ji^:" (p' -'^ 0r 1%:'

'^■. im ifi^. 1.1^:.' ii^if,: te i*ii--'*lMv^ A'.:iy:' -*ii?' ife: :%;■ *'■; ij. ti^: .^' li." % v^: t.

s

.T75

Thirteenth Series, No. 12 May 27, 1922

S^arlj^ra Ololbg? lulblm

Supplementing the Hillegas Scale

A Description of the Derivation and Use of the Nassau County Supplement to the Hillegas Composition Scale

By M. R. TRABUE, Ph.D.

PROFESSOR OF EDUCATION, UNIVERSITY OF NORTH CAROLINA

Published by

3(rart|ar0 (Ball^B^. (Halambm llnitirrBUg

525 Weet 1 20th Street New York Qty

M«

"oe'-aaft

Qlrarljprs CdaUrgf ?inlUtin

Thirteenth Series, No. 12 May 27, 1922

Published twelve times a year: Fortnightly in September, October, and May, and

monthly from November to April, inclusive. Entered as second class matter January

15, 1910, at the Post Office at New York, New York, under the Act of August 24, 1912. Acceptance for mailing at special rate of postage provided for in Section 1103,

Act of October 3, 191 7, authorized.

Supplementing the Hillegas Scale

A Description of the Derivation and Use of the Nassau County Supplement to the Hillegas Composition Scale

By M. R. TRABUE, Ph.D.

PROFESSOR OF EDUCATION, UNIVERSITY OF NORTH CAROLINA

reprinted from

Teachers College Record, Vol. i8, No. i, January 1917

revised edition

Published by

STparlirrH (EoUrgp, ©nlumbia llniuprHity

525 West 1 20th Street New York City

OCT 23 '22

^C1A683863

^0 I

SUPPLEMENTING THE HILLEGAS SCALE*

During the spring session of 1916, some of the advanced stu- dents in the department of educational administration at Teach- ers College were engaged in an educational survey of Nassau County, Long Island. > In the course of this survey the educa- tional achievements of public school pupils were measured by means of standard educational scales and tests. The measure- ments of the quality of English compositions written in Nassau County suggested very strongly the need for two supplements to the Hillegas Scale.' In the first place, there seemed to be a distinct need for a supplementary scale composed of composi- tions of the same general type as those written by Nassau County pupils, and, in the second place, it appeared desirable to set up some tentative standards indicating the quality of English compositions to be expected from the pupils of any given school grade. The discussion which follows is a report of certain efforts to supply these two supplements to the Hillegas Scale.

In spite of all the criticisms of and objections to the Hillegas Scale, the fact remains that it is one of the most useful measur- ing instruments in the whole field of education. The objective character of the scale has been extremely useful. By means of this scale it is now possible for one person to understand and to identify the quality of written English indicated by another person. Intelligent use of the Hillegas Scale by teachers and supervisors tends to place emphasis on the quality of the written composition rather than upon the personal relations of the writer and the teacher. Any supervisor who employs the scale

*Copyright, 19 17, by Teachers College.

^ Trabue, M. R., Report of a Survey of Public Education in Nassau County, New York. Albany: University of the State of New York, Bulletin No. 652, December i, 191 7.

' Hillegas, M. B., A Scale for the Measurement of Quality in English Composition. Published by the Bureau of Publications, Teachers College, New York City.

2 TEACHERS COLLEGE BULLETIN

with a group of his English teachers, and who then takes the trouble to discuss with such teachers in the light of this experi- ence just what constitutes general merit in English composi- tion writing, will be able to show as a result a much greater agreement of opinions and of school marks in this subject. In other words, the use of the Hillegas Scale calls attention to and defines fairly well what we mean by general merit in English composition.

One of the defects which is quite frequently pointed out by those who object to the Hillegas Scale is the fact that the first three samples of this scale are not the compositions of children, but are artificially constructed samples. To some of the other samples the objection is made that they are so short as to afford very little evidence of their true quality. As an example we may point out Sample 534, which represents Quality 5.85 on the scale.

Fluellen The passages given show the following characteristic of Fluellen: his inclination to brag, his professed knowledge of History, his com- plaining character, his great patriotism, pride of his leader, admired honesty, revengeful, love of fun and punishment of those who deserve it.

This confused sentence is hardly sufficient to give one a definite impression as to its quality. Objection has also been made to the fact that the compositions appearing on the scale are not all of the same type. Some teachers claim that they find it difficult to compare the quality of two such compositions as the one shown above and the one which follows it in the Hille- gas Scale (Sample 196, Quality 6.75).

Ichabod Crane Ichabod Crane was a schoolmaster in a place called Sleepy Hollow. He was tall and slim with broad shoulders, long arms that dangled far below his coat sleeves. His feet looked as if they might easily have been used for shovels. His nose was long and his entire frame was most loosely hung to-gether.

Some slight additional criticism has been based upon the in- equality of the intervals between the values of samples on the scale. The interval between the values of the first and second samples is, for example, 1.83 units, while the interval between

SUPPLEMENTING THE HILLEGAS SCALE 3

the second and third samples is only .77 of a unit, and the inter- val between the eighth and ninth samples is only .66 of a unit. Many of the objections made to the Hillegas Scale are of very little consequence other than that they offer excuses for not making use of the scale. It is nevertheless worth while to at-

NASSAU COUNTY SUPPLEMENT TO THE HILLEGAS SCALE FOR MEASURING THE QUALITY OF ENGLISH COMPOSITIONS

Directions for measuring: Compare the quality of your composition with the quality of the samples on the scale. Assign to your composition the numerical value of that evaluated sample which most nearly equals it in merit.

0.

.004

1.1

1.06

1.9

1.93

2.8

2.81

3.8

3.84

What I should like to do next Saturday I went going on to the Dox Saturdaye dnd day we the boys and I well going home and I well going the boys, and I will going these read in and they to night, and we or night. I well going a ground shalt and I gone out I will going to shea shouse and I will shoe or the skill of the shea of night.

I intend to mak a snou man and make an fort and fort snou ball at chidern and hau I whist ma frant carolyn cole what were me I will going to the mauiss on Saturday.

Georga will come went me.

at night I will going out went my mother to the marce

I will mak the snou man and the fort in the moning and in the aftermoon I will go to the mauies.

I whist there whest school on Saturday

one next S aturday I expect to go to the city leve next G aturday to see my ofriend archie king I am going to grow to the baning balys circus with hime next S aturday fefore I go I have to do my jobs feedsing the cows ard horse ard chinkens and geese next Saturday My friend is a very good fellow to go and see So my mother S iad " If I do my work during Easter week vacation I can go to the barning baley circus with, hime

Once a pon a time there was a girl. One day she asked me what I was going to do next Saterday so I said, "I am going to go for a swim." And she said, "thats

just where I am going to." next Saterday came we both went down together. We came home at noon time, after dinner we went to the picktures. There we had a good time. And then came home at night.

I would like to go out in the after noon and play catching the ball. Go over to Bertha's house and have a few girls to come with me and be on each others side. I have a tennis ball too play with. The game is that one person should stand quite aways from another person and throw the ball too one then another. Someone has to be in the middle and try too get the ball a way from someone then she takes this persons place who she caught the ball from. Then till every person has a chance.

TEACHERS COLLEGE BULLETIN

5.0

4.97

7.2

8.0

Next Saturday I should like to go away and have a good time on a farm. I should like to watch the men plowing the fields and panting corn, wheat, and oats and other things planted on farms.

Next Saturday I will go to the Pioneer meeting if nothing happens so that I cannot go. I should like to go swimming but it is not warm enough and I would catch a bad cold. I should like to go to my aunts and drive the horses, I do not drive without some older person with me, so I cannot go very often.

I should like to see my aunts cat and her kittens, too. I think I can, to.

I should like to join my girl friends, who are :?oing to the city on the 9:05 A. M. train. They are going shopping in the morning and will ^Q have lunch to-gether, then they are going to the Hippodrome After the Hippodrome, they are all going home to dinner to one of the girls houses, she lives on Riverside Drive so they expect to take the "Fifth Avenue Bus" up there. The evening will be devoted to

6 01 P'^ying games, .'dinging and dancing.

If I had a thousand dollars to spend, I think I would take a trip to San Francisco by train with the rest of the family, and stop at a sea-side hotel. It would be glorious to see the surf again, and to escape from the cold blustering weather of December for the balmy breezes of the ocean, and the whiff of orange blossoms.

We could take long drives under shady trees, visit the orange and olive groves and bathe in the surf. Think of bathing in the ocean n December!

Coming home again I should enjoy stopping at Yellow Stone Park. It would be lots of fun to camp out, and to ride over the prairies on frisky ponies. It would be very interesting to notice the change of climate as we got farther east, and to go to bed on the train one evening feeling warm, and waking up the next morn- ing feeling very chilly.

I am afraid by the time I would get home a thousand dollars

7 22 would be pretty well used up; but if not I would like to give a party.

One Sunday, towards the end of my summer vacation, I was in bathing at the Parkway Baths. In the Brighton Beach Motor drome, a few rods away, an aviation meet was going on. Several times one of the droning machines had gone whirring by over our heads, so that when the buzzing exhaust of a flier was heard it did not cause very much comment. Soon, however, the white planes of "Tom" Sopwith's Wright machine were seen glimmering above the grandstand. Everyone stood spellbound as he circled the track several times and then headed out to sea. He was seen to have a passenger with him. Suddenly, the regular hum of his motor was broken by severe pops, and the engine ran slower, missing fire badly. In response, to Sopwith's movements, the big flier tilted and swooped down to the beach from aloft like an eagle. The terrified crowd made a rush to get out of the way as the airship came on, but Sop- with could not land on the beach, but skimmed along close to the water instead. Suddenly his wing caught the water, and the big machine somersaulted and sank beneath the waves. The aviators soon came bobbing up and were taken away in a launch, but the g 00 accident will not soon be forgotten by those who saw it.

SUPPLEMENTING THE HILLEGAS SCALE

The courage of the panting fugitive was not gone; she was game to the tip of her high-bred ears; but the fearful pace at which she had just been going told on her. Her legs trembled, and her heart beat like a trip-hammer. She slowed her speed perforce, but still fled industriously up the right bank of the stream. When she had gone a couple of miles and the dogs were evidently gaining again, she crossed the broad, deep brook, climbed the steep left bank, and fled on in the direction of the Mt. Marcy trail. The fording of the river threw the hounds off for a time; she knew by their uncertain yelping, up and down the opposite bank, that she had a little respite; she used it, however, to push on until the baying was faint in her ears, and then she dropped exhausted upon the ground.

Note. — The first seven of the above compositions, values o to 6.0, were written during the month of April, 1916, by children in the elementary grades of the schools in Nassau County, New York. The last three compositions, values 7.2, 8.0 and 9.0, were selected from compositions which have previously been published by Professor E. L. Thorndike.^

The value assigned to " The Hunted Deer " (9.0) is that given it in the Thorndike Extension of the Hillegas Scale.' The value assigned to each of the other compositions, values o to 8.0, is in each case the median rating of 139 judges who employed as the basis of their ratings the Hillegas Scale for English Composition by Young People.*

The unit of quality is the median deviation from the median judgment of the group of 202 judges used by Dr. Hillegas in securing the final values of the compositions appearing on the Hillegas Scale. In less technical terms, the unit of quality is such a difference in quaUty as was recognized by exactly 75% of the original judges and not recognized by the other 25%. For general purposes the bold faced values are accurate enough, although more exact values are given in small type.*

tempt to eliminate in so far as possible some of these real or imaginary objections to our educational scales. The supple- ment to the Hillegas Scale which is here presented does not eliminate all of the objections, but it does reduce some of them to such an extent that one may at least hope to find an increas- ing number of composition teachers making use of objective measurement.

1 Enghsh Composition — 150 Specimens Arranged for Use in Psychological and Educational Experiments by Edward L. Thorndike.

" Thorndike Extension of the Hillegas Scale for Measuring the Quality of English Composition.

* Scale for the Measurement of QuaUty in English Composition by Young Peo- ple, by Milo B. Hillegas.

* Copies of the above Supplement, in a form convenient for use, may be pur- chased in any desired quantity. Price, 8 cents per copy by mail; 5 cents in quan- tities, postage extra. Published by Bureau of Publications, Teachers College, New York City.

6 TEACHERS COLLEGE BULLETIN

This supplement may be found somewhat less objectionable than the Hillegas Scale inasmuch as the objections mentioned above are not so applicable to the supplement as to the scale itself. It contains no artificial samples. Each sample on the supplementary scale is as true a representation as can be made on the printed page of a child's written composition. ^ Each of the samples appearing in the supplement is sufficiently long that one may readily obtain a real appreciation of its quality. Each of the compositions tends to be of the same general narra- tive type, which fact gives one a simple basis for the recogni- tion of improvement in quality as one passes upward from qual- ity o to quality 9. The first seven compositions are upon the same topic, '' What I Should Like to do Next Saturday," which is closely related to the type of composition one would use in writing letters. This is an added advantage, since letter writing is no doubt the form of composition for which public school pupils will have the greatest use in their lives outside the class room. The intervals between the values of the samples in the supplement are slightly less unequal to each other than are the intervals found in the original Hillegas Scale itself. The rela- tive intervals on the two forms of the scale are represented graphically in Fig. i (page 14).

In preparing the above supplement, the writer made use of compositions written by the pubUc school pupils in Nassau County, New York. Each composition in a group of fifty-five hundred written by these pupils had been rated and assigned a value^ on the Hillegas Scale by at least two trained judges, and in case the first two independent ratings differed, a third judgment had been made by still another judge. From this mass of roughly evaluated compositions, twenty-eight samples were selected upon the values of which the first two judges had agreed. In addition to the requirement that his own judg- ment and that of the first two judges should agree as to the quality of the compositions chosen, the writer also made it a rule not to take more than four compositions rated on the same sample of the Hillegas Scale. The purpose of this rule was, of

^ The sample valued at 9.00, it should be noted, is the only sample not written by a school child.

* In these first ratings no effort was made to assign values intermediate to those appearing on the Hillegas Scale.

SUPPLEMENTING THE HILLEGAS SCALE 7

course, to make sure that there would be in the list two or three samples of each quality on the scale. In order to meet this requirement for the upper end of the list, two compositions pubhshed and partially evaluated by Professor E. L. Thorndike were included in the list of thirty samples finally selected for the experiment.

Each of the thirty compositions was carefully mimeographed, being identified merely by the first letters of the name of the child who had written the composition. ^ About one hundred sets of the thirty compositions were prepared, each set being enclosed in an envelope to facilitate the administration of the experiment. A sheet of directions was also prepared and mim- eographed, asking that the compositions be arranged first of all in the order of their merit and then rated carefully on the Hillegas Scale. A sample sheet of directions, as it was finally filled out* and returned to the writer, is shown on page 12.

One hundred and thirty-nine different judges were used in securing the final ratings given each of the thirty samples. These judges were all college graduates who had had experience in teaching and administrative or supervisory school work, and were at the time of making these judgments members of classes in educational administration at Teachers College. One group of 65 judges were members of the advanced or practicum class studying intensively the problems of educational measurement during the summer session of 191 6, while the other group of 74 judges were members of a similar group during the winter session of 191 6-1 7. There is no reason to believe that one of these groups was by nature any less intelligent or any less able to distinguish merit in English composition than the other.

The first group of 65 judges were assigned the task of rating the thirty mimeographed compositions immediately following four hours of special class work, two hours one day and two hours the next, in which the derivation of the Hillegas Scale had been discussed and three or four sets of composition papers had each been rated two or three times on the Hillegas Scale.

^ Identifying letters were " made up " for the two samples borrowed from Thorndike.

* The judgments recorded on this sample sheet are copied in the first horizontal ine of Original Record Sheet No. 2.

TEACHERS COLLEGE BULLETIN

OQ "H

'HOOOO

lOOOOO

OOOlrtO

OOlftOO

0}f ^J

n irtrtioo

Ot^rH^OOl

irtincoNo

t^Tf OWN

OOCOlrtLON

incoco-tN

S on

N COminirt

NOIMOLO

mt^f-iTi>m

tOtOt^MlO

OWOOOIO

}S of

tOooNLn

OincCMN

t^O-^OOOi

o/V

010 05 00 00

!r>iN^ior~

O100N.-IO1

^MMCvJCO

OlOt^-*"-!

Off »7

t^OlTjiNCO

ff.o 97

.-KNCOMCO

lOLnoO'^

(MMOO'-l'-l

(OOli-HOLO

nyj vj

oioomo

TflNLONN

Tf rHiniMOO

0(7 »//

^DC^O-*00

^j "d

OH vyn

OOlONtOOO

to ■'t CO 00 in

to CI C^ 00 t^

00 (CIO SO t^

ocot^oco

COMCMMCM

[^.-(OOrttO

«/Y iO

M3Vi ou

»d "D

»a fH

>M. M3

>PV »/

1,0 OD

^a of

oj o-j

COCOM-"!"*

CO to C^ LOO CO •*'*C0^

CM CO CO-* 10

00 10000 1> CM 10 CO 10 in

LOOJtXDlO M' 10 00 CM CO

t>Ot>t><D ■"tlOCMCOCO

TjlTtOTj-O

lOLO'^lOrJ'

OCM00Ot> LO •*■*;£! CO

lOt>lO00«3 10 10 CO 10 10

lOOCMtOtO ^CO'-HCM.-l

a)toiot>o

■q.TtcO'^CM

(DOOCMO -Jf-^CM^CO

OOCJICMCJIIO LOlOCvlTjiCM

to 00 00 CM CM to-* --1 10 CM

.-lOtOlOO COCOCS1C0.-C

Tf to-* 00 00

•* CO CO CO CO

COOCOtDO ■"tCMCMCO"*

OiCMCM-*^ COlO-*-*C0

OCOOOCM

CO-^CO-*-"^"

ocoooo

CO CM CO CO CO

I>CMOLOtD CO CO •^■* CO

OO-^f-iOlO

Tj-cotrTno

tocMTjiioin

001>10->t.-i

•* CO rococo

0000(0 00 10 CO-* CO-*

t-C0CMlOt> •*•*-* CO CO

1-itDOOOCM 10 CO CO CO 10

m MD

(DlOlOCM.-(

iom-<ttDco

Ot>OOC0 05 lOlO-*lOi-(

CM (O (J) ■* CM (0-*CO->t-*

03 H3

-*t>oa)CM

CO •q"(0-*(0

05 --KD 10 00 LOtO^LOlO

OOOOCMOO ■*(D-*1010

t>oc7i--ic:5 in-*-* to 10

^J HD

OKOLOOO 101010(0-*

tOCTllOLOOO

to -^tn 10 10

»a m

(O 000 to 00 CO to to 10 to

t>0(Jli-00

10 tOLO tOLfi

101000CT5

Tttototnto

ns m

OCM-*t>.-l to (0 00(00

OOC-lOt/50

t> to to 10 1^

tOCMO--ilO

otntoooo

OOtOCMCMO

ooooto

<"S «3

OlOOCOrl'.-i lOlOOlOO

lOOOOitO OOOlOOtO t-OOLOtO 1 OOtOOOf-

00 Lo to too

Vtf «7

l>-*t>00)

to to to 00 to

tocM-*(Oto oo-*cri-*-* inootoo totoioooo

OlCOOtOO

vf pv

O LO <J) o o 0(0(0 l>t>

to 05 CM 000

otooioto

O O O 00 o

CMOOOCn

otooooto

IS »il

oooo-*io 00000

OIOIOOO I tooo-* OOOOOtOO OOOOtOOiOO

«S »H

tD-*Orf<-* 000 0)000

-"tOOi-ltOCM UOOOCM'-i OitOOOOO OChOOOO)

SUPPLEMENTING THE HILLEGAS SCALE

OOCOOtO —oooo

Ortt-Ttin

tOLnoNTf

CMi-iinooT}'

LnMoo— if-

PONOOt^f^

OOOOIOCO

MCOtNlO

MOiwoo ■<toot^ino)

mmtooiw

IM (M -H

COMTflOOO

^tOOOOlN

CT*'«f «0

MtOO)ON

omotof~

XI ^ lo r~ in

OOO-^LO

inoootn

0)03 05 CO ID

INi-iMM^

OOt^OOOO

00 -H LO LO (N

-^NCO.-iCO

I>LO.-iOOOO NCOMi-irH

M-*iMrqr-(

^ CO CM

OOCOIOLOC^ riCMCOCO-*

«3t>C>J!£)C0

i-'i-iCOM.-i

C^OOOOIO

rjcgM— ICO

■* 00 ^CtTO) NCMM-*CO

MCOCOIOM'

•-(COCOMCO

oo^]O)coc^0

COC^C^-HCO

CJ —■ "3- PJ CO

LOOlOOOOCM CO CO CO CO ;d

CO-^PJCMN

OllOCOOO^D —iMCO-hCv)

NOOCVJOlCO

in --I CO .-1 ■*

ocooKOLn

CO CO CM CO-*

(Ot~-t^C0t> CM CM CO CO CO

0)0;OCO^O CM'^LOCO'^

■•s"too<rio

t^-* 00 coo

CO'^ ■^LOtO

COOilO'^O O.-1-.^COLO

•^•^C^CMCO

10 CO CO -sf'-fl

Ot^LOrHOl

coTfcoioin

incO^CMlD

CMt^OlOOCJ)

CO CO CO CO CO

CM CO CM CO Tr

cMLnoco-*

COOOOtlO COCOUJCOrj-

t^O-*^DO CO-^UI-^LO

Ol-HCDCMt^

<D0OtD!£)i-l

cOTfcg-ttco

incMcoco-^

■"f O CM CO 'J'

O)-^00-*CM

CO-* CO com

tOC0U5LO;D

Ol-*C0tD-* COCO-flOLO

LO-t ->J"*t^

Ln-*oo>(N

CM 1/5 000-^ CO-* so COLO

OlCOCMt^OO

t^Ot-t^LO

-* in •* '^ lo

lOtCUDSOCM CO CO CM CO-*

SOOCOt^^

coTfinin ■*

(CCM05CMO in Tp^ln to

in in so-* to

t^t^t^inoo CM CO-* CO in

oooocjiin ■* in to CO to

t^ CO in in in

cMin-HOoo -* CO to in CO

in in to to c^ -* ■*tointo

CO) to 0)^00

-* •* msoto

OOCMOitOC^ C0-*tO'*tO

CM to CM t^ in

so-* in in in

t^CMOiCJlCM

co-*Lnr)'in

CM in t^ 00 00 to into in in

in •*t^ 00 CO CO to in in t^

05t~-int^cM in -* in in to

CO r~-* to CO

Tl<tOtO-*t~

— lOCMCOi-i

-*inc^tDLn

oot-~t^o ■* in-*totoc^

to f~ in 00—1 t- to in in t>-

00 CO CO 0)00 ■v-^tomin

CO O) --I c-J to to CO c^ to to

tot^ooinin to to to to r~

C00-Ht^05

t^OOOt^CM

soinin WC--

oot^toooo

oootot^oo intot^i>to

tntot-^CMt-* t^ t^ t> c^ t-^

C^CSt^CMt^

c^int>-c^c-»

to to to 00 00

t^OlOOOCM -*-*tD00 0O

ot^ t^inoo

CMt^inoc^

00-*tO t^r-

CMt^CMCMOi

t^tot^ooo

t^OOOOTl-O

•*0)0ino

'5'-*CJ)t^00

OOt>tDt>00

tOr^OOOt^

-*S0tOOltO

CMCMinO'-i

CO 00 to co^

00 1> in 00 CO c^t^t^tooo

oincM— I— I

t^I>t^O)0)

to CO CM CM 00

tooooot^t>

oocor^-HCO

to O) to 00 00

t> in to 0)00

OOtOt-'CJiCO

in in t^ 00 c:^

inocooco

t^O)00t^00

McocMinin

l^ 00 O) ■* 00

OCOCOOO)

oioomoor^

t^ CO CM o o

Sto «

""oOjs

1) I,

x: 1 m aj 3

o'^ c

in cj w c:.2 ^

^ Mt«

"2i:

<U w O > N

HJ

V £: j: tu M^

SS ™ .9mW

S^ '^ ECi ij o aj a

" c '"

" £ t^

=" DO

c!3 ij

O C3 cu

rag" "«o)- •«"Ln o ♦J

c	•^to
t«	3 3
CO -0=0
c:
Bl	Sm
o	k5
J3	Ma
-	b-o
1	js »	r/l
	t.t>-	C
u	CQtO	o
o	h «	m
^	<U Q	0
ri^	u
	o>i	b
	0.x	u

lO

TEACHERS COLLEGE BULLETIN

OQ ^U

Oi-iOTrLO

OOOi-iTtO OOOOIO OONC>3-H OOIOOO

o;/ ^j

i-HinLoooiM

OOl-^OO O-^NNOi CSlOOt-r^Tl" tDON'-i<£>

H o}{

<woomt^

i-nocOfHin

lOlrt— irf<r 00L0.-1OO r-UDlrtcOCJ

}S of

OMlrtMin

OOOmt-HM 0IN«5-<^0 ^tOONlO

D/V

ooinoo

(N M .-H

o^ooin

LnoLOi-HO

^^DCOCOO 00t^Ot>M OlONOHn

Off »7

rtrH coined

O)0in.-n£>

ff,0 37 SS

«/v »</

"(7 "H

^A "d

CO to 00 00 03

r-iOOCMtO

Oi/^ OJ^

■-H Tf ,-1 (M CO

.'/V ■'9

N^IV OS

oooojt^rj

LOIOC^NCO

»<:/ OJ

COCOi-iCJ>M<

Ttin-*i-ico

»ff .»iir

OOCSlt^MOO

t^-* (DID 00

CO CO ■^rt CO

«/tI V3

•*-*coLnTj<

ooo<DO<r>

«/v »/

Tj'tDCOOOO

•q- in tt tc T^

Oi-it^tDN inrf rf COM-

CD -ooint^

■^tDTfCOrC

I.O 03

(D05Tt"a'ir5

Tl'Tj'COtO-*

^a of

COt^tDincO 1/5 LOCO to-*

.-1,-HOOOOO

imntocoin

--ic<ii>ooco

•* Tt LC t> Tj-

oo-*oiooai

lOlD'^LT)-*

lOi-iintrti-i

■"tCOlOCOTj"

oj o-j

■1(7 V3

OILOOOOC^

■^LOLOLnin

lOOOCONtD ■*IOCOOOM<

t^t^OOl^D

co^tonn-

fj O) .-I t^ (J»

(DinTj-tDCO

COOtXT)--! 1/5'^ ■^^D-^

.-HOOOONN

O'S'Lnt^io

NOOCOO-H LO t^ -^ tn LO

^LOLOCOtD

N(D-HMCO

Tj-cgLOTj-LO

oONoomco lo ■* LO T)< -a-

oj HD

LO<DLOI>'t

LOlMOlLOr-l

LfltDCVUOCO

.-uMai-HCV) CD t^ 'J' CO to

ioooo^oot~ LO COLO -a- CO

■(^ HO

OOOOOOOl ^(DtOOO'*

lOLOt^l>00 LOLOtDtDtO

tXDLOtDCO

^OOOCOO 001>-*CO!D

»ff VJ^

■q-LOLOOLO

00 <M to COLD (DLOtDt^LO

t>t>LOCTit>

to to to to to

"s .«;¥

03 t^ O) "tf tj)

oirjc^otD

a^r^Ln-^io

t>l>t>00LO

OCOOOtDCO t^OOOOt^t^

t^t^tOLO'^

ffig n;5

t> t^r-oico

tOLOTj'ON

t^t^oooot^

t^i>t>ooc^

t>lO.-HLOtO

ooooooot^

t^NMLOO t^OOtDl>00

OOOOLO t- l>00LOtOt>

OOLOtDO

ootoc^r-tD

HS "7

OOMC^CO^ t>00<DOO0O

t3>OOOLOCO C^tvC^OOO

COCOI>.-iO OOOC^OIOO

OOt^OOLO OOC^tOt^LO

vf pv

OCOO-*LO O0 00t-O5t^

Ot--Ttt^00

cooot>oot^

00 00LOtDL>

cgLONt^i-i

00LD00t>tD

>S »H

COOOOOLO 0Ot>O0tJ)tO

OOOOLOCO OOOOt^t^tJl

Tf OONCOIO

oot>ooor~

^coot>ai

00 00 00 00 1>

lOtOO^LO 0> 00 05 00 00

OrjtDCqtD OlOl^tOtD

-JOCO'-ILO

oot^oit^to

*S »H

tOOCOf-N

t^oooooc^

t^LOLOOOl

t- 00 00 05 to

f-H'S'LOLOLO

t^i>O>00i:^

SUPPLEMENTING THE HILLEGAS SCALE

11

	2 o M O O U o H Q W en < CO < i-l w	0(7 «y	MWOO-*	OOIOCON i-H 1 I	-^OOCDO	OOOON	o-*ooo	^oo-o	OOOO
	OJV -fj/	NNioirsLn	iOOTi't^Ln	COIONNCO	NlOrHt>00	OtOLOt^OO	10OCML0 05 coco rH	OOtO^M
	y 0^/	■HMCMOOO rH 1 rHCO	COrtOlMOO	CTiMrHW*	■^OOOrHLO	C^JLOOOt^OS rtCO	^Nt-tNt^	Ot-O^
	fS of	O WMO5 00		lO-*COOO	COtOrtOOO	t>OtOCMLO •-ICSI <N	tOlOOt-O ^<N	lOO5 00rt
	opV	000-*OLO	OO'l'C^lOOC^	OOOlOCOLO	^cqOMOi	totot>ot>	'* 0)05'* 00	to 00 to to
	og »7	(Mr-i com	tnoiLOf<iO cO'Hcocorg	TftOOOOCM	OOtOOONlO rHcsi^pgcM	COOCO-^M	c<IOooo^^ CMCMrHrHC<l	lO CM to CO CM-HCM
	ff.0»7	ooocsiinoo	UDLOLntDM COCg (NW	00t0Ot>L0 ■-ICOCJrH.-l	LOOOCOOIN	^COPJOOO f^cor-irgco	OtOLOtO*	OtOO)00 rHrtCM
	«/V "c/	un'-iffiMoo	CO CO CO CO cj	001010 05 00	OI>lO-HO CMNlNcgcO	to 00 to CO 00 M ^C^4rt	O00tOO5 1> corqrtr^cM	OCMOOt- COCMCM
	Off 0//		lOOOOOtO^ rHCJCOCOIN	tocot^ot^ IM-*CSIC^1^	t^ 00 LOCO 00 rtcortc^csi	CM-^C^LOO) lOCO-HMrt	cMtoor-io *CMrococM	ooot- r-H'*CM
	'^«5		i-HOlMl>00	OOOOOON rqrtcv)ooc<i	to to o o to cQcoc^c^cg	■*005<010 iNCOrtNrH	Tf tOlNt>tO '*'*'* CM CM	cMtortin CO CM CO rH
	on ofv	i>ooot^m cocj.-iTj'co	0)LOCJ)f-CM CO^MCMCSI	t^ioooooo NcO'-i t^co	tOOOl^O) ^c^c^co	lO00'*t>LO CM CO -H (M fj	Olo—iloco CM'*'* CM CM	coototo ■*rHCO.-l
1	>fV '■9	MMNCOtJ'	M'-cOlOOt^ rfTtcOCOM	■*t>LnrqtD	l>LO00tOt^ CO'^CONCM	00 to 00 too rHCO-HCO!N	lO00-*00'* CO CT* -H CO	tOCOf-t- WCOCO-H
N^iv on		ot>tnoCTi tfCMTj-OCO	COO'* eg-*	t^COtOC^'* CO 00 CO-* CO	^totooto '*t^rH0O^	f-t-ooco-* •*-**-*'*	tooooo coc^]'*^
CO	»d oj	OOOO'HOO CO -g- CO (C -q-	OOCJlOOO coLOTpLnco	LOooO'^Ln MLOTf IN	—1'* 05 00 05 pqcocoioco	Cg^t>rtCO -*t>COC0N	005COtOCM ■*-*'* CO*	LO to 00 to CO CM CO CM
Q »; o o w	»g }d	OCT! 00 pq CO	'J'COlOt^CO	OHD^tOOO M-*CO(N-*	C^tOtDOO Tt^'^lO'*	0)M05C0t- rHOOMCOCO	t-O'^LOtO to to LOCO CO	LO(N'*tD ■* CO LOCO
>M MD	CO CO LOOM Tj-cMcoLOun	lO-*-H00-* lO-^Tf^DCO	Lnt^ot>io CO'* CO CO CO	OOOCOt>LO C0t>-*IMCO	CMOMOO NtO'**Ln	lOO)CMNL0 •*lOlO-*CO	C0lOO5'H CO*'* CM
	}Vi »[	O (C Ol 'sD Tf	incT>^Lnt> Tj'^LO'3'CO	t^LOLnoico core CO COIN	05 00'* 00 to rt-^TCc^co	ot^rtoor- CMtO^CO-*	tot-t-ioco TriO'*'*co	CMOtOO ■*LO'*CM
o o	1.0 oj	00t>P3CT>O Tj- Tj- -J. !£) in	■^OOCOrfO	Oioorjooo ■*tO'*C0'*	t-.t^^lOCv) ■*LO'*'*'*	to 03 to t^ t- lOLO-*C0tO	OOrtOOtOlO Lot- to to-*	t-toooio -*to*co
^a of	lOcoLniOLfi	t>-*ooai«o	coco-* too CO t^-* COLO	C0Ot>00C0 IOlOtI"*'*	to CO t^ CM to LO 00'* CO to	000 0501-- t>to-*to^	OOt-O) ■*C0'*^
03 01	in Tj. (£) to U3	I>-CqOC0C^3	O>CT)00r^LO COIOCO-*'*	OPJIOLOOO tolr^-^iNCO	CMOC0t'~00 tO00**lO	LOCOOOt-CM to to to-* CO	■*tooto CO to to CM
	^a ID	.-irjtoooin Tj< Ti- ■* tD ;d	CO^OOCOCTl ■*Ln untried	in ',o CO o lO ^to-^coto	MOOCOf- t>tO'*'*C0	'*lO'*'*LO tooo*coio	CM CM CM 00 to LOtOtOCT*	'* CO to CO
	OD M3	coLncj)t>.^	OOtOCTlOlCO M'tO'3't^-*	'^intDrg'* lOt>^'*'*	totooooto C^t^-^tOrP	tO'S'lOCMCM to 00-*'* to	CML0C0O5tO to to t- lO to	t-ot-t- tO to to CO
	^J HD	tOOOCOOOl tOLn<£>tr~(D	■^CMtflt^CO LOIDLOOCO	00'>*t^CO'* coc^-^-^c^a	O-lOLOtOt	COIOOOOO i>t>tointo	OtOr-^OOLO tOtOC-LOtO	OOtOLO t-t-lO'*
	»a m	tOLOtOtXD	OOOOOt^ <0 [> t£i tr^ rji	OCOLOOCO ■^•0^*00	COlOlOtO^ tXOLOtO ■*	t'^ to 00 05 00 to 00-* CO c^	tO'*Ot>0 to to to tot—	OLOt-O to to to '*
	ns ?fV	Ol C-- O 'X> o	COOOOO-H f~t^t»00t^	oi>o-<to t^t^LO-^c;!	^OOOCOf- t>05LOtO*	00O5t-l00W 00C^tOLOf~	LOOI>t-00 t-oot-c-to	t-LOt-lO t- t-t- to
	■WS "3	LOt>-mLnco lO(£lt>t^tD	loootooiin	OOOOOILO y5t>tDLOI>	lOCOLOLOlO 00 001>t>t>	O^L01>rt 00 05 LO to 00	t-005t-tO t-t-totot-	00<J50 O)05tOO)
	MH «7	ooo t~t^-* CD00t>t><£)	O O) 00 C^J 00	LOcorjoo-* (OOOCOIOOO	C0 05t>l>.05 00t>0t01>	t0 0001>05 t- 00 tor- 00	CM CO CM to CM 00 00 00tDt35	00 LO to CO OOOOt-O)
	of pv	t> 00 00 1.^ «D	00OOCJ5 10	'*CT>L0t^O t>t><oiot>	■^ooor-oo 00O001>t>	'*0050CM 05 05O00 00	C^LOCDOt- OOt-OOt-t-	CO CO 00 't 05 O) t— 05
	IS »n	T)<O00 00I>	NOLOOLO oooooocjit^	I>0001>0>	tOC0N'*l> 00 05 00 to 1>	t-'*MCOCM C0O5CO00O5	OOCMCOO 00 0)0)00 05	to CO (35 to 00 00 t- 00
	..s*//jjgSSSf^	oOLnoocooi	0-5t"LOOCO 00 05 Ul tool	OLOOOtOO oo05C^Lnoo	tOCMLOOCO 00 0)0) £35 00	CMCMt-C0 05 t-t- 05 05 00	COLO 00 to 00O)tOO5

12

TEACHERS COLLEGE BULLETIN

In connection with this rating of compositions in class, certain compositions had been read aloud to the class and a general discussion had been held, dealing not only with the merit of

Directions

Do all of this work independently. Do not let your judgment be influenced by anyone or by anything other than the quality of the compositions themselves. First:

Arrange the thirty compositions in the order of their merit, giving the best composition rank number i, the next best rank number 2, and so on down to the poorest, which should receive rank number 30. Do not give any two composi- tions the same rank.

Record your judgment in the proper place on the blank below by writing neatly (or printing, if your writing is quality 9 or less) the identification letters of the composition opposite the rank number to which you have assigned it.

Second:

Determine as accurately as possible the position of each of the thirty compo- sitions on the Hillegas Scale or on the Thorndike Extension of the Hillegas Scale.

You may assign values intermediate to those given on the scale if you feel rea- sonably sure of your judgment. (You may, for example, wish to assign a value of 05 to one composition, 10 to another, 15 to another, etc., even though there are no samples on the scale between o and 18.)

Record below the value you assign to each composition, and return your record and material to Mr. Trabue before consulting with anyone regarding your judg- ments.

Rank Number	Composition Lettered	Quality on Hillegas Sc.	Rank Number	Composition Lettered	Quality on Hillegas Sc.
I . . . .	Re St. .	82	16	Ri Be	^8
a. . . .	. ...AdJa....	80	17	... Ca Pe	. ??
I. . . .	. . . .He Si . ■ ■	7n	18. .. .
4. . . .	.... Eu Sw . . .	78	10. . . .	Ro McN.	2C
?. . . .	... .LuRh...	7c	20	Ha Da	2d
6	....MiSu....	60	21 . .	... Qu Fr	2 ?
7. . . .	. .. .Jo Dr....	c ?	22. . . .	.... Ma Ro .	21
8... .	. .. .ChCo....	CO	2^. . . .	Ma	20
0. . . .	.... Lo Co ....	40	2A	Jo Si	. 10
10. . . .	....ChFr....	48	2K. . . .	Ro R .	18
II. . . .	... .ThBe....	47	26	. . . .Pa Mu...	17
12. . . .	CoO'T...	46	27	. . .Le Bo	. . 16
i^. . . .	....ChDr....	4c	28	Le O'B .	Id
14. . . .	. ...JeMi....	44	20. . . .	....FrKo....	17
IK. . . .	Ch Wi. . . .	42	^0. . . .	. . .Ru Do . .	0

SUPPLEMENTING THE HILLEGAS SCALE

13

the particular compositions read but also with what constitutes general merit in English compositions. This group of 65 judges will therefore be referred to during the remainder of this dis- cussion as the " trained group." The ratings given by these judges on each of the thirty samples of composition will be found in Record Sheet No. r.

The second group of 74 judges were assigned the task of rat- ing the thirty mimeographed compositions before any discus- sion had been held in class either of the derivation or the use of the Hillegas Scale. This group of judges were asked to indi- cate on their record sheets whether or not they had ever used the Hillegas Scale before, but some of them neglected to make any statement regarding the matter. Where no statement was made, the writer grouped the records of those who had been in attendance at Teachers College previous to this year with the group who stated that they were acquainted with the scale, while records made by new students were grouped with those who reported that they were unfamihar with the scale. The judgments of the group who were " unfamiliar " with the scale are shown in Record Sheet No. 2, while the judgments of the " familiar " group are shown in Record Sheet No. 3.

The median 1 rating of each of the thirty compositions is probably the most accurate measure of its real position on the Hillegas Scale.

A graphic representation of the median value of each compo- sition is shown in Fig. i. An ideal scale would, of course, have one sample at o, another sample at i.o unit above o, another sample at 2.0 units above o, and so on. Not enough of the samples used in this study have values at exactly the right points to make possible such an ideal scale, but as close an approxima- tion to the ideal as was possible has been made. The values of the resulting supplementary scale are shown in Fig. i, to- gether with the values of the original Hillegas Scale and the values of all the compositions from which the samples of the supplement were selected.

'The median rating is the rating above and below which there were an equal number of ratings, — in this case, 69 judges rated higher and 69 rated lower than the median rating on each composition.

14 TEACHERS COLLEGE BULLETIN

The reader will observe in Fig. i that two of the thirty samples are very near quality i, two are very near quality 5, and two are very near quality 8. In so far as the median ratings are concerned, it would apparently matter very little which of the two compositions in each of these three pairs was retained as a part of the supplementary scale. In order to determine which one should be used in each case, the distribution of ratings on each composition of the three pairs was more carefully studied. It was found that although both compositions He Si and Re St have a median rating of 8.0, the judges had greatly overesti- mated or greatly underestimated the value of Re St in fewer

HILLEGAS p

30 Samples

10

Fig. I. Values on the Hillegas Scale of thirty compositions from which nine were selected for the Nassau County Supplement

cases than they had the value of He Si. In other words, the judges had agreed as to the value of composition Re St much more closely than they had agreed as to the value of He Si. In a similar manner, although both compositions Ch Dr and Lo Co have a median rating of approximately 5.0, the judges agreed more closely as to the value of Lo Co. Compositions Ma and Jo Si, likewise, are both very near quality i.o, but there were somewhat fewer extremely bad judgments on compo- sition Ma. It seems quite evident that to be of the greatest usefulness a scale should be composed of compositions about whose quahty there is the least possible disagreement. In the three cases, therefore, where two compositions fall at approxi- mately the proper point on the quality scale, that sample was used which showed the smallest number of very bad judgments.

SUPPLEMENTING THE HILLEGAS SCALE

IS

In order to see more clearly the significance of the above paragraph and to understand the next few pages of this dis- cussion, the reader should recall how the values of the Hillegas Scale were established. One unit on the Hillegas Scale is just such a difference between composition x and composition y as would cause 75 per cent of Dr. Hillegas' 202 original judges to declare that x was better than y, while the remaining 25 per cent of the judges declared that y was better than x. If 50 per cent had said that x were better than y and the other 50 per cent that y were better than x, then we should have been safe in saying that x and y were equal. A difference noticed by 60 per cent but not noticed by 40 per cent would probably be a real difference, but it would not be a very useful difference, for 4 persons out of 10 would not be able to recognize and employ the difference. A difference which is noticed by 3 out of 4 persons is, however, a really useful difference, and it also agrees with some of our statistical concepts.

Dr. Hillegas assumed, and there is every reason to believe that the assumption is perfectly sound, that judgments as to the quality of a composition will be distributed symmetrically on either side of its true value, — that just as many will over- estimate as under-estimate its value, and that large errors in either direction will balance each other and be much less fre- quent than small errors. If the true value of composition x is represented in Fig. 2 by the point marked o on the base line, the judgments as to its value would, according to the assump- tion, be distributed both to the right and to the left of o in diminishing frequencies represented by the surface enclosed between the heavy curved line and the base. This symmetrical surface representing the judgments on composition x, whose true value on the base line is o, is the well-known friend and servant of the statistician, known as the Normal Surface of Frequency.

Another assumption made by Dr. Hillegas is that the variabil- ity of judgments on one composition is exactly equal to the variability of judgments on another composition. In Fig. 2 let us suppose that a second composition y has a true value on the base line which is just sufficiently far below (to the left of) the true value of composition x that exactly 75 per cent of the

i6

TEACHERS COLLEGE BULLETIN

judges in distribution x assigned to x values above (to the right of) the true location of y. Then, if judges vary just as widely on one composition as on another, we should expect to find 75 per cent of the judges in distribution surface y rating y as below the true value of composition x on the base line. The sta- tistical name for the difference between the true value or median (50 per cent) point of a distribution and the point on the base which has 25 per cent of the distribution on one side and 75 per cent on the other is the Median Deviation (M.D. or P.E.). The second assumption may therefore be restated to read, " The Median Deviation of a distribution of judgments on one composition is equal to the M.D. of a distribution of judgments on another composition."

-5-4-3-2-10123456 Fig. 2. Assumed equal variability of judgments on all compositions

The statement may now be made, which might have been made at the very beginning of the discussion for those who are familiar with statistical terms, that the Median Deviation is the unit of difference employed in the Hillegas Scale. The sample composition which on the scale has a value of 2.6 was judged as better than 2.6 by 50 per cent and as poorer than 2.6 by the other 50 per cent of Hillegas' 202 judges. Twenty-five per cent of these judges would have judged it as poorer than a sample whose actual value was 1.6, and on the other extreme 25 per cent of these judges would have judged it better than a sample whose real value was 3.6. We may assume that if Dr. Hillegas' judges had included in their Hst the composition Re St used in this study, 50 per cent of them would have rated it as better than 8.0 and the other 50 per cent would have rated it as poorer than 8.0, for the median of 139 judgments places this sample Re St at 8.0. If the M.D. of judgments on one

SUPPLEMENTING THE HILLEGAS SCALE

17

composition is equal to the M.D. of judgments on another, we may further assume that 25 per cent of Dr. Hi]Ie;:^as' Judges would have rated Re St as poorer than 7.0 and another 25 per cent of them would have rated it as better than 9.0. It is quite surprismg, therefore, to find that only about 10 per cent of the 139 judges used in this study rated Re St as low as 7.0 and only about 10 per cent rated it as high as 9.0.

If the base line of the two curves in Fig. 3 is allowed to repre- sent M.D. units on the Hillegas Scale, the low rather fiat curve encloses the surface representing the general distribution of judgments to be expected (according to Dr. Hillegas) for any

Re St

Gsneral dls- trfbution & scale

r Re St ao&le — HeSl ecale

Fig. 3. Small variability of judgments on one composition

sample of composition, while the narrow high curve encloses the surface representing the actual distribution of judgments on composition Re St. whose median value was found to be 8.0. If the M.D. of the Re St distribution were used as the unit of a scale, in place of the M.D. of an assumed distribution common to all compositions, note how much shorter these scale units would be and how many more units would therefore be included between any two different points on the scale. Fig. 3 also shows the M.D. units that would result from using as the basis for a scale the distribution of judgments on composition He Si, whose

i8

TEACHERS COLLEGE BULLETIN

median value was also found to be approximately 8.0. If the dis- tribution on He Si had been shown it would have been only slightly higher and less wide than the general distribution assumed by Hillegas. Such large differences in the form of surfaces of distribution of judgments on two equally good compositions were certainly unexpected. They tend to make one critical of the assumption that the variability of judgments on one com- position is equal to the variability of judgments on any other. A rough distribution is given below in Table I of the 139 actual deviations from the true value for each composition in the three pairs of equal samples mentioned above.

TABLE I

Composi- tion called	Median value	DiSTRIB. FROM	OF Amounts of Deviation Final Median Value	Total
0 -S to to •4 -9	i.o i.s 2.0 2.5 3.0 3.S to to to to to to 1.4 1.9 2.4 2.9 3.4 3.9	devia- tion
He Si Re St	8.01 8.00	47 30 62 30	31 IS 9 4 2 I 37 S 2 I I I	139 139	.82s .487
Ma Jo Si	1.06 1.03	S3 67 SI 60	18 I 22 s I	139 139	.627 .S68
ChDr Lo Co	4.98 4-97	28 43 42 46	30 23 12 21 27 IS 6 2 I	139 139	.900 .766

' The median of these deviations was calculated from a much finer grouping than that shown in this table, hence the unusual M. D on Ma.

In connection with this table it was observed that the M.D. of judgments for each of the six compositions was less than one unit (i M.D. of Hillegas' judges). This fact had in it a sugges- tion that possibly the 139 judges used in this experiment agreed with each other more closely than the judges used by Dr. Hille- gas agreed among themselves. To test this possibility, the method used by Hillegas on the judgments of 202 judges was employed upon the judgments of the 139 judges used in this study. The resulting calculation is shown in Table II.

Table II, beginning at the top and reading across horizon- tally to the right {rf. Original Record Sheets 1,2, and 3) reads as follows: Comparing the values given by each individual

SUPPLEMENTING THE HILLEGAS SCALE

19

TABLE II

Result of Applying Hillegas' Method to 139 Ratings on the HiLLEGAS Scale

	No. of Records		Amount
	showing	Percent-	"Poorer"	Amount better
Relation of		age	( ^ )	than Ru Do
	"Is "Is	"Better"	\ M.D. /	(i=M.D.)
	Poorer" Better"
Ru Do to Ma	^33 6	4-3	2-55	Ma is 2.5s
Ma to Le Bo	131 8	5-8	2.34	Le Bo is 4 . 89
Le Bo to Ma Ro	120 19	13-7	1.62	Ma Ro is 6.51
Ma Ro to Ca Pe	117 22	15-8	1.48	Ca Pe is 7.99
Ca Pe to La Co	"4 25	18.0	1.36	Lo Co is 9.3s
Lo Co to Th Be	119 20	14.4	1.58	Th Be is 10.93
ThBe to Mi Su	129 10	7.2	2.17	Mi Su is 13 . 10
MiSu to Re St	123 16	"•5	1.78	Re St is 14.88

judge to compositions Ru Do and Ma, 133 gave Ru Do a lower rating than Ma; while 6 of the 139, or 4.3 per cent, gave Ru Do a higher rating than Ma; converting 4.3 per cent into terms of M.D. by means of the table used by Pro- fessor Hillegas, there are 2.55 M.D. units between Ru Do and Ma. Similarly, 8 of the 139 judges, or 5.8 per cent, decide that Ma is better than Le Bo, so it is therefore 2.34 M.D. worse than Le Bo. Le Bo being 2.34 units better than Ma, which is 2.55 units better than Ru Do, we must conclude that Le Bo is 4.89 units (2.55 + 2.34=4.89) better than Ru Do. It will be observed that Re St, the best composition, is 14.88 units above Ru Do. It will be remembered that the median values of these two compositions on the Hillegas Scale differ by exactly 8.00 units. The difference in quality between Ru Do and Re St actually remains the same, but if one accepts the M.D. of the 139 judges as a basis he finds 6.88 units more between them than he finds if he accepts the M.D. of Hillegas' 202 judges. This difference is shown graphically in Fig. 4.

RD M LB MR CP • • • • •

I I I I I

LC TB

MS RS

I I I

Median RATINGS vs.

Calculated VALUES

RD

LB

MR

CP

LC

TB

MS

RS

I I I I I I

O 1 Fig. 4,

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Linear projections of median ratings and of M.D. values for samples included in Nassau County Supplement

20 TEACHERS COLLEGE BULLETIN

The discrepancy between the two groups of judges may be due to the fact that the 139 judges had the opportunity of using an objective scale (the Hillegas Scale) in their arrangements of the thirty compositions, while the origina 202 judges had no common objective measure of merit in composition. The 139 judges may agree more closely with each other because they were all " school people." The fact that the thirty composi- tions were all of the same general type may have been partially responsible for the greater amount of agreement as to differ- ences in merit. Probably all of these factors combine with less obvious causes to produce greater agreement among the 139 judges in this experiment than existed among Hillegas' 202 judges.

In order to see whether the four hours' discussion and use of the Hillegas Scale in the summer session class had resulted in any increased ability to agree as to composition quality, Dr. Hillegas' method was used on the 65 " trained " judges' judg- ments and on the 74 " untrained " judges' judgments separately. The results of these calculations are shown in Table III. The reader will recall again that Re St is only 8.00 units above

TABLE III

Measured by

A. Median on Hille- gas Scale

B. M.D. of 74 Un- trained Judges

C. M.D. of 65 Trained Judges

Distance above zero of samples in Nassau County Supplement

0			0				^
C)		0 CQ	(<	0.	c">	cq	^	c^
0=;	0			c3	0 -1	S	%

o 1.06 1.92 2.80 3.84 4.96 6.00 7.21 8.00

O 2.22 4.17 5.41 6.87 8.33 9.71 11.66 13.39

o 3.2s 6.50 8.79 10.30 II. 55 13.38 15.88 17.71

Ru Do, if we use the M.D. of Hillegas' judges as the unit, while it is 13.4 units above Ru Do if the " untrained " group are used as the basis, and if the 65 " trained " judges are used as a basis the difference is increased to 17.7 units. As was said before, the " trained " group were probably not more capable by nature than the " untrained." Four hours' discussion and use of the scale had merely made them agree much more closely in their

SUPPLEMENTING THE HILLEGAS SCALE 2 1

judgments of composition quality, even though they rated the thirty compositions without ever having seen any of them be- fore and without conferring at all about them. These differ- ences are shown graphically in Fig. 5 for each interval between samples of the supplement.

RD-M M-LB LB-MR MR-CP CP-LC LC-TB TB-MS MS-RS

1 ^^ 1.0c Hi M ^m -88 ■^1.04 ^■■1.12 ^HililA ■^■1.21 m -79

Fig. 5. Influence of training upon discrimination of differences between samples.

In concluding this discussion of the derivation of the Nassau County Supplement to the Hillegas Scale, the writer must explain that he does not advocate discarding the Hillegas Scale. It was impossible, before the Hillegas Scale was devised, to know that judges would be more variable on one composition than on another. Now that we can measure such differences in variability fairly well, we should get together a large number of compositions upon which judges do agree very closely. Just because the median judgment places a sample at the right point on the quality scale is not sufficient excuse for using it. Only those samples upon which people agree closely should be used. In order to get a very perfect scale, it would possibly be well to train the judges until additional training would not result in greater agreement among them. If the variability of human judgment is not a constant on which our scales for comparison may be built, possibly the variability of the judgments of col- lege graduates having ten hours' training in doing this work would more nearly approximate a constant for reference. For all practical purposes at present, however, some form of the Hillegas Scale, crude though it is, will supply all our needs in this direction.

The need for the second supplement, a tentative statement of standards to be achieved in English composition, was empha- sized very strongly by the differences between the results being obtained by the same grades in different schools. To find fourth and fifth grade classes acliieving results distinctly superior to eighth grade classes in a neighboring school is rather surprising

22

TEACHERS COLLEGE BULLETIN

until one stops to consider that up to the present no one has told these teachers what quality of English composition to accept in their classes. Teachers probably determine their standards most commonly from the compositions pupils hand in during the first few meetings of the class. In very rare in- stances samples are preserved from year to year and used as a basis for grading compositions from later classes. In the large majority of cases, however, a new teacher has no way of learn- ing what to expect or require. The random results which she obtains are illustrated by the records of the eight classes in- cluded in Table IV. Two of these classes are fourth grades, two are fifth, two are eighth grades, two are senior classes in high school, and all of them are in the same county within ten

miles of each other.

TABLE IV

Records of Eight Classes

	No. of compositions rated on sample of		Median	75%
In	value	Total
class		No
	o 1.8 2.6 3.7 4.7 5.8 6.7 7-7		Quality
A	3 27 9	39	3.80	3-41
B	I 14 8 I	24	2.94	2	55
C	6 8 12 4	30	6.37	5	48
D	I 8 19 4	32	5-66	5	16
E	S 23 19 I	48	1.99	I	31
F	34 7 I	42	3-8i	3	48
G	6 6 I	13	6.38	S	84
H	12 17 5	34	3-46	2.87

A and H are fourth grades, E and F are fifth grades, B and C are eighth grades, and D and G are fourth year high school classes.

The above table reads from left to right as follows: Three pupils in Class A wrote compositions which were scored on 2.6, twenty-seven pupils wrote compositions which were scored on 3.7, and nine pupils wrote compositions of quahty 4.7, making altogether thirty-nine compositions, half of which were of quality 3.8, or better, and 75 per cent of which were of quality 3.41 or better.

Notice in the above table the difference between the two eighth grade classes or between the two fifth grade classes. No- tice that a fourth grade class (A) does work distinctly better

SUPPLEMENTING THE HILLEGAS SCALE

23

than an eighth grade class (B), and that a senior class in high school (D) is quite clearly inferior to an eighth grade class (C). Certainly no teacher would be satisfied to have her class doing work typical of lower grades unless she were ignorant that they were doing such work. Knowledge of how well a class should do and ability to know when it was actually doing that well would seem to be prerequisites for successfully teaching a subject.

The purpose of the standards proposed in the following para- graphs is merely to serve as a guide to teachers, so that they may not be left as they have been in the past without any definite idea of the achievements to which their classes should aspire. Better standards will doubtless be proposed by others, but the suggestions which follow will be worth while if they serve merely to call forth criticism and more satisfactory standards.

How well do " Seniors " in high school write? What quality of compositions would you accept from " Sophomores " in high school? These two questions were asked of advanced students in Educational Administration. The answers were expressed in each case by means of the list of thirty compositions employed in the first part of this study. Without knowing the exact value of any of the thirty compositions, thirty-six experienced super- visors and administrators each selected two compositions from the lot, one to represent the average or median achievement of high school senior classes, and the other to represent just barely passing quaHty for the same grade. Forty-six other members of the group selected two compositions each, one to represent the average quality obtained from sophomore classes in high schools they had known, and the other to represent just barely passing or 70 per cent compositions for the same high school classes. The distribution of estimates made is shown in Table V.

It would not be sufiicient, however, to take the estimates of even the most able group of educators in the world as a basis for standards of achievement. It is necessary to know in addition just how well pupils actually write. The median results of composition tests and measurements on the Hillegas Scale are shown in Table VI for a number of typical school

24 TEACHERS COLLEGE BULLETIN

ESTIMATI	iS OF Quality Sample He Si	TABLE V OF English Compositk IN High Schools Senior classes	3NS TO BE Exp: Sophomore Median (8S%) .... 2. . .	ECTED classes
Value 8.0I	Median (8S%)	Passing (70%)	Passing (70%)
8.00	Re Si	. . .3		1
7-68	Adja	. . .3		c
7.51	LuRh	. . .9	2	, 7
7.47	Eu Sw	...a	I	... 2 . .	2
7.22 6.01	Mi Su ThBe	•••9 . . .1	6.. . ! .	13	... .2 8
5-89	ChFr	. . .4	3	8	II
5.76	ChCo	. . .4	8	I .	Q
4.98	ChDr		4		.... 2
4.97	LoCo		c	2
4.86	JoDr
4-77	CoO'T		2	2
4.36	JeMi		2
4.16	ChWi				2
4-03	RiBe
3-84	CaPe		. .... I
3.66	RoMcN....		I		1
3.31	Gr Mi '
2.81	MaRo
2.67	QuFr
2.54	Ha Da		I
2.50	PaMti
2.40	LeO'B
1-93	LeBo
1.06	Ma
1.03	Jo Si
.80	RoR				I
.68	FrKo
.004	RuDo

Number of judges 36 36 46 46

Median 7.35 5.49 7.12 5.83

systems. The lower line of this table shows the tentative standard medians proposed by the writer as the result of this study. Expressed briefly, the proposed standards show the quahty of compositions to be expected from at least half of a normal class of American boys and girls at the end of any given school year.

Some of the medians given in Table VI are shown graphically in Fig. 6.

It will be observed in both the table and the graph that the

SUPPLEMENTING THE HILLEGAS SCALE

25

TABLE VI

Median Achievements in English Composition by Grades (Hillegas Scale)

School system

Median score attained in grade

IV V VI VII VIII '"^ ^^ 3rd 4th yr. yr. yr. yr.

Lead, South Dakota^

Newark, N. J. (i school)^

Ethical Cult. Sch., N. Y. C.\

Chatham, N. J^

Salt Lake City, Utah*

Butte, Montana*

Nassau County, N. Y.'

South River, N. P

Mobile County, Alabama* . . . .

Mobile, Alabama*

Fifty -four High Schools*

Tentative Standard Medians .

357 411 4 2-39 2.51 3 4.01 4 2-95 2.85 4 3.58 3.84 4 2.34 2.80 3 2.76 3.42 3 2-31 2.55 3 3.20 3.91 4 331 3-^5 4

■H 5	or 5-
5^ 4	33 5-
72 5	39 5-
10 4	02 5.
.61 5	16 6.
■41 3	77 4-
.82 4	18 4.
.78 4	75 5-
■34 4	22
.60 4	95

57

27

74 High

29 School

37

II

56 s.oo 5.2s 5.68 5.94

62 5. 18 5.02 5.95 6.30 5.56 6.38 6.05 6.77 6.69 6.93 7.24 7.54 4.99 5.88 6.38 6.69

3.5 4.0 4.5 s.o 5.5 6.0 6.5 6.9 7.2

' From Annual Report of Superintendent of Schools, 1915-16, Lead, S. D.

'' From unpublished studies made in the department of educational administration. Teachers College.

3 Calculated from the distributions published in Cubberley's School Organization and Administration. World Book Company, Yonkers-on-Hudson, N. Y., 1916.

•• From Strayer's Some Problems in City School Administration, World Book Com- pany, Yonkers-on-Hudson, N. Y., 1916.

6 From unpublished report of the Mobile School Survey by the Bureau of Municipal Research of New York City. The Mobile County scores are for the white pupils outside the city, while the Mobile scores are for the white pupils in the city schools of Mobile. The seventh is the highest elementary school grade in the Alabama schools.

"From master's essay of Albion Hale I3rainard, prepared m the department of educa- tional administration, Teachers College, 1916. High schools from over 35 different states are included.

standard is higher than the majority of the schools have actually achieved, although at each grade one or more school systems have shown higher achievements than the standard suggested. If some schools are able to meet this standard without making particular efforts to do so, many schools should be able to meet it when they make definite efforts to do so. It seems desirable, at least, to set high standards " to shoot at " rather than to make the standards conform more closely to the achievements which the present more or less haphazard teaching efforts secure.

There is, in addition to the above, a real value in having a secondary standard to indicate the width of the distribution of scores to be expected in a class. There will usually be a small

26

TEACHERS COLLEGE BULLETIN

8.00

7.00

6.00

5.00

4.00

S.OO

2.00

4

SCHOOL GRADE 7 8 9 10 11 12

							^^
				/	/ / / /	X	2^
		A	/ ■'/y		%	/	""
/	/	6,	> /	s*/
	/

NC-N»SiAUCa

Fig. 6

number of pupils in each class who will be unable to approach the median standard, and for them a lower " mark " may act as a stimulus. For this purpose, the writer suggests that 75 per cent of each class ought to write compositions of or better than the quality indicated below:

Quality of English Composition to be Equalled or Excelled by AT Least 75 Per Cent of the Pupils in a Class

In grade IV V VI VII VIII IX X XI XII

Quality.

30 3-5

VI

40 4.5

S.o 5.5 6.0 6.4 6.7

SUPPLEMENTING THE HILLEGAS SCALE

27

The qualities suggested above for 75 per cent of a class are in each case just .5 of a unit lower than the qualities suggested for 50 per cent of the class. This difference between the achieve- ment of half and the achievement of three-fourths of a class is practically the difference one actually finds in classes as they are at present. The chief sources of proof for this statement are the tables of distributions in Nassau County, New York, where the score on each paper was practically the median of independent ratings by three well-trained judges. It would be somewhat unsafe to use distributions based on a single rating of each paper by an untrained person. Taking the combined scores from thirty-three Nassau County schools and consider- ing the distributions of results as though each grade were a single large class, differences were calculated by the method indicated in Table VII.

TABLE VII

Differences Between Achievements of 50 Per Cent and of 75 Per Cent of a Grade in Nassau County

School grade	IV	V	VI	VII	VIII	IX	X	XI	XII
No. of compositions . Median (50%) score. 75% of grade above.	1046 2.76 2.25	930 3-42 2.72	894 3-82 3.16	741 4.18 3-49	607 4-56 3-79	554 5.00 4.42	332 5-25 4.60	237 5.68 4-95	i6s 5-94 5-42
Difference (50-75).. .	•51	.70	.66	.69	• 77	•58	•6S	•73	•52

As was shown earlier in this discussion, being in the same grade in different Nassau County schools does not mean doing the same sort of English composition work. The above differ- ences are larger than would be found if we were dealing with actual classes or recitation sections. By taking all those recita- tion sections in Nassau County containing more than 25 pupils in the fourth grade, 20 pupils in the eighth grade, and 12 pupils in the senior high school grade, the distributions of differences between the achievements of 75 per cent and the achievements of 50 per cent of the pupils in actual classes were found to be as shown in Table VIII.

From Table VIII it will be observed that the requirement is not too high, which suggests that the achievement of 75 per

28

TEACHERS COLLEGE BULLETIN

TABLE VIII

Frequency of Differences Between Achievements of 50 Per Cent AND OF 75 Per Cent of Actual Classes

Size of Differences	In grade In grade In grade IV VIII XII
. ^0—. ■?Q Hilleeas units	3 3 5 5 2 4 3 2 7 I I 2 2 3 I I
.40-. 49
. KO-. ^Q
. 60-. 60
•70-. 79 .80+
Median difference'	■59 -46 .52

' Calculated from actual cases rather than from above distributions.

cent of a class should be within .5 of a unit of the achievement of 50 per cent of the class.

For the aid of those who may be unfamiliar with the method of calculating the 50 per cent or the 75 per cent achievement of a class, when the scores have been assigned to the papers merely in terms of the nearest samples on the scale, the following table is given.

TABLE DC

To Facilitate the Calculation of the Achievements of English Composi- tion Classes Measured by the Nassau County Supplement to the HiLLEGAS Scale

Sample Actual value Range of step' Length of step Sample
0 I.I			. . 0 . . . . . . 1 . 06 . . . .	.... 0- .53 ... 54-1 -49- ■•		-53 .96		.. . . 0 . ... I.I
1-9 2.8			..1.93.... ..2.81....	. . . .1.5CJ-2.36. .. •- --2. 37-3. 32...		.87 .96		.. ..1.9 ....2.8
3-8 S-O			..3.84.... ..4.97....	....3.33-4.40... .. ..4.41-5.48. ..		1.08 1.08		....3.8 ....S-o
6.0 7.2			. .6.01. . . . . .7.22. .. .	. . . .5.40-6.61. . . . . . .6.62-7.61. . .		I -13 1. 00		....6.0 ....7.2
8.0 9-0			. .8.00 ..9.0 ....	7.62-8.50. .. ....8.51- ...		.89		....8.0 ....9.0

' The third decimal point is omitted from these tables.

To show the method of using this table we may take the following distribution of scores as an example, and calculate the achievement of 50 per cent and of 75 per cent of the class.

Scored on o i.i 1.9 2.8 3.8 5.0 6.0 7.2 Total

Frequency 3 16 25 4 48

SUPPLEMENTING THE HILLEGAS SCALE

29

The median achievement will be the quality above which there are 24 papers (48 -^ 2=24) and below which there are 24 papers. If we count the three which were rated on 1.9 and the 16 which were rated on 2.8 we will have only 19 papers, and in order to reach the middle of the distribution we shall need to count 5 of the 25 papers rated on 3.8. We may assume that the 25 papers are spread evenly over step 3.8, extending (see Table IX) from ^.;^t, up to 4.40 at equal intervals. If we count out 5 of the 25 we shall pass over one-fifth of this step which is 1.08 units long; in other words, we shall have gone upward from T,.^T, a distance of .22 of a unit (1.084- 5=. 2 2) and we shall therefore have arrived at quality 3.55, which is the median quality of the distribution.

In order to find the achievement made by 75 per cent of the class, we shall need to count only one-fourth of the way into the distribution, — that is, we shall need to count out the lowest 12 pupils (48-^-4=I2). After counting the 3 papers scored on 1.9 it will be necessary to count out 9 (12 — 3=9) of the 16 scored on 2.8. Assuming that the quaHties of the papers are distributed evenly over step 2.8 from quality 2.37 to quality 3.32, we shall count out nine-sixteenths or .56 of the step, which is .96 of a unit long. This will take us to a point .54 of a unit (.96X.56=.54) above quality 2.37, which will be quality 2.91, the achievement made by 75 per cent of the class.

The table for the original Hillegas Scale which corresponds in its use to Table IX is given here as Table X.

TABLE X

To Facilitate the Calculation of Class Achievements on the Hillegas Scale for English Composition

Step No. Value of sample Range of step Length of step Step No.

o 0.00- .91 91.

1.83 92-2.21 I -SO-

2 2.60 2. 22-3.14 93 2

3 3 69 3. 15-4. 21 1.07 3

4 4-74 4.22-5.29 1.08 4

5 5 85 5-3^6.30 1. 00 S

6 6.75 6.30-7.23 93 6

7 7 72 7.24-8.05 81 7

0 8.38 8.05-8.87 82 8

8 9.37 8.88- 9

30

TEACHERS COLLEGE BULLETIN

It is not proposed that any teacher should use the Hillegas Scale in any form as the basis for all her marks in EngUsh com- position. The scale is a measuring instrument and not a peda- gogical device. In her instruction a teacher may at one period be laying stress on punctuation and give her class marks accord- ing to their success in punctuation. At another period the teacher may be placing her emphasis upon the choice of words, and the class marks may well reflect this emphasis. But at some two or three periods during the year teachers should meas- ure the general results that are being obtained by this instruc- tion. For this purpose a number of teachers should cooperate with each other in rating papers on. some form of the Hillegas Scale. When such practices become more general, we may hope that results in English composition writing will become uni- formly better.

APPENDIX

More than five years have passed since the above discussion was first published. During this period a considerable amount of real progress has been made in the science of measuring the quality of English compositions. Professor Earl Hudelson^of the University of West Virginia has been one of the most fruit- ful workers in this field.^

He has found, for example, that pupils write compositions of a much lower quality when they use the " What I Should Like To Do Next Saturday " topic than when they write on such a topic as " My First Lie." The average results obtained from " My First Lie " are one full unit better than the average results from the topic on which pupils in Nassau County wrote. If an eighth grade teacher secures from her pupils compositions on the shorter topic, she should subtract i.oo from their median score before comparing it with the records made by pupils on the " Next Saturday " topic. Such facts indicate the necessity of having a uniform topic or set of topics for testing ability in English composition, and the importance of following uniform directions.

Directions for Obtaining Compositions Have each pupil clear his desk and provide himself with pen, ink, a sheet of the usual composition paper and a blotter. The pupils should be asked to write (at the top of the sheet) their names, ages (at last birthdays), and school grades. If papers from more than one room or school are to be scored at the same time, it would be well to obtain from each pupil before the test begins a written record that will fully identify his paper. While the pupils are writing their names and other desired informa- tion, the examiner should write one of the four following topics on the blackboard. Be sure to write plainly and where all can see.

The Funniest Thing I Ever Saw

The Saddest Event of My Life

The Happiest Hour of My Life

The Most Unpleasant Hour of My Life

'Hudelson, Earl, Aims, Methods and Measurements in English Composition.

32 TEACHERS COLLEGE BULLETIN

Call attention of the pupils to the topic in the following words : " I have written on the blackboard the topic for a story. You are to write a story about the [topic]. You may make up a story if you need to, but most of you can tell a real experience. You may have twenty minutes in which to write. See how interesting a story you can tell."

All papers should be collected at the end of twenty minutes, whether the pupils have finished writing or not. However, in assigning scores to the papers, nothing should be subtracted from any pupil's mark because of an incomplete story. Each paper should be marked on the quality of the writing actually done.

Compare the general quality of the pupil's composition with the general qualities of the various samples on the scale. Assign to each composition the numerical value of that sample on the scale which most nearly equals it in general merit. Go up the scale until you reach compositions that are distinctly better than the one being rated ; go down the scale until you find composi- tions that are distinctly poorer than the one being rated ; and then compromise on some value between these two limits. In order to secure accuracy, it would be wise to have each compo- sition rated on the scale by at least three competent judges, each making his rating without knowledge of what the others have assigned. The average or the middle rating of the three judges may be taken as the one to represent the final value of the composition.

Directions for Improving the Judgments of Teachers In order to improve the abilities of teachers to judge the gen- eral qualities of English compositions, the other twenty-one compositions (rated by the 139 judges but not included in the Nassau County Supplement) are reprinted here as practice ma- terial. Let each judge read one of these compositions, compare it with the samples on the scale, decide carefully the numerical value that seems appropriate to it, and then turn to the Key on the last page of this booklet and compare his judgment with the value quoted there.

One should not be satisfied to discover that he has over- estimated or under-estimated a composition, but he should care-

SUPPLEMENTING THE HILLEGAS SCALE 33

fully re-examine the composition and the scale to see why the median of 139 judgments gives it the value that appears in the Key. After discovering why one has made an error, he may try again on another composition, preferably not the one imme- diately following, because of the danger that he might remember its value from the Key.

A careful rating of ten or twelve of these practice composi- tions, followed in each case by a comparison with the Key and a thorough re-examination of the scale to see why one's ratings differ from those in the Key, should reduce one's " personal equation " considerably. One may consider himself a reasonably competent judge when the average difference between his ratings and the true values has been reduced to less than .5 of a unit.

The true values of the following compositions are given in the Key on page 38.

Ri Be 4.03

We have all ready secured bases and other things to have a good game of base- ball and if we finish the game in time we will go into the woods and make a hute and have a kind of party. We are going to have cake and crakers. We hope to have a good time for we are going to have games of all sorts ,

Ch Co 5.76

My desire for enjoyment next Saturday would be a camping out party where I could obtain plenty of fishing, hunting and water sports, of which I am fond of.

I would like to have my cousins accompany me, as they know a great deal about fishing, and know how to track wild game, owing to their previous expedi- tions which they made with their uncle last summer.

It seems to me that our journey would be all the more pleasanter if we traveled by canoe, and make one or two portages when possible.

I think this would be a pleasant way to spend next Saturday, as nothing is more injoying than to be out in the open country, where you can breath the fresh air of the woods and feel at home once more.

Ha Da 2.54

I intend to go next Saturday is to sleigh riding, and part of the time to go ice skating if there is any ice skating, do my lessons, do my other work, and help my mother^ help my brother, and sister.

And do my poia lessons, and take my ponia lesson, and go down celler and chop wood for my mother, go the to the store for errands for my mother, and wipe the dishes, and bring up coals, wood for my mother, if I can make a kite and when it dries I will fly it for a little while and come in if it get cold and get warm and go out for a nother Httle and do the same thing if there is any more thing to do I wiU do it.

Jo Dr 4.86

Next Saturday I should like to Play ball in the morning till about 10 o'cloack. Then I should like to go to the beach and go in for a swim. In the Afternoon I should like to go to the city and see the ball game between the giants and Phila- delphia. In the evening I would like to go to the the theatre.

^4 TEACHERS COLLEGE BULLETIN

Ch Dr 4.98

If the weather be clear, my intentions are all planned out for next Saturday.

In the morning I have made a promise to pitch, (for a team just selected of fairly good players), who are to play the " Happy Nine " of Bullshead. The game is called at nine o'clock and the players are to be there about eight, so as to give us a chance to get warmed up (as base ball players calle it).

I am a pitcher that can not be found every day of the year. As I have played in a great many games of base ball. I have been practicing all this week after school hours. And think I can do the trick.

Ch Fr 5.89

Next Saturday I should like to have a good game of baseball because I like that game very much.

I would not care to win a game by unfair means nor by a large score but some- thing like one to two or one to nothing in our favor with every man watching his own game and playing to the best of his ability.

I IDie baseball because you have to be quick to play a good game and keep your head when you are playing against odds or when your are in a tight place — for instance You are two runs ahead of your opponent and they have two men on base and a good batter up to bat in the last half of the nineth inning, — then a good many pitchers and players get discouraged thinking there are no chances to win, but the ones that stick to it until the last man is out is the man or team for me.

Qu Fr 2.67

I am going to play ball and help my mother and I am going riding, deliver papers

then with my money go by a new ball. then I will feed my rabbits and chickins after they are feed I am going to Scouts

an practice then when I come home I'll go play. At night go to the movies after movies go get a nickels worth candy

Ad Ja 7.68 What I should like to do when I leave High School

My highest ambition after I leave high school is to enter coUege and successfully complete my course there and then enter upon a career of service for others. From a child I have always longed to be among the poor and lowly not at first for the purpose of doing missionary work but because my life had had so much happiness that I can not bear to think of anyone being unhappy.

I have always had a great admiration for missionaries but their lives seemed to be so set apart, so far above anything I could become or hope to become. Not until a few months ago did the idea come to me with any great force but from that time on I was determined to become a missionary not only that I may share- my happiness with others, but to teach them of the loving God, believing that through Him, they may be brought into lives of happiness and usefulness.

Fr Ko .68

What I should do Saturday play boll would The flower hil in the afftermoon and haue A chatch would the chatch on the boll lot I am going to patch on the manhasset team We wone the gam 16 to 4 and thun we went home

Ro McN 3.66

Next Saturday being that I have an holiday, I expect to go to Washington D.C. to spend Saturday and Sunday looking at the Pension Building, Libraies and the Captail etc. Into which I am very much interested in.

I would like to see the large streets, and especially visit the cabinet the large room and hear the statesmen talking

SUPPLEMENTING THE HILLEGAS SCALE

35

Gr Mi 3.31

I should like to-do next Saturday to help my mother and clean all the house and sweep the floors and scrob. and wash windows and chop wood for her and carrie it in the house and obey her and my father. Go to school hight on monday morn- ing and to store and go out shopping with her. And if it is early mother and I would go to a show. Then I should help my big sister and clean house for her and wash windows and sweep and shake all the carpets and make the beds and open all the windows. Then mother and my sister in-law would give me fifty cent a piece.

Je Mi 4.36

I should like to go a way for a long journey to California. And see the nice fruit trees and see how large they are. It is Very nice out there. For I have friends out there and they say it is very nice?

There is large trees out there that they have made tunnels through and places where horses can go through

There is lots of apple, and Oranges in California They send a lot of fruit to New York. I an in Roslyn now and do not like it Much? I should like to stay there for three months but I think only can stay for a few Weeks. But all the same I like to go and see my friends?

Pa Mu 2.50

What I shall do Saturday

I shall go out to my aunt in Linderhurst

When I get out their, I will go out to the chickens and then see the two dogs

When uncle otto comes home from worke Saturday night. Wee will

get up early in the morning

and get the chickencoop

clean And in the might I will go to the move picturs with uncle otto

Le O'B 2.40

I am going to school tomorrow. Are you going to play with me.

I am going to the theatre to night. Are you going to night to. I am going home

If you will play with me I will go. Alright are you coming now to play Don't forget to come. " Yes, I won't forget to come. I am going to play school at home. I will go with you I won't with you. Are you playing with us No I ant playing with you.

I Am going to play bass ball. Are you going to play to. " Yes, I am going to play.

Co O'T 4.77

Next Saturday a number of girls in my class are going on a tramping trip, sometimes called a Nature Study trip. As a very merry and jolly bunch of girls are going, I should also like to.

And if I am not allowed to go on this Trip, I am making reckoning on spend- ing The day with my Aunt, For in The evening. Some friends of hers are planning to bring a surprise party for her.

But as it is I really do think I would rather go on The trip, For I am sure I would have more fun.

RoR .80

I intend to go slealy rady next Saturday and mach a baby and pout a lout of chilon on.

And aftir we done we will go home and mach a big fout and mach two and we will have a snou ball find and we will rush om the eneany and we will catep there fout and we will traif thin home and we will ceap ther fout fou a price and there will mach a nether and we will fout thin again and we will have a good time

36

TEACHERS COLLEGE BULLETIN

Lu Rh 7.51 What I should ^ike to do when I leave High School

When I am graduated from Oceanside High School, I should like to study Chemical Engineering in the Carnegie Institute of Technology at Pittsburgh, Pa. I should like to finish my work at college as soon as possible in order to begin the more serious business of life. This work has always held for me considerable fascination, and I am deeply interested in it. Material wealth and riches need not be the result of my choice of occupation, but rather satisfaction with my work, an interest in it and, above all, the realization that my life has not been wasted.

Ec Si 8.01

Saturday I should like to transport myself to one of inumerable glens dotting the Southern portion of our country. It would be a shady, cool retreat, where moss covered rocks form natural seats, — where a crystal brook would babble and murmur of the wood creatures, — of the plants, and of the spring, as it winds its way among the monarchs of the woods, — where birds would serenade me with their sweet, warbling melody. And there I should study the wonderful manifes- tations of the power of God, — the bubbles of snowy clouds floating across the infinite surface of the azure sky, — the animmals and their homes. And then, per- haps I should roam thro the leafy aisles of the wood and study the inumerable flowers and plants, and breathe nature-given balm — their perfume. The colorful pictures, — the sky with its exquisite tints, — the flowers, — the birds, — the brook, — would give me more pleasure and information than the greatest pictures ever painted. Such is my dream of a perfect day, — and to sleep beneath the gleam- ing stars set in the soft, dark velvet of the sky, and breathe the invigorating airs of the woods, — its perfect ending.

Jo Si i.oj

1 ever af ten noon Three olock want I go home I took my gulf club and play. And I had lot of fun and sum Time I play bast ball in the night. And The Saturday I go to the gulf, club want I go there we bast ball and. I make lot of money and in Sunday

Eu Sw 7.47 My future plans

As a general thing a boy of fifteen or sixteen has no definite plans for the future. His ambitions are extremely eccentric and in most cases absurd. I believe that one of youth's first ambitions is to become an actor. He attends some theatre or perhaps the circus and is deeply impressed by blase of everything. This passion usually last about two years. A boy's future depends upon his temperament and the impressions dictated by in the ordinary course of life. I am glad to say that I have emerged from the previously mentioned age and after careful con- sideration have come to the conclusion that a course at either West Point or An- napolis is the zenith of my ambitions. On the whole I prefer Annapolis but if_ by any chance I should be unable to secure an appointment there I would not object to West Point. Both Aacdemies offer the widest opportunities to a young man both from an educational and fundamental standpoint. I do not believe that a young man could at the present time chose a more honorable or opportune future.

Ch Wi 416

I am going to playball, in a corner lot near are house. I might go to Great Neck in the afternoon if it does not rain. I would like to go on a trip to New York. I think I can go. I would like to go in the woods and pick May flowers and Violets if they are out. I should like to go and see a ballgame, and a good one too. I might can go to the railroad station and see if it is a package there. I would like to see the Great Neck Scouts play ball tomorrow.

SUPPLEMENTING THE HILLEGAS SCALE

Additional Data on Standard Scores

37

The younger pupils in each grade will usually write better compositions than the older pupils, because they have somewhat greater academic ability than the older pupils. All results from academic or intelligence tests should therefore be tabulated by age within each grade} If a class happens to be composed of bright young children, the tentative standards given in the original discussion (page 19) will be too low; while if the class is com- posed of retarded older pupils, those standards will be found too high. The writer recommends, therefore, that only the normal-age group in each grade be used for comparing one school with another. The six-year-old pupils are to be considered normal for the first grade, the seven-year-olds for the second grade, and so on through all the grades. As a matter of actual fact it often takes more ability than an average pupil possesses to be in the eighth grade of the usual school at age fourteen, but theoretically fourteen-year-olds are of normal age in the eighth grade.

The statement of median scores given below includes, there- fore, not merely the tentative standards proposed by the writer five years ago, but also the average of the actual qualities of all compositions measured in each grade, and the median achieve- ments of pupils of normal age in each grade. Older pupils should not be expected to do as well as, and younger pupils should be expected to do better than, the normal-age group in each grade.

Median Achievement in English Composition by Grades

Grade IV V VI VII VIII IX X XI XII

Standard (Tentative) 3.5 4.o 4-5 5-° 5-S 6.0 6.5 6.9 7.2

Actual Average 3-° 3-6 42 4-7 5-3 5-2 5-9 6.3 6.7

Normal Age Pupils 3.0 3.6 4.1 4.6 5.1 5-5 5-9 6.3 6.7

The above scores represent the work of each grade at approxi- mately the middle of the school year. Some allowance would

'For evidence on this point see the writer's report on tests given in St. Paul (Minn.) School Survey and in the Baltimore (Md.) School Survey.

38

TEACHERS COLLEGE BULLETIN

need to be made in each grade for results taken at the beginning or at the end of the year.

Sample Ri Be	Value 4,0?
ChCo Ha Da	5 2	76 5/|
To Dr	4	86
ChDr ChFr . ..	4 c	98 89 67
QuFr	2

Key to Compositions on Pages 33 to 36

Value 7.68

Sample

Ad Ja

FrKo 68

Ro Mc 3.66

GrMi 3.31

JeMi 4.36

Pa Mu 2. 50

Le O'B 2.40

Sample Value

CoO'T 4.77

RoR 80

Lu Rh 7.51

He Si 8.01

Jo Si 1.03

Eu Sw 7.47

Ch Wi 4. 16

LIBRARY OF CONGRESS

0 019 840 738 0

_ - ^ _^ ■ ■% *Tl!^ '"li >ll

f ^1 -"L ^T^ '€ "^ ■■€ € ^ € -A '^€ € ^ •% ^'t"^% % 4i ^ ''M 4 ''^ % 4 -i §■.€.•€ M 4 4 4 4 ^ # # %. -m ♦li ^^k ^ •

■1 ^ ^ M .^:*.iL..,:t..^-t^.-C^.m ^^ ,,^.,4- %^.# ■-* '^ir '^^k ^ ■

^ .*^ -^ •.€ 4 -^'^^ C ATI €. :A .;4 -^^ i:4: # * ^* ^*> ^*4 '^ .

d .4 .;« . ,^4 ., C^M ..;A ^^ ^4 .^*^^ ^'ft. ., i, ,^^ ,^4 . * , # ;^ .^% ^** ^^

4 4 ■*! .'^ 'f jt '€ 4._ 4 fl .:^^:4 ^ ^4. 4 # .# 'm "^ ^ ^ ■

€ ^^ k1 -4 'm 4 € ^ ^C ^€44 *i- -^ #^ -# ■% "*! "?^ • 1 ^ ■•^l ^1 :€."€. M._-'^ :1 ,^4,^ '^i % ^i ^* •# #■ 4i/* '#,-^ *i -^ ft 4 4 ^^ €,^^ :<€ € 4 A' % A 4 .'^ ^ 4 % /^ ^F

' "^ ^#j ^1^ '^^^ % iCi; -^ ^^^ ^ -^l ¥{ 'j^ €[ ^ ^ ^4 :^ -'^ -4^ a} :^

%1 ^4 .%J ^*l :4 '^ ^ '1 7^ M -^ ':^ ^% ^^^ '«1- ifi ^^: fi '^ %

i € -M ^ M 4 ■^^ ^ M -^ ^*i ^*^ ^ 4 '^ 4 «^; -4 "^ '^ '4 '^

f 1^ 4 ^1 'm./m m m 4 ^^ m ''€ ^ ■€ '* m :^'AM/%

.1 M ^^ M ;^ >1 4 "ft ^ ^€ «] ^ -^ .^ s»l ^ M 4, .^ 1^ Ji 4

M 4

;• JX

n ^'^ >*1

■41 :^ m ■»#].

^ -^^^ ■"" '"^ ..^ '.^ :ii! :^'-^' m n 4 ^- ^ ^ ,,.,, .., 4 -^; -^ ^4 $1 -^ . ^ -^ 1 4 % ^^

1 ^ '^ ^ 4 *I ^^^4 ^€ *^>

tl '^ ^ ^'t M U M ■:0^ M ;^ :^ M- v®!' ^^ -^^ ^

^ -^ ^

^ ii 'ii

i' '-^ ^^! '^II 1

'M M # ^

^^1 m Ml '¥{ 'tf' "^

^l' -i\^ ^'%. M ^ :^ M-

m> "M ''€

. .:t ;* ,:^; ,^.. .-.

:$^ ^ -^ ^ ,t| ^; ■^1 <«J 11 ?!S)- .^ ml "M

fl m

M m

€i '^i ^i ^>*'

1' 'M ^

1 ^i .'l

^! ^:

#: II'

"1^1 m m M: m- ■

I m H '^' -^1 ^. ^' '"wi #5

■;;g '.>^i -^s;! '\^

h .?^J "J» ':% '^Ji^ '^,,,^ ...^ ^^^

' >3fi &i ^.^j ^i 'm '<fi '^^( ':;i ,

-> ^> #^. *X

;#:- 115 -m^- 'W^' '^ ^

'is "^t 'vf' ' '"" ^^^"

E;^. *:*! #^ M ^1

^- fr^^ ¥ !'■ t* {-

W. ¥ .> !^ l*^ 1?^

,, ,li: §. 'f': p; p p p

^ #. 'Ip fe- P' p- fr^ P^~. F"' P F- > r- i* ^ §*'■ fc^- 1^' .Ii^ \P' \f^ W ^: # # i^. :^ j^^ i ' m p^ ¥■ P' ¥' P- P P- if P \^' »*

# §^' # 'fe^ |:' i^ i^i-' -^ ^' w p ¥' P y- W fr ¥' W ^ W I^ \-

" '^^ f^ ^' ^ fr' .f^' #* 1^' -P P l^ P' 1^ 1^' P- #• l^- P^" ^' W' '^^

fr' 1^^#' fr'" '1^" P P p W^ P' ^'' *' ^'' ^^'' *"ffe p- \^- ^' ^^ ^. i#^.' |t- ^. fr'^ fi^: ff^ i^^'-W' W W. iP "'^ '■ '^ ^" E^ 1^^' ^'. W if ^r.:

l'^. -^|i^.. .[p .^ {t.> i f^^^*^ ^'^^ ^*' ^"" '■ ' "

-^^.-i^r ^^;-^. ¥' ¥. .^-.. P

■. 3^^ [?!■ 1^.-^ ii^ pr fr

, fr,>,, &v ^^. *; *v -^v. ^,

m p Ip' px f '■" ■■

•If; !#:" l^-' ^^^ f^^' I^ ^' (i^' p^" I? :% ftt j;*. ' ■% liv' , ,

is" -l^;. J#, r|j!. ^^^

■^- ^ ^^-^ *^' V }i^ .#; ^i I

Iff It

i', (# ,#

■. -.^^ 1«/r-. ^-, ).-■ o. >.-. y jlfe #) [liv [t^ !#. i|^: F 1§ j

■ '^^' # ..*. ilr. 'I^-: ii.:/ -W !^

^ -J' -^z i^'' te, *' ■!*'■'. fr.^.,.

f. ".«.•! 1>;. 'iav, i^Saf.v .*.. kS>.v 'itej. ' ' ^^^ f- '

^■^ #^ "'^If -^|s ^^%V 1%

(^/ it, fi^'' J^\" w %■ ^.;^ 'ii.^ it;' '-B' "''"

#''5j; ■ % ■%■ W ,•.

.-> pS' \m- m ;:5^' i*^ i^.' t^

'V. \t. i4|. %, ;;|, ;|^; ||^ ^-

;it m ;. .-.-. ...

k' '^^■' *^ 't^'' .:^v w U^ ;=

(i;;: 1*. i^'. Ife: li'' •(% %

B- # ^^ !f It il^ .i^' ,.;'

..; is-,.' V;^ ., U.,

' ^^: m. % #^ v# it' #• ;l^i^ !t^ ;#. % ]$.■ i

^^ %' f<|i. ' • »^: x%r ..vIj- ■^^; ft;'' %^ fi/' -i^i 1;%' i^j:, ;,

4* m'^-i'^'ii 'm' :g.; .^^ ^' -im m '%ir i^'-^- ,.., ,, ,.,. ... 11;^ 4s#' v' %:' vf.; -m ^# [i>^ '""'■■ '"^ ■ ■■■* - ■ •- - ^-^ ■'-••■■■

*l# i% i^i^^ ^|5/ ?i|te u^^ '■ ^' *M^ *^: 4%, %•■: ^# .J

!&/!%:. t|;.^ ^#/>^%^ •%'■'%' '^iJi '#:"" -(fe' \(|r' W "W ^*f 5W i-^' r.v If- ■'^: '?^- '^L 7 fe i|:' t^^; ^s k.^' %' '*^' -ilkr -ilJk^ 'p-i ^:' *> ' '^^ ' '®^ '^ ' ''^ " '*'■" ^■^•' '^'^ ''^■' 's- ■ ■ '&. I

^/ 1:1=, Itv" 1^ #\v ,„. •;.. ,j*v 'la. p^ \$i- i*>i^. te '^is^' 4^' 4^' -^^:r W h' '%' "f^Z-^i

m t^:- I* i#; 4^-' 4i:' ■ ■ ■,■ p^' 'U i^- ).>. i^" 1.. ,. ,- ... ,.,. ... .. ,

i^; lim " Afe %, il;. %■ ^.. i,,. :.-. (|tc ..(Ij ^.; |s^ j^,.. 1% ■%, l*^.. -V^, ,t.. .|: ^. \ ^

It^ 1^^ l^', |4i W 1%^ W ^:' ^" '#■ \r ft/' 5;^? i^'- .1% ^^- %■ ik^ ;|i ■ W if.'

iiii i$v m^- m^ #'^ -V. T^sz-'fife- ,1^' :p. # ,|^. |i> It:' It' p fi riN.' •?*•. ;r; i^ - V l4^\ 1%.. tfe; I^V' 1^ ■ #.%■%' ^1^ ■ %;-v\.?^. 1^" ^!^--.v|,-" itr ii ;fc ir ^i ;.., , 1^, i^. li^ ,%.,viiiv-<50- -^iiwjw: 'i^'^r!^^^ ■*••'#■ ilf*? ' ■ ' ' '"

*'-^ a;^ jfe^ t,g.; fe t#:. Mf fe' i*i^.' »l^^- '■-^*' ■■^«- ■•'^ ^

... v.. ;^. -i ^1^.' i'!^.

i&. ■ f ?i? . L%. ' '|i;, ^'^:; 'ji^; I

J> 9'^ f$- >■ ;>. -1*^ ' \*.- '" It- ' li: ,'fe ■4

i,c »i,n ' *3 ^ .'4 .^v*}' ■=* "'*

^I 4 -^ >I 4 ■% ' :^ M M^ ^ ^A 'A M i4 '# v»| «^ -^Af 'nil ^i

■^ 4 4 ^^^ € •'i 4 ^4 4. € €."^:.4 ^ 4 4 ^l >^' :^x ^ ■■* -^

t ^ 4 *f S ^€ 't •€ ft ^. •* M '^^ .4 ^4' *1: , »l *].'% "*)^ '^Ji;

^' q (4 -^ ■■€ ^ 4 t « t. '4 « ^ € '^ ■■' i' .^^^ :% "^k '^

0 ^ '4 -^ 4 € 'il '^^l '%. M 4 '^ ^ ^ ^-^l -4 4 ■'^ '^ir ^

;^ 1^ -^1 ^^ t 1 "a ^ € € '4 ^M ^^ 4 4 .^ *1 ^ % .# ^

€ '^ >3 4 '4 -W 4 '^ '^ ^M •*•! 4 ^^^s: ^ ^^■^' ^ ■# ^# 4 *i i,iA ^^ 4 -fl ^^ f ^4 * 't ;^^ VI -M :*l ^^ *^ v't] -1 ■:«! ^. -^ ,*1 < ^i! 4 € ^^*i' 4 •€ ^^ '* ^- ^t n X! ^ ^ :A A <^-4- ,*;*,*, ;^i^'.il 4 ^§1 *l '^ il t € ^ -^1 ^ «i ^^ ^ # '< ■^ 4 -ft '^^ -^

; €-'^ n 4 "1 '^ft ^'^ '^ n 4 '^ '^ ^€ ti 4 , . :^ .,^^*i.:€,

#i -^ ^ ^^ '^\ H M. §1 M ^^ ^1 ^- '^i -^ M ^i m- *): -'^ »l 4 ^ ":"■*! ^ ^j %; M- % "-4 '^^5^ '* >*l -'^ ^ -^l "ti ^- '^ '^ '^ ^ "^ "^ . .. .:n -^ -.1 ■ .^ ^ 'li t; '^fei .;^ .3^1 ^ Jil ^0^ .m % 4- 4' -4 ^'i

.. -. '^^ ^ -f] -^H '^ ^^; '4 ^4 % -^ ^ -^ % 4 %

^1 -^ n 4

^i m

,:■ m ^ #1

W\ ■^'- ,■
■*1	■■■■
	^^^^ ^^m9m	• ^^^« HHH
:: ■■	■ ■
^ «■ j ^^"
^ t 1	■■	^^^^^^^^
■■*i 1 . 1	■ ■
)■■■*.
Al i	^^^^^H ^^HH^^B
	«l
:^ 1 ■ ■	■ ■
H B
^^^^^H ^^^^^1
		^^^^Mi H^H^B

'■:«1 ?»i ,:^ ^ti *i •^. ^ ^^. ■

^:ie .^ :^ M ■* -4 ^€ "^ '' •'"' ^-"^ -^ 'iti. 'ij '^^ ■^' ■"■"

.#» #i

'^: 11.^ -1^

m, m ^ '^ ^ '.:tj ^m, 'm\ ^. i

^' '^i

"%; -^i

% 4

"ii V- %>.

■m\ "^t

''M -*^ M

■w m

	M ''^ '-^		••in mi		'^:- ^^: m 'M ^1 ■:^j. ¥1 1i			i:	■%^. m^. ^^ -m, %f ::^:. p^- 'll,- '^ ^^^ ". 'hi?: #%. #li #t 'W mt ^§
' *1 ^"t JS?'^ ^*' 'Ok-' * '-^tl *'i«^l • ■!*■
	1 m m				> v.^ :«»■• i.m, '.^f; ^^Wl "^ '^
			■is'		»i # *\ #; *■; .c*\ •;=£' -fei

>r ^^;i % -ii %

^ ^!^ '*i_''fl. '^^ ' t:l

^ ^l ■^L'^^

11 "^t

i§>^ ./viw. #s i*i> •^^'i "^

■^i'.ji#'^i aft •■