AN EXAMPLE OF AN ASSESSOR TRAINING PROGRAMME



‘Here is an example of a typical training programme for the selection and training of oral interviews/assessors on the regular teaching stuff at a large industrial training centre. The programme takes the form of a series of weekly one-hour meetings. Note how the staff who are going to use the oral test procedure are involved in its development right from the start.

Meeting 1: all interested staff are invited to attend. The aims and problems of oral tests are outlined, including the principles of validity and reliability.

Meeting 2: discussion of mark categories; which are desirable and which are actually realistic. Five broad categories are chosen; listening comprehension, vocabulary, accuracy, pronunciation, and communicative competence.

Meeting 3: draft rating scales are discussed using copies of several other scales as reference, taken from commercially published sources and standard reference books.

 Meeting 4: the broad test type and the most suitable elicitation techniques are discussed; a short list of six tests is chosen. A sample, taped interview is played; some practical points of the general procedure for test administration and marking are considered.

 Over the next two weeks, the assessors carry out a total of forty sample oral interviews, all recorded, at a variety of language levels and using the different techniques on the short list.

 Meeting 5: personal feedback from sample interviews. The most common and most serious problems are identified and discussed; corrective action is agreed upon.

 Subsequently, the test designer selects twenty of the taped interviews for rater-training on the basis of a range of levels, good sound quality, and the presence of typical problems. These tapes are copied and their order randomized. Over the next two weeks, each assessor marks each of these twenty tapes, and all their marks for each category for each test are collated in a single table of results. To encourage frank discussion, each assessor is allocated a letter code.

Meeting 6 : the test techniques are discussed further and two are chosen: an interview about the learner’s job and talking about a picture. The average length is set at about fifteen minutes per test. The normal test routine is discussed.

Meeting 7: the table of sample interview results is presented and discussed anonymously; nobody knows who the codes refer to. The average of all the assessors’ marks is used as the norm. Major deviations are discussed in some detail; why do certain mark categories, and certain levels, produce a wider range of scores than others? Each assessor is subsequently told only which letter code represents her own marks, so she can compare them with the norm.

Meeting 8: in the light of the sample tape marking, and the techniques decided on, the weighting of the five categories is discussed and a decision made. The rating scales are reviewed.’