More futility

Last weekend, I was participating in a large-scale foreign language skills assessment programme for the state-owned news agency Xinhua. This is supposedly an "annual" event, but in practice it only seems to occur every two or three years. Well, I think this is the fourth time I've done it in seven years.

It was a particularly frustrating experience this time. In the past, the English tests I'd been administering were quite a varied challenge, involving some reading, some straightforward Q&A, and a small amount of rather freer conversation with the examiner at the end - which allowed for a reasonable amount of differentiation of the candidates.

The marking scale was a bit narrow, and the marking guidelines almost non-existent. But still, the test itself was fairly sound, and it did enable us to produce some kind of meaningful ranking of the candidates' English ability (a high level of which can, in theory, give them access to pay bonuses, promotions, and coveted overseas postings).

Following the old Chinese adage of "If it ain't broke, break it", this year they decided to change everything.

There was no longer any meaningful interaction with the examiners at all, no element of conversational English; it was almost entirely a listening test. 

The first part was graded on sets of 'True or False' questions which were so badly written (the basis of most of them was in fact: here is a hugely long verbatim quote from the passage you just heard, which has a number of typos and omissions in it; one of them is deliberate, and is supposed to be significant - it might well not be, but you have to guess if we intended it to be!) and varied so wildly in their degree of difficulty that the results on this section were essentially random. With 2 marks for each "correct" answer, almost everyone scored either 6, 8, or 10 marks (with the benefit of a lot of subliminal prompting from the examiners!). So, in effect we had only a three-point marking scale. And the main performance differentiator here was blind luck - whether you got an easy set of questions, or happened to guess right on some impossible questions.

In the second part, the poor candidates were forced to listen to LONG extracts from a political speech and then "sum up the main points" they could remember. This was a ridiculously difficult and unstructured task. The only people who did reasonably well on it were the ones who were adept at writing notes quickly, and were thus able to repeat certain phrases or sentences from the speech word-for-word (though usually without any apparent comprehension of their significance; almost no-one was capable of responding to even the simplest of questions about the stuff they were regurgitating). The banding scheme here was again very narrow: almost everyone was classified as either 'Low Average' or 'High Average', with only a few being (rather over-generously) awarded a grade of 'Excellent'. So, this section was judged on a three-point scale, with the main performance differentiator being an ability to take dictation.

In 20 years of EFL teaching, that was the most WORTHLESS test I have ever seen. I am appalled, disgusted to think that the results of this fiasco might actually be used to determine the progress of people's careers, to judge if they are capable of undertaking an overseas posting. My molars are ground down to the bone.