Job Discussion Forums

[LongShiKong]

[Sashadroogie]

If the student is giving 'the right answer' then it seems to me that issues of validity are involved, i.e. is the test testing what it is supposed to be?

In any case, it seems to be more a case of examiners not correctly applying the criteria than a problem with test design. Which tests are you referring to by the way?

[LongShiKong]

[spiral78]

[LongShiKong]

This was an oral test. With no other instruction, I chose to interpret the 'acceptable' as meaning 'passable' a 50%, not 'perfect'. For many of these tests, the syllabus explicitly mapped out such responses as falling within the domain of expressive language.

[Sashadroogie]

If you are in a situation where you can/have to choose to interpret what the criteria are or mean than the entire test seems to be lacking everything: all types of validity, but especially construct validity. What skills was the test supposed to assess exactly? You have my sympathies.

Who made these tests? Professional materials designers? Or some unqualified teachers rushing around a staff room trying to cobble something together fast?

[johnslat]

Dear Sasha,

"Professional materials designers? Or some unqualified teachers rushing around a staff room trying to cobble something together fast?"

Aren't they the same people?


Regards,
John

[Sashadroogie]

Dear Johnslat

Certainly not! At least not in the hallowed halls of Cambridge. Perhaps in whichever place in China poor Longshikong is working.

Off-his-face-validity Sasha

[timothypfox]

Yes, my MS TESOL did. But, I grew very disillusioned with assessment when I worked for public school in NYC. My job was collecting data on ESL high school students with moderate disabilities. Invariable the assessments confused measuring the students language level with their various disabilities such as language delays, ADHD, lower level cognitive functions, and speech and language deficits.

And, don't get me started on standardized testing! I am of the generation that had none, and did just fine!

I now work with EFL high school students at a Japanese high school. Assessment is done through written tests either writing or listening. The written tests cover a topic they practiced in class. During classes their work is edited using a marking guide indicating the type of errors, so that they learn to correct their own mistakes.

I also have junior high classes that I give speaking tests one by one too. Sections include memorizing a part of 3 dialogues that they perform with me, an oral spelling test, proper oral response to question structures we practiced, and reading a short passage aloud with correct pronunciation. Each section is marked using a 5 level rubric (each level has a description of achievement), and the total is then added for a final score...

Having said all this - Japanese schools still teach heavily with the grammar translation approach even though US programs adhere more to the communicative approach to instruction and assessment. US schools are so slanted towards collaborative group work and student empowerment that high level student grow bored as beaurocrats push "no child left behind...." Love it , or lump it!

[LongShiKong]

[Sashadroogie]

Well, if this is a ready-made placement test, then I'm not sure that using it 'in light of syllabus expectation' is in keeping with the test's original purpose. And again, if not all examiners are applying the criteria consistently, whatever they may be, then expect skewed results.

There is always some degree of subjectivity in assessing written and spoken language. But what you are describing seems to transgress many basic norms in language testing. Have a closer look at Cambridge and IELTS exams and tests, and refer to the band descriptors. There are public versions available on the net. As you'll see, the criteria are quite well-defined, in most cases, and so long as examiners actually stick to the criteria they are supposed to apply, then reliability will remain high. (I wouldn't place much credence in whatever 'opinion polls' you've come across either...)

Can you tell us what test you and your franchise are using? Which industry publications? It may help to understand your situation a little better.

[LongShiKong]

[Sashadroogie]

I'm not sure that Hughes has said that - at least not in your quotation. Phrases like 'too often' are not the same as poor testing is the norm.

In your specific situation, this seems to boil down to a case where nobody who is designing/selecting the test materials is qualified to do so. It may even come down to lack of examiner training also. This would seem to me to be because teachers (and school administration) are not the same manner of being as test materials designers. Nor should they be. It's a specialised area - which is why it is probably not covered very much in most TEFL-related courses.

Teachers often confuse listening exercises designed to test comprehension with listening skills lessons designed to teach specific listening skills. However, it is also true that we can confuse teacher-training courses with qualifications in testing and assessment. Of course it is helpful for all teachers to be familiar with the basic principles of testing, and even to be able to put together a reasonable test themselves, for their own students etc. But designing a battery of test items for a whole school, where the stakes may be very high indeed... this is a very different matter entirely.

Being able to drop in technical terms like 'concurrent validity' into staff room conversations is not the same as being fully competent in a very technical area. G-theory? Classical true Score theory? Most teachers have not got any real clue about the intricacies of all this - and I include myself in that category. So, it is hardly surprising that teachers, according to Hughes, have a mistrust of testing: they have not ever experienced real testing by real testers. Instead they have had to deal with what you have described, with the predictable confusion arising therein.

Teaching and testing. Different areas.

[Sashadroogie]

[LongShiKong]