Job Discussion Forums

[erracht]

First to update: as per my recent query about placement tests, I have looked at some of the links provided and put together an elementary test, an intermediate test and and advanced test.

But then my colleague changed her mind and decided to give all students, regardless of what level they are expected to be, common tests. So, she suggested I do something like take one third of each of the three level tests and put together three general tests using one third of each original test for each of the new tests. So now I have three tests, each of them having, beside a fill-in-the-blanks passage, 18 questions, of which 6 are relatively easy, 6 are relatively intermediate and 6 are relatively advanced. The questions are multiple choice.

My question: how do we grade such a test? Do we count the easier questions for less points than the harder ones? If so, how do we calculate the level based on score? Suggestions?

[Justin Trullinger]

My question: How much are they paying you for this?

I mean really. It takes a lot of time, refining, and just plain work to come up with a funtioning level test. And, although I've got quite a lot of experience in level evaluation, I'm not as qualified to do it as a lot of the people in the language departments at the big universities and testing companies. There are better tests on the market than any I'm likely to write.

But, sometimes that isn't really possible. (I believe $$$ is the reason usually cited.) So tell me, how many different levels of English does your school offer? I'd try to take the same number of questions, 10 or so, from the set text or course sylabus of each level offered. (By the way, is there some time contraint on this test you're thinking of? It sounds like you've only got 25 or 30 questions total, which sounds low to me. More questions will decrease your chances of getting weird false results.) And I would give all questions equal value. (Sounds weird, but trust me, it works out in the end!)

So, to use your example: I'm assuming 3 levels, elementary, intermediate, and advanced, with 6 questions from each.

A student who gets: 0 to 6 total right is Elementary
7 to 12 total right is Intermediate
13-18 total right is Advanced

Remember, as this is multiple choice, the scale will skew at the bottom, because someone with no knowledge of English at all will generally get 25 to 33% correct. (Depending on the number of possible answers to each question.) So somebody with a score of six could as easily have a decent elementary level, or no level at all. I always include a brief oral interview to sort out the difference. Expanding your universe (a larger number of questions or levels) will reduce the chance of false results, but they happen. Make sure you have a back up mechanism to try to spot them.

The reason that all questions have the same value:

A) To reach advanced level in this example, a student would have to be correct on some or most of the advanced question, and all or most of the lower level questions. This student is really advanced.

B) An intermediate student would be expected to get some or most of the intermediate questions (let's say 4), and most or all of the elementary (Let's say 5), PLUS one third of the advanced questions(Because multiple choice exams mean everbody gets a few of everything! Let's say 2). In this example, a student who knows nearly all of the elementary questions and most of the elementary questions gets an 11. Intermediate. Which is where he or she belongs. If the advanced questions were worth more, the result would skew upwards, possibly into the next level.

C) The assumption on this type of exam is that every student will achieve a "positive" (better than the statistical average,which is probably 25 percent on multiple choice questions with 4 choices) on questions up to and including their level, and an "average" result (Maybe 25%) on all questions above their level. If you give more value to the higher level questions, you'll cause all results to skew upwards, by giving more value to these random guesses.


I hope that was clear! Tee hee. Now try to find one listening with questions in several different level. I'd probably make my own...

Good luck,
Justin

[gaijinalways]

Hmm, that sounds reasonable on the face of it (having the all the questions worth the same number of points), but why do tests like the TOEIC and TOEFL have weighted questions then? In other words, the points, if the questions are answered correctly whether for low, intermediate or advanced students will be the same, and I would guess that the advanced students have a much better chance of answering the advanced questions correctly (and probably the other ones as well).

Personally I don't weight sections of my tests equally, but my tests are not all multiple choice questions only, which I wouldn't recommend for a language evalution test. If you are using only multiple choice exams (perhaps easier to grade as they can be computer scanned), then at least level out the problem of guessing by having more choices, perhaps as many as 7-10 (one of my university professors had killer multiple choice exams with up to 16 answers with combined answers like 'if a is true and b and c are false' or ' only if d, g, i are sometimes true'..trying to find the best answer was a nightmare!), and also answers that are similar to make it more difficult for students to guess (using similar grammar, verbs, structures etc.).

[Justin Trullinger]

Very true. Where did you go to university? I think I knew that guy!


In order for the balance to come out, in a level evaluation test, if you weight some questions higher than others, you'll need to adjust the total number of questions that correspond to each level. And then the equations get COMPLICATED.

I want to emphasize the difference between a regular classroom exam, where all the students taking it will be of (approximately) the same level, and a level test, which might be taken by students of any level, and is meant to indicate what level a student should be in.

In a regular classroom exam, I always weight questions differently, and hope (sometimes in vain) that student results will form a band between 60% and 90%. In a level test, I don't weight questions, and know that the wider the band of results, the better my chances of a good level assesment.

Better yet, I would buy a good level test. Because writing the BEEP things makes my head hurt.

Regards,
Justin