Site Search:
 
Get TEFL Certified & Start Your Adventure Today!
Teach English Abroad and Get Paid to see the World!
Job Discussion Forums Forum Index Job Discussion Forums
"The Internet's Meeting Place for ESL/EFL Students and Teachers from Around the World!"
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Is the IELTS speaking test scripted?

 
Post new topic   Reply to topic    Job Discussion Forums Forum Index -> China (Job-related Posts Only)
View previous topic :: View next topic  
Author Message
hereinbeijing



Joined: 24 Dec 2007
Posts: 97

PostPosted: Wed Sep 13, 2017 2:06 pm    Post subject: Is the IELTS speaking test scripted? Reply with quote

Hello,
Is part three of the IELTS speaking test scripted, meaning that the Examiner can only choose from a list of questions or are they free to come up with questions?
Thanks in advance
Back to top
View user's profile Send private message
Tudor



Joined: 21 Aug 2009
Posts: 319

PostPosted: Thu Sep 14, 2017 1:52 am    Post subject: Re: Is the IELTS speaking test scripted? Reply with quote

hereinbeijing wrote:
Hello,
Is part three of the IELTS speaking test scripted, meaning that the Examiner can only choose from a list of questions or are they free to come up with questions?
Thanks in advance


There is a list of questions, but examiners are expected to paraphrase these and grade their language to the candidate's level. They are also expected to ask 'follow-up' questions based on the listed questions, which are to get the candidate to clarify, justify or elaborate on their answer. Alternatively, they may be used to get a candidate back on track in case of tangential answers.
Back to top
View user's profile Send private message
hereinbeijing



Joined: 24 Dec 2007
Posts: 97

PostPosted: Thu Sep 14, 2017 2:54 am    Post subject: Thanks Tudor Reply with quote

Tudor,
I greatly appreciate your help.
Have a good day.
Back to top
View user's profile Send private message
theoriginalprankster



Joined: 19 Mar 2012
Posts: 755

PostPosted: Thu Sep 14, 2017 4:06 pm    Post subject: Reply with quote

When dealing with candidates in the 4-6 range I was usually asleep by Part 3.

Sad, but true. When you've heard the same garbled nonsense for the 1150th time you can afford to go into auto-examiner mode.

Most examiners have made up their mind about the speaking score by the end of Part 1 or 2.
Back to top
View user's profile Send private message
papuadn



Joined: 19 Sep 2016
Posts: 131

PostPosted: Thu Sep 14, 2017 6:54 pm    Post subject: Reply with quote

theoriginalprankster wrote:
When dealing with candidates in the 4-6 range I was usually asleep by Part 3.

Most examiners have made up their mind about the speaking score by the end of Part 1 or 2.
So...4-6, that's a third of the range. Is it true the IELTS disposes with auditing procedures unless challenged? Does a statistically improbable batch of scores flag an examiner for audit?
Quote:
Sad, but true. When you've heard the same garbled nonsense for the 1150th time you can afford to go into auto-examiner mode.
A charitable read indicates you were over-worked, yet a meaningful discrimination is what the test is designed to do, what you were paid to do (underpaid), and what the client paid for-- garbled nonsense notwithstanding.

An implication of your account is that proficiency is a binary proposition and it most certainly is not.

It is true experienced examiners can accurately suspect an outcome early on as proficiency exams have an advantage in terms of the number of indicators, but neglecting auditing procedures is irresponsible given the stakes of these exams.
Back to top
View user's profile Send private message
airapets



Joined: 22 Jul 2007
Posts: 77
Location: The Magic Kingdom

PostPosted: Tue Sep 19, 2017 2:44 pm    Post subject: Reply with quote

Actually, part one is clearly scripted as is part two. Only part three allows deviation. There is plenty of auditing done of examiners' marks. As for a previous poster's comment about having a difficult time to focus, most answers for parts one and two are canned responses, that is they are memorized answers, and the English quality is mediocre to say the least. If both part 1 and part 2 are like this, and the plurality are in China, then the candidate's mark will nearly always be between 4 and 5.5.

You do not see nearly as much memorization of answers in other countries or among students who are not from China, thus making IELTS examining more interesting in other countries.
Back to top
View user's profile Send private message
Trevor Wadlow



Joined: 14 Oct 2007
Posts: 97
Location: china

PostPosted: Tue Oct 17, 2017 5:05 am    Post subject: Re: Is the IELTS speaking test scripted? Reply with quote

hereinbeijing wrote:
Hello,
Is part three of the IELTS speaking test scripted, meaning that the Examiner can only choose from a list of questions or are they free to come up with questions?
Thanks in advance



Part three contains Headings which the Examiner uses to form questions. It is this part where the Examiner really makes his/her assessment.
Back to top
View user's profile Send private message Send e-mail
theoriginalprankster



Joined: 19 Mar 2012
Posts: 755

PostPosted: Tue Oct 17, 2017 7:21 am    Post subject: Reply with quote

Quote:
Part three contains Headings which the Examiner uses to form questions. It is this part where the Examiner really makes his/her assessment
.

Honestly, by the end of Part 2, even Part 1, we've more or less got a score figured out within our head. The candidate would have to really shine, or bomb, for the score to change more than a band either way.
Back to top
View user's profile Send private message
Trevor Wadlow



Joined: 14 Oct 2007
Posts: 97
Location: china

PostPosted: Tue Oct 17, 2017 9:00 am    Post subject: Reply with quote

theoriginalprankster wrote:
Quote:
Part three contains Headings which the Examiner uses to form questions. It is this part where the Examiner really makes his/her assessment
.

Honestly, by the end of Part 2, even Part 1, we've more or less got a score figured out within our head. The candidate would have to really shine, or bomb, for the score to change more than a band either way.


This hasn't been my experience at all. A candidate can easily prepare for and recite for parts 1 and 2, giving the impression of fluency at least. As they cannot prepare for Part 3, such candidates come unstuck. I have seen many a candidate appear to be 6, only to plummet to 5 by part 3.
Back to top
View user's profile Send private message Send e-mail
papuadn



Joined: 19 Sep 2016
Posts: 131

PostPosted: Tue Oct 17, 2017 10:10 am    Post subject: Reply with quote

papuadn wrote:
Does a statistically improbable batch of scores flag an examiner for audit?
airapets wrote:
There is plenty of auditing done of examiners' marks.
I might should have said, "...an examiner's results for audit..." Because, from the point of view of design, two trained examiners in this thread are disputing the relative value of three sections as though any were superfluous.

I agree with airapets' overview that Chinese examinees too typically memorize responses (the fastest way to lower any proficiency score) as compared to other test takers, which goes a long way in explaining why similarly (if not identically) trained examiners differ in their estimation of what sections are the most "real" or valuable, or indicative of a valid result-- given the noise to signal ratio, much of any measure can be rendered superfluous.

And I want to qualify what I'm about to write by expressing how impressed I was by the IELTS (compared to the TOEFL) when I surveyed existing measures to design and then collaboratively implement a speaking metric for an employer. Yet in a research setting where I cut my teeth on similar measures, the convention of a sole examiner simultaneously soliciting and scoring speech was eschewed for reasons of bias and divided attention. Moreover, recordings of the output were evaluated by an average of multiple scorers.

Of course, administering an IELTS is not research, it's a business within academia, its procedure long ago tempered by rigorous method. But its tests are a product and their implementation is not without pressures and incentives. Neither is the British Council/CELA transparent in terms of which nations most pad a bottom line. The Council puts out lots of demographically related statistics in terms of relative (or normative) scores, but in terms of dollars and pounds, I can find nothing.

So when the OP and myself are assured that there is "plenty of auditing done", I am interested and curious about any mechanism within a procedure used by tax-funded governments and institutions of higher learning. I'm curious because my Chinese students report non-native speaker examiners with strong accents, examiners that are, themselves, scored by IELTS in order to be examiners.

I am understandably skeptical of a combination of the Council's opacity and its confidence in a product being beyond question given the substantially increased earnings by serving Chinese nationals.
Back to top
View user's profile Send private message
Volver



Joined: 27 Sep 2013
Posts: 165

PostPosted: Tue Oct 31, 2017 6:37 am    Post subject: Reply with quote

I was an IELTS examiner for years until I could not take it any longer. The posters who said they could usually figure out a candidate's level long before Part 3 are absolutely correct. China is the great land of mediocrity when it comes to IELTS - this country really rocks those 5s and 5.5s. After hearing several thousand of candidates at this level, an examiner can often discern a candidate's level in under a minute.

Having said the above, I learned to hold off on giving a score until part 3 as you really owed that to the candidate and you may be surprised. Sometimes, and I enjoyed these, a person would start off low and then work their way up as they got more comfortable with the test. Part 3 turned out to be at a higher level then Part 1. However, what usually happened is Part 1 and 2 were great. This was always a red flag for me so I would plan to ask more (than required) and higher level questions of my own during Part 3. 9 out of 10 times the candidate fell apart and showed their true English level. Here's your 5, now get lost.

V
Back to top
View user's profile Send private message
papuadn



Joined: 19 Sep 2016
Posts: 131

PostPosted: Wed Nov 01, 2017 6:46 am    Post subject: Reply with quote

Volver wrote:
I was an IELTS examiner for years until I could not take it any longer. ... [S]o I would plan to ask more (than required) and higher level questions of my own during Part 3. 9 out of 10 times the candidate fell apart and showed their true English level. Here's your 5, now get lost.
Audits are indispensable for reasons of fatigue and bias so succinctly expressed.

An evaluator, for whatever reason, deviating from procedure in terms of quantity or content of prompts has defined criteria. Any latitude for which IELTS training provides is conservative, by design, for reasons of consistency because reliability and normative analysis are jeopardized by significant deviation. In plainer words, not everyone took the same test-- but plainer words are limited in their utility, such as same and significant, approached and demonstrated through abstruse statistical methods. Any and all correlations my efforts demonstrated were crunched by a math department.

Were suspicions and intuition (or an abundance of similar examinees) sufficient to claim a 90% success rate with false positives and a "true English level", why design and construct a measure at all?
Back to top
View user's profile Send private message
papuadn



Joined: 19 Sep 2016
Posts: 131

PostPosted: Thu Nov 02, 2017 4:55 am    Post subject: Reply with quote

airapets wrote:
There is plenty of auditing done of examiners' marks.
Volver wrote:
I was an IELTS examiner for years until I could not take it any longer. ... [S]o I would plan to ask more (than required) and higher level questions of my own during Part 3. 9 out of 10 times the candidate fell apart and showed their true English level. Here's your 5, now get lost.
Does the former conclude the expressions of the latter merit audit? Singular and authoritative contributions, so simultaneously absolute and vague in their conclusion, are discouraging to legitimate discourse. I sincerely hope for a reply.

Goodhart's Law [WikiPedia]
An adage named after economist Charles Goodhart, which has been phrased by Marilyn Strathern as: "When a measure becomes a target, it ceases to be a good measure." This follows from individuals trying to anticipate the effect of a policy and then taking actions which alter its outcome.

Goodhart’s Law and Why Measurement is Hard
June 9, 2016 By David Manheim
    I think a useful heuristic for understanding where to use them is to look for the triad of intuition, trust, and complexity.

    Complex systems can only be managed using metrics, and once the metrics are put in place, everyone is being incentivized to follow the system’s logic, to the exclusion of the original goals. If you’re not careful with your metrics, you’re not careful with your decisions. And you can’t be careful enough.

    Measurement replaces intuition, which is often fallible. It replaces trust, which is often misplaced. It finesses complexity, which is frequently irreducible. So faulty intuition, untrusted partners, and complex systems can be understood via intuitive, trustworthy, simple metrics. If this seems reductive, it’s worth noting how successful the strategy has been, historically. Wherever and whenever metrics proliferated, overall, the world seems to have improved.

    Despite these benefits, measuring obscures, disrupts, and distorts systems. I want to talk about the limitations of metrics before expanding on some problems that are created when they are used carelessly, and then show why the problem with metrics — and algorithms that rely on them — isn’t something that can be avoided.
[emphasis in original]
Back to top
View user's profile Send private message
Volver



Joined: 27 Sep 2013
Posts: 165

PostPosted: Thu Nov 02, 2017 1:51 pm    Post subject: Reply with quote

All I know is that I got audited regularly and NEVER ONCE got a bad review.

V
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    Job Discussion Forums Forum Index -> China (Job-related Posts Only) All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


This page is maintained by the one and only Dave Sperling.
Contact Dave's ESL Cafe
Copyright © 2016 Dave Sperling. All Rights Reserved.

Powered by phpBB © 2001, 2002 phpBB Group

Teaching Jobs in China
Teaching Jobs in China