|
Job Discussion Forums
"The Internet's Meeting Place for ESL/EFL Students and Teachers from Around the World!"
|
| View previous topic :: View next topic |
| Author |
Message |
Chris21
Joined: 30 Apr 2006
Posts: 366
Location: Japan
|
 Posted: Mon Nov 24, 2008 10:11 am Post subject: |
 |
|
| Quote: |
So, you choose not to believe that scores from 145,000 Korean test takers means anything? Go find better figures. ETS is the place that developed the TOEIC. Best answer I can give you.
Many students in Japan are forced to take TOEIC, too, so you can't throw out data for that reason. |
The issue isn't whether I believe the reported mean of 145,000 Korean TOEIC scores. This issue is whether the 145,000 Korean sample and the 877,000 Japanese sample are representative of the same types of students. My contention, as said earlier, is that the Japanese sample covers a much broader swath of the population, thereby lowering the mean score.
Unfortunately since TOEIC does not release detailed information about who is taking the test, other than "Korean" or "Japanese" in this case, a comparison is impossible. Additionally, the TOEIC is an institutional test developed specifically for Japan, so it'll be used in more institutional settings than in Korea. So, yes... I would throw out the data for that reason (at least if it is being used to compare two samples against each other).
Forgive me if I'm mistaken, but don't you teach at a science university, Glenski? You should ask one of the statisticians in the math department about problems that arise from biased samples. |
|
| Back to top |
|
 |
Glenski
Joined: 15 Jan 2003
Posts: 12844
Location: Hokkaido, JAPAN
|
| Posted: Mon Nov 24, 2008 9:56 pm Post subject: |
|
|
Chris,
Why does a larger sample mean a stupider bunch of people?
Oh, and that report link DOES have a breakdown of sorts on who takes the TOEIC from some of the countries. Not as extensive as you might like, but it's not nothing at all.
Better to have some data than none. Want more precise numbers? Find them yourself. I did my best to please you. |
|
| Back to top |
|
|
Chris21
Joined: 30 Apr 2006
Posts: 366
Location: Japan
|
| Posted: Mon Nov 24, 2008 10:27 pm Post subject: |
|
|
| Quote: |
Chris,
Why does a larger sample mean a stupider bunch of people?
Oh, and that report link DOES have a breakdown of sorts on who takes the TOEIC from some of the countries. Not as extensive as you might like, but it's not nothing at all.
Better to have some data than none. Want more precise numbers? Find them yourself. I did my best to please you. |
I'm appreciative Glenski, I'm just saying that these numbers don't mean anything so they can't be used to support a claim that Koreans are better at English than Japanese. Perhaps they are, but that certainly can't be proven with these stats.
A larger sample size means a stupider bunch of people, as you put it, because these are not randomized samples. These samples are dictated by a variety of factors... for example, economics. Who can afford to take the test? Generally, Japanese per capita income is higher than that of Koreans, and they may be in a better position to pay for an expensive English test. Also, educational policy. Are the same tests administered the same way by both Korean and Japanese schools? There may be similarities, but Monbusho and the Korean equivalent are separate and unique entities, and thus, there will be differences in how TOEIC is administered.
So to recap, these samples are not random samples. Japan and Korea are not the same, and as a result, the samples will be biased towards factors unique to each country, and the results meaningless. |
|
| Back to top |
|
|
Glenski
Joined: 15 Jan 2003
Posts: 12844
Location: Hokkaido, JAPAN
|
| Posted: Tue Nov 25, 2008 3:47 am Post subject: |
|
|
| Chris21 wrote: |
| I'm appreciative Glenski, I'm just saying that these numbers don't mean anything so they can't be used to support a claim that Koreans are better at English than Japanese. Perhaps they are, but that certainly can't be proven with these stats. |
I disagree. The stats aren't perfect, but they do show that the people who took them average out with different scores.
| Quote: |
| A larger sample size means a stupider bunch of people, as you put it, because these are not randomized samples. These samples are dictated by a variety of factors... for example, economics. Who can afford to take the test? |
No, you are also incorrect here. Wo says that the test-taker is always the one who pays for the exam? In some cases, companies or even universities pay the cost.
I don't know what you are thinking here. You ask for a direct comparison of Korean and Japanese students (I think) who take TOEIC, then you say these are not randomized. If they were randomized, then you would not know specific categories of people and have no way at all of comparing. You seem to want it both ways here. What would please you?
| Quote: |
| Also, educational policy. Are the same tests administered the same way by both Korean and Japanese schools? There may be similarities, but Monbusho and the Korean equivalent are separate and unique entities, and thus, there will be differences in how TOEIC is administered. |
I don't know what you are driving at here, so I can't really answer. TOEIC is given in various centers throughout the country (or should I say countries?) with monitors on staff, time limits, etc. Why should there be any differences in how they are "administered", unless I don't really understand what you mean by that? Just how are they administered in Korea?
Looking at the ETS page (specific to neither Korea nor Japan), you can see this link:
http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=277f658c97aa4110VgnVCM10000022f95190RCRD&vgnextchannel=f64afe3f4c394110VgnVCM10000022f95190RCRD
It clearly states:
| Quote: |
What Is TOEIC?
The Test of English for International Communication� (TOEIC�) is an English language test designed specifically to measure the everyday English skills of people working in an international environment.
Who Takes It And Why?
Nonnative English speakers take the test to demonstrate English proficiency when applying for new positions and obtaining credentials.
Where Do People Take It?
The test is given at open public sessions, and at companies and language schools around the world.
Who Accepts It?
The test is widely accepted by corporations, English-language programs, and government agencies around the world.
* Corporations use TOEIC test to document progress in English training programs, recruit and promote employees, and put standard measurements in place across locations.
* English Programs use it to place students at the right learning levels, and show student progress and program effectiveness.
* Government agencies use it to document progress in English language courses, and to recruit, promote and hire employees.
Testing Format
The TOEIC test is a paper-and-pencil, multiple-choice assessment that uses audiocassettes, pictures, and written materials to evaluate English-language skills. |
Looks pretty neutral to me. No bias in any country. What am I missing here? |
|
| Back to top |
|
|
Chris21
Joined: 30 Apr 2006
Posts: 366
Location: Japan
|
| Posted: Tue Nov 25, 2008 6:03 am Post subject: |
|
|
| Quote: |
| I disagree. The stats aren't perfect, but they do show that the people who took them average out with different scores. |
Sorry for not explaining my points clearly enough. Let me try with an analogy. All the students at Keio University are given an English test. Also, the students in intermediate & advanced classes at Waseda University are given the same test. If the Waseda students get 90% and the Keio students get 88%, does this mean that Waseda students are better at English? You might say, as you did above, that "people who took them average out with different scores", but if the groups are biased from the outset, then it isn't a fair comparison. Your response is likely to be "well, Korea is not analagous to Waseda", but when a test is administered much more selectively in Korea, then to some degree it is analagous to Waseda. TOEIC is treated more seriously there because a) it is more expensive for people to take relative to their income b) it does not have the widespread acceptance that it does in Japan, making it more for "keeners" than in Japan.
| Quote: |
| No, you are also incorrect here. Wo says that the test-taker is always the one who pays for the exam? In some cases, companies or even universities pay the cost. |
Who pays for the exam is irrelevant. Whether it is the test-taker, the institution, or company that pays for the exam, somebody has to pay for it. If the exam is relatively cheap in Japan, it will be taken by more people that don't really care if they fail (because they haven't wasted that much money). Conversely, when the TOEIC is comparatively expensive, students will likely study more because they might not want to take the test again (since it represents a significant expense for themselves). To analagize again, wouldn't you study harder for a test that cost $500 than one that cost $1? I'm not saying this is the case with everyone, however, it seems very likely for some.
| Quote: |
| I don't know what you are thinking here. You ask for a direct comparison of Korean and Japanese students (I think) who take TOEIC, then you say these are not randomized. If they were randomized, then you would not know specific categories of people and have no way at all of comparing. You seem to want it both ways here. What would please you? |
I do not want it both ways. I want random samples. Random samples would please me. I'm confused as to what you are suggesting. Are you suggesting that Korean and Japanese is a satisfactory control for these two groups? You think that these groups are drawing from exactly the same pool of test-takers? Perhaps you mean that if test-takers were randomized, they would all be lumped into a single large sample? It is certainly possible to create two different groups, one from a mid-level Japanese university and one from a mid-level Korean university, and then test all students... that would give a far better indication than these current stats do. Of course there would still be issues to debate because how can you ever be certain that a mid-level Japanese and mid-level Korean university are equal, but at least there would be an attempt to create equivalent groups. With the data you provided, no such attempt has been made.
| Quote: |
| I don't know what you are driving at here, so I can't really answer. TOEIC is given in various centers throughout the country (or should I say countries?) with monitors on staff, time limits, etc. Why should there be any differences in how they are "administered", unless I don't really understand what you mean by that? Just how are they administered in Korea? |
If a test is administered in a Japanese university to all students as part of their final English grade, then it will be given to a some unmotivated and low-level students. If a test is administered in a Korean university as a part of a study abroad program, then I would assume more of the test-takers would be motivated and of a higher-level. This is what I mean by differences in the administration of the test.
Since the TOEIC test was originally developed in cooperation with the Japanese government, and a dispproportionate number of test-takers are Japanese, I would say it is far more likely for TOEIC to be widely administered across an entire Academic program in Japan, than in Korea. And more likely that the TOEIC would be for specialized uses in Korea. This is not to say that the TOEIC is never used in a broad way in Korea, but my belief is that it tends to skew towards better students in Korea.
| Quote: |
| Looks pretty neutral to me. No bias in any country. What am I missing here? |
The bias lies in who is taking the test in each country.
I don't want to sound snarky, but I think you would be well-served to discuss appropriate research design with a colleague. The flaw in comparing ETS' TOEIC scores between nationalities is HUGE, and any credible journal editor would decline a research submission with this methodological flaw instantly. |
|
| Back to top |
|
|
RyznFree
Joined: 10 Nov 2008
Posts: 27
|
| Posted: Sat Nov 29, 2008 4:31 pm Post subject: |
|
|
Many people have said on this board that it shouldn't be a problem finding a job in Tokyo if you come in March with the money to last a couple months. Great!
But, what about this "exodus" of Korean ESL teachers coupled with the recession? Are there still enough jobs in Tokyo to go around? |
|
| Back to top |
|
|
Glenski
Joined: 15 Jan 2003
Posts: 12844
Location: Hokkaido, JAPAN
|
| Posted: Sun Nov 30, 2008 12:20 am Post subject: |
|
|
Chris,
I guess there is just no pleasing you. I can't even follow your logic.
You want something to compare directly, yet you can't even agree that comparing Keio and Waseda is fair.
You also wrote:
| Quote: |
| Your response is likely to be "well, Korea is not analagous to Waseda", |
No, don't put such words in my mouth. I would not say that. In fact, I have said quite the opposite.
| Quote: |
| but when a test is administered much more selectively in Korea, then to some degree it is analagous to Waseda. |
How it is administered is something you have not answered. At least, that's the way I read it. You seem to want data that isn't there. Ask ETS. I've done my best.
| Quote: |
| it does not have the widespread acceptance that it does in Japan, making it more for "keeners" than in Japan. |
What is a keener? Someone who studies hard to use TOEIC for a specific purpose of getting a job or promotion or overseas assignment? All of those are done in Japan, yes. Why do they take it in Korea? What is the "acceptance" there, and why does it matter for the original question you asked? You can't find data that says country X people took the exam for the same reason as country Y people, and then compare their scores. ETS does its best to provide a breakdown of test takers. Contact them if you want more, but I doubt they will be able to satisfy you.
You talk about unfairness in conditions like being able or unable to pay for the exam, and then you say it doesn't matter. It does matter, and the price itself does matter. At 6000 yen per test (roughly US$60), I can show you many college students who balk at taking it just on principle alone, but there are also many who don't want to pay 1/3 that for a regular English textbook! Also, some places require that students take the test several times a year, which makes that "cheap" price a lot deeper dip into one's pockets (or Daddy's).
| Quote: |
| I do not want it both ways. I want random samples. Random samples would please me. |
Just how are you going to get that? I suggest you read that article again, and look only at the graphs, not the explanations or breakdowns. That's as random as you are going to get.
I'm outta here. |
|
| Back to top |
|
|
Chris21
Joined: 30 Apr 2006
Posts: 366
Location: Japan
|
| Posted: Sun Nov 30, 2008 12:29 pm Post subject: |
|
|
Glenski,
I thought I could recommend some good books that might help clarify some issues. Sorry for not properly citing them.
For effective research design, you might want to check out...
Creswell, J. (2008). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications.
Brown, JD. (1988). Understanding Research in Second Language Learning: A Teacher's Guide to Statistics and Research Design. Cambridge University Press.
And for a better understanding of how skewed samples can affect results, check out...
Field, A. (2008). Discovering Statistics Using SPSS (Introducing Statistical Methods series) Sage Publications.
Hope those help. |
|
| Back to top |
|
|
Brooks
Joined: 16 Jan 2003
Posts: 1369
Location: Sagamihara
|
| Posted: Mon Dec 01, 2008 2:18 am Post subject: |
|
|
I wonder if Chris has ever taught the TOEIC in Japan.
My wife has and overall the students don't do well.
She was even asked to help edit a TOEIC book that called "Very Easy".
Lots of Koreans may be more motivated since by taking the TOEIC they can get a job, or a better one. |
|
| Back to top |
|
|
Chris21
Joined: 30 Apr 2006
Posts: 366
Location: Japan
|
| Posted: Mon Dec 01, 2008 6:52 am Post subject: |
|
|
| Quote: |
| I wonder if Chris has ever taught the TOEIC in Japan. |
Yes, I have. I had to teach it to a basic level group of students who had very little interest. To my earlier point, more "bad" students are probably forced to take the TOIEC in Japan than in Korea. This is why the mean average provided by ETS is misleading.
| Quote: |
My wife has and overall the students don't do well.
She was even asked to help edit a TOEIC book that called "Very Easy".
Lots of Koreans may be more motivated since by taking the TOEIC they can get a job, or a better one. |
My contention has never been that Koreans were not better at English than Japanese. They may be... but I am not familiar with any research that has adequately explored this issue (although it might be out there). My contention continues to be that by comparing a sample of Japanese students that is relatively broad, against a sample of Korean students that is skewed towards better students, it is wrong to say "Koreans are better than Japanese at English". Further, anecdotal accounts are also lacking in reliability. |
|
| Back to top |
|
|
reasonJP
Joined: 17 Jul 2008
Posts: 48
|
| Posted: Mon Dec 01, 2008 7:53 am Post subject: |
|
|
Statisticians and linguists, like the good ol' combination of oil and water. For the record, I'm with Chris, despite nearly failing my maths degree before moving to English teaching.
Oh and read The Black Swan by some guy I can't remember. 's good. |
|
| Back to top |
|
|
Khyron
Joined: 10 Mar 2006
Posts: 291
Location: Tokyo Metro City
|
| Posted: Mon Dec 01, 2008 10:56 pm Post subject: |
|
|
After working in both Japan and Korea, in my observations in daily life, Koreans, on average, are much better at English than Japanese are.
I did not take into consideration TOEIC scores when making these observations.
Due to Koreans being better overall at English, the OP is having trouble finding work in Tokyo. Isn't that right? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
This page is maintained by the one and only Dave Sperling.
Contact Dave's ESL Cafe
Copyright © 2018 Dave Sperling. All Rights Reserved.
Powered by phpBB © 2001, 2002 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
