Just Curious: Why Does LSAC "Remove Items from Scoring"?

[Adrianna W.]

Anyone who took the Nov. 2018 LSAT or was following how that went down saw that in the Reading Comprehension section, the final question of the section (which was about the Carroll/Chen research regarding the way our universe may have originated) was removed from scoring. (This was already a 100-item test as written, and removing this question made it a 99-item test, which is not unheard of but pretty rare in the released LSATs.)

Out of curiosity, does anyone at PowerScore know what circumstances typically lead LSAC to do this? Do they do this with questions that huge majorities of students got wrong, such that, in combination with re-assessing the way the question was written, it indicates to them that something about the question was worded in a confusing or unfair way? If so, what do you imagine would be the percentage of students that would have to get a question wrong for the LSAC to review it for unclear wording and consider removing the question from scoring? Do they ever do this with questions a lot of people got right but that, after reviewing them, the testmakers/scorers conclude were still written in such a way that they shouldn't count towards people's scores?

What are some other reasons they might do this (redundancy, the answer is somehow contained and revealed in the question stimulus itself)?

This isn't particularly germane to the study process, I was just wondering if anyone had any insight about this.

[Dave Killoran]

Anyone who took the Nov. 2018 LSAT or was following how that went down saw that in the Reading Comprehension section, the final question of the section (which was about the Carroll/Chen research regarding the way our universe may have originated) was removed from scoring. (This was already a 100-item test as written, and removing this question made it a 99-item test, which is not unheard of but pretty rare in the released LSATs.)

Out of curiosity, does anyone at PowerScore know what circumstances typically lead LSAC to do this? Do they do this with questions that huge majorities of students got wrong, such that, in combination with re-assessing the way the question was written, it indicates to them that something about the question was worded in a confusing or unfair way? If so, what do you imagine would be the percentage of students that would have to get a question wrong for the LSAC to review it for unclear wording and consider removing the question from scoring? Do they ever do this with questions a lot of people got right but that, after reviewing them, the testmakers/scorers conclude were still written in such a way that they shouldn't count towards people's scores?

What are some other reasons they might do this (redundancy, the answer is somehow contained and revealed in the question stimulus itself)?

This isn't particularly germane to the study process, I was just wondering if anyone had any insight about this.

Hi Adrianna,

Thanks for the question! To be honest, LSAC doesn't release a lot of information about why they remove items from scoring, and they never release percentages on those questions. So, we cannot draw a specific conclusion about the exact models they used in removing questions. That said, we do know they remove questions that do not meet these three criteria:

1. is clear and unambiguous,
2. has one and only one best answer, where the best answer is the one among the choices provided that most accurately and most completely answers the question that is posed, and
3. meets the LSAC standards for fairness and sensitivity.

And that, "If the test taker demonstrates that the test question does not have one and only one best answer among the choices provided, corrective action is taken." Note: policies drawn from https://www.lsac.org/about/lsac-policie ... -questions.

So, there are different reasons a question might be removed, and in the case of the RC problem you cite, there's no way for us to know for sure what they saw in the question!

I know that's not the specific answer you sought, but we can't see into this particular box due to LSAC policies. Thanks!

[Adrianna W.]

Thanks for answering this, Dave!