Khan Academy

[allielynn]

I listened to the March Crystal Ball podcast, and I thought I heard them say that they would put in the notes which PT's are a part of Khan Academy's program. I can't find this in the notes. Can you please provide a link if theses are published anywhere?

[Dave Killoran]

I listened to the March Crystal Ball podcast, and I thought I heard them say that they would put in the notes which PT's are a part of Khan Academy's program. I can't find this in the notes. Can you please provide a link if theses are published anywhere?

Hi Allie,

We haven't finished posting the complete notes yet, but here's the list of Khan Academy’s 11 full PTs + 2.5 additional tests used as experimentals:

Initial/Diagnostic: PT 78 (June 2016)
Test 1: PT 66 (June 2012) with Exp RC PT 53 (Dec 2007)
Test 2: PT 68 (Dec 2012) with Exp LR1 PT 53 (Dec 2007)
Test 3: PT 70 (Oct 2013) with Exp LR2 PT 53 (Dec 2007)
Test 4: PT 69 (June 2013) with Exp LG PT 53 (Dec 2007)
Test 5: PT 54 (June 2006) with Exp LG PT 55 (Oct 2008)
Test 6: PT 76 (Oct 2015) with Exp RC PT 55 (Oct 2008)
Test 7: PT 79 (Sept 2016) with Exp LR1 PT 55 (Oct 2008)
Test 8: PT 81 (June 2017) with Exp LR2 PT 55 (Oct 2008)
Test 9: PT 64 (Oct 2011) with Exp LG PT 63 (June 2011)
Test 10: PT 67 (Oct 2012) with Exp RC PT 63 (June 2011)


Thanks!

[allielynn]

Thanks so much!
I ran into a question that I didn't know if anyone has seen. I took Khan Academy PT 1 and started to BR the EXP section 53 RC. I did the 3rd passage and realized it wasn't the same as online Khan has. Khan has a completely different passage with 7 questions, where the main point first question correct answer is "While Europen immigrant and African American mutual aid societies in the U.S. were similar in their purpose and activities, there is evidence that the African American societies were influenced by mutual aid traditions in West Africa." The passage 3 that I am seeing for RC 53 is a comparative reading passage with 5 questions that is over business interest and university research. Do you know which test the above mentioned passage comes from, so I can BR it? Also, this makes the RC section have 29 questions instead of 27, which would have an impact on overall score/timing of this section. Any ideas on why they changed it out and decided not to have a comparative reading passage?

[Jon Denning]

Hi Allie - thanks for the follow up! And what an interesting (and strange) discovery...

Here's what I can tell about Khan's PT1 and both sections of its RC:

The first RC is real and taken from June 2012 (PT 66), but even it exhibits an oddity: on the original June 2012 test the first passage was about Digital Publishing, but for some reason the Khan test has moved that to passage 4, and shifted the other three passages up one, so the original passage 2 is now first, etc! The overall content is the same, but the order, for no discernible reason, has been completely rearranged.

I have no idea why they'd do that—my suspicion is that Khan just made a mistake and haven't bothered to correct it—but it changes the fundamental nature of the section is a worrying way: with the passages reordered you're experiencing the test differently than those who originally took it...i.e. those for whom the original curve/scale was created. So immediately I'm forced to view this test as regrettably "other" than the authentic experience it's purported to represent

And the experimental RC section is, somewhat unbelievably, even worse!

What's happened in the second (exp) RC is they've replaced the entire third passage from December 2007 (PT 53)—a passage that was originally Comparative Reading with five questions (about Research Commercialization)—with a single passage that has seven questions, and doesn't appear to be from any released LSAT. The result is that this section now has 29 total questions, which is more than any RC section has ever had, and no Comparative Reading at all! It also has a passage (#3) that isn't on any available test, so trying to Blind Review it or even find explanations for it is impossible.

In other words, for no apparent reason, they've created an RC section that's too long, that's missing a passage construction that appears on every test, and that has a rogue passage taken from some mysterious source....possibly even a non-LSAT source, where Khan created a simulated passage themselves and decided to throw that in (I have no idea if that's the case, but given that it's not from any available LSAT I can't think of an alternative; I even checked the free LSAT India tests and it doesn't come from those either).

So that's the rundown of what you get in RC on Khan's PT1.

It also illustrates why I, and so many others in the industry, constantly caution students about relying too heavily on Khan as they prep. It's great that there is more free test content available—kudos to LSAC for taking the initiative there!—but the way in which it's presented and administered is so half-baked and sloppy that in many instances it's doing more harm than good. So by all means use it, but only as an occasional supplement, where you proceed with caution and recognize that it's often just a well-intentioned monument to mediocrity

[LakeShow]

Looks like they did some of this with the PrepTests for the digital site as well (familiar.lsac.org). I've only take PT 73 on there so far but the order of the sections was different than the version in the book (10 Actual, Official LSAT PrepTests Volume VI). The self-study portal order matches the book as well.

Jon, I'd be curious your thoughts on how that might affect scoring, ignoring the whole digital versus paper aspect? Online portal is LG, LR1, LR2, RC whereas actual administration is supposed to be RC, LR1, LG, LR2.

You guys cover some of this in the podcast but NOTE for anyone else that when you take the test in the online portal (not Khan) you get the results for each section right away, you have to manually End the section even after the time has run out, and you have to go back to the home screen and then select your next section in order to move forward with the test. Khan was much better as simulating the these aspects of what the actual test day will be like.

Best of luck to all!
-Mark

[Jon Denning]

Great points here, Mark!

It's certainly curious, isn't it? Why arbitrarily change the original? I wish I had an answer...or at least some plausible guess beyond just a wanton sloppiness that seems to have infected the whole affair.

As to how it affects scoring data, well I don't really know that either. My suspicion is that it'd have little impact, aside from how section orders favor certain individuals over others: having RC first vs last, for example, is going to alter the test experience from student to student, where some would find it beneficial while others are penalized. So I think what you'd likely see are overall results unaffected (in any notable sense) while person to person the outcomes could be far more dramatic, since section order is subjectively significant.

Still a total mystery why they'd intentionally alter the presentation though. Which again leads me to believe that we're witnessing these tests deployed without intent :/

[LakeShow]

Thanks Jon and well stated. What gets me is that LSAC still has the line "we will continue to add more in the weeks and months ahead" on their site in the Digital FAQ section (it's been there for many weeks without update) and they claim: "These practice tests provide an experience that mirrors the Digital LSAT." I feel bad for those people who aren't lucky enough to have discovered your podcast or some other source that explained what the test will actually be like on the tablet. S/o to you for getting to take a pilot and passing along that knowledge.