Digital LSAT

[Broncos15]

Hello,

I read the last blog post and was intrigued about the implications of a digital LSAT.

Given the large role of the LSAT in admissions - compared to other tests in other fields where in some grad programs for example the GRE is just a box you check off on the application checklist- i think there could potentially be some negative implications of a digital LSAT.

While undoubtedly, it would be much cheaper and logistically easier to use, i feel the cons outweigh the pros ( at least as a test taker)

I bring this up because the LSAT is designed to test candidates for specific skills needed for law school success ( part of why it is a big part of the admissions decisions)- In other grad programs Standardized test validity as not as big because the GRE is used for all fields as wide ranging from English to Engineering ( even Business schools have begun to accept GRE's)- so Admissions officers are aware the GRE and the mutipurpose use of it

-The first concern would be validity concerns- Since studies/experiments do not always translate into real life situations ( LSAC would be conducting some before implementing this drastic of a change)...how long would it take initially for the reasoning abilities of a 170 scorer on a paper exam to reflect as a 170 on a computerized exam since with anything human involved there is always a potential for error

Since some ( arguably the vast majority) of students like to annotate and highlight ( even if it is just a few notes) in Reading Comp and may "need" to do that to score well would be at a disadvantage compared to someone who did not need to highlight or make notes on a paper based LSAT------ ( i use the word validity concerns as the one of the purposes of the LSAT is to measure skills needed for law school , and is a stronger predictor than the GPA for 1l performance ....and as a result it would be foolish to suggest test taker A has does not have the skills to succeed in law school compared to test taker B if A 'needed" to annotate in the text and B didn't - since when purchasing textbooks and studying during law school students have a choice of which format to use .....the paper exam better reflects this freedom in choice- those that want to make notes on the exam can, and those who do not need to can choose not to annotate



Timing Issues- the LSAT is much more dependent on time than other standardized exams, and the test takers do acknowledge this in making accommodation request harder to obtain. And there is a bit of data to suggest people read a bit slower on computers than on paper in Reading Comp for instance compared to the GRE, there are much more questions on RC ( 28) in a larger amount of reading, with reading the answer choices takes up a chunk of time too.
Logic Games would also present a problem- even if lets say they provided scratch paper, 35 mins would present a challenge because of the spatial distance between computer screen and paper( looking up and down between the two) compared to the current paper format


LSAT Format less conducive to computers-- as mentioned before the larger amounts of Reading in RC ( in both passages and answer choices ) compared to the GRE would make it a bit less conducive as there is much more reading going on with the additional challenge of stricter time constraints......in LR you could not circle the conclusion or underline key words

I also feel the LSAT is less conducive because of the nature of the questions and answer choices in each of the sections compared to the GRE, Process of Elimination ( POE) is used more frequently by many test takers on the LSAT since the answer choices are less cut and dry than a Vocab word on the GRE.....on the LSAT many test takers look at all answer choices even after they pick the their answer (even if briefly).....Mentally blocking out ( since you can't physically cross out an answer choice) an entire sentence/a few lines of text in an LSAT answer is much harder than mentally blocking out a single word ( GRE vocab)


Lastly and perhaps more importantly How would admissions decisions and law school medians be affected by this change- i could see a challenge as an admissions officer trying to interpret and compare the exam scores in the pile of applications between students who took the paper LSAT and digital ( as scores are valid for 5 years) LSAT ..I also feel LSAT medians within schools would drop a few points under a digital LSAT because of the difficulties I mentioned ( unless LSAC made the test easier to compensate for these issues)



Thanks!

[Jon Denning]

Hey Broncos,

Thanks for the message! I figure since that was my blog post I'd perhaps be best suited to carry on the discussion

You're absolutely correct that the LSAT couldn't be made digital as-is! In other words, there's no way the test makers would think it reasonable or fair to suddenly just put the same content, format, and timing stipulations into a digital form; the migration to computer screen would produce wildly different (arguably worse) results than those currently seen, which essentially defeats the whole purpose.

My point was simply that administering tests digitally in some form is clearly desirable, as some of the biggest, and more widely-respected, tests like the GRE, GMAT, and MCAT have all moved successfully in that direction. Hence it makes sense that LSAC would consider a similar transition at some point in the future, and reconstruct the test (and possibly test content/guidelines) accordingly so that it's still a measure of what it purports to be.

Thanks again for the lengthy and insightful comments! And fear not. If you're in LSAT prep mode now there won't be any changes to the test that affect you, and certainly nothing as drastic as a computer-based version until you're already out of law school!

Jon

[Broncos15]

Thanks! As a follow up ( of course it is hypothetical and speculation) but do you think one reason why the LSAT out of the major graduate school standardized tests has remained in paper format is because of "if it aint broke-don't fix it"?


For instance the curve itself speaks to the rigor of the exam..If you scored an 88% out of 100 on a regular undergrad test, that would be solid but not valedictorian material...but on the LSAT getting 88 out of 100 questions right could give you a 170 ( a top 2% score)

A better example ...is how the LSAT devotes the last 7-8 points ( 172/173 and above) just to split hairs among the top 1%

This type of hard exam would provide better information to law school admissions counselors about applicant's preparedness for law school compared to an easy watered down LSAT...I'd imagine if LSAC did ever want to make a change they would want to keep approximately the same curve intact which would present another difficulty to keep in mind if they changed the format

[Dave Killoran]

Thanks! As a follow up ( of course it is hypothetical and speculation) but do you think one reason why the LSAT out of the major graduate school standardized tests has remained in paper format is because of "if it aint broke-don't fix it"?

Let me jump in here, if you don't mind Actually, I don't think this is their mentality. The fact that the test works allows them to feel no pressure to change immediately, but eventually this test will change.

When we go back in history, LSAC has a solid track record of considering a computerized version of this test. For example, check out just some of the research reports they've authored about computerized testing: http://www.lsac.org/lsacresources/research/all/ct. A number of years back I went to one of their law school forums where they were testing a computerized version of RC. It was very similar to what they have now, and could easily have been used to produce valid results. There was even a point where LSAC stated that a computerized test was likely to appear within 5 years or so, but they went back to the drawing board after running into some problems.

There are different ways to implement a computerized question structure on this test, including both sectional and question-by-question adaptability. The one thing that has been proven however, is that a computer format test can be made in such a way to reproduce the results of a paper test. So, that's not the issue here (and never really has been the issue once the basic stages were passed). There are bigger problems for LSAC in test distribution and administration, test security, and the required question pool. Those issues are what I think have held this back, and the efficacy of the paper test has allowed them to be relaxed about it. But when I look down the path, I don't see a world where the paper test persists. At some point the LSAT will either be supplanted by a different exam or it will transition to a computer format. That's just the way of the world

When the test moves to computer, it could be a good thing or a bad thing. If it uses a format like the GMAT where you are forced to go question-by-question, then it's a bad thing. That format penalizes intelligent test takers. If they choose a format like the GRE where you can move around inside the section, that would be much better. Eventually, though, we'll have a 3D online environment which will eliminate a lot of these issues. That might not be for another 20 or 30 years, but it's coming, and when it does the testing world as we know it will change radically.

I do think that if they move to a computer format any time in the next decade, you'll see the test come along with standard notation tools like highlighting and tags. The original exam I checked out years ago had a highlighting feature, so it was on their minds even then.

I guess that, in summary, the question to me isn't if they'll make a change, but when. There's a massive amount of extant research on CAT testing out there, and they can confidently make the move without worries of test validity. And they've made major moves before (as have have other tests), so even that isn't new to them. There will be a transition period as admissions officers use both tests, but they too will adjust. The good news here is that I don't perceive this as occurring any time soon (and the longer the better).

Last point: I don't think it matters very much to LSAC what students want in the test or what they think of it (and that statement isn't made as a criticism). As long as the test is predictive and reliable, law schools will continue to use it, and ultimately the law schools are the real clients and partners here that have to be satisfied. A computerized test will be different, and some parts of it will probably make people happier, and others will make people less happy. I think LSAC accepts that as part of the landscape they operate in.

Thanks for the comments—this is always an interesting topic to us!

[Dave Killoran]

Coincidentally enough, today LSAC released their bi-annual newsletter, and in it they said:

• “In a similar vein, we are considering ways to enhance our LSAT-orientation offerings
  online. We are in the early stages of this exploration and will update you as we know
  more. Leveraging of our knowledge base and inventory of LSAT items is looming large
  in our crystal ball.
  
  Of course, this perfectly segues into our vision of a Digital LSAT. This vision is still a bit
  blurry in the crystal ball, because there are as many or more hurdles to overcome as there has been progress made; fortunately, there will be repeated opportunities at the annual meeting to see the truly impressive results of our development efforts thus far. I encourage you to make time to go to one of the Digital LSAT sessions to see for yourself.”

Note that second paragraph. It's coming, but not for a few years still. But they are clearly getting a bit stoked about it!