#23 - When 100 people who have not used cocaine are tested

[Rachael Wilkenfeld]

Hi Ashpine,

Yes, our conclusion is that in a group of 100 people, most of those who test positive will be actual cocaine users. The key is that we don't know how many of the population are cocaine users. Let's say out of every million people, one has used cocaine. If you take a random sample of 100 people, you aren't likely to pull that one person. So that means that likely all the positives in that group of 100 would be false positives, not true positive. On the other hand, let's say that out of every million people, 999,999 have used cocaine. That means that it is likely in your group of 100 people that most of your positives are true positives. That is the flaw answer choice (C) is describing.

Hope that helps!

[simonsap]

Sampling bias.

eg. take a lie detector

Scenario 1) out of 100 people who never lied, 5 will be falsely accused of being liars
Scenario 2) out of 100 people who all lied, 1 will be falsely acquitted

argument says that if I take a 100 randoms and I run a lie detector on them, most of those that test as liars were indeed liars.
But what if my sample of 100 randoms were people who never actually lied, like in scenario 1? Then 3 or 4 out of 5 really were liars? Scenario 1 shows that this conclusion is wrong.

To make such a conclusion, we need to know how well our selected samples generalize into the population at large.

[ashpine17]

I feel that I’ve seen this error before. What is this called? Why would the argument maker think the conclusion makes sense? Is he assuming the people being tested are mostly if not all, are actual cocaine users?

[Rachael Wilkenfeld]

It's a problem with base rates, ashpine. The author thinks their argument makes sense. And it does, if the population of cocaine users is the same as the population of non-users. But that hasn't been established by the argument. While I don't know the exact rates, I would expect that there are fewer cocaine users in the world than non-users. Let's play with some numbers!

Pretend we have a population of 100,100. Of those people 100,000 are non-cocaine users and 100 are cocaine users. Based on the premises here, 5,000 non-users would test positive, while only 99 users would test positive. That definitely doesn't work with the conclusion.

Let's flip the numbers. We'll keep the population at 100,100. Now 100,000 are cocaine users and 100 are non-users. We'd have 99,000 positive tests from users, and 5 positive tests from non-users. That would work with the conclusion.

Now let's pretend we have an exactly equal population of users and non-users. 100 users, 100 non-users. 99 users would test positive, and 5 non-users. That also works with our conclusion.

It's only when the population is heavily weighted toward the non-users that the conclusion doesn't follow. However, we don't know the population size here, and we can't rule out the possibility that the population is mostly non-users.

Hope that helps!

[christinecwt]

Hi - can anyone help explain why Answer Choice D is incorrect? Many thanks!

[Adam Tyson]

Answer D is incorrect, christinecwt, for two reasons. First, the author does not ignore that fact! They cited it in their data, and their conclusion is unaffected by that 1% who have a "false negative" result. The second reason is that, even if the argument did ignore that 1%, that would have no affect on their argument about the vast majority. The real problem is that there could be a huge disparity between the two groups. Maybe a random and representative sample of the population would include almost entirely people who have never used cocaine? Then most positive tests would probably be false positives, even if all the actual users also tested positive.

Imagine this scenario:

.1% (one out of a thousand people) of the population has used cocaine. We select 1000 people at random and test them. If that group was representative of the whole, we would probably get about 50 positive tests, and 49 or 50 of them would be false positives. Regardless of whether the one actual user tested positive or not, there would be far more false positives than true positives! So even if the author had ignored the possibility of a false negative (our one cocaine user testing negative), their conclusion about most of the positive tests being correct would be very, very wrong.