13 December 2010

Something Smells Funny

In medicine, where a screening test has false positivies and false negatives such that the test brings with it risks as to its usefullness; the test is dropped.

For an example, consider the Telegraph report, "Prostate cancer test is too risky, say doctors" where,

"It has been claimed that the test leads to over-diagnosis as it cannot distinguish between cancer and other conditions, such as a benign enlargement of the prostate or a urinary infection. Nor is a negative result fully reliable, as it can miss a tumour and dangerously provide a false reassurance."

A test that has its effectiveness judged on the basis of its false positive and false negative rates seems sensible to me.

Now, imagine if someone said that they had trained a dog to sniff out cancer, would you believe them?

I doubt it. Nevertheless, if false positive and false negative rates were provided in support of the claim we might think differently.

So, why do we accept this screening test when it comes to drug screening? That is, using sniffer dogs to indicate whether or not someone is carrying drugs?

I've blogged about this in the past "Validity of Your Assay".

Someone was stopped and searched leaving a tube station on the basis of the behaviour of a sniffer dog towards him. He tried to seek a legal remedy and that was the last that I heard. Periodically I would search for the case but nothing came of my searches.

I thought of this case again when I read Leveson LJ's recent lecture about expert evidence. On the grapevine, I found that there was an attempt to get a hearing but the court would not allow legal aid. At which point the case ground into the dust.

So, we've got a situation where a screening test is used as the basis of reasonableness in justifying the stopping and searching of someone. However, no criteria that describes the validity of the test is given by those operating the test. Further, reports suggest that the screening test has a false positive rate of 75%.

Not only that but the screening test can be gamed either unwittingly, or wittingly.

Animals are sensitive to their handlers unwittingly giving them cues, see, Clever Hans. Cues which would simply be reflections of the handlers prejudices.

The handler could also train the dog to behave as though drugs were present on the basis of hidden cue.

All this because Release did not manage to have their case heard.

(The Guardian has another example).

Update 19th February 2011. It appears that someone else has grasped the significance of the Clever Hans effect to sniffer dogs. You can read about a study reported by the Economist on the Mind Hacks blog, Sniffing Out the Unconscious.

Why wasn't this study done years and years and years ago? Talk about science being manipulated by those who control the purse strings.

No comments:

Post a Comment