01 December 2009
Scientific Validity - does the principle support what it purports to show?
Scientific validity is a difficult and slippery idea; just because the scientific method is being applied, hypothesis followed by experiment to test the hypothesis, doesn't necessarily mean that the conclusion is valid.
Whether or not the process is valid will depend upon the elements that make it up, such as, what hypothesis or model is being used. Is there any bias present, is the data accurate, reliable? Are there any ethical considerations that may have skewed the data?
Hypothesis or Model
Consider the question, is there such a thing as drug addiction? In order to answer the question, we put some rats in a cage and give them two food/water supplies: one clean, one adulterated with something such as an opiate.
In this experimental set up we can collect data which shows that the rats eat the opiate supply in preference to the non-opiate supply. Which proves it, doesn't it?
What if the rats are taking the opiates in order to relieve themselves from living a miserable existence; alone, trapped in a tiny cage as they grow older and die.
What would happen if the rats could live in groups, with lots of space, in a pleasant interesting environment - would opiates be addictive in this case?
This was the situation in Rat Park, follow the link to find out what happened.
Here's another hypothesis - it is possible to tell from the bumps on your head your personality, character and general intelligence. At least, that was what was believed not very long ago.
Are the theoretical underpinnings of the research sound?
For example, is there such a thing as race (in an objective rather than subjective sense)?
Where is the funding coming from? Are you reporting to a political party and would you get sacked if you produced the "wrong" results?
Is the data from a sampling study? Was the sample size large enough, were the samples taken random and so representative of the whole?
Is the data reliable? For example, where and how were the temperatures taken? Is a fraction of a degree real or experimental error?
How about presenting data; is it all there, or has some been left out because it doesn't support the hypothesis?
Drug trials are conducted under ethical rules, instead of testing drugs on people (initially, that is) they are tested on animals. But is the animal an appropriate substitute?
There is a lot more to write than the above, what is written is just a small taste of the problems and difficulty associated with the idea.
Update - Climategate and Scientific Conduct, Derek's post on some work that has questionablel validity.
2nd Dec 2009 - Another note regarding validity from Org Prep Daily.