There are two things I'd like to discuss, but we'll split up the blog posts for brevity. The first is the trouble with scientific data, and the second is the problem with toxicity testing as it stands today (I'll give you a hint, it has to do with dose). Let's start with scientific data.
After reading Paul's most recent post I realized it would be prudent to discuss data (no not the cyborg on Star Trek). More specifically, discuss scientific data. Why? We need to come to an agreement here on the degree of accuracy for facts contained in this presentation. After a brief overview of how we get from raw data to the statements in this presentation I think you'll agree that a debate on these facts, in this format, is the equivalent of lowering global sea level using a 5 gallon bucket.
The source page data, and the links and footnotes contained herein are largely from NGO (non-governmental organization) websites. Others are simply summaries/interpretations of data on other websites, all from a 30,000 ft view of the science. The problem here is that the "data" on these source pages really are not data at all. Merely black and white summaries of someone's interpretation of data... which was presented in a summarized and interpreted form for them, which was summarized and interpreted for that person, who read it in the primary literature... which is merely a summary and interpretation of data. I knew we would get to the data eventually. Just like a game of telephone, the message can be altered drastically from its initial form. More importantly, the initial message (the primary literature) may have been a complete crock because of the poor quality of the science.
So let's go down the lifecycle of data starting at the end (the presentation in question):
1) There are over 100,000 synthetic chemicals in commerce today. Only a handful of these have even been tested for human health impacts and NONE of them have been tested for synergistic health impacts, that means when they interact with all the other chemicals we’re exposed to every day.22 - Annie Leonard: Story of Stuff
I know for a fact (but only because this is my field) that this statement is sheer distortion and parts of it are utterly false. But, let's look up her footnote #22 (that is the 22 at the end of that sentence). 22 leads us to...
2)“Of the more than 80,000 chemicals in commerce, only a small percentage of them have ever been screened for even one potential health effect, such as cancer, reproductive toxicity, developmental toxicity, or impacts on the immune system. Among the approximately 15,000 tested, few have been studied enough to correctly estimate potential risks from exposure. Even when testing is done, each chemical is tested individually rather than in the combinations that one is exposed to in the real world. In reality, no one is ever exposed to a single chemical, but to a chemical soup, the ingredients of which may interact to cause unpredictable health effects.” From Coming Clean Campaign’s Body Burden
information, retrieved 11/8/07 from http://www.chemicalbodyburden.
org/
Ok, so Annie got her information from the Coming Clean Campain's "Body Burden" information on chemicalbodyburden.org. I'd like to point something out here. Annie isn't getting her data from scientists at places like the National Academy of Sciences, or the National Toxicology Program, or even primary literature. Her information is coming from a group who's sole existence depends on lobbying for a particular cause. It doesn't seem like exactly an unbiased place to get one's information. Not exactly balanced information. Roughly the equivalent of watching Fox News to get a balanced viewpoint. But let's go to the site and see where they got the data to make that statement.
3) http://chemicalbodyburden.org/whatisbb.htm
Under the heading "What is Body Burden", about half way down you will find the statement that was used for Annie's proof. The reference for this statement? Non-existent. In fact, looking over this website gave me chills in general. There is a high-degree of spin and stretching of facts, applying facts incorrectly, and falsehoods. So how can we backtrack the data now? Well, if you go to the about us link on the page of this organization (http://chemicalbodyburden.org/aboutus.htm) you'll see that their scientific toxicology staff consists of 1 person, namely: Dr. Harmon, PhD, Toxics Campaign Scientist
Dr. Harmon is a member of greenpeace, and is helping out with this organization. I am sure that Dr. Harmon is a excellent scientist, but even the best scientist in the world would have trouble keeping up to date with all of the changing literature and all of the data needed to support the generalized statements made on the website. There is simply too much out there to be an expert of all.
So it looks like the reference trail has gone dry. But let's imagine where the data possible came from. We can put a big black box.
4) The Black Box
The black box here is the X number of individuals/interpretations between the statements on the chemical body burden site (#3) and primary literature (#5). So step 4 is potentially step 4, 4.1, 4.2, 4.21, 4.22 etc... before we get to step 5. Who knows how facts get changed here.
5) Finding Primary Literature
AKA Publications. If we do a NCBI pubmed search (http://www.ncbi.nlm.nih.gov/sites/entrez) on BFR, for example, we get 498 hits. Here is one example of the 498 hits: http://www.ncbi.nlm.nih.gov/pubmed/18315715?ordinalpos=2&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Enjoy reading the abstract. After that, read all of the pages of that publication. Now read the other 498 references. I'll wait.
After that read about dioxins. That'll produce 13,275 hits.
Actually, that is really too specific, do a search of toxic and you'll get 142,197 hits.
Why don't we start with reading 1 paper
6) Reading Primary Literature
For each of these publications you'll have to read the author's interpretation of the data, and look at the raw data yourself to determine if you agree with the author's assessment. In addition, the raw data in publications isn't really raw, but presented in a certain way by the author.
Based on my calculations, we have between 6-492 people standing between Annie's initial statement (remember Annie, she is in the story of stuff) and the data. That is a disturbing game of telephone.
Bottom line. You can find scientists who fall on either side of every single argument and fact in science. On top of this, many of the things Annie is saying is simply incorrect, as is her source. If you'd like to read something on mixtures, I recommend this: http://www.ncbi.nlm.nih.gov/pubmed/16081521?ordinalpos=2&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Therefore, having a debate on the validity of these facts is not a useful exercise. Suffice to say that there has been tremendous research on mixtures over the years, and it seems to be overall a phantom issue since only those with common MOA should even be in the realm of concern. On top of all of this is the subject of the next posting : The Dose.
Subscribe to:
Post Comments (Atom)
5 comments:
After reading this post I feel very similar to the guy towards the middle of Flowers for Algernon.
In brief, Charlie goes from mental retardation to very smart in a few chapters and now that he can think for himself he goes to someone he had hitherto considered "wicked smaht" who was a university professor in Geology I think. Charlie asked the professor a few questions and followed those up with more and was very quickly told by the professor that he was out of his realm of expertise and Charl would have to ask someone else who was an expert in this other topic to which the conversation had led which is very closely related to, but not quite the same as, his expertise.
Charlie goes to this other guy and the same thing happens 3 or 4 times until Charlie realizes that these people are all stupid. They don't really know anything other than this tiny slice of life that they spend their whole lives staring at and anything outside of that is as much a mystery to them as _everything_ was to Charlie before he became "wicked smaht".
However, the world simply can't behave like this. I'm not entirely sure which direction you're going with your really fun explanation of primary sources. I think you meant that you can't trust people so you should read the primary source examine the data and decide for yourself. But then you present the ridiculous nature of this task for even the most dedicated person on any mildly broad topic.
If that is what is required to acquire any knowledge on a topic, then no person could possibly keep up to date on enough primary sources to have any valid sourced opinion on _anything_.
In order for people to acquire knowledge in any area there needs to be some level of abstraction of the primary source information. That said, I think an alternative to your primary source theory is, something that the Stuff movie apparently failed to do, is to do research on a topic by gleaning information from a variety of sources, albeit none of them primary, but most usefully from sources with a variety of motivations for doing their own research to begin with.
I really can't blame Annie for not reading her own primary sources, but maybe reading one more study after reading one that she liked would have made any conclusions presented in the movie more solid. Also, (thanks Wifey for mentioning this) arguments in either direction are always much stronger when the person presenting their point indicates that they understand counter arguments and are prepared to refute them.
I really enjoyed reading your comment John. Your Flowers for Algernon analogy is spot on. Actually I was entirely agreeing with your points until you wrote in paragraph 4 "however, the world simply cannot behave like this."
This is exactly the manner with which the world behaves! That is, partly, my point. Perhaps not the extreme of FfA and Charlie, but there is quite a bit of truth to it.
Why? There is simply too much level of detail out there for anyone to possibly digest. So people pick their slices and become specialists. This is particularly true of academia, where you find your niche and dig in for several decades until you become one of the top 10 or so people in world on the topic.
It takes decades of reading understand all of the science in your particularly slice that came before you, while trying to keep up with the new research.
In industry, they largely take a different approach. Where I work, everyone has a slightly different specialty so we cover all of the bases when working as an extremely large unit. You bring people in when you need a particular specialty. Even here, people specialties are FAR broader than in academia.
My current specialty is developmental and reproductive toxicology. Let's compare this to my previous specialty in academia.
First we lose the phrase developmental. Now we've left with reproductive. But that implies male and female, and I was just doing male. Also, not just male but the testis. Not the whole testis either, mind you, but just the seminiferous tubules. And I wasn't really interested in the entire seminiferous tubules from both testes, just the one undergoing injury. Right so now we have a specialist in the seminiferous tubules of the left testis. Oh, and I wasn't interested in the Sertoli cells (supportive somatic cells), just the germ cells. And frankly, not all of the germ cells, just the meiotic ones. And not the secondary meiotic, just the primary. And frankly, since I was studying a transcription factor I was interested in the nuclei only. Oh, and I didn't give a damn about any species other than mouse.
Let's compare, shall we? Developmental and Reproductive Toxicologist vs. Male Mouse Left Testis Seminiferous Tubule Germ Cell Primary Spermatocyte Nucleus Toxicologist. Right.
All of this leads to my point. Yes there was a point in there. The point is that after reading paul's post it became clear that debating the scientific accuracy of the facts is a futile process. Because we would have to go to the primary literature to get the data, or at least summaries of primary literature published in scientific journals so as to remove the clear conflict of interest from the sources Annie used.
I could send 50 or so articles showing all of the work that has been done in the area of mixtures and BRF etc... but this venue doesn't seem appropriate for that discussion.
Instead I propose to move on to the way the data was presented, and request Paul to acquiesce that Annie failed to show the other side of the argument, at best, and also made some false statements.
Incidentally, my next posting on "The Dose" is intended as a general, hopefully brief, overview of the overly alarmist nature of high dose toxicology data.
Reza, you ignorant slut.
Oh wait, sorry, I thought I was on a cable news debate.
No, seriously, I think we may have missed each other on this and its my fault because I said "cannot behave like this". But I didn't say what "this" was or what I meant. I think in conversation this would have been clearer but I was lazy while writing it so it didn't come across. Let me try again.
The world _does_ work like this where people become experts in this tiny little slice of excrutiating detail and not know anything else in detail. This is exactly what you did and saw other people do in grad school. We've talked about this a few times about how people spend decades carving out their own little niches of expertise.
However, the world _can't_ work like this in the sense that most people don't and can't develop the depth of knowledge required in the many areas that any reasonably high level conversation requires to speak authoritatively and from primary sources on all aspects. For example, its not possible for a person to be an authoritative, primary source reading expert on, say, the environment. That's an outrageously large topic and there are not enough lifetimes to really know it all in that depth.
But what people can do, and do do (he he, I said dodo) is to roll up information to the next level of abstraction. People who collect the data and know their slice at the highest level of detail properly publish their data to their peers for, idealy, review and debate. Then, the many slices of research in the myriad highly specialized areas start to form a larger picture where another level of people, that consume the primary sources present a higher level of abstraction, such as books and detailed articles in topical magazines. These sources are then consumed to yet a higher level and are reported in the mainstream media and eventually end up as some jackasses talking points on MSNBC or something.
So its only the last part there that I think is the problem. I think the consumers of primary sources really serve a purpose to the overall debate by allowing people who can't possibly consume all of the primary data to get a summary of the issue. The problem that can happen here is that here, as with all levels of the chain, bias can be introduced.
This is why my main point about Annie's movie is not that she didn't use primary sources and should be flogged for that, but that she chose not to get her information from more than one, clearly biased source. It is that purposeful ignorance of dissenting viewpoints, and not the lack of primary sources that undermines the movie.
What I'm saying is that its simply not possible understand and discuss higher level topics, such as energy, environment, medicine, areas with large amounts of scientific backing by only reading primary sources. It is the distillation of data that allows people to participate and understand the world around them and it is just their responsibility to be judicious consumers of the data at whatever level they choose to consume it.
You can eat at McDonalds all the time and you can just read websites with bias built into the freakin domain name to understand what is going on around you. Its more or less the same thing.
John and I were talking this morning. And one of the interesting items that came up is that while there are people who spend their lives in the details of things, there are also people (like me) who are responsible for consuming details from those low-level experts, and providing a level of abstraction on top of that.
My success at what I do depends entirely on the ability to consume information that has been abstracted from lots of people like John, and then make educated "guesses" based on that abstracted data. I'm only successful if I have a high success rate.
Yet I don't know how the details of the code work, I just have to be able to make smart guesses.
I would be very scared if I needed to always go to the "primary data" - in the case of software - the source code - to be able to figure out how things work at a larger level.
Mike,
That is indeed the way companies in science work. Academic labs are usually different where the professor is in charge of knowing details on the primary source. The reason for this is that a large percentage of the primary data is crap (due to poor interpretation or poor scientific methods). So the prof cannot rely on grad students and postdocs for everything.
In scientific companies it works in the way you describe. You have people who are responsible for knowing details, and others who utilize these people as resources in solving problems.
I would argue, however, that the difference between this structure and what is going on in the story of stuff is one of understanding primary data from bottom to top. In other words, you rely on John to know the primary source, but you have a level of specialty where you understand the nuances of what he's talking about. Why? Because there was a time when you did a job like John's. So as one progresses up the ladder, they still retain that general technical knowledge but worry less about the details.
The story of stuff is the equivalent of having John explain the code to me. I don't understand it on the same level as Mike, therefore something is lost in the translation.
Post a Comment