There's an interesting follow on debate in the comments of Reza's last post "The Trouble with Scientific Data".
Enjoy!
Tuesday, March 11, 2008
Saturday, March 8, 2008
The Trouble with Scientific Data
There are two things I'd like to discuss, but we'll split up the blog posts for brevity. The first is the trouble with scientific data, and the second is the problem with toxicity testing as it stands today (I'll give you a hint, it has to do with dose). Let's start with scientific data.
After reading Paul's most recent post I realized it would be prudent to discuss data (no not the cyborg on Star Trek). More specifically, discuss scientific data. Why? We need to come to an agreement here on the degree of accuracy for facts contained in this presentation. After a brief overview of how we get from raw data to the statements in this presentation I think you'll agree that a debate on these facts, in this format, is the equivalent of lowering global sea level using a 5 gallon bucket.
The source page data, and the links and footnotes contained herein are largely from NGO (non-governmental organization) websites. Others are simply summaries/interpretations of data on other websites, all from a 30,000 ft view of the science. The problem here is that the "data" on these source pages really are not data at all. Merely black and white summaries of someone's interpretation of data... which was presented in a summarized and interpreted form for them, which was summarized and interpreted for that person, who read it in the primary literature... which is merely a summary and interpretation of data. I knew we would get to the data eventually. Just like a game of telephone, the message can be altered drastically from its initial form. More importantly, the initial message (the primary literature) may have been a complete crock because of the poor quality of the science.
So let's go down the lifecycle of data starting at the end (the presentation in question):
1) There are over 100,000 synthetic chemicals in commerce today. Only a handful of these have even been tested for human health impacts and NONE of them have been tested for synergistic health impacts, that means when they interact with all the other chemicals we’re exposed to every day.22 - Annie Leonard: Story of Stuff
I know for a fact (but only because this is my field) that this statement is sheer distortion and parts of it are utterly false. But, let's look up her footnote #22 (that is the 22 at the end of that sentence). 22 leads us to...
2)“Of the more than 80,000 chemicals in commerce, only a small percentage of them have ever been screened for even one potential health effect, such as cancer, reproductive toxicity, developmental toxicity, or impacts on the immune system. Among the approximately 15,000 tested, few have been studied enough to correctly estimate potential risks from exposure. Even when testing is done, each chemical is tested individually rather than in the combinations that one is exposed to in the real world. In reality, no one is ever exposed to a single chemical, but to a chemical soup, the ingredients of which may interact to cause unpredictable health effects.” From Coming Clean Campaign’s Body Burden
information, retrieved 11/8/07 from http://www.chemicalbodyburden.
org/
Ok, so Annie got her information from the Coming Clean Campain's "Body Burden" information on chemicalbodyburden.org. I'd like to point something out here. Annie isn't getting her data from scientists at places like the National Academy of Sciences, or the National Toxicology Program, or even primary literature. Her information is coming from a group who's sole existence depends on lobbying for a particular cause. It doesn't seem like exactly an unbiased place to get one's information. Not exactly balanced information. Roughly the equivalent of watching Fox News to get a balanced viewpoint. But let's go to the site and see where they got the data to make that statement.
3) http://chemicalbodyburden.org/whatisbb.htm
Under the heading "What is Body Burden", about half way down you will find the statement that was used for Annie's proof. The reference for this statement? Non-existent. In fact, looking over this website gave me chills in general. There is a high-degree of spin and stretching of facts, applying facts incorrectly, and falsehoods. So how can we backtrack the data now? Well, if you go to the about us link on the page of this organization (http://chemicalbodyburden.org/aboutus.htm) you'll see that their scientific toxicology staff consists of 1 person, namely: Dr. Harmon, PhD, Toxics Campaign Scientist
Dr. Harmon is a member of greenpeace, and is helping out with this organization. I am sure that Dr. Harmon is a excellent scientist, but even the best scientist in the world would have trouble keeping up to date with all of the changing literature and all of the data needed to support the generalized statements made on the website. There is simply too much out there to be an expert of all.
So it looks like the reference trail has gone dry. But let's imagine where the data possible came from. We can put a big black box.
4) The Black Box
The black box here is the X number of individuals/interpretations between the statements on the chemical body burden site (#3) and primary literature (#5). So step 4 is potentially step 4, 4.1, 4.2, 4.21, 4.22 etc... before we get to step 5. Who knows how facts get changed here.
5) Finding Primary Literature
AKA Publications. If we do a NCBI pubmed search (http://www.ncbi.nlm.nih.gov/sites/entrez) on BFR, for example, we get 498 hits. Here is one example of the 498 hits: http://www.ncbi.nlm.nih.gov/pubmed/18315715?ordinalpos=2&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Enjoy reading the abstract. After that, read all of the pages of that publication. Now read the other 498 references. I'll wait.
After that read about dioxins. That'll produce 13,275 hits.
Actually, that is really too specific, do a search of toxic and you'll get 142,197 hits.
Why don't we start with reading 1 paper
6) Reading Primary Literature
For each of these publications you'll have to read the author's interpretation of the data, and look at the raw data yourself to determine if you agree with the author's assessment. In addition, the raw data in publications isn't really raw, but presented in a certain way by the author.
Based on my calculations, we have between 6-492 people standing between Annie's initial statement (remember Annie, she is in the story of stuff) and the data. That is a disturbing game of telephone.
Bottom line. You can find scientists who fall on either side of every single argument and fact in science. On top of this, many of the things Annie is saying is simply incorrect, as is her source. If you'd like to read something on mixtures, I recommend this: http://www.ncbi.nlm.nih.gov/pubmed/16081521?ordinalpos=2&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Therefore, having a debate on the validity of these facts is not a useful exercise. Suffice to say that there has been tremendous research on mixtures over the years, and it seems to be overall a phantom issue since only those with common MOA should even be in the realm of concern. On top of all of this is the subject of the next posting : The Dose.
After reading Paul's most recent post I realized it would be prudent to discuss data (no not the cyborg on Star Trek). More specifically, discuss scientific data. Why? We need to come to an agreement here on the degree of accuracy for facts contained in this presentation. After a brief overview of how we get from raw data to the statements in this presentation I think you'll agree that a debate on these facts, in this format, is the equivalent of lowering global sea level using a 5 gallon bucket.
The source page data, and the links and footnotes contained herein are largely from NGO (non-governmental organization) websites. Others are simply summaries/interpretations of data on other websites, all from a 30,000 ft view of the science. The problem here is that the "data" on these source pages really are not data at all. Merely black and white summaries of someone's interpretation of data... which was presented in a summarized and interpreted form for them, which was summarized and interpreted for that person, who read it in the primary literature... which is merely a summary and interpretation of data. I knew we would get to the data eventually. Just like a game of telephone, the message can be altered drastically from its initial form. More importantly, the initial message (the primary literature) may have been a complete crock because of the poor quality of the science.
So let's go down the lifecycle of data starting at the end (the presentation in question):
1) There are over 100,000 synthetic chemicals in commerce today. Only a handful of these have even been tested for human health impacts and NONE of them have been tested for synergistic health impacts, that means when they interact with all the other chemicals we’re exposed to every day.22 - Annie Leonard: Story of Stuff
I know for a fact (but only because this is my field) that this statement is sheer distortion and parts of it are utterly false. But, let's look up her footnote #22 (that is the 22 at the end of that sentence). 22 leads us to...
2)“Of the more than 80,000 chemicals in commerce, only a small percentage of them have ever been screened for even one potential health effect, such as cancer, reproductive toxicity, developmental toxicity, or impacts on the immune system. Among the approximately 15,000 tested, few have been studied enough to correctly estimate potential risks from exposure. Even when testing is done, each chemical is tested individually rather than in the combinations that one is exposed to in the real world. In reality, no one is ever exposed to a single chemical, but to a chemical soup, the ingredients of which may interact to cause unpredictable health effects.” From Coming Clean Campaign’s Body Burden
information, retrieved 11/8/07 from http://www.chemicalbodyburden.
org/
Ok, so Annie got her information from the Coming Clean Campain's "Body Burden" information on chemicalbodyburden.org. I'd like to point something out here. Annie isn't getting her data from scientists at places like the National Academy of Sciences, or the National Toxicology Program, or even primary literature. Her information is coming from a group who's sole existence depends on lobbying for a particular cause. It doesn't seem like exactly an unbiased place to get one's information. Not exactly balanced information. Roughly the equivalent of watching Fox News to get a balanced viewpoint. But let's go to the site and see where they got the data to make that statement.
3) http://chemicalbodyburden.org/whatisbb.htm
Under the heading "What is Body Burden", about half way down you will find the statement that was used for Annie's proof. The reference for this statement? Non-existent. In fact, looking over this website gave me chills in general. There is a high-degree of spin and stretching of facts, applying facts incorrectly, and falsehoods. So how can we backtrack the data now? Well, if you go to the about us link on the page of this organization (http://chemicalbodyburden.org/aboutus.htm) you'll see that their scientific toxicology staff consists of 1 person, namely: Dr. Harmon, PhD, Toxics Campaign Scientist
Dr. Harmon is a member of greenpeace, and is helping out with this organization. I am sure that Dr. Harmon is a excellent scientist, but even the best scientist in the world would have trouble keeping up to date with all of the changing literature and all of the data needed to support the generalized statements made on the website. There is simply too much out there to be an expert of all.
So it looks like the reference trail has gone dry. But let's imagine where the data possible came from. We can put a big black box.
4) The Black Box
The black box here is the X number of individuals/interpretations between the statements on the chemical body burden site (#3) and primary literature (#5). So step 4 is potentially step 4, 4.1, 4.2, 4.21, 4.22 etc... before we get to step 5. Who knows how facts get changed here.
5) Finding Primary Literature
AKA Publications. If we do a NCBI pubmed search (http://www.ncbi.nlm.nih.gov/sites/entrez) on BFR, for example, we get 498 hits. Here is one example of the 498 hits: http://www.ncbi.nlm.nih.gov/pubmed/18315715?ordinalpos=2&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Enjoy reading the abstract. After that, read all of the pages of that publication. Now read the other 498 references. I'll wait.
After that read about dioxins. That'll produce 13,275 hits.
Actually, that is really too specific, do a search of toxic and you'll get 142,197 hits.
Why don't we start with reading 1 paper
6) Reading Primary Literature
For each of these publications you'll have to read the author's interpretation of the data, and look at the raw data yourself to determine if you agree with the author's assessment. In addition, the raw data in publications isn't really raw, but presented in a certain way by the author.
Based on my calculations, we have between 6-492 people standing between Annie's initial statement (remember Annie, she is in the story of stuff) and the data. That is a disturbing game of telephone.
Bottom line. You can find scientists who fall on either side of every single argument and fact in science. On top of this, many of the things Annie is saying is simply incorrect, as is her source. If you'd like to read something on mixtures, I recommend this: http://www.ncbi.nlm.nih.gov/pubmed/16081521?ordinalpos=2&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_RVDocSum
Therefore, having a debate on the validity of these facts is not a useful exercise. Suffice to say that there has been tremendous research on mixtures over the years, and it seems to be overall a phantom issue since only those with common MOA should even be in the realm of concern. On top of all of this is the subject of the next posting : The Dose.
Thursday, March 6, 2008
Annotated Script with Sources
The Story of Stuff website provides references and sources for their statements in the form of an annotated script for the video. The script can be found on their references page.
1) On the "100,000 chemicals and synergistic impacts" statement: I tracked the source of that statement (using the source noted in the annotated script to this page.) This appears to be a web site focused on pointing out what we don't know about the chemicals in circulation in the US. It's not as well sourced as I'd like especially on the specific statement about the number of synergistic impact studies that have been conducted. The lack of a secondary source doesn't necessarily make it wrong, but I'm definitely interested in seeing the studies you mentioned that suggest we do understand the synergistic impacts.
2) The sources on BFRs are on page 5 of the script. The sites they point to point to other studies. I read a bunch of them and my take away is that BRFs are leaching into the environment (wildlife) and people's bodies. The amounts are small (in people) but have increased about an order of magnitude in the last 30 years. Plenty of alternatives exist.
The thing I struggle with is the idea that flame retardants are going to save my life. If I'm still asleep in my bed while my bedroom is on fire I'm going to be much more concerned with smoke inhalation than getting burned by my bed.
3) A few of the BFR sources describe the breast milk studies. Unless the sources are just wrong the statement she's making is correct. They make a point to state that Breast milk is still the best choice and they even site sources to that point. (See page 5.)
4) Factory Workers: The point made in the video is that this is a global system. One of the challenges for US factory workers is they cost more and have more restrictions as a result of the safety standards required in the US. I think the safety standards are a great idea, but they are not a global standard occurrence. So while I agree that this does not appear to be a major issue in the States, countries like China count and if they are anything like the US during our industrial revolution then factory workers are getting the short end of the stick.
I'm out of time for this post so I'll follow up on your other points in a few days.
1) On the "100,000 chemicals and synergistic impacts" statement: I tracked the source of that statement (using the source noted in the annotated script to this page.) This appears to be a web site focused on pointing out what we don't know about the chemicals in circulation in the US. It's not as well sourced as I'd like especially on the specific statement about the number of synergistic impact studies that have been conducted. The lack of a secondary source doesn't necessarily make it wrong, but I'm definitely interested in seeing the studies you mentioned that suggest we do understand the synergistic impacts.
2) The sources on BFRs are on page 5 of the script. The sites they point to point to other studies. I read a bunch of them and my take away is that BRFs are leaching into the environment (wildlife) and people's bodies. The amounts are small (in people) but have increased about an order of magnitude in the last 30 years. Plenty of alternatives exist.
The thing I struggle with is the idea that flame retardants are going to save my life. If I'm still asleep in my bed while my bedroom is on fire I'm going to be much more concerned with smoke inhalation than getting burned by my bed.
3) A few of the BFR sources describe the breast milk studies. Unless the sources are just wrong the statement she's making is correct. They make a point to state that Breast milk is still the best choice and they even site sources to that point. (See page 5.)
4) Factory Workers: The point made in the video is that this is a global system. One of the challenges for US factory workers is they cost more and have more restrictions as a result of the safety standards required in the US. I think the safety standards are a great idea, but they are not a global standard occurrence. So while I agree that this does not appear to be a major issue in the States, countries like China count and if they are anything like the US during our industrial revolution then factory workers are getting the short end of the stick.
I'm out of time for this post so I'll follow up on your other points in a few days.
Monday, January 28, 2008
Reza Identifies Specific Issues
This is the final comment copied from the original blog post. Reza identifies specific issues he has with the Story of Stuff. It was originally posted Monday, January 21, 2008 10:15:00 PM
Sigh... where to begin. Ok John. You lucked out, Paul and I couldn't get together tonight because I was at a Jazz gig. So here we go:
The two issues are 1) lack of accuracy and 2) persuasiveness. I do agree with paul that these are coupled. Let's go through and talk about point 1 first.
Pretty much everything she states about toxic chemicals is innacurate. I'm going from memory here, but this is a list:
1) She said somethings like 100,000 chemicals with only a few tested and none tested for synergistic impacts. Holy crap where do I start with this one.
- a) Much like with pharma, you need to go through millions of dollars of testing to get anything approved for production. You first identify potential hazard (carcinogenicity, teratogenicity, etc...) at astronomic doses, and then do calculate what humans could potentially be exposed to in "worst case" scenarios. Then you make sure that there is at least 100X less (often more like 1,000 fold lower) of that chemical in the worst case scenario than the LOWEST dose that caused ABSOLUTELY no effects in animal models. This is a simplified version of Risk Assessment.
- b) Synergistic effects? People have done almost a decade of research on just that. Potential synergistic effects of mixtures. The results of the billions of dollars in research? Bupkis. nada. Scare tactics at best.
2) BFR are neutoxiCANTS. They also save lives because they stop your mattress from burning when your house is on fire. I don't think they use them in pillows anymore, regardless the doses are really low (see risk assessment above). Despite all of this, due to media pressure people are actively looking for alternatives to BRF.
3) Toxicants all are concentrated in breast milk? WHAT?!?!?! That is about as nonsensical a statement as I have ever heard. While there are cases of very few, VERY FEW, lipophilic compounds that pool in certain tissues including breast milk; HOWEVER, people don't just ignore this fact. This is all part of the risk assessment. Babies are often the sensitive populations, so the risk assessments are done on babies.
4) Factory workers. I can't speak for companies that are ONLY in places like China, but US companies and international companies also do risk assessment for factory workers. Again, phantom issues that she is blowing WAY out of proportion and completely distorting the truth.
5) As an example that you can relate to: She says that computers go obsolete every 2 years. But she looked into this, opened her desk top and saw that they change only a little piece (we assume she means the CPU) with everything else staying the same. She also claims conspiracy because they make the new CPUs incompatible with the old ones. Do you feel this is an accurate representation of the truth? I can tell you that this is 100X more accurate than everything she said about chemicals.
Ok, so that is some of the accuracy stuff, there is more but I am running out of time. what about persuasiveness?
I would argue that she shoots herself in the foot on this because she pulls in political stuff that will likely turn off all moderates and republicans. It turned me off. here is a list:
1) The whole showing government as a tank. Why did she even go there? Isn't this a story about how we should be less materialistic and use less stuff? She is causing much of her potential audience to tune out because of her extremist opinions. And don't give me the "my friends want me to make it a tank but I think government should be for the people, by the people," etc...
2) Business should be bigger than the government. Why would we want an enormous government that wastes so much money/energy/brain power, etc... Again, this is politics that causes her to get off message.
3) So the media is to blame for everything else? The TV tells me to buy something so I go do it? It has nothing to do with my vanity? I don't need the TV to convince me that a 29" flatscreen monitor for my computer is cool. It is just cool. She continues to put the blame on everything except the individual. Again, she gets off message and will lose people.
There are a lot of other examples, but in the end I couldn't get over the HUGE distortion of truth and scare tactics that were used in this film. I think that it will likely convince people who were already convinced, further polarize the people who think tree huggers and their anti-business ways suck, and just irritate scientists due to lack of accuracy.
Ok, I'm tired. Is that what you wanted John?
Paul, now that John had his fill I would be happy to discuss offline. I think it would make for an interesting conversation. We can even invite poopy-pants to the call. Peace brother.
R
Why Not Just Call?
Paul and Reza discuss why Paul wrote a novel instead of picking up the phone. This back and forth is taken from the original blog post comments.
Monday, January 21, 2008 8:04:00 AM: Reza said...
What!?!?!
I can't believe you have time to write that response but you don't have time to discuss it over the phone! Boo.
We are talking about two different things here:
1) accuracy (or lack thereof) and
2) effective communication and persuasiveness.
Let's talk offline. Setup a time.
R
Monday, January 21, 2008 10:51:00 AM: Paul said...
It was the night of the second greatest game in Giant's History!!! I realize that I don't care but I figured you'd be watching!!!
So yes, yes I'll call...
My starting position for our future conversation is that they are not completely separate topics. Accuracy, simplicity and effective presentation are inexorably linked.
Talk to you soon my brother!
Some Friends Voice a Preference
The next day a bunch of our friends got on our case about taking the debate offline. We agreed to keep posting, thus this blog was born.
Frankly I (Paul) was glad to keep it on-line. I've debated Reza live, and it's not a fair match. I've lost a debate with him where he agreed with me but took the other side just for fun.
Paul Responds with his Perspective
This was Paul's response to Reza which was posted in the original blog comments Sunday, January 20, 2008 10:10:00 PM.
I'm not sure if the world is getting more complex, or I'm just paying more attention. Either way I agree that it appears more complex. For many people complexity acts as non-starter, a barrier-to-entry for critical thought, involvement and activism.
Clearly I'm not talking about you, but rather people that take a more dispassionate approach to our environment and their impact on it. I'd argue that this dispassionate group is a large majority of our population. Moreover, If we are ever going to get to a sustainable way of life, I'd argue that this population needs a catalyst to get them to start thinking about how their behavior impacts the rest of the world.
One of the things that impressed me most about The Story of Stuff was how they boiled down complex issues into relatively simple terms. I frequently see people present highly detailed and technically flawless information and have their message and goal go down in flames as a result of presenting at the wrong level of abstraction for their audience.
A challenge I often have is to boil down a complicated topic into something my audience will 1) listen to 2) relate to and 3) be willing to act on. Keeping in mind that the average (attentive) audience will only remember ~20% of anything presented, it is critical that I stay high enough up in scope that I don't bore or lose them. That often means sacrificing some of the detail and possible technical purity of the topic at hand.
With that in mind, when I consider The Story of Stuff as a catalyst for activating the public I'm impressed with it on several counts:
- It manages to keep people's attention for ~20 minutes while sharing information unrelated to American Idol.
- It boils down important and complicated topics in ways that are easily consumable by the majority of the population.
- It does not provide a detailed solution, call out a specific boycott-able culprit or dictate a specific action , but rather encourages people to learn more and provides some resources to get them started. (This one is absolutely critical.)
- It encourages discussion, debate and likely learning, even among people that know the specifics of some or all of the topics presented.
Its surprisingly difficult to craft a simple message about a complex topic with absolute regard to the purity of the subject matter. Being able to do so is what separates people like Einstein and Richard Feynman from the pack.
So when I look at The Story of Stuff, I judge it by how well it achieves its goal of engaging, educating and inspiring people. And rather than exploiting necessary simplifications and calling for specific actions, The Story of Stuff compensates for simplifications by encouraging its audience to learn more.
It takes a profound courage put an idea out to the public, dare people to ignore that idea, and then tell people they need to go learn if they want an answer to the challenge that idea raises. And even if the creators of this video are called out on the specifics and checkmated in a public battle of details and specifics they will have achieved their goal. They will have made people think.
Reza's First Response
This is a copy of Reza's first two comments. They were originally posted Sunday, January 20, 2008 at 8:54:00 AM.
What a strange mix of facts, spinning, truth, scare tactics, good points, superficial views of the real issues, honesty, and sheer lies.
As the world and these issues become increasingly complex, it becomes more and more difficult to know what is real.
As an example of a topic you know something about, listen to her argument about "disposable" computers and the little tiny piece that changes. That one minor example of all of the distortions and spin on the truth. The part that bothers me is these things are intertwined with a really good message. It makes you wonder about the validity of the things we don't know about.
Subscribe to:
Posts (Atom)