Evaluating Science

As you assess what appears to be science in the news and the popular literature, it might be helpful for me to share with you some basic lessons that I have learned during my career in science.  While you may be well aware of some of the points I am about to make, others may not be quite so obvious.  Much of this was recently reinforced in my own mind in an Encore class, “What Science Is and What Science Isn’t”.  Dr. Hubitz, a Physics professor with many years experience in teaching and evaluating science, felt that the general public was often misled by what they believed to be valid science.  It was a great course from my point of view because I could relate a broad range of misinterpretations from flying saucers, chiropractic treatment and many other subjects to my own experience in interpreting results of various clinical trials in my work at Bristol-Myers Squibb.

I don’t intend to judge any particular concept here but I will simply cite recent interest in the reported benefits of particular diets as a generalized example to point out the series of intellectual layers that need to be addressed.   It’s sort of like an alien from another world arriving on earth never having seen any type of plant, trying to understand the role of a coconut in a palm tree.  He would need to start at the roots and work up through progressive layers of understanding.

To claim that reported results have come about through scientific investigation implies the use of the scientific method in achieving those results. The scientific method demands a well-defined sequence of steps beginning with a clearly defined hypothesis, e.g., such and such a food is harmful to one’s health.  In such a case the hypothesis should be refined as much as possible.  For example, what is the specific harmful effect believed to be caused by the food?  What population (demographics) is susceptible to the suggested harmful effect?  What dose of the food is believed to be harmful? etc.,etc.

Once a clear hypothesis is formulated, a well-controlled study is designed to test the hypothesis.  A homogeneous target population is identified, test conditions are established with sufficient numbers of subjects to yield statistically meaningful results and all potential variables are held constant, except the variable being tested and this is varied in a systematic way in order to study important parameters such as dose-response relationships.  Reproducible findings are then needed from other investigators to confirm the hypothesis.   It’s unlikely that there has ever been a human dietary study that meets these strict standards.  It would be nearly impossible to control all variables in a human’s life other than one specific dietary factor.  For this reason nearly all data regarding diets and health come from retrospective surveys.  Additionally, results of this kind are only correlations, which do not establish a cause and effect relationship.  That’s not to say that such results are of no value.  They may be quite informative, but we need to recognize that retrospective data, in general, are far less reliable than results obtained from well-controlled prospective studies.  It’s always a good idea to get a good handle on the full range of opinions.  Where is there nearly unanimous concurrence and where are there credible divergent opinions?

The perspective I would bring to this, based on my experience with numerous clinical trials of various new pharmaceutical therapies, is that even in the case of very carefully conducted, well designed, prospective trials, results can and often do vary from one study to the next.  In fact the typical NDA presented to the FDA for approval of a specific indication might contain five large trials of several hundreds or thousands of patients each, three of which are positive for the new drug, one negative and one neutral.  You virtually never get a totally clean picture, even with tightly controlled studies.  Human biology is incredibly complex and variable.  Responses to nearly all interventions reflect this.

Nearly all health care workers are familiar with the very powerful effect the human mind can have on health and disease, from the medicine men of primitive societies, to faith healers and to most doctors and nurses.  A positive mental attitude has often been observed to play a substantial role in recovering from an illness.  Conversely, there are many reports of voodoo and other “evil” spells associated with sickness and even death.  There is a whole field of medicine; “psychosomatics” that deals with mentally induced illness. While I must admit that little if any of this is based on trials conducted by the “scientific method”, most of us are personally familiar with the strong effect mental attitude has on our lives.  Thus, if you have a strong belief that a particular diet or behavior is beneficial to you, chances are it will be, at least via the psychosomatic route and you should usually follow your instincts.

Finally, there was a statement in a recent article on diet and health that I strongly agree with.  The best judge of what is good for you is your own body.  Listen to it and follow what it is telling you.  If you feel good and are happy with your choices and your life style, stay with it.  If you feel improvement is needed, look for those things that benefit you as an individual.  There is no such thing as a “one size fits all” lifestyle.  Keep in mind the powerful effect of a positive mental attitude.