Studs and Duds – The Rating System
Each sport supplement ingredient reviewed in this app has been evaluated for efficacy in two parts. The first examination is at the clinical level. This concerns any scientific data or actual human trials that can help us determine the value of said ingredient as a performance-enhancing supplement. The second evaluation is of empirical evidence. This is an informal assessment of common observations with the use of the ingredient as a standalone item. In both cases, attempts have been made to obtain the most reliable and relevant data available. A traditional rating system of 1 to 5 represents the strength of each evaluation, and the combined score is used for the overall rating.
CLINICAL EVIDENCE
The clinical study of a sport supplement ingredient can take on many different forms. Sometimes it is as simple as a test tube experiment, where the investigators are looking for a particular hormone or chemical to be produced when cells are incubated with the supplement. At other times, they can be as complicated as a full university investigation, where the supplement or a placebo is given to human volunteers and a detailed accounting of all results is noted. Generally, the more detailed and extensive a study is, the more valuable are its conclusions. A positive response in a double blind placebo controlled study on individuals with consistent exercise or athletic experience is considered the highest level of clinical validation or “proof” available for a sport supplement.
CLINICAL STUDIES SCORECARD
Understanding Clinical Studies
To understand why validation in a placebo-controlled study is valued so highly, we need to examine just how strict these studies are. To begin with, a reputable study is designed from top to bottom to eliminate all influence of bias. This goes as far as “group randomization,” which assures that neither the participants nor the investigators know who is actually taking the supplement and who is taking the placebo until the study is completed (“double blind” study). This eliminates any chance an investigator might subconsciously favor one group over the other. All relevant protocols in the study are also disclosed, so there should be no guessing as to how the study was conducted, and how measurements were taken. Credible studies are also peer reviewed, which means that other researchers in the same field have had the opportunity to critically evaluate its techniques before publication.
Strength in Numbers
Just as important as removing bias in a study, is removing the influence of random chance. After all, if you subject two groups of people to exercise, one will perform better than the other given the influence of individual genetic, dietary, and motivational factors. But you cannot draw a conclusion based solely on which groups did better — you need to be sure one group did better because of the supplement. By using large group sizes and measuring each of the individual responses, a reputable study tries to prove an association between the intervention (the supplement) and the observed effect (the result) mathematically. Generally, the larger and more homogenous (equal in makeup) the groups are, the less influence individual factors will have on the outcome, and the more certain one can be of any association.
Science takes its widest steps forward on a foundation of certainty. While strong associations can tell us a great deal, a “statistically significant” mathematical association is required before any credible study can report something as a positive result. This is usually defined as a 95% probability or greater that the result would be repeated under the same conditions. In other words, based on all the individual and group total responses, we must be 95% sure or better that the supplement was responsible for the improvements. This is a very high threshold. So high, in fact, that for years studies giving anabolic steroids to athletes routinely failed to prove they actually increased muscle mass or performance. Only when the studies became larger, and the protocols better tailored to noticing these improvements, was the medical community able to reconcile the observations of athletes.
Because of these strict standards, we hold clinical validation in extremely high regard. Very few sport supplements have actually achieved this level of efficacy proof, and the ones that have are usually regarded as products of extremely high reliability. Still, it is important that we not focus solely on statistically significant results. Very often the population sizes or protocols of a study are not strong enough to achieve significant figures. We are often left with strong positive associations that are just not quite “provable.” An association that is seen with an 80-95% confidence level is usually regarded as a “statistically strong trend” or a “non-significant improvement.” Such strong associations help researchers focus their studies in the future. In a field easily influenced by individual factors, we too can learn a great deal from statistically strong trends. As you read this app, please pay close attention to such details under the Clinical Studies section for each supplement.
Body Composition
On this same note, if you are looking for studies proving that a supplement will help you make significant improvements in total muscle mass, know these studies are few and far between. Changes in body composition can be very difficult to control in a clinical setting, as they are easily influenced by many individual and methodological factors. We all know that some people gain muscle mass more easily than others, regardless of supplementation. This makes proving effect across small group populations difficult. Generally, changes in strength and performance numbers are easier to demonstrate. An absence of statistically significant findings on body composition in a study, therefore, does not necessarily represent a lack of effect. With the understanding that underlying changes in body composition are likely to go unnoticed, we typically look for performance improvements during resistance training, and extrapolate how they will relate to body mass over time.
EMPIRICAL EVIDENCE
Clinical studies can be time-consuming to design and implement. They are also very expensive. It can take years before a detailed study is conducted on a popular sport nutrition ingredient, especially one that relates to a relevant group of exercise-experienced individuals. If and when they are conducted, it can often take time, and repeat investigations, to draw out statistically significant findings. The sports nutrition industry, on the other hand, is very fast paced and innovation driven. New supplements come out all the time. Because of this, we cannot rely solely on clinical evidence when trying to determine the value of a sport supplement. We must also examine the realworld responses people have been noticing from the ingredient.
Each supplement reviewed in this app also includes a summary of empirical evidence pertaining to its use. By its nature, empirical evidence, or evidence based solely on observation, is less reliable that substantiated clinical evidence. It is subject to interpretation, and always lends itself to some bias in the part of the interpreter. Also, when people have expectations about the effects of a product, they sometimes notice positive changes that are not actually there (“placebo effect”). Still, in a fast-paced industry such as this, empirical evidence is a fundamental asset provided you are able to rely on the source. In this regard, I have done my best to be objective and critical. Whenever possible, I drew on my own observations, as well as the observations of other researchers in the field whose opinions I value.
