Commercial home tests that analyze stool samples promise insights into a person’s gut microbiome. But how reliable are those results? A study published in Communications Biology delivers a sobering verdict: the tested microbiome analyses do not provide dependable information about the actual composition of the gut flora.

“The study is very elegantly designed and impressively demonstrates how unreliable the tested commercial microbiome tests are,” said Gregor Gorkiewicz, MD, professor of medical microbiome research at the Medical University of Graz, Graz, Austria, in comments to the Science Media Center Germany. “However, this issue has been a much-discussed topic in the medical-scientific community for some time now.”

Interest Meets Uncertainty

The study authors’ starting point: numerous publications link the intestinal microbiome to a range of diseases, from metabolic disorders to cancer and even neuropsychiatric conditions. Despite intensive research, there are currently neither approved routine clinical tests nor broadly established diagnostic standards for microbiome analysis. At the same time, companies are developing home tests for laypeople (direct-to-consumer [DTC]).

Those kits operate in a regulatory gray area between medical devices and lifestyle products. Although consumers often receive health-related assessments and individualized recommendations, the tests are not subject to the same validation and quality-assurance requirements as traditional in vitro diagnostics. Against this backdrop, researchers at the US National Institute of Standards and Technology (NIST) evaluated the analytical performance of seven DTC tests.

Reference-Based Analytical Evaluation

To exclude biological variability from individual donors, NIST researchers used a homogenized, well-characterized human stool reference material. They ordered three test kits from each provider and filled them with identical sample material. The companies were not informed about the evaluation so that the conditions would be as realistic as possible.

In addition, the NIST team analyzed samples of the same material in its own laboratory using 16S ribosomal RNA (rRNA) amplicon sequencing and whole-genome shotgun sequencing. The team also examined seven additional donor samples to compare the providers’ methodological variance with the biological variance between actual individuals.

Molecular Techniques

16S rRNA gene amplicon sequencing. Targeted sequencing of a marker gene is used to determine the composition of bacterial communities; it primarily provides information on which species are present and their relative abundances.

Whole-genome shotgun sequencing. A method used to analyze all the DNA in an organism or a sample; the DNA is randomly broken into many small fragments, sequenced, and then reassembled using bioinformatic tools.

Read depth. The number of times a DNA segment is sequenced; higher read depth increases accuracy and the ability to detect rare organisms.

Reporting cutoffs. Thresholds set to determine when an organism is listed in the report.

Sample Handling Differences

Sampling recommendations already differed substantially. Some providers advised collecting the sample from toilet paper, and others directly from the stool; some recommended using a stabilization buffer, and others did not. In all cases, samples were shipped at room temperature, ie, without refrigeration.

The molecular biology methods varied considerably. Three companies used 16S rRNA amplicon sequencing targeting different regions, and four used whole-genome shotgun sequencing. Sequencing depth ranged from under 20,000 reads to more than 10 million reads. Notably, greater sequencing depth did not consistently correlate with a higher number of identified species. This suggests that not only the amount of data but also the entire methodological pipeline shapes the results.

“The providers differed substantially in sample collection, buffer/shipping, sequencing approach (rRNA gene amplicon sequencing vs whole metagenome shotgun sequencing), read depth, and reporting cutoffs,” said André Gessner, MD, PhD, head of the Institute for Microbiology and Hygiene, University of Regensburg, Regensburg, Germany, in comments to the Science Media Center Germany. These methodological differences produced markedly different taxonomic profiles. “In some cases, variability between providers is as large as — or larger than — the biological variability between different donors,” Gessner said. “Reproducibility within individual providers was also inconsistent.”

Differing Reference Populations

Beyond taxonomic profiles, many providers offer health-related assessments and individualized recommendations. The reference populations underlying those interpretations vary — internal company databases, the Human Microbiome Project, or American Gut— as does the definition of a “healthy” microbiome.

In one instance, identical sample material was classified as “healthy” by two providers but as “unhealthy” by a third. Functional assessments and dietary recommendations differed accordingly. There were also large discrepancies in detecting potentially pathogenic organisms. For Clostridioides difficile, three providers reported detection and four did not.

“The paper addresses the core problem in the DTC microbiome market: consumers receive seemingly precise reports, even though the analytical basis is often not robustly comparable,” Gessner said. “While the authors used homogeneous reference material and replicates, the study does not determine which provider is ‘right’ — only that they diverge greatly.” The results are also not automatically generalizable to all available services, he added.

No Professional Endorsement

At present, overall, the systematic analysis shows that the methodological variability of commercial microbiome tests for laypeople reaches a magnitude comparable to biological differences between individuals. Without standardized quality criteria and transparent validation strategies, the clinical interpretability of such tests remains limited.

Gessner summed up the most important takeaways for physicians: “Several medical societies and international expert panels currently advise against the widespread use of microbiome tests in routine diagnostics. Besides the lack of standardization, again demonstrated in the current study, the reason is that concrete treatment recommendations based on microbiome findings are often not yet possible.”

Accordingly, a recent international consensus paper emphasizes that the evidence for widespread routine use of microbiome tests is currently insufficient. In Germany, the German Society for Gastroenterology, Digestive and Metabolic Diseases already pointed out the lack of comparability and limited clinical significance of commercial microbiome tests in 2018,” Gessner emphasized.

This story has been translated from the Medscape German edition.