I’ve succumbed to some clickbait titling here, I’m sorry, but we really need to have a discussion about this.
There’s a fallacy running wild in food, and it’s company owners, consumers, some government officials, and even some QA personnel that continue to believe that the 25 gram sample they tested or sent out to their “independent” laboratory coming back negative for [insert pathogen here] somehow means that their product is as sterile as an autoclave.
Outside of the news, I’ve had state departments of health ask me for end-item CoA’s in response to consumer concerns, and I’ve had a dozen start-ups send me a 4 month old CoA when I ask them about pathogen controls.
Yeah, congratulations on having 25 grams of product be free from salmonella, now do you actually clean your equipment?
You can not test your way into food safety, that’s why Pillsbury and NASA said a systemic approach to positive controls was needed to ensure diarrhea in space wouldn’t be on the list of obstacles astronauts faced. However, because test results often reveal what were usually long-term, underlying, systemic issues below the threshold of detection, many people have adopted the notion that if “failing” microbiological tests are bad, then “passing” ones must be good.
Let’s cover some reasons/I’m going to rant about this now:
1. Spontaneous generation was disproven in the late 1800’s, why are you so proud that your results support that?
I’d have about $3.50 if I had a quarter for every time someone did “shelf life” testing on their product and were extremely proud that their retained samples had no pathogens at day 10, and still had no pathogens at day 60!
It’s a miracle.
If there is no salmonella in your steak sample at day zero, where is it supposed to come from? This is a common misunderstanding that shelf life testing is safety based, when far more often shelf life is determined by spoilage, not pathogen growth. “But Austin,” you say, “what about listeria growth in the fridge?” You’re right! That is a safety-oriented shelf life consideration, and worth modeling by testing the same food over time…but you still need to have some listeria to start with. Pathogens do not spontaneously appear in foods after 3-7 days, but they may proliferate if they’ve contaminated the food.
Your product’s shelf life/resistance to pathogen growth/microbial reduction step is determined by the methodology and results of your testing, not by a CoA that says you still found no pathogens after 14 weeks. Good for you, you still haven’t made a food with C. bot controls. I know, this stuff is more complicated than paying for a lab test would lead you to believe.
2. Food contamination isn’t homogeneous, and your 25g sample is statistically meaningless.
Think about the biggest recalls/outbreaks you can remember. As per usual, I’m going to use my flagship ice cream outbreak. Here you have a situation where systemic, underlying contamination caused 10 hospitalizations and 3 deaths over a period of 5 years. Clearly there was underlying contamination sprinkled throughout the product, but never enough to cause an outbreak cluster that fired off public health officials. The vast majority of food contamination isn’t gross contamination of every single can/bottle/package/meal made or served, but varying concentrations that overlap zero or “none detected”. As always, food contamination is a public health issue tackled on a per-capita scale, rather than a personal safety risk.
Think about it, that ice cream company was one of the largest in the country, in 2017 alone they had $426,000,000 dollars in sales. Even if you assume that their average unit of ice cream sold for $100, that means they would kick out 4,260,000 units of ice cream in a single year, or 21,300,000 in 5 years. If the 10 illnesses identified represented only 1% of people who were actually affected, that would mean that only 1 in 213,000 products was contaminated to the point of causing illness. .00046% of products.
What were the odds that one container of ice cream sampled would come up positive?
The fact of the matter is, contamination comes from cross contact, dirty equipment seeding products with biofilms, condensation periodically dripping onto a conveyor belt, or a million other small events that aren’t exactly a blender of equal distribution, and the likelihood of finding that contamination with a single sample becomes harder and harder the lower that level of contamination.
Imagine with me a lot of 10,000 bags of turmeric.
That’s an awful lot, but not an unreasonable number of units for a small food company to produce in a single lot. Now during production of this turmeric, the cooling pan in my air conditioning/dehumidifier overflowed and dripped some salmonella-containing liquid onto the last 5% of products produced. Contaminating them at a level that is likely to cause illness.
Keep in mind, this means that if I ship this product, 500 people could become sick. Food safety is one of those arenas where absolute numbers matter, just because I produced 9,500 perfectly safe packages of spice doesn’t change the fact that today’s production could hospitalize an entire elementary school.
But that’s okay, because every time we produce a batch of spices, I select one at random to send out for salmonella testing, if it comes back negative, that means the lot is safe, right?
At 5% contamination, a single randomized sample will have only a 5% chance of actually being a contaminated pouch. In order to have 95% confidence that you’ll detect the salmonella at 5% contamination, you’ll have to pull many more samples. I highly recommend quality professionals becoming familiar with the statistics covered by ICMSF or WHO, but for those looking for a quick and dirty sampling scheme, we can use the rule of 3’s.
In this case, the number of samples needed for 95% confidence that 1 or more will actually contain a contaminated sample is given by n=3/p where n is the number of samples needed, and p is the prevalence of the defect you think you might find.
In this case, I’d need 60 randomized samples tested before I was confident…that the contamination level is probably less than 5%. See what I did there? At no point in this sampling will I be able to say that the lot has no Salmonella in it, I can only statistically support a certain level of presumed contamination, which gets lower the more products I sample.
To complete my ice cream example above, if there was Listeria in .00046% of products produced, the company would have had to pull 652,174 samples to have 95% confidence that there was no listeria contamination at that level.
So, when a public health agency calls your company to say there are reports of illnesses and that epidemiological evidence is suggesting your product is implicated in an outbreak, or that they pulled a sample of your product and it contained a pathogen, your test results are helpful but ultimately irrelevant. It takes only two illnesses to make an outbreak, how many samples did you pull exactly?
3. Your bad practices aren’t causing a problem today, but they will.
I maintain that one of the reasons FDA threw the CODEX HACCP rulebook out the window when they created the preventive controls rule was because companies used CCP’s to define “no sale” criteria, and didn’t treat pre-requisite programs as what they were, which was literally pre-requisite to the hazard analysis. If your equipment wasn’t clean or maintained, your hazard analysis was supposed to be considered null and void and the process not considered safe. What you ended up instead was things like pet food manufacturers putting products in a retort, so they stop handling their raw ingredients in a temperature-controlled environment since the cook CCP will be met. Maybe someone forgot to sanitize a conveyor belt, but because the CCP’s on the metal detector were met the company releases product produced on dirty equipment.
Rather than enforce the sanitary GMP’s, FDA instead said “fine, sanitation is now a CCP”.
After understanding how statistically meaningless individual samples are, I now often caution other companies and QA folks on the dangers of “curiosity swabbing”. This is where a well meaning employee sees a problem, maybe a gap in the floor, or condensate on a wall, and decides “I should swab that to see if it’s safe!”
There are only two outcomes of this decision.
- You find a pathogen and suddenly you need to evaluate an immediate corrective action which may include substantial construction, product holds, or other implications. Good for public health, harder for your business.
- You don’t find anything, you still know the situation is wrong and has the potential to be a problem, but now you have a test result that says “on wednesday the 29th at 2pm this 12 sq. inch area had no listeria in it”. It’s meaningless just as above, and now no one wants to fix it because you demonstrated that it wasn’t a hazard with that snapshot in time.
Here’s what I tell companies to do: Hey look, a crack on the floor that could harbor a pathogen. Let’s fix the damn crack.
Come up with a realistic timeline and fix the potential hazard, the testing does not add new information, you’ve identified a weakness, put a plan in place that makes it possible to fix. Focus on prevention, not swabbing to satisfy your curiosity about the status that hour.
FDA warning letters are riddled with companies who test every lot for “safety” while mouse droppings rain from the ceiling. Save that testing money and put some new door sweeps in instead.
4. Only negative results are legit, all positive results must have been sampling error
Have I said this stuff is complicated? It never ceases to amaze that when results come back as <10/none detected/absent the lab is in great shape doing great work. But the minute a positive result shows up someone must have contaminated the sample.
I mean, that’s the only possible explanation? Or if it was the state they must be part of a vast conspiracy to bring down your product.
We don’t have a repeat-until negative 3 more times procedure for all of the “good” results that come in day-to-day, nor confirmation testing to resuscitate damaged cells from drying processes etc when the numbers are non detected. Fascinating how food companies jump though hoops to get back to those good results by using another lab, another sample, another method.
Forget micro testing, any thermometer, scale, pH meter, or other measuring device, when providing readings that cause problems, must be malfunctioning, otherwise it would read in spec!
This isn’t to say that we should not do end-item/product testing
Listen, even if it isn’t statistically significant, if you’re testing every lot, and every lot is contaminated at some percentage, after a year of tests you’re probably going to see it eventually, the trick is responding appropriately, and not just by saying “throw that day away”. Positive results may very well reflect a deeper, sporadic issue that should alter your sampling plan. Finding a pathogen should trigger increased sampling to (hopefully) capture low prevalence that you may just not have seen until now. If you’ve been on skip-lot testing, go back to lot-by-lot, or even institute a 3 class sampling plan of an indicator organism.
End-item sampling is also incredibly important to public health, as it captures actual dangerous products in the marketplace that need to be addressed, and enforcement agents will sample enough of them to actually catch a lot of that low level contamination (but won’t find root cause that way). You’ll also notice that FDA/FSIS/CDC doesn’t announce every time they get a negative result with “hey everyone, so-and-so’s brand tested clean in our 25g sample, they’re safe!”
Here’s my point, companies should include microbiological testing in their food safety systems, but focus less on end-item testing and more on environmental search-and-destroy programs, raw material sampling, and sanitation validation. All of these programs are more likely to reveal systemic issues that will cause random, sporadic contamination with actionable results, rather than sampling each day and hoping nothing “bad” ever comes up and trying to pretend that “<10cfu/g” is some magical word that means your food is safe. A bag of peanuts in a million bags of chips is only 1ppm peanuts, but you can’t exactly serve that bag to someone with an allergy without consequence.
If you expect a result to be negative, why are you testing it? Use your dollars, time, and sense to find and fix food safety weaknesses, not provide empty assurance that all is well.
Additional sources linked within text.
World Health Organization. “Statistical aspects of microbiological criteria related to foods: a risk manager’s guide.” (2016).
FAO. “Hazard analysis and critical control point (HACCP) system and guidelines for its application”. (1997)
ICMSF. “Microorganisms in foods 7: microbiological testing in food safety management”. (2002)
T. Gojanovic. “Zero defect sampling”. (2007). http://asq.org/quality-progress/2007/11/basic-quality/zero-defect-sampling.html
FDA. “BAM appendix 2: Most probable number from serial dilutions”. (2010). https://www.fda.gov/Food/FoodScienceResearch/LaboratoryMethods/ucm109656.htm
[…] a larger scale certainly sounds like a business problem, not mine. Despite the additional nuance (and statistics) of that example, the consumer perspective isn’t wrong. It’s also not wrong to know […]