Safe-Cart: Real-Time Food Safety Intelligence

Safe-Cart App

Problem & Motivation

Information on food recalls and ingredients is public, but it is obfuscated by FDA recall conventions and long or difficult to read ingredient lists. Issues around food safety impact everyone. In the past year alone, the FDA issued around 1,500 food recalls, and a recent study estimated that out-of-pocket expenses for allergy related doctor visits cost a person around $1600 per year.

To address this issue, we have developed an app to improve FDA recall awareness and reduce exposure to unwanted ingredients, before unsafe food reaches the kitchen. Our app includes two key features. First, users can scan a barcode in-store to receive information about unwanted ingredients that do not fit their dietary preferences. Second, users can scan their grocery store receipts to build up a record of their commonly purchased items. The user will then receive an alert if one of their commonly purchased items is recalled, to raise awareness before they even enter the grocery store. We are not aware of any other tool that proactively warns users about recalls in this way.

Data Source & Data Science Approach

Our app is built on two modules: the recall intelligence module and the ingredient risk intelligence module.

Recall intelligence: The recall intelligence module allows users to scan receipts, stores their commonly purchased items, and proactively warns users if one of these items was recalled. The FDA issued about 1500 food recall alerts last year and their database of recalls is updated once a week. Most FDA recall alerts include lots of extraneous information beyond just the product name and the reason for recall. We cleaned this unwieldy data through a combination of deterministic rules and LLM prompts, to produce a better input into our alert system. And to process receipt images, our app uses a dual OCR process including AWS textract to extract information from the receipt. We save grocery store and product information, while ignoring extraneous text that is also included on a receipt.

In order to match grocery store items in receipt format with grocery store items in FDA recall format, we developed an ensemble method of fuzzy matching. We first identified the top four measures for calculating the similarity between the entire receipt text and the entire FDA recall text. These four measures were TF-IDF, rapid fuzz partial, rapid fuzz token set, and cross encoder. If one of those four similarity measures surpasses a tested threshold and the similarity between individual words in the receipt and the recall also passes a tested threshold, then and only then do we alert the user of an FDA recall match.

Ingredient Risk Intelligence: The ingredient risk intelligence module is the core decision-making system behind Scan to Safety. It analyzes a scanned product using two inputs: the user’s personal profile (food sensitivities, diet preferences, and location for matching to relevant food recalls) and the product’s ingredient list from Open Food Facts. Open Food Facts is a database of foods from across the globe. It includes about 800k items from the U.S. of which slightly more than half include ingredient information.

The system first cleans and normalizes messy ingredient text (e.g., nested parentheses, percentages, inconsistent formatting) into structured tokens. It then uses substring matching and curated exclusion rules to map ingredients to potential food sensitivities or diet preferences.

A two-layer decision framework drives the final output and user recommendation for purchasing the product:

• Layer 1 (Hard Stops): Immediately flags “Don’t Buy” if there is a recall or confirmed ingredient/diet violation
• Layer 2 (Soft Signals): If safe, evaluates weaker signals (e.g., additives) or missing ingredient list to return a “Caution” verdict and suggest for users to check the product label

The engine produces a deterministic verdict — OK, Caution, or Don’t Buy — ensuring reliability. An LLM is used only to translate recall information into clear explanations, never to make decisions.

Project architecture: This is our high-level project architecture, when a user takes a photo of a receipt or scans a barcode.

Evaluation

Both the recall intelligence module and ingredient risk intelligence module adhere to the same core evaluation principle: preserve recall over precision. Faced with this tradeoff, we would rather issue the user a spurious warning than neglect to provide them with information about potentially harmful food.

Recall intelligence evaluation: As noted above, we assessed and iterated through many individual text similarity measures as well as combinations of those measures in order to produce a single ensemble similarity measure. Notably, we found that adding a layer to our method in which we also calculate the similarity on a word by word basis helped us to increase precision without sacrificing recall. This ensemble measure was tested on actual FDA recall data and synthetic receipt data, and resulted in a recall of 0.8, precision of 0.8, and accuracy of 0.86. In the chart below, the dots in light green represent single similarity measures. The dark green dot in the upper right hand corner represents our ensemble method. 

Ingredient risk intelligence evaluation: The engine was evaluated on 40 hand-labeled test cases across allergens and diet scenarios, with iterative improvements:

• Version 1 (Exact Matching): Missed real-world variations → lower recall (81.5%)
• Version 2 (Added Substring Matching Layer): Achieved 100% recall but introduced false positives (e.g., “eggplant” → egg)
• Final Version (with Exclusion Rules): Balanced performance with 96.4% precision and 0.982 F1

As noted, a key design choice was prioritizing recall over precision — missing a harmful ingredient is riskier than issuing an occasional false warning.

Key Learnings & Impact

There were many technical challenges to overcome in creating this app. The first big challenge was sufficiently cleaning the FDA recall data. Most FDA recall alerts include lots of extraneous information beyond just the product name and the reason for recall. We used a combination of deterministic rules and LLM prompts to turn the prose-like FDA recall alerts into a clear and concise database of product names, recall locations, and product UPCs. The second challenge was establishing a process to match receipt text with FDA recall text. We used a combination of similarity measures at the phrase and word level to improve model recall, and we are not aware of an existing tool with this functionality. Lastly, the third challenge was balancing recall and precision in our user alerts. As noted above, we wanted to preserve recall over precision - it is much worse to neglect to warn a user about a harmful product than to send them a false alarm. However, we included some rule-based exclusions to improve precision so that we would not overwhelm users with alerts.

Overall, we hope that our tool will improve FDA recall awareness and reduce exposure to unwanted ingredients, before unsafe food reaches our users’ kitchens. With this product in hand, users can receive alerts about recalls before they enter the grocery store, and they can perform ingredient checks before they reach the checkout counter. Public food safety data becomes much more valuable when it is personalized, explainable, and available at the exact moment a shopping decision is made. Our goal is to help people make safer food choices with less effort and more confidence.

Acknowledgements

We would like to thank Joyce Shen and Korin Reid for their help in creating this app, as well as those in our class who filled our our initial survey on potential user needs.