Master of Information and Data Science (MIDS) alums Abhishek Shetty, Ana Melody Masis, Vijaykumar Krithivasan, Landon (Austin) Yurica, and Mehmet Inonu are the winners of the Sarukkai Social Impact Award, for their project, Perceive AI.
The Sarukkai Social Impact Award was established by Sekhar and Rajashree Sarukkai in 2021. It recognizes capstone and final projects with the greatest potential to solve important social problems.
The MIDS team created an assistive shopping app that uses real-time grocery item recognition to guide visually impaired users through stores with precision, providing clear directional audio guidance, a screen-reader-friendly interface, and built-in voice command functionality.
We spoke to the team to learn more —
What inspired your project? How did you decide on the concept?
Vijaykumar: The inspiration came from a shared desire to build technology that meaningfully improves daily life. We noticed that while navigation tools exist, they’re often not designed with the visually impaired in mind. One of our members, Melody, had a family member who struggled with maintaining independence in public spaces, while I had the experience of watching my high school physics teacher live with retinitis pigmentosa and navigate the everyday challenges of vision loss. These moments led us to ask: what if a phone camera could become a companion that helps you see the world? That question sparked the idea for our assistive vision app.
Melody: Growing up, I saw my mom struggle with low vision. This only escalated as she aged, and now she is fully reliant on others to get around safely. Walking around a store with her has been a learning experience on some of the challenges that our product can help solve for many others.
“There is so much opportunity to use what we’ve learned in this program to enrich and support our local communities and industries. It just takes people who are willing to build it. The Sarukkai Award validated what we were trying to do: not just build tech, but build impactful tech.”
What was the timeline or process like from concept to final project?
Vijaykumar, Austin, and Mehmet: The process evolved over several months:
Phase 1: Ideation and Research — We explored use cases, met with visually impaired users, and reviewed existing tech.
Phase 2: Data and Prototyping — We curated datasets of grocery products, outdoor objects, and daily-use items from both open sources and our own image captures. Early object detection experiments used transfer learning with YOLO models (starting from v3 and v5, later moving to larger versions for accuracy). For mobile deployment, we exported models into CoreML format and validated them on iOS using the standard vision request pipeline. On the app side, we built the initial camera-to-model flow in Swift and connected it to a simple text-to-speech system for real-time voice prompts.
Phase 3: Integration and Iteration — We connected the live camera feed with the detection pipeline and added backend connectivity through AWS Amplify for synchronization and data handling. The app structure was modular, separating detection, speech, and connectivity, making it easier to test and iterate. We introduced strategies such as frame throttling to reduce redundant uploads, device-specific tags for routing backend responses, and prioritized voice feedback where higher-value messages could override basic detection outputs. These iterations reduced noise and improved overall user experience.
Phase 4: User Testing and Refinement — We tested in grocery stores with visually impaired users, focusing on latency, clarity of audio feedback, and ease of interaction. Based on feedback, we added support for customizable detection lists built directly through voice input, synced with the backend, and used for filtering detections. On the backend, server functions parsed uploaded image metadata, ran inference, and returned contextual messages through WebSocket. Training refinements included addressing look-alike items that caused misclassifications and balancing dataset samples to improve accuracy in real-world testing.
How did you work as a team? How did you work together as members of an online degree program?
Abhishek: As a distributed team, we relied heavily on structured coordination. We used tools like GitHub, Figma, and XCode to manage tasks asynchronously. Weekly video check-ins helped us align priorities. We also set up a Slack channel dedicated to debugging, user feedback, and ideation. Clear roles (tech lead, product design, user research, AI engineer) and mutual trust between the five of us made our remote collaboration fun and seamless.
Austin: Trust and technical excellence were the bedrock of our team’s success. Within moments of meeting, it was clear each member was highly technical and motivated to help our local community. As professionals with diverse backgrounds, we coordinated weekly, owned our tasks, and delivered what we said we would.
How did your I School curriculum help prepare you for this project?
Abhishek: The fact that the five of us took different electives really helped give us a rounded set of skills for a capstone project. For example, Vijay took W281: Computer Vision, which helped with how we thought about depth estimation of items within our app. When I took W266: Natural Language Processing, I completed a final project on multimodal large language models (LLMs). This knowledge helped me with the implementation of a multimodal LLM within our app’s backend. All of us took W201: Research Design and Applications, where we learned how to effectively communicate data science projects to different stakeholders. This helped us develop a punchy presentation to summarize our project at the end of the semester.
Do you have any future plans for the project?
Austin: Yes! In May, we were accepted into SkyDeck’s Pad 13, a startup accelerator. We plan to continue developing the app into a standalone product. The roadmap includes:
- Expanding the object detection model to support more items (e.g., grocery aisles, traffic signs).
- Adding personalization (e.g., remembering frequently bought items).
- Launching a closed beta with visually impaired users in partnership with local organizations.
How could this project make an impact, or, who will it serve?
Melody: This app empowers people who are blind or visually impaired to navigate stores, streets, and unfamiliar spaces more safely and independently. In all, it serves:
- Visually impaired individuals seeking autonomy in daily life.
- Caregivers looking for tools to support loved ones.
- Retailers and cities interested in accessibility tech integration.
Ultimately, we hope it will become a companion that gives users greater confidence while navigating their world.
You were recognized with the Sarukkai Social Impact Award. How does that feel?
Austin: We’re incredibly grateful to be recognized for our work. There is so much opportunity to use what we’ve learned in this program to enrich and support our local communities and industries. It just takes people who are willing to build it. The Sarukkai Award validated what we were trying to do: not just build tech, but build impactful tech.
Additional info to share?
Mehmet: We’d love to connect with:
- Anyone working in accessibility, AI for good, or retail innovation.
- Volunteers willing to test the app and give us feedback.
- Partners in public transportation or smart cities.
We’re also exploring integrations with voice assistants and wearables (like smart glasses) for future versions.
