Visual and Thermal Imaging Camera-Based System for a Smart Cooking Assistant

Homelessness is a condition that not only deprives individuals of a place to live but also exposes them to various risks, including malnutrition. Personalized cooking recipe recommendation systems have the potential to guide homeless individuals or those recently out of homelessness towards making informed dietary choices. However, a significant gap in current systems is their failure to cater to individuals with minimal cooking experience, offering little to no guidance on the actual cooking process. Addressing this critical gap, the present work introduces an innovative system designed to function as a smart cooking assistant. This system is not merely a passive repository of recipes but an active participant in the cooking process. It is built on the premise of observing users as they cook, utilizing a combination of hardware and advanced machine learning software to guide them through each step of the recipe meticulously. The system's hardware infrastructure is centered around the Raspberry Pi mini-computer, a compact yet powerful device capable of integrating various sensors, including a standard camera, an infrared thermal camera, and a humidity and temperature sensor. These components are strategically mounted above the cooking area, specifically focusing on the stovetop where the cooking vessel is placed. This setup enables the system to continuously monitor the cooking process. The core of the system's software is a deep learning image classification model that is generated using Google Vertex AI's Transfer learning functionality and trained on a dataset of 330 images collected from cooking pasta, captured through a smartphone camera and custom-designed hardware. By applying this method, the algorithm efficiently interprets the series of images to precisely identify the current stage of cooking and offer timely and automated suggestions. By offering a personalized, interactive, and educational cooking experience, the system not only aims to improve the nutritional intake of homeless individuals but also empowers them with the skills and confidence needed to cook healthy meals.