Automatic food detection in egocentric images using artificial intelligence technology

<jats:title>Abstract</jats:title> <jats:sec id="S1368980018000538_abs1" sec-type="general"> <jats:title>Objective</jats:title> <jats:p>To develop an artificial intelligence (AI)-based algorithm which can automatically detect food items from images acquired by an egocentric wearable camera for dietary assessment.</jats:p> </jats:sec> <jats:sec id="S1368980018000538_abs2" sec-type="general"> <jats:title>Design</jats:title> <jats:p>To study human diet and lifestyle, large sets of egocentric images were acquired using a wearable device, called eButton, from free-living individuals. Three thousand nine hundred images containing real-world activities, which formed eButton data set 1, were manually selected from thirty subjects. eButton data set 2 contained 29 515 images acquired from a research participant in a week-long unrestricted recording. They included both food- and non-food-related real-life activities, such as dining at both home and restaurants, cooking, shopping, gardening, housekeeping chores, taking classes, gym exercise, etc. All images in these data sets were classified as food/non-food images based on their tags generated by a convolutional neural network.</jats:p> </jats:sec> <jats:sec id="S1368980018000538_abs3" sec-type="results"> <jats:title>Results</jats:title> <jats:p>A cross data-set test was conducted on eButton data set 1. The overall accuracy of food detection was 91·5 and 86·4 %, respectively, when one-half of data set 1 was used for training and the other half for testing. For eButton data set 2, 74·0 % sensitivity and 87·0 % specificity were obtained if both ‘food’ and ‘drink’ were considered as food images. Alternatively, if only ‘food’ items were considered, the sensitivity and specificity reached 85·0 and 85·8 %, respectively.</jats:p> </jats:sec> <jats:sec id="S1368980018000538_abs4" sec-type="conclusions"> <jats:title>Conclusions</jats:title> <jats:p>The AI technology can automatically detect foods from low-quality, wearable camera-acquired real-world egocentric images with reasonable accuracy, reducing both the burden of data processing and privacy concerns.</jats:p> </jats:sec>