Article Preview
TopIntroduction
Recently, we have witnessed the splendid development of wearable sensor technologies that have been utilized to support human lives, including the smartwatch and personal digital assistant (Seneviratne, et al., 2017). The small sizes and affordable prices of wearable sensors, high bandwidth of the Internet, and flexible cloud storage encourage people to record data more frequently than before. The massive generated data enables individuals, organizations, research institutions to analyze the data to answer real-time questions and detect patterns. Therefore, a vast of research has been investigated on several categories relating to human lives such as education (Dao., Dang-Nguyen., Kasem., & Tran-The., 2018), healthcare (Mosenia, Sur-Kolay, Raghunathan, & Jha, 2017), human-computer interface (Steil, Müller, Sugano, & Bulling, 2018), entertainment (Cao, Xie, & Chen, 2019), and security (Blasco, Chen, Tapiador, & Peris-Lopez, 2016). That leads to the fact that people can have more opportunities to understand their lives well due to daily recording data whose content conceals both cognitive and physiological information, and have a better understanding of their social behavior. The more data we collect, the better it is for us to understand what goes on in everyday life and how to understand our actions better.
In order to satisfy a part of demands mentioned above, several events have been organized focusing on finding a solution for two exciting topics: daily living understanding and lifelog moment retrieval (Gurrin, Joho, Hopfgartner, Zhou, & Albatal, Overview of NTCIR-12 Lifelog Task, 2016), (Gurrin, et al., 2017); (Dang-Nguyen, et al., Overview of ImageCLEFlifelog 2017: Lifelog Retrieval and Summarization., 2017); (Dang-Nguyen, et al., 2018). These events offer a large annotated lifelog collected from various sensors such as physiology (e.g., heartbeat, step counts), images (e.g., lifelog camera), location (e.g., GPS), user's tags, smartphone logs, and computer logs. Unfortunately, the results of proposed solutions from participants are far from expected. It means that lifelogging is still a mystical land that needs to be discovered more in the insights we can extract from lifelogs and how the accuracy of these insights is.
In this paper, we introduce a new interactive method that has two following main functions:
- 1.
Purify and enrich a sample set for searching using “querying by sample” by converting a textual query to image samples, allowing users to interfere with eliminating irrelevant samples, and enriching the samples set by interactive loops. We call these image samples “event's seeds.”
- 2.
Divide time-ordered images into atomic clusters whose content sharing similar content and context. Then, assign a whole cluster into one event if this cluster contains at least one sample image (i.e., event’s seeds) detected in function 1. Moreover, the watershed-based approach is applied to connect consecutive atomic clusters as long as at least one of them contains the event's seeds.
The contribution of the proposed method comes directly from these two functions. The first function makes sure the content of the given query is converted into the most semantic-related images, namely the event's seeds. The second function leverages the context of events to watershed-based gathering all clusters that share the same content and context with the event's seeds. Hence, instead of comparing each image with a querying image (sample) like other existing methods, we find the small group of images that share the same content with the query, and automatically gather all relevant atomic clusters to form the final results. Thus, our method is logically faster and has better accuracy. Moreover, by using interactive mode, such seeds can be modified by users to eliminate irrelevant seeds and enrich relevant seeds manually (e.g., adding more semantic content in querying).
TopThe popular techniques that have been applying to cope with LRMT challenge can be summarized as follows: