Two-step long short-term memory method for identifying construction activities through positional and attentional cues: Fid-Bau Portal

Two-step long short-term memory method for identifying construction activities through positional and attentional cues

Cai, Jiannan / Zhang, Yuxi / Cai, Hubo

Abstract Recognizing construction activities and involved working groups is critical to enhancing construction safety and improving productivity. Most existing studies use videos that only contain one activity with involved entities and rely solely on the spatial-temporal relationship among entities. However, in practice, many workers and machines co-exist and collaborate to accomplish different activities, and not all of them are relevant to the same activity, even though they are spatially close. This paper presents a two-step classification approach – working group identification followed by activity recognition, leveraging both positional and attentional cues, to recognize complex interactions and their involved entities from videos that contain different activities with multiple entities. The spatial and attentional states of individual entities are represented numerically, and the corresponding positional and attentional cues between two entities are computed. Long short-term memory (LSTM) networks are designed to (1) classify whether two entities belong to the same group, and (2) recognize the activities they are involved in. The newly created method is validated using two sets of construction videos. Identifying working groups before recognizing ongoing activities enables the exclusion of group-irrelevant entities and thus, improves the performance. Moreover, by leveraging both positional and attentional cues, the accuracy increases from 85% to 95% compared with cases using positional cues alone.

Highlights • Attentional cues are incorporated as critical features for working group identification and group activity recognition. • Numerical methods have been created to compute attentional and positional cues. • LSTM networks are designed to capture the temporal evolution of interactions among construction entities. • A two-step approach is proposed to identify construction working groups and recognize corresponding group activities. • A new method is presented to exclude group-irrelevant entities and improve the performance of group activity recognition.