Automated point cloud classification using an image-based instance segmentation for structure from motion: Fid-Bau Portal

Automated point cloud classification using an image-based instance segmentation for structure from motion

Saovana, Natthapol / Yabuki, Nobuyoshi / Fukuda, Tomohiro

Abstract Point cloud constantly gains popularity as a visualization tool in numerous fields including civil infrastructure scope. However, automatic point cloud classification for civil infrastructures such as piers is challenging due to untidy scenes, gigantic sizes, and image feature-rich objects that can generate many cloud points. Moreover, the lack of training point clouds and unrealistic synthetic data preventing deep learning to fully support the three-dimensional point cloud classification. This paper proposes Point cloud Classification based on image-based Instance Segmentation (PCIS), an automated point cloud classification based on two-dimensional digital images from a daily work basis. These images are processed into the pre-trained network in PCIS to generate mask images, which are later used to create three-dimensional masks based on the projection from the solved camera parameters. The cloud points located inside these masks are classified as the cloud point of interest. The experiment result showed that PCIS correctly classified the point cloud and achieved up to 0.96 F1-score from a one-class classification sample and 0.83 F1-score from a six-class classification sample in our validation process. Our study has proved that normal digital images can also be used to train deep learning to classify the three-dimensional point cloud.

Highlights • A point cloud classification based on image-based instance segmentation is proposed. • The classification used an automated mask projection from construction site images. • The inputs are segmented images and camera poses from structure from motion. • The validation results showed that our system achieved an F1-score of up to 0.96.