CloudUP

CloudUP – Upsampling Vibrant Color Point Clouds

using Multi-Scale Spatial Attention

(Accepted in IEEE Access 2023)

Younju Cho, Rimsha Tariq, Usama Hassan, Javed Iqbal, Abdul Basit, Hyon-Gon Choo, Rehan Hafiz, Mohsen Ali

Immersive Media Research Lab, Electronics and Telecommunications Research Institute, Daejeon, South Korea
Intelligent Machines Lab, Information Technology University, Lahore, Pakistan

Paper

Abstract

In recent years, there has been a noticeable increase in the inclination towards digitizing our surroundings, encompassing various domains such as virtual reality, cultural heritage conservation, and architectural representation. The computation of high-resolution three-dimensional (3D) colored point clouds and meshes holds significant importance for such applications. However, traditional structure-from-motion (SfM) techniques may produce sparse 3D point clouds when low-resolution input images are used, resulting in a low-quality mesh generation. Traditional point cloud upsampling techniques that improve the 3D point cloud resolution typically work on LiDAR-generated point clouds devoid of color information. Furthermore, most learned point cloud upsampling techniques compute graph features that capture local information by identifying a local neighborhood in a limited region around a point and hence may result in sub-optimal representation. To

address these limitations, we propose CloudUp, a colored 3D point cloud upsampling approach that utilizes multi-scale spatial attention. Specifically, we design a novel across-scale attention mechanism for computing Multi-scale Point-Cloud Features (MPF) that capture 3D shape information of the point with respect to its neighborhood. We further extract spatial neighborhood-guided color features used to predict the color for the upsampled points. The color prediction is trained with a content-preserving loss function that aims to maintain intricate details and vivid colors. Our color refinement pipeline is guided by a vibrant colored dataset (collected by us) to assist in preserving the 3D contents.

Proposed Approach

We propose a two-stream architecture for the up-sampling of colored point clouds. The first stream up-samples the points using a cross-attention-based Multi-scale Point-Cloud Feature (MPF) computed for each point in the input sparse-point cloud. The second stream consists of the color prediction module, which predicts and refines the colors for each point in the up-sampled point cloud. We try to preserve the color variation by introducing a color variance conformity loss to avoid low-resolution color generation. Features computed at different stages of the point upsampling stream are used to compute color features by color prediction scheme.

Results

Quantitative comparison of the proposed CloudUp with SOTA methods on SketchFab and LS-PCQA datasets.

Qualitative results for CloudUp points upsampling on SketchFab dataset. CloudUp upsamples the points precisely leading to accurate point clouds with preserved structure.

About Authors