GSrec

Surface Reconstruction from 3D Gaussian Splatting
via Local Structural Hints

ECCV 2024

Qianyi Wu1     Jianmin Zheng2     Jianfei Cai1,2    
1Monash University   2Nanyang Technological University  

Abstract

TL;DR: Monocular geometry guidance to augment 3DGS with normal attributes, then use neural implicit representation to joint optimize the moving least square field formed by the 3DGS.

This paper presents a novel approach for surface mesh reconstruction from 3D Gaussian Splatting (3DGS), a technique renowned for its efficiency in novel view synthesis but challenged for surface reconstruction. The key obstacle is the lack of geometry hints to regulate the optimization of millions of unorganized Gaussian blobs to align to the true surface. This paper introduces local structural hints during training to address the challenge. We first leverage the prior knowledge from monocular normal and depth estimations to refine the covariance and mean of Gaussian primitives, enhancing their organization and providing crucial normal information for surface extraction. However, due to the highly discrete nature of Gaussian primitives, such geometry guidance remains insufficient for the alignment with the true surface. We then propose to construct a signed distance field by a moving least square (MLS) function over the Gaussians in each local region. More importantly, we further propose to jointly learn a neural implicit network to mimic and regularize the MLS function. The joint optimization helps the optimization of Gaussian Splatting towards accurate surface alignment. Extensive experimental results demonstrate the effectiveness of our method in achieving superior mesh quality compared with the SoTA surface reconstruction for 3DGS.

Results Gallery



Replica

Scene surface reconstruction result of .


Scannet

Scene surface reconstruction result of Scannet .


DTU

Here are some surface reconstruction results of DTU dataset.

Related Links

Here are some concurrent works we found that also aim to improve the surface reconstruction quality of Gaussian Splatting, welcome to check them out (sorted by arxiv date):
  • GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction
  • DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing
  • 2D Gaussian Splatting for Geometrically Accurate Radiance Fields
  • GauStudio: A Modular Framework for 3D Gaussian Splatting and Beyond
  • 3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting
  • Surface Reconstruction from Gaussian Splatting via Novel Stereo Views
  • Gaussian Opacity Fields: Efficient and Compact Surface Reconstruction in Unbounded Scenes
  • High-quality Surface Reconstruction using Gaussian Surfels
  • RaDe-GS: Rasterizing Depth in Gaussian Splatting
  • PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction
  • Trim 3D Gaussian Splatting for Accurate Geometry Representation

  • It is also welcome to check our recent works on 3D scene representation, editing and understanding:
  • HAC. A 3DGS compression framework by using context model to achieve ~75x size reduction.
  • GScream. Perform object removal under 3D Gaussian Splatting with coherent geometry & texture and high efficiency.
  • PCF-Lift. Probalistic features can make the 2D panoptic segmentation lifting problem more robust with better performance.
  • ClusteringSDF. A clustering framework to extend ObjectSDF/++ to handle view-inconsistent mask.

  • BibTeX

    @inproceedings{Wu2024gsrec,
            author    = {Wu, Qianyi and Zheng, Jianmin and Cai, Jianfei},
            title     = {Surface Reconstruction from 3D Gaussian Splatting via Local Structural Hints},
            booktitle = {European Conference on Computer Vision},
            year      = {2024}
        }