https://github.com/graphdeco-inria/gaussian-splatting

Preface

Introduction

3D 高斯溅射是时下最新的一项计算机视觉/计算机图形学/深度学习技术，它是“神经渲染”领域的最新的成果。它引入了三项关键技术，使得在保证画质的同时，也能实现实用的训练时间，并能在 1080p 分辨率下达到实时渲染

Fundamental Theories

技术核心

• NeRF

  • 传统NeRF是把场景表示为一个连续函数 $F(\mathbf{x}, \mathbf{d})$，输入3D空间点 $\mathbf{x}$ 和视角方向 $\mathbf{d}$
  • 但NeRF渲染需要体积积分，训练和推理都很慢。

• SfM

  • 并没有直接在稠密体素上优化，而是先通过COLMAP/SfM得到稀疏点云。
  • 每个点都变成一个"高斯球体"来近似场景。

• Gaussian Splatting

  • 每个场景点被建模为一个3D Gaussian Ellipsoid，它们带有协方差矩阵 $\Sigma$
  • 渲染时并未逐点绘制，而是把这些 Ellipsoid 投射到2D图像上，得到平滑的 Splat

• Differentiable Rasterization

  • 渲染可微，使得通过反向传播来优化位置、形状、颜色、透明度等参数
  • 用的是类似 OpenGL 的快速 Rasterizer

基础数学模型

• 单个高斯定义

一个高斯分布中心在 $\mu \in \mathbb{R}^{3}$，协方差矩阵 $\Sigma \in \mathbb{R}^{3 \times 3}$：

这里：

• $\mu$：高斯的中心（3D点的位置）
• $\Sigma$：协方差矩阵，控制形状（球形/椭球）

• $G(\mathbf{x})$：某点在该高斯下的"贡献"

  ---

• 投影到 2D 图像平面

通过相机投影矩阵 $\Pi$，将 $\mu$ 和 $\Sigma$ 映射到图像平面：

其中 $J$ 是投影的雅可比矩阵。

这使得 3D 高斯在屏幕上变成一个 2D 高斯斑点（一个模糊的小椭圆）。

• 颜色与透明度合成

每个高斯还带颜色 $\mathbf{c}$ 和透明度 $\alpha$。

图像上的最终像素值是多个高斯 splat 的加权混合：

其中 $T_{i} = \prod_{j<i} (1-\alpha_{j})$ 表示前面高斯的透射率（类似体积渲染公式）。

• 优化目标

与真实照片对比，最小化差异（MSE、SSIM、LPIPS 等损失）。

通过反向传播，更新 $\mu,\Sigma,\alpha,\mathbf{c}$ 等参数，使得渲染结果与真实图像一致。

Build Step

Device & System Preparation

• NVIDIA GPU & GPU Drive
• ANY X86_64 CPU（High computing power better）
• Ubuntu 24
• Anaconda OR Miniconda
• Python 3.9（important）

About CUDA Version

• Use CUDA Version == 12.8

The default CUDA version on the Ubuntu24 system is 12.9, and latest version is 13.

However, using newer versions of CUDA Toolkit + PyTorch may cause various issues, such as requiring higher Python versions that are not supported by Gaussian Splatting.

Install

Review the Environment.yml before starting, as it contains all the modules required for use.

If you are using servers or computers located within China, please configure a network proxy or set up a download mirror in advance to avoid download failures.

Please ensure that your system has Git installed before executing the command.

Step 1: Git Clone

git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive

Note:--recursive must be included because the project has submodules.

Step 2: Create Conda Environment

conda create -n gaussian_splattingpython=3.9 ipython
conda activate gaussian_splatting

Step 3: Install CUDA Tookit

Please check if your CUDA Toolkit version is 12.8. If so, proceed to execute the final command in the "Third Step."

nvcc --version

Note: The CUDA version displayed by the nvidia-smi command refers to the driver version, not the CUDA Toolkit version.

If not, you will need to uninstall the current version and reinstall it. This guide will skip the uninstallation steps and focus solely on the installation process for version 12.8.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

conda install -c nvidia cuda-toolkit=12.4

Conda will install CUDA Toolkit 12.4, which remains backward-compatible with CUDA 12.8 instructions.

Step 4: Install the C++ Compilation Environment

sudo apt-get update 
sudo apt install build-essential ninja-build

Step 5: Install Additional Dependencies

conda install -c conda-forge plyfile
conda install tqdm

and important step:

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

Installation will be aborted if your Python version is not 3.9.

pip install opencv-python joblib

Finally, ensure numpy==1.22. If not, please uninstall the current numpy using the pip and reinstall version 1.22.

Step 6: Modify Modules

First, navigate to and enter your project directory. For example, my directory is ~/Code/gaussian-splatting.

cd ~/Code/gaussian-splatting

and edit file to include float.h: #include <float.h>

vim submodules/simple-knn/simple-knn.cu

Step 7: Compile Modules

pip install submodules/diff-gaussian-rasterization 
pip install submodules/simple-knn
pip install submodules/fused-ssim

Final Step

After saving the dataset, you can execute the command to begin training.

python train.py -s ./xxx