Build Gaussian Splatting Trainer for Ubuntu 24 & New CUDA SDK Version

https://github.com/graphdeco-inria/gaussian-splatting

YouTube Video:

Preface

Introduction

3D Gaussian Splatting is the latest technology in computer vision, computer graphics, and deep learning, representing the newest achievement in the field of "neural rendering." It introduces three key techniques that ensure high image quality while also achieving practical training times and real-time rendering at 1080p resolution.

Fundamental Theories

Technical Core

NeRF
- Traditional NeRF represents a scene as a continuous function $F(\mathbf{x}, \mathbf{d})$ , taking a 3D point $\mathbf{x}$ and a viewing direction $\mathbf{d}$ as input.
- However, NeRF rendering requires volume integration, making both training and inference slow.
SfM
- Instead of directly optimizing on dense voxels, it first obtains a sparse point cloud via COLMAP/SfM.
- Each point becomes a "Gaussian sphere" to approximate the scene.
Gaussian Splatting
- Each scene point is modeled as a 3D Gaussian ellipsoid with a covariance matrix $\Sigma$ .
- During rendering, instead of drawing point by point, these ellipsoids are projected onto a 2D image to produce smooth splats.
Differentiable Rasterization
- The rendering is differentiable, allowing optimization of position, shape, color, opacity, and other parameters via backpropagation.
- It uses a fast rasterizer similar to OpenGL.

Basic Mathematical Model

Single Gaussian Definition

A Gaussian distribution centered at $\mu \in \mathbb{R}^{3}$ with covariance matrix $\Sigma \in \mathbb{R}^{3 \times 3}$ :

G(\mathbf{x}) = \exp\left(-\frac{1}{2}(\mathbf{x} - \mu)^T \Sigma^{-1}(\mathbf{x} - \mu)\right)

Here:

$\mu$ : center of the Gaussian (position of the 3D point)
$\Sigma$ : covariance matrix, controlling the shape (spherical/ellipsoidal)
$G(\mathbf{x})$ : the "contribution" of a point under this Gaussian
Projection to the 2D Image Plane

Using the camera projection matrix $\Pi$ , map $\mu$ and $\Sigma$ to the image plane:

\mu_{2D} = \Pi(\mu), \quad \Sigma_{2D} = J \Sigma J^{T}

where $J$ is the Jacobian of the projection.

This turns a 3D Gaussian into a 2D Gaussian splat (a blurred small ellipse) on the screen.

Color and Opacity Compositing

Each Gaussian also carries a color $\mathbf{c}$ and opacity $\alpha$ .

The final pixel value on the image is a weighted blend of multiple Gaussian splats:

C(\mathbf{p}) = \sum_{i} T_{i} \alpha_{i} \mathbf{c}_{i}

where $T_{i} = \prod_{j<i} (1-\alpha_{j})$ represents the transmittance of preceding Gaussians (similar to the volume rendering formula).

Optimization Objective

Compare with real photos and minimize the difference (losses like MSE, SSIM, LPIPS).

Through backpropagation, update parameters such as $\mu,\Sigma,\alpha,\mathbf{c}$ so that the rendered result matches the real image.

Build Step

Device & System Preparation

NVIDIA GPU & GPU Driver
ANY X86_64 CPU (High computing power better)
Ubuntu 24
Anaconda OR Miniconda
Python 3.9 (important)

About CUDA Version

Use CUDA Version == 12.8

The default CUDA version on Ubuntu 24 is 12.9, and the latest version is 13.

However, using newer versions of CUDA Toolkit + PyTorch may cause various issues, such as requiring higher Python versions that are not supported by Gaussian Splatting.

Install

Review the Environment.yml before starting, as it contains all the modules required for use.

If you are using servers or computers located within China, please configure a network proxy or set up a download mirror in advance to avoid download failures.

Please ensure that your system has Git installed before executing the command.

Step 1: Git Clone

git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive

Note: --recursive must be included because the project has submodules.

Step 2: Create Conda Environment

conda create -n gaussian_splatting python=3.9 ipython
conda activate gaussian_splatting

Step 3: Install CUDA Toolkit

Please check if your CUDA Toolkit version is 12.8. If so, proceed to execute the final command in the "Third Step."

nvcc --version

Note: The CUDA version displayed by the nvidia-smi command refers to the driver version, not the CUDA Toolkit version.

If not, you will need to uninstall the current version and reinstall it. This guide will skip the uninstallation steps and focus solely on the installation process for version 12.8.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda-toolkit-12-8

conda install -c nvidia cuda-toolkit=12.4

Conda will install CUDA Toolkit 12.4, which remains backward-compatible with CUDA 12.8 instructions.

Step 4: Install the C++ Compilation Environment

sudo apt-get update 
sudo apt install build-essential ninja-build

Step 5: Install Additional Dependencies

conda install -c conda-forge plyfile
conda install tqdm

and important step:

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

Installation will be aborted if your Python version is not 3.9.

pip install opencv-python joblib

Finally, ensure numpy==1.22. If not, please uninstall the current numpy using the pip and reinstall version 1.22.

Step 6: Modify Modules

First, navigate to and enter your project directory. For example, my directory is ~/Code/gaussian-splatting.

cd ~/Code/gaussian-splatting

and edit file to include float.h: #include <float.h>

vim submodules/simple-knn/simple-knn.cu

Step 7: Compile Modules

pip install submodules/diff-gaussian-rasterization 
pip install submodules/simple-knn
pip install submodules/fused-ssim

Final Step

After saving the dataset, you can execute the command to begin training.

python train.py -s ./xxx