A tutorial demonstrates how to build an end-to-end hierarchical Neural Radiance Field using JAX, Flax, Optax, and the volume-rendering primitives from the jax3d library.
Building the synthetic scene
The process starts by creating a synthetic multi-view dataset derived from an analytic scene. This setup includes volumetric geometry and view-dependent radiance. The code uses sample_along_rays and volume_rendering to establish the forward rendering process. It then implements a NeRF model featuring positional encoding, skip connections, separate coarse and fine networks, and view-direction conditioning. Hierarchical importance sampling is handled via sample_piecewise_constant_pdf.
Training relies on JAX JIT compilation, Adam optimisation, exponential learning-rate decay, and gradient clipping. Evaluation covers novel-view synthesis using PSNR, depth and opacity visualisation, sampling diagnostics, 360-degree rendering, and marching-cubes geometry extraction.
Installing dependencies and configuring the environment
The following Python script sets up the JAX3D environment and installs required packages like etils, chex, flax, optax, and scikit-image. It clones the google-research/jax3d repository if the local directory does not exist.
The tutorial loads the volume_rendering module directly from the cloned repository to avoid pulling in unnecessary dependencies like gin or tfds.
Configuration parameters define a 64×64 resolution with 24 training views and 3 test views. The camera radius is set to 3.2 with a 40-degree field of view. Near and far clipping planes are 1.9 and 4.7 respectively. The model uses 64 samples for the coarse network and 64 for the fine network. Positional encoding is set to 10 degrees, with view-direction encoding at 4 degrees. The network width is 128 with a depth of 6 and a skip connection at layer 3. Training runs for 2,500 steps with an initial learning rate of 5e-4 decaying to 5e-6. The batch size is 2,048 rays.
If the system detects a CPU rather than a GPU, the script automatically switches to a smaller CPU-friendly configuration. This reduces the resolution to 40×40, limits training views to 14, and cuts the step count to 400.
Defining the camera model and scene geometry
The code establishes the camera model using pinhole intrinsics and look-at poses. It generates normalised world-space rays from each camera pose to provide the geometric foundation for the rendering pipeline. Camera placement follows an orbit-based system using golden-angle azimuths and monotone elevations to ensure well-spread views on a dome.
The analytic ground-truth scene contains soft-edged spheres, a patterned floor, and view-dependent specular radiance. Lighting is defined by a single light source at coordinates [0.55, 0.75, 0.85]. Three spheres are positioned at specific coordinates with defined radii and albedo colours.
The floor field calculation uses sigmoid functions to create soft edges based on x and y coordinates. A checkerboard pattern is applied to the floor texture using integer division of the floor coordinates.
Rendering the ground truth dataset
The script renders the synthetic multi-view dataset using JAX3D’s volume-rendering implementation. It samples along rays to generate depth and position data before computing density-weighted blends of the spheres and floor.
Training views are generated using the orbit function with a phase of 0.00. Test views use a phase of 0.41 and a specific elevation range. The code flattens the ray origins, directions, and colours into arrays for the ray pool. A pool of 2,048 rays is created for the training set.
Visualisation plots the first eight training views, displaying the rendered ground truth images alongside their camera coordinates.
