Trellis.2 fine-tuning scripts are now available for local and cloud use
The Hugging Face team has released scripts and containers to fine-tune the Trellis.2 model on custom 3D assets. The tools run locally or on Hugging Face Jobs and require minimal arguments if your dataset is updated to the Hub.
Introduction
3D generation serves robotics, simulation, media, and entertainment. These fields struggle to enforce strict rules and standardise model representations. File formats like GLB and PLY vary widely. Representations include neural radiance fields and 3D Gaussian splatting. Each format carries different advantages and disadvantages.
Trellis introduced a unified latent representation. Its decoders convert this representation into 3D models in the required output format. Users do not need separate models for each representation format.
The same team released Trellis.2 to improve the model’s representation of 3D structure. Neither model offers an easy way to adapt it to a specific task.
What we want to improve
A general-purpose model like Trellis.2 can err in details important for a specific application. A user may need accurate geometry for furniture models. Without fine-tuning for that task, the pretrained model produces geometry that is insufficiently accurate. We use a subset of GSO that contains 188 footwear objects for our experiment. Initial tests showed the pretrained model usually produced shoes that looked correct, but the soles were incorrect. We test whether a LoRA can make the soles more accurate.
Our goal is to use the Google Scanned Objects dataset to improve the performance of Trellis.2 on a specific object category. We explain which pipeline components to fine-tune and which to leave unchanged during training. The pipeline separates structure, detailed geometry, and materials. We will fine-tune only the components important for our task.
Choosing what to freeze matters as much as choosing what to fine-tune.
What will we use?
LoRA
LoRA, short for low-rank adaptation, fine-tunes a model by training small adapter matrices while keeping the original weights frozen. This reduces the number of trainable parameters and the memory needed for their gradients and optimizer state. We use Hugging Face’s PEFT library to add and train these adapters.
Hugging Face Jobs
Hugging Face Jobs runs containerized workloads on cloud hardware. We use it for preprocessing and training. To use your own GPU instead, see the local Docker commands below. Before you start, make sure that you have:
- A HF_TOKEN that you can obtain from here
- Your Hugging Face account must have access to DINOv3, because DINOv3 is a gated model
- If you use your own dataset, upload its meshes to the Hub before you start.
How Trellis.2 works
Before we train, let us look at how Trellis.2 generates a 3D object and where we will add LoRA adapters.
Trellis.2 has five main stages: remove the image background, encode the image, predict the coarse structure, generate detailed geometry, and generate materials. The pipeline then exports the result as a textured mesh.
Encoding the input image
First, BiRefNet removes the background from the input image. DINOv3-L then encodes the image into feature tokens. The generation models use these tokens to guide the 3D object.
Sparse Structure Generation
The sparse-structure flow DiT predicts the object’s coarse shape. It is a diffusion transformer that makes a small latent of size 16³ × 8. A variational autoencoder decoder then changes this latent into a 64 × 64 × 64 occupancy grid. Each voxel in the grid shows if that location is part of the object. A voxel is a pixel in three dimensions. We add LoRA adapters to this model because its output guides the geometry stage.
O-Voxel and Structured Latents
O-Voxel is the 3D format of Trellis.2. It is a sparse voxel grid. It keeps only the voxels on the object’s surface. Each voxel keeps local geometry, such as a mesh vertex, and material attributes. Because the format is small, a CPU changes a mesh into O-Voxel in seconds. A GPU changes O-Voxel back into a mesh in seconds.
Structured sparse latents come from Trellis. They keep 3D data in sparse tensors, which makes computation faster. The model uses only approximately 10% of the voxels in the grid, so it does not spend compute on empty voxels. Thus, the shape flow DiT and the texture flow DiT are sparse transformers. The sparse-structure flow DiT is a dense transformer.
Geometry Generation
The shape flow DiT uses the occupancy grid and the DINOv3 feature tokens to make a latent. Another VAE decoder then changes this latent into O-Voxel geometry at a resolution of 512 × 512 × 512.
We add LoRA adapters to the DiT at this stage because it controls the object’s shape.
Material Generation and Mesh Creation
In the last stage, the texture flow DiT uses the shape latent and the DINOv3 features as input. It defines the materials and textures of the geometry’s surface. A VAE decoder then changes the latent into base color, metallic, roughness, and alpha values for each voxel. We do not add LoRA adapters to this stage. Our dataset has diffuse color maps, but it lacks the separate material maps this stage needs for training. Also, our problem is the shoe’s shape. The base model already makes good colors and textures.
Finally, the pipeline combines the geometry and material attributes into a textured mesh and exports it as a GLB file.
Planning
Which components to fine-tune
We add LoRA adapters only to the sparse-structure flow DiT and the shape flow DiT. These DiTs control the object’s shape and geometry. We freeze the texture flow DiT to keep its current capabilities.
Our dataset contains diffuse color maps that include baked lighting effects. To train the texture flow DiT, we need separate material maps. The color maps in our dataset mix surface color with lighting. Thus, they are not correct targets for this training.
If we train the texture flow DiT on these maps, it can learn to show lighting as a surface property. It can also forget what it learned before, a problem called catastrophic forgetting.
Why use LoRA
Our subset has 188 shoes from the 1,030 objects in GSO. For a dataset this small, LoRA is a better choice than full fine-tuning. Full fine-tuning costs much more. It can also overwrite what the model already does well. Our configuration trains approximately 9.3 million adapter parameters in total for both targeted DiTs. We use rank 8 adapters in the attention and feed-forward layers of the sparse-structure flow DiT. We use rank 16 adapters in the attention layers of the shape flow DiT. The rank sets the size of the adapter matrices. A higher rank lets the adapter learn more.
What you will run
A 3D model setup can need many dependencies and installation steps. We supply two Docker images with the necessary software. One image is for preprocessing, and one is for fine-tuning.
To train your LoRA adapters and upload them to the Hub:
- Give the containers access to your data.
- Set the parameters for your task.
- Run the preprocessing container.
- Run the fine-tuning container.
Then we explain the fine-tuning script, its settings, and how each setting affects training.
Experiment overview
We will walk through three parts of the experiment:
- Prepare the data. Convert GLB or OBJ meshes into the latent representations Trellis.2 uses for training, and render the input images.
- Train the adapters. Run one job for each DiT, with the prepared dataset and a LoRA configuration.
- Explain the code. Examine how the containers load the data, add adapters, and train the models.
We save the preprocessed dataset so that all training runs can reuse it.
Walkthrough
Running on Hugging Face Jobs
Preprocessing
The preprocessing container uses Blender to render 16 views of each object. LoRA training will use these views as conditioning images. Blender runs without a display, through the xvfb-run utility. The container also processes each object through the preprocessing pipeline. See the footwear dataset here for an example intermediate output. For our 188 objects, the full run took approximately 1 hour on an a100-large job. A compact version of the preprocessing pipeline is also available in this companion repository.
export HF_TOKEN=hf_xxx
# Replace your-username with your Hugging Face username
hf jobs run -d --flavor a100-large --timeout 6h --secrets HF_TOKEN \
cr.hf.co/suvadityamuk/trellis2-finetuning-prep:latest \
trellis2-prep \
--meshes your-username/my-meshes \
--push-to your-username/my-meshes-trellis2 \
--private --holdout 0.05 \
--views 16 --workers 8For Hugging Face Jobs, we pass the following arguments:
| flag | meaning |
|---|---|
-d | Detach: print the job ID and URL, then return. To follow the job, runhf jobs logs -f <id> |
--flavor a100-large | The hardware. We use a100-large (80 GB). To see all options, run hf jobs hardware |
--timeout 6h | The job stops after this time. Both commands resume. Preprocessing skips finished objects. Training continues from the newest checkpoint in –push-to |
--secrets HF_TOKEN | Send your token (from HF_TOKEN or hf auth login) to the job as an encrypted environment Source Read original → Related articles   |