TorchNeuron, a new PyTorch plugin for AWS Trainium, launches into public beta on 20 October 2026. Developers install it via pip install torch-neuronx.
Native PyTorch support on Trainium
Previously, running models on AWS Trainium and Inferentia required the optimum-neuron library. That tool relied on torch-neuronx, which was built on PyTorch XLA. The setup demanded a compiler mindset: lazy tensors, static shapes, and graph recompilation on every break. optimum-neuron hid this complexity but required custom code for model classes and trainers.
TorchNeuron changes this. It integrates directly into PyTorch and exposes neuron as a native device. It supports eager execution, PyTorch’s own distributed primitives like DTensor and FSDP2, and torch.compile. Hugging Face libraries now treat Trainium like any other accelerator. Transformers, Diffusers, Accelerate, PEFT, and TRL run on it using the same classes and scripts as other hardware. Users should now use these libraries directly instead of Optimum Neuron.
Transformers
Inference
Text generation is now natively supported in Transformers for a wide range of models. TorchNeuron supports 89% of the top 100 most-used text-generation model architectures out of the box, compared to 46% with Optimum Neuron v0.4.5. Users simply pass the device_map=neuron argument to move their model to Neuron devices.
This example demonstrates text generation with tensor parallelism. It uses a static cache for generation, which is strongly recommended with Neuron to prevent frequent recompilations.
import os
import torch
import torch.distributed as dist
import torch_neuronx # noqa: F401
from torch.distributed import DeviceMesh
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
from transformers.distributed import DistributedConfig
from transformers.generation.configuration_utils import CompileConfig
MODEL_ID = "Qwen/Qwen3.8-27B"
MAX_SEQ_LEN = 1024
PREFILL_CHUNK_SIZE = 512
dist.init_process_group(backend="neuron")
world_size = dist.get_world_size()
model = AutoModelForCausalLM.from_pretrained(
MODEL_ID,
dtype=torch.bfloat16,
distributed_config=DistributedConfig(tp_size=world_size),
device_mesh=DeviceMesh("neuron", list(range(world_size))),
).eval()
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": "The future of AI is"}],
add_generation_prompt=True,
padding=True,
tokenizer_kwargs={"pad_to_multiple_of": PREFILL_CHUNK_SIZE},
return_tensors="pt",
)
generation_config = GenerationConfig(
max_new_tokens=128,
do_sample=False,
use_cache=True,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
cache_implementation="static",
max_cache_len=MAX_SEQ_LEN,
prefill_chunk_size=PREFILL_CHUNK_SIZE,
compile_config=CompileConfig(backend="neuron", fullgraph=False, dynamic=False),
)
with torch.no_grad():
output_ids = model.generate(
inputs["input_ids"],
attention_mask=inputs["attention_mask"],
generation_config=generation_config,
)
if dist.get_rank() == 0:
prompt_len = inputs["input_ids"].shape[1]
print(tokenizer.decode(output_ids[0, prompt_len:], skip_special_tokens=True))
Training
With Neuron as a native PyTorch device, training on Trainium uses the same Hugging Face libraries as any other accelerator: Transformers, Accelerate, PEFT, and TRL. Users no longer need a dedicated trainer or a Neuron-specific fork of the model code. A standard TRL fine-tuning script runs on a Trainium instance with only a few Neuron-specific settings.
This supervised fine-tuning script uses TRL’s SFTTrainer. If you have fine-tuned a model with TRL on GPUs before, it should look familiar.
import os
import torch
from datasets import load_dataset
from transformers.distributed import DistributedConfig
from trl import ModelConfig, ScriptArguments, SFTConfig, SFTTrainer, TrlParser
def main(script_args, training_args, model_args):
# The Neuron backend can't compile Dynamo's automatic dynamic shapes yet (https://github.com/huggingface/trl/issues/7660)
torch._dynamo.config.automatic_dynamic_shapes = False
training_args.model_init_kwargs = {
"revision": model_args.model_revision,
"trust_remote_code": training_args.trust_remote_code,
"attn_implementation": model_args.attn_implementation,
"dtype": model_args.dtype,
"device_map": None,
}
if int(os.environ.get("WORLD_SIZE", "1")) > 1:
# The model is sharded on the TP x FSDP grid as its weights are loaded
training_args.model_init_kwargs["distributed_config"] = DistributedConfig(
tp_size=int(os.environ.get("TP_SIZE", "1")),
fsdp_size=int(os.environ.get("FSDP_SIZE", "1")),
)
# Prepare dataset
train_dataset = load_dataset(
script_args.dataset_name,
name=script_args.dataset_config,
split=script_args.dataset_train_split,
)
# UltraChat also has a string `prompt` column, which makes TRL treat it as a prompt-completion dataset.
train_dataset = train_dataset.select_columns(["messages"])
# Training
trainer = SFTTrainer(
model=model_args.model_name_or_path,
args=training_args,
train_dataset=train_dataset,
)
trainer.train()
trainer.save_model(training_args.output_dir)
trainer.end()
if __name__ == "__main__":
parser = TrlParser((ScriptArguments, SFTConfig, ModelConfig))
script_args, training_args, model_args = parser.parse_args_and_config()
main(script_args, training_args, model_args)
Parallelism is handled natively by Transformers. Users put a DistributedConfig with the tensor-parallel (TP) and FSDP sizes in model_init_kwargs. SFTTrainer passes it to from_pretrained, and the model is sharded as its weights are loaded. The Trainer sees that the model is already distributed and does not wrap it again. Users do not need an --fsdp flag or an Accelerate FSDP config.
Nothing in the script is tied to Trainium: the same code runs on any hardware. A few things are worth knowing when running it on Trainium:
- Static shapes. The script disables automatic dynamic shapes in
torch.compileand uses wrapped packing to produce sequences of up to 2048 tokens. With TRL ≥ 1.16, batches on Neuron are padded to a multiple of 128 tokens, limiting the number of input shapes. Compiled graphs are reused for subsequent steps with the same shapes. torch.compileon. Passing--torch_compile trueis all it takes: on a Trainium instance,torch.compiledetects the device and picks the Neuron backend automatically, with nothing else to configure.- Async weight loading off. The built-in asynchronous weight loading in Transformers is not supported on Neuron, so for now users disable it with
HF_DEACTIVATE_ASYNC_LOAD=1.
The command below launches training with torchrun, which starts one process per NeuronCore. TP_SIZE and FSDP_SIZE set the parallelism layout, and their product is the number of processes: here 4 x 8 = 32, matching --nproc_per_node. The 32 processes form a 4 x 8 grid. Tensor parallelism splits each layer across the 4 cores of a TP group, and FSDP shards the parameters, gradients and optimizer state across the 8 TP groups.
TP_SIZE=4 FSDP_SIZE=8 HF_DEACTIVATE_ASYNC_LOAD=1 uv run torchrun --nproc_per_node=32 sft_neuron_example.py \
--model_name_or_path Qwen/Qwen3-8B \
--dataset_name HuggingFaceH4/ultrachat_200k \
--dataset_train_split train_sft \
--output_dir Qwen3-8B-sft-tp4-fsdp8 \
--dtype bfloat16 \
--max_length 2048 \
--packing true \
--packing_strategy wrapped \
--learning_rate 2e-5 \
--warmup_steps 0.03 \
--num_train_epochs 1 \
--logging_steps 2 \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--save_only_model true \
--torch_compile true
The test ran on 8 Trainium2 chips (32 NeuronCores), which is half a trn2.48xlarge instance.
Training runs to completion, and trainer.save_model writes a standard Transformers checkpoint. Nothing in it is tied to Neuron: users can push it to the Hugging Face Hub as is and load it on any other hardware.
Diffusers
Inference
Many pipelines from the Diffusers library are compatible with TorchNeuron in eager and compiled mode. FLUX.1-dev, FLUX.2, Qwen-Image, Qwen-Image-2.1 and H3 are also compatible with tensor parallelism to make the most of Neuron devices. This example shows how to use TP:
import torch, torch.distributed as dist, torch_neuronx # TP
from torch.distributed.device_mesh import DeviceMesh # TP
from diffusers import QwenImage21Pipeline, QwenImage21Transformer2DModel, TensorParallelConfig# Distributed devices initialization
dist.init_process_group(backend="neuron") # TP
dev = torch.neuron
Source Read original →
![Image generation models running locally on limited resources [P]](https://ai-maestro.online/wp-content/uploads/2026/05/image-generation-models-running-locally-on-limited-resources-768x432.jpg)
