Thinking Machines releases Inkling, a 1 trillion parameter multimodal model on Hugging Face

Thinking Machines has launched Inkling, a large multimodal language model available on Hugging Face. The system handles text, audio, and images, features agentic capabilities, and operates with a 1 million token context window. It is offered in full BF16 and a calibrated NVFP4 version. The release includes speculative MTP layers to speed up inference. Transformers, SGLang, and llama.cpp all have day-0 support.

Why Inkling is different

This is the first open model with approximately 1 trillion parameters and a 1 million token window to natively accept text, images, and audio. It was trained on 45 trillion tokens covering text, images, audio, and video. The focus is on reasoning across these modalities. The model is designed for domain adaptation through fine-tuning. The team has used the system for demos and architectural exploration. They believe it will support a new wave of multimodal reasoning applications.

Architecture and capabilities

Inkling is a decoder-only multimodal Mixture-of-Experts model. It has 975 billion total parameters and 41 billion active parameters. The architecture uses several distinct techniques.

• Decoder-only: The design supports causal autoregressive generation, similar to most current state-of-the-art LLMs.
• Multimodal: The system ingests text, audio, and images.
• Mixture of Experts (MoE): The feed forward networks inside each layer are sparse. Only 41 billion parameters are active at any given time. The model contains 256 experts.

Relative attention replaces RoPE, the usual method for injecting positional information in transformers. Inkling uses relative attention to encode position. Each attention layer learns position directly in the attention logits. Aside from key-query-values, there is a fourth projection producing a per-token, per-head relative feature R. This projection tensor is tweaked with distance information and propagated into the attention module.

Hybrid attention alternates between global attention and sliding window attention in the decoder layers. Global attention attends to the full context length at once. Sliding window attention attends to a fixed context window in a sliding fashion. The architecture follows a 5:1 pattern of sliding window to global attention layers. This scheme provides computational efficiency. The final layer uses global attention to build feature-rich representations.

Short convolution uses a distinctive short 1D convolution, or SConv, over the hidden states. SConv reads the current token and the previous W-1 hidden states, with W being the sliding window size. The logic is that SConv helps with local attention while freeing the attention and MoE modules from local representations.

MoE with shared experts sink routes scores to both routed experts and shared experts. Top-k selection is performed over 6 experts, plus 2 shared experts that are always active.

Vision understanding includes a simple hierarchical MLP patchifier with several linear layers. Each layer merges pixels progressively until the final layer produces one embedding per patch.

Audio understanding employs a discretized mel spectrogram. Each audio chunk of 100 ms is converted to the mel scale and then classified into the exact mel spectrogram bin.

The multimodal towers are relatively simple modules. Other models often use separate encoders for each modality. Each image patch passes through the image embedding tower. Each audio chunk passes through the audio embedding tower to get media embeddings. Image inputs include an additional temporal dimension for video processing. The team expects this to be useful for downstream fine-tuning but has not evaluated out-of-the-box video performance. The tower folds the patch grid, stacks a small local block of neighboring tokens into the channel dimension, and goes through hMLP. The audio waveform converts to mel scale, which is then classified into a discrete mel bin. These mel bin values embed in the audio embedding tower and the embeddings sum to construct the final audio input.

Inference support

Inkling includes day-0 transformers support and works in major inference engines like SGLang and vLLM.

The model is large. The bf16 checkpoint requires 2 TB of VRAM. The nvfp4 version requires 600 GB of VRAM. Users can try the model through serverless inference routers like Inference Providers. Local deployment with llama.cpp is possible using ggml quants.

Transformers

The easiest way to infer with transformers directly is to use the any-to-any pipeline. Users can use the 16 bit “thinkingmachines/Inkling” on Hopper or later GPUs. The quantized NVFP4 checkpoint “thinkingmachines/Inkling-NVFP4” works on Blackwell Nvidia GPUs. The latest version of transformers (5.14.0) was released today.

from transformers import pipeline

model_id = "thinkingmachines/Inkling"
# model_id = "thinkingmachines/Inkling-NVFP4"

pipe = pipeline("any-to-any", model=model_id)

After initializing the pipeline, users pass in the prompt as follows.

image_url = (
    "https://huggingface.co/datasets/merve/vl-test-suite/"
    "resolve/main/pills.jpg"
)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image_url,
            },
            {
                "type": "text",
                "text": "Do components in this supplement interact with each other?",
            },
        ],
    },
]
output = pipe(
    messages,
    max_new_tokens=2000,
    return_full_text=False,
    reasoning_effort="medium",
)
output[0]["generated_text"]

Going one level lower, users can use Auto classes. For inference, the AutoModelForMultimodalLM class handles models and the AutoProcessor class handles processors. For different reasoning tasks, the tokenizer takes in a reasoning_effort argument. Existing options for reasoning effort are “none”, “minimal”, “low”, “medium”, “high”, “xhigh”, and “max”.

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You should only answer with a number."},
    {"role": "user", "content": "What is 17 * 23?"},
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    reasoning_effort="high",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=2000)
generated_tokens = output[0][inputs["input_ids"].shape[1] :]
print(processor.decode(generated_tokens, skip_special_tokens=False))

For multimodal inference, users can use the same classes. The model card provides example snippets for each different modality.

Text with image inference

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
)

image_url = (
    "https://huggingface.co/datasets/merve/vl-test-suite/"
    "resolve/main/pills.jpg"
)
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": image_url,
            },
            {
                "type": "text",
                "text": "Do any of the components in this supplement interact?",
            },
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    reasoning_effort="medium",
    return_dict=True,
    return_tensors="pt",
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=2000)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

processor.parse_response(response)

Inkling also takes in audio input. The following inference snippet uses the same AutoModelForMultimodalLM class.

Text with audio inference

from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    dtype="auto",
    device_map="auto",
)

audio_url = (
    "https://huggingface.co/datasets/merve/vl-test-suite/"
    "resolve/main/example_audio.mp3"
)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe the following speech to text."},
            {
                "type": "audio",
                "audio": audio_url,
            },
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=512)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

processor.parse_response(response)

For realistic parallel deployment in a cluster of several nodes, users should refer to the Slurm section below.

SGLang

SGLang is one of the fastest deployment frameworks for Inkling at the time of release, as it includes a custom model implementation. The launch command below shards the model across 8 GPUs and serves an OpenAI-compatible API on port 30000.

pip install sglang

python3 -m sglang.launch_server \
 --model-path thinkingmachine/Inkling \
 --tp-size 8 \
 --served-model-name inkling \
 --host 0.0.0.0 \
 --port 30000

Match –tp-size to your GPU count. Add –mem-fraction-static (e.g. 0.85) if you need to leave more