Run a vLLM Server on HF Jobs in One Command

You can spin up a model for testing or batch generation using a single command.

If you require a managed, production-ready service instead, that is what Inference Endpoints are for. This guide covers the manual route.

Here is the process end to end.

Prerequisites

• A payment method or a positive prepaid credit balance. Jobs are billed per minute based on hardware usage.
• The huggingface_hub library version 1.20.0 or higher. Install it using this command:

pip install -U "huggingface_hub>=1.20.0"

• Log in locally with hf auth login.

Launch the server

The hf jobs run command is the Hugging Face equivalent of docker run. It uses the official vllm/vllm-openai image. You specify the hardware with --flavor and expose the vLLM port with --expose:

hf jobs run --flavor a10g-large --expose 8000 --timeout 2h \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000

The --expose 8000 flag routes the container port through HF’s public jobs proxy. The command prints the URL where the server is reachable:

✓ Job started
  id: 6a381ca1953ed90bfb947332
  url: https://huggingface.co/jobs/qgallouedec/6a381ca1953ed90bfb947332
Hint: Exposed ports are reachable at (requires an HF token with read access to the job):
  https://6a381ca1953ed90bfb947332--8000.hf.jobs

The string 6a381ca1953ed90bfb947332 is your job ID. Keep track of it, as you will need it later. We will use <job_id> as a placeholder for the rest of this guide.

Allow a couple of minutes for the weights to download and the system to boot. When the logs display Application startup complete, the server is live.

Query it from anywhere

vLLM supports the OpenAI API. Every request requires your HF token as a bearer token. The quickest method is curl:

curl https://<job_id>--8000.hf.jobs/v1/chat/completions \
  -H "Authorization: Bearer $(hf auth token)" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'

This returns standard OpenAI-style JSON. The response content appears in choices[0].message.content as "Hello! How can I assist you today? 😊".

Alternatively, point the OpenAI client at the exposed URL from Python and pass the token as the API key:

from huggingface_hub import get_token
from openai import OpenAI

client = OpenAI(
    base_url="https://<job_id>--8000.hf.jobs/v1",
    api_key=get_token(),
)
resp = client.chat.completions.create(
    model="Qwen/Qwen3-4B",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

Hello! How can I assist you today? 😊

Perform a quick health check before starting work. Run this curl command to list the model:

curl https://<job_id>--8000.hf.jobs/v1/models -H "Authorization: Bearer $(hf auth token)"

🔐 The endpoint is gated, not public. Every request must carry an HF token with read access to the job’s namespace. A plain browser visit will be rejected. The jobs proxy acts as your API gate: access is scoped to you and your organisation. This is fine for private use, but treat the URL with care. Do not share it expecting it to be open, and do not paste your token into untrusted places. If you require finer-grained or public access, set up a proper gateway instead. Alternatively, see the section below on HF Jobs or Inference Endpoints.

Clean up

Jobs are billed per second. Stop the server when you are finished:

hf jobs cancel <job_id>

The --timeout setting is a safety net that will auto-stop the server, but cancelling explicitly is cheaper. An a10g-large instance runs at $1.50/hour. Check hf jobs hardware for the full price list and select the smallest flavour that fits your model.

Going further: bigger models

The same command scales to much larger models. Select a more powerful --flavor and instruct vLLM to shard the model across GPUs using --tensor-parallel-size. For example, to run the 122B Qwen3.5 mixture-of-experts model on 2× H200:

hf jobs run --flavor h200x2 --expose 8000 --timeout 2h \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3.5-122B-A10B \
  --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 \
  --max-model-len 32768 --max-num-seqs 256

The --tensor-parallel-size argument must match the number of GPUs in the flavour. h200x2 requires 2, while h200x8 requires 8. Run hf jobs hardware to see what is available. Give bigger models a longer --timeout because they take longer to download and load. For large models, H200 flavours are usually the best value.

The --max-model-len 32768 --max-num-seqs 256 flags are specific to this model. Qwen3.5-122B uses a hybrid Mamba/attention architecture with a 256K-token default context. This setting does not leave enough memory for vLLM’s default batch settings. Capping the context length and concurrent-sequence count keeps the system within the GPUs’ memory. If a model fails to start with an out-of-memory or cache-block error, dialing these two settings down is the first step to fix it. Everything else, including the exposed URL, the OpenAI client, and token auth, stays exactly the same.

Going further: Chat with it in a UI

Prefer a chat window over curl? A few lines of Gradio code point at the same endpoint. Add --reasoning-parser deepseek_r1 to the vllm serve command so Qwen3’s thinking returns as a separate field (not necessary, but helpful), then run this code locally (you will just need the job ID):

import gradio as gr
from gradio import ChatMessage
from huggingface_hub import get_token
from openai import OpenAI

client = OpenAI(base_url="https://<job_id>--8000.hf.jobs/v1", api_key=get_token())

def chat(message, history):
    messages = [{"role": m["role"], "content": m["content"]} for m in history if not m.get("metadata")]
    messages.append({"role": "user", "content": message})
    stream = client.chat.completions.create(model="Qwen/Qwen3-4B", messages=messages, stream=True)

    thinking, answer = "", ""
    for chunk in stream:
        delta = chunk.choices[0].delta
        thinking += delta.model_extra.get("reasoning", "")
        answer += delta.content or ""
        out = []
        if thinking.strip():
            status = "done" if answer.strip() else "pending"
            out.append(ChatMessage(role="assistant", content=thinking, metadata={"title": "💭 Thinking", "status": status}))
        if answer.strip():
            out.append(ChatMessage(role="assistant", content=answer))
        yield out

gr.ChatInterface(chat).launch()

Run it, open http://127.0.0.1:7860, and chat. Reasoning streams into the collapsible panel, with the answer below.

Going further: SSH into the running server

Need to debug a startup failure, watch GPU memory, or tail logs interactively? You can open a shell straight into the running job. Launch it with --ssh and ensure your public key is registered at huggingface.co/settings/keys:

hf jobs run --flavor a10g-large --expose 8000 --timeout 2h --ssh \
  vllm/vllm-openai:latest \
  vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000

Then connect with the job ID:

hf jobs ssh <job_id>

You are now inside the container. You can run nvidia-smi, inspect the process, or poke at