Self-hosted UI-TARS-1.5-7B serving ladder for an open-weight agent

In one paragraph

The frontier route (EXP-012) cannot feed training, and the official OSWorld UI-TARS-1.5 agent is the open-weight agent we intend to run on both arms through the thin cua_adapter. That agent needs a served model, and the model must not become a hidden variable between arms, so model_serving puts one private endpoint on 127.0.0.1, serves the pinned checkpoint through the authors' own named TGI container pinned by digest, and climbs down a context ladder (65536, 49152, 32768) until a level passes every measured gate. The floor is 32768 because the frozen agent config needs at least 25,338 tokens (design value, arithmetic from public configs). A 24 GB L4 may not serve the frozen config at any level; the report would then name failed_prereq rather than quantize, drop history or shrink screenshots. Nothing has been served.

Setup

Pins (model_serving/pins.py)

• Model: ByteDance-Seed/UI-TARS-1.5-7B at revision 683d002dd99d8f95104d31e70391a39348857f4e; config.json must say Qwen2_5_VLForConditionalGeneration and bfloat16 with no quantization_config. Per-file hashes are not pinned in the repository (huggingface.co was unreachable from the authoring environment, "so none are invented"); plan records the hub's own file listing for review and download --expect-plan verifies every file against the hub's immutable-revision metadata (LFS sha256, git blob sha1, sizes), refusing a hub that reports a different commit.
• Server: ghcr.io/huggingface/text-generation-inference:3.2.1, always run by digest sha256:5f0caa6661f6b30681ff78704c84c2d7b43f529501fc91a700596cde4151c4e5 (index; amd64 manifest sha256:6a3e0a67...ef5e), read from the public registry. pull fails unless the pulled image carries that digest.
• Authors' settings (from the UI-TARS deploy guide, written for a 48 GB L40S): max input 65536, max batch prefill 65536, max total 65537, CUDA_GRAPHS=0, PAYLOAD_LIMIT=8000000.
• Runtime: local model directory mounted read-only with HF_HUB_OFFLINE=1, --dtype bfloat16, --num-shard 1, --usage-stats off, no --quantize, container port 80 published on 127.0.0.1 only (verified from docker port), served name uitars15-7b.

The ladder

Levels 65536, 49152, 32768, each a fresh container; the largest level that passes everything is chosen and written to serving-lock.json. Input bound is context - 1024 so max_tokens=1000 always fits. Each level must pass, with every claim either measured or reported unverified (an unverified claim fails the level):

• Configuration: /info (or the launcher's printed args in the container log) must give TGI 3.2.1, a bfloat16 dtype and exactly the requested limits; quantize is measured from the log when printed, otherwise recorded as "by construction only".
• No screenshot shrinking, proven from the model's own settings: the expected vision-token count per 1920x1080 image is computed from the downloaded preprocessor_config.json through the Qwen2-VL smart_resize and must equal the pin (2691, else the pins are stale); the endpoint's own prompt_tokens are differenced across text-only, one image and six images; one image must cost at least the expected count and six must cost exactly five more than one.
• Real agent shape: six distinct 1080p screenshots plus text, max_tokens=1000; the model's answer is recorded.
• The input bound is actually exercised: a calibrated filler must land within 5% of the declared bound and not above it; a request that does not reach the bound is a failure.
• Readiness has no fixed cutoff for a healthy start: the wait continues while the container's log keeps growing and fails only after a stall (--ready-stall-s, default 900 s) or a hard cap (default 4 h); per-request timeout default 3600 s (all design values, recorded in the report).

Why 32768 is the floor

The frozen agent config (cua_adapter.config, the official qwen25vl_normal runtime: history_n=5, 1920x1080 screenshots, no shrinking, max_tokens=1000) sends up to six screenshots of about 2691 vision tokens each plus text: at least 25,338 tokens (design value: 6 x 2691 + 8192 + 1000, "rough arithmetic only (not a feasibility proof)"). A lower context would force dropping history images or shrinking screenshots, and neither is ever applied.

Outcome vocabulary

• A passing level: serving-report.json plus serving-lock.json naming the context, the digest, the model manifest hash and every deviation (a context below the authors' 65536 is a recorded deviation, not a silent change).
• failed_prereq: "unquantized BF16 did not serve the frozen config on this GPU at any of the levels; options (a 48 GB GPU like the authors' L40S, a quantized checkpoint, fewer/smaller images) are each a recorded deviation needing an operator decision; none was applied."
• Neither is an agent result: "A passing ladder shows the endpoint can serve the frozen config; the CUA pass rate needs a real task episode and its checker."

What ran

Nothing on a GPU. The only live acts claimed are reading the image digest from the public registry and the operator reporting the GPU (1x NVIDIA L4, 22888 MiB usable).

Results

None. The back-of-envelope figures in SERVING.md (weights ~15 to 17 GiB in BF16, KV cache ~56 KiB per token, ~3.5 GiB at 65536; all estimates) are not results and are not shown as such.

Failures and surprises

None recorded.

Limitations

• The L4 is smaller than the authors' recommended L40S; the ladder exists because feasibility is unknown.
• Per-file model hashes are verified at download time against the hub, not pinned in the repository.
• GPU cost is accounted separately from environment cost; the serving report records GPU wall time only, never a price.
• The OpenAI-compatible route's handling of the model name is assumed; the probe sends uitars15-7b and fails if rejected.

Reproduce

python -m model_serving describe
python -m model_serving environment --model-dir /data/uitars
python -m model_serving pull
python -m model_serving plan --out /data/uitars-plan.json
python -m model_serving download --model-dir /data/uitars --expect-plan /data/uitars-plan.json
python -m model_serving ladder --model-dir /data/uitars --evidence /var/lib/drumsbench/serving-1
python -m model_serving start --model-dir /data/uitars --context <chosen>
DOUBAO_API_URL=http://127.0.0.1:18080/v1 DOUBAO_API_KEY=none-required python -m cua_adapter run ... --serving-lock .../serving-lock.json
python -m model_serving stop

ladder leaves no container behind and writes the report and, on success, the lock, which cua_adapter run --serving-lock verifies against the pins and records in episode.json.

Sources

• benchmark/model_serving/SERVING.md, benchmark/model_serving/pins.py, benchmark/model_serving/probe.py
• benchmark/cua_adapter/pins.py, benchmark/cua_adapter/config.py (the frozen agent config and REQUIRED_CONTEXT)
• benchmark/README.md, sections "UI-TARS agent adapter" and "Self-hosted UI-TARS-1.5-7B serving"
• Commits e611c65, 831c25e, 0e16bc5 (2026-10-25)