feat(vla): integrate LingBot VLA v2 structured serving and benchmarks - #28
Merged
Conversation
Fix the public official-checkpoint loader and bundle pinned RobotWin normalization statistics with public entry and profile coverage. Localize the Qwen3-VL adaptations, move the optional Triton MoE implementation behind the ops/kernel boundary, and keep PyTorch fallbacks for unsupported execution. Pin the upstream LingBot-VLA v2 revision in a preprocessing, velocity, and final-action parity comparator, and remove the tracked temporary source copy. Verification: - .venv-vla/bin/python -m pytest -q tests/unit/models/test_lingbot_vla_v2.py tests/unit/models/test_lingbot_vla_v2_loader.py tests/unit/pipelines/lingbot_vla_v2 (17 passed) - ruff check on new VLA ops, kernel, validation, and tests - ruff check --select F821 on modified VLA model modules - .venv-vla/bin/python -m py_compile on modified VLA modules - git diff --check
Capture LingBot-VLA v2 preprocessing, fixed noise, per-step flow state and velocity, and final canonical actions through the public TeleFuser pipeline. Record checkpoint, processor, input, runtime, shape, and dtype identities alongside each artifact. Harden the parity comparator to require complete artifact contracts, contiguous denoising steps, finite values, matching metadata, and strict or portable tolerances with first-failure reporting. Add CPU regression tests and document the local baseline workflow. Verification: 25 focused VLA tests passed in .venv-vla; ruff check and format checks passed; two real 6B H100 captures matched across 38 comparisons with max_abs=0.0.
Pin the LingBot-VLA v2 reference checkout and add an isolated upstream capture runtime for preprocessing, velocity, and final-action comparison. Commit the official RobotWin normalization statistics, align public image preprocessing and normalization precision, and record attention, MoE, and norm-stat identities in parity artifacts. Keep the upstream Triton MoE path enabled for production while providing a deterministic reference backend for strict cross-process comparison. Verification: .venv-vla/bin/python -m pytest tests/unit/models/test_lingbot_vla_v2_loader.py tests/unit/pipelines/lingbot_vla_v2/test_data.py tests/unit/pipelines/lingbot_vla_v2/test_robot_profile.py tests/unit/validation/test_lingbot_vla_v2_artifacts.py -q (24 passed); ruff check and ruff format --check passed for the changed VLA pipeline, tests, and validation scripts; git diff --cached --check passed.
Make the LingBot VLA v2 integration inference-only by rejecting training forwards, freezing policy parameters, and removing optimizer and training-parallel APIs. Remove action, depth, video, MoE balance, and router loss helpers while preserving checkpoint topology and inference prefix computation. Add regression coverage for the inference-only boundary and removed loader loss helpers. Verification:\n- 28 focused VLA tests passed\n- real LingBot VLA v2 6B checkpoint loaded and inferred on H100\n- strict upstream parity passed 38/38 comparisons with max_abs=0.0\n- targeted Ruff checks and git diff --check passed
Add a LingBot VLA v2-specific FastAPI application with bounded Base64 image validation, one process-local policy replica, serialized inference, readiness reporting, and normalized canonical action responses. Factor the official 6B runtime construction into a reusable VLA helper, share it with the offline CLI, document local .venv-vla startup and request usage, and cover the HTTP contract, validation, lifecycle cleanup, and request serialization. Verification: 33 focused LingBot VLA v2 tests passed; Ruff lint and format checks passed; git diff --check passed; a real 6B single-GPU HTTP request returned a finite 50x55 action chunk.
Add a structured task contract and result path while preserving the existing media task API. Wire LingBot VLA v2 into the native scheduler, pipeline pool, status and metrics APIs, and expose action inference through TFClient. Add the native pipeline entrypoint, service documentation, and coverage for routing, validation, pool passthrough, client encoding, lifecycle cleanup, and JSON result handling. Verification: 168 focused service and VLA tests passed; Ruff check and format check passed; strict pipeline validation reported SAFE; real 6B checkpoint smoke returned a finite 50x55 action chunk.
Keep the LingBot VLA v2 policy resident on its target GPU, warm the fixed-shape inference path before service readiness, and avoid per-request allocator cache eviction. Add a reproducible single-GPU benchmark with latency distributions and memory metrics, and wake idle native-service workers immediately when tasks arrive. Verified with 178 focused VLA and shared-service tests, ruff checks, real H100 benchmarks, HTTP inference, and 38-layer strict repeatability.
Remove trailing whitespace and the extra final blank line from the LingBot VLA v2 model sources so branch-level Git whitespace checks pass after syncing with main. No executable logic is changed. Verification: - 187 service and LingBot VLA v2 tests passed - 3 LingBot VLA v2 model tests passed - focused Ruff check and format check passed - git diff --check passed
Remove the unused V1 Qwen2.5-VL policy, legacy flow-matching implementation, dead resampler variants, demo code, and dormant global monkey-patch entry points. Keep the V2 inference path explicit through a compact FlowMatchingBase, retain checkpoint-compatible Qwen3-VL, action expert, alignment, and MoE modules, and preserve the models-to-ops-to-kernel boundary. Verification: - Ruff check and format check passed for all four model files - Python compilation passed - 190 service and LingBot VLA v2 tests passed - real 6B tensor replay produced 50x55 actions - strict 38-layer regression parity passed with max_abs=0 - git diff --check passed
Add a VLA-specific real HTTP workload for fixed-count, concurrent, and duration validation. Verify the native service contract, task lifecycle, finite 50x55 canonical actions, latency, throughput, and bounded result artifacts without changing shared service interfaces.\n\nDocument single-replica, multi-replica, and soak usage, and add focused CPU coverage for contract validation and concurrent requests.\n\nVerification:\n- ruff check tools/validation/validate_lingbot_vla_v2_structured_service.py tests/unit/validation/test_lingbot_vla_v2_structured_service.py\n- 27 focused pytest tests passed\n- real 6B native HTTP smoke passed\n- 20/20 single-replica 256x256 requests completed successfully
Freeze the LingBot VLA v2 structured request and action-result contracts, reject sensitive camera echoes, and report latency-window trends. Add opt-in process-tree RSS and per-GPU NVML sampling with bounded artifacts, document the validation workflow, and cover contract and sampler boundaries in unit tests. Verification: 33 focused pytest tests; ruff check; ruff format --check; git diff --check.
Add the LingBot VLA v2 native service to the declared-contract coverage registry and validate the run entrypoint name declared by PIPELINE_CONTRACT instead of assuming run_with_file. Verification: 65 VLA, structured service, and registry tests; ruff check; ruff format; git diff --check.
Add a native LingBot VLA v2 AIPerf endpoint and HTTP polling transport with a pinned workload, bounded action summaries, contract assets, and documentation. Add real-service fault validation for malformed requests, cancellation, replica termination, GPU release, and graceful pool capacity degradation. Remove the replica cancel-forwarder completion delay and convert dead-process IPC failures into replica eviction without changing public service contracts. Verification: 93 focused VLA/service tests passed; 153 non-LiveKit service tests passed; 19 AIPerf adapter tests passed; real two-replica structured runs completed 100/100 requests; fault validation passed 5/5; ruff, formatting, config validation, shell syntax, and git diff checks passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integrate LingBot VLA v2 base-model inference into TeleFuser with a native structured action service, regression validation, AIPerf workload support, and request-level replica
resilience.
Included
PIPELINE_CONTRACTTFClientVLA action supportValidation
93 passed: focused LingBot VLA v2, structured service, pipeline pool, and validation tests.153 passed: non-LiveKit native service tests.19 passed: AIPerf adapter tests.50 x 55100/100successful requests2.761 req/s0.723 s0.657 s5/5checks passed.git diff --checkpassed.Scope And Compatibility
Known Limitations
policy_verified=false; output is a normalized canonical50 x 55action chunk, not a physical robot command.