Skip to content

feat(swiftvr): add real-time restoration pipeline - #29

Merged
lzx1413 merged 3 commits into
mainfrom
swiftvr
Aug 10, 2026
Merged

feat(swiftvr): add real-time restoration pipeline#29
lzx1413 merged 3 commits into
mainfrom
swiftvr

Conversation

@lzx1413

@lzx1413 lzx1413 commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Port the released SwiftVR ReAE and transformer into TeleFuser as a stateful
causal restoration pipeline.

The integration preserves the released checkpoint format, one-step DiT,
causal ReAE state, chunk/flush semantics, and supports direct PIL frame output.
It also adds optional torch.compile, tf-kernel FP8Linear, selectable attention
backends, and three-stage execution with WorkerTensorChannel latent handoff.

Motivation

SwiftVR was not previously available through TeleFuser. This change provides a
production-oriented single-GPU pipeline and FlashVSR-style H100 example while
keeping the default path at least as fast as the released implementation.

The default TeleFuser QHD core path reaches 32.42 FPS versus 31.05 FPS for a
local run of the released implementation under the same checkpoint and setup.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Performance improvement
  • Code refactoring
  • Documentation update
  • Other

Changes Made

  • Ported the released SwiftVR ReAE and Wan transformer into TeleFuser.
  • Added causal stateful chunk processing, tail flushing, RoPE offsets, and
    session isolation.
  • Changed pipeline/session output to List[PIL.Image.Image] with pinned output
    conversion.
  • Added FlashVSR-style H100 offline example and model loading documentation.
  • Added Hugging Face checkpoint download instructions and expected file layout.
  • Added optional attention backend selection and torch.compile kernel fusion.
  • Added LiveAct-compatible tf-kernel FP8Linear W8A8 support.
  • Added three-stage encode/DiT/decode execution with WorkerTensorChannel.
  • Fixed generic SDPA layout copies by avoiding unnecessary contiguous transforms.
  • Added focused model, streaming, stage, example, attention, compile, and
    FP8Linear tests.

Testing

  • Focused unit tests pass
  • Manual H100 testing performed
  • Performance benchmarks updated

Test commands:

.venv/bin/python -m pytest \
  tests/unit/models/test_swiftvr_transformer.py \
  tests/unit/pipelines/swiftvr/test_example.py \
  tests/unit/pipelines/swiftvr/test_streaming.py \
  -q

ruff check telefuser/models/swiftvr_transformer.py \
  tests/unit/models/test_swiftvr_transformer.py \
  examples/swiftvr/swiftvr_restore_h100.py

ruff format --check telefuser/models/swiftvr_transformer.py \
  tests/unit/models/test_swiftvr_transformer.py \
  examples/swiftvr/swiftvr_restore_h100.py

git diff --check

Focused SwiftVR tests: 22 passed.

Manual H100 validation:

  • 81-frame 1080p generation: 34.78 FPS
  • Including H.264 encoding: 22.18 FPS
  • QHD default core path: 32.42 FPS
  • QHD torch.compile: 40.3-40.5 FPS
  • QHD tf-kernel FP8Linear: 35.8-36.1 FPS
  • QHD torch.compile + FP8Linear: 45.2-45.7 FPS
  • Three-GPU stage pipeline: 39.87-40.11 FPS

Checklist

  • Code follows the project's coding standards (ruff)
  • Pre-commit hooks pass (ruff, ruff-format)
  • All tests pass (pytest tests/) — full repository suite not run
  • New tests added for new functionality
  • Documentation updated
  • Commit messages are clear and descriptive
  • PR title follows the project convention:
    feat(swiftvr): add optimized streaming execution

Related Issues

N/A

Additional Notes

The published SwiftVR QHD result is 31.32 FPS on one H100. A local run of the
official implementation measured 31.05 FPS under the same checkpoint.

The recommended acceleration configuration is:

python examples/swiftvr/swiftvr_restore_h100.py \
  --model_root /data/SwiftVR \
  --attn_impl TORCH_SDPA \
  --compile_dit \
  --quantization tf-kernel-fp8

SageAttention remains selectable, but was slower for this workload:
30.8-31.1 FPS eager and 38.1-38.4 FPS with compile.

The first compile shape incurs approximately 44 seconds of one-time compilation
overhead. The stage-parallel result is a pipeline throughput measurement and is
not directly comparable to single-GPU latency or memory.

GPU Architecture Support

  • SM80 (Ampere, Ada Lovelace)
  • SM90 (Hopper H100)
  • SM100+ (Blackwell)

No new CUDA kernel implementation is added in this PR. Existing SageAttention
and tf-kernel FP8Linear backends are used and validated on H100.

Performance Impact

The default path is faster than the released implementation at QHD.

  • Official published: 31.32 FPS
  • Official local probe: 31.05 FPS
  • TeleFuser default: 32.42 FPS
  • TeleFuser compile: 40.3-40.5 FPS
  • TeleFuser compile + FP8Linear: 45.2-45.7 FPS

The FP8 output comparison against BF16 measured 52.24 dB PSNR, 0.00146 MAE,
and 0.015625 maximum absolute error on a QHD test request.

lzx1413 added 2 commits August 9, 2026 16:20
Port the released SwiftVR ReAE and transformer into a causal single-device pipeline with stateful chunk processing and pinned output conversion.

Add a FlashVSR-style H100 example, documentation, and focused model/session tests. The example warms full and tail shapes and sustains 1080p generation on one H100 without keeping the development validation scripts.

Verified with 16 SwiftVR unit tests, Ruff check/format, diff checks, and an 81-frame 1080p H100 run at 34.78 generation FPS and 22.18 FPS including H.264 encoding.
Keep the generic TORCH_SDPA path on transpose views instead of forcing contiguous copies. Add a unit test that verifies the SDPA backend receives transpose views for BSND input.
@lzx1413 lzx1413 mentioned this pull request Aug 10, 2026
31 tasks
Align the default offline path with the upstream causal benchmark and return PIL frames directly from pipeline sessions.\n\nAdd opt-in torch.compile, tf-kernel FP8Linear, selectable attention backends, and three-stage tensor-channel execution. Record H100 parity and acceleration measurements, including FP8 numerical validation.\n\nVerification: pytest tests/unit/models/test_swiftvr_transformer.py tests/unit/pipelines/swiftvr/test_example.py tests/unit/pipelines/swiftvr/test_streaming.py -q (22 passed); ruff check; ruff format --check; git diff --check.
@lzx1413
lzx1413 merged commit c1e1121 into main Aug 10, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant