[Bounty $1500] LLVC (Low-Latency Low-Resource Voice Conversion) bring up using TTNN APIs

Bounties
1

Tenstorrent Bounties

Issue ID: I_kwDOI9Wqc87XHw0e

:memo: Background

This bounty is for bringing up the LLVC (Low-Latency Low-Resource Voice Conversion) model using TTNN APIs on Tenstorrent hardware (Wormhole or Blackhole).

LLVC is a real-time voice conversion model from Koe AI specifically optimized for low latency and CPU efficiency. Published in 2023, key features include:

  • Ultra-low latency: Designed for real-time voice conversion with minimal delay
  • CPU-optimized: Efficient enough to run on CPU in real-time
  • Streaming support: True streaming inference with chunked processing
  • High quality: Natural voice conversion while maintaining low latency
  • Compatible architecture: Based on RVC/QuickVC with optimizations
  • Small model size: Low resource requirements for edge deployment
  • No F0 dependency: Optional pitch-independent conversion mode
  • MIT License: Free for commercial use

LLVC achieves real-time factor (RTF) < 0.1 on CPU, making it suitable for live applications like real-time dubbing, streaming, gaming voice chat, and interactive voice applications.

The goal is to enable this model to run on TT hardware for ultra-high-throughput, ultra-low-latency voice conversion optimized for real-time applications.

:bullseye: What Success Looks Like

A successful submission will fulfill all requirements in the following stages. Payout is made after all three stages are completed.

Stage 1 — Bring-Up

  • Implement LLVC using TTNN APIs (Python)
  • Implements the full generation pipeline:
    • Lightweight encoder (optimized for low latency)
    • Streaming-capable architecture
    • Efficient decoder
    • Optional pitch extraction (F0-based and F0-free modes)
    • Vocoder integration
  • Model runs on either N150 or N300 Tenstorrent hardware with no errors
  • Supports both modes:
    • Streaming mode: Real-time conversion with chunked processing
    • Non-streaming mode: Full-context conversion
  • Produces valid converted audio output
  • Output is verifiable (audio quality assessment, compare with PyTorch reference)
  • Achieves baseline throughput target:
    • At least 50 tokens/second for decoder generation
    • Real-time factor (RTF) < 0.3 for streaming mode
    • Latency < 100ms for streaming chunks
  • Accuracy evaluation:
    • Speaker similarity > 70% (cosine similarity)
    • Content preservation: WER < 3.0
    • Token-level accuracy > 95% against PyTorch reference
  • Audio quality: Natural prosody with minimal artifacts despite low latency
  • Clear instructions for setup and running the model

Stage 2 — Basic Optimizations

  • Use optimal sharded/interleaved memory configs for encoder-decoder layers
  • Implement efficient sharding strategy for:
    • Lightweight encoder layers
    • Decoder layers optimized for streaming
    • Convolutional layers with causal padding
    • Cached convolution states for streaming
    • Optional pitch extraction module
  • Fuse simple ops where possible (e.g., layer normalization, activation functions)
  • Store intermediate activations in L1 where beneficial
  • Use recommended TTNN/tt-metal flows for streaming audio models
  • Leverage TT library of fused ops for convolution blocks
  • Optimize chunk-based processing for streaming
  • Efficient state management for causal convolutions
  • Optimize vocoder integration

Stage 3 — Deeper Optimization

  • Maximize core counts used per inference
  • Implement deeper TT-specific optimizations:
    • Ultra-low latency streaming (minimize chunk processing time)
    • Efficient causal convolution with state caching
    • Flash Attention or equivalent where applicable
    • Minimize voice conversion latency per chunk
    • Batch processing for multiple concurrent streams
    • Pipeline encoder/decoder/vocoder stages
    • Optimize memory footprint for edge deployment
  • Minimize memory and TM (tensor manipulation) overheads
  • Explore look-ahead strategies for quality vs latency trade-off
  • Document any advanced tuning, known limitations, or trade-offs
  • Target stretched goals:
    • 100+ tokens/second generation speed
    • RTF < 0.1 for streaming mode (match CPU performance)
    • Latency < 50ms for streaming chunks
    • Support for 10+ concurrent streams

:compass: Guidance & Starting Points

  • Use the TTNN model bring-up tech report as your primary reference
  • Reference streaming audio model patterns in tt-metal for streaming architectures
  • Use the official LLVC repository for model architecture details
  • Refer to the LLVC paper (arXiv:2311.00873) for technical details
  • Check LLVC demo page for audio samples
  • Refer to TT Fused ops PR #29236 for optimization opportunities
  • The model architecture consists of:
    • Lightweight encoder: Optimized convolutional layers with reduced complexity
    • Cached convolutions: State management for causal streaming inference
    • Efficient decoder: Minimal layers for low latency
    • Optional F0 module: Pitch extraction when needed
    • Vocoder: HiFi-GAN or similar (can be shared with other models)
  • Key challenges:
    • Ultra-low latency requirements (< 100ms)
    • Streaming inference with causal convolutions
    • State management across chunks
    • Quality vs latency trade-off
    • Vocoder integration and optimization
    • Small model size constraints
  • Streaming parameters:
    • window_ms: Chunk size in milliseconds (affects latency)
    • extra_convert_size: Extra context for quality (affects latency)
  • Ask for help or file issues if ops are missing in TTNN

:magnifying_glass_tilted_right: Possible Approaches

  • Start from the official LLVC repository and port components sequentially:
    1. Lightweight encoder (optimized convolutions)
    2. Cached convolution layers for streaming
    3. Efficient decoder
    4. Optional pitch extraction (F0 mode)
    5. Vocoder integration
    6. Streaming inference pipeline with state management
  • Validate each submodule’s output against PyTorch reference before integration
  • Experiment with different sharding strategies:
    • Channel sharding for convolutions
    • Sequence length sharding for chunks
    • Minimize cross-chip communication
  • Use TTNN profiling tools to identify bottlenecks in:
    • Per-chunk processing time (critical metric)
    • Encoder latency
    • Decoder latency
    • State management overhead
    • Vocoder latency
  • Test diverse use cases:
    • Streaming mode with various chunk sizes (50ms, 100ms, 200ms)
    • Non-streaming mode for comparison
    • F0-based vs F0-free modes
    • Multiple concurrent streams
    • Real-time conversion scenarios
  • Open a draft PR early to get feedback on your approach

:bar_chart: Result Submission Guidelines

Beyond the model implementation itself, contributors must submit the following material as proof of work. However, feel free to open a PR at any time if you want us checking that you are on the right track. Just understand that payout is only made after all 3 stages are completed.

Deliverables:

  • Functional model implementation
  • Validation logs (output correctness)
  • Performance report + header for final review

Links:

:books: Resources

Model Resources

TT-Metal Resources