F0 curve manipulation: Optional external F0 curve file support
High quality: Natural-sounding voice conversion with artifact prevention
MIT License: Free for commercial use
Full deployment stack: Library, CLI, and API support with Docker
RVC enables high-quality voice conversion by combining VITS architecture with retrieval-based feature matching, making it suitable for singing voice conversion, dubbing, voice cloning, and personalized voice applications.
The goal is to enable this model to run on TT hardware for high-throughput, low-latency voice conversion across diverse use cases.
What Success Looks Like
A successful submission will fulfill all requirements in the following stages. Payout is made after all three stages are completed.
Stage 1 — Bring-Up
Implement RVC using TTNN APIs (Python)
Implements the full generation pipeline:
VITS encoder (posterior encoder)
Pitch extraction module (rmvpe or crepe)
Feature retrieval module (index-based)
Flow-based decoder
HiFi-GAN vocoder
Model runs on either N150 or N300 Tenstorrent hardware with no errors
Produces valid converted audio output
Output is verifiable (audio quality assessment, compare with PyTorch reference)
Achieves baseline throughput target:
At least 30 tokens/second for flow generation
Real-time factor (RTF) < 0.5 for typical utterances
Accuracy evaluation:
Speaker similarity > 75% (cosine similarity of speaker embeddings)
Content preservation: WER < 2.5
Token-level accuracy > 95% against PyTorch reference
Audio quality: Natural prosody, accurate pitch conversion, minimal artifacts
Clear instructions for setup and running the model
Stage 2 — Basic Optimizations
Use optimal sharded/interleaved memory configs for encoder-decoder layers
Implement efficient sharding strategy for:
VITS posterior encoder
Flow-based decoder layers
Pitch extraction module (rmvpe/crepe)
Feature retrieval index
HiFi-GAN vocoder layers
Fuse simple ops where possible (e.g., layer normalization, activation functions)
Store intermediate activations in L1 where beneficial
Use recommended TTNN/tt-metal flows for audio models
Leverage TT library of fused ops for convolution and attention blocks
Optimize feature index retrieval operations
Optimize pitch extraction and F0 manipulation
Optimize HiFi-GAN vocoder integration
Stage 3 — Deeper Optimization
Maximize core counts used per inference
Implement deeper TT-specific optimizations:
Efficient flow-based decoder computation
Flash Attention or equivalent for attention layers
Minimize voice conversion latency
Batch processing for multiple voice conversions
Optimize feature index search (retrieval operations)
Pipeline encoder/decoder/vocoder stages
Efficient pitch extraction and transposition
Minimize memory and TM (tensor manipulation) overheads
Explore caching strategies for feature indices
Document any advanced tuning, known limitations, or trade-offs
Target stretched goals:
60+ tokens/second generation speed
RTF < 0.2 for real-time applications
Support for batch processing (5+ concurrent conversions)
Validate each submodule’s output against PyTorch reference before integration
Use TTNN profiling tools to identify bottlenecks in:
Encoder processing time
Feature retrieval latency
Pitch extraction overhead
Flow decoder computation
Vocoder latency
Test diverse use cases:
Male-to-female and female-to-male conversion
Singing voice conversion
Different pitch transpositions (-12 to +12 semitones)
Various index rates (0.0 to 1.0)
Real-time conversion requirements
Open a draft PR early to get feedback on your approach
Result Submission Guidelines
Beyond the model implementation itself, contributors must submit the following material as proof of work.
However, feel free to open a PR at any time if you want us checking that you are on the right track.
Just understand that payout is only made after all 3 stages are completed.
Background
What Success Looks Like
Guidance & Starting Points
Possible Approaches
Result Submission Guidelines
Resources