[Bounty $2000] RT-DETR + RT-DETRv2 bring up using TTNN APIs

Bounties
1

Tenstorrent Bounties

Issue ID: I_kwDOI9Wqc87SoooR

:memo: Background

This bounty is for bringing up RT-DETR (Real-Time DEtection TRansformer) using TTNN APIs on Tenstorrent hardware (Wormhole or Blackhole). RT-DETR is an end-to-end transformer-based object detector designed for real-time inference. It eliminates non-maximum suppression (NMS) and uses a hybrid CNN–Transformer encoder with dynamic query selection to balance speed and accuracy. Later versions (e.g., RT-DETRv2) introduce bag-of-freebies training improvements and architecture refinements for even higher performance.

The goal is to enable efficient, end-to-end RT-DETR inference on TT hardware for high-throughput, low-latency object detection, showcasing the platform’s suitability for transformer-heavy vision workloads.

:bullseye: What Success Looks Like

A successful submission will fulfill all requirements below. Payout is made after all three stages are completed.

Stage 1 — Bring-Up

  • Implement RT-DETR using TTNN APIs (Python).
  • Runs on N150 or N300 without runtime errors.
  • Produces valid detections (bounding boxes + class labels) on sample images.
  • Output is numerically/verifiably aligned with a PyTorch reference (small validation set).
  • Accuracy evaluated on the COCO dataset, following YOLOv4 demo instructions.
  • Include clear environment setup and run instructions.

Stage 2 — Basic Optimizations

  • Use optimal sharded/interleaved memory configs for convolutional and attention layers.
  • Fuse simple ops where possible (e.g., Linear + Bias + Activation, or conv + activation).
  • Store intermediates in L1 when beneficial.
  • Follow recommended TTNN/tt-metal CNN + Transformer flows.
  • Use TT fused ops like relu with conv.
  • Profile encoder/decoder stages and eliminate redundant Tensor Manipulations (TMs).

Stage 3 — Deeper Optimization

  • Maximize core parallelism across encoder/decoder layers.
  • Implement deeper TT-specific optimizations (e.g., fused QKV + SDPA, operator reordering to reduce TM overheads).
  • Optimize hybrid CNN–Transformer fusion to minimize memory transfers.
  • Document any advanced tuning or known issues.

:compass: Guidance & Starting Points

:magnifying_glass_tilted_right: Possible Approaches

  • Port an existing PyTorch RT-DETR step-by-step (backbone → encoder → decoder → prediction heads).
  • Validate intermediate tensors against CPU/PyTorch reference.
  • Profile encoder attention and multi-scale fusion; apply fusion/reshape optimizations to reduce TM overheads.
  • Add decoder-layer control flags to demonstrate speed vs. accuracy scaling.
  • Use TTNN profiling tools to identify bottlenecks.
  • Open a draft PR early for feedback on approach and performance metrics.

:bar_chart: Result submission guidelines notes

Beyond the model implementation itself. Contributors must submit the following material as a proof of work. However, feel free to open a PR at any time if you want us checking you are on the right track. Just understand that payout is only made after all 3 stages are completed.

  • Functional model implementation
  • Validation logs (output correctness)
  • Performance report + header for final review

Links:

:books: Resources