This bounty is for bringing up facebook/sam2-hiera-tiny using TTNN APIs on Tenstorrent hardware (Wormhole or Blackhole).
SAM 2 is Meta’s foundation model for promptable visual segmentation (paper).
The goal is to enable this model to run on TT hardware in image mode — Hiera image encoder, prompt encoder, and two-way transformer mask decoder. Video tracking (memory encoder/attention/bank) is out of scope
What Success Looks Like
Implement the full image-mode pipeline in TTNN: image encoder, prompt encoder, and mask decoder
Support point, box, and mask prompts, matching the HF reference numerically (PCC)
Measure and report throughput/latency on device, performance repport
Stage 1 — Bring-Up
Implement sam2-hiera-tiny (image mode, 1024×1024 input) using TTNN APIs (Python)
Model runs on N150 or N300 with no errors
Produces valid masks on sample images; verify against the HF reference
Stage 2 — Basic Optimizations
Use optimal sharded/interleaved memory configs across encoder and decoder
Efficient sharding for patch embedding and transformer blocks
Fuse simple ops where possible
Store intermediate activations in L1 where beneficial
Stage 3 — Deeper Optimization
Maximize core utilization per inference
Minimize memory and tensor manipulation overheads
Guidance & Starting Points
Possible Approaches
Start from the HuggingFace Sam2Model / PyTorch reference and port layers one by one to TTNN, validating each submodule’s output against the reference before full integration.
Use TTNN profiling tools to identify bottlenecks and fusion opportunities.
Result Submission Guidelines
Beyond the model implementation itself, contributors must submit the following material as proof of work.
However, feel free to open a PR at any time if you want us checking that you are on the right track.
Just understand that payout is only made after all 3 stages are completed.
Background
What Success Looks Like
Guidance & Starting Points
Possible Approaches
Result Submission Guidelines
Resources