TTNN currently has two matmul APIs, ttnn.matmul and ttnn.experimental.minimal_matmul. Additionally, these ops have various configuration parameters.
For multi-device, additional CCLs like all_gather or reduce_scatter are needed.
You must create some infrastructure that given a particular input provides a configuration that is guranteed to be the most performant configuration. It may also give suggestions to the user that a different layout or memory configuration for the input would be more performant, if possible.
This infrastructure would be highly depdenent on the exact implementation for the matmul. So it must be re-usable & modular enough to be re-trained if the underlying code changes.
It would be great if this infrastructure could also be re-used for other ops.
What Success Looks Like
A successful submission will fulfill all requirements in the following stages. Payout is made after all stages are completed.
An API that looks exactly like torch.matmul, which accepts both single-device & multi-device input tensors. The API can decide how to handle the weights, by accepting host weights, & calls the most optimal matmul config.
Tests that show that the API always chooses valid configurations, without any errors.
Tests that show that the chosen configuration is the most performant. This is done by modifying the chosen configuration and showing that performance decreases.
Usage of the created API in any model & showcasing performance improvement.
Code on how to update the infrastructure when the underlying implementations change.
Stage 1 — Acceptance
A proposal on how this infra will work & how it will integreate into the existing code base. This is necessary for being assigned the work.
Proposal should mention the algorithm used & how it will address all the requirements mentioned above.
Stage 2 — Mid way Review
When you have something that works, but not production ready.
Stage 3 — Final Review
When the predictions are perfect, tests are implemented & at least one model has shown performance improvement using this API.
Guidance & Starting Points
Contact @sankarmanoj-tt
Possible Approaches
Usage of a relatively simple Deep Neural Network
Result Submission Guidelines
Beyond the model implementation itself, contributors must submit the following material as proof of work.
However, feel free to open a PR at any time if you want us checking that you are on the right track.
Just understand that payout is only made after all 3 stages are completed.
Deliverables:
API for most optimal matmul
Functional tests showing that the API works with correct outputs for any input
Background
What Success Looks Like
Guidance & Starting Points
Possible Approaches
Result Submission Guidelines