@mikiobraun: "Mat-Mat-Mult is O(n^3) if done naively on O(n^2) data that a..."

Back to 2023

Mikio Braun

@mikiobraun

Part of a thread

Mat-Mat-Mult is O(n^3) if done naively on O(n^2) data that already hints there is some potential for data re-use. Now the funny thing is that even if the matrices are too big for the cache, you can still re-order the computation to make it fit into the cache.

Aug 16, 2023 · 05:36

Full thread (5 tweets) →

Yeah, matrix operations are very interesting from a computational point of view. With LLMs it‘s also...

Aug 16, 2023 · 05:15

Mat-Mat-Mult is O(n^3) if done naively on O(n^2) data that already hints there is some potential for...

Aug 16, 2023 · 05:36

That way, even if your matrices are in the GBs, you can fully saturate whatever computing pipeline y...

Aug 16, 2023 · 05:37

I know that asymptotically, the exponent is more something like 2.37... https://t.co/Q8nYEyGmnA The ...

Aug 16, 2023 · 05:39

This is a really fascinating topic. I remember that GotoBLAS, an implementation in handwritten assem...

Aug 16, 2023 · 05:44