1
Trying to figure out how long you need to train on an RTX 3090 to get 10^26 floating point ops. 🤔
Back to archive
Thread
4 tweets
2
I still haven't made the math but isn't any cloud data center immediately above the threshold? Does the US administration know how cloud computing works?!
3
OK, let's do the math. According to en.wikipedia.org/wiki/GeForce_4…, the RTX 4090 has 73 TFLOPS (single precision), tera is 1e12 so that's 73e12 floating ops per second. A day has 86400 seconds, so that's 6.3e18 FLOPD. To get to 1e26, you'd need to run it for 43438 years.
GeForce RTX 40 series - Wikipedia
en.wikipedia.org
4
According to this medium.com/codex/gpt-4-wi…, GPT-3 was trained on 1024 GPUs for 34 days, even if those were 4090s, that'd be 2.2e23 floating ops, so yeah, still off by a factor of 500, but not much!
medium.com
· Access denied