Thread

OK, as much as I mistrust all the Microsoft and OpenAI papers on arxiv that I consider "science PR", this one looks like an interesting read: arxiv.org/pdf/2303.12712… (arxiv seems to be a bit slow right now, I guess they're having their slashdot moment... .)

Interestingly, I think what the current LLM approach has achieved is something that always seemed elusive, and that is to capture and extract "common sense." en.wikipedia.org/wiki/Cyc has been a symbolic AI project that attempted to do the same, through manually extracting rules.

Started in 1984, it is still ongoing for almost 40 years. Painstakingly trying to distill "common sense" so that machines can reason about reality. I see parallels to the way ML has completely changed machine translation here.

Machine translation used to be the hunt for an universal grammar and parsers/generators as intermediary representation to translate between languages. As you can image, given all the quirks of natural language, this was difficult.

Google was one of the first to use their massive treasure trove of bilingual data to shortcut all that manual work by exploiting statistics at scale. research.google/pubs/pub35179/

It seems we've achieved the same thing using large language models. Essentially by learning the sequential structure of data, we have somehow created a machine that generates that intermediary representation, that we don't understand but that seems to be good enough.

And at the same time, we extracted "common sense" that previously took decades to assemble manually. This is remarkable, although there are still more challenges to take. But that's for another thread :)