So far I was mostly lucky that data was already there, but the sample size is limited, of course. Regarding the data format, I‘m a fan of JSONL, one record as json per line. That was how we got the data from Twitter and it‘s a good format, quite UNIX friendly :)