If you look at a random paper (e.g. transformers), you'll see that there many design decisions that went into this, but no analysis of which of these choices really brought the biggest differences. At most there is a discussion of structural differences to existing work.