Yes, I‘ve noticed that as well. E.g. for recommendations one has to think about how to evaluate offline because one cannot just go back in time and show other results. But it pays off having some offline proxy metric. A/B tests take way too if you are still searching.