We currently already do this in runtime_build_and_test, we can reuse the
same technique in other workflows to speed them up.