ci: remove cache fallbacks

Remove the cache fallbacks in CircleCI's configuration as they cause
difficult to debug errors and provide little safety as compared to
their execution costs to populate.