Reverts "Fix memory leak in `TabPageSelector` (#147403)" (#147622)

Reverts: flutter/flutter#147403
Initiated by: chingjun
Reason for reverting: Causing an internal test to fail, see b/338159496 for details
Original PR Author: ValentinVignal

Reviewed By: {polina-c}

This change reverts the following previous change:
Part of https://github.com/flutter/flutter/issues/141198