Allow recurrences to affect the custom date field, or set up an independent custom date field recurrence so that the custom date field doesn't need to be reset each time a task recurrence triggers.