A task can fail in several places, and because data can't be submitted atomically across the entire system we need to have a mechanism for detecting data associated with failed processing attempts.
I propose that we generate a random identifier every time a worker receives a task, and once the worker has completed processing of the task it submits the random identifier to a list of successful processing attempts. I suggest calling this random identifier the processing identifier. Any data with a processing ID that isn't in the list of successful processing attempt can be discarded and considered failed.
A task can fail in several places, and because data can't be submitted atomically across the entire system we need to have a mechanism for detecting data associated with failed processing attempts.
I propose that we generate a random identifier every time a worker receives a task, and once the worker has completed processing of the task it submits the random identifier to a list of successful processing attempts. I suggest calling this random identifier the processing identifier. Any data with a processing ID that isn't in the list of successful processing attempt can be discarded and considered failed.