flip_api.private_services.stale_task_recovery

Scheduled job to recover tasks stuck in IN_PROGRESS state.

If a trust picks up a task but crashes before reporting the result, the task remains IN_PROGRESS forever. This module periodically resets such stale tasks back to PENDING so they can be re-dispatched.

Functions

`recover_stale_tasks`(→ int)	Reset stale IN_PROGRESS tasks back to PENDING, or mark them FAILED if retries are exhausted.
`retry_failed_post_processing`(→ int)	Retry post-processing for completed tasks that still need it.
`recover_stale_tasks_scheduled_task`(→ None)	Scheduled task entry point for stale task recovery and post-processing retry.

Module Contents

flip_api.private_services.stale_task_recovery.recover_stale_tasks(db: sqlmodel.Session) → int

Reset stale IN_PROGRESS tasks back to PENDING, or mark them FAILED if retries are exhausted.

A task is considered stale if it has been IN_PROGRESS for longer than TASK_STALE_TIMEOUT_MINUTES without a result being reported.

Tasks that have already been retried TASK_MAX_RETRIES times are marked as FAILED instead of being re-queued, preventing poison tasks from looping indefinitely.

Parameters:: db (Session) – Database session.
Returns:: Number of tasks recovered (re-queued or failed).
Return type:: int

flip_api.private_services.stale_task_recovery.retry_failed_post_processing(db: sqlmodel.Session) → int

Retry post-processing for completed tasks that still need it.

If a CREATE_IMAGING task completed but its post-processing (status persistence and email notifications) failed, this retries it.

Parameters:: db (Session) – Database session.
Returns:: Number of tasks retried.
Return type:: int

flip_api.private_services.stale_task_recovery.recover_stale_tasks_scheduled_task() → None: Scheduled task entry point for stale task recovery and post-processing retry.