flip_api.private_services.stale_task_recovery ============================================= .. py:module:: flip_api.private_services.stale_task_recovery .. autoapi-nested-parse:: Scheduled job to recover tasks stuck in IN_PROGRESS state. If a trust picks up a task but crashes before reporting the result, the task remains IN_PROGRESS forever. This module periodically resets such stale tasks back to PENDING so they can be re-dispatched. Functions --------- .. autoapisummary:: flip_api.private_services.stale_task_recovery.recover_stale_tasks flip_api.private_services.stale_task_recovery.retry_failed_post_processing flip_api.private_services.stale_task_recovery.recover_stale_tasks_scheduled_task Module Contents --------------- .. py:function:: recover_stale_tasks(db: sqlmodel.Session) -> int Reset stale IN_PROGRESS tasks back to PENDING, or mark them FAILED if retries are exhausted. A task is considered stale if it has been IN_PROGRESS for longer than ``TASK_STALE_TIMEOUT_MINUTES`` without a result being reported. Tasks that have already been retried ``TASK_MAX_RETRIES`` times are marked as FAILED instead of being re-queued, preventing poison tasks from looping indefinitely. :param db: Database session. :type db: Session :returns: Number of tasks recovered (re-queued or failed). :rtype: int .. py:function:: retry_failed_post_processing(db: sqlmodel.Session) -> int Retry post-processing for completed tasks that still need it. If a CREATE_IMAGING task completed but its post-processing (status persistence and email notifications) failed, this retries it. :param db: Database session. :type db: Session :returns: Number of tasks retried. :rtype: int .. py:function:: recover_stale_tasks_scheduled_task() -> None Scheduled task entry point for stale task recovery and post-processing retry.