Persist the identity before retrying
Generate an Idempotency-Key before the first POST and persist it with the exact body. If the connection fails after submission, retry that same operation with the same key and body. A new intended collection needs a new key.
Submitting a request normally returns an accepted response and request ID. It does not mean that collection has finished. Store the ID as the durable link between your queue and RunHarvester.
POST /api/v1/data/requests
GET /api/v1/data/requests/{id}
GET /api/v1/data/requests/{id}/resultPoll with a bounded policy
Pending, running, and solving_captcha represent active states. Poll using exponential backoff with jitter, adhere to server retry guidance if available, and enforce an application deadline. Halt polling when the status becomes completed, failed, or cancelled.
Retrieve a result only after completion. RESULT_NOT_READY is a state conflict, not extracted data. RESULT_EXPIRED means the retention window has passed; repeated downloads cannot recover it.
A failed-job retry is new work
Review the public error and trace before attempting to rerun a failed job. Address invalid inputs and unsupported extraction fields first; blind retries squander capacity. The retry operation assigns a new request ID while retaining the original failed request for diagnosis.
Keep the source and retry IDs together. Bound attempts and total cost, and reconcile returned billing metadata rather than assuming every failure is free or every cancellation refunds immediately.
- Retry transient transport failures with the original submission identity.
- Treat failed-job retry as a new task and track its new identifier.
- Download and validate successful results before their reported expiry.
- Send only non-sensitive IDs and public errors to support.