# Asynchronous scraping: polling, retries, and idempotency

Handle RunHarvester requests from submission to retained result. Distinguish network retries, failed jobs, incomplete results, and result expiry.

Canonical: https://runharvester.com/guides/async-scraping-retries

By: [RunHarvester](https://runharvester.com/about-us)

Updated: 2026-10-03

## Persist the identity before retrying

Generate an Idempotency-Key before the first POST and persist it with the exact body. If the connection fails after submission, retry that same operation with the same key and body. A new intended collection needs a new key.

Submitting a request normally returns an accepted response and request ID. It does not mean that collection has finished. Store the ID as the durable link between your queue and RunHarvester.

```
POST /api/v1/data/requests
GET /api/v1/data/requests/{id}
GET /api/v1/data/requests/{id}/result
```

## Poll with a bounded policy

Pending, running, and solving_captcha represent active states. Poll using exponential backoff with jitter, adhere to server retry guidance if available, and enforce an application deadline. Halt polling when the status becomes completed, failed, or cancelled.

Retrieve a result only after completion. RESULT_NOT_READY is a state conflict, not extracted data. RESULT_EXPIRED means the retention window has passed; repeated downloads cannot recover it.

- [HTTP statuses and request states](https://docs.runharvester.com/getting-started/first-request)

## A failed-job retry is new work

Review the public error and trace before attempting to rerun a failed job. Address invalid inputs and unsupported extraction fields first; blind retries squander capacity. The retry operation assigns a new request ID while retaining the original failed request for diagnosis.

Keep the source and retry IDs together. Bound attempts and total cost, and reconcile returned billing metadata rather than assuming every failure is free or every cancellation refunds immediately.

1. Retry transient transport failures with the original submission identity.
2. Treat failed-job retry as a new task and track its new identifier.
3. Download and validate successful results before their reported expiry.
4. Send only non-sensitive IDs and public errors to support.

## How long can I retrieve a result?

Data retention is governed by your specific service and plan settings. Make sure to review the expiry details provided and fetch any necessary results prior to expiration. If a Data API result expires, it will return RESULT_EXPIRED. Ensure you export necessary data to your personal storage instead of relying on retained results as a permanent archive.

[Documentation](https://docs.runharvester.com/getting-started/first-request)

## How do I retry without creating duplicate work?

Generate an Idempotency-Key before submission and store it with the exact request body. Reuse it when retrying the same submission after a network interruption. A new logical request requires a new key. Retrying a failed request creates a new request ID; preserve the original for diagnosis.

[Documentation](https://docs.runharvester.com/concepts/retries)

## What happens to credits after a failure or cancellation?

Requests that fail validation incur no fees. If a request fails completely, base debits are refunded; the final credit adjustment depends on the applicable billing rules. Cancellations in pending or running states may resolve at varying times. Cross-reference billing metadata and the returned token state to match up charges.

[Documentation](https://docs.runharvester.com/concepts/billing)

## Continue reading

- [Web scraping API](https://runharvester.com/api)

- [Browser Workflows](https://runharvester.com/browser-workflows)

- [FAQ](https://runharvester.com/faq)

[Source: RunHarvester documentation](https://docs.runharvester.com/concepts/retries)