Resources / Field guide

HTTP vs browser scraping: choose by the data you need

A practical method for selecting HTTP, browser, or auto acquisition: compare returned content, wait conditions, routing, and effective cost.

By · Updated

Start with the required output

Write down the fields or page content your application needs. Inspect the initial HTML: if it already contains those values, HTTP may be sufficient. If the values appear only after page execution or interaction, test browser mode.

JavaScript on a website is not itself a reason to use a browser. Some pages ship data in HTML or embedded JSON. Conversely, an HTTP success response can contain only an application shell with none of the requested records.

  1. Choose a small sample including typical, empty, and changed pages.
  2. Fetch the sample using HTTP and inspect the returned payload.
  3. For missing dynamic content, test browser mode with supported waits.
  4. Compare completeness, public errors, latency, and returned charge metadata.

Wait for meaningful content

With a dynamic catalog, monitoring a visible product container is typically more pertinent than tracking a generic page-load event. Apply the waits defined in the request contract and set a timeout. Be aware that a page may maintain background connections even once the content is delivered.

Check for empty-state and challenge pages as well as successful records. Record why a request failed so retry policy can distinguish transient failures from an invalid URL, schema, or access requirement.

Separate execution from egress

HTTP and browser describe acquisition behavior; residential options change the egress route. Start with auto when you want the service to select a suitable route, then use measurements to justify a more specific mode.

Do not compare modes solely by the base price. A route that returns unusable content creates downstream rework. Use billing metadata, valid-result counts, and a consistent sample when assessing a workflow.

A few useful answers

Common questions

When should I use HTTP, browser, or auto mode?

Choose HTTP if the required content appears in the initial response. Opt for browser mode when JavaScript execution or page interaction is necessary. Auto permits the service to select a route. Compare actual output and billing metadata on representative pages before committing to a fixed mode.

Read the documentation
Can RunHarvester scrape JavaScript-rendered pages?

Browser mode runs the page within a browser environment. Set up supported wait conditions for the desired content and verify the outcome. Rendering a page doesn't ensure all lazy-loaded elements are visible or that the target service accepted the request.

Read the documentation

Put it to work

Start with one real request.

Inspect the result. Refine your workflow. Choose a plan for your workload.