Scraping sites that fight back: Cloudflare, dynamic loading and empty HTML

The failure looks the same every time. You send a request, get back a 200 status code, and the HTML body is nearly empty. No listings, no prices, no images, just a shell of divs and a loading spinner that never resolves because nothing is actually loading anything, it's just markup waiting for JavaScript that never ran. This is where most agency builders lose a day or two on a scraping project, because the fix isn't one thing. It's a sequence of questions, and you have to work through them in order.

First question: is the content actually rendered client-side, or is the site blocking you outright? These look identical from the outside but need completely different fixes. A builder working on a client project to pull property listings, images, price, capacity, and location off a target site hit exactly this wall. The obvious tools failed for a more specific reason: they didn't support the search parameters the site needed passed through the URL. The fix wasn't a bigger scraper, it was smaller: build the URL by hand using the parameter that actually worked (road name, in that case, since postcode and suburb weren't supported), start with a single page working end to end, then expand once that's proven.

When the block is Cloudflare, not JavaScript

If you're getting a challenge page, a CAPTCHA, or a connection that just hangs, that's Cloudflare or a similar bot-detection layer, and no amount of regex or retry logic gets past it reliably. This is the point where a lot of people burn hours trying to fix it with better headers or rotating user agents. Don't. Reach for a scraping platform built to handle it. Apify comes up constantly in this exact spot, precisely because it maintains actors that already solve the Cloudflare problem for specific sites, rather than you rebuilding that solve yourself.

One builder trying to pull North Vancouver home listings off Zillow with raw HTTP requests and regex was fighting both problems at once, dynamic content and anti-bot defenses, and the regex kept breaking on page structure changes. Switching to an existing Zillow actor on Apify solved it in an afternoon instead of another week of patching.

Separate collection from extraction

Once you've got a tool that can actually reach the content, the next mistake is building one workflow that scrapes and parses and analyzes all in a single pass. When that pass fails partway through, on a rate limit, a layout change, a timeout, you lose the whole run and have no idea what you already had. Split it. Scrape and land the raw data first. Analyze or transform it in a separate step that reads from what you already collected. This came up again and again across different projects, an Instagram lead-gen build and a YouTube analytics build both got more reliable once collection and analysis were pulled apart into their own stages, each one retryable on its own.

Where the real cost sits

Apify's pricing runs per-thousand-results rather than per-scrape, typically somewhere between one and three dollars per thousand depending on the actor, and a low-cost monthly plan often nets a much larger usable credit balance than the sticker price suggests. That matters because scraping a full site catalogue, not just a sample page, is where projects blow their budget if nobody checked pricing before building.

Think about volume before you build anything. A test run of ten pages tells you nothing about what a nightly scrape of ten thousand listings will cost, or how the site's own rate limiting behaves at scale.

And one more thing worth deciding early: never scrape a site you don't need to. If the data owner will hand it to you directly, or expose an API, or even accept an upload, that's a maintenance burden you just avoided permanently. Scraped sites change their layout without warning, and every layout change is an outage in your pipeline until someone notices and fixes it.