VPS for web scraping: what matters for data collection
A scraper needs a server that runs around the clock, doesn't depend on your home internet connection and doesn't hit a traffic cap. But the right configuration depends heavily on how exactly you collect data. Before renting a VPS, work out which of two types your scraper belongs to.
HTTP scraper or headless browser
Scrapers built on direct requests — Scrapy, aiohttp, httpx, Go clients — download HTML or JSON and parse it. Most of the time they're waiting on the network, so a single core can handle hundreds of concurrent async requests, and 1–2 GB of memory is enough. The CPU load comes from parsing: lxml and regular expressions over thousands of pages a minute will noticeably heat up the CPU. For scripts like these a regular VPS for Python will do.
Headless browsers — Playwright, Puppeteer, Selenium — are needed where content is rendered by JavaScript. Every Chromium tab eats hundreds of megabytes of memory and a sizeable share of a core. Size your resources by the number of concurrent tabs: 2 cores and 4 GB can reasonably run a few, while dozens call for 8–16 GB and a matching number of cores.
When running Chromium in Docker, increase /dev/shm (the --shm-size option) — with the default 64 MB, tabs crash on heavy pages.
Traffic and port speed
Scraping is mostly inbound traffic, and its volume is easy to underestimate: a page with images and scripts weighs megabytes, and you fetch hundreds of thousands of them a day. Choose a plan with unmetered traffic — this filter is already applied on this page, and similar plans are collected in the unmetered traffic VPS section. Check the port speed too: "unlimited" at 100 Mbps and at 1 Gbps are very different things. In headless mode, turn off loading of images, fonts and video if you don't need them.
IP addresses
One server means one IPv4 address, and with active collection sites quickly start rate-limiting it. Hosts charge extra for additional IPv4 addresses and don't always provide them. Many plans include an IPv6 /64 subnet, but you can only use it on sites that accept IPv6 — see the IPv6 VPS selection. You can also run your own proxies on your server; see the VPS for a proxy server page.
Pick the location based on your target sites: many sites serve region-specific content, so a server in the same country sees the same pages as local visitors.
What not to do
- Don't hammer other people's sites until they go down: keep reasonable delays and follow the site's rules. Site owners send complaints to your host, and the host may shut your server down.
- Don't collect personal data without a legal basis — that carries separate legal liability.
- Don't keep results only on the server. Export them to a database or storage and make backups.
Prices in the table above are already shown in your currency, and the "Payment" column shows which methods each host accepts.