Web Scraping Legality: The Short Answer
Web scraping is not inherently illegal, but it is not unconditionally legal either. Its legality depends on four key factors: the jurisdiction you and the target website operate in, the type of data you collect (personal vs. non-personal), the method you use to access it (public pages vs. authenticated sessions), and your purpose for collecting it (commercial research vs. identity theft).
The common belief that "if it's public, I can take it" is FALSE under GDPR and increasingly challenged worldwide. Public availability of data does not automatically grant you the right to collect, store, and process it. This distinction is critical and has been reinforced by regulators and courts repeatedly since 2022.
The Four Pillars of Scraping Legality
Generally Permissible
- Scraping publicly available non-personal data (prices, product specs, weather)
- Academic and journalistic research on public data
- Competitive price monitoring of product listings
- Search engine indexing (established legal precedent)
High Legal Risk
- Scraping personal data without a GDPR lawful basis
- Bypassing authentication or technical access controls
- Mass collection of biometric data (faces, fingerprints)
- Overloading servers (potential DoS / CFAA violation)
The legal landscape continues to evolve. New regulations, court decisions, and enforcement actions emerge regularly. What follows is a detailed breakdown of the two most impactful regulatory frameworks (GDPR and CCPA), the court cases that define current precedent, and practical guidance for staying compliant.
GDPR & Web Scraping (EU/UK)
The General Data Protection Regulation (GDPR) is the most comprehensive data protection framework affecting web scraping worldwide. It applies to any organization processing personal data of individuals in the EU or UK, regardless of where the organization is based. If you scrape data that includes information about EU/UK residents, GDPR applies to you.
Key GDPR Concepts for Scrapers
Personal Data is Broadly Defined
Under GDPR, personal data includes any information relating to an identifiable natural person. This includes names, email addresses, phone numbers, usernames, photos, and critically, IP addresses are classified as personal data (PII) under GDPR. If your scraping activity collects any of these data points, even incidentally, GDPR applies.
You Need a Lawful Basis (Article 6)
GDPR requires one of six lawful bases for processing personal data: consent, contract, legal obligation, vital interests, public task, or legitimate interest. For scraping, the most commonly invoked basis is legitimate interest (Article 6(1)(f)), but this requires a balancing test against the rights of data subjects.
Public Availability Does Not Equal Consent
The fact that someone posted information on a public webpage does not mean they consented to having it scraped, aggregated, and processed. GDPR Article 9(2)(e) mentions data "manifestly made public," but this applies only to special category data and has been interpreted narrowly by regulators. The KASPR case made this point emphatically.
Right to Erasure Applies
Data subjects have the right to request deletion of their personal data (Article 17). If you scrape personal data, you must have mechanisms to identify and delete specific individuals' data upon request. This has significant technical implications for scraping operations that store large datasets.
Case Study: KASPR Fined EUR 240,000
In 2022, the French data protection authority (CNIL) fined KASPR EUR 240,000 for scraping LinkedIn contact data including phone numbers and email addresses. KASPR argued that the data was publicly available on LinkedIn profiles. CNIL rejected this argument, ruling that LinkedIn users had not "manifestly made public" their contact details in the GDPR sense, and that KASPR's legitimate interest did not outweigh the data subjects' rights to privacy. This case is a landmark warning for any business scraping professional social media profiles.
The practical implication for scraping teams is clear: if your scraping involves any personal data of EU/UK residents, you need to conduct a legitimate interest assessment, implement data minimization, provide transparency notices where feasible, and build systems to handle subject access and erasure requests. Ignoring these obligations creates exposure to fines of up to EUR 20 million or 4% of global annual turnover.
CCPA & Web Scraping (California/US)
The California Consumer Privacy Act (CCPA), as amended by the CPRA (effective January 2023), is the most significant state-level data protection law in the United States. While less comprehensive than GDPR, it creates important obligations for businesses that scrape data involving California residents.
CCPA Key Provisions for Scrapers
Broad Definition of Personal Information
CCPA defines "personal information" broadly to include identifiers, commercial information, internet activity, geolocation data, and inferences drawn from other data. IP addresses, browsing history, and search queries all qualify.
Right to Know and Delete
California consumers can request to know what personal information a business has collected about them and request its deletion. Businesses must respond within 45 days.
Right to Opt Out of Sale/Sharing
Consumers have the right to opt out of the "sale" or "sharing" of their personal information. Under CPRA, "sharing" includes sharing for cross-context behavioral advertising, even without monetary exchange.
Publicly Available Information Exemption
CCPA exempts "publicly available" information from government records. However, this exemption is narrow: it does not cover all publicly accessible data on the internet, only data from government sources.
Unlike GDPR, the CCPA does not require an explicit lawful basis before collecting personal information. Instead, it focuses on transparency (disclosure at or before collection), consumer rights (access, deletion, opt-out), and accountability. For scraping operations, this means you must disclose your data collection practices in your privacy policy and honor opt-out requests.
CCPA vs. GDPR: Key Differences for Scrapers
| Aspect | GDPR | CCPA/CPRA |
|---|---|---|
| Scope | EU/UK residents' data, any processor | CA residents, businesses meeting thresholds |
| Legal basis required? | Yes (6 lawful bases) | No (disclosure-based model) |
| IP as personal data? | Yes, explicitly | Yes (identifier) |
| Max penalty | EUR 20M or 4% revenue | $7,500 per intentional violation |
| Public data exemption | Very narrow | Gov records only |
Beyond CCPA, other US states are enacting their own privacy laws. Virginia (VCDPA), Colorado (CPA), Connecticut (CTDPA), Utah (UCPA), and over a dozen more states have passed or are considering data protection legislation. While there is no federal comprehensive privacy law yet, the trend toward state-level regulation means US-based scrapers must track an increasingly complex patchwork of requirements. The CFAA (Computer Fraud and Abuse Act) remains the primary federal statute relevant to scraping, which we cover in the legal precedents section below.
Key Legal Precedents
Court decisions and regulatory enforcement actions shape the practical boundaries of web scraping more than statutes alone. The following cases are essential knowledge for any team engaged in data collection at scale. The CFAA (Computer Fraud and Abuse Act), enacted in 1986, remains the primary US federal statute used to challenge web scraping. Originally designed to combat computer hacking, its application to web scraping has been debated extensively in courts.
hiQ Labs v. LinkedIn
2022Pro-scrapingUS Supreme Court (remanded)
The Supreme Court vacated the Ninth Circuit ruling and remanded the case for reconsideration in light of Van Buren. On remand, the Ninth Circuit reaffirmed that scraping publicly accessible data does not violate the CFAA. LinkedIn could not use the CFAA to block hiQ from scraping public profiles.
Significance: Established that accessing publicly available data on the internet is not unauthorized access under the CFAA.
Clearview AI
2022-2024Anti-scrapingMultiple EU DPAs
Clearview AI was fined by multiple European data protection authorities (France: EUR 20M, Italy: EUR 20M, Greece: EUR 20M, UK: GBP 7.5M) for scraping billions of facial images from social media without consent or a lawful basis under GDPR.
Significance: Demonstrated that scraping publicly visible data can still violate GDPR when it involves biometric/personal data processed without a legal basis.
Meta v. Bright Data
2024MixedUS District Court, N.D. California
A federal judge ruled that Bright Data did not violate Meta Terms of Service by scraping publicly available data from Facebook and Instagram. However, scraping data from logged-in sessions could constitute a breach of contract.
Significance: Reinforced the distinction between scraping public data (likely permissible) and scraping behind authentication walls (legally risky).
Van Buren v. United States
2021Narrowed CFAA scopeUS Supreme Court
The Supreme Court ruled 6-3 that the CFAA covers only those who access a computer without authorization, not those who have authorized access but use it for improper purposes. This narrowed the CFAA to a gates-up-or-down inquiry.
Significance: Narrowed the scope of the CFAA, making it harder to prosecute scraping of publicly accessible websites under federal computer fraud statutes.
KASPR (CNIL Decision)
2022Anti-scrapingCNIL (French DPA)
KASPR was fined EUR 240,000 by the French data protection authority (CNIL) for scraping LinkedIn contact data (phone numbers, emails) without a valid legal basis. CNIL found that the data subjects had not made their data manifestly public and KASPR lacked legitimate interest.
Significance: Clarified that scraping professional social media profiles does not fall under the "manifestly made public" exception in GDPR, and legitimate interest must be carefully assessed.
Key Takeaways from Legal Precedents
- Scraping publicly accessible data generally does not violate the CFAA (Van Buren, hiQ)
- Scraping behind authentication walls carries significantly higher legal risk (Meta v. Bright Data)
- GDPR can impose massive fines even when data is publicly visible (Clearview AI, KASPR)
- Biometric and sensitive personal data receive the strongest protections globally
- Terms of Service violations may give rise to breach of contract claims, even where CFAA does not apply
Ethical Scraping Best Practices
Beyond legal compliance, ethical scraping practices reduce your litigation risk, build sustainable data pipelines, and maintain your reputation in the industry. These practices go beyond what the law strictly requires but represent the standard that responsible data collection teams should follow.
Respect robots.txt
While robots.txt is a guideline, not legally binding (RFC 9309), respecting it demonstrates good faith. Courts have referenced robots.txt compliance in their rulings. Treat it as a minimum baseline, not a maximum boundary.
Implement Rate Limiting
Sending thousands of requests per second can constitute a denial-of-service attack, which is clearly illegal. Use reasonable delays (1-5 seconds), respect Retry-After headers, and limit concurrent connections to avoid server strain.
Avoid Personal Data Without Basis
Do not scrape personal data unless you have a documented lawful basis. If you need product prices, do not also harvest reviewer names and profile URLs. Apply data minimization as a default principle.
Be Transparent
Use identifiable User-Agent strings. Maintain a public page explaining your data collection practices. Provide contact information for website operators who want to discuss your scraping activity.
Data Minimization
Collect only the data you need for your stated purpose. Do not stockpile data "just in case." Implement retention schedules and delete data when it is no longer needed. Less data means less risk.
Honor Opt-Out Requests
Build mechanisms to handle data subject requests (GDPR erasure, CCPA opt-out). If a website operator or individual asks you to stop scraping their content or delete their data, comply promptly.
The Golden Rule of Ethical Scraping
Ask yourself: "Would I be comfortable if the website operator, the data subjects, and a regulator could all see exactly what I am collecting, how I am collecting it, and what I am doing with it?" If the answer is no, reconsider your approach. Ethical scraping is not just about avoiding fines — it is about building data operations that are sustainable, defensible, and respectful of others' digital presence.
How Proxy Type Affects Compliance
The proxy infrastructure you use for scraping carries its own compliance implications. Not all proxy providers source their IP addresses ethically, and using proxies from compromised devices or botnets can create additional legal liability beyond the scraping itself.
Ethical IP Sourcing Matters
The source of your proxy IPs is a compliance consideration that many teams overlook. There is a significant ethical and legal difference between proxy networks built from consenting device owners and those built from compromised systems.
Ethical Proxy Sources
- Real 4G/5G devices with user consent
- Datacenter IPs from legitimate providers
- ISP proxies from consenting partners
- Transparent sourcing with auditable consent
Problematic Proxy Sources
- Botnets or malware-infected devices
- SDK-based networks without clear consent
- Hacked routers or IoT devices
- "Free VPN" apps that sell user bandwidth
How Proxies.sx Approaches IP Sourcing
Proxies.sx operates a mobile proxy network built on real 4G/5G devices with explicit consent from device owners. Every IP in our pool comes from a genuine mobile carrier connection (T-Mobile, Verizon, AT&T, Vodafone, and others across 15+ countries). We do not use residential SDK networks, botnets, or hacked infrastructure.
This ethical sourcing model means that when you use Proxies.sx for your scraping operations, the proxy layer itself does not introduce additional compliance risk. Your scraping traffic routes through legitimately obtained mobile IPs via CGNAT, the same infrastructure that millions of regular mobile users share. You can review our privacy policy and terms of service for full details on our data handling practices.
When evaluating proxy providers for compliance-sensitive scraping operations, ask about their IP sourcing practices, whether they can provide documentation of consent, and whether their infrastructure has been involved in any legal actions or regulatory investigations. Transparency from your proxy provider is a key component of your overall compliance posture.
Scraping Compliance Checklist
Use this 10-point checklist before starting any new scraping project. While not a substitute for legal advice tailored to your specific situation, this checklist covers the fundamental compliance considerations that every scraping operation should address.
Identify the legal basis for data collection
Under GDPR, you need a lawful basis (legitimate interest, consent, contractual necessity, etc.). Document your basis before scraping begins.
Determine if personal data is involved
IP addresses, email addresses, names, usernames, and profile photos are all personal data under GDPR. If you are collecting any of these, additional obligations apply.
Check the website Terms of Service
While ToS violations alone may not create criminal liability, they can form the basis for breach of contract claims. Review and document ToS before scraping.
Respect robots.txt directives
Although not legally binding, ignoring robots.txt signals can be used as evidence of bad faith in litigation. Treat robots.txt as a baseline for respectful access.
Implement rate limiting and polite crawling
Do not overload target servers. Use reasonable delays between requests (1-5 seconds minimum). Excessive load can constitute a denial-of-service attack, which is clearly illegal.
Minimize personal data collection
Apply GDPR data minimization principles. Only collect the data you actually need. If you need pricing data, do not also scrape user reviews containing personal names.
Implement data retention policies
Do not store scraped data indefinitely. Define retention periods aligned with your stated purpose. Delete data when it is no longer needed.
Provide opt-out mechanisms where applicable
If you operate in a CCPA jurisdiction, consumers have the right to opt out of the sale of their personal information. Build mechanisms to honor these requests.
Use ethically sourced proxy infrastructure
Ensure your proxy provider obtains IP addresses from consenting device owners. Using proxies from botnets or non-consenting devices creates additional legal and ethical liability.
Conduct and document a Data Protection Impact Assessment
For large-scale scraping operations, GDPR Article 35 may require a DPIA. Document your assessment of risks, safeguards, and proportionality even if not strictly required.
Important Disclaimer
This checklist is provided for educational purposes and does not constitute legal advice. Web scraping legality varies by jurisdiction, target website, data type, and specific circumstances. For scraping operations involving personal data, cross-border data transfers, or regulated industries, consult with a qualified attorney who specializes in data protection law in your relevant jurisdictions.
Frequently Asked Questions
Scrape Responsibly with Ethically Sourced Proxies
Proxies.sx provides real 4G/5G mobile proxies from consenting device owners across 15+ countries. Start with a free trial: 1GB bandwidth + 2 ports. No credit card required.