Web scraping, the automated collection of data from websites, is a powerful tool for businesses looking to gain insights and intelligence. But is it legal? The answer is not always clear cut, as the legal landscape surrounding web scraping is complex and varies significantly by country and context.
In this article, we‘ll take a deep dive into the legality of web scraping in the United States, Europe, and beyond. We‘ll examine key court cases, statutes, and regulations that govern when and how web scraping can be conducted legally. We‘ll also explore the role of IP proxies in web scraping and discuss best practices for ensuring compliance with data privacy and anti-hacking laws.
Web Scraping Legal Landscape
United States
In the U.S., web scraping is governed by an overlapping patchwork of federal and state laws. Here are some of the key legal frameworks that come into play:
Computer Fraud and Abuse Act (CFAA)
The CFAA, 18 U.S.C. § 1030, is the primary federal anti-hacking statute. It prohibits intentionally accessing a computer without authorization or exceeding authorized access to obtain information.
There has been much debate in the courts about whether violating a website‘s terms of service constitutes "exceeding authorized access" under the CFAA. In the 2017 case of hiQ Labs, Inc. v. LinkedIn Corp., the Ninth Circuit Court of Appeals held that the CFAA does not prohibit scraping data that is publicly accessible, as that would grant website owners the power to block access to information in the public domain. However, the court noted that a website could expressly revoke authorization to access its data, at which point further scraping could violate the CFAA.
Other circuit courts have adopted a broader view of CFAA liability. For instance, in EF Cultural Travel BV v. Explorica, Inc., the First Circuit held that using a scraper to access and collect information in excess of the authority granted by a website‘s terms of use can constitute a CFAA violation.
The U.S. Supreme Court recently narrowed the scope of CFAA liability in Van Buren v. United States, holding that an individual "exceeds authorized access" under the CFAA when he accesses a computer with authorization but then obtains information located in particular areas of the computer that are off limits to him. This may limit the CFAA‘s applicability to web scraping going forward.
Copyright
Web scraping can also implicate copyright law, as a website‘s content and underlying code may be protected by copyright. In Ticketmaster Corp. v. Tickets.com, Inc., a court held that the use of a web crawler to copy event information from Ticketmaster‘s website did not constitute copyright infringement, as the scraper only copied factual information rather than creative content.
However, in Associated Press v. Meltwater U.S. Holdings, Inc., a court found that a news aggregator‘s scraping of AP articles was not a fair use under copyright law, as it allowed Meltwater to serve as a substitute for the original works.
Under the Digital Millennium Copyright Act (DMCA), circumventing technological measures that control access to copyrighted works is prohibited. So scraping a website in a manner that bypasses technical restrictions could give rise to a DMCA claim.
Breach of Contract
Many websites have terms of service that restrict data scraping. If a web scraper agrees to those terms and then violates them by scraping, they could be liable for breach of contract.
For example, in QVC, Inc. v. Resultly, LLC, a court allowed QVC‘s breach of contract claim against a scraper to proceed based on QVC.com‘s terms of use, which prohibited the use of bots and the reproduction of site content. However, some courts have held that merely posting terms on a website is not sufficient to bind visitors; the user must take an affirmative action to assent to the terms, like clicking an "I Agree" button.
Trespass to Chattels
Trespass to chattels is a common law tort that involves wrongfully interfering with someone‘s personal property. In the web scraping context, some courts have held that the use of bots that burden a website‘s servers or impair its functionality can constitute a trespass to chattels.
In the seminal case of eBay, Inc. v. Bidder‘s Edge, Inc., a court enjoined Bidder‘s Edge from using bots to scrape eBay‘s website on the grounds that the scraping activity was a trespass to chattels. More recently though, in Craigslist Inc. v. 3Taps Inc., a court held that the defendant‘s scraping of Craigslist‘s publicly accessible content did not constitute a trespass to chattels under California law.
Other Laws
Web scraping may also be restricted by other U.S. laws, such as:
- The CAN-SPAM Act, which prohibits scraping email addresses for the purpose of sending spam.
- The Computer Fraud and Abuse Act (CFAA), which prohibits unauthorized access to computers and networks.
- Section 5 of the Federal Trade Commission Act, which prohibits unfair and deceptive trade practices.
- State data privacy laws, which may require notice and consent before scraping personal information.
Europe
In the European Union, web scraping is primarily regulated by the General Data Protection Regulation (GDPR) and the Database Directive.
GDPR
The GDPR is a comprehensive data protection law that took effect in 2018. It governs the collection, use, and processing of personal data of EU residents.
Under the GDPR, personal data is defined very broadly to include any information relating to an identifiable person, such as their name, email address, IP address, or device ID. In order to legally scrape personal data, you must have a valid lawful basis for processing, such as:
- Consent from the data subject
- Legitimate business interests that are not outweighed by the data subjects‘ rights
- Compliance with a legal obligation
Penalties for violating the GDPR can be severe, up to 4% of a company‘s global annual revenue or €20 million. So GDPR compliance is crucial for anyone scraping websites with EU personal data.
Database Directive
The EU Database Directive 96/9/EC provides sui generis protection for databases that require a substantial investment to assemble. An entity that scrapes a significant portion of a protected database without permission could be liable for infringement.
In the 2004 case of The British Horseracing Board Ltd v. William Hill Organization Ltd, the European Court of Justice held that William Hill‘s scraping of a substantial part of BHB‘s horse racing database was an infringement under the Database Directive.
Other Countries
Many other countries have their own data protection and anti-hacking laws that may impact the legality of web scraping. For example:
- Canada‘s Personal Information Protection and Electronic Documents Act (PIPEDA) regulates the collection, use, and disclosure of personal information in the course of commercial activities.
- Australia‘s Privacy Act 1988 prohibits the collection of personal information without notice and consent.
- China‘s Cybersecurity Law requires network operators to obtain consent before collecting and using personal information.
- Brazil‘s General Data Protection Law (LGPD) is similar to the GDPR and imposes strict requirements for processing personal data.
Role of IP Proxies
IP proxies play an important role in many web scraping setups. A proxy server acts as an intermediary between the scraper and the target website, forwarding requests and responses.
There are several reasons why scrapers use proxies:
- To distribute scraping requests across multiple IP addresses and avoid rate limits or IP bans.
- To mask the scraper‘s true IP address and location for anonymity.
- To improve performance by routing requests through faster or geographically closer servers.
However, the use of proxies can also have legal implications. Some courts have held that the use of proxies or other tools to circumvent IP blocking is a violation of the CFAA. In Craigslist Inc. v. 3Taps Inc., the court found that 3Taps‘ use of proxies and other techniques to bypass Craigslist‘s IP blocks after receiving a cease and desist letter violated the CFAA.
The law is unsettled on this point though. In hiQ Labs, Inc. v. LinkedIn Corp., the Ninth Circuit questioned whether merely avoiding an IP block rises to the level of a CFAA violation, as opposed to more deceptive conduct.
From a GDPR perspective, the use of proxies could potentially be considered a form of automated decision-making that requires explicit consent from data subjects. It‘s an open question that has not been definitively resolved.
Impact and Prevalence of Web Scraping
Web scraping is extremely common in today‘s data-driven business landscape. According to a 2020 report by Opimas, the web scraping industry generated $2.5 billion in revenue in 2020 and is expected to grow to $6.5 billion by 2025.
A recent study by Imperva found that 41.6% of all internet traffic comes from bots and scrapers, with "good bots" like search engine crawlers accounting for 13.1%. Imperva also reported a 37.9% increase in web scraping attacks between 2018 and 2019.
The economic impact of web scraping is significant. On one hand, scrapers can unlock valuable data and insights that drive innovation, competition, and consumer choice. For instance, price comparison services rely heavily on scraping to aggregate offers and identify the best deals for shoppers.
But unrestricted scraping can also impose costs on website owners in the form of increased bandwidth usage, server crashes, intellectual property loss, and consumer privacy violations. In a 2018 survey by Distil Networks (now Imperva), 38.7% of businesses reported costs of $250,000 or more due to web scraping.
From a societal perspective, some argue that web scraping promotes transparency, knowledge sharing, and free speech. The ability to collect and analyze publicly available information can shed light on important issues and hold powerful interests accountable.
However, others contend that unfettered scraping can undermine privacy, enable surveillance and manipulation, and stifle investment in content creation. There are valid concerns about scrapers amassing detailed profiles on individuals and using that data for invasive marketing or other harmful purposes.
Compliance Best Practices
For companies engaged in web scraping, it‘s essential to develop a comprehensive compliance program to mitigate legal risks. Here are some key best practices:
Scraping Policies: Implement clear policies and procedures for web scraping that address issues like respecting robots.txt files, terms of service, and IP blocks; obtaining consent where required; and securely storing and disposing of scraped data.
Data Hygiene: Put in place technical and organizational measures to ensure data minimization, accuracy, security, and deletion in line with applicable laws like the GDPR. Only scrape what you really need and have a legal basis for.
Proxy Management: Carefully vet and monitor any proxy services used for web scraping. Ensure they have strong privacy and security practices. Avoid using proxies to circumvent website restrictions after being notified to stop scraping.
Legal Review: Involve legal counsel in the design and approval of web scraping projects. They can help assess the legal landscape, spot red flags, and craft appropriate terms of service and privacy policies.
Ethics and Transparency: Abide by ethical principles like respect for website owner preferences, transparency about scraping activity, and responsible use of scraped data. Consider signing on to codes of conduct like the Web Scraping Code of Ethics.
Monitor and Audit: Regularly monitor web scraping tools and proxies for misuse or anomalies. Conduct periodic audits to ensure compliance with internal policies and external laws.
By following these best practices, companies can reap the benefits of web scraping while minimizing legal and reputational risks. However, the compliance landscape is constantly shifting, so it‘s important to stay up-to-date on new developments.
The Future of Web Scraping Law
As web scraping continues to grow in importance and sophistication, the law will need to evolve to strike the right balance between the rights of scrapers, website owners, and data subjects. Some key areas to watch include:
- Further clarification of CFAA liability for web scraping, both through court rulings and potential legislative amendments.
- Refinement of GDPR guidance on when web scraping requires consent and how the legitimate interest balancing test applies.
- Proliferation of state and national data privacy laws that impose new notice, choice, and security requirements on scrapers.
- Development of industry self-governance frameworks and technical standards to promote ethical web scraping.
Ultimately, the goal should be to enable beneficial and innovative uses of web scraping while preventing abusive and harmful practices. This will require ongoing dialogue and collaboration among policymakers, technologists, legal experts, and other stakeholders.
Conclusion
Web scraping is a double-edged sword. On one hand, it is a powerful tool for extracting insights and driving innovation in the digital economy. On the other hand, it can be misused in ways that violate privacy, property rights, and principles of fair competition.
The legality of web scraping is a complex question that depends on a variety of factors, including the jurisdiction, the nature of the data being scraped, the scraper‘s methods and intents, and the website owner‘s stance. In the U.S., web scraping is governed by an patchwork of laws, from the CFAA to copyright to trespass to chattels, that have been applied in sometimes inconsistent ways by the courts. In the EU, the GDPR and Database Directive impose significant restrictions on the scrapers.
To comply with this complex and dynamic legal landscape, companies engaged in web scraping need to implement robust policies and processes for ensuring data privacy, security, and ethics. They need to carefully manage issues like proxy usage, robots.txt compliance, terms of service, and data retention. And they need to stay abreast of evolving laws and industry norms.
With the right approach, web scraping can be not only legally defensible, but socially beneficial. It can foster competition, transparency, and innovation while respecting the legitimate interests of website owners and data subjects. As the web continues to grow and evolve, the law of web scraping must strive to achieve that balance.