Glassdoor has become the go-to site for millions of job seekers, employees, and employers looking to research companies and gain insights into salaries, company reviews, interview processes, and more. With over 110 million reviews and salary reports across 2 million companies, Glassdoor‘s data is a goldmine for anyone interested in the labor market and workforce trends.
However, manually combing through Glassdoor to find relevant data can be extremely time-consuming and impractical, especially if you want to analyze data at scale. This is where web scraping comes in – it allows you to automatically extract large amounts of data from Glassdoor and compile it in a structured format for further analysis.
In this guide, we‘ll dive into why you might want to scrape data from Glassdoor, the challenges you may face, and provide a comprehensive tutorial on how to easily scrape Glassdoor company reviews and data without any coding using pre-built tools. Let‘s get started!
Why Scrape Data from Glassdoor?
Glassdoor data can provide immense value for a variety of purposes:
For Job Seekers: Glassdoor‘s company reviews, salary reports, and interview experiences submitted by current and former employees offer an inside look at what it‘s really like to work at a company. By analyzing this data, job seekers can identify top employers, compare salaries to ensure fair pay, and prepare for interviews. Scraping Glassdoor data allows job seekers to aggregate company insights at scale and make data-driven decisions in their job search.
For Employers: Companies can use Glassdoor data to benchmark themselves against competitors, identify areas for improvement, and attract top talent. By scraping Glassdoor reviews, companies can track employee sentiment over time, monitor their brand reputation, and gather feedback to address any issues. Scraped salary data can also inform compensation strategies to ensure they are offering competitive pay.
For Recruiters: Recruiters can leverage Glassdoor data to find top talent, craft compelling outreach messages, and gain a deeper understanding of their target companies and candidates. Scraping interview reviews can provide valuable insights into the hiring process at different companies. Salary data can help recruiters price roles competitively and understand the market value of candidates.
For Researchers: Glassdoor‘s vast dataset is a valuable resource for researchers studying labor market trends, employee satisfaction, company culture, diversity & inclusion, and more. By scraping and analyzing millions of data points from Glassdoor, researchers can uncover insights and trends that surveys and other methods may not capture.
Challenges of Scraping Glassdoor
While Glassdoor‘s data is immensely valuable, extracting it at scale comes with some challenges:
Login Walls: Many pages on Glassdoor require users to log in to access the full data. Scrapers need to handle authentication and maintain active sessions.
Bot Detection: Glassdoor has anti-bot measures in place to prevent scraping. Scrapers need to mimic human behavior, such as adding delays between requests and randomizing user agents.
Rate Limits: Glassdoor may throttle or block IP addresses making too many requests in a short period. Scrapers should add delays and limit concurrency.
Changing Page Structures: Glassdoor‘s pages may change over time, breaking scrapers. Scrapers need to be maintained to adapt to any HTML structure changes.
Legal and Ethical Concerns: Glassdoor‘s terms prohibit scraping. Scrapers should use data for analysis only, not republish it, and be mindful not to overload Glassdoor‘s servers.
Despite these challenges, there are still ways to scrape Glassdoor data safely and effectively. In the next section, we‘ll explore different methods and tools for extracting Glassdoor data.
Methods for Scraping Glassdoor Data
There are several ways to scrape data from Glassdoor, depending on your technical expertise and project needs:
1. Pre-Built Glassdoor Scrapers
The easiest way to scrape Glassdoor is to use pre-built Glassdoor scrapers that handle the heavy lifting for you. There are open-source scrapers written in Python, Node.js, and other languages that you can find on GitHub. These scrapers have built-in functionality to extract different types of data from Glassdoor, such as company reviews, salaries, jobs, and interviews.
Using a pre-built Glassdoor scraper is a good option if you have some coding knowledge and want a quick solution. However, the downside is that these scrapers may not be actively maintained, so they could break if Glassdoor changes their page structures. You also have less control over the extracted data and may need to modify the code for your specific use case.
2. Coding a Glassdoor Scraper from Scratch
If you have more advanced programming skills, you can code your own Glassdoor scraper from scratch using languages like Python or Node.js and libraries like Scrapy, BeautifulSoup, Cheerio, or Puppeteer.
The benefit of coding your own scraper is that you have full control and flexibility over the data you extract and can customize it to your exact needs. However, this requires a significant amount of development time and effort, especially to handle all the challenges of scraping Glassdoor at scale (authentication, pagination, rate limits, etc.).
3. Using Web Scraping Tools and Services
For non-technical users or those who want a faster, easier way to scrape Glassdoor, web scraping tools and services are the best option. These tools provide a user-friendly interface for setting up Glassdoor scrapers without any coding.
Some popular web scraping tools that work well for Glassdoor include:
- ParseHub
- Octoparse
- WebHarvy
- Import.io
- Scrapy Cloud
The advantages of using a web scraping tool are:
- No coding required – just point-and-click to configure your scraper
- Handles technical challenges like authentication, pagination, throttling, etc.
- Cloud-based options for large-scale scraping and automatic scheduling
- Pre-built Glassdoor templates for quick setup
- Structured exports to JSON, CSV, Excel for easy analysis
In the next section, we‘ll show you step-by-step how to scrape Glassdoor company reviews using our recommended no-code tool.
Step-by-Step Guide to Scraping Glassdoor Reviews
For this tutorial, we‘ll use Octoparse, a powerful no-code web scraping tool, to extract reviews from Glassdoor. Octoparse offers pre-built Glassdoor scrapers as well as advanced functionality like cloud-based scraping, IP rotation, scheduled crawls, and API access.
Note: Octoparse offers a free trial, but for large-scale scraping of Glassdoor, you‘ll likely need a paid plan for more pages/records per crawl and the Cloud-based features. Paid plans start at $75/month.
Step 1: Create a free Octoparse account and install the app
Go to the Octoparse website and sign up for a free account. Download and install the Octoparse app on your computer.
Step 2: Choose the pre-built Glassdoor Reviews template
In the Octoparse dashboard, click on "Template" and search for the "Glassdoor Reviews" template, or go to the Glassdoor category. This template will automatically configure a scraper to extract the following fields from Glassdoor reviews:
- Review date
- Reviewer job title
- Employment status
- Location
- Review title
- Number of helpful votes
- Pros text
- Cons text
- Advice to management
- Star ratings for overall, work-life balance, culture & values, diversity & inclusion, career opportunities, compensation & benefits, senior management
Click "Use Template" to create a new task using this template.
Step 3: Enter your target company‘s Glassdoor Reviews URL
In the newly created task, paste in your target company‘s Glassdoor Reviews URL in the format:
https://www.glassdoor.com/Reviews/[Company-Name]-Reviews-E[Company-ID].htm
The [Company-Name] and [Company-ID] fields should match the values in your target company‘s Glassdoor URL. For example, for Microsoft:
https://www.glassdoor.com/Reviews/Microsoft-Reviews-E1651.htm
Step 4: Configure pagination, delays, max pages/records
By default, the template will extract data from the first page of reviews only. To scrape more pages, configure the following settings:
- Pagination: Set "Max Page" to the number of pages of reviews you want to scrape, or leave as "0" to extract all pages.
- Delays: Add a 5-10 second delay between pages to avoid overloading Glassdoor‘s servers and prevent your IP from being blocked.
- Max Records: Set a maximum number of reviews to scrape, or leave as "0" for unlimited.
Step 5 (Optional): Configure proxy settings for large crawls
If you are scraping a large number of reviews (1000+), it‘s recommended to use Octoparse‘s built-in proxy rotation feature to avoid getting blocked by Glassdoor. You can connect your own list of proxy servers or purchase proxy IPs through Octoparse‘s proxy partners like Bright Data, IPRoyal, or ProxyCrawl.
Step 6: Run the Glassdoor scraper and export your data
Once you‘ve configured your Glassdoor scraper, click "Save & Run" to start the crawl. You can run it locally for small crawls, or in the Cloud for large jobs. Octoparse will automatically paginate through all the review pages and extract the data into a structured table.
When the crawl is complete, you can export your scraped Glassdoor reviews to CSV, JSON, Excel, or database destinations like MySQL or Google Sheets. And that‘s it! You‘ve successfully scraped Glassdoor reviews without any coding.
Best Practices for Ethical Glassdoor Scraping
When scraping Glassdoor or any website, it‘s important to do so ethically and responsibly to avoid negatively impacting the site or violating terms of service. Here are some best practices to keep in mind:
Respect Robots.txt: Check Glassdoor‘s robots.txt file and respect any directives on which pages are allowed to be scraped.
Don‘t Overload Servers: Add 5-10 second delays between requests and limit concurrent requests to avoid putting excessive load on Glassdoor‘s servers.
Use Proxies and Rotate IP Addresses: Rotate your IP address every few requests using proxies to avoid getting blocked.
Set User Agents and Referrers: Configure your scraper to send legitimate user agent strings and referrers to avoid looking like a bot.
Only Scrape Publicly Available Data: Don‘t attempt to scrape any data behind login walls that is not publicly accessible.
Use Scraped Data Internally Only: Respect Glassdoor‘s terms of service and only use scraped data for internal analysis, not for republishing or commercial purposes.
Regularly Monitor and Maintain Scrapers: Check your scrapers periodically to ensure they are still functioning correctly and not violating any terms of service. Update them as needed if page structures change.
By following these guidelines, you can scrape Glassdoor data safely and ethically to gain valuable insights and inform business decisions.
Conclusion
Glassdoor is an invaluable resource for companies, job seekers, recruiters, and researchers looking to gain insights into employee experiences, salaries, diversity, and company culture. By scraping Glassdoor data at scale, you can uncover trends and make data-driven decisions to improve your organization.
In this guide, we covered why you might want to scrape Glassdoor, the challenges involved, and how to easily extract Glassdoor reviews using a no-code tool like Octoparse. While there are other methods for scraping Glassdoor, like using pre-built scrapers or coding one from scratch, no-code tools offer the fastest and most user-friendly solution.
When scraping Glassdoor, remember to do so ethically by respecting their terms of service, adding delays between requests, rotating IP addresses, and using the data for analysis purposes only.
Armed with this knowledge, you‘re ready to start collecting valuable Glassdoor data to inform your business strategies and gain a competitive edge. Happy scraping!