Job hunting and hiring can be an arduous process for both job seekers and employers. Sifting through countless job boards and postings to find the right match takes significant time and effort. That‘s where job aggregators come in.
Job aggregators act as a one-stop shop, pulling together listings from all corners of the internet into a single, searchable database. They enable job seekers to find more relevant opportunities more quickly, while helping employers get their open roles in front of a larger audience of qualified candidates.
The global online recruitment market is massive and growing rapidly. It was valued at over $28 billion in 2019 and is projected to reach nearly $44 billion by 2027. In the US alone, traffic to job aggregator sites like Indeed and ZipRecruiter tops 200 million visits per month.
| 2019 Market Size (USD) | 2027 Projected Market Size (USD) | 2019-2027 CAGR | |
|---|---|---|---|
| Global | $28.68 billion | $43.39 billion | 7.1% |
| US | $8.97 billion | $13.78 billion | 7.4% |
While the space is crowded, the massive market size shows there are still opportunities for new entrants to gain a foothold. But exactly how do job aggregators work and how can you build a profitable one? Let‘s dive in.
What Is a Job Aggregator?
A job aggregator is a platform that collects job postings from a variety of sources, including job boards, company websites, online classifieds, and more. It then organizes and presents these listings in a user-friendly format, allowing job seekers to search for opportunities based on criteria like location, job title, company, salary, etc.
Benefits for Employers
For companies looking to hire, job aggregators offer expanded reach for their job ads. Instead of being limited to candidates who happen to visit their particular company website or niche job board, employers can get their postings in front of a much larger pool of potential applicants.
Many job aggregators also offer additional services to employers, such as featured company pages for branding, competitive analysis tools to see how their salaries and benefits stack up, and more.
On average, companies receive 34% more applications by posting jobs on aggregators compared to their own sites. And according to a study by Madgex, job aggregators drive 36% of external hires, more than any other source.
Benefits for Job Seekers
For job hunters, these platforms dramatically simplify the search process. Rather than checking dozens of different sites daily for new listings, job seekers can perform a single search to view all current opportunities that match their criteria.
Job aggregators also help job seekers learn more about potential employers by offering company reviews, salary data, and other insights. Some even provide personalized recommendations and analysis on how a candidate compares to other applicants.
A study by Pew Research found that 79% of job seekers use online resources in their most recent job search, and 45% cite job search sites as their top resource. By centralizing listings across the web, job aggregators have become an essential tool for modern job hunters.
How Job Aggregators Make Money
Job aggregators have several paths to monetization:
Subscription/Freemium Model – Offer basic features for free while charging a recurring fee for advanced capabilities like unlimited job postings, candidate resume access, etc.
Pay-Per-Click – Charge employers each time a job seeker clicks on one of their job listings. Prices are often determined in an auction format. Indeed‘s average cost per click in 2020 was $0.88.
Paid Placement – Allow employers to pay for priority placement at the top of search results. ZipRecruiter‘s "TrafficBoost" costs $149 for a 30-day featured job listing.
Promoted Company Profiles – Sell enhanced branding packages with features like videos, employee testimonials, etc. to help employers stand out. Dice offers "Employer Branding" packages starting at $699/month.
Hiring Services – Branch out beyond simple job ad aggregation to offer full-service hiring help, such as sourcing candidates, screening resumes, scheduling interviews, performing background checks, and more.
Candidate Services – Charge job seekers for extras like instant alerts for new postings, company research reports, resume critiques, and profile enhancement to increase visibility to recruiters.
The key is offering valuable tools and information that simplify, speed up, and improve the outcomes of the job search and hiring processes for both sides of the marketplace. The more efficiently you can connect employers with qualified candidates, the more you can charge for your services.
In 2019, the top 5 job search sites in the US posted a combined $4.4 billion in revenue, demonstrating the massive earning potential for leaders in this space.
| Site | 2019 Revenue |
|---|---|
| Indeed | $2.75 billion |
| ZipRecruiter | $430 million |
| CareerBuilder | $356 million |
| Glassdoor | $312 million |
| Monster | $262 million |
Building Your Job Aggregator
Now that we‘ve covered the basics of how job aggregators work and make money, let‘s explore the steps to creating a successful one:
1. Identify Your Niche
The first step is deciding what segment of the job market to target. Trying to be everything to everyone is a losing proposition. The most successful job aggregators focus on a particular niche.
That could mean catering to a specific industry, like tech, healthcare, finance, or hospitality. The upside of an industry focus is you can tailor your platform to the unique needs and preferences of that vertical.
For example, Dice is a job board specifically for technology professionals. By focusing solely on tech roles, they‘ve been able to build a highly targeted platform with features like skills search that reflects how software engineers and other tech workers prefer to job hunt.
Another approach is concentrating on a certain geographic area, such as a single country or city, or underserved smaller towns that the big players overlook. Local expertise and relationships can be a major competitive differentiator.
Lensa has found success by building individual sites for over a dozen US states and major cities, leveraging their local market know-how and employer partnerships to create a strong alternative to national job boards.
2. Aggregate Job Listings
The foundation of any job aggregator is the postings themselves. You need to pull in a large volume of high-quality, relevant listings from a broad spectrum of sources. The goal should be to create the most comprehensive database of jobs in your niche.
That means going beyond the easily accessible public feeds to include harder-to-find opportunities on smaller job boards, industry-specific sites, and company career pages. The more extensive your index of jobs, the more useful you‘ll be to job seekers.
However, building this database is easier said than done. Many websites have barriers in place to prevent scraping. Others might block your IP address if you make too many requests.
That‘s where tools like Octoparse and proxy services come in handy. Octoparse is a powerful web scraping platform that allows you to easily extract data from websites, even those using anti-scraping techniques.
It offers built-in templates specifically for scraping job postings from popular boards like Indeed, Glassdoor, and more. You simply enter your search criteria and let Octoparse do the rest. It also provides useful features like IP rotation, cloud scraping, and the ability to schedule recurring scraping jobs.
Proxies allow you to route your web scraping requests through IP addresses in different locations to avoid detection and bans. There are several types of proxies to choose from:
| Proxy Type | Pros | Cons |
|---|---|---|
| Datacenter | Lowest cost, fastest speed | Easiest to detect and block |
| Residential | Harder to detect as they come from real user devices | More expensive, can have performance and reliability issues |
| ISP | Highest anonymity, as they are issued by real Internet Service Providers | Most expensive, smaller pool of available IPs |
Using a combination of datacenter and residential proxies from providers like Bright Data or Proxy-Cheap along with Octoparse‘s scraping capabilities is a powerful way to build a comprehensive job listing database.
3. Keep Listings Fresh
Of course, scraping your initial database of job postings is just the beginning. To be truly useful to job seekers, you need to ensure your listings are always up-to-date. That means constantly refreshing your index with the latest opportunities.
Again, this is where automation tools are critical. With Octoparse, you can schedule your scraping tasks to run every few hours or even minutes, ensuring you always have the freshest data.
You can also take advantage of incremental scraping to only pull in new listings since your last update. This not only keeps your database current but also minimizes unnecessary strain on your servers and storage.
4. Offer Valuable Extras
Simply aggregating job listings likely isn‘t enough to stand out in the crowded market. To really attract employers and job seekers, you need to provide additional value-add features.
For employers, that could include things like enhanced company branding, featured placements, and tools for promoting their company culture and employee value proposition.
Glassdoor, for instance, allows companies to optimize their branded profile pages with elements like videos, employee reviews, and office photos to give candidates a better sense of their workplace and culture. They also offer managed job advertising services to help employers effectively target and attract talent.
On the candidate side, consider offering job search alerts, resume building tools, career advice content, salary and company insights, and personalized recommendations. The more you can do to help job seekers find their ideal role and employers find their ideal hire, the more indispensable your platform will become.
LinkedIn has had success with this approach. Beyond simply listing jobs, their platform provides career insights, skill assessments, professional networking, and learning opportunities to keep members engaged throughout their career journeys, not just when they‘re actively job seeking.
Putting It All Together
So what does this look like in practice? Let‘s walk through an example of using Octoparse along with proxies to scrape job listings from Indeed.
First, we‘ll navigate to Indeed and perform a search for software engineering roles in San Francisco. Then we‘ll right-click on one of the job titles and select "Inspect" to open up the browser‘s developer tools.
In the Elements tab, we can see the HTML of the job listing. The job title is an <a> element with the class "jobtitle". We can use this information to tell Octoparse what data to scrape.
Next, we‘ll open up Octoparse and create a new task. We‘ll enter the Indeed URL with our search criteria and select the "Auto-detect web page data" option. Octoparse will analyze the page and suggest data fields to extract. We‘ll select the job title, company, location, and description fields.
We‘ll also click the "Settings" button and configure Octoparse to use proxies. We can either import a list of proxies or enter the connection details for a proxy service like Bright Data. Using proxies will help prevent our scraper from being blocked by Indeed.
Finally, we‘ll save the task and let it run. Octoparse will navigate through the search results, grabbing the details for each job posting and compiling them into a spreadsheet for us.
We can schedule this task to run automatically to keep our database constantly updated with the latest jobs. And we can create similar tasks for other job boards and company sites we want to scrape.
Of course, building a job aggregator involves much more than just scraping data. You‘ll also need to build a user-friendly website, develop parsing algorithms to normalize and deduplicate your data, and much more.
But by leveraging powerful tools like Octoparse and proxies to automate the job discovery process, you‘ll have a strong foundation to build on.
Final Thoughts
Building a successful job aggregator requires a significant investment of time and resources. But for entrepreneurs who are able to identify an underserved niche, leverage the right tools and data sources, and create a truly differentiated platform, the potential payoff is huge.
By simplifying the job search and hiring process and making more efficient matches between candidates and employers, you can carve out a profitable position in this massive market. The keys are having a clear target audience, offering genuine value to both job seekers and employers, and leveraging technology to operate at scale.
Some potential challenges you may encounter include:
- Ensuring data quality and consistency when aggregating from many sources
- Protecting user data and privacy
- Preventing fraud and spam
- Complying with legal requirements around data collection and usage
- Differentiating from larger competitors with more resources
However, with careful planning and execution, these challenges are surmountable. Focus on building a great user experience, providing unique and valuable features, and constantly iterating based on feedback and data.
By following the strategies and best practices outlined in this guide and leveraging tools like web scraping and proxies, you‘ll be well on your way to creating the next big job search destination.