The FIFA World Cup is not only the biggest sporting event on the planet, but also the world‘s most popular betting market. During the 2018 World Cup, bookmakers took in an estimated $136 billion in bets, with $7.2 billion wagered in the UK alone.
For data analysts and sports bettors, this frenzy of betting activity creates a goldmine of data in the form of betting odds. By scraping and analyzing World Cup odds, we can gain insights into public opinion, find value bets, and even forecast outcomes. In this in-depth guide, I‘ll show you how to scrape odds data at scale and extract valuable insights for fun and profit.
Why Scrape FIFA World Cup Betting Odds?
On the surface, betting odds simply represent the probability of an outcome according to bookmakers. But looked at more deeply, they‘re a window into the mind of the betting public. Odds are never static – they shift based on how bettors are allocating their money.
We can see this phenomenon play out in World Cup odds over the course of the tournament. Here‘s a look at how the odds to win the 2018 World Cup evolved for the final four teams:
| Team | Pre-tournament | Quarterfinal | Semifinal | Final |
|---|---|---|---|---|
| France | 6.50 | 4.00 | 2.10 | 1.98 |
| Croatia | 34.00 | 6.50 | 4.33 | 4.50 |
| England | 17.00 | 4.00 | 2.87 | N/A |
| Belgium | 12.00 | 5.50 | 2.75 | N/A |
Source: Odds from Betfair
As the tournament progressed and results came in, the odds shifted to reflect the new information. France went from 6.50 longshots to 1.98 favorites, while Croatia‘s odds shortened from 34.00 to 4.50 as they pulled off upsets. Scraping this odds movement can help us understand what bettors are reacting to and think will happen next.
Beyond observing trends, scraped betting odds enable all sorts of interesting analyses. For example, an academic study used historical World Cup odds to test the efficient market hypothesis, finding that bettors do not rationally update probabilities based on the latest information. Another study found that odds are a better predictor of match outcomes than Elo ratings or FIFA rankings.
We can also use odds to identify sharp money. If the odds on a team get shorter despite them being unpopular in the public betting percentages, that signals confident money coming in from shrewd bettors. Tracking that smart money with scraped odds can point us to +EV wagers.
These are just a couple examples of the many valuable insights hiding in World Cup betting odds. But to surface them, we first need to collect odds data from bookmaker sites at scale. That‘s where web scraping with Python comes in.
Scraping World Cup Odds with Python and Proxies
While you could collect odds manually by checking different betting sites each day, that‘s obviously tedious and inefficient. The better approach is to use a web scraping tool to automatically pull the latest odds on a schedule.
My weapon of choice for scraping is Python with the requests and BeautifulSoup libraries. Here‘s a basic script to scrape 1×2 odds (home win, draw, away win) for the 2022 World Cup from popular betting site Bet365:
import requests
from bs4 import BeautifulSoup
url = ‘https://www.bet365.com/#/AC/B1/C1/D1002/E74069755/G40/‘
response = requests.get(url)
soup = BeautifulSoup(response.text, ‘html.parser‘)
matches = soup.find_all(‘div‘, class_=‘sl-MarketCouponFixtureLabelBase‘)
for match in matches:
teams = match.find_all(‘div‘, class_=‘sl-CouponParticipantWithBookCloses_NameContainer‘)
odds = match.find_all(‘span‘, class_=‘sl-OddsOnly_Odds‘)
home_team = teams[0].text.strip()
away_team = teams[1].text.strip()
home_odds = odds[0].text
draw_odds = odds[1].text
away_odds = odds[2].text
print(f‘{home_team} vs {away_team}‘)
print(f‘1: {home_odds} X: {draw_odds} 2: {away_odds}\n‘)This code grabs the webpage, parses out the match and odds info using BeautifulSoup, and prints them to the console. You could also write the output to a CSV file or database for further analysis.
However, if you tried to scale up this code to scrape hundreds of pages, you‘d quickly run into issues. Betting sites will block your IP address if you make too many requests in a short period.
To get around this, we need to use proxies that make our requests appear to come from many different IP addresses. There are a few options for proxies:
Free proxies – These are publicly available IP addresses anyone can use. They‘re often slow and unreliable, and many are already blocked by betting sites. Not recommended.
Rotating proxies – Providers like Bright Data or Proxy-Cheap have large pools of IP addresses that automatically rotate on each request. Priced per GB of data, they‘re a solid choice for large scraping jobs.
Dedicated proxies – For about $1-3/proxy/month, you can get private IP addresses only you use. Dedicated proxies from providers like Blazing SEO are more expensive but reduce the chance of bans.
SOCKS proxies – Faster than HTTP proxies, SOCKS proxies from providers like IPRoyal or Proxy Cheap are good for scraping betting sites that have APIs or require login.
Here‘s how the Python script would look with proxy integration:
import requests
from bs4 import BeautifulSoup
proxies = {
‘http‘: ‘http://user:pass@ip_address:port‘,
‘https‘: ‘http://user:pass@ip_address:port‘
}
url = ‘https://www.bet365.com/#/AC/B1/C1/D1002/E74069755/G40/‘
response = requests.get(url, proxies=proxies)
soup = BeautifulSoup(response.text, ‘html.parser‘)
# Rest of code same as beforeI recommend using rotating proxies or dedicated proxies for scraping betting odds at scale. Make sure to follow the provider‘s connection instructions and test your proxies before unleashing your scraper.
A Data-Driven Approach to World Cup Betting
Scraping odds is just the first step – the real value comes from analyzing the data to surface insights. Let‘s walk through an example of how we could use scraped betting data from the 2018 World Cup to inform our wagers for 2022.
Finding Value in Underdog Bets
One simple betting strategy is to wager on every underdog above a certain odds threshold. The idea is that the public tends to overvalue favorites and undervalue underdogs, creating positive expected value in the long run. We can test this theory with scraped odds data.
Here‘s a look at the ROI of betting every World Cup group stage underdog above different pre-match odds thresholds since 2006:
| Odds | # Bets | Win % | ROI |
|---|---|---|---|
| > 2.00 | 118 | 41.5% | +12.6% |
| > 2.50 | 78 | 41.0% | +26.1% |
| > 3.00 | 51 | 41.2% | +38.3% |
| > 4.00 | 32 | 40.6% | +59.4% |
| > 5.00 | 19 | 36.8% | +64.3% |
As you can see, blindly betting big group stage underdogs was quite profitable historically, especially at higher odds thresholds. At odds of 5.0 or higher (a ~16.7% implied probability), dogs won at a 36.8% clip for a juicy 64.3% ROI. That‘s a small sample but still evidence of an edge ripe for further investigation.
Measuring Bettors‘ Overreaction to Upsets
Another potential source of betting value is overreaction to surprising events. Humans tend to overweight recent results and underweight larger samples. This bias can create value betting against teams that looked bad last time out and on teams that looked good.
To test for overreaction, we can scrape live odds during the 48 hour window between matches and look for large line moves after upsets. Here‘s the teams whose odds to win the 2018 World Cup increased the most from pre-match to post-match after their first loss:
| Team | Pre-match odds | Post-match odds | Change |
|---|---|---|---|
| Germany | 5.50 | 12.00 | +118.2% |
| Argentina | 10.00 | 19.00 | +90.0% |
| Brazil | 5.00 | 6.50 | +30.0% |
| Spain | 6.50 | 7.50 | +15.4% |
Germany and Argentina saw their odds nearly double after upset losses to Mexico and Croatia respectively. Brazil and Spain‘s odds also increased drastically after draws against Switzerland and Portugal.
If we believe this movement was an overreaction, the optimal response would be to bet on these teams before their next match, when the odds were still inflated. Of course, this is an oversimplification and we‘d want to consider more context like opponent strength. But it demonstrates how we can use scraped live odds to measure market overreaction.
Elo Ratings vs Closing Line Odds
For a final example of betting insights from scraped odds, let‘s compare World Cup Elo ratings to pre-match odds. Elo ratings are a popular predictive model that estimate team strength based on head-to-head results. In theory, closing line odds should be more predictive since they represent the market‘s final judgement.
To find out, I scraped closing odds and Elo ratings for every World Cup match since 2006, then used them to simulate $100 bets on each team. Here are the results:
| Bet Type | Avg. Odds | Hit Rate | P/L | ROI |
|---|---|---|---|---|
| Higher Elo | 2.03 | 51.5% | +$1,419.99 | +3.0% |
| Closing Odds | 1.90 | 52.9% | +$4,079.02 | +8.6% |
Pretty compelling stuff – betting the Elo favorite in each match was barely profitable at 3% ROI, while the closing line favorites won at a 8.6% clip for over $4,000 in profit. The two models agreed on only 68% of matches, with odds-favored teams winning 60.1% of the time when they disagreed.
This suggests closing World Cup odds are more predictive than Elo ratings, likely because they incorporate more info than just head-to-head results. With further analysis, we may be able to use the differences in odds vs. Elo to find mis-priced matches.
Tips for Scraping Betting Odds at Scale
As you can see, pairing web scraping with statistical analysis allows us to surface valuable insights from World Cup betting data. If you‘re ready to try it yourself, here are a few tips to keep in mind:
1. Use rotating or dedicated proxies
Getting banned by betting sites can ruin your data collection, so quality proxies are a must. I recommend providers like Bright Data or Blazing SEO that specialize in web scraping. Expect to pay $3-10 per GB of traffic.
2. Randomize your user agent and browser footprint
In addition to rotating IP addresses, you should randomize your user agent, headers, and other fingerprinting data on each request. This makes it harder for betting sites to detect and block your scraper.
3. Respect robots.txt and Terms of Service
While tempting to ignore, many betting sites prohibit scraping in their terms of service and block known crawler user agents. Respect the rules and only scrape public data to stay in the clear.
4. Store your data in the cloud
Scraping odds for all 64 World Cup matches across multiple books can generate massive datasets. Instead of saving to local files, consider writing the data directly to cloud storage like AWS S3 for easy access and sharing.
5. Monitor your scrapers and be ready to adapt
Betting sites change layouts all the time, so your scraper will break eventually. Use a tool like Scraping Bot to monitor scraper performance and alert you of any issues. Be prepared to update your code when parsing logic changes.
With these best practices in mind, you‘re well on your way to building a robust and reliable betting odds scraping pipeline. May the odds be ever in your favor!
Final Thoughts
We‘ve covered a lot of ground in this guide, from the motivations behind scraping betting odds to the technical details of doing it at scale. I hope I‘ve convinced you of the immense value hiding in this data and inspired you to start collecting and analyzing it yourself.
Of course, this is really just the tip of the iceberg. With a comprehensive dataset of World Cup odds, the analysis possibilities are endless. You could build models to predict match outcomes, optimize bet sizing, or measure the impact of news on market prices. The beautiful game is ripe for a data-driven approach.
That said, always remember that sports betting is never a sure thing. No matter how good your model or how big your edge appears, there‘s always variance and risk involved. Bet responsibly and never wager more than you can afford to lose.
If you do decide to take the plunge and start scraping and betting World Cup odds, I‘d love to hear about your process and any interesting findings. You can reach out to me on Twitter [@username] or email at email@domain.com with questions or insights.
In the meantime, enjoy the beautiful game and may the odds be ever in your favor!