Medium.com has cemented itself as a hub of high-quality, long-form content since its 2012 founding by former Twitter CEO Evan Williams. It now attracts over 170 million monthly readers1 and has published over 20 million articles to date2, spanning topics from technology to creativity to entrepreneurship, authored by a mix of domain experts and impassioned amateurs.
This vast trove of textual data and engagement metrics, all consolidated in one open platform, is a potential gold mine for content strategists, market researchers, and data scientists eager to extract insights. But with millions of articles spread across the sprawling site, manual methods of data collection are infeasible.
Enter web scraping – the automated, programmatic extraction of data from websites. By deploying web scrapers to collect and parse data from Medium at scale, we can efficiently tap into the collective wisdom and unfiltered perspectives shared on the platform to drive business decisions and model language.
In this deep dive, we‘ll cover the top reasons and use cases for scraping data from Medium, the nuts and bolts of building a Medium scraper (for both coders and non-coders), and how to handle the cat-and-mouse game of rotating proxies and IP addresses to avoid hitting rate limits. Finally, we‘ll explore analytical approaches to convert your raw scraped Medium data into true insight gold. Let‘s jump in!
The Insight Gold Mine: 6 High-Value Applications of Medium Data
Medium‘s open, long-form format lends itself to a uniquely rich and insightful dataset for mining business intelligence. Here are six powerful applications of data scraped from Medium:
Content Strategy Inspiration: Analyzing data on the best-performing Medium posts in your niche – based on engagement metrics like claps, highlights, read ratio, and comment sentiment – provides concrete examples of topic, format, style, and headline combinations that resonate with target audiences to emulate.
Trend and Narrative Tracking: Applying techniques like topic modeling and named entity recognition to a corpus of Medium posts surfaces emerging trends, dominant narratives, and thought leader perspectives shaping your industry before they hit mainstream media.
Competitive Content Gap Analysis: Scraping and indexing your competitors‘ content on Medium helps identify whitespace opportunities where your brand can fill topic and format gaps, based on a data-driven picture of the content landscape.
Audience Demographic and Psychographic Insights: Analyzing the bios and post histories of your target Medium authors and commenters sheds light on audience demographics, professional backgrounds, content preferences, and pain points – all key inputs for persona development and customer journey mapping.
Influencer Identification and Outreach: Finding the most prominent voices and engaging content within your niche on Medium, based on a combination of follower counts, post performance, and comment sentiment, provides a qualified list for influencer marketing outreach.
Fine-Tuning Language Models: The informal yet contextually rich text data from Medium posts and comments serves as valuable training data for fine-tuning language models to better reflect target domains, user personas, and conversational styles for chatbot, content generation, and voice UI applications.
The unifying theme? Web scraping allows us to efficiently collect, quantify, and analyze the unstructured qualitative data of ideas, opinions, and conversations on Medium to inform business decisions.
The Legal Lay of the Land: Medium‘s Scraping Policies
Before we dive into the technical how-to of scraping Medium, it‘s essential to understand the legal and ethical bounds of scraping the site.
Medium‘s robots.txt file, which specifies the rules for bots and scrapers interacting with the site, currently allows crawling of most public post and profile pages. However, it explicitly disallows scraping of many pages like the homepage, tag pages, and post creation pages.
Additionally, Medium‘s Terms of Service prohibits scraping that "imposes an unreasonable load on our infrastructure" or "may harm our Users."
The key takeaway? Scrape responsibly and respectfully. Use rate limiting to space out your requests, cache already-scraped content to avoid duplicate requests, and don‘t scrape any pages blocked by Medium‘s robots.txt file.
Now, let‘s dive into the nuts and bolts of building a Medium scraper that balances data extraction with good web citizenship!
The Coder‘s Approach: Scraping Medium with Python and Scrapy
For data engineers and data scientists comfortable with Python, the popular Scrapy framework offers a powerful and flexible solution for scraping Medium.
Here‘s a high-level walkthrough of building a Medium scraper using Scrapy:
Install Scrapy:
pip install scrapyCreate a new Scrapy project:
scrapy startproject medium_scraperDefine the data fields to scrape in
items.py:import scrapy
class MediumPost(scrapy.Item):
title = scrapy.Field()
author = scrapy.Field()
date = scrapy.Field()
url = scrapy.Field()
text = scrapy.Field()
claps = scrapy.Field()
comments = scrapy.Field()
4. **Create a Spider to crawl and parse Medium posts in `medium_spider.py`:**
```python
import scrapy
from medium_scraper.items import MediumPost
class MediumSpider(scrapy.Spider):
name = ‘medium‘
start_urls = [‘https://medium.com/tag/machine-learning‘]
def parse(self, response):
for post_url in response.css(‘a[data-test-id="post-list-item-title"]::attr(href)‘).getall():
yield scrapy.Request(response.urljoin(post_url), callback=self.parse_post)
def parse_post(self, response):
post = MediumPost()
post[‘title‘] = response.css(‘h1::text‘).get()
post[‘author‘] = response.css(‘div[data-test-id="post-author"] a::text‘).get()
post[‘date‘] = response.css(‘time::attr(datetime)‘).get()
post[‘url‘] = response.url
post[‘text‘] = ‘ ‘.join(response.css(‘article p::text‘).getall())
post[‘claps‘] = response.css(‘div[data-test-id="post-clap-count"]::text‘).get()
post[‘comments‘] = response.css(‘div[data-test-id="responses-count"]::text‘).get()
yield postThis Spider starts on a specific Medium tag page (e.g. "machine-learning"), extracts the URLs of all individual posts listed, then visits each post page to extract the target fields like title, author, full text, claps, and comments.
- Run the Spider:
scrapy crawl medium -o medium_data.csvThis command kicks off the Medium scraper and outputs the scraped data to a CSV file for analysis.
However, running a scraper for any extended period poses the risk of your IP address getting rate-limited or blocked by Medium‘s anti-bot countermeasures. To avoid this, we need to incorporate a proxy rotation system.
Evading Detection: Proxy Rotation for Continuous Medium Scraping
When scraping Medium at scale, using the same IP address for all requests is a surefire way to get rate-limited or banned. The solution is proxy rotation – routing scraping traffic through a pool of IP addresses to distribute the request load.
Here are two popular approaches to integrate proxies into our Medium scraper:
- Scrapy-Rotating-Proxies Middleware: The scrapy-rotating-proxies package makes it easy to use a pool of proxies with a Scrapy Spider:
First, install the package:
pip install scrapy-rotating-proxiesThen, add the following settings to settings.py:
ROTATING_PROXY_LIST = [
‘proxy1.com:8000‘,
‘proxy2.com:8031‘,
‘proxy3.com:8032‘,
]
DOWNLOADER_MIDDLEWARES = {
‘rotating_proxies.middlewares.RotatingProxyMiddleware‘: 610,
‘rotating_proxies.middlewares.BanDetectionMiddleware‘: 620,
}This configuration pulls proxy IPs from a hardcoded list, then the middleware handles rotating through them for each request and checking if any proxies have been banned.
- Crawlera Smart Proxy Service: Crawlera is a paid smart proxy service that handles proxy rotation, CAPTCHAs, and other anti-bot countermeasures.
To use Crawlera with Scrapy:
Install the Crawlera middleware:
pip install scrapy-crawleraAdd the following settings to settings.py:
CRAWLERA_ENABLED = True
CRAWLERA_APIKEY = ‘YOUR_APIKEY‘
DOWNLOADER_MIDDLEWARES = {
‘scrapy_crawlera.CrawleraMiddleware‘: 610
}With proxy rotation in place, our risk of triggering Medium‘s anti-scraping measures is mitigated and we can kick off longer-running scraping jobs to collect a robust Medium dataset.
From Unstructured Text to Actionable Insights: Analyzing Scraped Medium Data
With a corpus of Medium text data in hand, the exciting work of mining it for insights can begin! Here are some powerful techniques to extract value from scraped Medium data:
Topic Modeling with LDA: Latent Dirichlet Allocation (LDA) is an unsupervised learning method that identifies latent topics across a collection of documents – perfect for uncovering the thematic landscape of Medium posts in a particular niche. Gensim‘s LdaMulticore module makes it easy to train an LDA topic model on scraped Medium text data.
Sentiment Analysis with VADER: Gauge the emotional resonance and reception of Medium content using the VADER Sentiment Analysis tool, which provides a sentiment polarity score (positive/negative) and intensity rating for input text. Apply it to the text of user comments on Medium posts to measure audience sentiment.
Named Entity Recognition with spaCy: Extract key people, places, and things mentioned across Medium posts using the spaCy library‘s pre-trained Named Entity Recognition (NER) model. This provides a data-driven view of the salient entities shaping a particular conversation.
Text Summarization with Sumy: Distill long-form Medium posts down to their key points using extractive summarization techniques like Luhn and LexRank from the Sumy package. This enables quickly digesting the key insights from a large corpus of scraped posts.
Content Engagement Analysis with Pandas: Combine scraped Medium text data with numerical engagement metrics like claps, highlights, read ratio, and comment sentiment in a Pandas DataFrame to analyze which topics, entities, post lengths, authors, and other content attributes drive the highest engagement.
The key is asking incisive questions of your scraped Medium data, then applying the appropriate analytical techniques to find the answers. The insights are there – web scraping gives you the data to mine them.
Translating Medium Insights to Business Impact
With a toolkit of web scraping and text analysis techniques at your disposal, how can you translate insights from Medium data into real business impact? Here are some thought-starters:
Content Strategy: Use topic modeling insights to prioritize high-engagement, rising-interest topics and formats for your content calendar. Replicate stylistic and structural elements of top-performing Medium posts in your own content.
User Research: Analyze Medium comment text to identify your target audiences‘ pain points, desires, and jobs to be done, then address them in your content and product marketing.
Competitor Benchmarking: Quantify your brand‘s share of voice and sentiment relative to competitors in your niche on Medium. Fill content whitespace that competitors are neglecting.
Thought Leadership: Identify the most influential voices and viral content in your industry on Medium, then contribute to those conversations and invite those thought leaders to collaborate on content.
Audience Building: Retarget your Medium post readers with relevant offers to drive subscriptions or sales. Use NER-extracted entities as keywords for PPC campaigns to reach audiences with demonstrated topic interest.
The applications are endless – it just takes a data-driven mindset and a willingness to ethically scrape, experiment, and iterate.
The Future of Medium Scraping: Challenges and Opportunities
As Medium‘s popularity and potential as a source of market and audience insight grows, so too will the competition to scrape its data at scale.
Anti-scraping countermeasures like CAPTCHAs, rate limits, and IP bans will only get more sophisticated, putting pressure on scrapers to adapt with rotating user agent strings, headless browsers, and CAPTCHA-solving services.
However, for data-driven organizations committed to gleaning insights from the collective wisdom of Medium in an ethical manner, the juice will continue to be worth the squeeze.
By providing an unfiltered window into the ideas and conversations shaping industries and a potent source of training data for natural language models, Medium data will only become more valuable to mine in the years to come.
The key is striking the right balance between data extraction and good web citizenship – rotating proxies, respecting robots.txt directives, limiting request rates, and delivering genuine value to Medium users with the insights gleaned from their data.
As the old adage goes, "with great power comes great responsibility." And as the power to scrape and analyze Medium data at scale grows, so too does the responsibility to steward that data ethically and translate it into insights that enrich the community.
Here‘s to the bright future of scraping Medium for insight gold! 🚀