Automating SEO Competitor Analysis with Python

Posted on

by

Simone De Palma

in

Add me as Preferred Source on Google
Add as preferred source on Google

How much of your competitor analysis is still manual?

Many SEO professionals spend hours exporting reports, crawling websites, checking robots.txt files, analysing schema markup, comparing search results and grouping keywords into topics.

While each task is valuable on its own, repeating the same process across multiple competitors quickly becomes time-consuming and difficult to scale.

Fortunately, Python allows repetitive analysis to be automated, turning hours of manual work into workflows that can be completed in minutes.

In this article, Iโ€™ll explore four Python frameworks based on Elias Dabbas Advertools library to automate key areas of competitor research:

Why automate competitor analysis?

Checking one competitorโ€™s robots.txt, scraping a SERP, or eyeballing a spreadsheet of schema markup is fine.

Checking five competitors across four different signals, every month, is not. Itโ€™s slow, itโ€™s inconsistent, and itโ€™s very easy to miss the pattern that actually matters because youโ€™re buried in tabs.

Over the last few months, Iโ€™ve been building out a small toolkit of Python scripts that turn this kind of competitor analysis into something repeatable: pull the data once, run it through the same script every time, and let a chart do the pattern-spotting for you.

Auditing Competitor Robots.txt at Scale

The humble robots.txt file is often overlooked during competitor research, yet it can reveal a surprising amount about a websiteโ€™s technical SEO strategy.

While its primary purpose is to instruct search engine crawlers which sections of a site should or shouldnโ€™t be accessed, many websites inadvertently expose valuable information such as XML sitemap locations, private directories, staging environments, internal search paths and even legacy sections that are no longer linked internally.

The script uses Advertools to fetch the robots.txt files of your competitors 

import advertools as adv
import pandas as pd
import time
robotstxt_urls = [
    'https://www.pomellato.com/robots.txt',
    'https://marcobicego.com/robots.txt',
    'https://robertocoin.com/robots.txt',
    'https://www.buccellati.com/robots.txt',
    'https://www.pasqualebruni.com/robots.txt'
]
robots_dfs = []
for url in robotstxt_urls:
    try:
        df = adv.robotstxt_to_df(robotstxt_url=url)
        df['robots_url'] = url
        robots_dfs.append(df)
    except Exception as e:
        print(f"Error processing {url}: {e}")
    time.sleep(7)  # be polite to servers
robots_df = pd.concat(robots_dfs, ignore_index=True)

Once every robots.txt file is in a single dataframe, the script splits the work into two tracks:

  • User-agent classification. A rule-based function buckets every declared User-agent line into a brand โ€” Google, Bing, OpenAIโ€™s GPTBot, Anthropicโ€™s ClaudeBot, Meta, Perplexity, and so on โ€” so you can immediately see which crawlers a competitor is addressing (and, more importantly, which AI crawlers theyโ€™ve chosen to block).
  • Disallow classification. A second classifier groups blocked paths into functional categories โ€” account/login areas, cart and checkout, search and navigation, CMS internals โ€” using keyword matching based on your target site.

The output can be viewed on a Plotly heatmap with competitors on the y-axis and category on the x-axis.

Here is the code to plot the blocked User-Agent from the competitor set

import plotly.express as px
heatmap_data = pd.crosstab(user_agent['robots_url'], user_agent['user_agent_bucket'])
site_names = [get_site_name(url) for url in heatmap_data.index]
fig = px.imshow(
    heatmap_data,
    labels=dict(x="User-Agents", y="Robots.txt", color="Count"),
    x=heatmap_data.columns,
    y=site_names,
    color_continuous_scale="viridis",
    text_auto=True
)
fig.update_layout(title="Distribution of Blocked User-Agents")
fig.show()

The same function, pointed at the Disallow table instead, produces a second heatmap showing which functional areas each competitor is hiding from crawlers.

These findings often reveal opportunities to improve your own crawl efficiency or identify content areas that competitors have chosen to prioritise or intentionally hide.

By automating this process, analysing fifty competitors takes virtually the same amount of time as analysing five.

โšกPro Tip

Rather than hand-writing dozens of elif clauses to classify hundreds of blocked paths, I exported the โ€œcontentโ€ column, pasted the list into Claude, and asked it to propose a rule-based classification and translate it into a Python function.

Itโ€™s a good example of using an LLM as a proper assistant for a tedious task that requires logic.

Iโ€™ll explain it more in-depth in this practical guide I made for the dear chaps in Athens SEO

Visualise SERP Competitors Overlap

Understanding who ranks for your target keywords is useful. Understanding how competitors overlap across hundreds of keywords, even more.

This script uses the Google Programmable Search Engine (PSE) API to retrieve search results and transforms them into a SERP heatmap

It calculates how frequently each domain appears, its average ranking position and its overall SERP coverage, making it easy to identify which competitors dominate an entire topic rather than individual keywords.

One of the biggest advantages of using the PSE API is that it offers a free, scalable and standardised way to analyse Googleโ€™s organic search results without scraping the SERPs directly.

โšกPro Tip: The Programmable Search Engine provides a proxy of Googleโ€™s search results. While results can be configured by language and location, they are not a 100% representation of live Google Search and do not include SERP features such as AI Overviews, Featured Snippets or other rich results.

By creating your own Custom Search Engine, you can quickly generate competitor SERP data and build a scalable workflow for analysing organic search visibility across your market.

Step Instructions
Connect to the Google Custom Search API
  1. Go to Google Cloud Console.
  2. Navigate to Credentials.
  3. Click Create Credentials โ†’ API Key (or select an existing key).
  4. Copy the generated API key and store it somewhere safe.
  5. Next, navigate to APIs & Services โ†’ Library.
  6. Search for Custom Search API.
  7. Ensure the API is enabled for your project.
Create a Custom Search Engine ID
  1. Go to the Programmable Search Engine page.
  2. Click Add to create a new Custom Search Engine (CSE).
  3. Add the competitor websites you want to analyse.
  4. Create the search engine.
  5. Copy the generated Search Engine ID (cx).
  6. Store the Search Engine ID in a secure location, as it will be required by the Python notebook.
 
import advertools as adv
serp_rankings = adv.serp_goog(
    q=["tailleur da donna", "abiti da cerimonia", "vestiti eleganti"],
    gl=['it'],
    key="YOUR_CUSTOM_SEARCH_API_KEY",
    cx="YOUR_CUSTOM_SEARCH_ENGINE_ID"
)
serp_rankings.to_csv('serp_export.csv', index=False)
top_domains = df['displayLink'].value_counts()[:num_domains].index.tolist()
top_df = df[df['displayLink'].isin(top_domains)]
summary = (df.groupby(['displayLink'], as_index=False)
             .agg({'rank': ['count', 'mean']})
             .assign(coverage=lambda d: d[('rank', 'count')] / d[('rank', 'count')].sum()))

Once youโ€™ve exported the SERP data, the script analyses that to uncover the competitive landscape.

The main function calculates how frequently each domain appears, its average ranking position, and its overall SERP coverageโ€”showing which websites occupy the most search results across your keyword set. The results are visualised in a heatmap, making it simple to identify the strongest competitors at a glance.

Clustering Competitor Keywords from SERP Overlap

The third framework is the most involved of the four, and it tackles a different question:

How do competitors rank and overlap with my target website on Google?

Pixel rank tracking addresses the first part of the question, and a pay-as-you-go wrapper like DataForSEO has the right API for it.

According to their documentation, SEO ranks are treated by the following parameters:

  • rank_absolute: Your total rank across the entire page (e.g., 10th item from the top, including ads, maps, and organic links)
  • rectangle_y: The exact pixel distance from the top of the screen to your link (how far a user has to scroll to see you).

My script is built on top of the assessment of the rectangle_y in relation to the rank_absolute parameters.

Since the cumulative pixel height of the SERP up to position 3 is approximately 717 px, the ranking thresholds can be defined as:

  • Top 3: < 600 px
  • Top 5: 600โ€“1,000 px
  • Top 10: 1,000โ€“2,000 px
  • Off 10: Else px

This is one of the advantages of pixel-based ranking. Unlike traditional ranking positions, pixel thresholds need to be defined manually because the height of Googleโ€™s SERPs varies significantly by industry and the types of SERP features displayed.

To determine your own thresholds, open a Google search results page in Chrome DevTools, switch to a mobile viewport, and enable the ruler to measure the pixel height of the results.

This allows you to create ranking thresholds that reflect the actual layout of the SERPs youโ€™re analysing.

With a bit of further data wrangling, you can plot the SERP ranking position bucket for your target site.

Beyond pixel rank tracking, the framework measures the overlap between SERPs.

Borrowing from Marco Giordanoโ€™s deep dive on SERP clustering, you can build clusters of related keywords and automatically assign each cluster a representative topic using TF-IDF.

Unlike traditional keyword clustering, which groups terms based on similar wording, the script clusters keywords according to how Google ranks them. If two keywords consistently return many of the same URLs in the search results, theyโ€™re grouped together because they likely share the same search intent.

The script also analyses domain overlap, showing how often competitors rank for the same keywords and highlighting which websites compete most directly.

Auditing Competitor Schema Markup Coverage

Many only check whether schema exists.

This framework goes much further by comparing which schema types competitors actually implement, allowing you to benchmark your own structured data strategy.

First, open a sample page and check the HTTP status code returned. While youโ€™re there, make sure the page is indexable by confirming it isnโ€™t marked with a noindex and doesnโ€™t point to another URL via a canonical tag. The analysis should focus only on pages that are eligible to appear in search results.

Next, export a structured data crawl for each competitor using Ahrefs Site Audit (or your preferred website crawler). This dataset will be used to compare schema markup implementations across competing websites at scale

Run a List mode crawl with Screaming Frog with all the structured data options configured and export structured_data_all.xlsx

The one caveat worth flagging before you run this: open the exported XLSX first and check that your type_columns list actually matches the schema-type headers in the file. 

import pandas as pd
df = pd.read_excel('/content/structured_data_all.xlsx')
type_columns = ['Type-1', 'Type-2', 'Type-3', 'Type-4', 'Type-5', 'Type-6',
                 'Type-7', 'Type-8', 'Type-9', 'Type-10', 'Type-11', 'Type-12']
df_types = df[['Address'] + type_columns]
melted = df_types.melt(id_vars='Address', value_vars=type_columns, value_name='Schema_Type').dropna()
grouped = melted.groupby(['Address', 'Schema_Type']).size().reset_index(name='Count')
pivot_table = grouped.pivot(index='Address', columns='Schema_Type', values='Count').fillna(0).astype(int)

You can then visualise the results in a simple competitor matrix showing which competitors implement each schema type.

This is the place to spot at a glance that a competitor is running Product and Review schema you havenโ€™t implemented yet, or that nobody in your competitive set has bothered with FAQPage markup.

Four Frameworks to Streamline Competitor Analysis

Competitor analysis has evolved far beyond comparing rankings or reviewing a handful of landing pages.

The four frameworks demonstrate how Python can turn a task that used to eat an afternoon into something you can re-run in minutes.

A few practical takeaways if youโ€™re adapting any of these for your own stack:

  • Keep credentials out of the notebook itself. Every API-driven notebook here (PSE, DataForSEO) should load its keys from Colab secrets or environment variables, not inline strings.
  • Let an LLM write the tedious classifiers. The robots.txt notebookโ€™s approach โ€” export the raw terms, ask Claude to propose a rule-based classification and translate it into pandas code โ€” is a pattern worth reusing anywhere youโ€™re facing hundreds of unique strings that need bucketing.
  • Adapt every framework to your sector. Although it will take some time to adjust to each new client, the cost is worth the automation for the next runs. The worst-case scenario is spending one extra hour after an LLM adjusting the code to suit your needs.

Summarise this post