Keyword clustering & intent for SEO

Below is a guide to taking a CSV of up to 50,000 keywords and clustering them by topic — with keyword intent applied as well.

Get the script — take a copy or use it now in Google Colab

Thank you to these geniuses

I've merged a couple of scripts together to build exactly what I wanted — huge shoutout to these two for their work.

Semantic Clustering Tool — LeeFootSEO, website: searchsolved.co.uk

Keyword Intents — Greg Bernhardt, article: Use Python to label query intent, entities and keyword count

What is keyword topic clustering?

Keyword topic clustering is a technique used in SEO to group related keywords based on their semantic meaning. This helps organise and optimise website content, allowing search engines to better understand the topic and intent behind the keywords.

By using Python libraries and algorithms, SEO practitioners can analyse large sets of keywords, identify patterns or similarities, and create clusters of keywords that share common themes or intent.

What is keyword intent and how can we categorise it?

Understanding keyword intent is crucial. Keyword intent refers to the underlying motivation or purpose behind a user's search query. By categorising keywords based on their intent, you can optimise content to better align with what users are actually looking for.

There are generally four main types of keyword intent: informational, navigational, transactional, and commercial investigation.

Informational intent

Users searching for knowledge or solutions to a specific query. Generally contain terms like who, what, when, where, why, how — or entities like a movie name or song name. Any time a user wants more information and isn't going to take action online, it's an informational keyword.

• What is a German Shepherd?
• My Chemical Romance
• Is Deception Bay a safe suburb?

Navigational intent

Users aiming to find a specific website or brand — they already have a destination in mind and use search to get there directly.

• Bunnings Warehouse
• Kong dog toys
• Semrush login

Transactional intent

Users ready to buy or perform a specific action — high intent to engage in a commercial activity.

• Buy dog food
• Cheap blue dress
• Books for sale

Commercial investigation

Users comparing options before making a purchase — gathering information, evaluating alternatives.

• Compare dog toys
• Ahrefs vs Semrush
• Best technical SEO tool

My issue with standard keyword intent categorisation

Whenever a tool automatically applies intent to keyword data, I constantly find it doesn't quite fit. This is part of the reason I built intent categorisation into the script myself.

I also find that commercial and transactional overlap significantly — across both pet ecommerce and financial websites, I tend to target them together rather than treating them as distinct buckets.

"Navigational" as a grouping causes its own problems too. Tools tend to lump your own brand names, product brand names, and specific product names together, which isn't useful for content planning. Because of these nuances — and possibly because I'm a control freak — I prefer to set the intent categories myself.

How to use the Google Colab script

Written for complete beginners, including SEOs who have never coded before.

Google Colab is a free coding notebook. Open the script, take a copy (the same way you'd copy any Google Drive file), then work through each block of code from the top by clicking the play button next to each step.

CSV structure

Your file must be a CSV (not Excel) with two columns: keywords in the first, search volumes in the second.

Download an example CSV

Decide on your keyword intent categories

These are my defaults — adjust them to suit your vertical. The script looks for these words when deciding intent.

# Define your intent categories
informative = ['what', 'who', 'when', 'where', 'which', 'why', 'how', 'can ']

transactional = ['buy', 'order', 'purchase', 'cheap', 'price', 'discount', 'shop', 'sale',
                 'offer', 'snuffle mat', 'pet crate', 'food', 'toy', 'feeder', 'collar',
                 'bed', 'hardness', 'ball', 'carrier', 'litter', 'bowl', 'best', 'top',
                 'review', 'comparison', 'compare', 'vs', 'versus', 'guide', 'worm treatment']

branded = ['royal canin', 'revolution', 'science diet', 'bravecto', 'balance life',
           'black hawk', 'adaptil']

Step 1: Run pip

Click the play button next to the code. This installs the required libraries. You'll get a green tick when it's done.

This script uses four libraries:

Sentence Transformers — A Python framework from sbert.net that computes sentence and text embeddings, then compares similarities to cluster keywords.

Pandas — Analyses and manipulates data; handles our data frames throughout.

Chardet — Detects character encoding of the CSV so it can be read correctly.

Detect delimiter — Detects the delimiter of the CSV (comma, pipe, semicolon, etc.).

!pip install sentence_transformers polyfuzz chardet detect-delimiter

Step 2: Run Python imports

import sys
import time
import pandas as pd
import chardet
import codecs
from detect_delimiter import detect
from google.colab import files
from sentence_transformers import SentenceTransformer, util

Step 3: Upload CSV keyword & volume file

• Keywords in the first column
• Search volumes in the second column
• CSV format only
• Recommended maximum: 50,000 rows

# Upload the keyword export
upload = files.upload()
input_file = list(upload.keys())[0]  # get the name of the uploaded file

Step 4: Set cluster accuracy, size & sentence transformer

You can change these values to suit your needs.

cluster_accuracy — 0–100. Higher = tighter clusters, but more keywords end up unclustered.

min_cluster_size — Minimum number of keywords in a cluster. Lower = tighter groups.

Sentence Transformer — Defaulted to the faster model. The higher-quality option works better on a paid Colab subscription.

cluster_accuracy = 85  # 0-100 (100 = very tight clusters, more unclustered keywords)
min_cluster_size = 2
# transformer = 'all-mpnet-base-v2'  # best quality
transformer = 'all-MiniLM-L6-v2'     # 5x faster, still good quality

Pre-trained model options: sbert.net/docs/pretrained_models.html

Step 5: Run CSV checker

Confirms which character encoding your CSV uses and adjusts if needed.

acceptable_confidence = .8
contents = upload[input_file]

codec_enc_mapping = {
    codecs.BOM_UTF8: 'utf-8-sig',
    codecs.BOM_UTF16: 'utf-16',
    codecs.BOM_UTF16_BE: 'utf-16-be',
    codecs.BOM_UTF16_LE: 'utf-16-le',
    codecs.BOM_UTF32: 'utf-32',
    codecs.BOM_UTF32_BE: 'utf-32-be',
    codecs.BOM_UTF32_LE: 'utf-32-le',
}

encoding_type = 'utf-8'
is_unicode = False

for bom, enc in codec_enc_mapping.items():
    if contents.startswith(bom):
        encoding_type = enc
        is_unicode = True
        break

if not is_unicode:
    guess = chardet.detect(contents)
    if guess['confidence'] >= acceptable_confidence:
        encoding_type = guess['encoding']

print("Character Encoding Type Detected", encoding_type)

Step 6: Create DataFrames

with open(input_file, encoding=encoding_type) as myfile:
    firstline = myfile.readline()
myfile.close()
delimiter_type = detect(firstline)

df = pd.read_csv((input_file), on_bad_lines='skip', encoding=encoding_type, delimiter=delimiter_type)
count_rows = len(df)
if count_rows > 50_000:
    print("WARNING: You may experience crashes when processing over 50,000 keywords. Consider smaller batches.")
print("Uploaded keyword CSV successfully!")
dfkeyword = df

first_column_name = df.columns[0]
second_column_name = df.columns[1]

if first_column_name != "Keyword":
    df.rename(columns={first_column_name: "Keyword", second_column_name: "Search Volume"}, inplace=True)

cluster_name_list = []
corpus_sentences_list = []
df_all = []

corpus_set = set(df['Keyword'])
corpus_set_all = corpus_set
cluster = True

Step 7: Run clustering — this can take a while

cluster_accuracy = cluster_accuracy / 100
model = SentenceTransformer(transformer)

while cluster:
    corpus_sentences = list(corpus_set)
    check_len = len(corpus_sentences)
    corpus_embeddings = model.encode(corpus_sentences, batch_size=256, show_progress_bar=True, convert_to_tensor=True)
    clusters = util.community_detection(corpus_embeddings, min_community_size=min_cluster_size, threshold=cluster_accuracy)

    for keyword, cluster in enumerate(clusters):
        print("\nCluster {}, #{} Elements ".format(keyword + 1, len(cluster)))
        for sentence_id in cluster[0:]:
            print("\t", corpus_sentences[sentence_id])
            corpus_sentences_list.append(corpus_sentences[sentence_id])
            cluster_name_list.append("Cluster {}, #{} Elements ".format(keyword + 1, len(cluster)))

    df_new = pd.DataFrame(None)
    df_new['Cluster Name'] = cluster_name_list
    df_new["Keyword"] = corpus_sentences_list
    df_all.append(df_new)
    have = set(df_new["Keyword"])
    corpus_set = corpus_set_all - have
    remaining = len(corpus_set)
    print("Total Unclustered Keywords: ", remaining)
    if check_len == remaining:
        break

df_new = pd.concat(df_all)
df = df.merge(df_new.drop_duplicates('Keyword'), how='left', on="Keyword")

df = df.sort_values(by="Search Volume", ascending=False)
df['Cluster Name'] = df.groupby('Cluster Name')['Keyword'].transform('first')
df.sort_values(['Cluster Name', "Search Volume"], ascending=[True, False], inplace=True)
df['Cluster Name'] = df['Cluster Name'].fillna("zzz_no_cluster")

col = df.pop("Keyword")
df.insert(0, col.name, col)
col = df.pop('Cluster Name')
df.insert(0, col.name, col)

df.sort_values(["Cluster Name", "Keyword"], ascending=[True, True], inplace=True)

uncluster_percent = (remaining / count_rows) * 100
clustered_percent = 100 - uncluster_percent
print(clustered_percent, "% of rows clustered successfully!")

Step 8: Apply intents

Update the intent categories below to match your vertical. These are Greg's defaults as a starting point:

# Define your intent categories
transactional = ['buy', 'order', 'purchase', 'cheap', 'price', 'discount', 'shop', 'sale', 'offer']
commercial = ['best', 'top', 'review', 'comparison', 'compare', 'vs', 'versus', 'guide', 'ultimate']
informational = ['what', 'who', 'when', 'where', 'which', 'why', 'how']
custom = ['brand variation 1', 'brand variation 2', 'brand variation 3']

Then update the intent application code if you've changed the category names:

df.loc[df['Keyword'].str.contains('|'.join(transactional), case=False, na=False), 'Intent'] = df['Intent'] + ' Transactional'
df.loc[df['Keyword'].str.contains('|'.join(commercial), case=False, na=False), 'Intent'] = df['Intent'] + ' Commercial'
df.loc[df['Keyword'].str.contains('|'.join(informational), case=False, na=False), 'Intent'] = df['Intent'] + ' Informational'
df.loc[df['Keyword'].str.contains('|'.join(custom), case=False, na=False), 'Intent'] = df['Intent'] + ' Custom'

Final step: Download your clustered CSV

df_intents.to_csv('clustered.csv', index=False)
files.download("clustered.csv")

Put the output into a pivot table to filter by intent and find your next content topic to target.