While generative models have made significant strides in generating creative content ideas, most examples fail to ground these ideas in real-world search demand and trends. Keyword research is essential to fuel generative models with ideas that cut through the noise. By analyzing high-performing keywords, and trends in search queries, content creators can develop ideas tailored to what searchers want now.

Leveraging keyword research helps generate content that solves problems, addresses recent news topics, and capitalizes on current interests—ultimately producing a steady stream of impactful content. And with generative AI, we can use these keyword insights to produce content at scale.

In this blog post, we’ll build a simple Python application to generate content ideas based on keyword research. We’ll use two Cohere endpoints: Embed and Chat.

We’ll show snippets of the code in this article, but you can find the complete Google Colab notebook here.

*In a nutshell, three steps are involved: getting the keywords, grouping them into topics, and generating ideas from these topics.*

Step 1: Get a list of High-performing Keywords

First, we need to find a supply of high-traffic keywords for a given topic. Many keyword research tools are available, including Google Keyword Planner, which is free to use.

These keyword research tools provide various information and statistics in their reports, but we’ll only need two types of information: the keywords and the search volume for these keywords.

Let’s say we are interested in generating content ideas around the topic “Remote Teams.” To this end, we can use Google Keyword Planner's “Discover new keywords” feature to get a list of high-performing keywords related to the term.

For convenience, the keyword list that we are using in this article is available here.

*Getting a list of high-performing keywords related to a particular search term in Google Keyword Planner*

Step 2: Group the Keywords into Topics

We now have a list of keywords, but this list is still raw. For example, “managing remote teams” is the top-ranking keyword in this list. However, many similar keywords are further down the list, such as “how to effectively manage remote teams.”

In any keyword research output, there are bound to be many similar keywords, such as these, which means that we need to be able to compare and distill them into broader themes.

Embed the Keywords

The first step is to turn each keyword into a text embedding. A text embedding is a list of numbers (also called a vector) that provides a numerical representation of the contextual meaning of the text. This enables use cases that involve comparing passages of text, such as what we are doing here with clustering. Some other use cases made possible by this are search, recommendation, and classification.

The Cohere Embed endpoint turns a text input into a text embedding. Its usage is straightforward – we call the co.embed()method by passing a list of text inputs and getting back the corresponding embeddings. method by passing a list of text inputs, and get back the corresponding embeddings.

import cohere
co = cohere.Client("COHERE_API_KEY")

def embed_text(texts):
  output = co.embed(
                texts=texts,
                model='embed-english-v3.0',
                input_type="search_document",
                )
  return output.embeddings

embeds = np.array(embed_text(df['keyword'].tolist()))

Cluster the Keywords into Topics with scikit-learn

We then use these embeddings to cluster the keywords. A common term used for this exercise is “topic modeling.” Here, we can leverage scikit-learn’s KMeans module, a machine learning algorithm for clustering.

The input to the module is the embeddings. We also need to define the number of clusters to be created. In this example, we choose four clusters, but there is no right or wrong number to use.

The implementation is as follows.

from sklearn.cluster import KMeans
NUM_TOPICS = 4
kmeans = KMeans(n_clusters=NUM_TOPICS, random_state=21, n_init="auto").fit(embeds)
df['topic'] = list(kmeans.labels_)

The output we get is the assigned topic for each keyword, ranging from 0 to 3 (as we had defined four clusters).

Here are some example topic assignments (see the right-hand column). The first and last items are similar, so they are grouped in the same topic.

keyword	volume	topic
managing remote teams	1000	0
remote teams	390	1
collaboration tools for remote teams	320	1
online games for remote teams	320	3
how to manage remote teams	260	0
…	…	…	…

And here’s an example list of keywords that belong to that topic (the theme is quite clearly about managing remote teams):

'keywords': 'managing remote teams, how to manage remote teams, leading remote teams, managing remote teams best practices, remote teams best practices, best practices for managing remote teams, ... [truncated for brevity]

Generate Topic Names

We now have each keyword assigned to a topic, but we don’t have a topic name other than the 0-4 integers, which are not very informative. Given these keywords, it would be nice to get a representative name for each topic.

For this, we can use the Chat endpoint to generate a topic name for that cluster.

topic_keywords_dict = {topic: list(set(group['keyword'])) for topic, group in df.groupby('topic')}

# Function to generate a topic name based on keywords
def generate_topic_name(keywords):
    # Construct the prompt
    prompt = f"""Generate a concise topic name that best represents these keywords.\
Provide just the topic name and not any additional details.

Keywords: {', '.join(keywords)}"""
    
    # Call the Cohere API
    response = co.chat(
        model='command-r',
        message=prompt,
        preamble="")
    
    # Return the generated text
    return response.text

Here is the same table updated with the topic names (see the right-hand column).

keyword	volume	topic	topic_name
managing remote teams	1000	0	Remote Team Management
remote teams	390	1	Remote Team Tools and Tips
collaboration tools for remote teams	320	1	Remote Team Tools and Tips
online games for remote teams	320	3	Remote Team Fun
how to manage remote teams	260	0	Remote Team Management
…	…	…	…

And here are all the four generated topic names:

Remote Team Management
Remote Team Tools and Tips
Remote Team Resources
Remote Team Fun

Step 3: Generate Blog Post Ideas for Each Topic

Now that we have the keywords nicely grouped into topics, we can proceed to generate the content ideas.

Take the Top Keywords from Each Topic

The list can get very long depending on how many keywords you import. So, it’s probably a good idea to generate content ideas based not on all the keywords but only on the best-performing ones.

So, here we can implement a filter to take just the top N keywords from each topic, sorted by the search volume. In our case, we use 10.

TOP_N = 10

top_keywords = (df.groupby('topic')
                        .apply(lambda x: x.nlargest(TOP_N, 'volume'))
                        .reset_index(drop=True))

Create a Prompt with These Keywords

Next, we create a function to generate blog post ideas. It takes in a string of keywords, calls the Chat endpoint with a prompt to generate three blog ideas, and returns the generated text.

def generate_blog_ideas(keywords):
  prompt = f"""{keywords}\n\nThe above is a list of high-traffic keywords obtained from a keyword research tool. 
Suggest three blog post ideas that are highly relevant to these keywords. 
For each idea, write a one paragraph abstract about the topic. 
Use this format:
Blog title: <text>
Abstract: <text>"""
  
  response = co.chat(
    model='command-r',
    message = prompt)
  return response.text

And that’s it! We now have a list of content ideas informed by what readers actually want to read.

Let’s look at a sample topic and get a feel of what the output looks like. Below are the blog post ideas generated for the Remote Team Management topic.

1. Blog title: "Leading Remote Teams: Strategies for Effective Management"
   Abstract: Effective management of remote teams is crucial for success, but it comes with unique challenges. This blog will explore practical strategies for leading dispersed employees, focusing on building a cohesive and productive virtual workforce. It will cover topics such as establishing clear communication protocols, fostering a collaborative environment, and the importance of trusting and empowering your remote employees for enhanced performance.

2. Blog title: "Remote Teams' Best Practices: Creating a Vibrant and Engaging Culture"
   Abstract: Building a rich culture in a remote team setting is essential for employee engagement and retention. The blog will delve into creative ways to foster a sense of community and connection among team members who may be scattered across the globe. It will offer practical tips on creating virtual rituals, fostering open communication, and harnessing the power of technology for cultural development, ensuring remote employees feel valued and engaged.

3. Blog title: "Managing Remote Teams: A Comprehensive Guide to Training and Development"
   Abstract: Training and developing remote teams present specific challenges and opportunities. This comprehensive guide will arm managers with techniques to enhance their remote team's skills and knowledge. It will explore the latest tools and methodologies for remote training, including virtual workshops, e-learning platforms, and performance coaching. Additionally, the blog will discuss the significance of ongoing development and how to create an environment that encourages self-improvement and learning.

And here’s another example for the Remote Team Tools and Tips topic:

1. Blog title: "The Ultimate Guide to Building Effective Remote Teams"
   Abstract: Building a cohesive and productive remote team can be challenging. This blog will serve as a comprehensive guide, offering practical tips and insights on how to create a united and successful virtual workforce. It will cover essential topics such as building a strong team culture, utilizing collaboration tools, and fostering effective communication strategies, ensuring remote teams can thrive and achieve their full potential.

2. Blog title: "The Best Collaboration Tools for Remote Teams: A Comprehensive Review"
   Abstract: With the rapid rise of remote work, collaboration tools have become essential for teams' productivity and efficiency. This blog aims to review and compare the most popular collaboration tools, providing an in-depth analysis of their features, ease of use, and benefits. It will offer insights into choosing the right tools for remote collaboration, helping teams streamline their workflows and enhance their overall performance.

3. Blog title: "Remote Retrospective: A Guide to Reflect and Grow as a Remote Team"
   Abstract: Conducting effective retrospectives is crucial for remote teams to reflect on their experiences, learn from the past, and chart a course for the future. This blog will focus on remote retrospectives, exploring different formats, techniques, and free tools that teams can use to foster continuous improvement. It will also provide tips on creating a safe and inclusive environment, encouraging honest feedback and productive discussions.

Final Thoughts

In this article, we looked at how to generate content ideas grounded on actual user demand and recency; all this is informed by keyword research results. We can even go on to generate a complete blog post given these abstracts, but that deserves an article of its own, so let’s leave that for another time!

This was possible via a combination of two Cohere endpoints: Embed and Chat. And quite likely, this will not be the last time you’ll see this combo in the wild. What one provides perfectly complements the other—text understanding and text generation, respectively—making this combination extremely useful in real-world applications.

Get started by creating a free account on Cohere.