Using Generative AI to Increase SEO: Automating SEO Description Generation

In this post, I’ll explain how you can leverage generative AI to automate the process of creating SEO-friendly descriptions for your website’s Markdown content and documentation sits.

1. Prerequisites

Ensure you have the following prerequisites:

- Python installed on your system
- Required libraries: openai, markdown, beautifulsoup4
- An API key from OpenAI (you can sign up for an API key at OpenAI https://www.openai.com/

Install the necessary libraries by running the following command:

pip install openai markdown beautifulsoup4

2. Implementing the Script

import os
import json
import markdown
from bs4 import BeautifulSoup

# Set your OpenAI API key here
from openai import OpenAI
import os

# Initialize the OpenAI Client with your RunPod API Key and Endpoint URL
client = OpenAI(
    api_key="",
    base_url=f"",
)


def generate_seo_description(content):
    prompt = [
        {
            "role": "system",
            "content": "Add your description here"
        },
        {
            "role": "user",
            "content": f"Add your description here\n\n{content}"
        }
    ]
    r = client.chat.completions.create(
        model=MODEL_NAME,
        messages=prompt,
        max_tokens=60
    )
    response = r.choices[0].message.content.strip()
    return response

def extract_metadata(content):
    lines = content.split("\n")
    metadata = {}
    if lines[0] == "---":
        end = lines[1:].index("---")
        meta_lines = lines[1 : end + 1]
        for line in meta_lines:
            if ":" in line:
                key, value = line.split(":", 1)
                metadata[key.strip()] = value.strip()
        return metadata, lines[end + 2 :]
    return metadata, lines


def update_metadata(metadata, content):
    meta_lines = ["---"]
    for key, value in metadata.items():
        meta_lines.append(f"{key}: {value}")
    meta_lines.append("---")
    meta_lines.append("")
    return "\n".join(meta_lines) + "\n".join(content)


def process_file(filepath, processed_files):
    if filepath in processed_files:
        print(f"Skipping {filepath} (already processed)")
        return

    try:
        with open(filepath, "r", encoding="utf-8") as file:
            content = file.read()

        metadata, body = extract_metadata(content)

        html_content = markdown.markdown("\n".join(body))
        text_content = BeautifulSoup(html_content, "html.parser").get_text()
        seo_description = generate_seo_description(text_content)
        metadata["description"] = seo_description
        new_content = update_metadata(metadata, body)

        with open(filepath, "w", encoding="utf-8") as file:
            file.write(new_content)

        print(f"Updated {filepath}")
        print(f"Description: {seo_description}")

        processed_files.append(filepath)
        save_progress(processed_files)

    except Exception as e:
        print(f"Error processing {filepath}: {str(e)}")


def load_progress():
    try:
        with open("progress.json", "r") as file:
            return json.load(file)
    except FileNotFoundError:
        return []


def save_progress(processed_files):
    with open("progress.json", "w") as file:
        json.dump(processed_files, file)


def process_directory(directory, processed_files):
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith(".md"):
                filepath = os.path.join(root, file)
                process_file(filepath, processed_files)


# Set the directory you want to process
directory_to_process = "."

# Load the progress from the previous run
processed_files = load_progress()

# Process the directory
process_directory(directory_to_process, processed_files)

The script to automate the process of generating SEO descriptions for Markdown files. Here’s a breakdown of the script’s functionality:

1. It retrieves the OpenAI API key and initializes the OpenAI client.
2. It defines a `generate_seo_description` function that takes the content of a Markdown file and generates an SEO-friendly description using the OpenAI API.
3. It defines utility functions like `extract_metadata`, `update_metadata`, and `process_file` to handle the extraction and updating of metadata in the Markdown files.
4. It implements a `process_directory` function that recursively processes all Markdown files in a specified directory and its subdirectories.
5. It includes error handling and progress tracking to resume processing from where it left off in case of interruptions or errors.

To use the script, follow these steps:

1. Set your OpenAI API key in the script.
2. Specify the directory containing your Markdown files in the `directory_to_process` variable.
3. Run the script, and it will process all the Markdown files in the specified directory, generate SEO descriptions, and update the metadata in each file.

3. Customizing the SEO Description Generation

The script uses the OpenAI API to generate SEO descriptions based on the content of the Markdown files. You can customize the generation process by modifying the `generate_seo_description` function.

The `generate_seo_description` function takes the content of a Markdown file and sends a prompt to the OpenAI API. The prompt includes instructions for generating a concise and SEO-friendly description limited to 160 characters. You can adjust the prompt and instructions to fine-tune the generated descriptions according to your requirements.

4. Monitoring and Refining

After running the script and generating SEO descriptions for your Markdown files, it’s essential to monitor the impact on your website’s SEO performance. Use tools like Google Analytics and Google Search Console to track your organic traffic, keyword rankings, and user engagement metrics.

Regularly review the generated descriptions to ensure they accurately represent your content and align with your SEO goals. If needed, you can iteratively refine the script and the prompts to improve the quality and relevance of the generated descriptions.

5. Automating the Process

To streamline the SEO description generation process, you can automate the running of the script. Here are a few options:

• Set up a cron job or scheduled task to run the script periodically, ensuring that new Markdown files are processed regularly.
• Integrate the script into your content management system (CMS) or build pipeline, triggering the generation of SEO descriptions whenever new content is added or updated.

By automating the process, you can save time and effort while maintaining up-to-date and optimized SEO descriptions for your website’s content.