Make your website discoverable by AI assistants like ChatGPT, Claude, and Perplexity. This guide covers the llms.txt standard, Lynkow's automatic Markdown generation, and a complete Next.js 15 implementation that exposes your content in machine-readable formats.

1. What is llms.txt and why it matters

The problem

Large language models struggle with HTML-heavy websites. When an AI assistant tries to answer a question using your site, it has to parse navigation bars, footers, JavaScript bundles, and ad blocks just to find the actual content. Most of that HTML is noise. The result: your content is either missed entirely or poorly understood.

The solution

llms.txt is an emerging standard -- like robots.txt, but for AI. It is a plain-text Markdown file served at /llms.txt that gives language models a structured index of your site's content. Instead of crawling and parsing HTML, an AI can read your llms.txt and immediately understand what your site offers.

Lynkow auto-generates three complementary endpoints from your published content:

Endpoint

Purpose

Size

/llms.txt

Summary index with links to individual articles as Markdown URLs

Small (a few KB)

/llms-full.txt

Every published article and page concatenated into one Markdown document

Can be large (hundreds of KB)

/{path}.md

Any individual article or page served as clean Markdown

One article at a time

A typical llms.txt looks like this:

Markdown
# My Tech Blog

> A blog about web development, AI, and developer tools.

## Sitemap
- [Sitemap XML](https://example.com/sitemap.xml)

## Blog
### Tutorials
- [Getting Started with Next.js](https://example.com/tutorials/getting-started-nextjs.md): Learn how to build your first Next.js application from scratch.
- [Advanced TypeScript Patterns](https://example.com/tutorials/advanced-typescript.md): Deep dive into conditional types, mapped types, and template literals.

### News
- [Product Launch 2026](https://example.com/news/product-launch-2026.md): Announcing our new developer platform.

## Pages
- [About Us](https://example.com/about.md): Learn about our team and mission.
- [Pricing](https://example.com/pricing.md): Plans and pricing for all team sizes.

Why this matters for SEO

AI-powered search is growing fast. When someone asks ChatGPT or Perplexity a question, these systems look for structured, machine-readable content to cite. Sites that serve llms.txt get cited more often and more accurately. This is the equivalent of adding a sitemap for Google -- except it is for the AI search engines that are gaining market share every quarter.

2. Serving llms.txt

Create a route handler that returns the summary index of your site's content.

TypeScript
// app/llms.txt/route.ts
import { lynkow } from '@/lib/lynkow'

export async function GET() {
  const content = await lynkow.seo.llmsTxt()

  return new Response(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  })
}

The response contains a Markdown document with your site name, description, and links to every published article and page as .md URLs. Lynkow groups articles by category and includes excerpt descriptions when available.

Cache aggressively -- content changes are infrequent, and a 24-hour TTL (86400 seconds) keeps the load minimal while ensuring updates propagate within a day.

3. Serving llms-full.txt

The full version concatenates every published article and page into a single Markdown document. AI systems that want to ingest your entire site in one request use this endpoint.

TypeScript
// app/llms-full.txt/route.ts
import { lynkow } from '@/lib/lynkow'

export async function GET() {
  const content = await lynkow.seo.llmsFullTxt()

  return new Response(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  })
}

For content-heavy sites, this response can be large (hundreds of KB or more). Caching is essential here. If your site has hundreds of articles, consider increasing the cache duration or using on-demand revalidation via webhooks (see the Webhooks & Revalidation guide).

4. Serving individual articles as Markdown

Each article and page can be served as standalone Markdown at its public URL path with a .md extension. For example, an article at /blog/getting-started becomes available at /blog/getting-started.md.

The SDK's seo.getMarkdown() method takes the content path and returns clean Markdown. The SDK appends .md to the path automatically -- you pass the path without the extension.

How paths work

  • Blog articles use Content.path directly. This path already includes the locale prefix for multi-language sites (e.g., /fr/guides/getting-started).

  • Pages use Page.path. For multi-language sites, prepend the locale: /${page.locale}${page.path}.

Route handler

Create a catch-all route that intercepts any URL ending with .md:

TypeScript
// app/[...path]/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ path: string[] }> }
) {
  const { path } = await params
  const fullPath = '/' + path.join('/')

  // Only handle .md requests
  if (!fullPath.endsWith('.md')) {
    return new Response('Not Found', { status: 404 })
  }

  // Strip the .md extension -- the SDK adds it back internally
  const contentPath = fullPath.slice(0, -3)

  try {
    const markdown = await lynkow.seo.getMarkdown(contentPath)

    return new Response(markdown, {
      headers: {
        'Content-Type': 'text/markdown; charset=utf-8',
        'Cache-Control': 'public, max-age=3600, s-maxage=3600',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}

With this in place, every published article and page on your site is accessible as Markdown. A request to https://example.com/blog/my-article.md returns clean Markdown with the article title, metadata, and body content.

Separating the .md route from page routes

If the catch-all above conflicts with your existing [...slug] page routes, place the .md handler in a middleware instead:

TypeScript
// middleware.ts
import { NextRequest, NextResponse } from 'next/server'

export function middleware(request: NextRequest) {
  const { pathname } = request.nextUrl

  // Rewrite .md requests to a dedicated API route
  if (pathname.endsWith('.md') && !pathname.startsWith('/api/')) {
    const url = request.nextUrl.clone()
    url.pathname = '/api/markdown' + pathname
    return NextResponse.rewrite(url)
  }

  return NextResponse.next()
}

export const config = {
  matcher: ['/((?!_next|favicon.ico).*)'],
}

Then create the API route:

TypeScript
// app/api/markdown/[...path]/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ path: string[] }> }
) {
  const { path } = await params
  const fullPath = '/' + path.join('/')

  if (!fullPath.endsWith('.md')) {
    return new Response('Not Found', { status: 404 })
  }

  const contentPath = fullPath.slice(0, -3)

  try {
    const markdown = await lynkow.seo.getMarkdown(contentPath)

    return new Response(markdown, {
      headers: {
        'Content-Type': 'text/markdown; charset=utf-8',
        'Cache-Control': 'public, max-age=3600, s-maxage=3600',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}

5. Multi-language support

Lynkow generates locale-specific versions of llms.txt and llms-full.txt. Each locale gets its own index containing only content published in that language.

Locale-specific llms.txt

Pass the locale option to get content for a specific language:

TypeScript
// app/[locale]/llms.txt/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ locale: string }> }
) {
  const { locale } = await params

  try {
    const content = await lynkow.seo.llmsTxt({ locale })

    return new Response(content, {
      headers: {
        'Content-Type': 'text/plain; charset=utf-8',
        'Cache-Control': 'public, max-age=86400, s-maxage=86400',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}

This serves:

  • /en/llms.txt -- English content index

  • /fr/llms.txt -- French content index

  • /es/llms.txt -- Spanish content index

Locale-specific llms-full.txt

Same pattern for the full document:

TypeScript
// app/[locale]/llms-full.txt/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ locale: string }> }
) {
  const { locale } = await params

  try {
    const content = await lynkow.seo.llmsFullTxt({ locale })

    return new Response(content, {
      headers: {
        'Content-Type': 'text/plain; charset=utf-8',
        'Cache-Control': 'public, max-age=86400, s-maxage=86400',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}

Default locale fallback

When no locale is specified (the routes from sections 2 and 3), the SDK returns content in the site's default locale. You do not need to pass a locale for single-language sites.

Individual Markdown articles in multi-language sites

For blog articles, the Content.path returned by the SDK already includes the locale prefix. Pass it directly to seo.getMarkdown():

TypeScript
// Content.path = '/fr/guides/getting-started'
const markdown = await lynkow.seo.getMarkdown(content.path)
// Fetches: /fr/guides/getting-started.md

For pages, prepend the locale to the page path:

TypeScript
// page.path = '/about', page.locale = 'fr'
const pagePath = `/${page.locale}${page.path}`
const markdown = await lynkow.seo.getMarkdown(pagePath)
// Fetches: /fr/about.md

6. Linking from robots.txt

Tell AI crawlers where to find your llms.txt by referencing it in robots.txt. This is the convention that emerging LLM crawlers look for.

If you already serve robots.txt through Lynkow (see the SEO & Analytics guide), you can enhance it:

TypeScript
// app/robots.txt/route.ts
import { lynkow } from '@/lib/lynkow'

const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || 'https://example.com'

export async function GET() {
  const robotsTxt = await lynkow.seo.robots()

  // Append llms.txt references
  const enhanced = [
    robotsTxt.trim(),
    '',
    `# LLM-readable content`,
    `# See https://llmstxt.org for the specification`,
    `Sitemap: ${SITE_URL}/llms.txt`,
  ].join('\n')

  return new Response(enhanced, {
    headers: {
      'Content-Type': 'text/plain',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  })
}

The output will look like:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

# LLM-readable content
# See https://llmstxt.org for the specification
Sitemap: https://example.com/llms.txt

7. Linking from HTML head

Add a <link> tag to your root layout so browsers and crawlers can discover your llms.txt from any page on your site.

TypeScript
// app/layout.tsx
import { ReactNode } from 'react'

const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || 'https://example.com'

export const metadata = {
  metadataBase: new URL(SITE_URL),
  other: {
    'llms.txt': `${SITE_URL}/llms.txt`,
  },
}

export default function RootLayout({ children }: { children: ReactNode }) {
  return (
    <html lang="en">
      <head>
        <link
          rel="alternate"
          type="text/plain"
          href="/llms.txt"
          title="LLM-readable content"
        />
        <link
          rel="alternate"
          type="text/plain"
          href="/llms-full.txt"
          title="LLM-readable content (full)"
        />
      </head>
      <body>{children}</body>
    </html>
  )
}

For multi-language sites, include locale-specific links:

TypeScript
// app/[locale]/layout.tsx
import { ReactNode } from 'react'

interface Props {
  children: ReactNode
  params: Promise<{ locale: string }>
}

export default async function LocaleLayout({ children, params }: Props) {
  const { locale } = await params

  return (
    <html lang={locale}>
      <head>
        <link
          rel="alternate"
          type="text/plain"
          href={`/${locale}/llms.txt`}
          title={`LLM-readable content (${locale})`}
        />
        <link
          rel="alternate"
          type="text/plain"
          href={`/${locale}/llms-full.txt`}
          title={`LLM-readable content - full (${locale})`}
        />
      </head>
      <body>{children}</body>
    </html>
  )
}

8. Testing

Verify llms.txt

Use curl to check each endpoint:

Bash
# Summary index
curl -s https://example.com/llms.txt

# Full content
curl -s https://example.com/llms-full.txt

# Individual article (note the .md extension)
curl -s https://example.com/blog/my-article.md

# Locale-specific
curl -s https://example.com/fr/llms.txt

What to check

  1. llms.txt should contain your site name as an H1, a description blockquote, and a list of links to .md URLs grouped by category.

  2. llms-full.txt should contain the full Markdown body of every published article and page, separated by horizontal rules (---).

  3. Individual .md endpoints should return the article title as an H1, publication date, category, tags, and the full body as Markdown.

  4. Response headers should include Content-Type: text/plain; charset=utf-8 (or text/markdown; charset=utf-8) and appropriate Cache-Control values.

Check with AI tools

Test that your llms.txt is actually discoverable:

  • Perplexity: Ask it a question about your site content. Sites with llms.txt tend to get cited.

  • ChatGPT with browsing: Ask it to summarize your website. It should find and use the llms.txt.

  • llmstxt.org: The specification site maintains a directory and validator.

Local development

During development, test against your local server:

Bash
# Start the dev server
npm run dev

# Test endpoints
curl -s http://localhost:3000/llms.txt
curl -s http://localhost:3000/llms-full.txt
curl -s http://localhost:3000/blog/my-first-post.md

9. Complete implementation

Here is every file you need, consolidated for easy reference.

SDK client

TypeScript
// lib/lynkow.ts
import { createClient } from 'lynkow'

export const lynkow = createClient({
  siteId: process.env.NEXT_PUBLIC_LYNKOW_SITE_ID!,
  fetchOptions: {
    next: { revalidate: 60 },
  },
})

Route handlers

TypeScript
// app/llms.txt/route.ts
import { lynkow } from '@/lib/lynkow'

export async function GET() {
  const content = await lynkow.seo.llmsTxt()

  return new Response(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  })
}
TypeScript
// app/llms-full.txt/route.ts
import { lynkow } from '@/lib/lynkow'

export async function GET() {
  const content = await lynkow.seo.llmsFullTxt()

  return new Response(content, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  })
}
TypeScript
// app/[locale]/llms.txt/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ locale: string }> }
) {
  const { locale } = await params

  try {
    const content = await lynkow.seo.llmsTxt({ locale })

    return new Response(content, {
      headers: {
        'Content-Type': 'text/plain; charset=utf-8',
        'Cache-Control': 'public, max-age=86400, s-maxage=86400',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}
TypeScript
// app/[locale]/llms-full.txt/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ locale: string }> }
) {
  const { locale } = await params

  try {
    const content = await lynkow.seo.llmsFullTxt({ locale })

    return new Response(content, {
      headers: {
        'Content-Type': 'text/plain; charset=utf-8',
        'Cache-Control': 'public, max-age=86400, s-maxage=86400',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}

Markdown catch-all with middleware

TypeScript
// middleware.ts
import { NextRequest, NextResponse } from 'next/server'

export function middleware(request: NextRequest) {
  const { pathname } = request.nextUrl

  // Rewrite .md requests to the markdown API route
  if (pathname.endsWith('.md') && !pathname.startsWith('/api/')) {
    const url = request.nextUrl.clone()
    url.pathname = '/api/markdown' + pathname
    return NextResponse.rewrite(url)
  }

  return NextResponse.next()
}

export const config = {
  matcher: ['/((?!_next|favicon.ico).*)'],
}
TypeScript
// app/api/markdown/[...path]/route.ts
import { NextRequest } from 'next/server'
import { lynkow } from '@/lib/lynkow'

export async function GET(
  request: NextRequest,
  { params }: { params: Promise<{ path: string[] }> }
) {
  const { path } = await params
  const fullPath = '/' + path.join('/')

  if (!fullPath.endsWith('.md')) {
    return new Response('Not Found', { status: 404 })
  }

  const contentPath = fullPath.slice(0, -3)

  try {
    const markdown = await lynkow.seo.getMarkdown(contentPath)

    return new Response(markdown, {
      headers: {
        'Content-Type': 'text/markdown; charset=utf-8',
        'Cache-Control': 'public, max-age=3600, s-maxage=3600',
      },
    })
  } catch {
    return new Response('Not Found', { status: 404 })
  }
}

Enhanced robots.txt

TypeScript
// app/robots.txt/route.ts
import { lynkow } from '@/lib/lynkow'

const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || 'https://example.com'

export async function GET() {
  const robotsTxt = await lynkow.seo.robots()

  const enhanced = [
    robotsTxt.trim(),
    '',
    '# LLM-readable content',
    '# See https://llmstxt.org for the specification',
    `Sitemap: ${SITE_URL}/llms.txt`,
  ].join('\n')

  return new Response(enhanced, {
    headers: {
      'Content-Type': 'text/plain',
      'Cache-Control': 'public, max-age=86400, s-maxage=86400',
    },
  })
}
TypeScript
// app/layout.tsx
import { ReactNode } from 'react'

const SITE_URL = process.env.NEXT_PUBLIC_SITE_URL || 'https://example.com'

export const metadata = {
  metadataBase: new URL(SITE_URL),
}

export default function RootLayout({ children }: { children: ReactNode }) {
  return (
    <html lang="en">
      <head>
        <link
          rel="alternate"
          type="text/plain"
          href="/llms.txt"
          title="LLM-readable content"
        />
        <link
          rel="alternate"
          type="text/plain"
          href="/llms-full.txt"
          title="LLM-readable content (full)"
        />
      </head>
      <body>{children}</body>
    </html>
  )
}

Summary

Feature

SDK Method

Route

Content-Type

Summary index

lynkow.seo.llmsTxt()

app/llms.txt/route.ts

text/plain

Full content

lynkow.seo.llmsFullTxt()

app/llms-full.txt/route.ts

text/plain

Locale summary

lynkow.seo.llmsTxt({ locale })

app/[locale]/llms.txt/route.ts

text/plain

Locale full

lynkow.seo.llmsFullTxt({ locale })

app/[locale]/llms-full.txt/route.ts

text/plain

Single article

lynkow.seo.getMarkdown(path)

app/api/markdown/[...path]/route.ts

text/markdown

Robots.txt

lynkow.seo.robots()

app/robots.txt/route.ts

text/plain

HTML discovery

<link rel="alternate">

app/layout.tsx

--