Don't Redirect Crawlers by Language: URL-Based Locale and hreflang With the Next.js 16 Proxy

The Problem: Spanish Pages That Search Engines Never See

My portfolio has six pages with a Spanish URL and an English sibling:

• /publicaciones and /posts
• /proyectos and /projects
• /contacto and /contact
• /privacidad and /privacy
• /herramientas and /tools
• /tools/distribucion-financiera and /tools/financial-distribution

The Spanish versions existed, rendered fine in my browser and were listed in the sitemap. But none of them were getting indexed. The reason was a piece of "smart" locale detection I wrote months earlier.

What You'll Learn

• Why language redirects based on Accept-Language hide pages from crawlers
• What Google recommends instead
• How to replace middleware.ts with the Next.js 16 proxy.ts convention
• How to let the URL decide the UI language without redirecting anyone
• A side note about 404 pages that return HTTP 200

The Old Middleware

This was the strategy, in the middleware's own words: "Amazon-like". If the visitor had a locale cookie, respect it. Otherwise, read Accept-Language and redirect to the sibling page:

function detectAcceptLanguage(header: string | null): Locale {
  if (!header) return 'en'
  const primary = header.split(',')[0]?.trim().toLowerCase() ?? ''
  return primary.startsWith('es') ? 'es' : 'en'
}

export function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl
  const pair = PAGE_PAIRS[pathname]
  if (!pair) return NextResponse.next()

  // Honor explicit cookie — user already chose
  const cookieLocale = req.cookies.get(LOCALE_COOKIE)?.value
  if (cookieLocale === 'es' || cookieLocale === 'en') {
    return NextResponse.next()
  }

  // No cookie yet → use Accept-Language as the hint
  const detected = detectAcceptLanguage(req.headers.get('accept-language'))
  const targetPath = pair[detected]
  if (pathname === targetPath) {
    return NextResponse.next()
  }

  const url = req.nextUrl.clone()
  url.pathname = targetPath
  return NextResponse.redirect(url)
}

For a human with a Spanish browser it worked well. Now look at it from a crawler's point of view.

Why Crawlers Hit the Wall

Crawlers like Googlebot and Bingbot typically do not send an Accept-Language header, and they do not keep cookies between requests. So for them:

1. No cookie, so skip the first branch
2. No Accept-Language, so detectAcceptLanguage returns 'en'
3. /publicaciones is not the English path, so redirect (307) to /posts

Every Spanish URL redirected to its English sibling, every time. From the search engine's side, the Spanish pages simply did not exist as indexable documents.

Google's guidance on multilingual sites says exactly this: avoid automatically redirecting users based on their perceived language, because it can stop both users and crawlers from reaching all versions of your site. Use hreflang to tell search engines which URLs are translations of each other, and let people switch language with a visible link.

Middleware Is Now Proxy

While fixing this, the Next.js 16 build reminded me of something else:

The "middleware" file convention is deprecated. Please use "proxy" instead.

Next.js 16 renamed middleware.ts to proxy.ts, and the exported function from middleware to proxy. Same idea, new name. So the fix was also a good moment to rename the file.

The New Proxy: Never Redirect, Just Tell the App

The new rule is simple: the URL decides the language. /publicaciones is Spanish, /posts is English, for everyone, including bots. No redirects.

The proxy only annotates the request with an x-locale header that the app can read later:

import { NextResponse, type NextRequest } from 'next/server'
import { PAGE_PAIRS, type Locale } from '@/lib/locale-routes'

/**
 * Pins the UI locale to the URL on locale-specific routes.
 *
 * No redirects: crawlers send no Accept-Language, so redirecting them to the
 * sibling page made the Spanish URLs unindexable. Visitors switch language
 * with the toggle; hreflang tells search engines about the pair.
 */

const CONTENT_PREFIXES: Record<string, Locale> = {
  '/posts/': 'en',
  '/projects/': 'en',
  '/publicaciones/': 'es',
  '/proyectos/': 'es',
}

function pathLocale(pathname: string): Locale | null {
  const pair = PAGE_PAIRS[pathname]
  if (pair) {
    if (pair.es === pair.en) return null
    return pathname === pair.es ? 'es' : 'en'
  }
  for (const [prefix, locale] of Object.entries(CONTENT_PREFIXES)) {
    if (pathname.startsWith(prefix)) return locale
  }
  return null
}

export function proxy(req: NextRequest) {
  const locale = pathLocale(req.nextUrl.pathname)
  if (!locale) return NextResponse.next()

  const requestHeaders = new Headers(req.headers)
  requestHeaders.set('x-locale', locale)
  return NextResponse.next({ request: { headers: requestHeaders } })
}

Two details worth pointing out:

• NextResponse.next({ request: { headers } }) forwards a modified request to the app. The header is not sent to the browser; it only exists for server code that runs during this request.
• Pages that are the same in both languages (like /) return null, so they keep using the cookie or the browser language.

Individual posts and projects are covered too, through the path prefixes (/publicaciones/... is always Spanish).

Reading the Locale in the App

On the server side, getLocale() now checks the header first. The order is: URL, then cookie, then Accept-Language:

export async function getLocale(): Promise<Locale> {
  // Set by proxy.ts on locale-specific URLs; the URL beats the cookie.
  const h = await headers()
  const pathLocale = h.get('x-locale')
  if (pathLocale === 'es' || pathLocale === 'en') return pathLocale

  const cookieStore = await cookies()
  const cookieVal = cookieStore.get(LOCALE_COOKIE)?.value
  if (cookieVal === 'es' || cookieVal === 'en') return cookieVal

  const accept = h.get('accept-language') ?? ''
  return accept.toLowerCase().startsWith('es') ? 'es' : 'en'
}

The root layout uses it for <html lang>, so a Spanish URL always produces lang="es", even if your cookie says English. That matters: a page whose content is Spanish but whose lang attribute says English sends mixed signals.

And the pages declare their pair with hreflang through the Metadata API:

alternates: {
  canonical: URL,
  languages: {
    en: PATH_EN,
    es: PATH_ES,
    'x-default': PATH_EN,
  },
},

Verifying It as a Bot

I requested the six Spanish URLs with a Bingbot user agent and no Accept-Language:

curl -s -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)' \
  https://www.fredhii.com/publicaciones

All six returned 200 with <html lang="es">. Before the change, the same request returned a 307 to the English page.

Side Note: 404 Pages That Return 200

While testing, I also requested a post slug that does not exist and got HTTP 200 instead of 404. The page showed the not-found UI, but the status code was wrong.

The cause is the loading.tsx boundary. With a loading boundary, the response starts streaming before the page decides it is not found, and once the status line is sent, it cannot change. Next.js compensates by injecting:

<meta name="robots" content="noindex" />

So search engines will not index the page, which is the part that matters for SEO. I also tried dynamicParams = false; for routes that render dynamically, it did not change the status code. I decided the noindex behavior is acceptable rather than removing the loading state.

Key Takeaways

• Do not redirect by language: crawlers usually send no Accept-Language and no cookies, so they always land on the default locale
• Make the URL the source of truth for the language, and use hreflang to connect the pairs
• In Next.js 16, middleware.ts is now proxy.ts; you can pass data to the app by modifying request headers in NextResponse.next
• Check your pages with a bot user agent, not just your browser
• Under a loading.tsx boundary, not-found pages may return 200 with a noindex meta tag