javascript 41 lines · 6 steps

Environment-aware robots.txt in Next.js

A robots convention route that blocks crawlers off production and applies fine-grained rules once live.

Explained by highlit
Next.js Intermediate seo environment-config web-crawlers conditional-logic
1import type { MetadataRoute } from 'next'
2 
3const BASE_URL = process.env.NEXT_PUBLIC_SITE_URL ?? 'https://example.com'
4 
5export default function robots(): MetadataRoute.Robots {
6 const isProduction = process.env.VERCEL_ENV === 'production'
7 
8 if (!isProduction) {
9 return {
10 rules: {
11 userAgent: '*',
12 disallow: '/',
13 },
14 }
15 }
16 
17 return {
18 rules: [
19 {
20 userAgent: '*',
21 allow: '/',
22 disallow: ['/admin/', '/api/', '/checkout/', '/*?*sort=', '/draft/'],
23 },
24 {
25 userAgent: 'GPTBot',
26 disallow: '/',
27 },
28 {
29 userAgent: 'CCBot',
30 disallow: '/',
31 },
32 {
33 userAgent: 'Googlebot',
34 allow: '/',
35 crawlDelay: 1,
36 },
37 ],
38 sitemap: `${BASE_URL}/sitemap.xml`,
39 host: BASE_URL,
40 }
41}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1A robots.ts file lets Next.js generate robots.txt from typed code instead of a static file.
  2. 2Gating rules on the deploy environment keeps preview and staging URLs out of search indexes.
  3. 3An array of rules can target individual user agents differently, from full access to total blocks.

Related explainers

javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
javascript 
import { useState, useEffect, useCallback, useRef } from 'react';
 
const cache = new Map();
const inflight = new Map();

Building a stale-while-revalidate hook in React

caching request-deduplication custom-hooks
Advanced 10 steps
javascript 
import { useEffect, useRef, useState } from 'react';
 
export function useDelayedFlag(active, delay = 300) {
  const [visible, setVisible] = useState(false);

Delaying a loading spinner with a React hook

custom-hooks debouncing cleanup
Intermediate 8 steps
javascript 
const SWIPE_THRESHOLD = 80;
const MAX_TRANSLATE = 120;
 
export function attachSwipeToDismiss(element, onDismiss) {

Building a swipe-to-dismiss gesture in JS

touch-events gesture-detection dom-manipulation
Intermediate 10 steps
javascript 
const { pool } = require('./db');
 
function withTransaction() {
  return async (req, res, next) => {

A per-request transaction middleware in Express

middleware database-transactions connection-pooling
Advanced 7 steps
javascript 
function collapseConsecutiveLogs(lines, { keyFn = (l) => l.message } = {}) {
  const groups = [];
 
  for (const line of lines) {

Collapsing consecutive log lines in JavaScript

grouping run-length-encoding data-transformation
Intermediate 7 steps

Share this explainer

Here's the card — post it anywhere.

Environment-aware robots.txt in Next.js — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code