javascript 34 lines · 6 steps

Sharding a large sitemap in Next.js

Split a catalog too big for one sitemap into paginated shards, each generated on demand from the database.

Explained by highlit
Next.js Intermediate sitemap pagination seo database code-splitting
1import type { MetadataRoute } from 'next'
2import { db } from '@/lib/db'
3 
4const URLS_PER_SITEMAP = 50_000
5const BASE_URL = 'https://example.com'
6 
7export async function generateSitemaps(): Promise<{ id: number }[]> {
8 const total = await db.product.count({ where: { published: true } })
9 const shards = Math.ceil(total / URLS_PER_SITEMAP)
10 
11 return Array.from({ length: shards }, (_, id) => ({ id }))
12}
13 
14export default async function sitemap({
15 id,
16}: {
17 id: number
18}): Promise<MetadataRoute.Sitemap> {
19 const products = await db.product.findMany({
20 where: { published: true },
21 orderBy: { id: 'asc' },
22 skip: id * URLS_PER_SITEMAP,
23 take: URLS_PER_SITEMAP,
24 select: { slug: true, updatedAt: true, imageUrl: true },
25 })
26 
27 return products.map((product) => ({
28 url: `${BASE_URL}/products/${product.slug}`,
29 lastModified: product.updatedAt,
30 changeFrequency: 'weekly',
31 priority: 0.7,
32 images: product.imageUrl ? [product.imageUrl] : undefined,
33 }))
34}
01 / 01
STEP 01

Walkthrough

Space play ←→ step click any line
Three takeaways
  1. 1Sitemaps have a 50,000-URL limit, so large datasets must be split across multiple files.
  2. 2generateSitemaps enumerates shard IDs while the default export renders each shard independently.
  3. 3Passing the shard id into skip/take turns pagination into a clean map from index to URL slice.

Related explainers

javascript 
function evaluate(expression) {
  const tokens = tokenize(expression);
  let pos = 0;
 

Building a recursive descent calculator

parsing recursion operator-precedence
Intermediate 8 steps
javascript 
import { useState, useEffect, useCallback, useRef } from 'react';
 
const cache = new Map();
const inflight = new Map();

Building a stale-while-revalidate hook in React

caching request-deduplication custom-hooks
Advanced 10 steps
javascript 
import { useEffect, useRef, useState } from 'react';
 
export function useDelayedFlag(active, delay = 300) {
  const [visible, setVisible] = useState(false);

Delaying a loading spinner with a React hook

custom-hooks debouncing cleanup
Intermediate 8 steps
javascript 
const SWIPE_THRESHOLD = 80;
const MAX_TRANSLATE = 120;
 
export function attachSwipeToDismiss(element, onDismiss) {

Building a swipe-to-dismiss gesture in JS

touch-events gesture-detection dom-manipulation
Intermediate 10 steps
python 
from django.db import connections
from django.db.utils import OperationalError
from django.core.cache import caches
from django.http import JsonResponse

Building a health check endpoint in Django

health check monitoring database
Intermediate 7 steps
javascript 
const { pool } = require('./db');
 
function withTransaction() {
  return async (req, res, next) => {

A per-request transaction middleware in Express

middleware database-transactions connection-pooling
Advanced 7 steps

Share this explainer

Here's the card — post it anywhere.

Sharding a large sitemap in Next.js — share card
Made with highlit — turn any snippet into a walkthrough like this in about a minute.
Explain your code