Today was the first time the site left my laptop and tried to become a real website on my home server — and it fought me the whole way. The AI had confidently written the Docker setup, the deploy workflow, and the config, and all of it looked right. Then reality hit: fourteen separate bugs, one after another, each invisible until the exact moment it wasn't. A DNS record pointing at the wrong thing. Files owned by the wrong user. A stale Docker volume quietly ignoring every fix. Images that returned 400 no matter what I threw at them. And my personal favorite — the same error message showing up three different times for three completely different reasons. There was even a stretch of five straight commits chasing a fix that was architecturally impossible, which the AI happily helped me attempt anyway. Fixing it wasn't about writing more AI-generated code — it was about vibe engineering my way through: reading source maps by hand, curling endpoints to see what the server actually said, and trusting the evidence over the confident-sounding suggestion. This is the story of all fourteen bugs, in the order they broke me.
TLDR
- Did: Walked through full production deploy to the home server. DNS, Docker, GitHub Actions, R2, database restore.
- Bug 1 — Wrong DNS record type: Setup doc said CNAME to a DDNS hostname. Server uses pfSense Dynamic DNS updating GoDaddy A records directly. Fixed: A record, no intermediary.
- Bug 2 — importMap.js missing in Docker build: Payload generates
importMap.jsat dev time; it's gitignored so it doesn't exist in CI. Build failed withModule not found: Can't resolve './admin/importMap.js'. Fixed: addednpm run generate:importmapbeforenpm run buildin the Dockerfile. - Bug 3 — PAYLOAD_SECRET missing during build: Next.js pre-renders ISR pages at build time. Those pages call Payload. Payload requires
PAYLOAD_SECRET. No secret in CI → build failed. Fixed: addedisBuildPhasecheck insrc/lib/payload.tsto return empty data whenNEXT_PHASE === 'phase-production-build'. - Bug 4 — EACCES on
.next/cacheand.next/server: App runs asnextjsuser (uid 1001) but files were copied into the image as root. ISR cache writes failed with permission denied. Fixed: added--chown=nextjs:nodejsto allCOPYcommands in the Dockerfile runner stage. - Bug 5 — Stale root-owned volume survived the chown fix: Added
next_cachenamed volume and--chownin the same pass, but the volume had already been created as root by a previousdocker compose up. Docker doesn't re-seed a volume that already exists. Had todocker volume rm herfoodblog_next_cacheto force recreation from the updated image. - Bug 6 — R2 hostname not in
next/imageremotePatterns: Browser got 400 on every image. Next.js image optimizer rejects external hostnames not explicitly allowlisted.pub-xxx.r2.devwas never added toremotePatternsinnext.config.ts. First fix usedR2_PUBLIC_URLenv var dynamically — didn't work becausenext.config.tsis evaluated at build time, not runtime. Second fix: static*.r2.devwildcard pattern. Still didn't work — see Bug 9. - Bug 7 —
generate:importmapsilently failing withoutPAYLOAD_SECRET: The importMap build fix from earlier stopped working after an unrelated change triggered a fresh Docker layer. Root cause:payload generate:importmapneeds to initialise the Payload config, which requires a non-emptyPAYLOAD_SECRET. Without it, the command exits 0 but produces no file.npm run buildthen fails with the sameModule not found: Can't resolve './admin/importMap.js'error as before. Fixed: setENV PAYLOAD_SECRET=build-time-placeholderandENV DATABASE_URI=...placeholderin the builder stage of the Dockerfile. These values only exist during the build and never reach the running container. - Bug 8 — Wrong deploy path in workflow: Updated the deploy directory to
/data/herfoodblogin the docs but missed it in.github/workflows/deploy.yml. Fixed before it caused a runtime failure. - Bug 9 —
*.r2.devwildcard not matching in Next.js: After deploying the wildcard fix and clearing the image cache volume, images were still 400. Verified: R2 image returned 200 via direct curl. Compiled config confirmed*.r2.devwas inremotePatterns. App container could reach R2 directly. Curled/_next/imagedirectly and got"url" parameter is not allowed— wildcard present in config but not matching at runtime. Fixed: replaced wildcard with the exact hostnamepub-00cfedb2e09340258e621c3f06b2792d.r2.dev. - Also fixed: All four social media profile URLs were wrong (Facebook, Pinterest, Instagram, YouTube — all pointing to a
herfoodbloghandle that doesn't exist). - Diagnostic red herring:
docker compose exec db psql -U payloaddefaults to connecting to a database named after the user (payload), which doesn't exist. The real database isherfoodblog_db. Always add-d postgresor-d herfoodblog_dbwhen using psql this way. - Round 3 — React error #418 unminified: Site up but browser console showed
Minified React error #418. Wanted full error messages. Three webpack-based approaches tried in sequence:optimization.minimize = false,resolve.aliasto dev builds,NormalModuleReplacementPlugin. All silently did nothing. Root cause discovered by reading source maps: build uses Turbopack (turbopack:///[project]/...prefix), so the entirewebpackcallback innext.config.tsis never called. Triedcpof dev react-dom over production file in Dockerfile — broke the build because the dev file wraps all exports in"production" !== process.env.NODE_ENV && (function() {...})(), making it a no-op in production. Gave up. Conclusion: swapping React to dev builds in a production container is a dead end. Debug the hydration error locally withnpm run devinstead. - Bug 11 — React hydration error #418 (AdSense): After giving up on unminified errors, identified root cause by running
npm run devlocally and reading the console. Error #418 never appeared locally — AdSense got 403 on localhost and never initialised. That absence was the clue: production-only error = AdSense actually loading. Reading the React source confirmed error #418 with "HTML" args means extra DOM nodes found during hydration (not a text mismatch). The<ins class="adsbygoogle">was SSR-rendered. AdSense auto-scans for<ins>elements on load and injects children into them. React 19 uses concurrent hydration (scheduled viaMessagePort) — AdSense can modify the<ins>between scheduler ticks. Fix: render<ins>client-only (useState(false)+useEffectsets mounted, null until then). Server and initial client both render null → no mismatch possible. AdSense initialises cleanly after mount. Confirmed fixed in production. - Bug 12 — Duplicate
<MobileMenu>in layout: Static analysis of server-rendered HTML found an orphan<button aria-label="Open menu">sitting between</header>and<main>.<MobileMenu>was rendered both inside<Header>and as a sibling in<FrontendLayout>. Removed from layout; Header already includes it. - Bug 13 — R2 images still 400 —
pathnamemissing from remotePatterns: Even with the exact hostname inremotePatterns, images returned 400. The compiled config had{"protocol":"https","hostname":"pub-xxx.r2.dev"}— nopathnamefield. Next.js requires an explicitpathnameto match any path. Fixed: addedpathname: '/**'. - Bug 14 —
generate:importmapsilently skips in CI:payload generate:importmaphas a "no new imports found" optimisation — if the file exists and nothing changed, it skips writing. In CI the file doesn't exist (was gitignored), but the command still exits 0 without writing. Turbopack then fails withModule not found: Can't resolve '../importMap'. Fixed: removedimportMap.jsfrom.gitignoreand committed it. Update and re-commit it when adding custom Payload components.
What got built
This was the first real deployment session. Content work is done, the site works locally, and it was time to carry it to the home server. The setup guide (documents/home_server_setup.md) was written and reviewed in the previous session — today was about actually following it.
Before touching the server
Before SSHing in, we reviewed the setup doc. Found the first bug immediately: it said to create a CNAME record pointing dev.herfoodblog.com at a DDNS hostname (the DuckDNS pattern). The server uses pfSense, which has a built-in Dynamic DNS client that updates GoDaddy A records directly via the API. A CNAME to a DDNS intermediary is the right pattern when you don't have that — if pfSense is handling the dynamic update, you just need an A record. Updated the doc and moved on.
The GitHub Actions build: three failures
Pushed to main to trigger the build. It failed immediately.
Failure 1: importMap.js
Module not found: Can't resolve './admin/importMap.js'
Payload generates importMap.js when the dev server starts. It's gitignored — rightly so, it's a build artifact. But the Docker build has no dev server, so the file never exists. Fix: run npm run generate:importmap in the Dockerfile before npm run build.
RUN npm run generate:importmap && npm run build
Failure 2: PAYLOAD_SECRET during SSG
Error: missing secret key. A secret key is needed to secure Payload.
Failed to collect page data for /recipes/[slug]
Next.js pre-renders pages at build time — ISR pages too, not just SSG. The recipe page's generateStaticParams calls getAllRecipeSlugs() which calls getPayload() which requires PAYLOAD_SECRET. That env var doesn't exist in the CI build environment, and there's no database there anyway.
The fix is in src/lib/payload.ts. Next.js sets process.env.NEXT_PHASE to 'phase-production-build' during next build. Every data-fetching helper now checks this and returns empty data:
const isBuildPhase = process.env.NEXT_PHASE === 'phase-production-build'
export async function getPublishedRecipes(...) {
if (isBuildPhase) return emptyList
// ... normal implementation
}
This means the build produces pages with empty shells. At runtime, ISR revalidation fills them in on first request. For a blog on a home server, this is fine — the first visitor to each page pays a small latency cost, everyone after gets the cached version.
The recipe [slug] page also needed dynamicParams = false changed to true with a revalidate = 3600 added, so recipe pages are generated on demand rather than pre-built.
Build 3 passed. The image was pushed to GHCR.
On the server: two more bugs
Bug: DATABASE_URI pointing to localhost
The app container started, connected to the proxy-tier network, picked up its env file — and immediately crashed trying to connect to Postgres at localhost:5432. The .env.local had been written with localhost as the database host. Inside Docker Compose, the host is db (the service name). Changed to @db:5432 and restarted.
Bug: EACCES on .next/server/app/recipes
After the database connection fixed, ISR tried to write its page cache and hit:
Error: EACCES: permission denied, mkdir '/app/.next/server/app/recipes'
The Dockerfile copies everything into the runner image, but the COPY commands were running as root. The app runs as nextjs (uid 1001). Files owned by root, process running as non-root — writes fail. Fix: add --chown=nextjs:nodejs to every COPY in the runner stage:
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
Also added a next_cache named volume in docker-compose.yml for /app/.next/cache so the image optimiser has a writable location that survives container restarts.
Bug: Stale volume ignored the chown fix
After pulling the new image with --chown, the EACCES error on .next/cache/images persisted. The next_cache volume had been created by an earlier docker compose up — before the --chown fix — and Docker doesn't re-seed an existing named volume from the image. The old root-owned volume just kept mounting over the correctly-owned directory.
Fix: delete the volume and let Docker recreate it from the updated image.
docker compose down
docker volume rm herfoodblog_next_cache
docker compose up -d
This is a Docker gotcha that's easy to miss: named volumes are initialised once from the image on first creation. After that they're opaque to image updates. If you change file ownership in the image, any volumes that were seeded before the change have to be manually removed.
R2 images returning 400
After the permission errors cleared, every image on the site returned 400 in the browser console:
/_next/image?url=https%3A%2F%2Fpub-xxx.r2.dev%2FIMG_8470-scaled.jpg&w=1920&q=75 400 (Bad Request)
Next.js's image optimizer refuses to proxy images from external hostnames unless they're explicitly listed in remotePatterns in next.config.ts. The R2 public URL was never added. Fixed by reading R2_PUBLIC_URL from the environment at startup and extracting the hostname dynamically:
if (process.env.R2_PUBLIC_URL) {
try {
const { hostname } = new URL(process.env.R2_PUBLIC_URL)
remotePatterns.push({ protocol: 'https', hostname })
} catch {}
}
This way the config works for any R2 bucket URL without hardcoding a hostname, and stays valid even if the bucket URL changes later.
Except it didn't work. Deployed, still 400. The reason: next.config.ts is evaluated at next build time, not at next start time. The env var approach assumes the config is read at runtime — it isn't. The config is compiled into the build output. At build time in CI, R2_PUBLIC_URL is not set, so remotePatterns compiles to []. The running container never gets a chance to read the env var. Fixed properly with a static wildcard:
remotePatterns: [
{ protocol: 'https', hostname: '*.r2.dev' },
]
Bug: generate:importmap silently exits 0 without a secret
After the R2 fix triggered a new build, the importMap error came back:
Module not found: Can't resolve './admin/importMap.js'
We had fixed this before by adding npm run generate:importmap to the Dockerfile. It had been working. Now it stopped. The difference: a fresh Docker layer cache meant generate:importmap was actually running again, and this time we could see it was producing no output. The command exits 0 — success — but writes no file.
The cause: payload generate:importmap initialises the Payload config before scanning for components. Payload config initialisation validates PAYLOAD_SECRET. With no secret set in the builder stage, Payload catches the error internally and exits cleanly without generating anything. No error, no file.
Fix: set placeholder env vars in the builder stage:
ENV PAYLOAD_SECRET=build-time-placeholder
ENV DATABASE_URI=postgresql://payload:placeholder@localhost:5432/placeholder
RUN npm run generate:importmap && npm run build
These ENV declarations only exist in the builder stage. The runner stage (FROM node:20-alpine AS runner) starts fresh — it gets real values from .env.local via Docker Compose at container startup. No secrets are baked into the final image.
Social media URLs
All four social links in the header, footer, and recipe page had placeholder handles (herfoodblog) instead of the real ones. Fixed to the correct URLs for Facebook, Pinterest, Instagram, and YouTube.
Round 2: the wildcard that wasn't
After the *.r2.dev fix was deployed, images were still returning 400. The obvious next guess was a stale volume — cleared next_cache, did docker compose pull, came back up. Still 400.
Time to actually diagnose instead of guess. Three questions:
1. Is the R2 image accessible?
curl -I "https://pub-00cfedb2e09340258e621c3f06b2792d.r2.dev/IMG_8470-scaled-800x533.jpg"
HTTP/1.1 200 OK. Image is there.
2. Did the fix actually make it into the running container?
docker compose exec app node -e "const c = require('/app/.next/required-server-files.json'); console.log(JSON.stringify(c.config?.images))"
Output confirmed "remotePatterns":[{"protocol":"https","hostname":"*.r2.dev"}]. Pattern is compiled in.
3. Can the container reach R2?
docker compose exec app wget -q -O /dev/null "https://pub-00cfedb2e09340258e621c3f06b2792d.r2.dev/IMG_8470-scaled-800x533.jpg" && echo "OK"
OK. Container has outbound internet access.
All three pass. The pattern is there, the image is accessible, the container can reach it — and it's still 400. So curl the image optimizer directly to get the actual error text:
curl -s "https://dev.herfoodblog.com/_next/image?url=https%3A%2F%2Fpub-00cfedb2e09340258e621c3f06b2792d.r2.dev%2FIMG_8470-scaled-800x533.jpg&w=1920&q=75"
"url" parameter is not allowed
That's the exact message Next.js returns when the hostname isn't in remotePatterns. The pattern is compiled in but not matching. *.r2.dev isn't working as a wildcard in this version of Next.js.
Fix: swap the wildcard for the exact hostname.
remotePatterns: [
{ protocol: 'https', hostname: 'pub-00cfedb2e09340258e621c3f06b2792d.r2.dev' },
]
New build pushed. Images loaded.
React error #418 — five commits to learn it's impossible in prod, then actually fixing it
With the site up and images loading, the browser console had a new error:
Uncaught Error: Minified React error #418; visit https://react.dev/errors/418?args[]=HTML&args[]=
at rX (react-dom-client.production.js:2898:15)
Error #418 is a hydration mismatch — server-rendered HTML doesn't match what React expects on the client. The args HTML and `` (empty string) point somewhere in the root layout. But the "Minified React" prefix means React's production bundle is active and error messages are stripped to numeric codes. Need the dev build to get a useful component stack.
Attempt 1: Added productionBrowserSourceMaps: true and webpackConfig.optimization.minimize = false to next.config.ts. Source maps started working. Minification did not change.
Attempt 2: React ships pre-built inside node_modules/next/dist/compiled/react-dom/. The client.js entry point checks process.env.NODE_ENV and requires either react-dom-client.production.js or react-dom-client.development.js. Added resolve.alias to redirect react/jsx-runtime and react-dom/server to dev builds. Still react-dom-client.production.js in the stack trace.
Attempt 3: Webpack's DefinePlugin replaces process.env.NODE_ENV at compile time, so the dev branch in client.js is dead-code eliminated before any alias can intercept it. Switched to NormalModuleReplacementPlugin, which fires at module resolution time after DCE:
new webpack.NormalModuleReplacementPlugin(
/react-dom-client\.production\.js$/,
(resource) => {
resource.request = resource.request.replace('production', 'development')
}
)
Still react-dom-client.production.js. Three commits, zero effect.
Root cause: Read the source map files in .next/static/chunks/. Every source entry was prefixed turbopack:///[project]/.... The build uses Turbopack — next build in Next.js 16 defaults to Turbopack. The entire webpack callback in next.config.ts is never called for production builds. Every change made to that callback over the past four commits was silently irrelevant.
Attempt 4: Bypass the bundler entirely — cp the dev file over the production file in the Dockerfile before the build:
RUN cp node_modules/next/dist/compiled/react-dom/cjs/react-dom-client.development.js \
node_modules/next/dist/compiled/react-dom/cjs/react-dom-client.production.js
RUN npm run generate:importmap && npm run build
Build failed immediately: Module not found: Can't resolve '../importMap'. Same symptom as the original importMap bug. Root cause: the dev file wraps its entire body in "production" !== process.env.NODE_ENV && (function() { ... })(). With NODE_ENV=production set in the builder stage, the file is a complete no-op — no exports at all. Payload imports React during generate:importmap initialisation; an empty react-dom module breaks it; it silently exits 0 with no file; npm run build then fails on the missing importMap.
Result: Reverted the cp. The build works again. Getting React dev builds into a Next.js production container is a dead end — React's own dev/prod split is the wall, and there's no clean way around it without rebuilding in NODE_ENV=development which produces a completely different Next.js output. The right tool for debugging this hydration error is npm run dev locally.
Also changed the image tag from :latest to :debug mid-debugging to rule out the server pulling a cached image, and removed the SSH auto-deploy step from the GitHub Actions workflow since server pulls are done manually.
Actually fixing the hydration error
Running locally with npm run dev produced no #418 error at all. That was the clue. AdSense gets 403 on localhost and never initialises. The error is production-only because in production AdSense actually loads.
Error #418 with args HTML and "" means extra DOM nodes found during hydration — not a text mismatch. Tracing the React source confirmed the throw site (popHydrationState line 3010) fires when nextHydratableInstance is non-null after React finishes a component: the real DOM has more children than the virtual DOM expects.
The sequence in production:
- Server renders
<ins class="adsbygoogle">as an empty element - Browser receives HTML, React starts concurrent hydration (scheduled via
MessagePort) - AdSense script loads (even with
lazyOnload, a cached script can initialise quickly) - AdSense auto-scans for
<ins>elements and injects iframe children into them - React's hydration scheduler picks back up and reaches the
<ins>— finds children it didn't expect - Error #418
The race is real because React 19 uses cooperative scheduling. Hydration can be interrupted and resumed across ticks. AdSense doesn't know about React's scheduler.
Fix: render <ins> client-only. Added useState(false) and a useEffect that sets it to true after mount. The component returns null until mounted. Server renders null; initial client render also null; hydration matches perfectly. After mount, <ins> appears and AdSense initialises it normally.
const [mounted, setMounted] = useState(false)
// eslint-disable-next-line react-hooks/set-state-in-effect
useEffect(() => { setMounted(true) }, [])
if (!process.env.NEXT_PUBLIC_ADSENSE_CLIENT_ID || !mounted) return null
// then render <ins>
Also found and removed a duplicate <MobileMenu> in the frontend layout. It was already rendered inside <Header> — the layout was rendering it again as a sibling, producing two hamburger buttons on mobile and an orphan <button> in the SSR HTML.
Deployed, checked production console: #418 gone.
Round 3: still 400 with the exact hostname
Back to the images. Despite the exact hostname in remotePatterns (Bug 9 fix), R2 images were still returning 400. The compiled config confirmed the hostname was there:
{"remotePatterns":[{"protocol":"https","hostname":"pub-00cfedb2e09340258e621c3f06b2792d.r2.dev"}]}
No pathname field. The Next.js image optimizer treats pathname as a required match criterion — without it, no path matches. Fix: add pathname: '/**':
remotePatterns: [
{
protocol: 'https',
hostname: 'pub-00cfedb2e09340258e621c3f06b2792d.r2.dev',
pathname: '/**',
},
],
CI build failure: importMap.js not written
The build triggered by the pathname fix failed immediately:
Error: Turbopack build failed with 3 errors:
Module not found: Can't resolve '../importMap'
Module not found: Can't resolve '../importMap'
Module not found: Can't resolve './admin/importMap.js'
The exact same symptom as the original importMap bug (Bug 2). But that was already fixed — npm run generate:importmap runs in the Dockerfile before npm run build, with PAYLOAD_SECRET and DATABASE_URI set. Why is it back?
Running it locally:
Generating import map
No new imports found, skipping writing import map
payload generate:importmap has a "no new imports found" optimisation — if the file exists and the config hasn't changed, it skips writing. Locally the file is on disk (gitignored but present). In CI the checkout never has it. But the command doesn't check whether the file exists before deciding to skip — it sees no diff from the current state, exits 0, writes nothing. The build then fails because the file is missing.
Fix: remove importMap.js from .gitignore and commit it. The file is deterministically generated from payload.config.ts — it won't drift unless you add a custom Payload component. When you do, run npm run generate:importmap locally and commit the result. This is the same pattern as payload-types.ts: generated, but committed so CI never has to regenerate it.
# After adding a custom Payload component:
npm run generate:importmap
git add "src/app/(payload)/admin/importMap.js"
git commit -m "chore: regenerate importMap.js"
AI flops
The setup doc didn't survive first contact
We reviewed home_server_setup.md carefully in the previous session. It still had a wrong DNS approach (CNAME instead of A record), four wrong env var names, and a CNAME to a non-existent DDNS host for the DEPLOY_HOST GitHub secret. Most of these were caught before SSHing in — but the DATABASE_URI localhost mistake slipped through to runtime.
The Dockerfile had no chown
This is a standard Next.js Docker pattern — the official Next.js Docker example always uses --chown on COPY. We missed it. The error is obvious in hindsight: non-root user, root-owned files, writable directories needed for ISR. Should have been in the original Dockerfile.
The chown fix didn't apply to the existing volume
Added the --chown and rebuilt. Still got the EACCES. Spent time re-checking the Dockerfile before realising the volume was the problem — it was created before the fix and Docker doesn't update it. The lesson: when changing file ownership in a Docker image, any named volumes that overlap with those paths need to be recreated manually. There's no warning; Docker just silently keeps using the old volume.
R2 remotePatterns never added
The next/image optimizer's hostname allowlist requirement is documented, but it's easy to forget when all local dev uses /api/media/file/ (Payload's local storage path, which is in localPatterns). R2 images use a completely different hostname that only appears in production. No test or lint catches a missing remotePatterns entry — you only find out when every image on the live site returns 400.
next.config.ts is build-time, not runtime
The first attempt at the R2 fix read R2_PUBLIC_URL from process.env dynamically and extracted the hostname. Reasonable assumption — the config file is TypeScript, it runs in Node.js, env vars should be available. Wrong. next.config.ts is evaluated once during next build and the result is compiled into the server output. Runtime env vars never reach it. The fix was to use a static wildcard pattern, which is one line and works for any R2 bucket.
generate:importmap swallows its own failure
payload generate:importmap exits 0 even when it can't do its job. If PAYLOAD_SECRET is missing, Payload initialises, catches the validation error internally, and exits cleanly — no output, no error code, no file. The next command in the chain (npm run build) then fails with a confusing module-not-found error that looks identical to the original importMap problem. Took two rounds of the same error to realise generate:importmap was the one failing silently, not the build.
importMap.js is underdocumented
Payload's documentation doesn't prominently explain that importMap.js must be generated before next build in a CI context. It's an auto-generated file that works transparently in local dev (the dev server creates it on startup) but silently breaks Docker builds. The fix is one line, but finding it required reading the error carefully and understanding that Payload generates this file at dev time.
*.r2.dev wildcard compiled in but not matching
The static wildcard *.r2.dev was supposed to be the clean fix — no hardcoded bucket URL, works for any R2 bucket. It wasn't. The pattern appeared correctly in required-server-files.json. The container could reach R2. The image existed. Everything pointed to the fix working. It didn't. Next.js is returning "url parameter is not allowed" which is its exact message for a hostname not in remotePatterns. Whether this is a version-specific bug or the wildcard syntax isn't actually supported for .r2.dev is unclear. The pragmatic fix — exact hostname — took one line and one build.
pathname missing from remotePatterns
After spending a full debugging round confirming the exact hostname was in remotePatterns and images were still 400, the fix was to add one field: pathname: '/**'. The Next.js docs show both hostname and pathname in examples but don't prominently state that omitting pathname will cause all paths to fail to match. The hostname-only entry compiles in without warnings, the optimizer rejects all requests without explaining why the pattern didn't match.
generate:importmap "no new imports" treated as success in CI
payload generate:importmap returned exit code 0 without writing the file. The build then failed with the same Module not found: Can't resolve '../importMap' error as the very first deployment bug. Looked identical to Bug 2 recurring — same error, same symptom — but the cause was different. Bug 2 was PAYLOAD_SECRET missing; Bug 14 was the "no new imports" optimisation treating a non-existent file as up-to-date. Two different failure modes producing the same error message. The tell was running the command locally and seeing "skipping writing import map" in the output — which never appeared in the CI logs because CI only shows the npm run build failure, not the successful-but-silent generate:importmap run.
The root category: importMap.js was treated as a build artifact (gitignored) when it should be treated as a generated source file (committed). The distinction matters when CI does fresh checkouts. payload-types.ts is correctly committed; importMap.js was not; now it is.
Four commits of webpack config changes that never ran
When the hydration error appeared, the assumption was that the production build uses webpack — reasonable, since the webpack callback in next.config.ts has been in the file the whole time and nothing broke. Wrong. Next.js 16 defaults to Turbopack for next build. The webpack callback is only called when webpack is the bundler. Turbopack ignores it entirely, silently.
So three different webpack approaches — optimization.minimize, resolve.alias, NormalModuleReplacementPlugin — were each committed, built, deployed, and confirmed to have no effect, before the actual bundler was identified. The tell was already in the repo: .next/static/chunks/*.js.map source entries all prefixed with turbopack:///[project]/. That file was there from the first local build. Reading it earlier would have saved three round trips through GitHub Actions.
The cp broke the build — React's dev file is a production no-op
After identifying Turbopack as the bundler, the next idea was to bypass all bundler configuration entirely by cp-ing the dev react-dom file over the production one before the build. Seemed solid: the bundler reads the file by path, we give it dev content, done.
Broke the build. The react-dom development file wraps its entire body in "production" !== process.env.NODE_ENV && (function() { ... })(). With NODE_ENV=production set in the Docker builder stage, the file exports nothing. Payload's generate:importmap loads React internals during config initialisation, gets an empty module, silently exits 0, no importMap is generated, and npm run build fails with the same module-not-found error as the very first deployment bug.
Five commits across this debugging arc. The conclusion is boring: you can't get React dev builds in a production Next.js container without changing the entire build mode. Debug hydration errors locally.
AI wins
isBuildPhase pattern is clean
The solution to the PAYLOAD_SECRET problem — checking NEXT_PHASE at the top of src/lib/payload.ts and returning empty data — is four lines that solve the problem permanently for all future builds. No database in CI needed, no build secrets to manage, no changes to the rendering architecture. The site builds without credentials and fills in at runtime.
Caught the deploy path mismatch before it caused a failure
When fixing the PAYLOAD_SECRET build issue, noticed that .github/workflows/deploy.yml still had /opt/herfoodblog after we'd changed the deploy directory to /data/herfoodblog in the docs. Caught and fixed in the same commit. Would have caused a silent deploy failure (the SSH step would fail with "no such directory") that looked like a network issue.
Systematic diagnosis on the persistent 400
When the wildcard fix didn't work, instead of guessing at another fix, stopped and ran three targeted checks: verify the image is accessible on R2, verify the compiled config has the pattern, verify the container has network access. All three passed. That ruled out every plausible cause except the wildcard matching itself. Then curled the /_next/image endpoint directly to get the exact error text. One curl showed "url parameter is not allowed" — the hostname allowlist check, not a network issue, not a cache issue. That made the fix obvious.
End of day
Done: site running at dev.herfoodblog.com with no hydration errors. importMap.js committed, remotePatterns has pathname: '/**', CI build passing. Waiting for server pull to confirm images load.
The honest take: fourteen bugs, two categories. The first ten were all in the gap between local dev and a Docker container running as a non-root user — visible only under real deployment conditions. The next four were harder: five wasted commits trying to get unminified React errors in production (architecturally impossible), the actual hydration fix in one commit, and two more rounds of the same importMap symptom caused by different root causes each time. The recurring lesson from the importMap bugs is that "same error" doesn't mean "same cause" — the error message is Module not found: Can't resolve './admin/importMap.js' every time, but Bug 2 was a missing PAYLOAD_SECRET, Bug 7 was a broken react-dom module, and Bug 14 was a generator that silently skipped because it thought nothing changed. Each time the fix was different.
Comments
Post a Comment