Sal's

I published my drafts: lessons learned and steps taken

As I mentioned, I accidentally published all my draft posts. You know, the ones I cheekily listed at the bottom of my ripgrep post. All public and in my RSS feed. Dangit.

What happened

I can’t know for sure, but here’s what I suspect happened.

Sloppiness on my part

When Hugo builds my production site, it puts the files in the public/ subdirectory. I then upload that subdir to Cloudflare Pages with wrangler.

When Hugo runs its local dev server, which renders all my draft posts, it puts the files in the public/ subdirectory.

See where I’m going with this?

I have to assume that, when I ran my production deploy, the public/ folder was dirty with the dev server’s version of the site. Talk about a rookie mistake.

I’d been keeping the server running on purpose because I ran a pre-deploy check that scraped the site for broken links. I think I’d been passively assuming that the dev server stuck its build somewhere else, and I was too buried in all the Hugo learnings to think too hard on it. Plus, everything had been working fine for the first couple days, so it slipped off my list of concerns.

Rollbacks are nice

I was struggling to get my fix deployed (see The plot thickens section below), but fortunately the Cloudflare dashboard came to my rescue. It has a history of my deployments, and with a click I can instantly roll the site back to any of the recent ones. I found the latest one that didn’t have my drafts, and rolled back to it.

Adding guardrails

I then spent a lot of time adding scripts and automation to prevent this sort of thing from happening again.

This is all part of my Makefile now, so make deploy_staging and make deploy_prod automatically do the cleaning, building, double checking, and deploying.

The plot thickens: Cloudflare Pages not updating

At this point I’m thoroughly gunshy about deploying to prod given my recent misadventures, so I used my staging deployment to test all these checks out. And I found a stubborn problem: broken files in staging weren’t getting updated when I’d redeploy.

For example, the <link rel="alternate"... element in my topmost index.html file — the element that tells RSS readers where to find my RSS feed — had localhost as its URL domain. And no matter how many times I re-deployed the corrected site, it wouldn’t change. I tried wrangler’s --skip-caching flag. I tried purging the cache in the Cloudflare Dash. I checked my local public/index.html probably over thirty times to confirm it was correct. I tried downgrading wrangler to an older version, since it’s had several updates recently. I had a dozen conversations with Gemini about it.

No matter what, each time I’d deploy, my staging site’s homepage content wouldn’t budge.

Eventually I tried deploying the site to a net-new Cloudflare Pages preview site. The new site did not have the problem. Okay, good, I’m not crazy.

Then I shoved some changes into my homepage and re-deployed to staging. Aha! That fixed the problem in staging.

So, I suspect I’ve hit some bug in how Cloudflare manages updates. I assume the deploy logic thinks it already has the latest version of the file despite the fact that the local one has different content. But if I make additional changes to the local file, it seems I can unstick it.

To address this, I added one more script to my deployment pipeline. This script adds a comment with a timestamp to the bottom of my public/index.html and public/index.xml files after they are built by hugo. The comments look like this:

<!-- build 1786050421 -->

They should be invisible to the reader, but in theory, they will nudge the Cloudflare logic to update those two pages on every deploy.

This seems like a reasonable time to cite the old classic:

There are only two hard things in Computer Science: cache invalidation and naming things. — Phil Karlton (via Martin Fowler)

Back to GitHub-triggered deploys?

In the midst of all this, I said screw it, this is too wonky and time consuming, I’ll just go back to having Cloudflare build the site from pushes to GitHub.

I set all that up, but the build was failing because Cloudflare Workers uses an older hugo binary. I set the HUGO_VERSION env var as guided by the Cloudflare docs, but it still used an old version. Then I remembered I’d also have to get dart-sass installed in the worker for it to be able to compile my CSS. Gah.

So I gave up on this and went back to solving the problem above.

Maybe things work now

My Makefile is now labyrinthian. It might work. Who knows.

And to you bloggers who use Bear, Micro.blog, Write.as, Pure Blog, etc. … you may now nod and smile in smug satisfaction with your choices.

(Although, this sort of firefighting is kinda fun. Once I the fixing-it part, anyway.)

#Hugo #This-Blog