Updating public Next.js environment variables without rebuilds

Photo by Ibrahim Boran

Introduction

If you've ever tried building a Next.js app with environment variables you've probably run into unexpected behavior of one kind or another. One of the most common gotchas of Next.js is how environment variables are handled during build-time vs run time, particularly public variables. This means that not all environment variables are evaluated when you think they are, and changing certain variables can have no effect at all.

Here we offer a slightly hacky yet workable solution that allows modifying public environment variables at runtime without having to rebuild your app. We've adapted this solution from approaches used by PostHog and Cal.com.

Build time vs Runtime environment variables in Next.js

Next.js supports both build time and runtime environment variables which can be accessed by the Node.js server. This allows us to reference an env var in our server-side code, and it will behave as expected. Lets say we have an env var MY_VALUE=foo and the following code in our Next app:

const value = process.env.MY_VALUE

Building our application and then updating MY_VALUE=bar at runtime will mean value will be bar, which is the expected behavior.

However, this is not the case for public environment variables. These are environment variables prefixed with NEXT_PUBLIC_ and made available to client-side code that can run in a user's browser. Let's say we have a public env var NEXT_PUBLIC_VALUE=foo at build time, and the following code in our app:

const value = process.env.NEXT_PUBLIC_VALUE

This time, the value of the env var is evaluated only at build time as foo and inlined into our JavaScript bundle. This means updating this value after the app is built has no effect. Updating our env to NEXT_PUBLIC_VALUE=bar does nothing, and const value will be foo. This is problematic for a number of reasons, not least of which is that it exacerbates the confusion between server and client code, but that is a discussion for another time.

The primary issue with how Next.js evaluates public env vars is that you need to rebuild your entire app every time you want to change them, which almost defeats the purpose of environment variables to begin with. You cannot build once, deploy many because each user running your app may need different public env var values, but the build process effectively hardcodes them into your JavaScript bundle. You can't take a single build and run it in multiple environments like staging and production either, since again you may need different env vars for things like OAuth client IDs, API endpoints etc, and so you are forced to rebuild your entire app for each deployment environment.

The solution

We are going to get around this problem by employing a combination of placeholder strings and a bash script to find-and-replace values in our compiled code before our app starts. This will allow us to build our app once but run it with arbitrary public environment variables.

Step 1: Build the app with placeholder public variables.

In this example, we are building our app in a Docker container. We're going to add placeholder variables to be used at build-time. We are using strings prefixed with BAKED_ so that we can identify them later. In this example, we have 2 public environment variables: NEXT_PUBLIC_BACKEND_API_URL and NEXT_PUBLIC_GITHUB_CLIENT_ID

# Set placeholder environment variables in Dockerfile
ARG NEXT_PUBLIC_BACKEND_API_URL=BAKED_NEXT_PUBLIC_BACKEND_API_URL
ARG NEXT_PUBLIC_GITHUB_CLIENT_ID=BAKED_NEXT_PUBLIC_GITHUB_CLIENT_ID

# Build our app
RUN yarn build

The strings BAKED_NEXT_PUBLIC_BACKEND_API_URL and BAKED_NEXT_PUBLIC_GITHUB_CLIENT_ID will now be inlined into our complied JavaScript in all instances where we reference process.env.NEXT_PUBLIC_BACKEND_API_URL or process.env.NEXT_PUBLIC_GITHUB_CLIENT_ID respectively.

Step 2: Replace hardcoded placeholders with real variables at runtime.

Now that we have hardcoded values in place of our public environment variables prefixed with BAKED_, all we need to do is a simple find-and-replace. Here's a bash script that does just that:

#!/bin/bash
# replace-variables.sh

# Define a list of environment variables to check and replace
VARIABLES=("NEXT_PUBLIC_BACKEND_API_URL" "NEXT_PUBLIC_GITHUB_CLIENT_ID")

# Check if each variable is set
for VAR in "${VARIABLES[@]}"; do
    if [ -z "${!VAR}" ]; then
        echo "$VAR is not set. Please set it and rerun the script."
        exit 1
    fi
done

# Find and replace BAKED values with real values
find /app/public /app/.next -type f -name "*.js" |
while read file; do
    for VAR in "${VARIABLES[@]}"; do
        sed -i "s|BAKED_$VAR|${!VAR}|g" "$file"
    done
done

We'll call this script replace-variables.sh. It's going to first check that the real environment variables NEXT_PUBLIC_BACKEND_API_URL and NEXT_PUBLIC_GITHUB_CLIENT_ID exist, and then replace our hardcoded values with the real ones in the compiled JavaScript in the /public and /.next folders.

When copy and pasting the script above, make sure to replace the VARIABLES list with your actual public env var names.

Step 3: Integrating the script with the start process

Now all that's left to do it make sure this script runs before we start our Next.js app. We'll add a start.sh script to take care of that:

#!/bin/sh
# start.sh

# Replace runtime env vars and start next server
sh scripts/replace-variable.sh && 
yarn start

Let's make sure we start our Docker container using start.sh:

CMD ["/app/scripts/start.sh"]

That's it! Now when we start up our container with docker run we will automatically replace our pre-baked public environment variables with the real ones at runtime. This allows us to pre-build our app image once and distribute it via an image registry like DockerHub to be run with arbitrary public env vars.

Benefits of this approach

The main upside of this approach is avoiding multiple rebuilds of our app, and being able to easily distribute and re-use a pre-built app container. Not only will you save on CI runs and build minutes, but if you're looking to avoid vendor lock-in with Vercel then this solution should help make your build pipeline simpler.

This also arguably makes handling public environment variables more intuitive, since we can reference these env vars in our code without having to worry about at what stage of the build or deployment they will be evaluated at.

Potential limitations and considerations

The heart of this fix is a bash script that is doing a find-and-replace on our entire compiled JavaScript, so there is always the chance that something is inadvertently replaced and breaks our application code. Choosing unique variables names and prefixes will help mitigate this risk for the most part, but it's important to keep in mind.

This script is also going to increase the start-up time of our containers slightly, but that is well worth the trade-off when saving hours on app rebuilds in my opinion.

Advanced use cases

This approach can also be adapted for more advanced use-cases. In the Phase Console, we needed to support optional env vars that may not be required for users who are self-hosting, so we have modified replace-variables.sh to process 2 separate lists of variables: MANDATORY_VARS and OPTIONAL_VARS.

#!/bin/bash
# replace-variables.sh

# Define a list of mandatory environment variables to check
MANDATORY_VARS=("NEXT_PUBLIC_BACKEND_API_BASE" "NEXT_PUBLIC_NEXTAUTH_PROVIDERS")

# Define a list of optional environment variables (no check needed)
OPTIONAL_VARS=("APP_HOST" "NEXT_PUBLIC_POSTHOG_KEY" "NEXT_PUBLIC_POSTHOG_HOST" "NEXT_PUBLIC_GITHUB_INTEGRATION_CLIENT_ID")

# Infer NEXT_PUBLIC_APP_HOST from APP_HOST if not already set
if [ -z "$NEXT_PUBLIC_APP_HOST" ] && [ ! -z "$APP_HOST" ]; then
    export NEXT_PUBLIC_APP_HOST="$APP_HOST"
fi

# Check if each mandatory variable is set
for VAR in "${MANDATORY_VARS[@]}"; do
    if [ -z "${!VAR}" ]; then
        echo "$VAR is not set. Please set it and rerun the script."
        exit 1
    fi
done

# Combine mandatory and optional variables for replacement
ALL_VARS=("${MANDATORY_VARS[@]}" "${OPTIONAL_VARS[@]}")

# Add NEXT_PUBLIC_APP_HOST to the list for replacement
ALL_VARS+=("NEXT_PUBLIC_APP_HOST")

# Find and replace BAKED values with real values
find /app/public /app/.next -type f -name "*.js" |
while read file; do
    for VAR in "${ALL_VARS[@]}"; do
        if [ ! -z "${!VAR}" ]; then
            sed -i "s|BAKED_$VAR|${!VAR}|g" "$file"
        fi
    done
done

If you would like any help adapting this approach for your needs, or have any questions, feedback or comments for us, feel free to reach out to us on GitHub or Slack.