Building a Progressive Web App with Next.js - Part II

This is a two-part article and this is the second part where we integrate Service Workers, purge unused CSS, setup custom server-side caching, etc. in our app. The first part is live here where we create a simple web app using Next.js.

In this part of the tutorial, we start to make our Next.js app progressive in nature. We have gone through everything concerned with the user interface. We have made the components, pages and data fetching logic. You can take a look at the first part of the article if you haven't already or need a refresher. If you are fairly comfortable with React or Next.js, you may prefer to inspect the code instead for a quicker understanding. The repository is available on Github.

Service Worker Registration

We will pick up from where we left off earlier. Let's look at the OfflineSupport component where we register our Service Worker that we will generate during build using Google's Workbox.

OfflineSupport.js

import React, { PureComponent } from 'react';

class OfflineSupport extends PureComponent {
  componentDidMount() {
    if ('serviceWorker' in navigator) {
      navigator.serviceWorker
        .register('/sw.js')
        .then(() => console.log('service worker registered.'))
        .catch(err => console.dir(err));
    }
  }

  render() {
    return null;
  }
}

export default OfflineSupport;

Here, we create a Component by extending React.PureComponent. This is so because effectively we will not be rendering anything in this component and hence don't want to trigger any unnecessary update scenarios for it as well. We return null from its render method to avoid rendering anything. We need to register the Service Worker when this component is mounted, therefore we implement the necessary logic in the componentDidMount lifecycle method. We first check if the browser has support for Service Workers. If it does we try to register the service worker i.e. the sw.js file. If the registration is successful, we console.log a statement reflecting that otherwise, we log an error to the console using the common then-and-catch based flow for Promises. You can also use some notification or toast type component to convey the message here.

Extending Configuration using next.config.js

CSS and Workbox Config

We are done with the user-facing code now. We have also created the Service Worker registration logic. Now, we need to generate the sw.js file that we intend to register in the OfflineSupport component. We will install an npm package next-workbox-webpack-plugin to help simplify using Workbox with Next.js.

You may remember that we imported two CSS files in our Next.js app but Next.js doesn't support CSS bundling by default. We will need to extend its configuration to be able to use CSS. We will use @zeit/next-css package that does it with great ease.

$ npm i -D next-workbox-webpack-plugin @zeit/next-css

Next.js allows custom configuration via next.config.js file under the project root directory. We will use it in our case too. You can either export out a function or an object that will be used to generate new configuration. Our configuration will be very straightforward here, thanks to all these great packages.

@zeit/next-css exports a withCSS function that can be invoked and exported out.

const withCSS = require('@zeit/next-css');module.exports = withCSS();

This will be enough to support using CSS (without CSS Modules, which we are not using here anyway). This config will generate or output a single CSS file to .next/static/style.css which we will serve from /_next/static/style.css in our Header component using a <link>.

But we need to customise the webpack config further to generate our Service Worker code. To do that, we can pass a custom Next.js configuration as an object parameter to the withCSS function.

We will pass it an object with a webpack method that takes in two parameters itself - the first being the default config as object named config and the second one is the options object that we will destructure to get the following values - isServer (true if the config is being run for the  server and false if it is for the client), buildId (a randomly generated ID value for the current Next.js build) and dev (true if it is a development build otherwise false). We will use this destructured variables to conditionally attach custom build configuration.

For example, we use isServer and dev to add the NextWorkboxPlugin instance for generating the Service Worker code only when bundling a production server build.

next.config.js

const withCSS = require('@zeit/next-css');
const NextWorkboxPlugin = require('next-workbox-webpack-plugin');

module.exports = withCSS({
  webpack(config, { isServer, buildId, dev }) {

    const workboxOptions = {
      clientsClaim: true,
      skipWaiting: true,
      globPatterns: ['.next/static/*', '.next/static/commons/*'],
      modifyUrlPrefix: {
        '.next': '/_next',
      },
      runtimeCaching: [
        {
          urlPattern: '/',
          handler: 'networkFirst',
          options: {
            cacheName: 'html-cache',
          },
        },
        {
          urlPattern: /[^3]\/movie\//,
          handler: 'networkFirst',
          options: {
            cacheName: 'html-cache',
          },
        },
        {
          urlPattern: new RegExp('^https://api.themoviedb.org/3/movie'),
          handler: 'staleWhileRevalidate',
          options: {
            cacheName: 'api-cache',
            cacheableResponse: {
              statuses: [200],
            },
          },
        },
        {
          urlPattern: /.*\.(?:png|jpg|jpeg|svg|gif)/,
          handler: 'cacheFirst',
          options: {
            cacheName: 'image-cache',
            cacheableResponse: {
              statuses: [0, 200],
            },
          },
        },
      ],
    };

    if (!isServer && !dev) {
      config.plugins.push(
        new NextWorkboxPlugin({
          buildId,
          ...workboxOptions,
        }),
      );
    }

    return config;
  },
});

For next-workbox-webpack-plugin to work, it needs only one required parameter that is the buildId that we destructured earlier. We can also pass it any valid Workbox config options (i.e. any options we can pass to workbox-webpack-plugin) to further customize our setup.

We store the Workbox related config in a separate variable named workboxOptions and later use Object Spread syntax to pass the values as parameters to the plugin.

We add our custom plugin setup by accessing the config.plugins array which already holds the plugins that are configured by default. We add our extra ones by simply pushing our plugin instance.

Let's go through the options that we are using -

1. clientsClaim- This option determines whether the service worker should start controlling any existing clients as soon as it activates.

2. skipWaiting - This option determines whether the service worker should skip the waiting lifecycle stage.
   
   We set both the options to true to have quicker the iteration of Service Worker updates.

3. globPatterns - Files matching this pattern are included in the precache manifest for pre-caching purposes. 
   
   This is usually not needed as Workbox by default precaches all assets generated by webpack. But while developing this application, it seemed to skip some files like style.css and .js files under the commons directory that's why we have included this option.

4. modifyUrlPrefix - This option is used to modify the precached URL mappings.
   
   We use it to replace .next in the precached URLs to /_next/ because that is the location our app expects the assets to be served from. This only affects the assets precached using the globPatterns options. All other precached assets already have their URLs prefixed correctly with /_next/

5. runtimeCaching - This option accepts an array of objects containing configuration to generate Service Worker code to handle runtime caching. Each object accepts properties like urlPattern (string or RegExp to match requests against that URL), handler (strings corresponding to built-in caching strategies) and options (this object accepts a lot of other properties like cacheName, expiration, cacheableResponse, etc.).
   
   We use it to cache our HTML response for the index (using a string value for urlPattern) and the movie (using RegEx) pages. We implement a networkFirst strategy in which we will try to fetch the response from the network first, failing which we respond using the cached response. The cached response will echo the last successful network response. This is crucial for making our app offline browsable. This is our 'html-cache' as the cacheName suggests.

We also cache the responses from the TMDB API using a staleWhileRevalidate caching strategy. In this strategy, we respond from the cache on request if a cached response is available but also fire off a network request which is used to update the cache. This not only helps to make our offline experience possible but also causes data loading to be faster.

Lastly, we cache all the images we need to display using the cacheFirst strategy because images are very network heavy resources and we don't want to re-fetch them again and again. Also, the images are not likely to update very often. This is the last part that makes our app look closest to its fullest even when offline.

The service worker code and all its related dependencies will be generated inside the workbox directory under the static directory at the project root location. This is taken care of by the plugin itself.

Web App Manifest

To make our into an "installable" PWA, we need to add a Web App Manifest file. It is a simple JSON file (manifest.json) that contains a number key-value pair providing information about the app and how it should behave when "installed" on a user's device. This is the file that is required for the browser to pop that "Add to Home Screen" prompt to the users.

We will use a package named webpack-pwa-manifest for generating the manifest.json file.

$ npm i -D webpack-pwa-manifest

We import the plugin to our next.config.js and push an instance of it to the config.plugins array similar to how we did it for next-workbox-webpack-plugin. We pass it an object as the parameter with properties corresponding to the information that we want in our Web App Manifest like name, short_name, description, icon, background_color, orientation, etc. We also pass in some extra properties that are related to the way the file needs to be generated. For example, we are not using an HTML template or a plugin like html-webpack-plugin so we don't need HTML injection, hence we set inject to false. We also don't want to generate fingerprints, so we set that to false as well. We use the icons array property and use the path module to set the location of our favicon.ico file. The plugin will generate the necessary icon files for each of the sizes we provid in the sizes array. Lastly, the includeDirectory , start_url and publicPath are set to the values that are needed to properly serve the generated files to our app.

Our next.config.js should look something like this now.

const withCSS = require('@zeit/next-css');
const NextWorkboxPlugin = require('next-workbox-webpack-plugin');
const WebpackPwaManifest = require('webpack-pwa-manifest');
const path = require('path');

module.exports = withCSS({
  webpack(config, { isServer, buildId, dev }) {
    
    const workboxOptions = {
      clientsClaim: true,
      skipWaiting: true,
      globPatterns: ['.next/static/*', '.next/static/commons/*'],
      modifyUrlPrefix: {
        '.next': '/_next',
      },
      runtimeCaching: [
        {
          urlPattern: '/',
          handler: 'networkFirst',
          options: {
            cacheName: 'html-cache',
          },
        },
        {
          urlPattern: /[^3]\/movie\//,
          handler: 'networkFirst',
          options: {
            cacheName: 'html-cache',
          },
        },
        {
          urlPattern: new RegExp('^https://api.themoviedb.org/3/movie'),
          handler: 'staleWhileRevalidate',
          options: {
            cacheName: 'api-cache',
            cacheableResponse: {
              statuses: [200],
            },
          },
        },
        {
          urlPattern: /.*\.(?:png|jpg|jpeg|svg|gif)/,
          handler: 'cacheFirst',
          options: {
            cacheName: 'image-cache',
            cacheableResponse: {
              statuses: [0, 200],
            },
          },
        },
      ],
    };

    if (!isServer && !dev) {
      config.plugins.push(
        new NextWorkboxPlugin({
          buildId,
          ...workboxOptions,
        }),
        new WebpackPwaManifest({
          filename: 'static/manifest.json',
          name: 'Next PWA',
          short_name: 'Next-PWA',
          description: 'A Movie browsing PWA using Next.js and Google Workbox',
          background_color: '#ffffff',
          theme_color: '#5755d9',
          display: 'standalone',
          orientation: 'portrait',
          fingerprints: false,
          inject: false,
          start_url: '/',
          ios: {
            'apple-mobile-web-app-title': 'Next-PWA',
            'apple-mobile-web-app-status-bar-style': '#5755d9',
          },
          icons: [
            {
              src: path.resolve('static/favicon.ico'),
              sizes: [96, 128, 192, 256, 384, 512],
              destination: '/static',
            },
          ],
          includeDirectory: true,
          publicPath: '..',
        })
      );
    }

    return config;
  },
});

Purging Unused CSS and Autoprefixing CSS

Since we are using a CSS framework there will be a lot of CSS that we send out to the users but is not being used in our app. To make our app bundle smaller, we want to remove the unused CSS. We can do that easily using packages like purify-css. To make things easier, we will use a super simple webpack loader named css-purify-webpack-loader that uses purify-css to purge unused CSS and enable us to serve leaner CSS bundle.

$ npm i -D css-purify-webpack-loader

We have already seen how we can add extra plugins in our next.config.js. Now, we will take a look at how we can add loaders. This portion is a little tricky. For plugins, we just needed to push the instances of new plugins to the config.plugins array but for loaders, it is a bit different because there can be various loaders for different kind of files. For example, babel-loader for .js files, css-loader for .css files, etc. We need to specifically find the CSS specific setup and add our loader to the existing setup.

Now, config.module.rules is an array that includes all the loaders setup. To find the loader setup for CSS we will run the Array.prototype.find method on the config.module.rules array and check if any of the object items in the array has a RegEx value for the test field that tests true against a .css file name.

This way we will find the CSS specific loader object and then we will push our css-purify-webpack-loader specific data to its use array property.

next.config.js

const withCSS = require('@zeit/next-css');
const NextWorkboxPlugin = require('next-workbox-webpack-plugin');
const WebpackPwaManifest = require('webpack-pwa-manifest');
const path = require('path');

module.exports = withCSS({
  webpack(config, { isServer, buildId, dev }) {
    // Fixes npm packages that depend on `fs` module
    config.node = {
      fs: 'empty',
    };

    if (!isServer) {
      config.module.rules.find(({ test }) => test.test('style.css')).use.push({
        loader: 'css-purify-webpack-loader',
        options: {
          includes: ['./pages/*.js', './components/*.js'],
        },
      });
    }

    const workboxOptions = {
      clientsClaim: true,
      skipWaiting: true,
      globPatterns: ['.next/static/*', '.next/static/commons/*'],
      modifyUrlPrefix: {
        '.next': '/_next',
      },
      runtimeCaching: [
        {
          urlPattern: '/',
          handler: 'networkFirst',
          options: {
            cacheName: 'html-cache',
          },
        },
        {
          urlPattern: /[^3]\/movie\//,
          handler: 'networkFirst',
          options: {
            cacheName: 'html-cache',
          },
        },
        {
          urlPattern: new RegExp('^https://api.themoviedb.org/3/movie'),
          handler: 'staleWhileRevalidate',
          options: {
            cacheName: 'api-cache',
            cacheableResponse: {
              statuses: [200],
            },
          },
        },
        {
          urlPattern: /.*\.(?:png|jpg|jpeg|svg|gif)/,
          handler: 'cacheFirst',
          options: {
            cacheName: 'image-cache',
            cacheableResponse: {
              statuses: [0, 200],
            },
          },
        },
      ],
    };

    if (!isServer && !dev) {
      config.plugins.push(
        new NextWorkboxPlugin({
          buildId,
          ...workboxOptions,
        }),
        new WebpackPwaManifest({
          filename: 'static/manifest.json',
          name: 'Next PWA',
          short_name: 'Next-PWA',
          description: 'A Movie browsing PWA using Next.js and Google Workbox',
          background_color: '#ffffff',
          theme_color: '#5755d9',
          display: 'standalone',
          orientation: 'portrait',
          fingerprints: false,
          inject: false,
          start_url: '/',
          ios: {
            'apple-mobile-web-app-title': 'Next-PWA',
            'apple-mobile-web-app-status-bar-style': '#5755d9',
          },
          icons: [
            {
              src: path.resolve('static/favicon.ico'),
              sizes: [96, 128, 192, 256, 384, 512],
              destination: '/static',
            },
          ],
          includeDirectory: true,
          publicPath: '..',
        })
      );
    }

    return config;
  },
});

postcss.config.js

module.exports = {
  plugins: [require('autoprefixer')],
};

We need to use the glob file pattern for the includes property on the options object to specify the files that need to considered and scanned for selectors, class names, etc. during the CSS purifying process. In our case, it will be the .js files under the components and the pages directory.

$ npm i -D autoprefixer

Finally, we need to add autoprefixer to have better cross-browser support. Since postcss-loader is already setup in Next.js, it will be extremely easy to integrate autoprefixer. All we need to do is create a postcss.config.js file at the project root and add autoprefixer module as plugin in the config object that we export out.

Final structure of the server side webpack config's module object will be something like this:

webpack.config.module.json

{
  "rules": [
    {
      "test": {},
      "include": [
        "/path/to/project/next-pwa"
      ],
      "exclude": {},
      "use": {
        "loader": "next-babel-loader",
        "options": {
          "dev": false,
          "isServer": false
        }
      }
    },
    {
      "test": {},
      "use": [
        {
          "loader": "/path/to/project/next-pwa/node_modules/extract-text-webpack-plugin/dist/loader.js",
          "options": {
            "id": 1,
            "omit": 0,
            "remove": true
          }
        },
        {
          "loader": "css-loader",
          "options": {
            "modules": false,
            "minimize": true,
            "sourceMap": false,
            "importLoaders": 1
          }
        },
        {
          "loader": "postcss-loader",
          "options": {
            "config": {
              "path": "/path/to/project/next-pwa/postcss.config.js"
            }
          }
        },
        {
          "loader": "css-purify-webpack-loader",
          "options": {
            "includes": [
              "./pages/*.js",
              "./components/*.js"
            ]
          }
        }
      ]
    }
  ]
}

Creating Custom Server and ssr-caching

Last but not the least, we will add the custom server for supporting all the customizations we have added to our Next.js app and sprinkle in some performance boost using ssr-caching using the lru-cache package. We will use express to build the server.

$ npm i express lru-cache

First, we will import all the packages we will need on the server which includes express, next, lru-cache and the join method from the built-in path module.

Then, we set our PORT and dev variable for the server port number and the flag indicating development or production build respectively. Next, we create a Next.js server invoking the next method with the dev flag passed to it as an object parameter. We also set up the handle variable with the return value of app.getRequestHandler method.

We also create a ssrCache object by instantiating a new cache object with max cache limit set to 20 and max expiration age set to 5 minutes.

server.js

const { join } = require('path');
const express = require('express');
const next = require('next');
const cache = require('lru-cache'); // for using least-recently-used based caching

const PORT = 8000;
const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev });
const handle = app.getRequestHandler();

const ssrCache = new cache({
  max: 20, // not more than 20 results will be cached
  maxAge: 1000 * 60 * 5, // 5 mins
});

app.prepare().then(() => {
  const server = express();

  server.get('/', (req, res) => {
    renderAndCache(req, res, '/');
  });

  server.get('/movie/:id', (req, res) => {
    const queryParams = { id: req.params.id };
    renderAndCache(req, res, '/movie', queryParams);
  });

  server.get('*', (req, res) => {
    if (req.url.includes('/sw')) {
      const filePath = join(__dirname, 'static', 'workbox', 'sw.js');
      app.serveStatic(req, res, filePath);
    } else if (req.url.startsWith('static/workbox/')) {
      app.serveStatic(req, res, join(__dirname, req.url));
    } else {
      handle(req, res, req.url);
    }
  });

  server.listen(PORT, err => {
    if (err) throw err;
    console.log(`> Live @ https://localhost:${PORT}`);
  });
});

async function renderAndCache(req, res, pagePath, queryParams) {
  const key = req.url;

  // if page is in cache, server from cache
  if (ssrCache.has(key)) {
    res.setHeader('x-cache', 'HIT');
    res.send(ssrCache.get(key));
    return;
  }

  try {
    // if not in cache, render the page into HTML
    const html = await app.renderToHTML(req, res, pagePath, queryParams);

    // if something wrong with the request, let's skip the cache
    if (res.statusCode !== 200) {
      res.send(html);
      return;
    }

    ssrCache.set(key, html);

    res.setHeader('x-cache', 'MISS');
    res.send(html);
  } catch (err) {
    app.renderError(err, req, res, pagePath, queryParams);
  }
}

We use this ssrCache object for caching the render results of the pages so that subsequent requests can be served up faster. We create an async function renderAndCache that is responsible for handling the serving of the page responses either from the cache or by rendering. We will get into how it works after we discuss the route based request handling logic first.

We start to write the request handling logic by creating an express server. We handle the home route "/" by sending the response we get from invoking renderAndCache using req, res and pagePath. We handle the movie route bit differently because it is a parameterized route. Also, if you remember Next.js expects the parameterized route to have a structure like this /movie?id=1 i.e. using query parameters but we want to prettify it as /movie/1. Therefore, we extract the parameter from the end of the path and pass it on to the renderAndCache method in the form of query parameters along with the usual req, res and pagePath parameters.

Lastly, we implement a match all route to handle the serving of Service Worker specific files from the static directory at the project root as static assets using the app.serveStatic method available on the Next.js server instance. We leave all the unattended routes to be handled by the Next.js server's built-in request handler that we stored in the handle variable earlier.

Now, let's come back to the renderAndCache method which accepts a req, res, pagePath and queryParams parameter. When it is invoked for a movie route queryParams is used to understand which specific movie page needs to be rendered otherwise it is undefined and of no purpose for the rendering of the home route. We first use the url on the req object as key to check if any cache exists for the URL. If it does exist, we send back the cache as response and add a header key "x-cache" with the value "HIT" representing the response was from the cache. Otherwise, inside a try block, we invoke the asynchronous renderToHTML method on the app instance to render the page and its constituent React components into HTML. We then store the rendered response in cache with the url as key and send it back as the response with "MISS" as the value for the "x-cache" header key this time. If any error occurs inside the try block during the rendering, we return the Next.js default error component instead of using the renderError method.

Conclusion

Our Progressive Web App is ready, so let's audit it.

This is the audit result. As you can see we are still lagging a bit behind on the performance front. This article was meant to provide a base from which you can build your own PWAs with ease. The goal was to introduce you to the aspects that you need to focus on, features that you need to implement and libraries or packages that may help you get things done fast when building PWAs. There is always room for improvement. So, for your own experimentation, you can try to improve the performance score of the app. One easy performance gain would come from removing the render-blocking CSS. PWAs are amazing so we all must take a closer look at them and make our apps more progressive.