How to configure incremental static site generation

Introduction#

When content changes inside a CMS it is very rare that all pages will be affected. If your site contains a large number of pages, the ability to identify specific pages which have been affected will make a clear impact on costs and productivity. A Jamstack architecture needs to maintain a full static representation of your site and so Uniform provides the ability to generate these changed pages without building the entire site.

Without this mechanism in place, any change in the CMS will have to trigger full site re-generation. This can be quite an expensive process, from compute/resource and time perspective. Business users are used to seeing the effect of publishing in seconds, not in minutes or hours.

This is especially true for most Sitecore implementations, where the process of page re-generation involves fetching the page-level content along with all the datasource content and render all HTML fragments (if MVC mode is used). Depending on the amount of page-level items, complexity and efficiency of the existing presentation layer, this process could significantly delay the static site generation process and become adoption blocker for the Jamstack architecture. This is why Uniform for Sitecore implements the Incremental Static Site Generation (ISSG) capability.

This capability is not enabled by default since it always requires a solution-specific configuration. This guide provides the walkthrough of how to enable and configure this capability.

In contrast with the Incremental Static Regeneration feature of Next.js, which relies on the caching strategy commonly known as “stale-while-revalidate” and requires Node servers to re-generate pages after cache expires, the native ISSR capability of Uniform works differently. It does not require the special infrastructure capabilities to support this, will only regenerate when content is actually changed and therefore simplifying the delivery layer of your app by removing the need of this additional layer of cache management.

Supported configurations#

  • This feature is supported for both MVC and JSS-based Sitecore solutions.
  • In terms of the Deployment Service configuration, only the Self-hosted Deployment Service is supported at the moment.

If you are running a managed Deployment Service in Netlify, this feature is not supported due to the atomic and immutable deploys of Netlify, read more about it here. In this case, we recommend configuring Incremental Content Sync, which handles the most of the heavy lifting associated with fetching content from Sitecore incrementally, which allows to run last mile HTML static site generation against a fast blob storage using parallel threads, which scales well for large content volume solutions.

Prerequisites#

  1. You must have the Deployment Service configured using the Self-hosted Deployment Service option. Follow this guide if you haven't done it yet.
  2. Access to Sitecore instance and ability to deploy new configuration files.

Phases of the Static Site Generation#

In order to understand how this capability works, it is also important to understand these different phases of a Static Site Generation: Build, Export, and Deploy:

  • Build phase: responsible for compiling all application code into a bundle. The bundle typically contains front-end artifacts - .js, .css files, etc. This resulting bundle is what runs the the rendering of the app during the (Static) Export phase and during app rehydration client-side.

    You need a new build and therefore, a new bundle each time a code is changed in source control, each time you release.

    When using Next.js, this phase is executed during next build command.

  • Export phase: is where the build-time static site generation runs. It takes the application bundle produced during the Build phase, and combines data from Sitecore CMS (fetched via Uniform APIs), producing the "last mile output" - the HTML files containing pre-rendered pages along with the references to the .js and .css artifacts from the Build phase.

    You can run the Export phase multiple times independently from the Build phase (as long as the Build phase is performed at least once). Also, as long as the .next folder is persisted, the Build phase can be run multiple times without generating new .js and .css artifacts.

    When using Next.js, this phase is executed with next build. In Next.js version 10, however, the Export phase also happens during the next build command execution, and next export phase simply copies the necessary files to the output folder in such a way that it can be served by a simple file server. It is still useful to think about these are two distinct phases, even though the way these phases are triggered in recent versions of Next.js is different.

  • Deploy phase: this phase combines the artifacts of the Build and Export phases, which is the whole statically generated app, and deploys it to a target delivery environment (typically, a CDN, or a combination of a Blob Storage and CDN).

/img/how-to-configure-issg/build-export-deploy.png

When the Incremental Static Site Generation capability of Uniform is configured and there are any page-level item changes detected after the Sitecore Publishing executed, the Build phase is skipped (since the app code artifacts haven't changed) and the Export phase is performed only for the pages that were actually changed. The Deploy phase takes the resulting HTML files and redeploys them to the target hosting environment.

How does the dependent page detection work?#

The job of providing which pages were changed and therefore need to be regenerated is the delegated to the the Uniform Connector that is listening to the Sitecore Publishing events and initiates the Incremental Static Site Generation:

/img/how-to-configure-issg/incremental_static_site_generation.png

Since some publishing operations may change items that are not necessarily pages, there is an extensible and configurable process in place to identify which dependent pages need to be re-exported.

Here is a more detailed flow of how this mechanism works:

  1. The process is kicked off at the end of the Sitecore Publishing operation by the TriggerDeployment processor of the Sitecore publish pipeline.

  2. Matching Site Configuration is resolved based on the Publishing operation specifics - the language, the target database. If no matching Site Configuration is found, this process skips any further work.

  3. If the Deployment Scope service is configured as incremental alternative logic of page detection will kick in:

    1. The timestamp of the previous incremental static site export is resolved. This timestamp is written into a .timestamp.txt file during static site deployment and is fetched from the location where the site is hosted from, for example: https://container-name.z##.web.core.windows.net/.timestamp.txt

      If the timestamp is not found, the process will skip incremental processing and resolve all page-level items that are included into the scope of a given Site Configuration.

    2. If the timestamp is found, retrieve all items that have been changed in the target web database since the last time this operation was executed.

      Conceptually, think about it as a SQL query being executed on the target database of the publish operation that was executed, similar to this one: SELECT [ID] FROM [Items] WHERE [Updated] ≥ 'last-timestamp' The actual implementation is somewhat different than this.

    3. For each of the changed items, the page-level items that are dependent on these items are resolved. This is done by the Uniform Connector executing the getDependentPages pipeline, described here in greater detail.

      Due to solution specific nature of this area, this pipeline will likely require some tweaking. Consult the Common Use Cases below for more info.

  4. Pass the list of resolved page-level item paths to the Deployment Service to kick off the Incremental Static Site Generation (ISSG) process . If the incremental mode is not configured (which is the default), this process will re-export all the pages in a given Site Configuration.

  5. The changes pages are re-generated and the new timestamp is written as .timestamp.txt (since v4.0.1, the timestamp file is generated in separate folder) and deployed along with the rest of the files after the static file upload succeeds.

Configuration#

The incremental mode can only be configured via configuration files and cannot be configured via Sitecore items.

Step 1: configure incrementalDeploymentScopeService#

Add <deploymentScopeService set:ref="uniform/services/incrementalDeploymentScopeService" /> into your Site Configuration's <deployment /> element:

<configuration xmlns:set="<http://www.sitecore.net/xmlconfig/set/>">
<sitecore>
<uniform>
<siteConfigurations>
<siteConfiguration name="mysite">
<deployment>
<deploymentScopeService set:ref="uniform/services/incrementalDeploymentScopeService" />
</deployment>
</siteConfiguration>
</siteConfigurations>
</uniform>
</sitecore>
</configuration>

Notice that set: suffix was used in front of a few attributes. It is a best practice, making the order in which the config files are loaded by Sitecore not important.

Step 2: configure deploymentStateService#

In order to be able to retrieve the timestamp of previous site deployment, add the following element under your Site Configuration's <deployment /> element:

<deploymentStateService set:ref="uniform/services/webClientDeploymentStateService">
<PublicUrl>https://www.mysite.com</PublicUrl>
</deploymentStateService>

The value within <PublicUrl /> has to correspond to the location where your site is being deployed. For example, if it is deployed to Azure Blob Storage, the value will look something like this:

<PublicUrl>https://your-container-name.zAA.web.core.windows.net</PublicUrl>

The complete set of additions to the site configuration will look like this:

<configuration xmlns:set="<http://www.sitecore.net/xmlconfig/set/>">
<sitecore>
<uniform>
<siteConfigurations>
<siteConfiguration name="mysite">
<deployment>
<deploymentScopeService set:ref="uniform/services/incrementalDeploymentScopeService" />
<deploymentStateService set:ref="uniform/services/webClientDeploymentStateService">
<PublicUrl>https://www.mysite.com</PublicUrl>
</deploymentStateService>
</deployment>
</siteConfiguration>
</siteConfigurations>
</uniform>
</sitecore>
</configuration>

Step 3: Tweak getDependentPages pipeline (if needed)#

Review the Developer Reference here that describes the getDependentPages pipeline. Consult the Common Use Cases section below to identify if your solution needs further configuration tweaks, which can either be a matter of adjusting the values of default processors or implementing your own processors.

The importance of the Sitecore LinkDatabase#

The Sitecore LinkDatabase and its health is an important factor in ensuring the Incremental SSG works as expected. Most logic in dependent page detection relies on the LinkDatabase to resolve the page-level item dependencies on datasource items that get published while the page-level item itself remains intact.

It is therefore essential to keep the LinkDatabase healthy and monitor log files for any exceptions related to the LinkDatabase. We also recommend rebuilding it periodically during your maintenance windows and after each new release of any back-end functionality and run the Broken Links Report to spot any broken links that need fixing.

Besides that, additional measures that you can take place:

  1. Prevent pages with broken links from being approved via workflow by using the standard Sitecore workflow action.
  2. Make sure the Sitecore business users are aware that when the Breaking Links dialog appears after item deletion, they must not leave the item links broken as this could leave to unexpected results.

When Incremental rules must be overridden#

Some changes made after publishing may require a full re-export. For example, changes in global navigation items, footer links, etc. Changes that typically require full cache clear of your presentation layer. Detection of these cases is possible with the feature called FullDeploymentRules, allowing to force full static site generation when a certain condition is met.

Configuring FullDeploymentRules#

  1. Locate the <deploymentScopeService /> element under your site configuration's <deployment /> section.

  2. Add the <fullDeploymentRules hint="raw:AddFullDeploymentRule"></fullDeploymentRules> block.

  3. Add at least one rule from the list of supported ones, for example:

    <sitecore>
    <uniform>
    <siteConfigurations>
    <siteConfiguration name="uniform-mvc-kit">
    <deployment>
    <deploymentScopeService ref="uniform/services/incrementalDeploymentScopeService" singleInstance="false">
    <fullDeploymentRules hint="raw:AddFullDeploymentRule">
    <ItemIdRule1 type="Uniform.ExcludeRules.ItemIdRule, Uniform.Content.Sitecore" id="{A4446E41-9DE2-446C-A3FE-C9986A255A9E}" />
    </fullDeploymentRules>
    ...

The name of the XML element (ItemIdRule1 in the example above) must be unique and the value doesn't matter. The most important part is the type attribute, as it defines the type of the rule and other attributes that specify values for a given rule.

If any of the configured rules match at least one item in the scope, the full static site generation will be triggered.

Types of rules avilable:

  • Item ID based rule, where the id attribute must contain the item id.

    <ItemIdRule1 type="Uniform.ExcludeRules.ItemIdRule, Uniform.Content.Sitecore" id="{A4446E41-9DE2-446C-A3FE-C9986A255A9E}" />
  • Field Value rule that checks a particular field value on the item.

    • where the field attribute must contain either a field id or a field name
    • where the value attribute must contain the string value of the field.
    <FieldValueRule1 type="Uniform.ExcludeRules.FieldValueRule, Uniform.Content.Sitecore" field="{830EB23C-A667-4BDF-8E67-C93DDE710728}" value="My Albums" />

Creating a custom rule#

  1. Add implementation of Uniform.ExcludeRules.IIncludeExcludeRule interface (sourced from Uniform.Abstractions Nuget package) and implement the bool Applies(Item item) method.

    namespace HabitatUniform.FilteringRules
    {
    public class IsNavigationItem : Uniform.ExcludeRules.IIncludeExcludeRule
    {
    public bool Applies(Item item)
    {
    // add your business logic here
    return true;
    }
    public void Init(XmlNode configNode)
    {
    // add any parameter parsing here
    }
    }
    }
  2. Register the implementation in configuration:

    <deploymentScopeService ref="uniform/services/incrementalDeploymentScopeService" singleInstance="false">
    <fullDeploymentRules hint="raw:AddFullDeploymentRule">
    <NavigationItems type="HabitatUniform.FilteringRules.IsNavigationItem, Habitat" />
    </fullDeploymentRules>
    </deploymentScopeService>

Common Use Cases#

Consider the following Common Use Cases below, some of these require modification of the getDependentPages pipeline, some don't.

In any case, it is worth remembering that in order for the Incremental Static Site Generation to take place, there must be at least one change detected in target database after publishing. Sometimes a Sitecore Publish operation may not actually end up modifying any items in the target database. This means that there will be no changes to process, so the Incremental Static Site Generation will be skipped in this case.

Use Case 1: When a shared datasource item is changed#

Let's imagine a fairly common use case where a global item Global Hero that is referenced by both Home and Album page below via a data source link established on the layout definition of both page-level items:

/img/how-to-configure-issg/Untitled.png

The Navigate → Links button allows seeing the two page-level items referring to Global Hero:

/img/how-to-configure-issg/Untitled1.png

After modifying the Global Hero and publishing it (no matter via what kind of publishing), the standard implementation of the getDependentPages pipeline, specifically, the CheckLinkDatabaseReferrers processor will facilitate the discovery of both page-level items and adding them to the scope of the next static site generation process.

If the Deployment Service is configured correctly, you should be able to see the following messages during the publishing operation in your Next.js application console, indicating that both Home and Album page-level items were re-exported after the Global Hero item was published:

03/03-13:54:32 info: Starting export for 2 page(s):
/album
/
...

This use case does not require any customizations, the only requirement is a healthy state of the Sitecore LinkDatabase.

Use Case 2: When a datasource child item is changed#

Expanding on the Use Case 1, sometimes a datasource item is a composite of mutliple items. This is typical for components, such as carousel, galleries, accordions, etc.

Consider the following example:

  1. The Home page has a "Slideshow" rendering on the layout that references the Main slideshow datasource item:

    /img/how-to-configure-issg/Untitled2.png

    The "Main slideshow" datasource item actually doesn't have any fields.

  2. It's the child items (Slide 1 and Slide 2) that carry the content for the Slideshow component, so these are the items that are typically changed:

    /img/how-to-configure-issg/Untitled3.png

The problem#

If the Slide 1 item is changed, there is no direct reference from any page-level item, so without some tweaks it is not possible to detect that the Home page needs to be re-exported if any of the slide items are changed during publishing.

The solution#

Create a config patch for the existing CheckParentItemLinkDatabaseReferrers processor (see it's description here), where you need to register your own AncestorByTemplateLocator locator:

<processor type="Uniform.Pipelines.GetDependentPages.CheckParentItemLinkDatabaseReferrers, Uniform.Deployment.Incremental">
<param desc="mapNodeService" ref="uniform/services/mapNodeService" />
<abortIfFound>false</abortIfFound>
<Locators hint="raw:AddLocator">
<slideshow
type="Uniform.Pipelines.GetDependentPages.EvaluatedItemLocators.AncestorByTemplateLocator, Uniform.Deployment.Incremental"
itemtemplateid="{TEMPLATE-ID-OF-GALERY-CHILD-ITEM}"
parenttemplateid="{TEMPLATE-ID-OF-GALERY-FOLDER}" />
</Locators>
</processor>

This locator will activate only if the itemtemplateid matches the template id of the Slide item:

/img/how-to-configure-issg/Untitled4.png

and will look up the ancestor item that is expected to have the reference in LinkDatabase to the page-level items using the value of the parenttemplateid attribute:

<slideshow
type="Uniform.Pipelines.GetDependentPages.EvaluatedItemLocators.AncestorByTemplateLocator, Uniform.Deployment.Incremental"
itemtemplateid="{TEMPLATE-ID-OF-GALERY-CHILD-ITEM}"
parenttemplateid="{TEMPLATE-ID-OF-GALERY-FOLDER}" />

The complete config patch could look like this, where itemtemplateid is the the template id of the Slide items and parenttemplateid is the template id of the Main slideshow item.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
<sitecore>
<pipelines>
<group name="uniform" groupName="uniform">
<pipelines>
<getDependentPages>
<processor type="Uniform.Pipelines.GetDependentPages.CheckParentItemLinkDatabaseReferrers, Uniform.Deployment.Incremental">
<Locators hint="raw:AddLocator">
<slideshow type="Uniform.Pipelines.GetDependentPages.EvaluatedItemLocators.AncestorByTemplateLocator, Uniform.Deployment.Incremental"
itemtemplateid="{ADF1D613-BEE4-44D0-9B6F-32261DE37E35}"
parenttemplateid="{A87A00B1-E6DB-45AB-8B54-636FEC3B5523}" />
</Locators>
</processor>
</getDependentPages>
</pipelines>
</group>
</pipelines>
</sitecore>
</configuration>

It's worth mentioning that the ancestor lookup will not stop until it finds the matching item by parenttemplateid meaning that if you have another container on the path, the logic will skip it if it's template id doesn't match and the Main slideshow will be located:

/img/how-to-configure-issg/Untitled5.png

Another common scenario is Sitecore Forms, where the changes may apply to the form subitems, however it is the ancestor form item (Contact us) that is linked to page-level items:

/img/how-to-configure-issg/Untitled6.png

In this case additional locator configuration will look like this:

  • itemtemplateid is the id of the /sitecore/templates/System/Forms/Fields/Text template.
  • parenttemplateid is the id of the /sitecore/templates/System/Forms/Form template
<processor type="Uniform.Pipelines.GetDependentPages.CheckParentItemLinkDatabaseReferrers, Uniform.Deployment.Incremental">
<Locators hint="raw:AddLocator">
<form-text-field type="Uniform.Pipelines.GetDependentPages.EvaluatedItemLocators.AncestorByTemplateLocator, Uniform.Deployment.Incremental"
itemtemplateid="{FC18F915-EAC6-460A-8777-6E1376A9EA09}"
parenttemplateid="{6ABEE1F2-4AB4-47F0-AD8B-BDB36F37F64C}" />
</Locators>
</processor>

Use Case 3: When a child item under a known folder is changed#

Consider another common scenario. An item called Some settings item stored in a local component content folder. This item doesn't have a reference in LinkDatabase from the ancestor page-level item Home.

/img/how-to-configure-issg/Untitled7.png

Even if there is no entry in LinkDatabase, no problem! We can use the CheckIfLocalPageDatasource processor, which accept the LocalPageDatasourceFolderTemplate which will be the template ID of the Data item that stores all local content for our pages. This means that there is any item changed that is stored in this folder, the parent page-level item (Home in this case) will be added to the list of Dependent Pages.

<processor type="Uniform.Pipelines.GetDependentPages.CheckIfLocalPageDatasource, Uniform.Deployment.Incremental">
<abortIfFound>false</abortIfFound>
<LocalPageDatasourceFolderTemplate>{A87A00B1-E6DB-45AB-8B54-636FEC3B5523}</LocalPageDatasourceFolderTemplate>
</processor>

Full config patch could look like this:

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
<sitecore>
<pipelines>
<group name="uniform" groupName="uniform">
<pipelines>
<getDependentPages>
<processor type="Uniform.Pipelines.GetDependentPages.CheckIfLocalPageDatasource, Uniform.Deployment.Incremental">
<abortIfFound>false</abortIfFound>
<LocalPageDatasourceFolderTemplate>{A87A00B1-E6DB-45AB-8B54-636FEC3B5523}</LocalPageDatasourceFolderTemplate>
</processor>
</getDependentPages>
</pipelines>
</group>
</pipelines>
</sitecore>
</configuration>

Use Case 4: When an item is changed that has a rendering with a code reference to#

Let's imagine we have a configuration item Chatbot Setting that stores some presentation settings for a given rendering called Chatbot.

/img/how-to-configure-issg/Untitled8.png

/img/how-to-configure-issg/Untitled9.png

The reference to this item is established in the presentation code, so even Sitecore is not aware of this dependency.

This is a use case for a custom getDependentPages pipeline processor. Check the reference guide for more info on how to implement it.

The job of this processor will be to check if the args.EvaluatedItem matches the criteria (by inspecting it's template id for example) and then using Sitecore APIs (LinkDatabase for example or search query) to locate all pages that use the Chatbot rendering on the layout. Those pages will be added to args.DependentPages list, and the rest will be taken care of the Uniform Connector infrastructure.