Skip to main content

How to configure Content Sync

This guide is going to use Azure Blob Storage as the target provider. As of version 4, Uniform for Sitecore officially supports Azure Blob Storage and Tencent Cloud Object Storage (COS) as targets. That being said, adding a new provider is not a huge undertaking and can be done as an extension to Uniform for Sitecore. Contact support if you need to make it work on a platform that we do not support at the moment.

Prerequisites#

  1. You have Uniform Connector installed and configured.
  2. You completed the How to create a Site Configuration guide.
  3. You completed the How to add a Deployment Service to a Site Configuration guide.
  4. An account with Azure with a valid subscription.

Introduction#

The purpose of the Content Sync capability is to remove the need for the Static Site Generation (SSG) process (depicted below as "2. run SSG") to make direct calls to the Sitecore instance, where the Uniform APIs as hosted. This is how the process works without Content Sync enabled:

/img/how-to-configure-content-sync-azure-blob-storage/Uniform_API_fetch_without_content_sync.png

When the Content Sync feature is enabled, the Static Site Generation (SSG) process can fetch from the file storage outside of Sitecore:

/img/how-to-configure-content-sync-azure-blob-storage/Uniform_API_fetch_with_contentsync.png

Incremental Content Sync#

Ability to update Content Sync storage incrementally requires the Incremental Static Site Generation feature to be configured. Once that is in place, Content Sync will follow the same rules as described in the Incremental Static Site Generation guide. The only difference is the location of the timestamp that stores the last successful datetime of the Content Sync operation. In this case, it will be stored along with the synced content files in a file named .content.timestamp.txt.

Benefits of Content Sync#

  1. The most obvious one is that your Sitecore instance can be behind a firewall, which is quite common if you use a Sitecore Content Management role as the source for static builds. This capability can be quite important if you are running the builds that perform static site generation inside a managed cloud build environment (using Netlify or Vercel, for example) and your infrastructure settings do not allow for direct access to the Sitecore Content Management server.
  2. Reduce the load on the Sitecore instance during the Static Site Generation phase on large sites. When running such an operation on large sites, this process can be quite taxing on a Sitecore server.
  3. Speed up the SSG phase. When running exports on thousands of pages, it is much faster to go against static file storage than a live Sitecore instance. When using Next.js, this approach allows to increase the number of threads for the static site generation process and achieve much faster export times.
  4. Content Sync can be configured in incremental mode, allowing to sync only pages that are changed during the publishing process. Check this guide for more info.
  5. Allow developers to work against a snapshot copy of the Uniform APIs without connectivity to a Sitecore instance, which speeds up development.

Supported configurations#

  1. As of version 4, Content Sync is only supported for MVC-based solutions, where Uniform Page Service is used.
  2. As of version 4, you can push to Azure Blob Storage and Tencent Cloud Object Storage (COS).

How is Content Sync triggered?#

There are three ways to trigger Content Sync:

  1. From GUI (usually used during initial configuration)
  2. After Sitecore publishing (after the Deployment Service is configured).
  3. By running export of your app. When Content Sync is configured, making a request to a Uniform API will reload its synced state in configured storage.

See the "How to trigger Content Sync" chapter below for more info.

Configuring Content Sync#

First of all, let's create a storage account in your Azure subscription and one container to host the files of the Content Sync operation. We will need the name of the container and the value of the Connection String from the "Access keys" tab of your Storage Account. If you do not have yet an Azure storage that you are going to use for Content Sync, you can create it by following Step 1. Create an Azure Blob Storage from this article.

Using Azure Blob Storage#

Option 1: use item-based configuration#

  1. Find your Site Configuration item in master database under /sitecore/system/Uniform/SiteConfigurations

    If you haven't configured any Site Configuration yet, you can follow Defining a Site Configuration using Sitecore items instructions.

  2. After you've created a Site Configuration item, find an item Configuration under it, and observe different Insert Options for this item. For Content Sync, you will need to select "Configure: Azure Blob for Pages [AzureBlobPageContentService]":

    /img/how-to-configure-content-sync-azure-blob-storage/Untitled1.png

  3. Provide the name of the container created earlier and the paste in the value of the connection string:

    /img/how-to-configure-content-sync-azure-blob-storage/Untitled2.png

    We do not recommend storing the connection string value on the item. Instead, you can provision an environment variable on the machine where your Sitecore instance runs and reference it in your field using the following syntax: $(env:ENVIRONMENT-VARIABLE-NAME)

    For example:

    /img/how-to-configure-content-sync-azure-blob-storage/Untitled3.png

    This definition will work in the same way, as long as you have an environment variable with a name UNIFORM_CONTENT_SYNC_CONNSTRING.

  4. Verify Content Sync configuration by visiting https://your-sitecore-instance/sitecore/admin/uniform?siteConfiguration=your-site-configuration-name. If everything is configured correctly, you should be able to see your newly added configuration in the response:

    <siteConfiguration xmlns:d1p1="http://www.sitecore.net/xmlconfig/" name="uniform-mvc-kit" d1p1:source="{5C1F5684-5E68-4492-8DDA-8504C64055F1}">   <site inherits="uniform-mvc-kit" d1p1:source="{5C1F5684-5E68-4492-8DDA-8504C64055F1}" />      <ContentSync>         <PageContentSyncService ref="uniform/services/azureBlobPageContentSyncService" d1p1:source="{5356B3CC-9728-45F8-BEC8-95BCB92DE859}">            <ContainerName>content</ContainerName>            <ConnectionString>your-connection-string</ConnectionString>         </PageContentSyncService>      </ContentSync></siteConfiguration>

  5. Trigger the Content Sync operation and verify it is successful (using GUI is easiest). See the "How to trigger Content Sync" chapter below.

Option 2: use config file-based configuration#

  1. Create a Sitecore config patch file.

  2. The contents of this file must add a new <contentSync /> section for your site configuration under <sitecore>/<uniform>/<siteConfigurations>/<siteConfiguration name="your-site-name"> The referencing type fo the pageContentSyncService needs to point to the Azure Blob implementation: uniform/services/azureBlobPageContentSyncService:

    <contentSync>    <pageContentSyncService set:ref="uniform/services/azureBlobPageContentSyncService">        <ContainerName>content</ContainerName>        <ConnectionString>connection-string</ConnectionString>     </pageContentSyncService></contentSync>

    The full config file patch for site name "uniform-mvc-kit" may look like this:

    <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">    <sitecore>        <uniform>            <siteConfigurations>                <siteConfiguration name="uniform-mvc-kit">                    <contentSync>                        <pageContentSyncService set:ref="uniform/services/azureBlobPageContentSyncService">                            <ContainerName>content</ContainerName>                            <ConnectionString>connection-string</ConnectionString>                        </pageContentSyncService>                    </contentSync>                </siteConfiguration>            </siteConfigurations>        </uniform>    </sitecore></configuration>

    We do not recommend storing the connection string values in configuration, instead, provision an environment variable on the machine where your Sitecore instance runs and reference it in your field using the following syntax: $(env:ENVIRONMENT-VARIABLE-NAME)

    <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">    <sitecore>        <uniform>            <siteConfigurations>                <siteConfiguration name="uniform-mvc-kit">                    <contentSync>                        <pageContentSyncService set:ref="uniform/services/azureBlobPageContentSyncService">                            <ContainerName>$(env:CONTENTSYNC_CONTAINER_NAME)</ContainerName>                            <ConnectionString>$(env:CONTENTSYNC_CONNECTION_STRING)</ConnectionString>                        </pageContentSyncService>                    </contentSync>                </siteConfiguration>            </siteConfigurations>        </uniform>    </sitecore></configuration>

  3. Verify https://your-sitecore-instance/sitecore/admin/uniform?siteConfiguration=your-site-name if everything is configured correctly, you should be able to see your newly added configuration in the response:

    <siteConfiguration xmlns:d1p1="http://www.sitecore.net/xmlconfig/" name="uniform-mvc-kit" d1p1:source="{5C1F5684-5E68-4492-8DDA-8504C64055F1}">   <site inherits="uniform-mvc-kit" d1p1:source="{5C1F5684-5E68-4492-8DDA-8504C64055F1}" />      <ContentSync>         <PageContentSyncService ref="uniform/services/azureBlobPageContentSyncService" d1p1:source="{5356B3CC-9728-45F8-BEC8-95BCB92DE859}">            <ContainerName>content</ContainerName>            <ConnectionString>your-connection-string</ConnectionString>         </PageContentSyncService>      </ContentSync></siteConfiguration>

  4. Trigger Initial Content Sync operation and verify it is successful. See the "How to trigger Content Sync" chapter below.

How to trigger Content Sync#

Option 1: from GUI in Content Editor#

  1. Activate the Uniform tab in the Content Editor ribbon by right-clicking the ribbon and selecting Uniform.

  2. Locate your site configuration in Content Editor under /sitecore/system/Uniform/SiteConfigurations. After you've selected it in the content tree, run "Initial Content Sync":

    /img/how-to-configure-content-sync-azure-blob-storage/Untitled4.png

  3. This operation will create a Content Sync history item and open the tab with an app displaying the progress of the operation:

    /img/how-to-configure-content-sync-azure-blob-storage/Untitled5.png

Option 2: from the publishing pipeline#

Content Sync is intended to be triggered at the end of the Sitecore Publishing pipeline execution.

This requires the Deployment Service to be configured. Make sure that you have configured your Deployment Service using this guide.

We recommend using this approach during release time (by triggering Sitecore publishing from your CI/CD pipeline) and during normal operation of the Content Management server, picking up changes in the Sitecore publishing pipeline and syncing those into Azure Blob Storage.

Option 3: by running static export of your app#

When Content Sync is enabled, requests to Uniform content APIs (Map and Page services) will always reload the synced copies of the output in target blob storage. Try making the following requests and see the corresponding files in blob storage being updated:

  • https://sitecore-host/uniform/api/content/<your-site-name>/map?uniform_token=<your-token>
  • https://sitecore-host/uniform/api/content/<your-site-name>/page?uniform_token=<your-token>

Steps:

  1. Make sure to disable preview mode in your Next.js app by setting the UNIFORM_OPTIONS_PREVIEW=0 environment variable.
  2. Make sure the UNIFORM_API_URL is pointing to your Sitecore instance: UNIFORM_API_URL=https://your-sitecore-instance
  3. Run npm run export
  4. This will populate the Azure Blob Storage container with the latest content from Map and Page service.

This approach is not compatible with the incremental Content Sync, as the timestamp will not be written into the target Blob Storage.

Verifying that Content Sync works#

After you've triggered Content Sync via any of 3 options above, open Sitecore logs and verify that there are no errors or warnings associated with the Content Sync operation and that the timestamp is written into the blob storage at the end of the operation:

Writing timestamp '2021-03-20 21:27:41.821', URL: https://ABC.blob.core.windows.net/content/uniform/api/content/yoursite/.content.timestamp.

ManagedPoolThread #11 14:27:42 INFO  ==  UNIFORM  ==   Scope.Pages(site: "uniform-mvc-kit"): 541348 14:27:42 INFO  ==  UNIFORM  ==   Scope.Media(site: "uniform-mvc-kit"): 0692: 23476 14:27:42 INFO  ==  UNIFORM  ==   Thread #0: Prefetching page 0/5, site: uniform-mvc-kit, path: 693: 26228 14:27:42 INFO  ==  UNIFORM  ==   Thread #2: Prefetching page 2/5, site: uniform-mvc-kit, path: thanks694: 22040 14:27:42 INFO  ==  UNIFORM  ==   Thread #1: Prefetching page 1/5, site: uniform-mvc-kit, path: contact735: 23476 14:27:43 INFO  ==  UNIFORM  ==   Thread #0: Completed - prefetching item 0/5, site: uniform-mvc-kit, path: 736: 23476 14:27:43 INFO  ==  UNIFORM  ==   Thread #0: Prefetching page 4/5, site: uniform-mvc-kit, path: album748: 26228 14:27:43 INFO  ==  UNIFORM  ==   Thread #2: Completed - prefetching item 2/5, site: uniform-mvc-kit, path: thanks749: 22040 14:27:43 INFO  ==  UNIFORM  ==   Thread #1: Completed - prefetching item 1/5, site: uniform-mvc-kit, path: contact750: 22040 14:27:43 INFO  ==  UNIFORM  ==   Thread #1: Prefetching page 5/5, site: uniform-mvc-kit, path: login755: 23476 14:27:43 INFO  ==  UNIFORM  ==   Thread #0: Completed - prefetching item 4/5, site: uniform-mvc-kit, path: album759: 22040 14:27:43 INFO  ==  UNIFORM  ==   Thread #1: Completed - prefetching item 5/5, site: uniform-mvc-kit, path: login761: ManagedPoolThread #11 14:27:43 INFO  ==  UNIFORM  ==   Writing timestamp '2021-03-20 21:27:41.821', URL: https://uniformdocs.blob.core.windows.net/content/uniform/api/content/uniform-mvc-kit/.content.timestamp.txt

Configure Content Sync on publishing trigger#

Content Sync operation is tied to Sitecore publishing, which requires your site to have Deployment Service configured for a given site configuration. Learn more about Site Configurations here.

If the deployment service is configured:

  1. Ensure your deployment service is up and running (if it's self hosted, start up your next.js app)
  2. Trigger a publish operation (single item republish would work).

    If you do not have Incremental Static Site Generation enabled, any publishing process will trigger full Content Sync for all configured sites. It is therefore highly recommended to enable incremental mode for production.

  3. After publishing, the following log messages will appear on the last page of the Publish wizard, indicating that Content Sync took place and all pages for your given site are exported:
Job started: Publish to 'web'INFO [Deploy] Checking all sites sequentially (1).INFO Fetching URI: http://localhost:3000/uniform/api/service/status?uniform_token=12345INFO Uniform service is ready, version: 4.0.210309-6Items created: 0Items deleted: 0Items updated: 1Items skipped: 1INFO [Deploy] Checking all sites sequentially (1).INFO [Deploy] Starting deployment for site "uniform-mvc-kit"WARN Cannot retrieve timestamp, URL: https://uniformdocs.blob.core.windows.net/content/uniform/api/content/uniform-mvc-kit/.content.timestamp.txtINFO Fetching URI: https://unfrmtestcm.dev.local/sitecore/service/keepalive.aspxINFO Fetching URI: https://unfrmtestcm.dev.local/uniform/api/content/uniform-mvc-kit/map?uniform_token=12345INFO GetMediaScope(site: "uniform-mvc-kit"): Media container could not be resolvedINFO Clearing all Uniform memory cachesINFO Starting content and media deployment for site "uniform-mvc-kit"INFO Scope.Pages(site: "uniform-mvc-kit"): 5- /- /album- /contact- /login- /thanksINFO Prefetch is in progressINFO Writing timestamp '2021-03-20 22:20:15.942', URL: https://uniformdocs.blob.core.windows.net/content/uniform/api/content/uniform-mvc-kit/.content.timestamp.txtINFO Fetching URI: http://localhost:3000/uniform/api/service/status?uniform_token=12345INFO Uniform service is ready, version: 4.0.210309-6INFO Fetching URI: http://localhost:3000/uniform/api/deployment/status/f?uniform_token=12345...INFO Fetching URI: http://localhost:3000/uniform/api/deployment/status/f?uniform_token=12345INFO The site has been deployed: uniform-mvc-kitJob ended: Publish to 'web' (units processed: 2)

Advanced settings#

Adjusting thread count#

Consider adjusting the number of threads for Content Sync operation, this may speed up the operation dramatically. Beware that depending on how taxing the page rendering is, setting this to a high value can cause more harm than good, overwhelming your Sitecore instance:

<setting name="Uniform.ContentSync.PrefetchData.ThreadsCount" value="4" />

Adjusting the amount of retries#

Content Sync operation is performing page rendering. If your Sitecore instance is unstable, it may take a few retries to get a response. By default the retry count is 10, which should be sufficient, but if you see the timeouts in the logs, feel free to adjust this setting: 

<setting name="Uniform.ContentSync.PrefetchData.RetryCount" value="10" />

FAQ#

Q: Can Content Sync update files incrementally?#

A: Content Sync is not configured as incremental by default since this requires the Incremental Static Site Generation to be configured. Learn more here.

Q: Do I need to trigger sync after a new release?#

A: Most likely yes, if the release changes the presentation of your site (updated markup, updated CSS, then you must re-run Content Sync together with your next Sitecore release).

Q: Can Content Sync support a different target hosting platform (AWS S3, etc.)?#

A: The implementation of the content sync provider is extensible, so you are able to create the one that works with your platform. Contact support for more information.

Q: How fast is Content Sync?#

A: This depends on how fast your Sitecore instance can render pages, the state of your HTML cache, the number of pages you have, and how many concurrent threads your Sitecore instance can support. This can take seconds, or minutes, depending on all these variables.

Q: Can I trigger Content Sync via API?#

A: This is not supported out of the box. We recommend triggering Sitecore publish that initiates Content Sync.

Q: How do I fully re-sync content for my Site Configuration?#

A: This can be done by deleting the /uniform/api/content/[site-configuration-name] folder and/or the /.content.timestamp.txt file from your blob container and then running any publishing operation that results in at least one item being published (single item publish of the home page would do).

What's next?#

You can now run static export against the Content Sync'd Azure Blob Storage. Once you configured Content Sync and after the initial sync is successful, you can adjust the value of UNIFORM_API_URLto your Azure Storage Container Blob URL (e.g. https://ABC.blob.core.windows.net/content), meaning that next time static site export runs, it can go against that blob storage, not your Sitecore instance.

  1. Configure Media Sync using this guide.