Adventures in Sitecore Partial Html Cache Clearing

Sitecore offers us the HtmlCacheClearer and the HtmlCacheClearAgent. The former will clear the entire HTML cache for sites defined within the handlers sites property at the end of a publish, the latter is disabled by default in the scheduler but if enabled will clear the entire HTML cache for every site hosted in the solution based on the interval you specify.

Neither of these are particularly desirable. Clearing the entire HTML cache at the end of every publish can massively reduce the performance boost you can gain from using HTML cache if publishing is a regular activity. It also clears caches for every site configured regardless if there were changes to the sites content or not.

Clearing the entire HTML cache for every site on a schedule has some advantages in that we can hold onto the cache for longer (based on the interval you set, which can be loosely tied to acceptable delays in seeing content changes go live) but again it’s indiscriminate in that it clears the cache whether the content has changed or not. Scheduled clearance can also create a “hill effect” in CPU usage on your content delivery servers.

CPU-Hills

The spikes you can see above are a direct consequence of setting the HTML cache to clear every 30 minutes for a site that makes heavy use of HTML cache. This can be compounded if you have multiple content delivery servers all clearing at the same time and it’s extremely difficult (perhaps impossible) to synchronise the interval based scheduler across delivery servers to try and offset them so this solution also does not scale very well at all.

And what about deployments, if we go down to a single delivery server while upgrading the other the spikes after cache clearance can get pretty uncomfortable.

Well the good news is we can get rid of this problem by extending Sitecore so that it only clears the HTML cache for the sections of the site we specify, within the sites we specify and when we specify.

First I’ll point out how to access the HTML cache for a given site:

// skipping null checks for brevity
var cache = Factory.GetSiteInfo("siteName").HtmlCache;

HtmlCache inherits from Sitecore.Caching.CustomCache and if you’ve ever created your own cache before by inheriting from CustomCache you will know that you need to generate a cache key to add an entry.

Sitecore generates this cache key for every rendering in the mvc.renderRendering pipeline and GenerateCacheKey processor and the cache key is a concatenation of (depending on the caching settings on the rendering item):

  • Controller name
  • MVC area
  • Data source path
  • Language
  • Querystring
  • Rendering parameters
  • Device

And so on. If you want to see all of the keys that are currently in the HTML cache to understand the format you can call:


var keys = cache.InnerCache.GetCacheKeys();

If you want to customize the way cache keys are generated or add additional context to the cache keys you can replace Sitecore’s GenerateCacheKey processor with your own. For example if you are working with dynamic pages (or virtual urls) and want to be able to cache the renderings on these pages even though they don’t have a traditional data source to vary by, you could do something like:


public class GenerateDynamicCacheKey : GenerateCacheKey
{
    protected override string GenerateKey(Rendering rendering, RenderRenderingArgs args)
    {
        var key = base.GenerateKey(rendering, args);

        // get a dynamic identifier i.e. product id

        return AppendProductId(key);
    }
}

Now we can set renderings to cacheable on dynamic pages which vary by the product being displayed.

This is where things get interesting. As well as being able to clear the entire HTML cache for a site by calling cache.Clear()CustomCache, which HtmlCache inherits from exposes two additional methods:

  1. public virtual void RemoveKeysContaining(string value);
  2. public virtual void RemovePrefix(string prefix);

The names of these methods should make it fairly self-explanatory what they do and if we think back to the example of appending the product id to the end of the cache key well now we’re in a position where we could add a handler to the publish:end:remote event that extracts the item just published, check to see if it’s a product and if so remove the HTML cache entries with keys containing the product id:

protected void OnPublishEnd(object sender, EventArgs args)
{
    // null checks skipped for brevity
    var item = (Event.ExtractParameter(args, 0) as Publisher).Options.RootItem;

    if (item.IsProduct()) // extension method
    {
        Factory.GetSiteInfo("siteName").HtmlCache.RemoveKeysContaining(item["Product Id"]);
    }
}

So now we can hold on to the precious HTML cache for all of our product pages and only clear the cache for individual products when they’re updated.

Conversely we may still want to clear all non-product related caches on a schedule. First let’s take a look at what RemoveKeysContaining actually does. Well it calls InnerCache.RemoveKeysContaining(value). InnerCache is a property that implements ICache, which turns out to be Sitecore.Caching.Cache, which in turns inherits from Cache. The RemoveKeysContaining method in Sitecore.Caching.Cache looks like this:


public void RemoveKeysContaining(string keyPart)
{
    base.Remove((string key) => key.IndexOf(keyPart, StringComparison.InvariantCultureIgnoreCase) > -1);
}

So essentially RemoveKeysContaining is really a helper method on the Cache class, which internally creates a predicate that gets passed into the Remove method of the generic cache class it inherits from.

Ok, great, so this means we can create our own predicate. Well how about when we generate our own cache keys as described earlier we add an identifier to all product related caches $”{productId}.product“.

This would enable us to do something like the following in our scheduled, non-product related cache clearer:

// agent registered in the scheduling section of the Sitecore config
// siteName(s) to clear could be defined in the agent config
public void Run
{
    var cache = Factory.GetSiteInfo("siteName").HtmlCache;
cache.InnerCache.Remove((string key) => !key.EndsWith(".product"));
}

That simple. So now we have individual product page caches clearing only when the product is modified and non-product related caches clearing based on an interval.

Hopefully it’s evident now that using a combination of:

  1. Customised HTML cache keys
  2. Custom predicates for removing specific entries based on their key

You can really begin to fine tune your HTML cache clearing strategy to to clear entries only when it’s absolutely necessary.

Happy optimization!

Advertisements

Sitecore Publishing Service Field Casing

There is a bug confirmed by Sitecore support for version 2.2.1 (revision 180807) of the publishing service, where changes to fields where the only change is the casing of the field value are not promoted to publishing targets. This is a bit of an edge case but it can manifest itself if for example you are storing class/assembly names and newing up objects using Sitecore’s ReflectionUtil:

var instance = ReflectionUtil.CreateObject(
                item["Assembly"],
                item["Class"],
                new object[] { }) as IMyInterface;

Now let’s say someone went against class naming conventions and created a class called JSONParser and then down the line that class got renamed to JsonParser, well the publishing service would not promote that change to your publishing targets and object instantiation on the delivery servers would begin to fail.

The issue is due to the collation of the database (Latin1_General_CI_AS).

They did supply a working patch that is pretty straight forward to apply, you only need to modify the publishing service itself (1 dll, 1 xml config). Support ticket number 290996.

5 Ways to Learn Sitecore Development

Even though Sitecore Symposium was 3 weeks ago I’m still processing the experience and the takeaways I got from the event, every year is always different in this regard. One of the things that struck me this year was the broad levels of experience within the development community. Many of us have been doing this for over 10 years, some of us 15+ but as the platform continues to grow and thrive there’s a whole wave of developers relatively new to the platform coming through. Which basically inspired this back to basics guide on what I find to be the best ways to grow your Sitecore development skills. If you’re getting started or looking to upgrade those skills, I hope this helps!

1. Learn from Sitecore

So basically Sitecore is built on Sitecore, which means chances are the customisation/extension that you are trying to build already has solid examples in terms of technical approach and best practices that Sitecore are using themselves. This is true of:

  • Pipelines
  • Event handlers
  • Scheduled tasks
  • DI patterns
  • Ribbon buttons
  • Workflow actions
  • Patch config files
  • Rules engine conditions and actions

This is certainly not an exhaustive list and of course this means that a decompiler is your friend. Planning on extending a pipeline with a custom processor? Take a look at an existing processor in the pipeline, what is it doing? What is it NOT doing? What kind of properties are available on the pipeline args?

2. Patch All The Things

Or to put it another way, know what you changed and have an easy way to turn it off if you need to. It can’t be stressed enough that any modifications you make to Sitecore should be applied to Sitecore using patch files. This has huge benefits in terms of troubleshooting, change control, environment specific transforms, transparency, upgrades and so on. There are plenty of resources out there that break down Sitecore’s patching syntax. Why is this about learning Sitecore development? Because starting off with best practices is a lot easier than “un-learning” bad practices and modifying Sitecore’s configuration files directly is up there with worst of bad practices.

3. Master Sitecore YouTube Channel

We all learn in different ways, some people are visual, some prefer to read and of course some prefer to digest video content. There are so many great ways to consume Sitecore training materials in video format including:

  • The Sitecore Virtual User Group Conference
  • Live streams of user groups from all over the globe
  • Sitecore’s official YouTube channel

I single out Master Sitecore because it seems increasingly that video content from other sources is being aggregated on this channel, for example live recordings from this years SUGCON in Berlin. I highly recommend you check out some of the material on here as it’s contributed from a wide range of sources in the community.

4. Sitecore Documentation

Ok, this might seem like an obvious thing to mention BUT Sitecore have come on leaps and bounds in the last few years in the quality of the documentation they provide and we all acknowledge the awesome work being done by Martina Welander to make documentation a first class citizen at Sitecore. What’s interesting is that as well as the general documentation that’s available on doc.sitecore.net there are “documentation portals” for specialised subjects such as:

And the information provided in those is definately well worth a read. We have also recently had the promise from Sitecore that the move to two releases a year means that every release will be a “complete” release, which includes full documentation.

One other thing to pay some attention to from a documentation perspective is the release notes on dev.sitecore.net for the core CMS and optional modules, this in combination with the known issues sections can provide some much needed insight to what you can expect from the platform.

5. The Sitecore Community!

Last but by no means least, if you haven’t noticed already Sitecore has an incredible network of community contributors all over the globe on various channels:

  • Individual blogs
  • Sitecore Stack Exchange
  • Twitter (#Sitecore, #SitecoreUG, #SitecoreSym, #SitecoreMVP…)
  • Open source projects
  • Slack
  • User groups
  • Conferences

The insights being shared by the community are invaluable, get involved! One way I found to get involved in the community initially was to think of parts of Sitecore that I wanted to learn more about and build a small module that does something useful and uses those parts of the platform… and then.. well, share it. It’s a good confidence booster, which can also launch you into things like speaking at user groups. The community is always looking to hear from new speakers (as well as the familiar faces).

Happy Sitecore… ing.

processItem Pipeline in Relation to FXM

I’ve talked a lot recently about the processItem pipeline and how it can be extended to automatically associate site visitors to Sitecore profiles without having to tag all of your page items in the content tree, in fact it was the theme of my Sitecore Symposium talk in Orlando.

What I have discovered fairly recently is that FXM (Federated Experience Manager) uses the exact same pipeline to register goal conversions, page events etc. on external sites. If you haven’t heard of FXM before, essentially FXM allows you to do “Sitecore like things” on sites that are not running on Sitecore i.e.

  • Track user activity
  • Inject Sitecore managed content into external sites
  • Personalise content injected from Sitecore on external sites

Not an exhaustive list but you get the point. It’s not a silver bullet however and it’s far from perfect but let’s say you want to use the FXM APIs in a non-website context, let’s say mobile apps, to track user activity and goal conversions or do the same kind of profile mapping that I talk about in my Symposium talk. Well, because FXM uses the processItem pipeline within a page tracking context you absolutely can, even though FXM has no back-end UIs that will allow users to manage tracking in a non-website context.

So how exactly? Well firstly when you create an external site in FXM you end up with a “Domain Matcher” item, which defines mappings between the external sites host name and the FXM site it relates to and language mapping rules. This allows you to tell Sitecore how to intepret FXM requests from external sites with the freedom to define how the external site provides that information, could be in the request headers for example.

fxm-domain-matcher

As you begin to set “filters” for the external site using the FXM version of the experience editor (not available for mobile apps of course), for example where the external site address being tracked equals a specific page then register the goal conversion “subscribed to newsletter”, filter items are created as children of the domain matcher. The goals to register etc. are then stored against the tracking field of the child filter item.

The RegisterEventsForMatchedPagesProcessor within the tracking.matchpages pipeline defined in Sitecore.FXM.config then basically iterates over the child filter items that match the current FXM external site tracking request, creates an instance of ProcessItemArgs and passes it into the processItem pipeline.

fxm-events-processor

So again, it would be a relatively trivial thing to extend either the FXM tracking.matchpages or processItem pipeline to read from a repository of rules based profile map items to associate the current visitor with Sitecore profiles even within a non-Sitecore and non-website context.

I am currently working on a project where we will quite likely take this approach, I will update this post with my findings.

Happy profiling!

Sitecore Symposium 2018 Slides

symposium-2018

I had a great time presenting my talk “Taxonomy & Rules Based Profile Mapping” at this years Sitecore Symposium in Orlando, the best comment I had after the talk was a gentleman who came over after and said “thanks so much, that was one of the best developer talks of the conference so far”, it really validated all the hard work and consideration that goes into these things.

I’ve uploaded the slides from the talk for your reference as a pptx so you have the notes, get your download on:

Taxonomy & Rule Based Profile Mapping

The link to the GitHub repository containing the source code is in the slides but if you just want that you can grab it here. I still need to work on the documentation but it’s coming.

It’s been two years in a row speaking at Symposium (last year was mobile app development with the Sitecore Xamarin SDK), let’s see if I can make it three (wherever it may be)!

Sitecore EntityService with ItemService Style Token Authorization

Sitecore’s ItemService and EntityService have been around for a while now and there is a fair amount of community content out there (and Sitecore’s documentation) that describe the role of the two service layers and how to configure them so I won’t reinvent those wheels here.

What I have recently come to discover however is that the EntityService, whilst providing better support for working with custom business objects and writing custom WebAPI controllers does not support authorization out-of-the-box. The settings in Sitecore.Services.Client.config that relate to authorization all apply to the ItemService only.

What I wanted for my project, was the flexibility of the EntityService with (JWT) token based authorization. Best of both worlds.

Ian Graham wrote an excellent article on enabling token based authorization, which you can read here. The configuration steps mentioned there are all relevant to this scenario so consider them a pre-requisite.

After configuring token based authorization and setting up an EntityService you will immediately notice that you can send a request to your service without a token header and it will return a 200 response and the expected JSON data. So how can we use the out-of-the-box token generation and validation that Sitecore provides with the ItemService within the context of EntityService controllers?

Well first, you need to understand that Sitecore’s implementation uses the class Sitecore.Services.Infrastructure.Sitecore.Security.ConfiguredOrNullTokenProvider, which is instantiated in the constructor of Sitecore.Services.Infrastructure.Sitecore.Security.TokenDelegatingHandler


public TokenDelegatingHandler() : this(new ConfiguredOrNullTokenProvider(new SigningTokenProvider()), new UserService())
{
}

The TokenDelegatingHandler is configured in Sitecore.Services.Client.config


<api>
    <services>
        <configuration type="...">            
            <delegatingHandlers hint="list:AddDelegatingHandler">
                <delegatingHandler desc="TokenDelegatingHandler">Sitecore.Services.Infrastructure.Sitecore.Security.TokenDelegatingHandler, Sitecore.Services.Infrastructure.Sitecore</delegatingHandler>
            </delegatingHandlers>
        </configuration>
    </services>
</api>

The TokenDelegatingHandler uses the following method to validate a token passed in the request header


private void AttemptLoginWithToken(HttpRequestMessage request)
{
    if (request.Headers.Contains("token"))
    {
        ITokenValidationResult tokenValidationResult = this._tokenProvider.ValidateToken(request.Headers.GetValues("token").FirstOrDefault<string>());
        
        if (tokenValidationResult.IsValid)
        {
            if ((
                from c in tokenValidationResult.Claims
                where c.Type == "User"
                select c).Count<Claim>() == 1)
            {
                string value = tokenValidationResult.Claims.First<Claim>((Claim c) => c.Type == "User").Value;
                this._userService.SwitchToUser(value);
            }
        }
    }
}

What’s important here is that an instance of HttpRequestMessage is passed into this method.

Now let’s create a custom authorize attribute that inherits from System.Web.Http.AuthorizeAttribute. In this class override the IsAuthorized method, which is passed an instance of HttpActionContext and let’s give it a constructor that creates a new privately available instance of ConfiguredOrNullTokenProvider just like the TokenDelegatingHandler (except we don’t also need the UserService, we just want to know if the token is valid or not).

The HttpActionContext instance passed into the IsAuthorized method gives us access to the HttpRequestMessage (which the TokenDelegatingHandler uses to locate the token header, so we can do the same. Putting it all together


public class TokenAuthorizeAttribute : AuthorizeAttribute
{
    private readonly ITokenProvider _tokenProvider;

    public TokenAuthorizeAttribute() : this (new ConfiguredOrNullTokenProvider(new SigningTokenProvider()))
    {

    }

    public TokenAuthorizeAttribute(ITokenProvider tokenProvider)
    {
        _tokenProvider = tokenProvider ?? throw new ArgumentNullException("tokenProvider");
    }

    protected override bool IsAuthorized(HttpActionContext actionContext)
    {
        if (!actionContext.Request.Headers.Contains("token"))
            return false;

        var tokenValidationResult = _tokenProvider.ValidateToken(actionContext.Request.Headers.GetValues("token").FirstOrDefault());           

        return tokenValidationResult.IsValid;
    }
}

Now we simply decorate our entity service controller with our custom token authorization attribute


[ServicesController]
[EnableCors(origins: "*", headers: "*", methods: "GET")]
[TokenAuthorize]
public class ProductController : Sitecore.Services.Infrastructure.Sitecore.Services.EntityService<Product>
{

}

Now you can request tokens the same way you do with the ItemService:

Request Sitecore token using Postman

And pass the value of the token response back to Sitecore in the token header when making requests to your entity service controllers or get a 403 response.

As highlighted on Sitecore Stack Exchange you will get a 500 server error response if you pass a token that has expired (expiry is 20 minutes by default but configurable in the setting Sitecore.Services.Token.Authorization.Timeout), well certainly in 8.2 Update 2 and Update 3 you will, I haven’t tested other versions. If you don’t care about that, you are done here.

If however, you do want to get a proper 403 response for expired tokens (like me and the OP on Stack Exchange) read on.

To make this error go away we have to (unfortunately) do some slightly dirty things. First we need to replace the default implementation of TokenDelegatingHandler, which is straight forward to do because it’s just configuration


<api>
    <services>
        <configuration>
            <delegatingHandlers hint="list:AddDelegatingHandler">
                <patch:delete />
            </delegatingHandlers>
            <delegatingHandlers hint="list:AddDelegatingHandler">
                <delegatingHandler desc="TokenDelegatingHandler">MyHandler, MyAssembly.dll</delegatingHandler>
            </delegatingHandlers>
        </configuration>
    </services>
</api>

In our custom handler we want to replace the default SigningTokenProvider with our own implementation in two of the constructors


public class MyHandler : TokenDelegatingHandler
{
    public MyHandler()
        : base(new ConfiguredOrNullTokenProvider(new MySigningTokenProvider()), new UserService())
    {

    }

    protected MyHandler(HttpMessageHandler innerHandler)
        : base(innerHandler, new ConfiguredOrNullTokenProvider(new MySigningTokenProvider()), new UserService())
    {

    }
}

This is where it gets gory, yes you guessed it, no virtual methods, private members etc so we have to copy paste a bunch of Sitecore’s code and replace the bit responsible for the 500 error which as the thread on Stack Exchange explains is in the logging of SecurityTokenInvalidLifetimeException exceptions raised. I’ve thrown up a gist so you don’t have to go through the pain of tidying it all up.

That’s it. Sitecore’s EntityService with ItemService style token authorization and proper 403 responses for expired tokens.

Happy… securing.

Sitecore & Xamarin for Mobile Apps

I had the pleasure of presenting this topic at Sitecore Symposium this year in Las Vegas. The aim of the topic was to demonstrate the steps required to get started with Xamarin development in conjunction with Sitecore’s Mobile SDK.

I have turned this presentation into an article, which you can find here:

https://www.valtech.com/blog/sitecore–xamarin-for-mobile-apps/

You can also download the slides from the presentation here:

Everyone gets an app!