Automatically Reduce Google Analytics Payload Length

Here we are, reunited with customTask. This time, we’ll put this wonderful mechanism to work for a very, very good cause. One of the lesser known “features” of Google Analytics is that when the payload size (the request body that is actually sent to Google Analytics with each request) goes past a certain limit, specifically 8192 bytes, the hit is aborted without warning. This can come as a surprise, because there’s no indication anywhere in Google Analytics that you are missing hits because of this.

The solution? A nice little customTask script, which recursively whittles away parts of the payload until the length is just below the cap.

The idea is that you will be able to specify a list of payload parameters to be removed one by one from the payload until the size is safely below the maximum of 8192. First things that will go are some of the default dimensions that no one really uses, such as ‘Java Enabled’ and ‘Document Encoding’. Next, you can list all the Custom Dimensions you want to get rid of, in order of priority.

Finally, you’ll be able to list the Enhanced Ecommerce impression parameters that you want to remove from each impression in the payload. The script will stop as soon as the length is below 8192, so you’ll most likely end up with some impressions having more details than others.

Why stop at impressions? Because, from experience, that’s where most problems lie. It shouldn’t be too difficult to customize the script to also include Promotions and even “regular” Enhanced Ecommerce product objects.

Once you’ve implemented this in your Enhanced Ecommerce tags, you should never see the following error in the console (if you have Google Analytics Debugger activated, or if you have Universal Analytics running in debug mode):

I recommend implementing this customTask together with my solution for sending the payload length as a custom dimension. This way you’ll be able to detect if you’re approaching the maximum length of the payload with your hits, and you can then proceed to implement this payload length limiter when necessary.

For an alternative solution, take a look at this excellent article by the awesome Dan Wilkerson from the equally awesome Bounteous blog.

The customTask variable

To build the customTask, you need to create a new Custom JavaScript variable. Name it something like {{customTask - reduce payload length}}.

Note! If you already have a customTask implemented in your Enhanced Ecommerce tags, you’ll need to combine the code below with the existing customTask script you already have. For tips on how to do this, consult this guide.

Add the following code into the customTask variable body:

function() {
  return function(customModel) {
    // Add any other default parameters you want to strip from the payload
    // into this array in order of priority. 
    // For more details, visit: https://bit.ly/2MOhjBD
    var defaultParams = ['&je', '&de', '&sd', '&vp', '&sr'];
    
    // List the (regular) Custom Dimensions you want to strip from the
    // payload in order of priority. Leave the array empty if you don't
    // want to remove Custom Dimensions. 
    // For more details, visit: https://bit.ly/2MM6O1O
    var customDims = ['&cd198', '&cd199', '&cd200'];

    // List the impression object parameters you want to strip from the
    // payload in order of priority. Only list the field specifier, so
    // instead of '&il1pi2va', write 'va', and instead of '&il1pi4cd3', write
    // 'cd3'. For more details, visit: https://bit.ly/2KehCHF
    var impressions = ['va', 'br', 'ca', 'ps', 'nm'];
    
    // Don't touch the code below.
    var maxLength = 8192,
        globalSendTaskName = '_' + customModel.get('trackingId') + '_sendHitTask',
        originalSendHitTask = window[globalSendTaskName] = window[globalSendTaskName] || customModel.get('sendHitTask');
  
    customModel.set('sendHitTask', function(sendModel) {
    
      var ga = window[window['GoogleAnalyticsObject']];
    
      // Only log if in analytics debug mode.
      var log = function(msg) {
        if ('dump' in ga) {
          window.console.log.apply(window.console, [msg]);
        }
      };
    
      var hitPayload = sendModel.get('hitPayload');
    
      var removeKeys = function(hitPayload, keys) {
        var key, regex;
        while(hitPayload.length >= maxLength && keys.length) {
          key = keys[0];
          log('--> Removing ' + key);
          regex = new RegExp(key + '=[^&]+', 'gi');
          keys.shift();
          hitPayload = hitPayload.replace(regex, '');
          log('--> New length: ' + hitPayload.length);
        }
        return hitPayload;
      };
    
      var removeImpressions = function(hitPayload, keys) {
        var key, regex, oldKey;
        while(hitPayload.length >= maxLength && keys.length) {
          if (key !== keys[0]) {
            key = keys[0];
            log('--> Removing &ilNpiN' + key + ' from impression objects');
          }
          regex = new RegExp('&il\\d+pi\\d+' + key + '=[^&]+', 'i');
          if (!regex.test(hitPayload)) {
            keys.shift();
          }
          hitPayload = hitPayload.replace(regex, '');
        }
        log('--> New length: ' + hitPayload.length);
        return hitPayload;
      };

      // If over payload length, remove default parameters.
      if (hitPayload.length >= maxLength) {
        log('Payload too long (' + hitPayload.length + '), removing default keys...');
        hitPayload = removeKeys(hitPayload, defaultParams);
      }
      // If over payload length, remove custom dimensions.
      if (hitPayload.length >= maxLength) {
        log('Payload still too long (' + hitPayload.length + '), removing Custom Dimensions...');
        hitPayload = removeKeys(hitPayload, customDims);
      }
      // If over payload length, clean up impression objects.
      if (hitPayload.length >= maxLength) {
        log('Payload still too long (' + hitPayload.length + '), cleaning up Impressions...');
        hitPayload = removeImpressions(hitPayload, impressions);
      }
    
      // Send the modified payload.
      sendModel.set('hitPayload', hitPayload, true);
      originalSendHitTask(sendModel);
    });
  };
}

Make sure you edit the defaultParams, customDims, and impressions arrays to include the keys you want to purge from the dataLayer. Note that you will need to list the keys using their Measurement Protocol parameter names rather than what their analytics.js field names are. So instead of 'encoding', for example, you need to use '&de'.

The keys I have added to the defaultParams array are:

Naturally, you might want to preserve some of these, e.g. &vp or &sr. Feel free to edit the defaultParams array as you see fit.

When adding items to the impressions array, the syntax is slightly more complex. You should only add the actual parameter specifier, which is the key that comes after &ilNpiN. So if the parameter name for Product Impression Variant is &il2pi1va, you’d type just 'va' into the array. Or, if you wanted to purge the Product-Scoped Custom Dimension from index 3 from your impression objects, instead of typing '&il1pi1cd3', you’d type 'cd3 into the array.

Note that for defaultParams and customDims you need to type the full parameter name with the leading &.

Add the variable to your tag

This variable should be added to the tag which sends the Enhanced Ecommerce payload with impression objects to Google Analytics. To add it to the tag, either edit the Google Analytics Settings variable in the impression tag, or add the field directly to the tag.

Browse to More Settings -> Fields to set, and add the following field:

Field name: customTask
Value: {{customTask - reduce payload length}}

Like so:

And that’s it for the setup! Now, if your payload is over 8192, the script will first chop out the parameters specified in the defaultParams array. Then it will gobble up the dimensions in the customDims array. Finally, if the payload is STILL too long, it will proceed to clean up your impressions objects by removing the fields listed in impressions one by one, impression by impression.

How to debug

If you walked through the code, as I’m sure you did, you might have noticed how I use a custom method named log() to send some debug data to the console. The thing is, I don’t want to pollute the JavaScript console with debug messages when Google Analytics is not in debug mode, so I do a simple check to see if the debug version of analytics.js is loaded:

if ('dump' in window['ga']) {}

Basically, if the global ga object has the method dump, I’m assuming the user is using the debug version of analytics.js.

NOTE! If someone has figured a more robust way to check for the debug version of analytics.js, please let me know! Thank you.

So, if you want to see debug messages, you can set the library to debug mode by browsing to More Settings -> Advanced Configuration -> Use Debug Version in your Google Analytics tag or Google Analytics Settings variable, and setting it to true. Or, you can use the Google Analytics Debugger browser extension.

Once you have the debug library loaded, you can see debug messages such as these if the payload exceeds the maximum length of 8192:

In these debug messages you can see how the script walks through your payload, stripping out parameters where necessary.

Summary

I wish analytics.js had a more elegant way of handling payloads that are too large. Just dropping them is slightly overkill, in my opinion. analytics.js could, for example, compile a custom hit to GA when the payload maximum size is breached, and this would turn into a warning or notification in the Google Analytics UI. This way you’d at least have a clue that something is amiss.

So the customTask solution presented here is, as is typical for scripts that I write, a bandaid for something that will hopefully be remedied natively by the library itself in the future.

Naturally, the “best” way to avoid payloads that are too large is to only send relevant data to Google Analytics. In the case of impressions, for example, it doesn’t really make sense to send every single impression on the page in one hit. Rather, you’d want to only send impressions that the user has actually viewed. Luckily, I might just have a guide coming up that will show you how to track viewed impressions only.

Let me know in the comments if you find this useful or if you have suggestions for improving the method!