Support CNAME Cloaking for tracking scripts to reduce DNI interference from ad blockers
I've seen estimates that up to 50% of visitors are using browser-layer ad blockers (Ghostery, AdBlock, uBlock, etc.) or DNS-layer (PI-hole, etc.) ad blocking.
I personally use the Ghostery browser plugin, which blocks CallRail's DNI script when I visit our client sites.
Clearly, this is a big problem - we're missing call and attribution data from a big chunk of our visitors.
Defeating ad blockers should be a top priority for CallRail because less interference by ad blockers = more tracking numbers served = more calls tracked = more billable minutes.
CNAME Cloaking
To defeat browser-layer blocking (which is what most people are using), we can use CNAME cloaking.
Some analytics providers are already offering it as a feature: the tenant chooses or is assigned a subdomain, then creates a CNAME record on their own domain with that subdomain as the target. From that point on, the .js is served from that cloaked domain, which isn't on any block lists.
And because browser-layer ad blockers have limited access to the request and DNS resolution, they can't compare the CNAME target to blacklists.
Test Results
Despite this not being officially supported by CallRail, my experiment so far has produced exciting results.
I created a CNAME record "crcdn" with a target of "cdn.callrail.com", then changed a test site to call "crcdn.test-site.com/..swap.js" rather than "cdn.callrail.com/..swap.js".
When I visit the test site, Ghostery reports that it is blocking CallRail - but swap.js is loaded and the DNI still happens! The tel: link target has been replaced with the tracking number that corresponds to the URL parameter I used in the test.
Using Chrome dev tools, I can see that the request that Ghostery actually blocking was made to js.callrail.com to start the session. That request was initiated by swap.js after it loaded.
With an officially supported CNAME cloaking feature, what I'm imagining could happen is that when swap.js is generated for that company, the references to js.callrail.com and cdn.callrail.com are replaced by the chosen cloaking domain.
In this way, none of CallRail's tracking requests would be blocked at the browser layer.
This leads me to an important caveat - despite getting DNI working against Ghostery, there could be all sorts of potential for unexpected behavior and inaccurate CallRail session data, because:
- CallRail's tracking cookies appear to have been set from the cloaked domain - though this may not actually be a problem, because the keys/values still appear in the output of document.cookie() when I run it in console of the test site.
- Ghostery is still blocking any subsequent requests to *.callrail.com after loading swap.js, which interferes with CallRail session creation and update (CNAME cloaking those domains would fix this issue)
Defeating DNS-layer Blocking
This is another topic, but I think it would take a hybrid server-side/client-side DNI solution where everything is served from the first-party domain
A barebones implementation would be to expose an API endpoint which accepts a URL parameter value and returns the corresponding swap targets+values, then devs could update the DOM as needed based on the API response.
Comments
Thank you so much for your explanation & detailed write-up! We are definitely aware of the growing share of privacy browsers, ad blockers, and stricter cookie policies and their impacts on our script.
The suggestion of a CNAME cloaking is a good one! To the point in your caveat, I think we'd have a lot of edge cases to test for and ensure we can still support, including some of the more complex features of our snippet. We would also need to build some sort of registration process for the domains of interested customers.
We have also considered the server-side DNI approach you mentioned. That solution puts a lot of work on the customer side to be able to pass our snippet the params it would need to determine the right number to swap (as well as needing to code for handling the swap on the site).
TLDR; You can guarantee we'll be exploring this problem more in the future, and we will seriously consider the suggestions you've made here.
Hi Adam, thanks for your reply! I'm thrilled to hear that CallRail is exploring this increasingly important issue. Aside from the increased tracked call volume - this sort of feature is something that a competitor could leverage quite effectively.
I'm sure there's a lot of variation between the contents of swap.js based on the individual company's configuration, version, etc. There were significant differences just in the two swap.js responses I looked at.
That said, it seems to me that the issue is simply cloaking the domain. In swap.js, there are explicit references to *.callrail.com domains:
If swap.js came back with those references (and any other references elsewhere) replaced, I'm hoping this would make everything "just work" regardless of any other variance in the script.
At whatever point you're generating and caching[1] the swap.js content, just replace your .callrail.com domain names with the customer's cloaked names.
You could keep things simple initially by putting the onus on customers for cloak domain selection and all DNS record setup and verification.
That way, it would just need to be a company-level input for the domain names to use in place of js.callrail.com and cdn.callrail.com.
Is this viable?
I was thinking we could implement this on our own server-side: Remove the swap.js reference from our header and call it server-side on each request, then transform the response (replacing the callrail.com domains), and write that transformed JS into the document, which is then sent to the client. The problem is, we don't want to bake a blocking non-cacheable external request into every request to our applications/websites. So at first glance, this roll-your-own solution isn't practical.
I agree, a full-featured server-side DNI offering would add great engineering and support complexity. However, the implementation I mentioned could be a low-investment way to offer numerous options for developers.
If there was an API endpoint we could POST the visitor's query string to[2] and get a response with the tracking number to use[3], then we'd render the document with the tracking number already in place.
In our applications, we already have logic which determines the href value for our call to action links, so it's really not much more work to incorporate that response data so we can show the tracking number in the component when it is first rendered.
This approach would also work well with typical JS frontends that populate component slot data from API calls.
And as noted, server-side DNI would be the ultimate solution, defeating even DNS-layer ad blocking.
-
[1] I'm assuming that swap.js is cached at some layer by CallRail, rather than being freshly generated for each request. This could be incorrect.
[2] Rather than posting the full string, the responsibility would be on developers to first process the full string so its parameters and order exactly match those defined in the DNI triggers. This would also include removing always-unique params such as "gclid". So for simplicity, the CallRail API endpoint would expect to receive only a value that either matches to a trigger or doesn't.
[3] Disclaimer: This is only a good idea if your application is caching the responses from CallRail for re-use on the next occurrence of that query string, so you don't have to make a blocking external API call on every request.
Really appreciate the thorough response and everything you've investigated and outlined here. We will be continuing to investigate and look to make future enhancements for how this functionality could be improved for both developer and non-developer type users. No current timeline as of yet, but stay tuned!
Thanks Adam!
This is a critical need, it sounds like things are going to progress slowly.
I'm talking to my devs about some additional approaches. I'd like to beta test on some client sites and then maybe roll out a SaaS offering that provides this service for other companies who are tired of losing a huge chunk of their call data to ad blockers.
Seems CR already may already have some preliminary support in place for domain cloaking. I just today realized that the white label agency tools add-on supports tracking scripts from a custom domain:
I really like this general approach, and I do think it's something we'll be looking at in upcoming roadmap discussions. However, you're correct to assume this isn't something we can fast track right now. There's a few other considerations our team is navigating with this at the moment.
1. There's a case to be made that CallRail domains should be removed from these adblocker lists because blocking the loading of swap.js does in some cases break functionality on customer sites. We haven't explored this thoroughly yet, but it is an angle successfully argued by Marketo in the past. At the crux of this issue is CallRail being lumped in with third-party analytics tools (which are generally much more despised), despite really being first-party analytics (used by the business, but never sold or shared).
2. CNAME cloaking is a clever solution, but it does have infrastructure implications on our side. For starters, we need to support SSL certs for all those CNAMEs, because the SNI and the HTTP Host header will indicate the cloaked CNAME rather than the currently-expected js.callrail.com. This isn't an insurmountable problem – as you indicate others have done this – but it does require a rearchitecture of our ingress infrastructure to get all of these on a smaller set of IPs. We do support something similar for custom white-label domains, but we've found it to scale poorly (which is why white label is either an add-on or requires higher plan tier).
3. Not everyone will want this particular solution. We're also exploring server-side solutions, as well as solutions which proxy the request through the origin server. (Our Wordpress plugin has support for this at the moment.)
That all said, I can assure you this is on our minds. Increased ad-blocker adoption will reduce the traffic for which attribution is available, and that's a problem for customers that we need to fix. Out interests are aligned here, so this is definitely not going unnoticed.
[Edit:] Probably should have mentioned I'm the CTO here. 🙂
Hi Elliot, thanks for the thoughtful response! I really appreciate the attention to this issue.
I totally agree, but playing devil's advocate, list maintainers may counter with two things:
1. No functionality is broken, it's just that the default phone number is displayed rather than the desired tracking number
2. swap.js seems to be doing a lot more than simply swapping numbers - there's visitor session polling, form capture code, etc.
To that second point, I wonder if you'd have more luck with list maintainers by decoupling the DNI from everything else, perhaps serving two different scripts? That said, I realize this may not be practical or even possible.
In my experiment, SSL "just worked." All I did was create an unproxied CNAME record in Cloudflare pointing crcdn.test-site.com to cdn.callrail.com. That said, while I had no issues accessing swap.js via HTTPS on the cloaked domain, that doesn't mean there's no unexpected behavior serving it from a different domain that is unknown to CallRail.
I recently installed both Fathom Analytics and Simple Analytics on a client site, so I could test out their CNAME cloaking features.
With Simple Analytics, after choosing a subdomain name, I had to click to generate the SSL cert for it, and there was a warning that it could take some time for the cert to generate. I noticed that with Fathom Analytics, that was no mention of SSL certs and everything was working instantly. I'm not sure if Fathom handles cert issuance in the background without telling the user, or if their configuration just doesn't require a new cert.
Isn't this a good thing? Or could the mismatch would cause issues or unexpected behavior?
To my understanding, browser ad blocker plugins aren't able to access the DNS request layer. Of course DNS-layer blockers can, but CNAME cloaking won't work at that layer anyway.
Perhaps a tweak to the tracking snippet could help with this.
Along with the call to swap.js, include some JS which checks for the presence of the CallRail window object on document ready. If it isn't found, assume an ad blocker interfered, and fetch swap.js from the cloaked domain.
This approach would allow you to continue serving 50-70% of the requests as usual, falling back to the alternative infrastructure only when necessary.
Great to know, thank you! While we typically build sites using Laravel, I glanced at the WP plugin code and think we could possibly adapt it.
My concern with fetching swap.js server-side was I assumed that the request needed to come from the user in order for CallRail to obtain their IP and perhaps other data (not sure what is used), but it appears that's not an issue after all.
I'm thrilled to say that after experimenting with some different approaches to this problem, we've come up with something much more effective than CNAME cloaking.
While my primary concern was dynamic number insertion, in my initial testing this solution also gets the subsequent swap_session.js request and the poll.js requests past ad blockers. So everything seems to work, not just DNI.
We don't use the form capture feature, so I haven't tested that, but the requests to form_capture.js should work as well.
We're leveraging e few techniques to do this, some at the DNS level and some at the web server level.
This solution should defeat virtually everything, including ad blocker browser plugins and DNS-level blockers.
We're testing this on a client site now, rolling out to a few more clients next week.
To implement this, we simply added a short JS snippet to the client site (doesn't replace CallRail's snippet) and some A records on their domain.
Our solution only fires if CallRail didn't load (presumably because the script was blocked), and it transforms responses so that everything continues to look first-party.
I'd love to talk to someone at CallRail about this, and hopefully offer it as a service to other CallRail customers!
Thank you so much for the follow up. That is great news that you've been able to come up with this solution for you and your clients. I am going to get with some members of our Product team this week, and will follow up with you next week to further the discussion.
Thanks for being a valued member of the CallRail Community!
- Adam
Thank you Adam!
I attended the event in Tampa this Wednesday, and had the chance to demo this for some of the team. That was awesome, I really appreciate the opportunity.
We're currently implementing an additional defense (filename and path randomization for the cloaked swap.js), which should be ready next week.
Once that's done, we'll roll it out to a few more client sites to continue testing.
We're also working on logging, so we'll be able to know how many requests are hitting the service. I think that by comparing that to the total site requests, we can determine what percentage of visitors are using an ad blocker.
That is great to hear! So glad you were able to attend the Tampa event and discuss your solution with the team. Looking forward to hearing how the additional testing goes!
Hi Adam,
Last week, our proxied requests started failing. I was able to resolve the underlying DNS-related issue with some help from Stackoverflow, and everything is working again.
The filename randomization is already tested, though we aren't using it in the POC yet.
As for testing form tracking - I don't see why it wouldn't work, but we don't use that feature on any client sites. I have another agency that has expressed interest in testing this feature, perhaps they can provide that confirmation.
Something I'd like to explore is proxying everything through a single subdomain rather than three, so users of the service would only need to create one A record. I think this is possible, but may not be able to know for sure without confirming some things with CallRail.
I've had R&D on pause pending an update from Laura on the the product team.
That said, the domain registration UI, and billing functionality is ~75% complete. The remaining request logging setup could be completed quickly. (We'd likely filter all known bots and crawlers from this reporting.)
The final piece would be building out the AWS infrastructure and setting up our automations, but that can be done within a few days as well.
If CallRail ultimately does not want to promote a third-party service such as this, then I would just scrap these plans and continue privately using the solution as-is for our own clients.
Either way, I'd love to hear back soon!
Really appreciate the update, and am impressed by all your efforts on this over the last couple months. After many internal discussions regarding this on both the Product and Development teams, we have been unable to add this to our immediate roadmap. However, as we discussed previously, your solution is something you should offer your clients as needed!
Thanks Adam!
I've enjoyed the opportunity to explore this problem, and have learned quite a bit in the process. This was an immensely satisfying solve.
We've rolled this out to four client sites (and I'm already sleeping better).
The next step is to implement analytics, so we can see what percentage of visitors swap.js fails to load for. Once that's in place and I can get a few weeks of data, I'll share the findings.
I'm quite interested to compare the percentages between browsers and device types. I could always be wrong, but I believe the desktop numbers may be shocking enough to get the Finance team discussing this too. 😊
Hey all,
Thanks for the really detailed discussion here! I too would be heavily supporting of offering a server-side option for getting a tracking number. We see anywhere from 20-40% of calls we expected to be tracked fail due to swap.js failing.
Hi Brian,
Thanks for sharing that.
We've continued tracking data from a few client sites.
Since 9/13/22, we've tracked visits from 16,435 distinct IP addresses where Callrail was not blocked (the "calltrk" object existed at the time our script fire after page load).
We've had 2,482 distinct IPs where the script was blocked. So we're seeing about 13% of visitors blocking CallRail.
This was and continues to be an absolutely massive problem.
Something important to note is that for sites with a lot of PPC and display ad traffic, the numbers won't be as bad, whereas for sites with predominantly organic traffic, they will be worse.
This is because the people who are using ad blockers are rarely even seeing PPC or display ads and won't reach the site to begin with.
Please sign in to leave a comment.