I’ve been busy with a huge MongoDB xDB (Sitecore 8.2) upgrade to a Sitecore SQL Server xDB (9.3 in my case), where I’ve run into a nasty issue with the Sitecore xConnect Search Indexer (IndexWorker) reporting incompatible tokens, due to a different set of shards. The IndexWorker logs were full with the ‘Sitecore.Xdb.Collection.Failures.IncompatibleSyncTokensException: Tokens are incompatible, they have different set of shards’ exception.
After the migration was done successfully, 3 local databases were restored onto the higher environments:
The first thing to-do then was checking for the existing server and database names:
Once that was checked, we updated the server names (and possibly the database names if they differed from the local environment) on the restored databases on the higher environment.
However after that I ran into the situation that when trying to start (or re-start) the xConnect IndexWorker (xConnect Collection Search Windows Service), the logs were full with the above exceptions.
So Google to the rescue here. I ended up with this Stack Exchange thread & with a blog-post by João Neto on XConnect IndexWorker error ‘Tokens are incompatible, they have different set of shards.’. Before explaining that the above 2 threads didn’t work for me and what I did to overcome the problem, let’s have an explanation on the situation first.
Sitecore stores the xDB Index Token (which is a Base64 encoded string and which contains the database server names and database names of the xDB collection shards) in the _xdb & xdb_rebuild Solr indexes. You can see this Base64 encoded string when you for example have your Solr instance and select the _xdb collection and query on the following id:xdb-index-token. Do this as well for the _xdb_rebuild Solr collection.
If there are no values for xdbtokenbytes_s (could be that only the _xdb index collection returns a value and the _xdb_rebuild doesn’t), then that’s okay, if you have a value for xdbtokenbytes_s, we need to check what that Base64 Encoded string contains in terms of the SQL Server name and database names. Since you’re reading this blog-post you’ve probably run into this xDB Index Token issue as well, and probably for 100% certainty say that the Base64 encoded string contains an invalid value in your specific environment.
So when downloading this script and executing it with the xdbtokenbytes_s value, we could see that on our higher environments it was containing the incorrect servername.
So far so good. After checking the incorrect value, we should request a rebuild of the xDB index. So in the xConnect Collection Search IndexWorker folder: <xConnect Collection Search service root>\App_Data\jobs\continuous\IndexWorker we ran the Sitecore.XConnectSearchIndexer.exe -rr (alternatively Sitecore.XConnectSearchIndexer.exe -requestrebuild) command.
The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process. But like for João Neto the issue persisted. He mentioned that the List Manager application also stores synchronization tokens. But for me I had no children under /sitecore/system/List Manager/ListOperations.
In order to get rid of the previously added xDB Index Token to the xdb & xdb_rebuild Solr collections (apparently for me the request rebuild didn’t update it, even though we stopped the IndexWorker service and restarted it), the last fallback for me was just deleting it, instead of having the service update it. Maybe the next best thing if you’re running into this issue will be directly deleting it (without going through the checks of converting the Base64 encoded string with that Powershell script 😉) from Solr.
So here’s what you need to do in the Solr dashboard. Make sure to select the _xdb collection (and if the xDB_rebuild collection also has the incorrect value, then execute the above query twice) and make sure to select ‘XML’ as Document Type. Then paste in the following:
Then submit the document. This will delete the ‘incorrect’ xDB Index Token. You can always check this again by executing the previous Solr query of id:xdb-index-token. You should have no results then.
With this removed, we can successfully start the Sitecore xConnect Search Indexer again without any errors in the windows service logs. The IndexWorker service will then automatically register a correct xDB Index Token after a while when the IndexWorker requests a rebuild.
Hope this blog-post serves as an addition to the other posts on the nasty issue with the incompatible tokens due to a different set of shards.