Advanced synchronization
When you define a content type, by default, all items of that type are distributed to all mobile devices. While this setup is simple and easy to get started, it does not scale in case you rely too heavily on content items. Let's assume, that you have a content type 'gebaude' (which is German for 'building') where you track all the buildings that are inspected. In case you have 1000 of those items, all 1000 will be transferred to all iCL Filler apps of your inspectors.
While several thousand items may not be an issue, in case you create and track defects during your inspections, this number can rise significantly until the synchronization process gets slow. Apart from this fact, it is very likely, that not all of your inspectors really need all defects to be present on their device.
1. Enabling filteredSynchronization
Usually, content items are only needed in the app if they are used by an inspection (referenced by a task).
In order to enable this behavior, you can simply turn on Filtered sync for specific content types by specifying filteredSynchronization:true in the content types json definition.
Once configured, items of type gebaude will only be send to your users devices, if they do have a task (using the field itemsToInspect or 'Inspected objects') for them.
{
"$type": "contentType",
"name": "gebaude",
"displayName": "Gebäude",
"canBeInspected": true,
"titleFieldName": "bezeichnung",
"icon": "fa-building",
"filteredSynchronization":true,
"fields": [...]
...
}
The reason why this is specified per content type is, that you may have a content type building that needs to be filtered because you have a lot of buildings in your system, but there may also be a content type building category which only holds a very limited amount of items (e.g. 100) and therefore does not require any sync filters to be applied.
Keep in mind that while sync filters reduce the amount of data that is transferred to the users devices, it considerably raises query complexity and therefore the load on the database engine. Therefore, enable filtered synchronization with caution/care - it is not a silver bullet!
You will see then, that the checkbox in filtered sync will be checked for this type
Now, while the items of any non-filterd will always be transferred to all devices, filtered content types are only synchronized if they satisfy any of the following rules:
1. The content item is contained in itemsToInspect
If there is any task synchronized to a users device that refers to a filtered content item in itemsToInspect, the item will be synchronized.
Example: You create a task for inspecting a building, and add this building to the task as itemToInspect, then the building will be sent to the device as well.
2. The content item is used in task data
If there is any task synchronized to a users device, which contains additional data (task data which is used to pre-fill checklists) that refers to a filtered content item, the item will be synchronized.
Example: You create a task (via the REST interface or excel import) that contains the unique id of a building B which is to be filled into a checklist field, then this identity will be recognized and the building will be sent to the device as well
3. The content item is used within a checklist
If an inspection is synchronized to a users device, which contains a checklist, that refers to an item in a 'content item' field, that item will be sent to the device as well.
Example: A building A existed on your device because it fulfilled one of the other rules. In the checklist, you manually select building A using a 'content item' field. Then, even if the original rule is not fulfilled anymore, building A will still remain on your device, because it is used in the checklist.
4. A defect items parent fulfills any of the rules above
A defect item is referring some content item that fulfills any of the above rules. As defects are always synchronized along with their parent items, it too will be sent to the device.
Example: A building A was inspected some time ago. During this inspection, you noted several defects which were created and assigned to A when you completed the inspection. Now, you are again inspecting this building. As the defects are sent along with A, you will be able to see all already existing defects and be able to determine if any new came up and/or if any of the existing ones have already been resolved.
This functionality of 'synchronizing related items' only works for defects. If you have a building with five floor items associated with it, these floors will not be synchronized automatically.
Now that you know what rules affect whether an item is synchronized, we need to provide you with some technical background information, which will help you to fully understand the principles behind filtered synchronization and how it affects your system.
As with inspections and tasks, whether or not some object (inspection, task, content item, file, ...) is synchronized to a users device is defined using sync rules. These are evaluated every minute in the background by the job called IEvaluateSyncRulesJob.Execute.
Now, whenever a filtered content item is fulfilling one of the rule above, it is marked as to be synchronized by the IEvaluateSyncRulesJob.Execute background job.
This marking is then valid for 24 hours.
This means, that every day at midnight, the marking of those items has to be renewed (done by the background job IReEvaluateInspectionSyncRulesJob.Execute)
This also means, that if an item is considered to not fulfill any sync rule anymore, it will still be transferred to the users device until the marking eventually expires after 24 hours. This strategy was chosen as it scales better for many items.
Also, be aware that all filtered content items are synchronized per team if a task is self-assignable. This means, that if a self-assignable task assigned to 'Antony' of team 'building inspections' is referring to a building A, then this building will not only be synchronized to the device of 'Antony', but also to the devices of all other inspectors of team 'building inspections'. Again, this strategy was chosen as it delivered the best of both worlds concerning query complexity and amount of data transferred (bytes).
2. Custom sync filters
In some cases, the above mentioned Filtered sync approach is not sufficient.
2.1 Synchronize related items using synced
There may be cases, where you have a set of content types that are related and, therefore, should always be synchronized together.
For example, you may want to plan an inspection for a building. During this inspection, the user should go through all fire extinguishers of said building and verify they are without flaws and present. Now, it would make sense to have a content type gebaude which represents the building and you'd like to create a task for it. The fire extinguishers (represented by a separate content type feuerloescher) should be synced along with the building they belong to - you do not want to have to add all x extinguishers as Inspected object to the task.
To achieve this, you can use our advanced sync filter functionality where you define, that items of type feuerloescher should be synced to a users device, whenever their parent building (gebaude) is synced.
{
"$type": "contentType",
"name": "feuerloescher",
"displayName": "Feuerlöscher",
"canBeInspected": true,
"titleFieldName": "bezeichnung",
"filteredSynchronization":true,
"fields": [
...
,{
"$type": "contentField",
"type": "ContentItem",
"name": "buildingid",
"displayName": "Building",
"isRequired": true
}
],
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "synced",
"field": "buildingid"
}
}
],
"relationships": [
{
"$type": "contentRelationship",
"name": "Building",
"displayName": "Building",
"foreignKeyField": "buildingid",
"targetContentType": {
"$type": "contentTypeRef",
"id": "1eee7b9b-66d7-4760-b75b-03cd1a085bbd",
"version": 1
}
}
]
...
}
When defining a hierarchical sync rule, the system assumes that all items that depend on a synchronized item should be sent to the same receiver.
Therefore, you do not have to specify a receiver in the syncrule.
You can, however, override this behavior by specifying one.
Notice the definition of syncRules where we define, that this content type is synced, whenever the building is synchronized. This works, because we specify the field buildingid in our synced filter, which basically says: Look at the value of the field buildingid - if an item with that identity is synced, also sync this feuerloescher item.
Also notice that for this to work, you need to stay aligned with the following rules:
- you can only use
syncRulesif your content type has filteresSynchronization enabled - you can only use the
syncedrule if your content type (here:feuerloescher) defines arelationshipwith another content type - in some cases, the
relationshipdoes not define the exact type of the target:
{
"relationships": [
{
"$type": "contentRelationship",
"name": "Building",
"displayName": "Building",
"foreignKeyField": "buildingid",
"targetContentType": {
"$type": "contentTypeRef",
"version": 1
}
}
]
...
}
In this case, the system will assume that all content types with builtInType 0 (zero) that have filteredSynchronization enabled are valid targets.
This makes it very comfortable to e.g. define a content type for defects that can belong to any type of parent and should be synchronized along with it, without having to update all possible target types whenever a new suitable content type is added or an old one removed.
However, in rare cases, this can lead to a cyclic dependency error. In such cases, you can explicitly define all possible target types using the targetContentTypeIds array:
{
"relationships": [
{
"$type": "contentRelationship",
"name": "Building",
"displayName": "Building",
"foreignKeyField": "buildingid",
"targetContentType": {
"$type": "contentTypeRef",
"version": 1
},
"targetContentTypeIds": [
"1eee7b9b-66d7-4760-b75b-03cd1a085bbd",
"396205e4-a6d7-4713-88a7-7e55010e2967",
...
]
}]
}
Note that this rule does not apply to content types with "builtInType":6 (defects).
- The parent type of a rule using
synced(in this casegebaude) must have filtered synchronization enabled. This one is obvious as if that were not the case, allfeuerloescheritems would always be synced to all devices which is basically the same behavior as not using filtered synchronization in the first place. - The parent type of a rule using
syncedmust not be a defect ("builtInType":6) - You must not define cyclic dependencies using the
syncedrule. In case the typegebaudehad (for whatever reason) a relationship tofeuerloescher(e.g. to store the main extinguisher of a building...apologies for this nonsensical example) and you'd define that a building should besyncedwhenever thefeuerloescheris synchronized, this would create a cyclic dependency (gebaude->feuerloescher->gebaude). Note, that our system will give you an error if you do have some cycle anywhere in your content types hierarchy
As with the filtered synchronization, the rules you define per content type are evaluated every minute by the job called IEvaluateSyncRulesJob.Execute and expire after 24 hours.
The sync rules are evaluated in dependency order. For this, we analyze all content types, their rules and dependencies and evaluate them in that order. In our example, feuerloescher depends on gebaude (because it uses the synced rule). gebaude does not have a dependency on any other type, so it is evaluated first. feuerloescher is evaluated last. Because of this dependency order evaluation, the system
- must know all valid types of a relationship (therefore the introduction of the field
targetContentTypeIds) - and the dependency graph must not have a cycle.
2.2 Synchronize based on filters and receivers
Another case is that you may want to define if a content item is synchronized and who it is sent to based on its attributes.
For example, you may want to distribute orders to all field agents as long as they are not completed and not cleared.
You could of course link them to tasks to get synchronized, but maybe your agents do not want to work task-based.
In such cases, you can define custom sync filters.
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
}
}
]
2.2.1 Filter expressions
To create a filter, you can use the following building blocks.
The filter object:
Allows to define a filter expression to reduce the number of results
Example:
{
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
}
The following oeprators are available
| Name | Type | Description | SQL translation |
|---|---|---|---|
| Eq | any | Compares two values for equality | = |
| Neq | any | Compares two values for inequality | <> |
| Lt | any | Lower than | < |
| Lte | any | Lower than or equal | <= |
| Gt | any | Greater than | > |
| Gte | any | Greater than or equal | >= |
| Startswith | string | True if field starts with the provided value | Like '...%' |
| Endswith | string | True if field ends with the provided value | Like '%...' |
| Contains | string | True if field contains the provided value | Like '%...%' |
| In | array | True, if the field matches any item in the provided array | IN (...)' |
| NotIn | array | True, if the field matches no item in the provided array | IN (...)' |
| IsNull | any | True, if the provided field is null | is null' |
| IsNotNull | any | True, if the provided field is not null | is not null' |
The group object
Allows to combine multiple filter objects logically.
The logic can be either AND or OR
Example:
{
"$type": "group",
"logic": "AND",
"operators": [
{
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
},
{
"$type": "filter",
"field": "agent_team",
"operator": "Contains",
"value": "order agents WEST"
}
]
}
2.2.2 Specifying a receiver
As you can see, the previous example defined a constant filter, which will synchronize all orders as long as their order_status is not equal to "completed" or "cleared".
It did not define to which users the items should be synchronized. As that information cannot be copied from any parent item (as with the synced rule), we need to somehow define the receiver.
We could, of course, omit the receiver option anyways. In that case, all items matching the filter will be synchronized to all users.
For this reason, you can define a receiver attribute.
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
},
"receiver": {
"$type": "receiver",
"type":"Team", -- the default!
"filter": {
-- a typical filter expression
"$type": "filter",
"field": "title", -- the title of the team
"operator": "Eq",
"value": "order agents WEST"
}
}
}
]
This specifies, that any non-complete order should be synchronized to a team called "order agents WEST".
A receiver can have two types:
"Team"specifies, that the receiver is a team. The filter expression can therefore use only attributes of a team, which areName Type Description idguid the unique id of a team isactiveboolean indicating if the team is active or obsolete titlestring the title of the team "User"specifies, that a specify user is the receiver. The filter expression can therefore only use attributes of a user, which areName Type Description idnumber the unique id of a user externalidstring a unique external id for the user usernamestring the username namestring the user's first name surnamestring the user's last name emailaddressstring the e-mail address of the user isactiveboolean indicating if the user is active or not
The default type of receiver is "Team"
A receiver can use the same filter expressions that are described here
In case you need to use a field of the content type in your receiver filter expression, you can use fieldref.
The fieldref will automatically point to fields of the content type.
Read more here
Combining sync rules
As there are multiple teams which should all receive their own order, we can determine the team by looking at the field agent_team.
The following example shows:
Synchronize any unfinished orders that contain the text order agents WEST in their field agent_team to the team order agents WEST
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "group",
"logic": "AND",
"operators": [
{
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
},
{
"$type": "filter",
"field": "agent_team",
"operator": "Contains",
"value": "order agents WEST"
}
]
},
"receiver": {
"$type": "receiver",
"type":"Team", -- the default!
"filter": {
-- a typical filter expression
"$type": "filter",
"field": "title",
"operator": "Eq",
"value": "order agents WEST"
}
}
}
]
In this case, you can add one or more synchronization rules.
If you only define one rule, it is considered being the default rule and does not need a name.
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
}
}
]
If you define multiple sync rules you have to name them (or all but the default rule), so that the system knows which rule to apply:
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"]
}
},
{
"$type": "syncRule",
"name":"bycategory"
"filter": {
"$type": "filter",
"field": "order_category",
"operator": "Eq",
"value":"technical"
}
}
]
2.2.3 using fieldref
As you saw in the previous example, creating a separate sync rule per team is possible, but does not scale well.
If possible, it would be better, if the content type contained a field team which contains the team the order should be synchronized to:
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "filter",
"field": "order_status",
"operator": "NotIn",
"value":["completed", "cleared"],
"comment": "...only synchronize if the order is not yet completed"
},
"receiver": {
"$type": "receiver",
"type":"Team", -- the default!
"filter": {
"$type":"filter",
"comment":"here, the fields are the one of the team!",
"field": "title",
"operator":"Eq",
"value":{
"$type":"fieldref",
"field":"agent_team",
"comment":"fieldref is used to reference the field 'agent_team' of the content type 'order'"
}
}
}
}
]
Whenever you specify sync rule's filter, the field names used refer to the fields of the content type itself.
However, when specifying the filter of the receiver, the used field names refer to the fields of a
TeamorUserobject.If you need to use a field of the content type in your
receiver.filteryou need to use afieldref
Combining a hierarchical sync rule with constant filters
Lastly, it is also possible to combine synced rules and normal filter expressions
In this case the system assumes that all items that depend on a synchronized item should be sent to the same receiver.
Therefore, you do not have to specify a receiver in the syncrule.
You can, however, override this behavior by specifying one as in the following example
"filteredSynchronization": true,
"syncRules": [
{
"$type": "syncRule",
"filter": {
"$type": "group",
"logic": "AND",
"operators": [
{
"$type": "filter",
"field": "status",
"operator": "Neq",
"value": "removed"
},
{
"$type": "synced",
"field": "buildingid"
}
]
},
"receiver": {
"$type": "receiver",
"type":"Team", -- the default!
"filter": {
"$type": "filter",
"field": "title",
"operator": "Eq",
"value": "order agents WEST"
}
}
}
]