Synchronizer Traffic

Traffic accounting, fees, and monitoring on the Global Synchronizer

Synchronizer usage incurs operational costs for SVs. For amortizing these costs, improving the incentives for operating an SV, and making Denial Of Service attacks on the network unattractive, synchronizer usage beyond a certain free base rate level is charged with so-called synchronizer fees paid in Canton Coin. For supporting this functionality, sequencers keep track of available and consumed traffic for each synchronizer member (most relevantly: for all validator participants). Whenever both the free (base rate) and paid-for (extra) traffic are exhausted for a participant, attempted writes are denied by the sequencer.

SVs, or more specifically SV participants and mediators, have unlimited traffic and are therefore not subject to synchronizer fees themselves. However, it is part of the responsibilities of SV operators to jointly ensure that the fee configuration of the Global Synchronizer is sound and in line with the operational objectives of the network.

Traffic accounting; what counts as traffic?

Sequencers keep track of the traffic used by validator participants. Traffic in this context refers to all messages from participants that have to be sequenced, i.e., messages that the group of sequencers has to order, persist (up to a pruning interval), and deliver to recipients (typically mediators and other participants).

Most prominently, traffic is used for Daml workflows as part of the Canton transaction processing protocol. This includes:

• Confirmation requests; sent when a participant initiates the committing of a ledger transaction.
• Confirmation responses; sent for participants that host stakeholders of a transaction.

Note that not only custom Daml workflows count towards traffic spend but also automated “built-in” workflows such as rewards collection.

In addition to Daml workflow-related messages, participants also use traffic for, among other things:

• Submitting topology transactions, for example for allocating new parties or vetting newly uploaded DAR packages.
• Exchanging periodic ACS commitments to ensure that they are in sync.

Internally, sequencers maintain various counters per validator participant that correspond to the more easily visible traffic parameters introduced next. On a high level, sequencers keep track of:

• The available base rate traffic balance. The base rate is defined as a burst amount over a time window, so that even when fully depleted the available base rate traffic balance recovers fully after a (“window”-long) period of inactivity.
• The available extra traffic balance. This is traffic that has been paid for. The extra traffic balance is only consumed during times at which the base rate traffic balance is fully depleted; i.e., the base rate traffic balance is always consumed first.

When neither base rate nor extra traffic balance is available to a participant, the sequencer will deny further attempts to submit messages for sequencing until either the base rate traffic balance recovers or the extra traffic balance for the participant gets topped up. Note that as a safeguard against top-up deadlocks, the validator app will under some conditions abort ledger submissions even if there is still some traffic balance left. This only applies to submissions from the validator app itself! If you run another application against the validator we recommend monitoring the traffic balance and pausing that app if you run out of traffic.

Traffic accounting is “by participant”; all parties hosted on the same participant share the same traffic balance.

Traffic parameters

How traffic balance interacts with transaction submission

Here’s an example of traffic accounting in practice for submitting a transaction:

1. Transaction submitted.
2. The traffic manager (on the sequencer) checks your node’s base rate (i.e. free) traffic balance and the traffic that you have purchased.
3. Included in your base rate traffic is that accrued since your last submission. This accrual is time-linear, calculated to reach burstAmount after burstWindow time.
4. If you have enough base rate traffic, your submission is sequenced. Your base rate traffic balance is drawn down for the transaction and your extra traffic balance is unchanged.
5. If you do not have enough base rate traffic, but your extra traffic is enough to pay the difference, your submission is sequenced. Your base rate traffic balance is depleted and your extra traffic balance is drawn down for the remainder.
6. If neither of the above applies, the submission will fail.
7. Regardless of any situation, if you don’t submit transactions for burstWindow, your base rate traffic balance will be fully replenished to burstAmount. This will, at minimum, allow you to buy traffic again to avoid getting stuck in the situation where you don’t have enough traffic to top up your extra traffic.

A successful sequencing does not necessarily mean the transaction is fully accepted. See traffic_wasted below for explanation of these cases and how best to avoid them.

Traffic top-ups; how does one “buy” traffic?

“Buying traffic” resolves around two main components:

1. On-ledger MemberTraffic contracts (Daml API docs) that track each validator’s traffic state and are updated (atomically) whenever CC is spent for buying traffic. Most importantly, on-ledger MemberTraffic contracts track the total amount of extra traffic purchased for that validator.
2. In-sequencer traffic state that tracks both available and spent traffic. SVs update the in-sequencer traffic state based on the MemberTraffic state they observe on ledger, thereby ensuring that paid traffic fees are translated into actual traffic balance increases. Sequencers also update the in-sequencer traffic state themselves, whenever traffic is consumed (see traffic_accounting).

The validator app contains built-in top-up automation that automatically buys traffic to meet preconfigured throughput needs. Operators configure a target throughput (bytes per second) and a minimum top-up interval (seconds); the automation buys target throughput × minimum top-up interval bytes whenever both of the following conditions are met:

• the extra traffic balance has fallen below that total top-up amount, and
• at least the minimum top-up interval has elapsed since the last top-up.

The latter condition is a safeguard: it gives the operator a chance to intervene before CC is rapidly spent down in the case of a bug or attack causing runaway submissions. This is a tradeoff for the operator to manage: a shorter interval reduces the risk of traffic running out between top-ups, but increases the risk of CC being spent quickly before the operator has time to notice it; a longer interval gives the operator more time to notice problems, but increases the risk of running out of traffic. Both failure modes occur in practice: a node can stop transacting either because it ran out of traffic, or because it ran out of CC to purchase more.

Additionally, to prevent top-up transactions from failing due to an already depleted traffic balance, the validator app will abort ledger submissions if the balance has fallen below a predefined amount.

For configuring the built-in top-up automation, please refer to the Kubernetes validator deployment guide or the corresponding Docker-compose one. Configuring alternative methods for buying traffic, e.g., using third-party services, exceeds the scope of this documentation.

Wasted traffic

Wasted traffic is defined as synchronizer events that have been sequenced but will not be delivered to their recipients. For validators, which are subject to traffic fees, wasted traffic implies that traffic has been charged for a message that was ultimately not delivered. Not all failed submissions result in wasted traffic: wasted traffic only occurs whenever a synchronizer event is rejected after sequencing but before delivery. Some level of wasted traffic is expected and unavoidable, due to factors such as:

• Submission request amplification. Participants that use BFT sequencer connections retry submission requests after a timeout to ensure speedy delivery in the face of nonresponsive sequencers; if processing was simply slower than usual but the sequencer was not faulty, the duplicate request counts as wasted traffic.
• Duplication of messages within the ordering layer, typically linked to transient networking issues or load spikes.
• Duplication of submissions on the participant/app side, for example when catching up after restoring from a backup or after some crashes.

Validator perspective

Validator operators are encouraged to investigate the causes of repeatedly failing submissions. As stated above, not all failed submissions result in wasted traffic, and some wasted traffic is unavoidable. Attention is warranted, however, if the rate of wasted traffic increases significantly at some point in time.

The Splice distribution contains a Grafana dashboard about Synchronizer Fees (validator view), to assist in monitoring traffic-related metrics. The Rejected Event Traffic panel on this dashboard is especially relevant for determining the rate of wasted traffic. (Hover on the ⓘ symbols in panel headers for precise descriptions of the shown data.)

SV perspective

SV operators are encouraged to monitor wasted traffic across all synchronizer members, as reported for example by sequencer metrics, to detect cases where wasted traffic increases significantly and/or in a global manner. The Splice distribution contains a Grafana dashboard about Synchronizer Fees (SV view) that can be helpful, as well as an alert definition that focuses on validator participants.

Note that wasted traffic is less relevant for SVs themselves, as SV components have unlimited traffic. Note also that SV mediators and sequencers waste traffic as part of their regular operation: They heavily use aggregate submissions where sequencers collect messages from a group of senders and only deliver a single message per recipient once a threshold of individual submissions has been sequenced; sequenced individual submissions beyond the aggregation threshold count as wasted traffic. All that said, should an SV component suddenly exhibit a significant increase in wasted traffic, this likely points to an actual issue that should be investigated.