Skip to main content

IEEE 802.1 TSN toolbox: how each mechanism extends the Ethernet bridge

· 14 min read
PhD candidate, LyRIDS (ECE) × CEDRIC (Cnam)

Time-Sensitive Networking (TSN) is a set of IEEE 802.1 standards for time-sensitive transmission over Ethernet. This post goes through the tools one by one, always asking the same question: which part of an ordinary Ethernet bridge does this tool change?

The IEEE 802.1Q bridge: architecture and data-plane actionsIEEE 802.1Q bridge ↗Control planeSRP (Qat), TSN configuration (Qcc), path control (Qca)frame inframe outIngressPSFP (Qci)FRER (CB)Relayforwardingexplicit treesQueuingCBS (Qav)ATS (Qcr)Tx selectiongates (Qbv)CQF (Qch)MAC / PHYpreemption(Qbu, 802.3br)Common time: gPTP (802.1AS)
Figure 1. TSN is not a new bridge: each tool extends one stage of the existing IEEE 802.1Q bridge. Each box names the stage and the TSN tools added to it (802.1 standard in brackets).Source: own drawing, after the bridge architecture, data-plane actions and QoS function chain in J. Farkas, TSN Basic Concepts, DetNet–TSN workshop, 11 November 2018, slides 7, 8 and 10.
Abbreviations used in this post (17)
AbbreviationStands forMeaning
ATSAsynchronous Traffic ShapingIEEE 802.1Qcr: per-stream token-bucket shaping at every hop using local clocks only.
AVBAudio Video BridgingThe predecessor of TSN: gPTP, the credit-based shaper and stream reservation, combined for audio and video (profile: IEEE 802.1BA).
CNCCentralized Network ConfigurationThe 802.1Qcc entity that computes paths, schedules and reservations for the whole network and configures the bridges.
CQFCyclic Queuing and ForwardingIEEE 802.1Qch: two queues per class alternate every cycle, so a frame advances one hop per cycle.
DetNetDeterministic NetworkingThe IETF architecture and working group for bounded-latency, low-loss flows over IP and MPLS (RFC 8655).
FRERFrame Replication and Elimination for ReliabilityIEEE 802.1CB: frames are numbered, sent over disjoint paths, and duplicates are removed.
gPTPgeneralized Precision Time ProtocolThe profile of PTP defined by IEEE 802.1AS, in which every bridge takes part in time distribution.
IEEEInstitute of Electrical and Electronics EngineersThe professional association whose Standards Association publishes the IEEE 802 LAN/MAN standards.
IPInternet ProtocolThe network-layer protocol of the Internet (IPv4, RFC 791; IPv6, RFC 8200).
IS-ISIntermediate System to Intermediate SystemA link-state routing protocol (ISO/IEC 10589, RFC 1195); the control plane of SPB and PCR.
MACMedium Access ControlThe lower sublayer of the data link layer, which frames data and controls access to the medium. A MAC address identifies a network interface.
NETCONFNetwork Configuration ProtocolThe IETF protocol for installing and reading device configuration described in YANG (RFC 6241).
PSFPPer-Stream Filtering and PolicingIEEE 802.1Qci: stream filters, stream gates and flow meters that enforce each stream’s contract at ingress.
RESTCONFREST-based configuration protocolAn HTTP-based protocol for access to YANG-modelled data (RFC 8040).
SDNSoftware-Defined NetworkingAn architecture that separates the control plane, run by a logically centralised controller, from the data plane of the network elements (RFC 7426).
TSNTime-Sensitive NetworkingThe set of IEEE 802.1 standards that give Ethernet bounded latency, time synchronisation and high reliability.
YANGYet Another Next Generation (data modelling language)The language in which network configuration and state are modelled for NETCONF and RESTCONF (RFC 7950).

1. Definition​

TSN is the set of standards produced by the TSN task group of IEEE 802.1 to provide deterministic services over Ethernet (IEEE 802) networks. Most of them are extensions of Bridges and Bridged Networks (IEEE 802.1Q), the standard for bridges and virtual LANs, and aim at very low latency and high availability [1][2]. The organisation behind them is described in the IEEE 802.1 post.

Figure 1 is the picture to keep in mind. A bridge already has a control plane and a data plane made of ingress processing, a relay, queues, transmission selection and a MAC [3]. Every TSN tool extends one of those existing stages. This is why TSN traffic and ordinary traffic can share a bridge: the eight priorities of 802.1Q are kept for backward compatibility, and the tools are added around them [2].

The bridge itself (its control and data planes, the actions of each port and the steps of the forwarding process) is described in the bridge post.

The TSN standards at a glance​

TSN is not one standard but a family. Each standard covers one function, and a system designer combines the ones a system needs, which is how TSN lets IT and OT traffic share one Ethernet connection [5]. The table follows the overview of the main TSN standards in [5], with each name linked to its post in this series and two standards added that complete the set.

StandardNameWhat it addsGroup
802.1AS-2020Timing and SynchronizationA common time in every bridge and end station (gPTP)Time synchronisation
802.1QavForwarding and Queuing Enhancements for time-sensitive streamsThe credit-based shaper, which spreads reserved traffic over timeBounded latency
802.1QbvEnhancements for Scheduled TrafficGates opened and closed on a timetable (time-aware shaper)Bounded latency
802.1QbuFrame PreemptionExpress frames interrupt preemptable ones (with 802.3br in the Ethernet MAC)Bounded latency
802.1QchCyclic Queuing and ForwardingTwo queues per class alternate, one hop per cycleBounded latency
802.1QcrAsynchronous Traffic ShapingPer-stream token-bucket shaping without a clock (not in [5])Bounded latency
802.1QciPer-Stream Filtering and PolicingEach stream checked against its contract at ingressReliability
802.1CBFrame Replication and Elimination for ReliabilityCopies over disjoint paths, duplicates removedReliability
802.1QcaPath Control and ReservationExplicit, non-shortest paths installed through IS-ISReliability
802.1QatStream Reservation ProtocolDistributed admission control, hop by hopResources and configuration
802.1QccStream Reservation Protocol enhancements and TSN configurationCentralised configuration with a CUC and a CNCResources and configuration

On top of these base standards, profiles select the tools and settings for one application: 802.1BA for audio and video, 802.1CM for mobile fronthaul, IEC/IEEE 60802 for industrial automation and 802.1DG for vehicles. The rest of this post goes through the four groups in turn.

Why standard Ethernet is not enough​

Ordinary network equipment has no notion of time, and nothing at the link layer prevents congestion. Even with eight strict priorities, a frame of the highest priority can wait behind a lower-priority frame whose transmission has already started [2]. TSN addresses both problems.

Basic terms​

From the point of view of 802.1 there are two kinds of device, bridges and end stations [3]:

TermMeaning
TalkerThe end station that is the source of a stream
ListenerAn end station that receives a stream
StreamA unidirectional flow of data from one talker to one or more listeners
BridgeThe relay between LANs; its data plane has a MAC relay and at least two ports

Three kinds of traffic then coexist on the network: time-triggered traffic, rate-constrained streams and best-effort traffic [2].

A bridged network of edge and core bridges with talkers, listeners and three streams
Figure 1b. The basic components on the original slide: talkers and listeners at the edge, edge and core bridges in between, and three streams crossing the bridged network. Each bridge is drawn with its bridge ID and port IDs.Source: J. Farkas, “TSN Basic Concepts,” DetNet–TSN workshop, IEEE 802.1, 11 November 2018, slide 6; cropped, otherwise unchanged.

2. Four groups of tools​

TSN components: synchronisation, reliability, latency and resource management, with the standards of each group
Figure 2. The TSN toolbox: four groups of standards and projects, which together give zero congestion loss and bounded latency. Projects whose name starts with “P” were still in progress in 2018.Source: J. Farkas, “TSN Basic Concepts,” DetNet–TSN workshop, IEEE 802.1, 11 November 2018, slide 5; cropped, otherwise unchanged.

The task group presents its standards as a toolbox with four groups: time synchronisation, bounded low latency, ultra reliability, and dedicated resources with their management interfaces [3][4]. A tool can be used on its own, but the full guarantee comes from combining them [2]. The sections below follow this grouping.

3. Time synchronisation: 802.1AS​

In depth: the 802.1AS post.

IEEE 802.1AS defines the generalised Precision Time Protocol (gPTP), a tightly constrained profile of IEEE 1588 that distributes time through the network itself with sub-microsecond precision [2]. The 2020 revision improved the accuracy of time measurement and added multiple time domains for redundancy [2]; work on the revision also covered link aggregation, faster change-over between grandmaster clocks, and redundant paths and grandmasters [4].

In Figure 1, time is the bar under the whole data plane. Gates, cyclic queuing and time-gated policing are meaningless without it, whereas the credit-based shaper and asynchronous shaping do not need it.

4. Bounded low latency​

Credit-Based Shaper: 802.1Qav​

In depth: the credit-based shaper post.

The oldest shaper, inherited from AVB, spreads the frames of a class evenly over time so that bursts do not overflow buffers downstream. Each shaped queue has a credit, counted in bits, and may transmit only while its credit is not negative. The slope at which credit accumulates is the bandwidth reserved for the class [2].

AVB defines two shaped classes. Class A has a 125 µs transmission period and a target worst-case latency of 2 ms over seven hops; class B has a 250 µs period and a 50 ms target. Together they may use at most 75 % of the link bandwidth [2]. Combined with stream reservation, the shaper gives a delay below 250 µs per bridge [4], which is too large for many control applications [2] and motivated the tools that follow.

Scheduled traffic: 802.1Qbv​

gate, critical queuegate, other queuesframes on the linkcycleopenclosedclosedopencriticalotherothercycleopenclosedclosedopencriticalotherotherguard band: no new non-critical frame may start
Figure 3. Scheduled traffic (802.1Qbv). Gates open and close on a repeating cycle; closing the non-critical gates in advance (guard band) keeps the critical window free.Source: own drawing, after J. Farkas, TSN Basic Concepts, DetNet–TSN workshop, 11 November 2018, slide 11, and J. Farkas, Introduction to IEEE 802.1, 15 May 2017, slides 31 and 33.

In depth: the Time-Aware Shaper post.

A transmission gate is associated with each queue. A queue whose gate is closed is not considered by transmission selection; a queue whose gate is open is served by its usual algorithm. Gates follow a list that repeats periodically, relative to the common time base [3][4]. The result divides time on the link into windows reserved for given priorities, in the manner of time-division multiple access [2].

This mechanism is the Time-Aware Shaper of the analysis literature. The service it offers to a queue is derived in the service curves post.

Frame preemption: 802.3br and 802.1Qbu​

express frame becomes readywithout preemptionpreemptable frameexpressexpress frame waitswith preemptionfragment 1expressfragment 2 (remainder)time
Figure 4. Frame preemption (802.3br and 802.1Qbu). The operation is local to one link: the fragments are reassembled at the next hop.Source: own drawing, after J. Farkas, Introduction to IEEE 802.1, 15 May 2017, slides 29 and 30, and J. Farkas, TSN Basic Concepts, DetNet–TSN workshop, 11 November 2018, slide 12.

In depth: the 802.3br post for the Ethernet side and the 802.1Qbu post for the bridge side.

A gate does not help if a long frame is already on the wire when the critical window opens. Preemption lets an express frame suspend the transmission of a preemptable frame, which resumes afterwards. Each traffic-class queue is assigned one of the two statuses [4]. The MAC is split into an express MAC and a preemptable MAC above a merge sublayer; the minimum fragment is 64 bytes including its check sequence [4].

Preemption reduces delay variation for express traffic and returns to preemptable traffic the bandwidth that a guard band would otherwise waste. It is local to a link, not a form of IP fragmentation [3].

Cyclic Queuing and Forwarding: 802.1Qch​

cycle 1cycle 2cycle 3cycle 4bridge A sendsframebridge B sendsframebridge C sendsframetime
Figure 5. Cyclic Queuing and Forwarding (802.1Qch). A frame received in one cycle is transmitted in the next, so latency depends only on the cycle time and the number of hops.Source: own drawing, after J. Farkas, Introduction to IEEE 802.1, 15 May 2017, slides 35 and 36, and the Wikipedia article on TSN.

In depth: the CQF post.

Two buffers per class are filled and drained in alternate cycles, synchronised across the network. Frames received during one cycle are sent during the next, so a frame stays one cycle at each hop [4]. Latency then depends only on the cycle time and the hop count, not on the topology or on the other flows [2].

Asynchronous Traffic Shaping: 802.1Qcr​

In depth: the ATS post.

Asynchronous Traffic Shaping (ATS) achieves zero congestion loss without time synchronisation. Traffic is re-shaped at every hop, and urgent traffic is prioritised over relaxed traffic. It resembles per-flow shaping, with one important difference: all streams going from one input port to one output port share a queue, and the appropriate shaper state is applied to each frame in front of that queue [3]. The shaper applies a token-bucket rule using local clocks and assigns each frame an eligibility time; strict-priority selection then operates on eligible frames [3].

ATS reuses the per-stream stages shown in the next figure. Its network calculus model, the interleaved regulator, is discussed in the TSN shapers post.

5. Ultra reliability​

Per-Stream Filtering and Policing: 802.1Qci​

incomingframeStream filterstream ID, priority,countersStream gateopen or closed,sets internal priorityFlow meter (PSFP)bandwidth profile,red / yellow / greenATS shapertoken bucket,eligibility timeQueuing
Figure 6. Per-stream processing ahead of the queues. PSFP (802.1Qci) meters and drops; ATS (802.1Qcr) reuses the filter and gate but delays each frame to an eligibility time instead.Source: own drawing, after J. Farkas, TSN Basic Concepts, DetNet–TSN workshop, 11 November 2018, slides 13 and 15.

In depth: the PSFP post.

Guarantees computed for well-behaved streams are worthless if one talker exceeds its contract. PSFP protects the network against bandwidth violation, malfunction and attack by taking decisions per stream at the ingress [3]. A frame passes through three elements [3][4]:

  1. a stream filter, which identifies the stream and selects the gate and meter to apply;
  2. a stream gate, open or closed on a time schedule, which can also set the internal priority of the frame;
  3. a flow meter, which enforces a bandwidth profile and marks frames red, yellow or green.

In the terms of this series, PSFP is what makes the declared arrival curve an enforced one.

Frame Replication and Elimination: 802.1CB​

TalkerBridge N1replicatepath 1path 2Bridge N2eliminateListener3 2 13 2 13 2 13 2 1one copy per path:one failure, no loss
Figure 7. Frame Replication and Elimination for Reliability (802.1CB): numbered frames are sent on two maximally disjoint paths and the extra copies are deleted.Source: redrawn after J. Farkas, TSN Basic Concepts, DetNet–TSN workshop, 11 November 2018, slide 16.

In depth: the FRER post.

FRER avoids frame loss caused by equipment failure. It is a per-frame 1+1 (or 1+n) redundancy: frames are numbered, sent on two or more maximally disjoint paths, then recombined with the extra copies deleted. There is no failure detection and no switch-over, hence no interruption when a path fails [3].

Path control: 802.1Qca​

Disjoint paths have to be installed. 802.1Qca extends the IS-IS routing protocol beyond shortest-path trees with explicit trees, in a hybrid arrangement: IS-IS provides topology discovery and default paths, while one or more controllers define the explicit trees [3].

IS-IS shortest path tree and an explicit tree installed by a controller between edge bridges
Figure 7b. Path control (802.1Qca) on the original slide: IS-IS keeps the shortest path trees, and a controller (Ctrl) adds an explicit tree between edge bridges EB1 and EB2 through core bridges CB1 and CB2.Source: J. Farkas, “TSN Basic Concepts,” DetNet–TSN workshop, IEEE 802.1, 11 November 2018, slide 17; cropped, with neighbouring slide text, logos or navigation icons blanked.

6. Dedicated resources and configuration​

Stream reservation: 802.1Qat​

In depth: the stream reservation post.

The Stream Reservation Protocol performs distributed admission control. A talker advertises a stream; the advertisement propagates towards the listeners while the worst-case latency is accumulated; a listener that accepts triggers the reservation of bandwidth back along the path. Streams are identified by a 64-bit stream identifier, made of a 48-bit MAC address and a 16-bit unique identifier [2].

TSN configuration: 802.1Qcc​

The fully distributed, centralised network and distributed user, and fully centralised configuration models
Figure 8. The three configuration models of 802.1Qcc: fully distributed, centralized network and distributed user, and fully centralized.Source: J. Farkas, “TSN Basic Concepts,” DetNet–TSN workshop, IEEE 802.1, 11 November 2018, slide 18; cropped, with neighbouring slide text, logos or navigation icons blanked.

In depth: the SDN post.

802.1Qcc enhances the reservation protocol and defines three configuration models [3]:

  • fully distributed: end stations and bridges exchange requirements hop by hop;
  • centralised network, distributed user: end stations still signal to the network edge, but a central entity computes and configures;
  • fully centralised: a Centralized User Configuration entity collects the requirements of the end stations and passes them to a Centralized Network Configuration entity.

The CNC has a view of the whole network, performs reservation and scheduling centrally, and configures bridges through NETCONF or RESTCONF with YANG data models [2]. This is the architecture assumed by most work on online reconfiguration, including the reconfiguration post: the CNC is the place where an admission test must run.

7. Summary​

GroupToolStandardStage of the bridge it extendsNeeds time sync
TimegPTP802.1ASCommon time base–
LatencyCredit-Based Shaper802.1QavQueuingNo
LatencyScheduled traffic802.1QbvTransmission selectionYes
LatencyFrame preemption802.3br, 802.1QbuMACNo
LatencyCyclic Queuing and Forwarding802.1QchTransmission selectionYes
LatencyAsynchronous Traffic Shaping802.1QcrIngress and queuingNo
ReliabilityPer-Stream Filtering and Policing802.1QciIngress portFor time-gated use
ReliabilityFrame Replication and Elimination802.1CBIngress and egress portsNo
ReliabilityPath control802.1QcaControl planeNo
ResourcesStream reservation802.1QatControl planeNo
ResourcesTSN configuration802.1QccControl planeNo

References​

  1. IEEE 802.1, Time-Sensitive Networking (TSN) Task Group. Web page
  2. Wikipedia, "Time-Sensitive Networking." Article
  3. J. Farkas, "TSN Basic Concepts," DetNet–TSN workshop, 11 November 2018. Slides
  4. J. Farkas, "Introduction to IEEE 802.1: Focus on the Time-Sensitive Networking Task Group," 15 May 2017. Slides
  5. J. Steinheider (NXP Semiconductors), "The Fundamentals of Time-Sensitive Networking," All About Circuits, industry article, 19 January 2021, Figure 1 and section "TSN Standards." Article