Control Plane IO

Open/R is peer-to-peer (P2P) connected. Neighboring instances exchanges the packets over connected links to make routing happen. This documentation provides overview of the packets and their signatures. Also highlights how Operators can ensure reliable Open/R communication in the network.

Control Plane Packets

Every data packet that is exchanged between Open/R instances is termed as Control Plane Packet in this context. There are two types of communication channels between neighboring Open/R instances, namely Neighbor Discovery and KvStore Synchronization.

Neighbor Discovery Packets

Spark is a neighbor discovery module of Open/R. It exchanges packets with neighboring Spark instances to discover and main neighbors. The packets exchanged have following characteristics

  • Internet Protocol is IPv6

    • Source address is IPv6 link-local address of source interface

    • Destination address is ff02::1 (IPv6 Link-Local Multicast)

    • TOS = 0xc0 (192) aka DSCP = 48 (Can be altered by OpenrConfig)

  • Transport Protocol is UDP

    • Source port is 6666 (Can be altered by OpenrConfig)

    • Destination port is 6666 (Can be altered by OpenrConfig)

Sample packets headers

11:30:26.223517 IP6 (class 0xc0, hlim 255, next-header UDP (17) payload length: 48) fe80::d8c4:97ff:fed0:5b14.6666 > ff02::1.6666: [udp sum ok] UDP, length 40

11:30:26.472102 IP6 (class 0xc0, hlim 255, next-header UDP (17) payload length: 148) fe80::d8c4:97ff:fece:2f5b.6666 > ff02::1.6666: [udp sum ok] UDP, length 140

11:30:26.963406 IP6 (class 0xc0, hlim 255, next-header UDP (17) payload length: 48) fe80::d8c4:97ff:fece:2f5b.6666 > ff02::1.6666: [udp sum ok] UDP, length 40

KvStore Synchronization Packets

KvStore is a distributed-replicated database. It communicates with neighboring KvStore instances to ensure consistency of data across all nodes within an area. The packets exchanged between KvStore have following characteristics

  • Internet Protocol is IPv6

    • Source address is IPv6 link-local address of source interface

    • Destination address is IPv6 link-local address of neighbor interface

    • TOS = 0xc0 (192) aka DSCP = 48 (Can be altered by OpenrConfig)

  • Transport Protocol is TCP

    • Source port is system generated by connect() syscall

    • Destination port is 2018, Thrift Server. (Can be altered by OpenrConfig)

Sample Packet headers

11:30:57.520356 IP6 (class 0xc0, hlim 64, next-header TCP (6) payload length: 73) fe80::d8c4:97ff:fed0:5b14.33642 > fe80::d8c4:97ff:fece:2f5b.2018: Flags [P.], cksum 0x8f05 (correct), seq 4248320327:4248320368, ack 3582705193, win 296, options [nop,nop,TS val 3417356247 ecr 3550908423], length 41

11:30:57.520788 IP6 (class 0xc0, hlim 64, next-header TCP (6) payload length: 76) fe80::d8c4:97ff:fece:2f5b.2018 > fe80::d8c4:97ff:fed0:5b14.33642: Flags [P.], cksum 0x013d (correct), seq 1:45, ack 41, win 239, options [nop,nop,TS val 3550928086 ecr 3417356247], length 44

11:30:57.520804 IP6 (class 0xc0, hlim 64, next-header TCP (6) payload length: 32) fe80::d8c4:97ff:fed0:5b14.33642 > fe80::d8c4:97ff:fece:2f5b.2018: Flags [.], cksum 0x851d (correct), seq 41, ack 45, win 296, options [nop,nop,TS val 3417356248 ecr 3550928086], length 0

Prioritizing Control Plane IO

Underlying Hardware or ASIC will be responsible for forwarding both Data Plane and Control Plane packets. Under stressful situations, high load of data plane packets may leads to drop of control plane packets. This can disrupt the core routing operations within network, making situation even worse with packet blackholing or routing loops.

Hence, is is important to prioritize the Control Plane packets over Data Plane packets in HW/ASIC, to ensure reliable delivery of Control Plane packets. This is usually achieved by introducing multiple priority queues. Special ACL rules in HW match the packets with control plane packet signature and re-directs them to the highest priority queue.

To facilitate the easy matching, every packet exchanged between Open/R instances is marked with specific TOS (or DSCP) value as described above. This value can be configured via OpenrConfig.

Client Packets

Open/R exposes Thrift RPC APIs for remote network operations e.g. retrieving network state or issuing drain/undrain commands. These are miscellaneous packets and not Open/R specific. Internet Protocol can be IPv4 or IPv6.

FBOSS’s SwitchAgent - Control Plane IO & WARM boot

FBOSS is an open-sourced switch operating system. Open/R is natively supported on FBOSS. FBOSS’s SwitchAgent is a core service that supports Control Plane IO so that protocols such as Open/R can talk to peer Open/R instances inband for their functioning. SwitchAgent does so by creating a TUN interface on Linux corresponding to each VLAN of the host. The VLAN status (UP/DOWN) is reflected on this Linux interface as well. The interfaces are named with the fboss prefix e.g. fboss4001. FBOSS’s SwitchAgent holds an FD to these VLAN so that it can multiplex packets between physical VLANs & their Linux counterpart.

Open/R uses the netlink protocol to subscribe links from Linux. And creates UDP/TCP sockets to send/receive packets to/from these interfaces.

This means Open/R can function if and only if SwitchAgent is functioning on the system. Both Open/R & SwitchAgent support WARM mode for service upgrades, we term it as Graceful Restart (GR). However, when SwitchAgent is undergoing restart the Open/R GR would be broken because all Linux interfaces would go down and Control Plane IO will be halted. To overcome this, we first intend to wrap SwitchAgent WARM restart with Open/R’s Graceful restart.

This is achieved by systemd’s BindsTo, After, and WantedBy directives of Open/R’s service spec. This will automatically bring down Open/R before SwitchAgent goes down and brings up Open/R after SwitchAgent comes up. In rare cases when SwitchAgent is crashing in a loop, you may also see Open/R restarting in a loop. In such cases systemd logs from /var/log/messages would be helpful in identifying service restart sequence & possible cause.