# Control Plane IO Open/R is peer-to-peer (P2P) connected. Neighboring instances exchanges the packets over connected links to make routing happen. This documentation provides overview of the packets and their signatures. Also highlights how Operators can ensure reliable Open/R communication in the network. ## Control Plane Packets --- Every data packet that is exchanged between Open/R instances is termed as `Control Plane Packet` in this context. There are two types of communication channels between neighboring Open/R instances, namely `Neighbor Discovery` and `KvStore Synchronization`. ### Neighbor Discovery Packets `Spark` is a neighbor discovery module of Open/R. It exchanges packets with neighboring Spark instances to discover and main neighbors. The packets exchanged have following characteristics - Internet Protocol is IPv6 - Source address is IPv6 link-local address of source interface - Destination address is `ff02::1` (IPv6 Link-Local Multicast) - `TOS = 0xc0` (192) aka `DSCP = 48` (Can be altered by OpenrConfig) - Transport Protocol is UDP - Source port is `6666` (Can be altered by OpenrConfig) - Destination port is `6666` (Can be altered by OpenrConfig) #### Sample packets headers ``` 11:30:26.223517 IP6 (class 0xc0, hlim 255, next-header UDP (17) payload length: 48) fe80::d8c4:97ff:fed0:5b14.6666 > ff02::1.6666: [udp sum ok] UDP, length 40 11:30:26.472102 IP6 (class 0xc0, hlim 255, next-header UDP (17) payload length: 148) fe80::d8c4:97ff:fece:2f5b.6666 > ff02::1.6666: [udp sum ok] UDP, length 140 11:30:26.963406 IP6 (class 0xc0, hlim 255, next-header UDP (17) payload length: 48) fe80::d8c4:97ff:fece:2f5b.6666 > ff02::1.6666: [udp sum ok] UDP, length 40 ``` ### KvStore Synchronization Packets `KvStore` is a distributed-replicated database. It communicates with neighboring KvStore instances to ensure consistency of data across all nodes within an area. The packets exchanged between KvStore have following characteristics - Internet Protocol is IPv6 - Source address is IPv6 link-local address of source interface - Destination address is IPv6 link-local address of neighbor interface - `TOS = 0xc0` (192) aka `DSCP = 48` (Can be altered by OpenrConfig) - Transport Protocol is TCP - Source port is system generated by `connect()` syscall - Destination port is `2018`, Thrift Server. (Can be altered by OpenrConfig) #### Sample Packet headers ``` 11:30:57.520356 IP6 (class 0xc0, hlim 64, next-header TCP (6) payload length: 73) fe80::d8c4:97ff:fed0:5b14.33642 > fe80::d8c4:97ff:fece:2f5b.2018: Flags [P.], cksum 0x8f05 (correct), seq 4248320327:4248320368, ack 3582705193, win 296, options [nop,nop,TS val 3417356247 ecr 3550908423], length 41 11:30:57.520788 IP6 (class 0xc0, hlim 64, next-header TCP (6) payload length: 76) fe80::d8c4:97ff:fece:2f5b.2018 > fe80::d8c4:97ff:fed0:5b14.33642: Flags [P.], cksum 0x013d (correct), seq 1:45, ack 41, win 239, options [nop,nop,TS val 3550928086 ecr 3417356247], length 44 11:30:57.520804 IP6 (class 0xc0, hlim 64, next-header TCP (6) payload length: 32) fe80::d8c4:97ff:fed0:5b14.33642 > fe80::d8c4:97ff:fece:2f5b.2018: Flags [.], cksum 0x851d (correct), seq 41, ack 45, win 296, options [nop,nop,TS val 3417356248 ecr 3550928086], length 0 ``` ## Prioritizing Control Plane IO --- Underlying Hardware or ASIC will be responsible for forwarding both `Data Plane` and `Control Plane` packets. Under stressful situations, high load of data plane packets may leads to drop of control plane packets. This can disrupt the core routing operations within network, making situation even worse with packet blackholing or routing loops. Hence, is is important to `prioritize` the `Control Plane` packets over `Data Plane` packets in HW/ASIC, to ensure reliable delivery of `Control Plane` packets. This is usually achieved by introducing multiple priority queues. Special ACL rules in HW match the packets with control plane packet signature and re-directs them to the highest priority queue. To facilitate the easy matching, every packet exchanged between Open/R instances is marked with specific `TOS` (or `DSCP`) value as described above. This value can be configured via OpenrConfig. ### Client Packets Open/R exposes Thrift RPC APIs for remote network operations e.g. retrieving network state or issuing drain/undrain commands. These are miscellaneous packets and not Open/R specific. `Internet Protocol` can be IPv4 or IPv6. ## FBOSS's SwitchAgent - Control Plane IO & WARM boot --- FBOSS is an open-sourced switch operating system. Open/R is natively supported on FBOSS. FBOSS's `SwitchAgent` is a core service that supports Control Plane IO so that protocols such as Open/R can talk to peer Open/R instances inband for their functioning. `SwitchAgent` does so by creating a TUN interface on Linux corresponding to each VLAN of the host. The VLAN status (UP/DOWN) is reflected on this Linux interface as well. The interfaces are named with the `fboss` prefix e.g. `fboss4001`. FBOSS's `SwitchAgent` holds an FD to these VLAN so that it can multiplex packets between physical VLANs & their Linux counterpart. Open/R uses the `netlink` protocol to subscribe links from Linux. And creates UDP/TCP sockets to send/receive packets to/from these interfaces. This means Open/R can function if and only if SwitchAgent is functioning on the system. Both Open/R & SwitchAgent support WARM mode for service upgrades, we term it as Graceful Restart (GR). However, when SwitchAgent is undergoing restart the Open/R GR would be broken because all Linux interfaces would go down and Control Plane IO will be halted. To overcome this, we first intend to wrap SwitchAgent WARM restart with Open/R's Graceful restart. This is achieved by `systemd`'s `BindsTo`, `After`, and `WantedBy` directives of Open/R's service spec. This will automatically bring down Open/R before SwitchAgent goes down and brings up Open/R after SwitchAgent comes up. In rare cases when SwitchAgent is crashing in a loop, you may also see Open/R restarting in a loop. In such cases `systemd` logs from `/var/log/messages` would be helpful in identifying service restart sequence & possible cause.