Multicast and the Markets with Brian Nigito

Market Structure and Exchange Evolution

• Modern securities exchanges are heavily regulated, electronic platforms that have replaced physical trading floors with purely electronic intermediaries over the last 20 years.
• Exchange rule sets have shifted from being forgiving of electronic timing to mandating continuous, low-latency access to quotations, fundamentally altering participation dynamics.
• The exchange ecosystem includes not only trading firms but also clearing houses (guarantors), other exchanges (which trade against each other's quotes), and market data providers.
• Core transactional messages are limited to orders (buy/sell instructions), cancellations, and executions (confirmations of trade).
• The "order book" is the fundamental data structure comprising unsatisfied orders, which are added, removed, or matched to generate executions.
• Reliability and fairness in data dissemination are critical constraints; losing track of a transaction or introducing latency disparities can result in significant financial loss and regulatory issues.
• Competition among participants drives an arms race for speed and determinism, as faster data access translates directly to better pricing and execution opportunities.

Networking Protocols: TCP vs. UDP and Multicast

• Order Entry (Private Stream): Uses TCP due to the need for reliability and ordered delivery, which is non-negotiable for trade confirmations and cancellations.
• Market Data (Public Stream): Uses UDP multicast because it allows a single message to be distributed to thousands of recipients simultaneously via switch hardware, rather than sending individual copies.
• Multicast Advantage: Network switches perform packet copying in specialized hardware (electrically repeating bits), making the process orders of magnitude faster and more deterministic than software-based unicast replication on a server.
• Co-location: Firms locate servers physically near exchange switches to minimize network hops, enabling the use of unreliable UDP and multicast over private, low-loss local area networks.
• Failure of Multicast on Open Internet: Multicast is ineffective on the public internet due to routing complexities and lack of support; it is viable only in controlled, private environments like financial exchanges.
• Adoption Barriers: While protocols like UFO (UDP for Orders) exist, widespread adoption is stalled by the high interoperability cost of maintaining multiple protocol stacks across dozens of exchanges.

Architecture and Performance Optimization

• The Sequencer Model: Many high-performance exchanges utilize a "sequencer" (often a single machine/core) that serializes all events and distributes them via multicast, simplifying concurrency and ensuring deterministic ordering.
• Specialized Recovery: Since standard reliable multicast is too slow for high-frequency trading, exchanges build custom recovery layers that add sequence numbers and specific retransmission logic for lost packets without blocking the main stream.
• State Compression: Proprietary protocols allow traders to replay only necessary state snapshots (e.g., 20% of the day) to bring systems online quickly, bypassing the need to buffer and replay entire message streams.
• Mechanical Sympathy: High-performance engineering requires a deep understanding of physical hardware limits, including CPU instruction pipelines, memory latency, and network card capabilities, rather than relying on high-level abstractions.
• Zero-Copy I/O: Modern systems eliminate data copying by having network cards write directly to user-space memory buffers, avoiding kernel-space copies that incur significant latency overhead.
• Interrupt vs. Polling: Systems have shifted from interrupt-driven I/O to user-space polling to reduce latency and CPU overhead, checking for data availability rather than waiting for hardware interrupts.

Hardware and Future Trends

• Custom Hardware: As data rates approach 17–25 Gbps, firms increasingly rely on FPGAs and custom programmable network cards to filter and aggregate market data at wire speed.
• L1 Crosspoint Switches: Layer 1 crosspoint devices provide electrical replication with latencies of 3–5 nanoseconds, significantly faster than traditional store-and-forward (7–10 µs) or cut-through (300–500 ns) switches.
• Serialization Delay: At high speeds (e.g., 10 Gbps to 25 Gbps), the physical time required to push a packet onto the wire (approx. 1 nanosecond per byte) becomes a dominant latency factor that cannot be bypassed.
• PCIe vs. Network Latency: The latency of moving data between cores (approx. 100 ns) and across PCIe (300–600 ns) is approaching the latency of network transmission in highly optimized co-location environments.
• RDMA and InfiniBand: While Remote Direct Memory Access (RDMA) offers zero-copy, CPU-independent data movement, its reliability mechanisms (e.g., flow control backpressure) can introduce unpredictable latency spikes in high-volume multicast scenarios.
• Standardization of High-Speed Tech: Technologies previously niche to trading, such as user-space polling (IO_uring) and 25 Gbps signaling, are gradually being standardized in Linux and commodity hardware, though higher signaling rates can introduce latency via forward error correction.
• Commodity Hardware Evolution: The industry faces a shift from "relentless commodity growth" (x86) back toward specialization, as Moore's Law slows and the value of domain-specific hardware (GPUs, FPGAs) increases for ultra-low latency tasks.