A streaming analytics platform for trading desks under load.
STACK
A streaming analytics platform that ingests market data, computes real-time aggregates, and serves dashboards to trading desks — under sustained load with sub-second latency targets.
The trading desk’s existing dashboards ran on hourly batch jobs — by the time a metric appeared, it was already stale for a fast-moving market. They needed sub-second visibility into positions and market aggregates, at volumes existing tooling had never been tested against.
Replaced the batch pipeline with a streaming architecture: market data lands in Kafka, gets aggregated in near-real-time, and lands in ClickHouse for sub-second dashboard queries — with the dashboards themselves subscribing to live updates instead of polling.
Every market data event lands in Kafka first, decoupling ingestion volume spikes from downstream processing capacity.
Chosen specifically for its columnar storage and real-time aggregation performance at the query volumes trading desks generate.
Built the Kafka ingestion layer and the ClickHouse aggregation pipeline.
Shipped live-subscribing dashboards and load-tested against 5x peak volume.
Decoupling ingestion from processing via Kafka meant a downstream slowdown never became an ingestion-side data loss problem.
Load testing against realistic trading-hours traffic patterns — not synthetic even load — surfaced bottlenecks a steady-load test never would have.
If this project resonates with what you're building, let's talk. I take on a limited number of projects each quarter.
hello@martinsai.name.ng · Response within 24h