0.3.0 Release + Benchmarking Suite

This release refreshes the query engine, swaps in a faster memory allocator, and adds two new ways to get data into a pipeline, along with many other smaller improvements and bug fixes.

This is a breaking change, as plugin interfaces were updated. All plugins will need to be recompiled with the latest streamling-plugin crate.

In our next release, we will be introducing more patterns and operators for complex enrichments, and partitioned processing for better CPU utilization.

Benchmarks

We created a benchmarking suite to catch regressions and compare Streamling against Bento and Flink on the same workloads.

The clearest way to see the difference is throughput per unit of memory. Streamling led with roughly 40x Flink on the Kafka workload and about 3x on the heavier Parquet → ClickHouse workload. On raw throughput, Streamling matched or surpassed Flink on Kafka and surpassed Bento by 2-5x across the board, while Flink took the lead on single-replica ClickHouse writes.

Streamling was built for stable and efficient performance on pipelines that power high-stakes financial use-cases, and in persistent production settings, stable and efficient resource usage is a key anchor for uptime. This benchmark will help make sure Streamling doesn't regress.

Kafka + SQL Projection

This benchmarks a read from Kafka (Avro via Schema Registry) → select every payload column plus lower(country) and one arithmetic column → discard data. It's focused on decoding and multi-transform data passing over sink writes.

Streamling leads on both throughput and efficiency, and the gap widens as machines get smaller:

Flink had different pod configurations as we wanted to make use of the parallelism settings to reflect real-world usage, which required 4 GB minimum. It was unable to run on the lowest machine setting.

Bento was the most memory-light, but was CPU-bound through the tests, resulting in lower throughput.

Streamling consistently had the lowest CPU usage and best memory-to-throughput ratio, resulting in the highest resource efficiency generally. However, Streamling has room to improve. The next release will focus on efficient data partitioning in order to have many internal 'copies' of the pipeline, allowing us to fully saturate resources.

Parquet -> ClickHouse benchmark

This benchmarks a much heavier workload: field-boundary GeoParquet (~218 GiB / 3.2B rows across 1000 files) read from a public S3 bucket → rename and flatten the nested fields, hex-encode the WKB geometry, derive an md5-based dedup key → ClickHouse Cloud ReplacingMergeTree.

This is a test against single deployment efficiency. The benchmarks use a nightly build that supports some parallel processing.

Flink wins on absolute throughput by saturating more of the provisioned CPU as parallelism is dialed up. Streamling uses substantially less CPU and memory for the same workload, but leaves headroom unused in a single replica.

However, in real usage, Streamling users spin up multiple replicas when higher throughput is needed. In production we've observed that this scales linearly, and given the resource headroom in these results, we expect multi-replica Streamling can surpass Flink's single-deployment throughput. We did not include a multi-replica mode in this suite, since the goal for the next release is to bring those efficiencies inside one deployment.

Bento could not be tested at comparable resource limits. The Parquet path materializes each whole S3 object as a single in-memory batch, so the densest file alone needs 30+ GB live. It required a 110 GB limit on a dedicated node (70 GB observed peak) to finish at all, while the other engines stream the same data inside 13 GB. This test will be re-run when the streaming parquet source is available.

Full results are in the benchmarking suite.

Updates in 0.3.0

DataFusion v54

Streamling now builds on the latest Apache DataFusion (v54), picking up its query-planning and execution improvements across the board.

Significant Kafka Source improvements

Avro decoding moved onto Arrow's native Avro support with avro-arrow. Records now decode straight into Arrow record batches instead of being parsed row by row and reassembled.

In production at Goldsky, pipelines reading from Kafka Avro topics saw an immediate 20% improvement with less CPU being used and higher throughput for the same workload and pipeline size.

Memory usage efficiency

After doing extensive benchmarking on mimalloc v2, jemalloc, and mimalloc v3, we found that mimalloc v3 lowers allocation latency and improves multi-threaded throughput, which translates to steadier performance under load and much lower memory usage.

Combined with an audit on memory usage and zero-copy enforcement across the codebase, we saw significant memory reductions for all running pipelines.

In Goldsky's cloud, which runs thousands of Streamling pipelines, an example customer saw a memory efficiency improvement from 31 GB to 4 GB on a workload that writes large (up to 1m rows) batches in real-time into ClickHouse.

New sources: File and Kafka JSON

Two new ways to read data:

• File source — point a pipeline at local or object-store files (Parquet, JSON, CSV, or Avro). Run it as a bounded source for backfills and replays, or as a continuous source that keeps watching the path for new data.
• Kafka JSON — the Kafka source now decodes JSON payloads in addition to Avro, so you can consume JSON topics without a schema registry.

Both look like any other source in your pipeline YAML:

sources:
  # Backfill from newline-delimited JSON files.
  seed.events:
    type: file
    path: /data/events
    format: json
    primary_key: id
    mode:
      type: bounded

  # Consume a JSON Kafka topic — no schema registry required.
  live.events:
    type: kafka
    topic: events.clicks
    data_format: json
    primary_key: id
    schema:
      id: string
      url: string
      user_id: string

ClickHouse Sink Optimization

Streamling's native ClickHouse sink sends Arrow format to ClickHouse, allowing for huge savings by skipping serialization. ZSTD and LZ4 are now added as compression codecs on top of Arrow, with ZSTD yielding significant efficiency improvements for most scenarios.

For large batches (100k rows per batch), sink flush times went down by 50%.

Previously, GZIP was used, but after testing with customer workloads the team found that ZSTD provided the best performance generally.

For small batches, sink flush latency is slightly higher. Combined with the network egress savings users will likely see, ZSTD was chosen as the default, but LZ4 and GZIP remain as options.

If you value the fastest writes over bandwidth savings, for example in low-latency or co-located deployments where you aren't network-bound, you can disable compression or select LZ4. In practice the gains are usually marginal.

Upgrade

Install or upgrade with the one-line bootstrap:

curl -fsSL https://streamling.dev/install.sh | bash

The full changelog is on GitHub.