最后一章

There were two main reasons for Flink’s rise to prominence:
Its rapid adoption of the Dataflow/Beam programming model, whichput it in the position of being the most semantically capable fully open source streaming system on the planet at the time.
 
Followed shortly the reafterby its highly efficient snapshotting implementation (derived from research in Chandy and Lamport’s original paper “Distributed Snapshots: Determining Global States of Distributed Systems” which gave it the strong consistency guarantees needed for correctness.
The simple fact that Flink now had the capability to provide exactly-once semantics along with native support for event-time processing was huge at the time. But it wasn’t until Jamie Grier published his article titled “Extending the Yahoo! Streaming Benchmark” that it became clear just how performant Flink was. In that article, Jamie described two impressive achievements:
Building a prototype Flink pipeline that achieved greater accuracy than one of Twitter’s existing Storm pipelines (thanks to Flink’s exactly-once semantics) at 1% of the cost of the original.
 
Updating the Yahoo! Streaming Benchmark to show Flink (with exactly-once) achieving 7.5 times the throughput of Storm (without exactly-once). Furthermore, Flink’s performance was shown to be limited due to network saturation; removing the network bottleneck allowed Flink to achieve almost 40 times the throughput of Storm.
 
In summary, Flink’s rapid rise to stream processing juggernaut can be attributed primarily to three characteristics of its approach: 
incorporating the best existing ideas from across the industry (e.g., being the first open source adopter of the Dataflow/Beam Model) 
bringing its own innovations to the table to push forward the state of the art (e.g., strong consistency via snapshots and savepoints, streaming SQL) 
doing both of those things quickly and repeatedly. Add in the fact that all of this is done in open source, and you can see why Flink has consistently continued to raise the bar for streaming processing across the industry.
An example here is the state snapshotting mechanism in Flink, or savepoints, which we discussed earlier. Flink is still the only publicly available streaming system to support snapshots in this way, but there’s a proposal in Beam to provide an API around snapshots because we believe graceful evolution of pipelines over time is an important feature that will be valuable across the industry. If we were to magically push out such an API today, Flink would be the only runtime system to support it. But again, that’s okay. The point here is that the industry as a whole will begin to catch up over time as the value of these features becomes clear. And that’s better for everyone.
The core vision for Beam is built around its value as a portability layer, and one of the more compelling features in that realm is its planned support for full cross-language portability.Though not yet fully complete(butlanding imminently), the plan is for Beam to provide sufficiently performant abstraction layers between SDKs and runners that will allow for a full cross-product of SDK × runner matchups. In such a world, a pipeline written in a JavaScript SDK could seamlessly execute on a runner written in Haskell, even if the Haskell runner itself had no native ability to execute JavaScript code continued to raise the bar for streaming processing across the industry.
MapReduce—scalability and simplicity
By providing a simple set of abstractions for data processing on top of a robust and scalable execution engine, MapReduce allowed data engineers to focus on the business logic of their data processing needs rather than the gnarly details of building distributed systems resilient to the failure modes of commodity hardware.
Hadoop—open source ecosystem
By building an open source platform on the ideas of MapReduce, Hadoop created a thriving ecosystem that expanded well beyond the scope of its progenitor and allowed a multitude of new ideas to flourish.
Flume—pipelines, optimization
By coupling a high-level notion of logical pipeline operations with an intelligent optimizer, Flume made it possible to write clean and maintainable pipelines whose capabilities extended beyond the Map → Shuffle → Reduce confines of MapReduce, without sacrificing any of the performance theretofore gained by contorting the logical pipeline via hand-tuned manual optimizations.
Storm—low latency with weak consistency
By sacrificing correctness of results in favor of decreased latency, Storm brought stream processing to the masses and also ushered in the era of the Lambda Architecture, where weakly consistent stream processing engines were run alongside strongly consistent batch systems to realize the true business goal of low-latency, eventually consistent results.
Spark—strong consistency
By utilizing repeated runs of a strongly consistent batch engine to provide continuous processing of unbounded datasets, Spark Streaming proved it possible to have both correctness and low-latency results, at least for in-order datasets.
MillWheel—out-of-order processing
By coupling strong consistency and exactly-once processing with tools for reasoning about time like watermarks and timers, MillWheel conquered the challenge of robust stream processing over out-of-order data.
Kafka—durable streams, streams and tables
By applying the concept of a durable log to the problem of streaming transports, Kafka brought back the warm, fuzzy feeling of replay ability that had been lost by ephemeral streaming transports like RabbitMQ and TCP sockets. And by popularizing the ideas of stream and table theory, it helped shed light on the conceptual underpinnings of data processing in general.
Cloud Dataflow—unified batch plus streaming
By melding the out-of-order streamprocessing concepts from MillWheel with the logical, automatically optimizable pipelines of Flume, Cloud Dataflow provided a unified model for batch plus streaming data processing that provided the flexibility to balance the tensions between correctness, latency, and cost to match any given use case.
Flink—open source stream processing innovator
By rapidly bringingthe power of out-of-order processing to the world of open source and combining it with innovations of their own like distributed snapshots and its related savepoints features, Flink raised the bar for open source stream processing and helped lead the current charge of stream processing innovation across the industry.
Beam—portability
By providing a robust abstraction layer that incorporates the best ideas from across the industry, Beam provides a portability layer positioned as the programmatic equivalent to the declarative lingua franca provided by SQL, while also encouraging the adoption of innovative new ideas throughout the industry.