crazeliner.blogg.se - Teambeam github

TEAMBEAM GITHUB SOFTWARE

7:20 If you need low latency - below 10ms, for example - Spark isn’t there yet.7:10 The initial implementation was a bit of a hack - by running micro-batches, you can gain the appearance of streaming support.7:00 Obviously Spark is older and much more established, but the streaming support is newer.6:40 If you’d asked me a year or two ago, I wouldn’t have advised these, but we’ve since realised that Flink fills a couple of important niches.6:20 I’m concerned with the paradox of choice - if you go into a shop to buy a new refrigerator then you panic because you don’t know which one to buy - so I hesitate to give too many choices.6:00 Lightbend’s historic strength is in micro-service development with a reactive platform using tools like Akka, Play and Lagom and of course Scala as well.5:50 We’re seeing a convergence of architectures where people need a lot more services in a streaming context.5:35 It bundles tools like Kafka, Spark and Spark Streaming - but we also provide HDFS because people do want to use this platform for batch processing as well.

5:30 The analogy here is that we’re doing something similar, but for stream processing.

5:20 But you tend to go with a commercial distribution (like Cloudera, or Hortonworks) because you get support.

5:05 If you know Hadoop, you have a curated distribution of a bunch of tools, each of which plays a particular role.

What is the Lightbend fast data platform?

4:30 It’s not only a competitive advantage to move from batch processing to stream-oriented processing, but sometimes you have no choice.

4:00 Increasingly people need answers quicker - you can’t just capture that data and process it later, but you need to respond at scale.

3:30 It’s a made up title - I might be the only VP of fast data engineering in the world - but I’m focussing on building stream-oriented tooling for people in the big data world.

What is fast data engineering at Lightbend?

3:05 When the big data concept kicked off, it seemed like a natural fit for my maths skills and was an interesting space to get into.

TEAMBEAM GITHUB SOFTWARE

2:40 When I left, the data science concept wasn’t fully formed, and I had been writing software in part because I was doing a lot of programming to do calculations for my work.2:30 You learn a lot about probabilities and statistics with physics anyway, and not being intimidated in big problems that are hard to solve - so it’s a pretty natural transition.2:05 It was a long circuitous arc - today, a lot of people who come from that background (often physics in general) often jump into the big data world or data science.What got you from nuclear physics into big data?