polyzos / stream-processing-with-apache-flink Goto Github PK

Hi there- thank you so much for this repo, and the forthcoming book! I'm looking forward to it.

I want to ask a question about how the networking is working to clarify my understanding on the Flink cluster setup. Please let me know if GitHub is the wrong place to ask this question, and if there is anywhere else that I could ask it!

I understand that the JobManager has two ports exposed to the host machine, 8081 for the web interface and 9249 for Flink internal metrics reporting. Each of the TaskManager containers then map from 9249 internally to 9250 and 9251 respectively to similarly report metrics to the host.

But don't the two TaskManager containers also have to communicate with each other (as per the documentation for standalone clusters, and as shown in this deep dive)?

Many thanks again, and apologies if this is a straightforward question!

When I tried kafka got this error
ERROR ==> Kafka haven't been configured to work in either Raft or Zookeper mode. Please make sure at least one of the modes is configured.

To solve I had to add KAFKA_CFG_PROCESS_ROLES: controller,broker environment variable

Hi there

I started up the Docker environment and everything came up fine from what I can tell. However when I try to execute the redpanda setup script I get the following message.

unable to create topics [accounts]: unable to dial: dial tcp 127.0.0.1:9092: connect: connection refused

I also tried running the rpk cluster info from the container and received the following.

unable to request metadata: unable to dial: dial tcp 127.0.0.1:9092: connect: connection refused

I'm running on an Intel based MacBook Pro and using colima in place of Docker Desktop, if that makes a difference.

Thank you

Hi,

I'm going through the book and I think the lookup join query is incorrect.

SELECT
    transactionId,
    t.accountId,
    t.eventTime_ltz,
    TO_TIMESTAMP_LTZ(updateTime, 3) AS updateTime,
    type,
    amount,
    balance,
    districtId,
    frequency
FROM transactions AS t
    JOIN accounts FOR SYSTEM_TIME AS OF t.eventTime_ltz AS a
    ON t.accountId = a.accountId;

As it is mentioned in the intro, a processing time attribute is required, however, in the example an event time field is used. On the Flink Dashboard it is also visible that the executed query is doing a temporal join. Switching to a field which is defined as PROCTIME() turns the query into a lookup join.