These are simple exercises where there are pipelines and functions defined, that explore the usage of Apache Beam
mvn package
java -jar target/pipelines-samples-0.1-shaded.jar
mvn package -Pflink-runner
cd flink/flink-1.11.0
./bin/flink run /Users/user}/{somePath}/pipeline-samples/target/pipelines-samples-0.1-shaded.jar --runner=FlinkRunner
Package the jar file as a fat.jar - dependencies included - using the shade plugin
mvn package -Pflink-runner
This will create a jar file in the /target/pipelines-samples-0.1-shaded.jar directory
From a windows Power shell command, start flink with docker compose, with the following commands:
set COMPOSE_CONVERT_WINDOWS_PATH=1
docker-compose up -d
Then bring up the Flink UI, I configured it for port 8888 localhost:8888
upload the pipelines-samples-0.1-shaded.jar file, add the program argument :
--runner=FlinkRunner
You should be able to run your job, and see the results:
export GOOGLE_APPLICATION_CREDENTIALS="/Users/{user}/{somePath}/XXX_credentials.json"
gcloud auth application-default login
mvn package -Pdataflow-runner
java -jar target/pipelines-samples-0.1-shaded.jar --runner=DataflowRunner --project=deloitte-beam-284202 --tempLocation=gs://deloitte-beam-sandbox/temp/ --region=us-west1
mvn package -Pflink-runner
scp -i ~/.ssh/keypair.pem ./target/pipeline-samples-0.1-shaded.jar ec2-user@ec2-xxx-xxx-xxx:/home/hadoop
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent