Comments (4)
The error indicates that there was an issue while trying to run kinesis_consumer.py
which could be for a variety or reasons. That executable may not be present in your PATH
or the shebang line may be set wrong for your system/environment. I suggest trying to set the executableName
and the shebang within it to absolute paths. There are several suggestions in this issue that may help.
From that thread, @venkat-vs-id said:
I got the same problem and this is what I found
http://venkat-echo.blogspot.com.au/2015/01/kinesis-trouble-shooting.html
- you can have absolute path or relative it doesnt matter.
- it is not necessary to set the PATH if you are using absolute path
- if you had use the (consumer)sample.py as a template, then fix the shebang (ie.. origina sample.py had #!env python... It is for this the KCL lib threw the error file or directory not found. change this to the actual python path like (#!/usr/bin/env python). Then the file or folder not found error is gone.
from amazon-kinesis-client-python.
My properties file looks like this
# The script that abides by the multi-language protocol. This script will
# be executed by the MultiLangDaemon, which will communicate with this script
# over STDIN and STDOUT according to the multi-language protocol.
executableName = kinesis_consumer.py
# The name of an Amazon Kinesis stream to process.
streamName = twitter-stream
# Used by the KCL as the name of this application. Will be used as the name
# of an Amazon DynamoDB table which will store the lease and checkpoint
# information for workers with this application name
applicationName = TwitterRTAnalyzer
# Users can change the credentials provider the KCL will use to retrieve credentials.
# The DefaultAWSCredentialsProviderChain checks several other providers, which is
# described here:
#
AWSCredentialsProvider = DefaultAWSCredentialsProviderChain
# Appended to the user agent of the KCL. Does not impact the functionality of the
# KCL in any other way.
processingLanguage = python/2.7
# Valid options at TRIM_HORIZON or LATEST.
# See http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html#API_GetShardIterator_RequestSyntax
initialPositionInStream = TRIM_HORIZON
# The following properties are also available for configuring the KCL Worker that is created
# by the MultiLangDaemon.
# The KCL defaults to us-east-1
#regionName = us-east-1
# Fail over time in milliseconds. A worker which does not renew it's lease within this time interval
# will be regarded as having problems and it's shards will be assigned to other workers.
# For applications that have a large number of shards, this msy be set to a higher number to reduce
# the number of DynamoDB IOPS required for tracking leases
#failoverTimeMillis = 10000
# A worker id that uniquely identifies this worker among all workers using the same applicationName
# If this isn't provided a MultiLangDaemon instance will assign a unique workerId to itself.
#workerId =
# Shard sync interval in milliseconds - e.g. wait for this long between shard sync tasks.
#shardSyncIntervalMillis = 60000
# Max records to fetch from Kinesis in a single GetRecords call.
#maxRecords = 10000
# Idle time between record reads in milliseconds.
#idleTimeBetweenReadsInMillis = 1000
# Enables applications flush/checkpoint (if they have some data "in progress", but don't get new data for while)
#callProcessRecordsEvenForEmptyRecordList = false
# Interval in milliseconds between polling to check for parent shard completion.
# Polling frequently will take up more DynamoDB IOPS (when there are leases for shards waiting on
# completion of parent shards).
#parentShardPollIntervalMillis = 10000
# Cleanup leases upon shards completion (don't wait until they expire in Kinesis).
# Keeping leases takes some tracking/resources (e.g. they need to be renewed, assigned), so by default we try
# to delete the ones we don't need any longer.
#cleanupLeasesUponShardCompletion = true
# Backoff time in milliseconds for Amazon Kinesis Client Library tasks (in the event of failures).
#taskBackoffTimeMillis = 500
# Buffer metrics for at most this long before publishing to CloudWatch.
#metricsBufferTimeMillis = 10000
# Buffer at most this many metrics before publishing to CloudWatch.
#metricsMaxQueueSize = 10000
# KCL will validate client provided sequence numbers with a call to Amazon Kinesis before checkpointing for calls
# to RecordProcessorCheckpointer#checkpoint(String) by default.
#validateSequenceNumberBeforeCheckpointing = true
# The maximum number of active threads for the MultiLangDaemon to permit.
# If a value is provided then a FixedThreadPool is used with the maximum
# active threads set to the provided value. If a non-positive integer or no
# value is provided a CachedThreadPool is used.
#maxActiveThreads = 0
and I am getting an error message
Mar 25, 2015 8:14:41 AM com.amazonaws.services.kinesis.multilang.MultiLangRecordProcessor stopProcessing
SEVERE: Failed to start client executable
java.io.IOException: Cannot run program "kinesis_consumer.py": error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at com.amazonaws.services.kinesis.multilang.MultiLangRecordProcessor.startProcess(MultiLangRecordProcessor.java:274)
at com.amazonaws.services.kinesis.multilang.MultiLangRecordProcessor.initialize(MultiLangRecordProcessor.java:112)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitializeTask.call(InitializeTask.java:74)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:48)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:23)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 9 more
Command issued was
`amazon_kclpy_helper.py --print_command --java /usr/bin/java --properties sample.properties`
Any ideas. The python file is chmod +x and it follows exactly as the sample application suggested.
from amazon-kinesis-client-python.
Thanks Kurtis, that indeed worked. I fixed the shebang and tried with absolute paths
`amazon_kclpy_helper.py --print_command --java /usr/bin/java --properties /home/sachin/sample.properties`
from amazon-kinesis-client-python.
It must be due to java cannot find it from path.
besides shebang, I make use of helper's "-c" feature to add my app.py explicitly, and I simply keep the rest relative as the sample.
from amazon-kinesis-client-python.
Related Issues (20)
- Advanced configuration for KCL MultiLangDaemon (DynamoDB/Kinesis endpoints etc) HOT 2
- Aggregating records from MultiLangDaemon workers (2.x) (Python)
- KCL client consistently throws exception when in use with localstack HOT 1
- Warnings on an Internal Service Error HOT 2
- How to get video data from signaling channel HOT 1
- Is cross account/cross region possible
- How to get metadata from Amazon Kinesis Video Streams?
- Need new release > 2.0.6 to solve vulnerabilities in dependencies HOT 6
- Kinesis Python going to sleep HOT 1
- Using Python KCL library for mutli-stream processing HOT 1
- Support STSAssumeRoleWithWebIdentitySessionCredentialsProvider? HOT 1
- MultiLang Daemon for KCL 2.0 for python - how to increase the SDK timeouts? HOT 1
- AWSCredentialsProvider issue when using named profile
- The security token included in the request is invalid HOT 1
- No such file or directory: 'pom.xml' Error when installing 2.1.0 HOT 2
- Is calling checkpoint() with no args on shutdown correct behaviour?
- Make Maven repository configurable in download_jars HOT 1
- CreateProcess error=193, %1 is not a valid Win32 application
- CheckpointInput has no attribute action HOT 1
- Is it possible to run applications utilizing amazon_kclpy in debug mode?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-kinesis-client-python.