Comments (10)
There are a few dependencies besides Kafka that may need snapshot builds: https://github.com/confluentinc/kafka-connect-hdfs/wiki/FAQ#when-i-try-to-compile-i-get-an-error-about-a-missing-snapshot-dependency
from kafka-connect-hdfs.
Sorry for getting late, got busy. However the steps you mentioned still do not work for me @ewencp
I am on Apache Maven 3.3.9 .The file is present in my local repo, installed it directly, but jar has some issue..
On building kafka connect hdfs, it threw the same error,
[WARNING] The POM for io.confluent:kafka-connect-avro-converter:jar:3.1.0 is missing, no dependency information available
Downloading: http://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/3.1.0/kafka-connect-avro-converter-3.1.0.jar
Downloading: https://repo.maven.apache.org/maven2/io/confluent/kafka-connect-avro-converter/3.1.0/kafka-connect-avro-converter-3.1.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.084 s
so I installed it from my local repo
mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-connect-avro-converter -Dversion=3.1.0-SNAPSHOT -Dpackaging=jar -Dfile=/.m2/repository/io/confluent/kafka-connect-avro-converter/3.1.0-SNAPSHOT/kafka-connect-avro-converter-3.1.0-SNAPSHOT.jar.lastUpdated
but the build throws error in opening zip file.
../.m2/repository/io/confluent/kafka-connect-avro-converter/3.1.0-SNAPSHOT/kafka-connect-avro-converter-3.1.0-SNAPSHOT.jar; error in opening zip file
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
Please let me know if I am missing something.thanks.
from kafka-connect-hdfs.
@rekhajoshm You should not need to manually execute any install-file
commands. It also looks like you are installing the wrong file -- a lastUpdated
file is not the jar itself. It is an internal file used by Maven to track when it last tried to download a file.
Did the build and install of the confluentinc/schema-registry repository complete without errors? That is the missing dependency but it is included in the instructions I linked to.
from kafka-connect-hdfs.
Thanks for the quick reply @ewencp .Yes the mvn install on schema-registry had finished successfully.And I have done these steps multiple times now.
[INFO] kafka-serde-tools-package .......................... SUCCESS [ 0.797 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
I undid the install of .lastUpdated jar, but the avro connector issue still persists for connect-hdfs.
Downloading: http://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/3.1.0-SNAPSHOT/maven-metadata.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.516 s
[INFO] Finished at: 2016-06-13T11:24:57-07:00
[INFO] Final Memory: 17M/309M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project kafka-connect-hdfs: Could not resolve dependencies for project io.confluent:kafka-connect-hdfs:jar:3.1.0-SNAPSHOT: Could not find artifact io.confluent:kafka-connect-avro-converter:jar:3.1.0-SNAPSHOT
from kafka-connect-hdfs.
@rekhajoshm Can you paste a few more lines from above the info you pasted? That'll show whether the version info is as expected and the path that at least a few of the jars got installed to.
Are you on the master branch and pulled from all the repositories to make sure you're on matching versions? I just ran this all from scratch, having removed my .m2/repository
entirely and moving my settings.xml
(since that picks up our internal maven) and it seems to make it through everything fine. There was one dependency issue on our external Maven repository on rest-utils, but I'm fixing that up in confluentinc/rest-utils#41 and that doesn't look related to your issue.
from kafka-connect-hdfs.
Thanks @ewencp for following up.
I am on master branch on schema-registry and it pulls kafka-connect-avro-converter to local maven but version is 2.0.0-SNAPSHOT.
When I try building up kafka-connect using 2.0.0-SNAPSHOT version for avro, connect-hdfs works.But I suspect this might mean loss of functionality(which I will discover later) and also if instead of 2.0.0-SNAPSHOT, why 3.0.0-SNAPSHOT is not installed correctly.
Please let me know if I am missing something.Thanks!
updated connect-hdfs snippet:
io.confluent
kafka-connect-avro-converter
2.0.0-SNAPSHOT
schema-registry install log:
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ kafka-connect-avro-converter ---
[INFO] Building jar: LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:2.6:test-jar (default) @ kafka-connect-avro-converter ---
[INFO] Building jar: LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ kafka-connect-avro-converter ---
[INFO] Installing LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT.jar to LOCAL_PATH/.m2/repository/io/confluent/kafka-connect-avro-converter/2.0.0-SNAPSHOT/kafka-connect-avro-converter-2.0.0-SNAPSHOT.jar
[INFO] Installing LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/pom.xml to LOCAL_PATH/.m2/repository/io/confluent/kafka-connect-avro-converter/2.0.0-SNAPSHOT/kafka-connect-avro-converter-2.0.0-SNAPSHOT.pom
[INFO] Installing LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT-tests.jar to LOCAL_PATH/.m2/repository/io/confluent/kafka-connect-avro-converter/2.0.0-SNAPSHOT/kafka-connect-avro-converter-2.0.0-SNAPSHOT-tests.jar
[INFO] BUILD SUCCESS
from kafka-connect-hdfs.
@rekhajoshm Can you run git pull
? The master branch of schema-registry definitely isn't 2.0.0 anymore and hasn't been for awhile -- see https://github.com/confluentinc/schema-registry/blob/master/pom.xml#L10
from kafka-connect-hdfs.
Thanks @ewencp .my bad. thought i had the latest in these checkouts.Anyhow did a refresh of all repos @ewencp - Kafka, Avro converter, common, rest-utils now and get an issue with kafka-clients on rest-utils build.
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.apache.kafka:kafka_2.11:jar:test:0.10.0.0-cp1 is missing, no dependency information available
[WARNING] The POM for org.apache.kafka:kafka-clients:jar:test:0.10.0.0-cp1 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] rest-utils-parent .................................. SUCCESS [ 0.430 s]
[INFO] rest-utils ......................................... FAILURE [ 0.187 s]
[INFO] rest-utils-test .................................... SKIPPED
[INFO] rest-utils-example ................................. SKIPPED
[INFO] rest-utils-package ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.729 s
[INFO] Finished at: 2016-06-13T13:07:44-07:00
[INFO] Final Memory: 12M/309M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project rest-utils: Could not resolve dependencies for project io.confluent:rest-utils:jar:3.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.kafka:kafka_2.11:jar:test:0.10.0.0-cp1, org.apache.kafka:kafka-clients:jar:test:0.10.0.0-cp1: Failure to find org.apache.kafka:kafka_2.11:jar:test:0.10.0.0-cp1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
from kafka-connect-hdfs.
That's the issue being addressed in confluentinc/rest-utils#41
from kafka-connect-hdfs.
Have locally taken your change, and woohoo, hdfs connect is working now!
Thanks for your patience @ewencp in working through this issue.
from kafka-connect-hdfs.
Related Issues (20)
- using wrong user/keytab while there are multiple hdfs-sink connections HOT 1
- template file isn't committed and uploaded to storage when using AvroFormat
- java.util.ConcurrentModificationException during task rebalancing HOT 1
- log4j update schedule HOT 1
- Hive table does not match column names present in the parquet data
- Exception when reading Decimal types written by connector
- Hive Merge Feature
- Incremental Co-operative Rebalancing Support for HDFS Connector
- Error after install and unistall connect-transforms
- Adding Hive partition threw unexpected error
- HDFS2 connect compatibility with HDFS3 server
- CVE-2021-34538 HIGH vulnerability HOT 2
- Task is being killed and will not recover until manually restarted
- Allow to limit retry write errors by timeout
- Kafka Issue while running on docker and adding new connector HOT 1
- can't build because repo conjars is down
- multiple keytab kerberos issue HOT 1
- OzoneFileSystem
- Non-resolvable parent POM io.confluent:common:[7.7.0, 7.7.1) for io.confluent:kafka-connect-storage-common-parent:11.2.9
- [2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:237) java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.open(java.util.Collection)" because "this.hdfsWriter" is null
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-hdfs.