salsify / avro-schema-registry Goto Github PK

View Code? Open in Web Editor NEW

90.0 90.0 21.0 402 KB

Implementation of the Confluent Schema Registry API as a Rails application

License: MIT License

Ruby 91.04% CSS 0.53% HTML 7.68% Dockerfile 0.71% Procfile 0.03%

avro hacktoberfest schema-registry

avro-schema-registry's People

Contributors

Stargazers

Watchers

avro-schema-registry's Issues

Response body not conforming with confluent documentation

When doing a GET /subjects/something/versions/1
this is returned

{
    "name": "something",
    "version": 1,
    "schema": "{\"type\": \"string\"}"
}

according to the confluent documentation I should get this:

{
    "subject": "something",
    "version": 1,
    "schema": "{\"type\": \"string\"}"
}

that is, the "name" property should be called "subject".

This means that we can't use the confluent client code defined in io.confluent.kafka.schemaregistry.client to interact with the server.

Is this something that you would consider changing?

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape was resolved to 1.1.0, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape', in any of the sources.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape was resolved to 1.1.0, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape', in any of the sources.

Bundler could not find compatible versions for gem "rails":
  In Gemfile:
    rails (~> 5.1.4, <= 5.2.2)

    heroku_rails_deploy (>= 0.4.1) was resolved to 0.4.5, which depends on
      rails

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Subject name cannot start with a number (does not conform with Confluent SR).

When I register a new schema under the subject which name starts with a number, I get 404 response.

Request:

POST https://avro-schema-registry.salsify.com/subjects/1startswithnumber/versions
Content-Type: application/json
Authorization: Basic YXZybzphdnJv
{
  "schema": "{\"type\":\"string\"}"
}

Response:
404 Not Found

With the subject starting with character, it works:

Request:

POST https://avro-schema-registry.salsify.com/subjects/startswithcharacter/versions
Content-Type: application/json
Authorization: Basic YXZybzphdnJv
{
  "schema": "{\"type\":\"string\"}"
}

Response:

{
    "id": 2
}

In comparison, Confluent Schema Registry accepts subjects with names starting with numbers.

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape was resolved to 1.1.0, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape', in any of the sources.

Bundler could not find compatible versions for gem "rails":
  In Gemfile:
    rails (~> 5.1.4, <= 5.2.2)

    heroku_rails_deploy (>= 0.4.1) was resolved to 0.4.5, which depends on
      rails

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Adding or removing a default does not register a new schema version

I've been noticing some confusing behavior with the schema registry. Specifically, we have some schemas which have added default values for some fields. Our expectation was that adding these values would register a new version with the registry. But it appears that the defaults are being stripped out, and the schema is being equated to the version with no defaults and therefore not getting registered.

Is this the expected behavior? If so, how is it possible to maintain FULL-TRANSITIVE compatibility? It was our understanding that defaults were required for that compatibility level.

Steps to reproduce -

Register a schema with all required fields and no defaults (id = 1)
Add defaults to one or more fields
Attempt to register the new schema
Returned id will still be 1

Incompatible with confluent rest-proxy.

Bug

Confluent rest-proxy generates a schema name with a hyphen. (e.g schema-value or schema-key). As far as I know, there is no trivial way to remove this hyphenated suffix before sending the schema name to the registry, so it is sent, and the schema registry complains about the URI.

Using the rest proxy I do:

curl -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" \
      -H "Accept: application/vnd.kafka.v2+json" \
      --data '{"value_schema": "{\"type\": \"record\", \"name\": \"User\", \"fields\": [{\"name\": \"name\", \"type\": \"string\"}]}", "records": [{"value": {"name": "testUser"}}]}' \
      "kafka-rest-proxy:8082/topics/Chips"

I get the following error:

{"error_code":40801,"message":"Schema registration or lookup failed io.confluent.rest.exceptions.RestException: Schema registration or lookup failed
io.confluent.rest.exceptions.RestException: Schema registration or lookup failed
\tat io.confluent.kafkarest.AvroRestProducer.produce(AvroRestProducer.java:104)
\tat io.confluent.kafkarest.ProducerPool.produce(ProducerPool.java:171)
\tat io.confluent.kafkarest.resources.TopicsResource.produce(TopicsResource.java:147)
\tat io.confluent.kafkarest.resources.TopicsResource.produceAvro(TopicsResource.java:135)
\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
\tat java.lang.reflect.Method.invoke(Method.java:498)
\tat org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
\tat org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143)
\tat org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
\tat org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
\tat org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
\tat org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
\tat org.glassfish.jersey.internal.Errors.process(Errors.java:315)
\tat org.glassfish.jersey.internal.Errors.process(Errors.java:297)
\tat org.glassfish.jersey.internal.Errors.process(Errors.java:267)
\tat org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
\tat org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
\tat org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
\tat org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
\tat org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:408)
\tat org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:583)
\tat org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:524)
\tat org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:461)
\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
\tat org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
\tat org.eclipse.jetty.server.Server.handle(Server.java:499)
\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
\tat java.lang.Thread.run(Thread.java:745)
Caused by: java.net.MalformedURLException: For input string: \"$(CP_SCHEMA_REGISTRY_SERVICE_PORT)\"
\tat java.net.URL.<init>(URL.java:627)
\tat java.net.URL.<init>(URL.java:490)
\tat java.net.URL.<init>(URL.java:439)
\tat io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:152)
\tat io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:229)
\tat io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:320)
\tat io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:312)
\tat io.confluent.kafka.schemaregistry.client.rest.RestService.registerSchema(RestService.java:307)
\tat io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.registerAndGetId(CachedSchemaRegistryClient.java:115)
\tat io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.register(CachedSchemaRegistryClient.java:154)
\tat io.confluent.kafka.serializers.AbstractKafkaAvroSerDe.register(AbstractKafkaAvroSerDe.java:136)
\tat io.confluent.kafkarest.AvroRestProducer.produce(AvroRestProducer.java:93)
\t... 48 more
Caused by: java.lang.NumberFormatException: For input string: \"$(CP_SCHEMA_REGISTRY_SERVICE_PORT)\"
\tat java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
\tat java.lang.Integer.parseInt(Integer.java:569)
\tat java.lang.Integer.parseInt(Integer.java:615)
\tat java.net.URLStreamHandler.parseURL(URLStreamHandler.java:216)
\tat java.net.URL.<init>(URL.java:622)
\t... 59 more
"}

In the logs from the avro-schema-registry, I see a request made to a URI like:

/subjects/chips-value/versions

Which when routed to will produce a screen like this:

Versus this route, which gives the expected output:

/subjects/chips_value/versions

It appears this is an error comes from hashie disliking the hyphen in the first URI when trying the mount the Schema API. Note there is no issue if I sub out the avro-schema-registry for the confluent schema registry.

Add support for the delete API

I believe this has been released in the Confluent Schema Registry 3.2.1 and is documented in this PR:

confluentinc/schema-registry#560

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape (>= 0, <= 1.2.2) was resolved to 1.2.2, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape (>= 0, <= 1.2.2)', in any of the sources.

Bundler could not find compatible versions for gem "rack-mount":
  In Gemfile:
    grape (>= 0, <= 1.2.2) was resolved to 0.1.3, which depends on
      rack-mount (~> 0.6.13)

Could not find gem 'rack-mount (~> 0.6.13)', which is required by gem 'grape (>= 0, <= 1.2.2)', in any of the sources.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape (>= 0, <= 1.2.2) was resolved to 1.2.2, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape (>= 0, <= 1.2.2)', in any of the sources.

Bundler could not find compatible versions for gem "rack-mount":
  In Gemfile:
    grape (>= 0, <= 1.2.2) was resolved to 0.1.3, which depends on
      rack-mount (~> 0.6.13)

Could not find gem 'rack-mount (~> 0.6.13)', which is required by gem 'grape (>= 0, <= 1.2.2)', in any of the sources.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Update release for this project

The latest release v.0.12.0 was created in February 2018. We pinned against that release and were exposed to a security vulnerability that has been fixed in master.

It would be awesome if releases could be cut more frequently. We have currently pinned against the HEAD in master.

bundle install fails due to mimemagic dependency constraint

We're seeing

[2021-04-07T18:11:29.754Z] Step 5/15 : RUN gem install bundler --no-document && bundle install --jobs 20 --retry 5
[2021-04-07T18:11:29.754Z]  ---> Running in 48db59532dc3
[2021-04-07T18:11:30.011Z] Successfully installed bundler-2.2.15
[2021-04-07T18:11:30.011Z] 1 gem installed
[2021-04-07T18:11:32.717Z] Fetching gem metadata from https://rubygems.org/.........
[2021-04-07T18:11:32.717Z] Your bundle is locked to mimemagic (0.3.3), but that version could not be found
[2021-04-07T18:11:32.717Z] in any of the sources listed in your Gemfile. If you haven't changed sources,
[2021-04-07T18:11:32.717Z] that means the author of mimemagic (0.3.3) has removed it. You'll need to update
[2021-04-07T18:11:32.717Z] your bundle to a version other than mimemagic (0.3.3) that hasn't been removed
[2021-04-07T18:11:32.717Z] in order to install.

when building the Dockerfile. Seems like you need to bump the mimemagic dependency to 0.3.6. mimemagicrb/mimemagic#98

Remove fingerprint v1

Add support for getting unescaped schema

The confluent schema-registry will return the unescaped schema via this endpoint

GET /subjects/(string: subject)/versions/(versionId: version)/schema

[question] can I use it for XSD or JSON schemas as well?

Upgrade to Rails 5.1

I created a branch for this: https://github.com/salsify/avro-schema-registry/tree/rails-5.1

I wasn't going to create a PR for this yet because I'm not actively using the schema registry yet.

Alpine base image

We're scanning images in our kubernetes cluster for CVE's and this one keep causing alerts.
Would you consider switching the base image to an alpine image?

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape was resolved to 1.1.0, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape', in any of the sources.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

Delete all versions from a schema

Great tool.
I need to create from scrash schemas in unit tests and deleting schemas is a must.
Does your API fully conforms with Confluent's? How can I delete a schema and all its versions?
e.g

Delete version 3 of the schema registered under subject "Kafka-value"

$ curl -X DELETE http://localhost:8081/subjects/Kafka-value/versions/3
  3

Delete all versions of the schema registered under subject "Kafka-value"

$ curl -X DELETE http://localhost:8081/subjects/Kafka-value
  [1, 2, 3, 4, 5]

Dependabot can't resolve your Ruby dependency files

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler could not find compatible versions for gem "rack-accept":
  In Gemfile:
    grape (>= 0, <= 1.2.2) was resolved to 1.2.2, which depends on
      rack-accept

Could not find gem 'rack-accept', which is required by gem 'grape (>= 0, <= 1.2.2)', in any of the sources.

Bundler could not find compatible versions for gem "rack-mount":
  In Gemfile:
    grape (>= 0, <= 1.2.2) was resolved to 0.1.3, which depends on
      rack-mount (~> 0.6.13)

Could not find gem 'rack-mount (~> 0.6.13)', which is required by gem 'grape (>= 0, <= 1.2.2)', in any of the sources.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

salsify / avro-schema-registry Goto Github PK

avro-schema-registry's People

Contributors

Stargazers

Watchers

Forkers

avro-schema-registry's Issues

Bug

Delete version 3 of the schema registered under subject "Kafka-value"

Delete all versions of the schema registered under subject "Kafka-value"

Recommend Projects

Recommend Topics

Recommend Org