biznetgio / restknot Goto Github PK

REST API for Knot Service

License: MIT License

Python 96.86% Dockerfile 1.08% Shell 0.80% Just 1.26%

dns knot-dns knotd rest-api agent-server knot-server dns-management

restknot's Introduction

Manage DNS records with asynchronous and simple APIs.

RESTKnot provide a high-level asynchronous API to existing Knot DNS server. This project consists of three applications: RESTKnot agent, RESTKnot API, and RESTKnot CLI. A user can create DNS record through web API provided by RESTKnot API, or as command line app using RESTKnot CLI. Both of them send command to RESTKnot agent which will be translated into Knot DNS action.

Features

Asynchronous operation
Created default DNS records when adding new zone.
Untangle all related record when deleting zone with single API.
Prevent wrong RDATA format with validation support.
Prevent record lost by checking RDATA contents before adding any record.

Take the tour

Create New Zone

curl -X POST \
  http://localhost:5000/api/domain/add \
  -H 'X-API-key: 123' \
  -F user_id=001 \
  -F zone=example.com

Edit a Single Record

curl -X PUT \
  http://127.0.0.1:5000/api/record/edit/10 \
  -H 'x-api-key: 123' \
  -F zone=bar.com \
  -F owner=@ \
  -F rtype=NS \
  -F rdata=one.exampledns.com. \
  -F ttl=3600

Delete a Zone

curl -X DELETE \
  http://localhost:5000/api/domain/delete \
  -H 'X-API-Key: 123' \
  -F zone=example.com

Project information

restknot's People

Contributors

Stargazers

Watchers

Forkers

riszkymf isnuryusuf sayyidisal w796933 eumss apititan simhaonline ulibaka jonxie5

restknot's Issues

[Feature Request] Multi broker support

RESTKnot-api rely on dpkp/kafka-python to communicate with kafka.
Currently, we are experimenting with this.

Supporting multiple brokers only needs small changes to the codebase. Since kafka-python already accepts a list of bootstrapping servers.

Our experiment is to create a topic with replication factor as 3.
Spin up 3 Kafka broker instances with 1 zookeeper. Connecting kafka-python to one of the brokers. Start sending the message while shutting down each broker until it left only one. While running the experiment we expected the producer to be reliably able to send messages even if some nodes go down.

We start by spinning 3 brokers, then tell RESTKnot-api to send a message. It works.

Now we shut down, one of the brokers. And RESTKnot-api still able to works well.

The last step is to shut down another broker, and it still works reliably well.

Turns out, our experiment runs successfully. But one thing to note that: kafka-python fails to connect if one of the brokers in the bootstrap server list is down. I will check if this also happens with the Kafka CLI client, it if is, then the problem relies on the Kafka-python.

We are also planning to experiment with multiple zookeepers.

Unable to add MX records

I am unable to add MX records in Knot. Every time when I try to add new MX records it says 'error: (parser failed)'. Please help me to resolve this.

Using the current restknot-agent. Zone is not created in Knot 3.1.2

Error saat memasukkan value ¼ ke endpoint /content (record type: MX)

<title>KeyError: 'default' // Werkzeug Debugger</title> <script src="?__debugger__=yes&cmd=resource&f=jquery.js"></script> <script src="?__debugger__=yes&cmd=resource&f=debugger.js"></script> <script type="text/javascript"> var TRACEBACK = 140470742903664, CONSOLE_MODE = false, EVALEX = true, EVALEX_TRUSTED = false, SECRET = "atWuLJfXD5h7cxQk5VE7"; </script>

builtins.KeyError

KeyError: 'default'

Traceback (most recent call last)

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1997, in `call`

 error = None

 ctx.auto_pop(error)

 def __call__(self, environ, start_response):

 """Shortcut for :attr:`wsgi_app`."""

 return self.wsgi_app(environ, start_response)

 def __repr__(self):

 return '<%s %r>' % (

 self.__class__.__name__,

 self.name,

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1985, in `wsgi_app`

 try:

 try:

 response = self.full_dispatch_request()

 except Exception as e:

 error = e

 response = self.handle_exception(e)

 except:

 error = sys.exc_info()[1]

 raise

 return response(environ, start_response)

 finally:

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/init.py", line 273, in `error_router`

 if self._has_fr_route():

 try:

 return self.handle_error(e)

 except Exception:

 pass # Fall through to original handler

 return original_handler(e)

 def handle_error(self, e):

 """Error handler for the API transforms a raised exception into a Flask

 response, with the appropriate HTTP status code and body.

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in `wrapped_function`

 # Wrap exception handlers with cross_origin

 # These error handlers will still respect the behavior of the route

 if options.get('intercept_exceptions', True):

 def _after_request_decorator(f):

 def wrapped_function(*args, **kwargs):

 return cors_after_request(app.make_response(f(*args, **kwargs)))

 return wrapped_function

 if hasattr(app, 'handle_exception'):

 app.handle_exception = _after_request_decorator(

 app.handle_exception)

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1540, in `handle_exception`

 # if we want to repropagate the exception, we can attempt to

 # raise it with the whole traceback in case we can do that

 # (the function was actually called from the except part)

 # otherwise, we just raise the error again

 if exc_value is e:

 reraise(exc_type, exc_value, tb)

 else:

 raise e

 self.log_exception((exc_type, exc_value, tb))

 if handler is None:

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/_compat.py", line 32, in `reraise`

 from io import StringIO

 def reraise(tp, value, tb=None):

 if value.__traceback__ is not tb:

 raise value.with_traceback(tb)

 raise value

 implements_to_string = _identity

else:

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1982, in `wsgi_app`

 ctx = self.request_context(environ)

 ctx.push()

 error = None

 try:

 try:

 response = self.full_dispatch_request()

 except Exception as e:

 error = e

 response = self.handle_exception(e)

 except:

 error = sys.exc_info()[1]

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1614, in `full_dispatch_request`

 request_started.send(self)

 rv = self.preprocess_request()

 if rv is None:

 rv = self.dispatch_request()

 except Exception as e:

 rv = self.handle_user_exception(e)

 return self.finalize_request(rv)

 def finalize_request(self, rv, from_error_handler=False):

 """Given the return value from a view function this finalizes

 the request by converting it into a response and invoking the

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/init.py", line 273, in `error_router`

 if self._has_fr_route():

 try:

 return self.handle_error(e)

 except Exception:

 pass # Fall through to original handler

 return original_handler(e)

 def handle_error(self, e):

 """Error handler for the API transforms a raised exception into a Flask

 response, with the appropriate HTTP status code and body.

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in `wrapped_function`

 # Wrap exception handlers with cross_origin

 # These error handlers will still respect the behavior of the route

 if options.get('intercept_exceptions', True):

 def _after_request_decorator(f):

 def wrapped_function(*args, **kwargs):

 return cors_after_request(app.make_response(f(*args, **kwargs)))

 return wrapped_function

 if hasattr(app, 'handle_exception'):

 app.handle_exception = _after_request_decorator(

 app.handle_exception)

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1517, in `handle_user_exception`

 return self.handle_http_exception(e)

 handler = self._find_error_handler(e)

 if handler is None:

 reraise(exc_type, exc_value, tb)

 return handler(e)

 def handle_exception(self, e):

 """Default exception handling that kicks in when an exception

 occurs that is not caught. In debug mode the exception will

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/_compat.py", line 32, in `reraise`

 from io import StringIO

 def reraise(tp, value, tb=None):

 if value.__traceback__ is not tb:

 raise value.with_traceback(tb)

 raise value

 implements_to_string = _identity

else:

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1612, in `full_dispatch_request`

 self.try_trigger_before_first_request_functions()

 try:

 request_started.send(self)

 rv = self.preprocess_request()

 if rv is None:

 rv = self.dispatch_request()

 except Exception as e:

 rv = self.handle_user_exception(e)

 return self.finalize_request(rv)

 def finalize_request(self, rv, from_error_handler=False):

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1598, in `dispatch_request`

 # request came with the OPTIONS method, reply automatically

 if getattr(rule, 'provide_automatic_options', False) \

 and req.method == 'OPTIONS':

 return self.make_default_options_response()

 # otherwise dispatch to the handler for that endpoint

 return self.view_functions[rule.endpoint](**req.view_args)

 def full_dispatch_request(self):

 """Dispatches the request and on top of that performs request

 pre and postprocessing as well as HTTP exception catching and

 error handling.

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/init.py", line 480, in `wrapper`

 :param resource: The resource as a flask view function

"""

 @wraps(resource)

 def wrapper(*args, **kwargs):

 resp = resource(*args, **kwargs)

 if isinstance(resp, ResponseBase): # There may be a better way to test

 return resp

 data, code, headers = unpack(resp)

 return self.make_response(data, code, headers=headers)

 return wrapper

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/views.py", line 84, in `view`

 The arguments passed to :meth:`as_view` are forwarded to the

 constructor of the class.

"""

 def view(*args, **kwargs):

 self = view.view_class(*class_args, **class_kwargs)

 return self.dispatch_request(*args, **kwargs)

 if cls.decorators:

 view.__name__ = name

 view.__module__ = cls.__module__

 for decorator in cls.decorators:

File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/init.py", line 595, in `dispatch_request`

 decorators = self.method_decorators

 for decorator in decorators:

 meth = decorator(meth)

 resp = meth(*args, **kwargs)

 if isinstance(resp, ResponseBase): # There may be a better way to test

 return resp

 representations = self.representations or OrderedDict()

File "/home/centos/RESTKnot/api/app/middlewares/auth.py", line 40, in `decorated_function`

 access_token = request.headers['Access-Token']

 stored_data = redis_store.get('{}'.format(access_token))

 if not stored_data:

 return response(400, message=" Invalid access token ")

 return f(*args, **kwargs)

 return decorated_function

File "/home/centos/RESTKnot/api/app/controllers/api/content.py", line 37, in `post`

 @login_required

 def post(self):

 json_req = request.get_json(force=True)

 command = utils.get_command(request.path)

 command = "zn_"+command

 init_data = cmd.parser(json_req, command)

 respons = dict()

 if init_data['action'] == 'insert':

 table = init_data['data'][0]['table']

 fields = init_data['data'][0]['fields']

 ct_rep = fields['nm_content']

File "/home/centos/RESTKnot/api/app/helpers/cmd_parser.py", line 29, in `parser`

 except Exception:

 variabel_check = None

 if variabel_check:

 data_variabel[variabel_key] = json[i][parameters_key][variabel_key]

 else:

 data_variabel[variabel_key] = endpoint[i][parameters_key][variabel_key]['default']

 tagfields[parameters_key]=data_variabel

 parameter=dict()

 for key in tagfields:

 try:

 fields_check = tagfields["fields"]

KeyError: 'default'

This is the Copy/Paste friendly version of the traceback. You can also paste this traceback into a gist:

<textarea cols="50" rows="10" name="code" readonly>Traceback (most recent call last): File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__ return self.wsgi_app(environ, start_response) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app response = self.handle_exception(e) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/__init__.py", line 273, in error_router return original_handler(e) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function return cors_after_request(app.make_response(f(*args, **kwargs))) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception reraise(exc_type, exc_value, tb) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise raise value.with_traceback(tb) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/__init__.py", line 273, in error_router return original_handler(e) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function return cors_after_request(app.make_response(f(*args, **kwargs))) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise raise value.with_traceback(tb) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper resp = resource(*args, **kwargs) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask/views.py", line 84, in view return self.dispatch_request(*args, **kwargs) File "/home/centos/RESTKnot/API/env/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request resp = meth(*args, **kwargs) File "/home/centos/RESTKnot/api/app/middlewares/auth.py", line 40, in decorated_function return f(*args, **kwargs) File "/home/centos/RESTKnot/api/app/controllers/api/content.py", line 37, in post init_data = cmd.parser(json_req, command) File "/home/centos/RESTKnot/api/app/helpers/cmd_parser.py", line 29, in parser data_variabel[variabel_key] = endpoint[i][parameters_key][variabel_key]['default'] KeyError: 'default'</textarea>

The debugger caught an exception in your WSGI application. You can now look at the traceback which led to the error. If you enable JavaScript you can also use additional features such as code execution (if the evalex feature is enabled), automatic pasting of the exceptions and much more.

Brought to you by DON'T PANIC, your friendly Werkzeug powered traceback interpreter.

Console Locked

The console is locked and needs to be unlocked by entering the PIN. You can find the PIN printed out on the standard output of your shell that runs the server.

PIN:

Invalid Hostname does not return error

hostname: -A0c or A0c- or A.-0c or A-.0c

above entry shall return an error because it is invalid hostname

Edit record delete previous records

Edit Record:

www. IN A 1.2.3.4
www. IN A 4.3.2.1

if one of them edited, it will delete other records.

Send command tidak memberikan error jika datanya salah

type: A
host: salahdata
address: apa ini

seharusnya ini error karena bukan ipv4 -- di knot tidak terbentuk jika dibaca dengan api zone_read -- tapi saat menembak api, returnnya sukses:

/sendcommand

stdClass Object
(
[count] => 3
[data] => Array
(
[0] => stdClass Object
(
[data] => stdClass Object
(
)

[description] => Array
(
[0] => stdClass Object
(
[zone-begin] => Array
(
[0] => stdClass Object
(
[sendblock] => stdClass Object
(
[cmd] => zone-begin
[zone] => kalambukmun.xyz
)

)

[1] => stdClass Object
(
[receive] => stdClass Object
(
[type] => block
)

)

[result] => 1
[status] => Command Execute
)

[1] => stdClass Object
(
[data] => stdClass Object
(
)

[description] => Array
(
[0] => stdClass Object
(
[zone-set] => Array
(
[0] => stdClass Object
(
[sendblock] => stdClass Object
(
[data] => "apa ini"
[ttl] => 86400
[rtype] => A
[cmd] => zone-set
[zone] => kalambukmun.xyz
[owner] => salahdata
)

)

[1] => stdClass Object
(
[receive] => stdClass Object
(
[type] => block
)

)

[result] => 1
[status] => Command Execute
)

[2] => stdClass Object
(
[data] => stdClass Object
(
)

[description] => Array
(
[0] => stdClass Object
(
[zone-commit] => Array
(
[0] => stdClass Object
(
[sendblock] => stdClass Object
(
[cmd] => zone-commit
[zone] => kalambukmun.xyz
)

)

[1] => stdClass Object
(
[receive] => stdClass Object
(
[type] => block
)

)

[result] => 1
[status] => Command Execute
)

)

[code] => 200
[status] => success
[message] => Operation succeeded
)

Migrate to GitHub Actions

Github actions provide many features that TravisCI is lacking.
One most important thing is: we want to build our docker image
manually so we can supply any argument we wish.

One example would be supplying --build-args argument on docker image build. So we can pass its build version. This leads us to be able to implement showing build version in a service endpoint. It makes us sure that we deployed the correct version.

Wrong value returned success

When add record, any value inputted always returned success. Despite the value is wrong

Provision several records

I am interested in running Knot DNS as a slave. In my setup, there is a daemon that figures out which zones have been added or deleted, and which ones are out of date. Thus, I need a way to tell Knot DNS to

add a new zone, and either a) immediately start a zone transfer (AXFR), or b) hand over all the records to the API, and get them provisioned all at once (not RRset by RRset),
delete a zone (trivial),
trigger a zone transfer (AXFR) for modified zones.

Does RESTKnox support the first and the third case? I could not find anything about "bulk record provisioning" in the documentation, nor did I find anything about triggering zone transfers. Is this out of scope?

Character on value of TXT record maybe not escaped correctly

Cannot add TXT record for following value:
"
'
"agus
'agus
@`!Aa"Bb#Cc$Dd%Ee&Ff'Gg(Hh)Ii*Jj+Kk,Ll-Mm.Nn/Oo0Pp1Qq2Rr3Ss4Tt5Uu6Vv7Ww8Xx9Yy:Zz;[{<|=]}>^~?

Knot DNS socket issue

We always experience this socket issue.
There will always a time when the libknot refuses to works.
The occurrence is once/twice every month.
We can not reproduce the issue, but it always happens.
Usually when there are many requests to the libknot. Sometimes it also happens even the request is low.

When it happened. Most of the time knotd refuses to start.
We need to export the zones, remove some of the data in var/lib/knot, import, then start the knotd again.
It is our "hack" for a couple of 16 months.
It happens so often, that we write the "known problem" for this case.

We have tried to upgrade to v3.0.4 but the problem persists.

KnotCtlError: connection reset (data: None)
ValueError: Can't connect to knot socket: operation not permitted (data: None)
ValueError: Can't connect to knot socket: OS lacked necessary resources (data: None)

The solution for the 3 error message above is:

restart the python app. if it's failed ->
then the knotd must be stopped. restart knotd ->
if it's refused. then export all zones -> remove all data in /var/lib/knot/confdb, timers and journal. Then start knotd.

Fortunately, the third step always works.
But I hope there will be a better solution (stopping it from happening).

Thanks.

knotc says "(not exists)" for remote id even if it's in the config

Knot DNS socket issue experiment on Knot 3.1.1

We have a lot of socket errors on Knot 2.9.4. The most common one is:

OS lacked necessary resources (data: None)

The errors always happen in one of 12 servers, as we explained priviously. So we believe that simply upgrading the Knot version doesn't solve the issue.
This error occurred frequently that we have our documentation and recovery scripts. It happens once every two weeks.

Now we are trying ~~desperately~~ the latest version 3.1.1 hoping that this issue disappeared. Unfortunately, we don't find any mentions of the related issue in the changelog, and the developer stated that the issue is still there. Nevertheless, we try it anyway.

Preparations

We upgrade all the knot serves to 3.1.1, Python libknot to 3.1.2.1, and refactor the agent to match the latest changes in the libknot.

First experiment

The error happens directly after the agent starts. We reported the issue through the Knot Gitter channel and Daniel Salzman (Current Knot DNS maintainer) suggested that we pass B (blocking) to the knotc send_block command.

We agreed:

 def send_block(
             ttl=ttl,
             rtype=rtype,
             data=data,
-            flags=flags,
+            flags="B",
             filter=filter_,
         )
         resp_ = ctl.receive_block()

The solution doesn't work at first. But after we restart (docker-compose down && docker-compose up -d) the agent, the issue disappeared.

We try to create 3 new domains to reproduce the issue, but no error happened. Then try to simulate where multiple users create many new domains simultaneously, as we suppose that this event will likely trigger the error in the socket. We shut down the agent, create 8 new domains, then start the agent. This time, The Kafka broker will push those 8 new domain messages so fast that the socket will fail. Turns out, it is not. The socket is fine.

Now, we remove our previous changes (the B) and do the same previous experiment ( 3 domains step-by-step, 8 domains simultaneously). If the error occurred, we can make sure that B was the solution. Sadly, both experiment doesn't fail the socket too.

We try to produce the error directly using knotc, as this is reproducible in 2.9.2.

[centos@knot-slave-2 knot]$ sudo knotc -t 30 reload
OS lacked necessary resources (data: None)

Unfortunately, this doesn't happen in 3.1.1. Otherwise, we will pass the B after the error and see the result.

At this point, we are not 100% sure that B was the solution.
Now, we deploy the agent some with B and some are not. We will try to abuse the socket more, and see the result.

At the time when the DNS standards were written, some rules were set out to govern their use. RFC 1912 and RFC 2181 set out that:

SOA and NS records are mandatory to be present at the root domain

CNAME records can only exist as single records and can not be combined with any other resource record ( DNSSEC SIG, NXT, and KEY RR records excepted)

This excludes a CNAME being used on the root domain, as the two rules would contradict each other.

According to above specification, we shall disallow CNAME record which owner is root domain. This applies on add record and edit record case.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.