Giter Club home page Giter Club logo

whoisd's Introduction

Openprovider Whois Daemon

A quick and easy way to setup your own WHOIS server with Elasticsearch/MySQL storage

Whois Daemon complies with the requirements of ICANN

Build Status Contributions Welcome

Whois Daemon represents a light server which provide fast way to produce whois information.

Testing

make test

Check of demo working state

make check

Running

Deploy and run into Kubernetes:

make deploy

Locally in docker:

make run

As daemon:

sudo whoisd install
sudo whoisd start

This will bring up whoisd listening on port 43 for client communication.

Examples

Examples of using Elasticsearch and MySQL backends in examples/README.md

Usage

whoisd - Whois Daemon

Usage:
  whoisd install | remove | start | stop | status
  whoisd [ -t | --test ] [ -option | -option ... ]
  whoisd -h | --help
  whoisd -v | --version

Commands:
  install           Install as service (is only valid for Linux and Mac Os X)
  remove            Remove service
  start             Start service
  stop              Stop service
  status            Check service status

  -h --help         Show this screen
  -v --version      Show version
  -t --test         Test mode

Options:
  -config=<path>    Path to config file (used in /etc/whoisd/whoisd.conf)
  -mapping=<path>   Path to mapping file (used in /etc/whoisd/conf.d/mapping.json)
  -host=<host/IP>   Host name or IP address
  -port=<port>      Port number
  -work=<number>    Number of active workers (default 1000)
  -conn=<number>    Number of active connections (default 1000)
  -storage=<type>   Type of storage (Elasticsearch, Mysql or Dummy for testing)
  -shost=<host/IP>  Storage host name or IP address
  -sport=<port>     Storage port number
  -suser=<name>     User name for storage login
  -spassword=<***>  Password of storage user
  -base=<name>      Storage index or database name
  -table=<name>     Storage type or table name

Config

The config file should be in /etc/whoisd/whoisd.conf. Of course possible to load config settings from any other place through -config option. If config file is absent, used predefined configuration below:

{
  "host": "0.0.0.0",
  "port": 43,
  "workers": 1000,
  "connections": 1000,
  "storage": {
    "storageType": "Dummy",
    "host": "localhost",
    "port": 9200,
    "user": "test",
    "password": "test",
    "indexBase": "whois",
    "typeTable": "domain"
  }
}

NOTE: Valid storage types: Elasticsearch, Mysql, Dummy. Dummy storage has two records for testing: "example.tld" and "google.com". You can test it:

whois -h localhost example.tld

or

whois -h localhost google.com

These fixtures placed in "storage" package directory.

Mapping

All required fields for whoisd must be defined in the mapping file. The mapping file represent all fields in your database as key names in the whoisd. The mapping file should be in /etc/whoisd/conf.d/mapping.json. It possible to load mapping file through -mapping option. The context of the mapping file is described below:

[
  {
    "TLDs": ["eu"],
    "Fields" : {
      "01": {
        "key": "Domain Name: ",
        "name": ["name"],
        "related": "name"
      },
      "02": {
        "key": "Registry Domain ID: ",
        "name": ["domainId"],
        "related": "name"
      },
      "03": {
        "key": "Registrar WHOIS Server: ",
        "value": ["whois.yourwhois.eu"]
      }
    }
  },
  {
    "TLDs": ["com", "net"],
    "Fields" : {
      "01": {
        "key": "Domain Name: ",
        "name": ["name"],
        "related": "name"
      },
      "02": {
        "key": "Registry Domain ID: ",
        "name": ["domainId"],
        "related": "name"
      },
      "03": {
        "key": "Registrar WHOIS Server: ",
        "value": ["whois.yourwhois.eu"]
      }
    }
  }
]
  • "TLDs" - a list of TLDs which accepted by Whois Daemon for specified fields
  • "Fields" - a list of fields from "01" to last number "nn" in ascending order
  • "key" - a label for the field (preinstalled config file has keys according to ICANN requirements)
  • "value" - use it if the field has constant value (not defined field from the database)
  • "name" - a name of the field in a database, if the field has not constant value ("value" is not defined)
  • "related" - a name of the field in a database through which a request for
[
  {

      "06": {
        "key": "Creation Date: ",
        "name": ["creationDate"],
        "format": "{date}",
        "related": "name"
      },
      "06": {
        "key": "Expiration Date: ",
        "name": ["expirationDate"],
        "format": "{shortdate}",
        "related": "name"
      }

  }
]
  • "format" - special instructions to indicate how to display field, the examples shown below
  • "{date}" - used in the format to indicate that the field is a 'date' and need special formatting of the date RFС3339
  • "{shortdate}" - used in the format to indicate that the field is a 'date' and need special formatting of the date in short format like '2006.02.03'
[
  {

      "12": {
        "key": "Domain Status: ",
        "name": ["domainStatus"],
        "multiple": true,
        "related": "name"
      },

      "52": {
        "key": "Name Server: ",
        "name": ["name"],
        "multiple": true,
        "related": "nsgroupId",
        "relatedBy": "nsgroupId",
        "relatedTo": "nameserver"
      }

  }
]
  • "multiple" - if this option is set to 'true', each value will be repeated in whois output with the same label like that:
Name Server: ns1.example.com
Name Server: ns2.example.com
Name Server: ns3.example.com
  • "relatedBy" - a name of the field in a database through which related a request for
  • "relatedTo" - a name of the table/type in a database through which made a relation
[
  {

      "13": {
        "key": "Registry Registrant ID: ",
        "name": ["handle"],
        "hide": true,
        "related": "ownerHandle",
        "relatedBy": "handle",
        "relatedTo": "customer"
      },

  }
]
  • "hide" - if this option is set to 'true' and value is empty the field will not shown in whois output
[
  {

      "14": {
        "key": "Registrant Name: ",
        "name": ["name.fullName"],
        "related": "ownerHandle",
        "relatedBy": "handle",
        "relatedTo": "customer"
      },

      "40": {
        "key": "Tech Name: ",
        "name": ["name.firstName", "name.lastName"],
        "related": "techHandle",
        "relatedBy": "handle",
        "relatedTo": "customer"
      }

  }
]
  • "name": ["name.fullName"] - use dot notation for embedded fields (MySQL storage not allowed)
  • "name": ["name.firstName", "name.lastName"] - all values of the fields will be joined by default
[
  {

      "01": {
        "key": "domain: ",
        "name": ["name"],
        "format": "{idn}",
        "multiple": true,
        "related": "name"
      },
      "21": {
        "key": "Registrant Phone: ",
        "name": ["phone.countryCode", "phone.areaCode", "phone.subscriberNumber"],
        "format": "{string}.{string}{string}",
        "related": "ownerHandle",
        "relatedBy": "handle",
        "relatedTo": "customer"
      }

  }
]
  • "format": "{idn}" - indicate that the field "name" (domain name) need to decode according to IDN in UTF-8
  • "format": "{string}.{string}{string}" - indicate that the fields ["phone.countryCode", "phone.areaCode", "phone.subscriberNumber"] need special formatting "{string}.{string}{string}" (they are not simple joined)
  • {string} - represent one string field in format option
[
  {

      "14": {
        "key": "Registrant Name: ",
        "name": ["name.firstName", "name.lastName"],
        "related": "ownerHandle",
        "relatedBy": "handle",
        "relatedTo": "customer",
        "replacedBy": "15"
      },
      "15": {
        "key": "Registrant Organization: ",
        "name": ["companyName"],
        "related": "ownerHandle",
        "relatedBy": "handle",
        "relatedTo": "customer"
      },

  }
]
  • "replacedBy" - it contains the number of field that replaces this field. If the field is referenced by "replacedBy" has non-empty value, then this field will not show in whois output, because this field should be replaced by the field specified in 'ReplacedBy'. As a result will be show "Registrant Name: " or "Registrant Organization: " but not all together.
[
  {

      "55": {
        "key": "",
        "value": [""],
        "format": ">>> Last update of WHOIS database: {date} <<<"
      }
  }
]
  • an example of formating where used undefined tag "{date}", because a name of the field has not present, "{date}" will be replaced by CURRENT date in RFC3339 format

Todo

  • in memory storage
  • Rest API
  • update storage records by Rest API

Contributors (unsorted)

All the contributors are welcome. If you would like to be the contributor please accept some rules.

  • The pull requests will be accepted only in "develop" branch
  • All modifications or additions should be tested

Thank you for your understanding!

License

MIT License

whoisd's People

Contributors

ereslibre avatar pecharmin avatar takama avatar tespio avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

whoisd's Issues

Problems with MySQL backend connection.

Hi,

I'm trying to link this up with a MySQL back-end, but I receive an error when doing the test.

whoisd -t

[SERVICE]: 2018/04/04 09:47:38 whoisd started on 0.0.0.0:43
[SERVICE]: 2018/04/04 09:47:38 Used storage Mysql on 10.22.160.38:3306
[STORAGE]: 2018/04/04 09:47:38 query:
[STORAGE:ERROR]: 2018/04/04 09:47:38 storage.go:79: Empty query
[SERVICE]: 2018/04/04 09:47:38 Read bytes: 10
not found

I had sort of the same error when doing the make test.

make test

Segmentation Fault or Critical Error encountered. Dumping core and aborting.
dep ensure

  • fmt

  • lint

  • vet

  • test
    ? github.com/openprovider/whoisd/cmd [no test files]
    === RUN TestClientHandling
    flag provided but not defined: -test.v
    Usage of whoisd:
    -base string
    (default "whois")
    -config string

    -conn int
    (default 100)
    -host string
    (default "localhost")
    -mapping string

    -port int
    (default 9800)
    -shost string
    (default "localhost")
    -spassword string
    (default "test")
    -sport int
    (default 9900)
    -storage string
    (default "dummy")
    -suser string
    (default "test")
    -t (default true)
    -table string
    (default "domain")
    -test
    (default true)
    -work int
    (default 100)
    [STORAGE]: 2018/04/04 09:47:08 query: google.com
    --- PASS: TestClientHandling (0.01s)
    PASS
    coverage: 40.0% of statements
    [CLIENT]: 2018/04/04 09:47:08 pipe google.com true
    ok github.com/openprovider/whoisd/pkg/client 1.025s coverage: 40.0% of statements
    === RUN TestConfig
    flag provided but not defined: -test.v
    Usage of whoisd:
    -base string
    (default "whois")
    -config string

    -conn int
    (default 100)
    -host string
    (default "localhost")
    -mapping string

    -port int
    (default 9800)
    -shost string
    (default "localhost")
    -spassword string
    (default "test")
    -sport int
    (default 9900)
    -storage string
    (default "dummy")
    -suser string
    (default "test")
    -t (default true)
    -table string
    (default "domain")
    -test
    (default true)
    -work int
    (default 100)
    --- PASS: TestConfig (0.01s)
    PASS
    coverage: 90.8% of statements
    ok github.com/openprovider/whoisd/pkg/config 1.018s coverage: 90.8% of statements
    === RUN TestMapper
    --- PASS: TestMapper (0.00s)
    PASS
    coverage: 60.0% of statements
    ok github.com/openprovider/whoisd/pkg/mapper 1.009s coverage: 60.0% of statements
    === RUN TestService
    flag provided but not defined: -test.v
    Usage of whoisd:
    -base string
    (default "whois")
    -config string

    -conn int
    (default 100)
    -host string
    (default "localhost")
    -mapping string

    -port int
    (default 9800)
    -shost string
    (default "localhost")
    -spassword string
    (default "test")
    -sport int
    (default 9900)
    -storage string
    (default "dummy")
    -suser string
    (default "test")
    -t (default true)
    -table string
    (default "domain")
    -test
    (default true)
    -work int
    (default 100)
    [SERVICE]: 2018/04/04 09:47:09 whoisd started on localhost:9800
    [SERVICE]: 2018/04/04 09:47:09 Used storage dummy on localhost:9900
    [STORAGE]: 2018/04/04 09:47:09 query: google.com
    [SERVICE]: 2018/04/04 09:47:09 Read bytes: 3005
    [CLIENT]: 2018/04/04 09:47:09 pipe google.com true
    --- PASS: TestService (0.03s)
    PASS
    coverage: 48.3% of statements
    ok github.com/openprovider/whoisd/pkg/service 1.043s coverage: 48.3% of statements
    === RUN TestDummySearch
    --- PASS: TestDummySearch (0.00s)
    === RUN TestDummySearchRelated
    --- PASS: TestDummySearchRelated (0.00s)
    === RUN TestDummySearchMultiple
    --- PASS: TestDummySearchMultiple (0.00s)
    === RUN TestDummySearchEmpty
    --- PASS: TestDummySearchEmpty (0.00s)
    === RUN TestElasticSearchEmpty
    --- PASS: TestElasticSearchEmpty (0.01s)
    === RUN TestMysqlSearchEmpty
    --- PASS: TestMysqlSearchEmpty (0.00s)
    === RUN TestStorage
    [STORAGE]: 2018/04/04 09:47:08 query:
    [STORAGE:ERROR]: 2018/04/04 09:47:08 storage.go:83: Empty query
    [STORAGE]: 2018/04/04 09:47:08 query: aaa
    [STORAGE]: 2018/04/04 09:47:08 query: google.com
    [STORAGE]: 2018/04/04 09:47:08 query: mmm
    [STORAGE]: 2018/04/04 09:47:08 query: eee
    --- PASS: TestStorage (0.00s)
    PASS
    coverage: 62.2% of statements
    ok github.com/openprovider/whoisd/pkg/storage 1.031s coverage: 62.2% of statements
    ? github.com/openprovider/whoisd/pkg/version [no test files]

Can anyone shed any light on the Empty query for storage.go? I do not see any network traffic going out to my MySQL server. Is there a forum set up where users can talk about issues and setups?

TIA,
Todd

Cannot install

whois@whois:~/whoisd$ make check
/bin/sh: 1: docker: not found
/bin/sh: 1: docker: not found
go get -u github.com/golang/dep/cmd/dep
go: go.mod file not found in current directory or any parent directory.
'go get' is no longer supported outside a module.
To build and install a command, use 'go install' with a version,
like 'go install example.com/cmd@latest'
For more information, see https://golang.org/doc/go-get-install-deprecation
or run 'go help get' or 'go help install'.
make: *** [Makefile:126: bootstrap] Error 1

What can i do ?

how to use multiple ip in host /etc/whoisd/whoisd.conf ?

example file whoisd.conf is
{
"host": "0.0.0.0",
"port": 43,
"workers": 1000,
"connections": 1000,
"storage": {
"storageType": "Dummy",
"host": "localhost",
"port": 9200,
"user": "test",
"password": "test",
"indexBase": "whois",
"typeTable": "domain"
}
}

but, how to use multiple ip?
example IPv4 10.0.0.100 && IPv6 fddf::100
i tried array but error
[SETUP:ERROR]: 2021/02/15 17:43:07 whoisd.go:52: Loading mapping file was unsuccessful - json: cannot unmarshal array into Go struct field Record.Host of type string

{
"host": ["10.0.0.100", "[fddf::100]"],
"port": 43,
"workers": 1000,
"connections": 1000,
"storage": {
"storageType": "Dummy",
"host": "localhost",
"port": 9200,
"user": "test",
"password": "test",
"indexBase": "whois",
"typeTable": "domain"
}
}

I apologize for my English
Regards Florian

Create anti affinity rule for replica set

Due to service using nodePort and external traffic to local node we need to have anti affinity rule. Because we should avoid to seat more than one pod on every node.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.