Giter Club home page Giter Club logo

denominator's Introduction

Denominator

Portable control of DNS clouds

Denominator is a portable Java library for manipulating DNS clouds. Denominator has pluggable back-ends, including AWS Route53, Neustar Ultra, DynECT, Rackspace Cloud DNS, OpenStack Designate, and a mock for testing. We also ship a command line version so it's easy for anyone to try it out. Denominator currently supports basic zone and record features, as well GEO (aka Directional) profiles. See Netflix Tech Blog for an introduction.

Build Status

Command line

For your convenience, the denominator cli is a single executable file. Under the hood, the cli uses airline to look and feel like dig or git.

Android

There's no official android client, yet. However, we do have an Android Example you can try out.

Binaries

If you are using OSX, Homebrew has a built-in installer for denominator.

Here's how to get denominator-cli 4.6.0 from bintray

  1. Download denominator
  2. Place it on your $PATH. (ex. ~/bin)
  3. Set it to be executable. (chmod 755 ~/bin/denominator)

Getting Started

Advanced usage, including ec2 hooks are covered in the readme. Here's a quick start for the impatient.

If you just want to fool around, you can use the mock provider.

# first column is the zone id, which isn't always its name!
$ denominator -p mock zone list
denominator.io.          denominator.io.                                          admin.denominator.io.                86400
$ denominator -p mock record -z denominator.io. list
denominator.io.                                    SOA   3600   ns1.denominator.io. admin.denominator.io. 1 3600 600 604800 60
denominator.io.                                    NS    86400  ns1.denominator.io.

Different providers connect to different urls and need different credentials. First step is to run ./denominator providers to see how many -c args you need and what values they should have:

$ denominator providers
provider   url                                                 duplicateZones credentialType credentialArgs
mock       mem:mock                                            false
clouddns   https://identity.api.rackspacecloud.com/v2.0/       true           apiKey         username apiKey
designate  http://localhost:5000/v2.0                          true           password       tenantId username password
dynect     https://api2.dynect.net/REST                        false          password       customer username password
route53    https://route53.amazonaws.com                       true           accessKey      accessKey secretKey
route53    https://route53.amazonaws.com                       true           session        accessKey secretKey sessionToken
ultradns   https://ultra-api.ultradns.com:8443/UltraDNS_WS/v01 false          password       username password

Now, you can list your zones or records.

$ denominator -p ultradns -c my_user -c my_password zone list
--snip--
netflix.com.
--snip--
$ denominator -p ultradns -c my_user -c my_password record --zone netflix.com. list
--snip--
email.netflix.com.                                 A     3600   192.0.2.1
--snip--

Code

Denominator exposes a portable model implemented by pluggable Providers such as route53, ultradns, dynect, clouddns, or mock. Under the covers, providers are Dagger modules. Except for the mock, all current providers bind to http requests via feign, which keeps the distribution light.

Binaries

The current version of denominator is 4.6.0

Denominator can be resolved as maven dependencies, or equivalent in lein, gradle, etc. Here are the coordinates, noting you only need to list the providers you use.

<dependencies>
  <dependency>
    <groupId>com.netflix.denominator</groupId>
    <artifactId>denominator-core</artifactId>
    <version>${denominator.version}</version>
  </dependency>
  <dependency>
    <groupId>com.netflix.denominator</groupId>
    <artifactId>denominator-clouddns</artifactId>
    <version>${denominator.version}</version>
  </dependency>
  <dependency>
    <groupId>com.netflix.denominator</groupId>
    <artifactId>denominator-dynect</artifactId>
    <version>${denominator.version}</version>
  </dependency>
  <dependency>
    <groupId>com.netflix.denominator</groupId>
    <artifactId>denominator-route53</artifactId>
    <version>${denominator.version}</version>
  </dependency>
  <dependency>
    <groupId>com.netflix.denominator</groupId>
    <artifactId>denominator-ultradns</artifactId>
    <version>${denominator.version}</version>
  </dependency>
</dependencies>

Getting Started

Creating a connection to a provider requires that you have access to two things: the name of the provider, and as necessary, credentials for it.

import static denominator.CredentialsConfiguration.credentials;
...
DNSApiManager manager = Denominator.create("ultradns", credentials(username, password)); // manager is closeable, so please close it!

The credentials are variable length, as certain providers require more that 2 parts. The above returns an instance of DNSApiManager where apis such as ZoneApis are found. Here's how to list zones:

for (Zone zone : manager.api().zones()) {
  for (ResourceRecordSet<?> rrs : manager.api().recordSetsInZone(zone.id())) {
    processRRS(rrs);
  }
}

If you are running unit tests, or don't have a cloud account yet, you can use mock, and skip the credentials part.

DNSApiManager manager = Denominator.create("mock");

The Denominator model is based on the ResourceRecordSet concept. A ResourceRecordSet is simply a group of records who share the same name and type. For example all address (A) records for the name www.netflix.com. are aggregated into the same ResourceRecordSet. The values of each record in a set are type-specific. These data types are implemented as map-backed interfaces. This affords both the strong typing of java and extensibility and versatility of maps.

For example, the following are identical:

mxData.getPreference();
mxData.get("preference");

Resource record sets live in a Zone, which is roughly analogous to a domain. Zones can be created on-demand and deleted.

Use via Dagger

Some users may wish to use Denominator as a Dagger library. Here's one way to achieve that:

import static denominator.CredentialsConfiguration.credentials;
import static denominator.Dagger.provider;
...
// this shows how to facilitate runtime url updates
Provider fromDiscovery = new UltraDNSProvider() {
  public String url() {
    return discovery.urlFor("ultradns");
  }
};
// this shows how to facilitate runtime credential updates
Supplier<Credentials> fromEncryptedStore = new Supplier<Credentials>() {
  public Credentials get() {
    return encryptedStore.getCredentialsFor("ultradns");
  }
}
DNSApiManager manager = ObjectGraph.create(provider(fromDiscovery), new UltraDNSProvider.Module(), credentials(fromEncryptedStore))
                                   .get(DNSApiManager.java);

Third-Party Providers

Denominator also operates with third-party DNS providers such as DiscoveryDNS.

Providers are looked up by name using ServiceLoader. Given the provider is in the classpath, normal lookups should work.

// Given the discoverydns jar is in the classpath
DNSApiManager manager = Denominator.create("discoverydns");

Third-party support is also explained in the CLI readme.

Build

To build:

$ git clone [email protected]:Netflix/denominator.git
$ cd denominator/
$ ./gradlew clean test install

Intellij Idea IDE

Generate the Idea project files:

$ ./gradlew idea

Import the project:

  1. File > Import Project...
  2. Preferences > Compiler > Annotation Processors > Check "Enable annotation processing"

Run the live tests:

  1. Choose a live test (e.g. CloudDNSCheckConnectionLiveTest)
  2. Right click and select "Create CloudDNSCheckConnectionLiveTest"
  3. VM options: -ea -Dclouddns.username=<username> -Dclouddns.password=<password>
  4. Working directory: /path/to/denominator/clouddns

Feedback and Collaboration

  • For high-level updates, follow denominatorOSS on Twitter.
  • For questions, please tag denominator in StackOverflow.
  • For bugs and enhancements, please use Github Issues.
  • For email discussions, please post to the user forum
  • For discussions on design and internals, please join #denominator on irc freenode or post to the dev forum

LICENSE

Copyright 2013 Netflix, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

denominator's People

Contributors

adriancole avatar etoews avatar garethbowles avatar jdamick avatar jkschneider avatar quidryan avatar randgalt avatar rspieldenner avatar santhosh-tekuri avatar stonse avatar thesurlydev avatar wnagele avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

denominator's Issues

record set versioning

At netflix, we'd like to be able to restore the state of a record set in reaction to operator error or data failure. In order to do so, we need a versioned api with pluggable support for cassandra (and maybe git).

Support list ResourceRecordSets by name

Currently the only way to query a specific DNS name is though ResourceRecordSetApi.getByNameAndType(String name, String type). However, there are use cases where the user does not know the type. If we can have another API in ResourceRecordSetApi that query with only the DNS name parameter, it can save the list API call to find the types.

Iterator<ResourceRecordSet<?>> listByName(String name) can avail this.

Tasks:

  • implement Iterator<ResourceRecordSet<?>> listByName(String name)
  • implement listByName in mock
  • implement listByName in ultradns
  • implement listByName in route53
  • implement listByName in dynect
  • list records by name using kwargs. ex. denominator record -z foo.com list --name hostname.foo.com

atomic implementation of applyRegionsToNameTypeAndGroup for UltraDNS

When testing the ultradns-ws provider, Netflix noticed that migrating territories from one directional group to another to results in a drop of traffic and slow recovery. The root cause is how updates must occur considering territory overlaps are illegal.

Ex. first remove argentina from group a. next add argentina to group b.

Advice is to wrap the two commands into a transaction, which may eliminate some traffic loss scenarios.

support get RecordSet by name and type

while nice to be able to dump the entire zone via list(), we should be able to retrieve a single RecordSet uniformly.

Optional<ResourceRecordSet<?>> getByNameAndType(String name, String type) can avail this.

Tasks:

  • implement Optional<ResourceRecordSet<?>> getByNameAndType(String name, String type)
  • implement replace in mock
  • implement replace in ultradns
  • implement replace in route53
  • implement replace in dynect
  • get record using kwargs. ex. denominator record -z foo.com get --name hostname.foo.com --type A

allow provider-specific args in the cli

currently, in order to specify credentials, we have to use the abstract -c flag for each part.

ex. denominator -p ultradns -c username -c password zone list

If we instead group on provider, we can detect what the exact values should be.

ex. denominator route53 --accessKey ... --secretKey ... zone list

In this case, airline would read Provider.defaultCredentialSupplier() to see if args are even required, or Provider.getCredentialTypeToParameterNames() to see what the argument names should be.

cover more record types in BaseProviderLiveTest

ultradns takes significantly different code paths if a record is an A or a CNAME. There are also slightly different paths A vs AAAA. Right now, BaseProviderLiveTest only tests multiple records which are A. In order to get better coverage, we need to at least add tests for CNAME manipulation, if not also AAAA.

Support for config files

It would be nice if this supported config files, like a .denominator file in your $HOME. This way you don't have to pass in credentials everytime.

support load balanced CNAME records

For example, we would like to read AWS route53 records with multiple CNAMEs.

multi-value (load-balanced) ResourceRecordSets:

Tasks:

  • implement base tests for round-robin cnames in mock
  • implement round-robin cnames in ultradns
  • implement round-robin cnames in route53
  • implement round-robin cnames in dynect

release version 1.0.2

It's time to release Denominator v1.0.2

  • Includes resource record set CRUD and TTL operations for DynECT, Route53, CloudDNS, and UltraDNS
  • Includes a command-line client with EC2 hooks for instance metadata
  • cut 1.x branch
  • bump master to 1.1.0-SNAPSHOT
  • have @allenxwang test 1.x branch
  • release 1.0.2 from 1.x branch, publishing to maven central
  • point the homebrew formula to use maven central location of the fat jar as download source (http://dl.bintray.com/content/netflixoss/denominator/denominator-cli/release/1.0.2/denominator?direct)
  • update bintray binary including version number
  • communicate via irc, google groups, twitter etc.

support apply ttl by name and type

Lowering ttl by recordset allows resolvers to become more "aware", prior to dns changes or migration. Let's add the ability to achieve this, granular to record name, type, so that users can prepare for the road ahead.

void applyTTLToNameAndType(String name, String type, UnsignedInteger ttl).

Tasks:

  • implement void applyTTLToNameAndType(String name, String type, UnsignedInteger ttl)
  • implement applyTTLToNameAndType in mock
  • implement applyTTLToNameAndType in ultradns
  • implement applyTTLToNameAndType in route53
  • implement applyTTLToNameAndType in dynect
  • applyTTL using kwargs. ex. denominator record -z foo.com applyTTL --name hostname.foo.com --type A --ttl 0

release denominator 1.1.1

It's time to release Denominator v1.1.1 maintenance release

  • release 1.1.1 from 1.x branch, publishing to maven central
  • add denominator-cli version 1.1.1 to bintray, uploading the fat jar as denominator-cli/release/1.1.1/denominator
  • point the homebrew formula to use maven central location of the fat jar as download source (http://dl.bintray.com/content/netflixoss/denominator/denominator-cli/release/1.1.1/denominator?direct) per issue #134
  • verify that 1.1.1 is in maven central
  • communicate via irc, google groups, twitter etc.

support idempotent replace RecordSet

We have 2 projects that need to replace the entire recordset for a given name and type, and without regard to prior values. One project needs to replace the CNAME pointing to Amazon ELB instances. Another needs to replace all existing A records for a name that points to many Amazon EIPs.

In both cases, an idempotent replace(ResourceRecordSet) is desired, particularly as providers can optimize delete/add records into the same transaction.

Tasks:

  • implement ResourceRecordSetApi.replace(ResourceRecordSetApi<?> in)
  • implement replace in mock
  • implement replace in ultradns
  • implement replace in route53
  • implement replace in dynect
  • replace rrset using kwargs. ex. denominator record -z foo.com replace --name hostname.foo.com --type A --data 1.2.3.4

implement record add/remove for ultradns

Here's how to add a record

ResourceRecord toAdd = rrBuilder().name("www." + zoneName).type("A").ttl(1800).rdata("1.2.3.4").build();

String guid = ultra.getResourceRecordApiForZone(zoneName).create(toAdd);

Here's how to remove a record

api =  ultra.getResourceRecordApiForZone(zoneName);
toDelete = api.listByNameAndType("www." + zoneName, "A").tryFind(rdataEquals("1.2.3.4");
if (toDelete.isPresent())
   api.delete(toDelete.get().getGuid());

Dynect API throws exception on concurrent reads from the same client

When there are concurrent reads from a same client for the same name and type, Dynect API throws an exception:

org.jclouds.dynect.v3.DynECTExceptions$JobStillRunningException: This session already has a job running
at org.jclouds.dynect.v3.handlers.DynECTErrorHandler.handleError(DynECTErrorHandler.java:47)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:182)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:152)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:131)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:97)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:59)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:119)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
at com.sun.proxy.$Proxy148.listByFQDNAndType(Unknown Source)
at denominator.dynect.DynECTResourceRecordSetApi.exisingRecordIdsByNameAndType(DynECTResourceRecordSetApi.java:237)
at denominator.dynect.DynECTResourceRecordSetApi.existingRecordsByNameAndType(DynECTResourceRecordSetApi.java:184)
at denominator.dynect.DynECTResourceRecordSetApi.getByNameAndType(DynECTResourceRecordSetApi.java:67)
at com.netflix.dns.RevisionPersister$1.run(RevisionPersister.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.jclouds.http.HttpResponseException: command: GET https://api2.dynect.net/REST/TXTRecord/allen.com./testnlb.allen.com HTTP/1.1 failed with response: HTTP/1.1 400 OK; content: [{"status": "failure", "data": {}, "job_id": 316077550, "msgs": [{"INFO": "token: This session already has a job running", "SOURCE": "API-B", "ERR_CD": "OPERATION_FAILED", "LVL": "ERROR"}]}]
at org.jclouds.dynect.v3.handlers.DynECTErrorHandler.handleError(DynECTErrorHandler.java:45)

Error handling - expected exceptions from cardinality violations

For the case where an add/remove on the ResourceRecordSetApi fails. The providers need a standard way to express errors that occur that are specific to a particular record type or provider record type.

As an example, in base DNS implementations you can only have 1 CNAME record at a particular label, www. in the case below.

Existing: www.denominator.io CNAME to www1
Then trying to add: www.denominator.io CNAME to www2
This should throw an exception such as:

TargetExistsException (which extends IllegalStateException)

Other implementations may allow this behavior through different provide specific extensions, round-robin answers, "pools", etc. - In this case there may be other errors that would occur, these should also throw TargetExistsException or IllegalStateException as appropriate.

release denominator 1.1.3

It's time to release Denominator v1.1.3 which supports use as a Dagger library and adds the ability to override provider urls.

  • update README.md and denominator-cli/README.md
  • release 1.1.3 from 1.x branch, publishing to maven central
  • add denominator-cli version 1.1.3 to bintray, uploading the fat jar as denominator-cli/release/1.1.3/denominator
  • update the homebrew formula to use bintray location of the fat jar as download source (http://dl.bintray.com/content/netflixoss/denominator/denominator-cli/release/1.1.3/denominator?direct) per issue #134
  • verify that 1.1.3 is in maven central
  • communicate via irc, google groups, twitter etc.

add default path and environment variable to define configuration file path

I propose the following default path for the configuration file: ~/.denominator/config.yml

Let's also add an environment variable, DENOMINATOR_CONFIG, or something similar to override the default configuration file path.

The existing -C, --config CLI arg will override both the default path and env. variable.

This will further simplify the number of args required to run commands against providers.

allow zone format syntax in CLI

This is both for recordset as well bulk operations:

Tasks:

  • add record using zone format. ex. denominator record -z foo.com add hostname.foo.com 3600 IN A 1.2.3.4
  • bulk add record using zone format. ex. denominator record -z foo.com add --file cnames.txt

analyze existing record data in netflix zones

Relating to issue #7 we need to analyze all resource record types used across Route 53, UltraDNS and DynECT to ensure the model we design is as simple as possible yet covers all existing DNS records in Netflix. In this case, we should categorize washed information from live production.

Let's take care to hunt down the purpose of any outlying record types, such as those with private or provider-specific types (ex. 65280).

Release Denominator 1.1.4

It's time to release Denominator v1.1.4 which increases performance, better supports geo changes, and adds a config file to the cli.

  • update README.md and denominator-cli/README.md
  • release 1.1.4 from 1.x branch, publishing to maven central
  • add denominator-cli version 1.1.4 to bintray, uploading the fat jar as denominator-cli/release/1.1.4/denominator
  • Update the homebrew formula with the latest bintray content
  • verify that 1.1.4 is in maven central
  • communicate via irc, google groups, twitter etc.

implement add/remove record for dynect

Here's how to add a record

import static org.jclouds.dynect.v3.domain.rdata.AData.a;
...
api = context.getApi().getRecordApiForZone(zoneName);

   CreateRecord<AData> record = CreateRecord.<AData> builder()
                                            .fqdn("www." + zoneName)
                                            .type("A")
                                            .rdata(a("1.2.3.4"))
                                            .build();
      Job job = null;
      while (true) {
         try {
            job = api.scheduleCreate(record);
            break;
         } catch (JobStillRunningException e) {
            // consider backoff.
            continue;
         }
      }
      checkState(job.getStatus(), Status.SUCCESS);
      // note regardless of above nothing happens until publish occurs
      // ex. session expiration or loss loses the change
      zoneApi().publish(zoneName);

Here's how to remove a record

api = context.getApi().getRecordApiForZone(zoneName);

toDelete = api.listByFQDNAndType(record.getFQDN(), record.getType()).tryFind(rdataEquals("1.2.3.4");

api =  ultra.getResourceRecordApiForZone(zoneName);
toDelete = api.listByNameAndType("www." + zoneName, "A").tryFind(rdataEquals("1.2.3.4");
if (toDelete.isPresent()) {
   Job job = api.delete(toDelete.get());
   // job might be null 
   checkState(job.getStatus(), Status.SUCCESS);
   // like above, note delete doesn't occur until publish
   zoneApi().publish(zoneName);
}

backends advertise RR types they support

While most zones contain a subset of common record types such as SOA or A, not all DNS services support the universe of record types. It should be possible to determine what types of records an implementation supports before attempting to create a record.

remove requirement for `provideThis()`

per square/dagger#176 we can unwind some complexity that results in implementors having to add provideThis() to their Provider subclasses in the following fashion:

One thing you could experiment with is ObjectGraph.plus(). Perhaps you could build your application in two phases: build a small inner object graph with everything you need during bootstrap, then build upon that to create a more complete object graph.

backfill mock tests on ultradns

test coverage on the ultradns provider is significantly less than other providers. here's a list of missing tests in UltraDNSResourceRecordSetApiMockTest

add tests

  • addFirstACreatesRoundRobinPoolThenAddsRecordToIt
  • addFirstAAAARoundRobinPoolThenAddsRecordToIt
  • addFirstAReusesExistingEmptyRoundRobinPool
  • addFirstAAAAReusesExistingEmptyRoundRobinPool
  • addSecondAAddsRecordToExistingPool
  • addSecondAAAAAddsRecordToExistingPool
  • addExistingADoesNothing
  • addExistingAAAADoesNothing
  • addFirstNormalCreatesRecord
  • addSecondNormalCreatesRecordWithExistingTTL
  • addExistingNormalRecordDoesNothing

remove tests

  • removeOnlyRecordAlsoRemovesPool

add ability to access version at runtime

Denominator needs to be able to emit its version, at least in CLI output.

Note I only think we need to consider semantic version and snapshot y|n

One idea is to add the following inner class to denominator.Denominator and read values from some resource.

public static enum Version {
    INSTANCE;

    public final int majorVersion;
    public final int minorVersion;
    public final int patchVersion;
    public final boolean snapshot;
    // implement here.
}

Problem while listing DNS entries in rackspace cloud dns

Thread [main] (Suspended (breakpoint at line 42 in Absent)) 
    Absent.get() line: 42   
    RecordsToPagedIterable.markerToNextForArg0(Optional<Object>) line: 52   
    RecordsToPagedIterable(Arg0ToPagedIterable<T,I>).apply(IterableWithMarker<T>) line: 73  
    RecordsToPagedIterable(Arg0ToPagedIterable<T,I>).apply(Object) line: 61 
    Functions$FunctionComposition<A,B,C>.apply(A) line: 210 
    InvokeHttpMethod.invoke(Invocation) line: 97    
    InvokeHttpMethod.apply(Invocation) line: 80 
    InvokeHttpMethod.apply(Object) line: 51 
    FunctionalReflection$FunctionalInvocationHandler<T>.handleInvocation(Object, Method, Object[]) line: 119    
    FunctionalReflection$FunctionalInvocationHandler<T>(AbstractInvocationHandler).invoke(Object, Method, Object[]) line: 70    
    $Proxy72.list() line: not available 
    CloudDNSResourceRecordSetApi.list() line: 33    
    ResourceRecordSetCommands$ResourceRecordSetList.doRun(DNSApiManager) line: 58   
    ResourceRecordSetCommands$ResourceRecordSetList(Denominator$DenominatorCommand).run() line: 132 
    Denominator.main(String[]) line: 74 
  @Override public Object get() {
    throw new IllegalStateException("Optional.get() cannot be called on an absent value");
  }

Only happens in a domain with lots of entries. Small domains work well.

support delete RecordSet by name and type

we need to be able to delete a record set without having to know its prior values.

void deleteByNameAndType(String name, String type)

Tasks:

  • implement void deleteByNameAndType(String name, String type)
  • implement replace in mock
  • implement replace in ultradns
  • implement replace in route53
  • implement replace in dynect
  • get record using kwargs. ex. denominator record -z foo.com delete --name hostname.foo.com --type A

change to the apache jclouds distribution (1.6.1)

jclouds is now an apache incubator project. All group ids change from org.jclouds to org.apache.jclouds

migrating to a snapshot release implies adding the following repository to the top-level build.gradle

maven { url 'https://repository.apache.org/content/groups/snapshots/' }

If only using one jar, we'd need to be careful and exclude inheriting others. for example

-  compile     'org.jclouds.provider:ultradns-ws:1.6.0'
+  // only use snapshot with ultradns
+  compile('org.apache.jclouds.provider:ultradns-ws:1.6.1-SNAPSHOT') {
+    exclude group: 'org.apache.jclouds'
+  }

support adding basic records

Routine DNS work in netflix relates to adding records with the same ttl and simple rdata. For example, most of the time, we are adding A, CNAME, or AAAA, or NS values with TTL 3600, and then checking with dig to see if the updates occurred.

Let's make it very easy to add simple records as this can get early feedback on denominator.

Tasks:

  • implement ResourceRecordSetApi.add(ResourceRecordSetApi<?> in) (issue #36)
  • implement ResourceRecordSetApi.remove(ResourceRecordSetApi<?> in)(issue #36)
  • implement add/remove record for mock (issue #36)
  • implement add/remove record for ultradns (issue #34)
  • implement add/remove record for route53 (issue #33)
  • implement add/remove record for dynect (issue #35)
  • add record using kwargs. ex. denominator record -z foo.com add --name hostname.foo.com --type A --rdata 1.2.3.4

Create live tests for each record type

Quick testing with a complex record (in this case MX) showed errors on multiple providers. Even if not all tests pass, we should know which record types work per-provider

multi-value (round-robin) ResourceRecordSets:

  • AAAA
  • A
  • MX
  • NS
  • PTR
  • SPF
  • SRV
  • TXT

single-value ResourceRecordSets:

  • AAAA
  • A
  • CNAME
  • MX
  • NS
  • PTR
  • SPF
  • SRV
  • SSHFP
  • TXT

Dynect APIs throw "login: There was a problem with your credentials" errors from time to time

Once a while I observe this type of error if I keep using Dynect APIs from my application. Usually a restart will fix it.

org.jclouds.http.HttpResponseException: command: GET https://api2.dynect.net/REST/Zone HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{"status": "failure", "data": {}, "job_id": 305900967, "msgs": [{"INFO": "login: IP address does not match current session", "SOURCE": "BLL", "ERR_CD": "INVALID_DATA", "LVL": "ERROR"}, {"INFO": "login: There was a problem with your credentials", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}]
at org.jclouds.dynect.v3.handlers.DynECTErrorHandler.handleError(DynECTErrorHandler.java:45)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:182)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:152)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:131)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:97)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:59)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:119)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
at com.sun.proxy.$Proxy95.list(Unknown Source)
at denominator.dynect.DynECTZoneApi.list(DynECTZoneApi.java:19)

org.jclouds.http.HttpResponseException: command: GET https://api2.dynect.net/REST/TXTRecord/allen.com./test.allen.com. HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{"status": "failure", "data": {}, "job_id": 305897866, "msgs": [{"INFO": "login: IP address does not match current session", "SOURCE": "BLL", "ERR_CD": "INVALID_DATA", "LVL": "ERROR"}, {"INFO": "login: There was a problem with your credentials", "SOURCE": "BLL", "ERR_CD": null, "LVL": "INFO"}]}]
at org.jclouds.dynect.v3.handlers.DynECTErrorHandler.handleError(DynECTErrorHandler.java:45)
at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:182)
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:152)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:131)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:97)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:59)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:119)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
at com.sun.proxy.$Proxy146.listByFQDNAndType(Unknown Source)
at denominator.dynect.DynECTResourceRecordSetApi.exisingRecordIdsByNameAndType(DynECTResourceRecordSetApi.java:237)
at denominator.dynect.DynECTResourceRecordSetApi.existingRecordsByNameAndType(DynECTResourceRecordSetApi.java:184)
at denominator.dynect.DynECTResourceRecordSetApi.replace(DynECTResourceRecordSetApi.java:158)

when adding, replace ttl on all records in the set, if specified

We should update our javadocs and tests to prove that if someone specifies a new ttl when adding a record to a set, all existing records ttl are updated accordingly.

ResourceRecordSetApi.add(String name, String type, UnsignedInteger ttl, Map<String, Object> rdata)

Support Directional RRSets

Essentially clients who are bound by a group of region codes will receive answers different than other clients.

CLI parsing overhead seems a bit excessive

Low priority issue, but open for any interested party to attack.

Using guava stopwatch, I tracked the time to populate and build airline CliBuilder.build() at 108ms. The time to execute Cli.parse is 21ms. The parsing time is an order of magnitude less than processing, but together they amount to 130ms... 1/3 of the time inclusive of launching the JVM when using the mock provider!

Right now, remote provider calls include a heavy overhead due to the jclouds dependency and how its guice is wired. This overhead is so large that addressing the jclouds dep is a lower hanging fruit wrt performance and will be tackled in a separate issue.

That said, we'll at some point want to address CLI parsing overhead, particularly as jopt-simple overhead is 13ms or so. My suspicion is that the runtime annotation processing in airline is the cause for this overhead, and a Dagger-like approach could rid the excess.

expose Provider.url

In order to test beta versions of DNS services, and support portable ones like OpenStack, we need to expose the url of the provider, and make that possible to override.

Proposal is to add

    /**
     * The base API URL of the DNS Provider. Typically, an http url, such as
     * {@code https://api/v2}. For in-memory providers, we expect the scheme to
     * be {@code mem}. For example, {@code mem://mock}. Encoding credentials in
     * the URL is neither expected nor supported.
     */
    String getUrl();

implement record add/remove for route53

Here's code to add a record to a set.

ResourceRecordSet toAdd = ResourceRecordSet.builder().name("www.denominator.io.").type("A").add("1.1.1.1").build();
Change creating = resourceRecordSetApi.create(toAdd);

// replace below with your favorite polling loop, or.. just let it ride
checkState(Predicates2.retry(new Predicate<Change>() {
         public boolean apply(Change input) {
            return route53Api.getChange(input.getId()).getStatus() == INSYNC;
         }
      }, 600, 1, 5, SECONDS).apply(creating), "%s never synced!", creating);

same process to remove, except s/resourceRecordSetApi.create/resourceRecordSetApi.delete, and you could get a null change if the record was already removed.

Add ZoneApi.iterator() and enforce RRSetFactory.create(id)

In both route53 and clouddns, it is possible to create two zones with the same name. This leads to a loss of data, as any zones after the first won't be visible.

The following form will need to be used in order to facilitate this:

Zone {
  String name;
  Optional<String> id;
}

It would be expected that some services don't have an id, so usage would play out like so..

rrsets = dnsApi.resourceRecordSetApiForZone(zone.idOrName()).iterator();

implement add/remove record for mock

Here's how to add a record

import static denominator.model.ResourceRecordSets.a;
...
record = a("www.denominator.io.", "1.2.3.4");

rrs = FluentIterable.from(data.get(zoneName)).tryFind(nameAndType("www." + zoneName, "A"));
if (rrs.isPresent()) {
   record = a(rrs.getName(), rrs.getType(), rrs.getTTL(), ImmutableList.builder().addAll(rrs.get()).addAll(record).build());
   data.remove(zoneName, rrs.get());
}
data.put(zoneName, record);

removing is the same, except without the put.

release denominator 1.1.2

It's time to release Denominator v1.1.2 which updates to Dagger 1.0.0

  • release 1.1.2 from 1.x branch, publishing to maven central
  • add denominator-cli version 1.1.2 to bintray, uploading the fat jar as denominator-cli/release/1.1.2/denominator
  • point the homebrew formula to use maven central location of the fat jar as download source (http://dl.bintray.com/content/netflixoss/denominator/denominator-cli/release/1.1.2/denominator?direct) per issue #134
  • verify that 1.1.2 is in maven central
  • communicate via irc, google groups, twitter etc.

release denominator 1.1.0

It's time to release Denominator v1.1.0 including geo resource record set support!

  • bump master to 1.2.0-SNAPSHOT
  • bump 1.x to 1.1.0-SNAPSHOT
  • release 1.1.0 from 1.x branch, publishing to maven central
  • add denominator-cli version 1.1.0 to bintray, uploading the fat jar as denominator-cli/release/1.1.0/denominator
  • point the homebrew formula to use maven central location of the fat jar as download source (http://dl.bintray.com/content/netflixoss/denominator/denominator-cli/release/1.1.0/denominator?direct) per issue #134
  • verify that 1.1.0 is in maven central
  • add a tech blog on what's the news
  • communicate via irc, google groups, twitter etc.

2114 Exception from UltraDNS when GeoResourceRecordSetApi.listByNameAndType against a hostname with a space in it.

When executing GeoResourceRecordSetApi.listByNameAndType against a hostname with a space in it, a 2114 error code is raised.

Looking at the wire log, it seems that while getResourceRecordsOfDNameByType soap command succeeds, but getDirectionalDNSRecordsForHost throws the 2114 code.

Workaround is to filter out any hosts that have spaces as they can't safely be queried until this is sorted on the server side. Another workaround would be to remove hosts with spaces in them.

add mandatory test case to list more than 100 records

In many apis, pagination is used to deal with lists of records (or record sets) beyond 100. After record 100, you'd need to pass a marker or the like to a new http GET operation. This is documented in Route53 and CloudDNS api guides

The problem causing issue #145 wasn't caught by our tests as we don't currently exercise providers enough to avoid latent pagination bugs. Pagination bugs render a provider completely useless. We need to add a mandatory test to ensure all providers can operate on realistically sized domains before being merged into denominator.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.