sebdah / dynamic-dynamodb Goto Github PK
View Code? Open in Web Editor NEWDynamic DynamoDB provides auto scaling for AWS DynamoDB
Home Page: http://dynamic-dynamodb.readthedocs.org/
License: Apache License 2.0
Dynamic DynamoDB provides auto scaling for AWS DynamoDB
Home Page: http://dynamic-dynamodb.readthedocs.org/
License: Apache License 2.0
If "always-decrease-rw-together" is set to true, there is a chance for the requested provisioning change to be set to that of the current tables. Because of that among other reasons... there are times when a request is made to change a tables provisioning to the same values it is currently provisioned.
In the "update_throughput" function in "dynamic-dynamodb/core/init.py" I would recommend a simple check that determines if the requested provisioning values are both equal to the current provisioning values.
This snippet would do nicely around line 305-306 in the aforementioned file:
if read_units == table.read_units && write_units == table.write_units:
return
This may include many things, but at least all formatting tags must be numbered.
Split main function to separate Python file so the DynamicDynamoDB class gets its own file.
Restructure the Dynamic DynamoDB code base
Say for instance I have a table with a write throughput of 2, and in my configuration, that table's minimum write throughput is set to 5. When attempting to increase to that minimum write throughput, amazon will throw an error, because you can only increase table's throughput up to twice the amount of the current throughput. Might I suggest something along the lines of:
return min( get_table_option(table_name, 'min_provisioned_reads'), (current_provisioning * 2) )
return min( get_table_option(table_name, 'min_provisioned_writes'), (current_provisioning * 2) )
When returning a minimum provisioned read or write value so that the table can at least get closer to it's minimum provisioned value.
I suppose this is an edge case, but I'd rather give you the option to ignore it instead of the issue never being heard.
It seems that dynamic-dynamo stumbles over certain usage ratios.
This only happens with certain tables, not sure what makes the difference.
Example output:
ubuntu@blah:~$ dynamic-dynamodb --dry-run -c PROD_HashKnownByAccount.conf
2013-04-04 14:38:01,342 - dynamic-dynamodb - dryrun - INFO - Consumed reads: 811
2013-04-04 14:38:01,342 - dynamic-dynamodb - dryrun - INFO - Provisioned reads: 1000
2013-04-04 14:38:01,342 - dynamic-dynamodb - dryrun - INFO - Read usage: 82%
2013-04-04 14:38:01,385 - dynamic-dynamodb - dryrun - INFO - Consumed writes: 15
2013-04-04 14:38:01,386 - dynamic-dynamodb - dryrun - INFO - Provisioned writes: 200
2013-04-04 14:38:01,386 - dynamic-dynamodb - dryrun - INFO - Write usage: 8%
Traceback (most recent call last):
File "/usr/local/bin/dynamic-dynamodb", line 7, in <module>
main.main()
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/main.py", line 174, in main
dynamic_ddb.run()
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 148, in run
self.ensure_proper_provisioning()
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 207, in ensure_proper_provisioning
throughput['write_units']))
ValueError: Unknown format code 'd' for object of type 'str'
with this config:
# AWS region to use
region: us-east-1
# How often should Dynamic DynamoDB monitor changes (in seconds)
check-interval: 300
# -------------------------------------------------------------------------------
[table: PROD_HashKnownByAccount]
# -------------------------------------------------------------------------------
#
# Read provisioning configuration
#
# Thresholds for scaling up or down the provisioning (%)
reads-upper-threshold: 100
reads-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 20
decrease-reads-with: 80
# Maximum and minimum read provisioning
# Dynamic DynamoDB will not provision any more or less reads than this
min-provisioned-reads: 10
max-provisioned-reads: 1500
#
# Write provisioning configuration
#
# Thresholds for scaling up or down the provisioning (%)
writes-upper-threshold: 100
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-writes-with: 30
decrease-writes-with: 60
# Maximum and minimum write provisioning
# Dynamic DynamoDB will not provision any more or less writes than this
min-provisioned-writes: 10
max-provisioned-writes: 500
Add --log-file
command line option in order to support writing to log file when using only command line options.
Enhance configuration option validation so that e.g. it is not possible to increase writes with more than 100% for example.
It would be nice to make this more dynamic and listen on an sqs queue for alerts and proactively react to them, instead of polling cloudwatch...
I can have a go at this, but cant promise it will be very elegant ;)
Min/max limits seems to be read improperly from configuration files
On line 46 in calculators.py in the function increase_reads_in_percent, you have
increase = int(
float(current_provisioning)*(float(percent)/100+1))
updated_provisioning = current_provisioning + increase
it should just be
increase = int(float(current_provisioning)*(float(percent)/100))
updated_provisioning = current_provisioning + increase
or alternatively
updated_provisioning = int(float(current_provisioning)*(float(percent)/100+1))
Add support for max read/write provisioning
Would be nice to have an ability to log through syslog, in daemon mode, e.g.:
dynamic-dynamodb --instance foo --syslog dyndyn \
--table-name foo --check-interval 300 --daemon start \
Where dyndyn
is a facility for syslog.
Mixup of read and writes provisioing in scaling
Make it possible to configure AWS credentials on command line
Add support for maintenance windows. If specified Dynamic DynamoDB should only scale up/down during this window.
Multiple maintenance windows should be possible to add.
Send e-mail when DynamoDB limits are reaches (e.g. 2 scale down's per day).
I'm getting this exception:
2013-07-21 10:12:28,421 - dynamic-dynamodb - INFO - foo - Consumed read units: 0%
2013-07-21 10:12:28,427 - dynamic-dynamodb - INFO - foo - Scaling down reads is not done when usage is at 0%
2013-07-21 10:12:28,484 - dynamic-dynamodb - INFO - foo - Consumed write units: 40%
Jul 21 12:12:28 conveyer.rultor.com rultor: Traceback (most recent call last):
Jul 21 12:12:28 conveyer.rultor.com rultor: File "/usr/local/bin/dynamic-dynamodb", line 26, in <module>
Jul 21 12:12:28 conveyer.rultor.com rultor: dynamic_dynamodb.main()
Jul 21 12:12:28 conveyer.rultor.com rultor: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/__init__.py", line 68, in main
Jul 21 12:12:28 conveyer.rultor.com rultor: core.ensure_provisioning(table_name)
Jul 21 12:12:28 conveyer.rultor.com rultor: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/__init__.py", line 30, in ensure_provisioning
Jul 21 12:12:28 conveyer.rultor.com rultor: table_name)
Jul 21 12:12:28 conveyer.rultor.com rultor: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/__init__.py", line 214, in __ensure_provisioning_writes
Jul 21 12:12:28 conveyer.rultor.com rultor: get_table_option(table_name, 'decrease_writes_with'))
Jul 21 12:12:28 conveyer.rultor.com rultor: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/calculators.py", line 113, in decrease_writes_in_percent
Jul 21 12:12:28 conveyer.rultor.com rultor: current_provisioning)
Jul 21 12:12:28 conveyer.rultor.com rultor: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/calculators.py", line 31, in get_min_provisioned_writes
Jul 21 12:12:28 conveyer.rultor.com rultor: (current_provisioning * 2)))
Jul 21 12:12:28 conveyer.rultor.com rultor: TypeError: int() argument must be a string or a number, not 'NoneType'
This is how I call the tool:
dynamic-dynamodb \
--table-name foo \
--reads-upper-threshold 90 \
--reads-lower-threshold 30 \
--increase-reads-with 50 \
--decrease-reads-with 50 \
--writes-upper-threshold 90 \
--writes-lower-threshold 40 \
--increase-writes-with 50 \
--decrease-writes-with 70
Handle DynamoDB provisioning exceptions cleaner
I believe you can only adjust your DynamoDB throughput up to twice per day, so why the need for this library?
When dyn-dyn is started in daemon mode, it does not update the logfile. It works when started interactively.
Handle ResourceInUseException better. This exception is thrown when down scaling of a table is in progress when the next down scaling is performed.
2013-04-26 08:53:00,837 - dynamic-dynamodb - INFO - prod-displays-20130426 - Consumed read units: 20%
2013-04-26 08:53:00,898 - dynamic-dynamodb - INFO - prod-displays-20130426 - Consumed write units: 54%
2013-04-26 08:53:00,898 - dynamic-dynamodb - INFO - prod-displays-20130426 - Changing provisioning to 25 read units and 500 write units
2013-04-26 08:53:00,905 - dynamic-dynamodb - WARNING - prod-displays-20130426 - Not performing throughput changes when table is in UPDATING state
2013-04-26 08:53:00,913 - dynamic-dynamodb - ERROR - prod-displays-20130426 - Unhandled exception: ResourceInUseException: Attempt to change a resource which is still in use: Table is being updated: prod-displays-20130426. Please file a bug report at https://github.com/sebdah/dynamic-dynamodb/issues
Today Dynamic DynamoDB is more or less handling configuration files OR command line options. This should be changed so that combinations are supported. I think that command line options should have precedence over configuration file options.
The pid file name is badly named, update it to something that makes sense
For some table's access patterns, scaling the table in percent does not make sense and scaling by fixed amounts would make more sense.
Growth of table traffic isn't always exponential but can also be linear.
Mix up between percent and units when getting the consumed read units.
I'm getting this error once per every 100-200 runs of dynamic-dynamo (approximately):
2013-08-13 18:17:13,792 - dynamic-dynamodb - ERROR - Failed connecting to DynamoDB: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials
: 2013-08-13 18:17:13,792 - dynamic-dynamodb - ERROR - Please report an issue at: https://github.com/sebdah/dynamic-dynamodb/issues
: Traceback (most recent call last):
: File "/usr/local/bin/dynamic-dynamodb", line 22, in <module>
: import dynamic_dynamodb
: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/__init__.py", line 26, in <module>
: import core
: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/__init__.py", line 6, in <module>
: import dynamodb
: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/dynamodb.py", line 56, in <module>
: DYNAMODB_CONNECTION = __get_connection_dynamodb()
: File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/dynamodb.py", line 23, in __get_connection_dynamodb
: configuration['global']['region'])
: File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb/__init__.py", line 65, in connect_to_region
: return region.connect(**kw_params)
: File "/usr/local/lib/python2.7/dist-packages/boto/regioninfo.py", line 63, in connect
: return self.connection_cls(region=self, **kw_params)
: File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb/layer2.py", line 152, in __init__
: validate_certs=validate_certs)
: File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb/layer1.py", line 92, in __init__
: validate_certs=validate_certs)
: File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 554, in __init__
: host, config, self.provider, self._required_auth_capability())
: File "/usr/local/lib/python2.7/dist-packages/boto/auth.py", line 686, in get_auth_handler
: 'Check your credentials' % (len(names), str(names)))
: boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV4Handler'] Check your credentials
Add --log-level
option to command line
Stack trace:
Traceback (most recent call last):
File "./dynamic-dynamodb", line 7, in <module>
main.main()
File "/Users/sebastian/git/dynamic-dynamodb/dynamic_dynamodb/main.py", line 174, in main
dynamic_ddb.run()
File "/Users/sebastian/git/dynamic-dynamodb/dynamic_dynamodb/dynamic_dynamodb.py", line 148, in run
self.ensure_proper_provisioning()
File "/Users/sebastian/git/dynamic-dynamodb/dynamic_dynamodb/dynamic_dynamodb.py", line 160, in ensure_proper_provisioning
read_usage_percent = self._get_consumed_reads_percentage()
File "/Users/sebastian/git/dynamic-dynamodb/dynamic_dynamodb/dynamic_dynamodb.py", line 322, in _get_consumed_reads_percentage
float(consumed_reads) / float(self._get_provisioned_read_units()) * 100))
File "/Users/sebastian/git/dynamic-dynamodb/dynamic_dynamodb/dynamic_dynamodb.py", line 365, in _get_provisioned_read_units
table = self.ddb_connection.get_table(self.table_name)
File "/usr/local/lib/python2.7/site-packages/boto/dynamodb/layer2.py", line 360, in get_table
response = self.layer1.describe_table(name)
File "/usr/local/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 217, in describe_table
return self.make_request('DescribeTable', json_input)
File "/usr/local/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 118, in make_request
retry_handler=self._retry_handler)
File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 845, in _mexe
status = retry_handler(response, i, next_sleep)
File "/usr/local/lib/python2.7/site-packages/boto/dynamodb/layer1.py", line 158, in _retry_handler
data)
boto.exception.DynamoDBResponseError: DynamoDBResponseError: 400 Bad Request
{u'message': u'Requested resource not found: Table: prod-sssions-201304 not found', u'__type': u'com.amazonaws.dynamodb.v20111205#ResourceNotFoundException'}
I just started dyn-dyn on a set of table that was manually configured to 2000-5000 CU per table (for a migration job). The actual usage now will be more in the 5-15 CU range.
dynamic-dynamodb fails on many of these tables as below:
2013-05-21 07:33:03,880 - dynamic-dynamodb - INFO - PROD_AccountRelation - Consumed read units: 9%
2013-05-21 07:33:03,921 - dynamic-dynamodb - INFO - PROD_AccountRelation - Consumed write units: 1%
2013-05-21 07:33:03,921 - dynamic-dynamodb - INFO - PROD_AccountRelation - Changing provisioning to 550 read units and 550 write units
2013-05-21 07:33:04,185 - dynamic-dynamodb - INFO - Provisioning updated
2013-05-21 07:33:04,345 - dynamic-dynamodb - INFO - PROD_PostrefsRepostLookup - Consumed read units: 1%
2013-05-21 07:33:04,480 - dynamic-dynamodb - INFO - PROD_PostrefsRepostLookup - Consumed write units: 2%
2013-05-21 07:33:04,480 - dynamic-dynamodb - INFO - PROD_PostrefsRepostLookup - Changing provisioning to 85 read units and 42 write units
2013-05-21 07:33:04,853 - dynamic-dynamodb - INFO - Provisioning updated
2013-05-21 07:33:04,957 - dynamic-dynamodb - INFO - PROD_ReactionUserRepost - Consumed read units: 7%
2013-05-21 07:33:05,007 - dynamic-dynamodb - INFO - PROD_ReactionUserRepost - Consumed write units: 4%
2013-05-21 07:33:05,007 - dynamic-dynamodb - INFO - PROD_ReactionUserRepost - Changing provisioning to 22 read units and 22 write units
2013-05-21 07:33:05,252 - dynamic-dynamodb - INFO - Provisioning updated
2013-05-21 07:33:05,305 - dynamic-dynamodb - INFO - PROD_OverallCounts - Consumed read units: 2%
2013-05-21 07:33:05,342 - dynamic-dynamodb - INFO - PROD_OverallCounts - Consumed write units: 2%
2013-05-21 07:33:05,342 - dynamic-dynamodb - INFO - PROD_OverallCounts - Changing provisioning to 33 read units and 33 write units
2013-05-21 07:33:05,370 - dynamic-dynamodb - WARNING - PROD_OverallCounts - Scaling limit exeeded. The table can only be scaled down twice per day.
2013-05-21 07:33:05,430 - dynamic-dynamodb - INFO - PROD_AccountList - Consumed read units: 1%
2013-05-21 07:33:05,479 - dynamic-dynamodb - INFO - PROD_AccountList - Consumed write units: 2%
2013-05-21 07:33:05,479 - dynamic-dynamodb - INFO - PROD_AccountList - Changing provisioning to 1300 read units and 33 write units
2013-05-21 07:33:05,506 - dynamic-dynamodb - WARNING - PROD_AccountList - Scaling limit exeeded. The table can only be scaled down twice per day.
2013-05-21 07:33:05,560 - dynamic-dynamodb - INFO - PROD_HashtagFanout - Consumed read units: 1%
2013-05-21 07:33:05,610 - dynamic-dynamodb - INFO - PROD_HashtagFanout - Consumed write units: 0%
2013-05-21 07:33:05,610 - dynamic-dynamodb - INFO - PROD_HashtagFanout - Scaling down writes is not done when usage is at 0%
2013-05-21 07:33:05,610 - dynamic-dynamodb - INFO - PROD_HashtagFanout - Changing provisioning to 650 read units and 100 write units
2013-05-21 07:33:05,637 - dynamic-dynamodb - WARNING - PROD_HashtagFanout - Scaling limit exeeded. The table can only be scaled down twice per day.
2013-05-21 07:33:05,697 - dynamic-dynamodb - INFO - PROD_PostrefsFeatured - Consumed read units: 1%
2013-05-21 07:33:05,789 - dynamic-dynamodb - INFO - PROD_PostrefsFeatured - Consumed write units: 0%
2013-05-21 07:33:05,789 - dynamic-dynamodb - INFO - PROD_PostrefsFeatured - Scaling down writes is not done when usage is at 0%
2013-05-21 07:33:05,789 - dynamic-dynamodb - INFO - PROD_PostrefsFeatured - Changing provisioning to 65 read units and 200 write units
2013-05-21 07:33:05,818 - dynamic-dynamodb - WARNING - PROD_PostrefsFeatured - Scaling limit exeeded. The table can only be scaled down twice per day.
2013-05-21 07:33:05,876 - dynamic-dynamodb - INFO - PROD_HashtagCounts - Consumed read units: 2%
2013-05-21 07:33:05,918 - dynamic-dynamodb - INFO - PROD_HashtagCounts - Consumed write units: 0%
2013-05-21 07:33:05,918 - dynamic-dynamodb - INFO - PROD_HashtagCounts - Scaling down writes is not done when usage is at 0%
2013-05-21 07:33:05,919 - dynamic-dynamodb - INFO - PROD_HashtagCounts - Changing provisioning to 33 read units and 50 write units
2013-05-21 07:33:05,947 - dynamic-dynamodb - WARNING - PROD_HashtagCounts - Scaling limit exeeded. The table can only be scaled down twice per day.
2013-05-21 07:33:06,021 - dynamic-dynamodb - INFO - PROD_HashtagSubscription - Consumed read units: 0%
2013-05-21 07:33:06,021 - dynamic-dynamodb - INFO - PROD_HashtagSubscription - Scaling down reads is not done when usage is at 0%
2013-05-21 07:33:06,083 - dynamic-dynamodb - INFO - PROD_HashtagSubscription - Consumed write units: 0%
2013-05-21 07:33:06,084 - dynamic-dynamodb - INFO - PROD_HashtagSubscription - Scaling down writes is not done when usage is at 0%
2013-05-21 07:33:06,084 - dynamic-dynamodb - INFO - PROD_HashtagSubscription - No need to change provisioning
2013-05-21 07:33:06,126 - dynamic-dynamodb - INFO - PROD_AccountCounts - Consumed read units: 8%
2013-05-21 07:33:06,171 - dynamic-dynamodb - INFO - PROD_AccountCounts - Consumed write units: 12%
2013-05-21 07:33:06,171 - dynamic-dynamodb - INFO - PROD_AccountCounts - Changing provisioning to 163 read units and 33 write units
2013-05-21 07:33:06,195 - dynamic-dynamodb - WARNING - PROD_AccountCounts - Scaling limit exeeded. The table can only be scaled down twice per day.
2013-05-21 07:33:06,277 - dynamic-dynamodb - INFO - PROD_PostrefsByUserWithoutReposts - Consumed read units: 1%
2013-05-21 07:33:06,323 - dynamic-dynamodb - INFO - PROD_PostrefsByUserWithoutReposts - Consumed write units: 1%
2013-05-21 07:33:06,324 - dynamic-dynamodb - INFO - PROD_PostrefsByUserWithoutReposts - Changing provisioning to 65 read units and 325 write units
2013-05-21 07:33:06,347 - dynamic-dynamodb - WARNING - PROD_PostrefsByUserWithoutReposts - Scaling limit exeeded. The table can only be scaled down twice per day.
The error message is wrong, I was able to size the CUs down manually in the AWS console (and am 100% sure that nobody had touched provisioning yet).
Config File:
# How often should Dynamic DynamoDB monitor changes (in seconds)
check-interval: 300
# Circuit breaker configuration
# No provisioning updates will be made unless this URL returns
# a HTTP 200 OK status code
#circuit-breaker-url: http://my.service.com/v1/is_up
#circuit-breaker-timeout: 500
[logging]
# Log level [debug|info|warning|error]
log-level: info
# Log file (comment out to get only console output)
log-file: /var/log/dynamic-dynamodb.log
# -------------------------------------------------------------------------------
[table: PROD_PostrefsForUser]
# -------------------------------------------------------------------------------
# Units to increase or decrease reads with, must be either percent or units
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 300
max-provisioned-reads: 5000
min-provisioned-writes: 200
max-provisioned-writes: 3000
[table: PROD_AccountCounts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_AccountList]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_AccountRelation]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_HashIsAccount]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_HashKnownByAccount]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_HashtagCounts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_HashtagFanout]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_HashtagSubscription]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_OverallCounts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostInfo]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostReaction]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsByUserTextComments]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsByUserWithReposts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsByUserWithoutReposts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsFeatured]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsForUser]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsPingPong]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsRepostLookup]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_PostrefsThread]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_Posts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_ReactionCounts]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_ReactionUserAbuseReport]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_ReactionUserDislike]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_ReactionUserLike]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_ReactionUserRepost]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
[table: PROD_RescrapeSpool]
# -------------------------------------------------------------------------------
increase-reads-unit: percent
decrease-reads-unit: percent
increase-writes-unit: percent
decrease-writes-unit: percent
# Thresholds for scaling up or down the provisioning
reads-upper-threshold: 90
reads-lower-threshold: 40
writes-upper-threshold: 90
writes-lower-threshold: 30
# How many percent should Dynamic DynamoDB increase/decrease provisioning with (%)
increase-reads-with: 30
decrease-reads-with: 35
increase-writes-with: 30
decrease-writes-with: 35
# Dynamic DynamoDB will not provision any more or less Provisioning than this
min-provisioned-reads: 20
max-provisioned-reads: 5000
min-provisioned-writes: 10
max-provisioned-writes: 3000
# Maintenance windows (in UTC)
#maintenance-windows: 22:00-23:59,00:00-06:00
# Other settings
# Allow down scaling when at 0% consumed reads
#allow-scaling-down-reads-on-0-percent: true
#allow-scaling-down-writes-on-0-percent: true
# Restric scale down to only happend when BOTH reads AND writes are in need
# of scaling down. Set this to "true" to minimize down scaling.
#always-decrease-rw-together: true
We are currently using dynamic-dynamo in our production servers, and due the behaviour of our systems, dynamic-dynamo is continuously changing the provisioning resources asigned to our DynamoTables.
However, we are seeing something really weird: the configured cloudwatch alarm limit also changes when the provisioned throughput is changed.
For example, we have a table provisioned originally with 100 reads, with an alarm configured with a 80% of the capacity. This table has been updated up and down for some days, and now the alarm is configured to the 27%.
We don't know yet if this is an issue from AWS or from dynamic dynamo, but in both cases it would be really nice if Dynamic Dynamo could set the alarm in the original status.
Any service can go down for whatever reason. In this case, DynamoDB Tables might no longer get any traffic. If an outage lasts long enough, provisioning might scale down to the min-provisioned-values. Since it makes sense to increase provisioning in smaller steps (since upscale-requests are not limited), it might take a long time after such an outage to get the table provisioning back to the required "normal" level.
Idea: Allow configuring a URL and max-response-time that is checked every check-interval before the dynamic-dynamo does its magic.
This might be a simple api endpoint that returns 200 if the service is up. If it responds, the script does its magic. If it does not return 200 or responds too slow, dynamic-dynamodb should stop downscaling the provisioning to allow speedy retrun to production throughput after an outage.
Handle DynamoDB exceptions when updating throughput.
Hi,
currently when dynamic-dynamodb starts in daemon mode, it creates a pid file on this location:
daemon = DynamicDynamoDBDaemon('/tmp/dynamic-dynamodb.pid')
Since this is harcoded, it is not possible to run multiple instances od dynamic-dynamodb.
Regards,
It should be possible to configure logging more in detail. E.g. logging to file should be possible as well as log level etc.
ubuntu@helo:~$ sudo dynamic-dynamodb -c PROD_AccountCounts.conf
2013-04-11 07:34:45,617 - dynamic-dynamodb - INFO - Consumed reads: 34
2013-04-11 07:34:45,630 - dynamic-dynamodb - INFO - Provisioned reads: 200
2013-04-11 07:34:45,630 - dynamic-dynamodb - INFO - Consumed read capacity: 17%
2013-04-11 07:34:45,630 - dynamic-dynamodb - INFO - Reads lower threshold: 30%
2013-04-11 07:34:45,630 - dynamic-dynamodb - INFO - Reads upper threshold: 80%
2013-04-11 07:34:45,666 - dynamic-dynamodb - INFO - Consumed writes: 7
2013-04-11 07:34:45,680 - dynamic-dynamodb - INFO - Provisioned writes: 18
2013-04-11 07:34:45,680 - dynamic-dynamodb - INFO - Consumed write capacity: 39%
2013-04-11 07:34:45,681 - dynamic-dynamodb - INFO - Writes lower threshold: 30%
2013-04-11 07:34:45,681 - dynamic-dynamodb - INFO - Writes upper threshold: 95%
2013-04-11 07:34:45,720 - dynamic-dynamodb - INFO - Reached provisioned reads min limit: 200
2013-04-11 07:34:45,720 - dynamic-dynamodb - INFO - Changing provisioning to 200 reads and 18 writes
Traceback (most recent call last):
File "/usr/local/bin/dynamic-dynamodb", line 7, in <module>
main.main()
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/main.py", line 229, in main
dynamic_ddb.run()
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 146, in run
self.ensure_proper_provisioning()
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 210, in ensure_proper_provisioning
throughput['write_units'])
File "/usr/local/lib/python2.6/dist-packages/dynamic_dynamodb/dynamic_dynamodb.py", line 483, in _update_throughput
table.update_throughput(int(read_units), int(write_units))
File "/usr/local/lib/python2.6/dist-packages/boto/dynamodb/table.py", line 245, in update_throughput
self.layer2.update_throughput(self, read_units, write_units)
File "/usr/local/lib/python2.6/dist-packages/boto/dynamodb/layer2.py", line 405, in update_throughput
'WriteCapacityUnits': write_units})
File "/usr/local/lib/python2.6/dist-packages/boto/dynamodb/layer1.py", line 261, in update_table
return self.make_request('UpdateTable', json_input)
File "/usr/local/lib/python2.6/dist-packages/boto/dynamodb/layer1.py", line 118, in make_request
retry_handler=self._retry_handler)
File "/usr/local/lib/python2.6/dist-packages/boto/connection.py", line 845, in _mexe
status = retry_handler(response, i, next_sleep)
File "/usr/local/lib/python2.6/dist-packages/boto/dynamodb/layer1.py", line 155, in _retry_handler
response.status, response.reason, data)
boto.dynamodb.exceptions.DynamoDBValidationError: DynamoDBValidationError: 400 Bad Request
{'message': 'The provisioned throughput for the table will not change. The requested value equals the current value. Current ReadCapacityUnits provisioned for the table: 200. Requested ReadCapacityUnits: 200. Current WriteCapacityUnits provisioned for the table: 18. Requested WriteCapacityUnits: 18. Refer to the Amazon DynamoDB Developer Guide for current limits and how to request higher limits.', '__type': 'com.amazon.coral.validate#ValidationException'}
ubuntu@helo:~$
Increase in percent does not add to current provisioning properly
It would be awesome if one could "queue" a table's provisioning decreases until decreases for both reads AND writes would be due.
Background: One can only send two downscaling requests per day -- not two for reads and two for writes. Optionally downscaling reads and writes simultaneously would increase efficiency of using this limitation.
If downscaling provisioning fails since two changes where made on the given day already, upscaling isn't executed either.
Example:
Result: The Bump to 200 Write-CUs is not happening since the downscaling of the Read-CUs failed.
Expected behavior: The increase in provisioned capacity should be executed nevertheless.
This is dangerous bug, since dyn-dyn stops scaling up again even though upscaling is ALWAYS possible.
Dynamic DynamoDB is only scaling down when consumed capacity > 0%. That has been implemented as a security enhancement, but in some cases (where the table really has low traffic) it makes sense to scale down at 0%.
Create an option to make 0% down scaling possible.
It would be great if Dynamic DynamoDB could handle more than one table at a time.
Add support for regular Python configuration files. Here's a suggested configuration:
[global]
#aws-access-key-id: AWS_ACCESS_KEY
#aws-secret-access-key-id: AWS_SECRET_KEY
region: us-east-1
check-interval: 300
[table: my-table]
# Read provisioning configuration
reads-upper-threshold: 90
reads-lower-threshold: 30
increase-reads-with: 50
decrease-reads-with: 50
#min-provisioned-reads: 100
#max-provisioned-reads: 500
# Write provisioning configuration
writes-upper-threshold: 90
writes-lower-threshold: 30
increase-writes-with: 50
decrease-writes-with: 50
#min-provisioned-writes: 100
#max-provisioned-writes: 500
It would be fantastic if Dynamic DynamoDB could handle more than one table at a time. Here's a suggested configuration:
[global]
#aws-access-key-id: AWS_ACCESS_KEY
#aws-secret-access-key-id: AWS_SECRET_KEY
region: us-east-1
check-interval: 300
[table: my-table]
# Read provisioning configuration
reads-upper-threshold: 90
reads-lower-threshold: 30
increase-reads-with: 50
decrease-reads-with: 50
#min-provisioned-reads: 100
#max-provisioned-reads: 500
# Write provisioning configuration
writes-upper-threshold: 90
writes-lower-threshold: 30
increase-writes-with: 50
decrease-writes-with: 50
#min-provisioned-writes: 100
#max-provisioned-writes: 500
It would then be possible to define more than 1 [table: x]
section and Dynamic DynamoDB could iterate over those sections.
Improper scaling under certain circumstances results in bad read/write provisioning.
Support for daemonizing Dynamic DynamoDB
Even if "allow-scaling-down-reads-on-0-percent: true" is in the config file, dyndyn does not do so:
2013-05-17 23:35:31,866 - dynamic-dynamodb - INFO - PROD_PostrefsByUserTextComments - Scaling down reads is not done when usage is at 0%
Update --help
docs
Update config file docs
Add --version
flag to dynamic-dynamodb command
dynamic-dynamodb --version
Calling the above from a terminal results in:
Traceback (most recent call last):
File "/usr/local/bin/dynamic-dynamodb", line 22, in <module>
import dynamic_dynamodb
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/__init__.py", line 26, in <module>
import core
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/__init__.py", line 6, in <module>
import dynamodb
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/core/dynamodb.py", line 7, in <module>
from dynamic_dynamodb.log_handler import LOGGER as logger
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/log_handler.py", line 23, in <module>
import config_handler
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/config_handler.py", line 4, in <module>
CONFIGURATION = config.get_configuration()
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/config/__init__.py", line 63, in get_configuration
cmd_line_options = command_line_parser.parse()
File "/usr/local/lib/python2.7/dist-packages/dynamic_dynamodb/config/command_line_parser.py", line 114, in parse
print 'Dynamic DynamoDB version: {0}'.format(dynamic_dynamodb.version())
AttributeError: 'module' object has no attribute 'version'
Not a terrible huge issue, just a little inconvenient...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.