Comments (16)
karlfreeman
commented on May 27, 2024
I'm using it roughly like so:
dynamodb_query = dynamodb_table.batch_get(:all, feed)
unless dynamodb_query.nil?
dynamodb_query.each { | activity |
parse_activity(activity)
}
end
from aws-sdk-ruby.
lsegal
commented on May 27, 2024
Can you provide a backtrace for the above exception?
from aws-sdk-ruby.
karlfreeman
commented on May 27, 2024
Sure, not much to the back trace though. Thanks for the quick reply!
gems/aws-sdk-1.8.0/lib/aws/dynamo_db/batch_get.rb:150 • each
gems/aws-sdk-1.8.0/lib/aws/dynamo_db/batch_get.rb:169 • each_attributes
/usr/local/api/releases/20130124112142/lib/unmagnify/me/activity/index.rb:51 • each
/usr/local/api/releases/20130124112142/lib/unmagnify/me/activity/index.rb:51 • response
bundler/gems/goliath-ea70864f86a9/lib/goliath/api.rb:227 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/simple_aroundware_factory.rb:77 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/validation/default_params.rb:39 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/validation/default_params.rb:39 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/params.rb:66 • block in call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/validator.rb:40 • safely
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/params.rb:64 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/default_mime_type.rb:26 • call
/usr/local/api/releases/20130124112142/lib/unmagnify/middleware/path_params.rb:18 • block in call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/validator.rb:40 • safely
/usr/local/api/releases/20130124112142/lib/unmagnify/middleware/path_params.rb:15 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/validation/request_method.rb:29 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
/usr/local/api/releases/20130124112142/lib/unmagnify/middleware/cors.rb:23 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
gems/rack-1.4.3/lib/rack/content_length.rb:14 • call
gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:114 • call
gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:91 • block in new
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/builder.rb:56 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/builder.rb:56 • block (3 levels) in build
gems/http_router-0.9.7/lib/http_router.rb:165 • call
gems/http_router-0.9.7/lib/http_router.rb:165 • process_destination_path
(eval):178 • block (2 levels) in inject_root_methods
(eval):168 • catch
(eval):168 • block in inject_root_methods
(eval):107 • block in inject_root_methods
gems/http_router-0.9.7/lib/http_router/node/root.rb:68 • []
gems/http_router-0.9.7/lib/http_router.rb:118 • block in call
gems/http_router-0.9.7/lib/http_router.rb:118 • catch
gems/http_router-0.9.7/lib/http_router.rb:118 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
/usr/local/api/releases/20130124112142/lib/unmagnify/middleware/cors.rb:23 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/favicon.rb:26 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/rack/async_middleware.rb:73 • call
gems/rack-1.4.3/lib/rack/content_length.rb:14 • call
gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:114 • call
gems/async-rack-0.5.1/lib/async_rack/async_callback.rb:91 • block in new
bundler/gems/goliath-ea70864f86a9/lib/goliath/request.rb:163 • call
bundler/gems/goliath-ea70864f86a9/lib/goliath/request.rb:163 • block in process
from aws-sdk-ruby.
karlfreeman
commented on May 27, 2024
I thought perhaps #161 might fix this, however its close but no cigar. Still having
NoMethodError: undefined method 'each_pair' for nil:NilClass
at https://github.com/aws/aws-sdk-ruby/blob/master/lib/aws/dynamo_db/batch_get.rb#L150 sporadically.
from aws-sdk-ruby.
jmclachlan
commented on May 27, 2024
I wonder if this is all related. I randomly get this error quite a bit when using DynoDB with threads.
undefined method tables' for <AWS::DynamoDB>:AWS::DynamoDB /task/__gems__/gems/aws-sdk-1.8.5/lib/aws/record/hash_model.rb:86:indynamo_db_table'
/task/gems/gems/aws-sdk-1.8.5/lib/aws/record/hash_model/finder_methods.rb:28:in find_by_id' /task/__gems__/gems/aws-sdk-1.8.5/lib/aws/record/scope.rb:97:infind'
/task/gems/gems/aws-sdk-1.8.5/lib/aws/record/hash_model/finder_methods.rb:77:in `find'
from aws-sdk-ruby.
karlfreeman
commented on May 27, 2024
Looking at that stack trace it doesn't appear to be related. However don't hold me to it 😉
from aws-sdk-ruby.
trevorrowe
commented on May 27, 2024
@jmclachlan Are you calling AWS.eager_autoload! before you start your concurrency? The SDK makes heavy use of autoload, which is not thread-safe. The AWS.eager_autoload! method forcibly loads all library files across the SDK (this can be slow when they are not cached). This generally clears up threading issues though.
from aws-sdk-ruby.
jmclachlan
commented on May 27, 2024
Nope. You are right. I think this is just a more general issue with the base client initialization and threads. My bad.
from aws-sdk-ruby.
jmclachlan
commented on May 27, 2024
@trevorrowe Awesome. That fixes everything. Thanks a bunch.
from aws-sdk-ruby.
karlfreeman
commented on May 27, 2024
@trevorrowe Is there anything more I can do to help figure out why these nil's keep cropping in response.data['Responses']
from aws-sdk-ruby.
trevorrowe
commented on May 27, 2024
I can not verify this yet (I will though), but it looks like DynamoDB is not returning the data at 'Responses'. My guess is the response is returned with no data, and ALL of the request items are returned as unprocessed keys. If this is the case, the fix is a one-liner ((response.data['Responses'] || {}).each_pair).
from aws-sdk-ruby.
trevorrowe
commented on May 27, 2024
I should also add, this could be happening if you are getting throttled.
from aws-sdk-ruby.
karlfreeman
commented on May 27, 2024
There's a good chance that we're getting throttled every now and again so that could be the cause.
from aws-sdk-ruby.
karlfreeman
commented on May 27, 2024
I'll also add I've just seen this happen on response.data["Items"] too
from aws-sdk-ruby.
trevorrowe
commented on May 27, 2024
@karlfreeman Sorry for the long absence on this issue. In both cases, it would be possible to coerce the nil value to an empty array. I'm afraid by doing this, we could mask some other issue.
If you are able to reproduce this issue somewhat reliably it would be SUPER helpful if you could enable http wire logging. This would dump the entire http response body (that is getting JSON parsed) and would answer the question if the key is missing from the source, or if there is some issue in the SDK causing this buggy behavior.
You can enable http wire logging like so:
# send the log output somewhere more useful than stdout, like a file
require 'logger'
AWS.config(:http_wire_trace => true, :logger => Logger.new($stdout))You can reduce the scope of what requests get wire-logged by creating a ddb instance that logs instead:
ddb_without_wire_trace = AWS::DynamoDB.new
ddb_with_wire_trace = AWS::DynamoDB.new(:http_wire_trace => true, :logger => Logger.new(...))Thanks!
from aws-sdk-ruby.
lsegal
commented on May 27, 2024
I'm going to close this issue since there has been no feedback in the last few months. @karlfreeman, if you are still having problems with DynamoDB in your application, please follow @trevorrowe's instructions and provide some more detailed logging of your requests so we can narrow down the problem. Thanks!
from aws-sdk-ruby.
Related Issues (20)
- SDK throw ArgumentError when the "workgroup_name" input is valid in Aws::RedshiftDataAPIService::Client HOT 3
- [SES] Custom verification email template no redirect HOT 9
- SQS QueuePoller sometimes returns dupe messages, raises BatchEntryIdsNotDistinct HOT 1
- DynamoDB #query method returns response with items = nil HOT 18
- Aws::S3::PresignedPost does not support tagging field HOT 3
- Multipart copy for Glacier
- CognitoIdentityCredentials is not including the logins set when getting credentials, making all requests unauthenticated HOT 5
- `Aws::BedrockRuntime::Types::<X>` doesn't have `event_type=` HOT 3
- Listing shared RDS Clusters raises an ArgumentError HOT 6
- Errors downloading files from s3 are propagated to incorrect thread HOT 8
- aws-sdk-sqs raises NoMethodError when messages not found HOT 14
- Generate RBS HOT 15
- aws-partitions 1.856.0 no longer be found in that source HOT 8
- aws-sigv4/eventstream released a minor version bump after removing support for Ruby 2.3/2.4 HOT 5
- [aws-sdk-sqs] Aws::SQS::Types::SendMessageBatchResult#failed returns nil HOT 8
- aws-sdk-sesv2 API :next_token Symbol key literal doesn't work as expected (but String key does) HOT 4
- S3 Control Client - Unable to Parse URI HOT 6
- AWS Bedrock ReadTimeout NetworkingError TCPSocket closed HOT 3
- Tagging resources seems to be broken HOT 6
- Aws::Errors::MissingCredentialsError exception in ECS when using TaskRoleArn HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-sdk-ruby.