Comments (9)
clarkie
commented on May 26, 2024
2
I've found some time to have another look at this. It looks like the scan is limiting the number of items scanned before evaluating the filter expression rather than the other way round. The AWS docs state:
In a response, DynamoDB returns all the matching results within the scope of the Limit value. For example, if you issue a Query or a Scan request with a Limit value of 6 and without a filter expression, DynamoDB returns the first six items in the table that match the specified key conditions in the request (or just the first six items in the case of a Scan with no filter). If you also supply a FilterExpression value, DynamoDB will return the items in the first six that also match the filter requirements (the number of results returned will be less than or equal to 6).
From my tests above I've logged the params sent to dynamodb and the response data:
params: { ExpressionAttributeNames: { '#email': 'email', '#age': 'age' },
ExpressionAttributeValues: { ':email': 'test1', ':age': 18, ':age_2': 22 },
FilterExpression: '(#age BETWEEN :age AND :age_2) AND (begins_with(#email, :email))',
TableName: 'vogels-int-test-users' }
data: { Items:
[ { name: 'Test 0',
id: 'userid-12',
acceptedTerms: false,
email: '[email protected]',
age: 22,
roles: [Object] },
{ name: 'Test 2',
id: 'userid-11',
acceptedTerms: false,
email: '[email protected]',
age: 21,
roles: [Object] },
{ name: 'Test 1',
id: 'userid-10',
acceptedTerms: false,
email: '[email protected]',
age: 20,
roles: [Object] } ],
Count: 3,
ScannedCount: 227 }
✓ should return all users with age between 18 and 22 and with email beginning with test1
params: { ExpressionAttributeNames: { '#email': 'email', '#age': 'age' },
ExpressionAttributeValues: { ':email': 'test1', ':age': 18, ':age_2': 22 },
FilterExpression: '(#age BETWEEN :age AND :age_2) AND (begins_with(#email, :email))',
Limit: 1,
TableName: 'vogels-int-test-users' }
data: { Items: [],
Count: 0,
ScannedCount: 1,
LastEvaluatedKey: { id: '458f4273-28b8-4a9f-80ee-7535cfa5877e' } }
3) should return 1 user with age between 18 and 22 and with email beginning with test1
You can see in the second test that the ScannedCount is 1 which is the limit value.
Another example, this time with .limit(10):
{ ExpressionAttributeNames: { '#email': 'email', '#age': 'age' },
ExpressionAttributeValues: { ':email': 'test1', ':age': 18, ':age_2': 22 },
FilterExpression: '(#age BETWEEN :age AND :age_2) AND (begins_with(#email, :email))',
Limit: 10,
TableName: 'vogels-int-test-users' }
{ Items: [],
Count: 0,
ScannedCount: 10,
LastEvaluatedKey: { id: '0dfdc4be-1289-45e9-a67f-10e82bfcd98e' } }
@Alex0007 can you check my logic and understanding of the docs before I raise an issue on https://github.com/aws/aws-sdk-js Thanks
from dynogels.
clarkie
commented on May 26, 2024
2
Actually. It looks like there's some confusion over what the docs say and this may be expected behaviour: http://stackoverflow.com/questions/38778506/why-dynamodb-scan-with-limit-and-filterexpression-not-return-the-items-that-matc
from dynogels.
clarkie
commented on May 26, 2024
2
Yeah, I think so. I'll update the docs to make it a little clearer.
from dynogels.
clarkie
commented on May 26, 2024
1
I've created a couple of passing and breaking tests: https://github.com/clarkie/dynogels/compare/limit-1-bug
The results (interspersed with the console.logs of the array lengths)
91
✓ should return all users with age between 18 and 22
1
✓ should return 1 user with age between 18 and 22
24
✓ should return all users with email beginning with test1
0
1) should return 1 user with email beginning with test1
3
✓ should return all users with age between 18 and 22 and with email beginning with test1
0
2) should return 1 user with age between 18 and 22 and with email beginning with test1
So it looks like it's an issue with a subset of the where clauses when combined with limit
from dynogels.
clarkie
commented on May 26, 2024
1
In fact, in the travis logs it shows as them all failing (https://travis-ci.org/clarkie/dynogels/jobs/147401027):
9
✓ should return all users with age between 18 and 22
0
1) should return 1 user with age between 18 and 22
7
✓ should return all users with email beginning with test1
0
2) should return 1 user with email beginning with test1
3
✓ should return all users with age between 18 and 22 and with email beginning with test1
0
3) should return 1 user with age between 18 and 22 and with email beginning with test1
from dynogels.
clarkie
commented on May 26, 2024
Are you saying that you're not getting a single item back?
Are you calling .exec()?
Can you share your schema?
from dynogels.
Alex0007
commented on May 26, 2024
Schema
import { location } from './common'
var schema = {
id: vogels.types.uuid(),
active: Joi.boolean().default(true),
user: Joi.string().guid().required(),
type: Joi.string().valid('billing', 'pickup'),
address: location
}Are you saying that you're not getting a single item back?
Correct. I'm calling exec((err, data) => {}) and receive no items back. When i remove .limit(1) call i get correct items
from dynogels.
clarkie
commented on May 26, 2024
Hmm, that's interesting. Can you put together a failing test case and I can start to look into it.
Thanks
from dynogels.
Alex0007
commented on May 26, 2024
So the behavior is expected and nothing to fix here? C:
from dynogels.
Related Issues (20)
- Does not enforce required() on nested property HOT 6
- What is the best way to convert a list of objects HOT 1
- Provide access to the AWS request object
- Model.validate not in latest release HOT 1
- Add transactions support HOT 1
- A method which returns the raw DynamoDB params for creating an item
- When I do an .updateAsync() with ReturnValues I recieve a type error
- Item.set doesn't work as advertised HOT 3
- Security question?
- while updating the model getting validationException HOT 1
- Project is dead? HOT 27
- using reserved word in schema causes silent error
- Enable On-Demand Billing Mode
- Cannot use "startKey" when using an index HOT 1
- 'updatedAt' field doesn't change once set.
- ValidationException on .destroy() method HOT 3
- Update joi dependency to use @hapi/joi HOT 6
- Keep getting Error: Invalid schema content when I use define HOT 3
- Stale library? HOT 1
- Is it possible to run dynogels with version 3 of AWS
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dynogels.