Comments (26)
bytekast
commented on July 21, 2024
1
awesome. thanks!
from dynamodb-onetable.
bytekast
commented on July 21, 2024
1
looks like it worked! appreciate it!
from dynamodb-onetable.
mobsense
commented on July 21, 2024
1
The log: true reveals ALL ;-)
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Can you please simplify down your samples to be stand-alone. I'd like to get a code snippet that we can run.
The samples/typescript/src/index.ts has a sample.
let from = new Date()
from.setMonth(from.getMonth() - 1)
let invoices = await Invoice.find({}, {
tunnel: { between: {
gs1sk: [`invoice#${from.toISOString()}`, `invoice#${new Date().toISOString()}`],
} },
index: 'gs1',
follow: true,
})
from dynamodb-onetable.
bytekast
commented on July 21, 2024
sure. here's a simple version:
const driverId = '12345'
const tunnel = { begins: { sk: 'MyEvent#MyModel#MyOperation#2021' } }
const results = await EventModel.find({ driverId }, { tunnel, index: 'gs2' })
Here's the error:
{"errorType":"TypeError","errorMessage":".for is not iterable","stack":["TypeError: .for is not iterable"," at Xt.tunnelProperties (/var/node_modules/dynamodb-onetable/dist/mjs/Model.js:841:29)"," at Xt.transformProperties (/var/node_modules/dynamodb-onetable/dist/mjs/Model.js:817:14)"," at Xt.prepareProperties (/var/node_modules/dynamodb-onetable/dist/mjs/Model.js:770:24)"," at Xt.queryItems (/var/node_modules/dynamodb-onetable/dist/mjs/Model.js:636:27)"," at Xt.find (/var/node_modules/dynamodb-onetable/dist/mjs/Model.js:528:27)"," at sO (/var/src/db.ts:84:36)"," at oO (/var/src/resolver.ts:7:38)"," at Runtime.handler (/opt/nodejs/node_modules/@sentry/serverless/src/awslambda.ts:274:18)"," at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"]}
from dynamodb-onetable.
bytekast
commented on July 21, 2024
same error here:
const driverId = '12345'
const tunnel = { begins: { sk: 'MyEvent#MyModel#MyOperation#2021' } }
const results = await table.queryItems({ driverId }, { tunnel, index: 'gs2' })
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Thank you. Found the issue. We'll patch this.
The fix is to change Object.values to Object.entries
- for (let [kind, settings] of Object.values(params.tunnel)) {
+ for (let [kind, settings] of Object.entries(params.tunnel)) {
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Fixed in 1.5.5
from dynamodb-onetable.
bytekast
commented on July 21, 2024
Hi @mobsense - it worked with the examples above.
However, when it is a composite primary key, it doesn't seem to work.
const market = 'denver'
const driverId = '12345'
const tunnel = { begins: { sk: 'MyEvent#MyModel#MyOperation#2021' } }
const results = await table.queryItems({ market, driverId }, { tunnel, index: 'gs1' })
No errors but not getting the results. Doing the "manual" document client query returns the correct results.
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Can you use params.log == true with a logger as described in the README and post the actual dynamodb query.
Also please post your schema. Hard to know what is not working right without seeing the schema and the actual query command being generated.
from dynamodb-onetable.
bytekast
commented on July 21, 2024
I think I know what's going on here....
const tunnel = prefixFilter ? { begins: { sk: prefixFilter } } : null
const results = await EventModel.find({ model, modelId }, { tunnel, index: 'gs1', reverse: true, log: true })
model is an attribute used both in the partition key and the sort key. when tunnel is null, the sort key gets auto populated with the model value even though that isn't the intended behavior.
here's the trace:
1626412031269 2021-07-16T05:07:11.268Z 47a243ed-af3d-4995-bd5f-6432fec89ada INFO info Dynamo "find" "Event" {
"trace": {
"cmd": {
"ExpressionAttributeNames": {
"#_0": "sk",
"#_1": "model",
"#_2": "modelId",
"#_3": "gs1pk",
"#_4": "_type"
},
"ExpressionAttributeValues": {
":_0": {
"S": "#route"
},
":_1": {
"S": "route"
},
":_2": {
"S": "Yi39uLPSRgNjymCcH"
},
":_3": {
"S": "model#route#Yi39uLPSRgNjymCcH"
},
":_4": {
"S": "Event"
}
},
"FilterExpression": "(#_1 = :_1) and (#_2 = :_2) and (#_4 = :_4)",
"KeyConditionExpression": "begins_with(#_0, :_0) and #_3 = :_3",
"TableName": "argus",
"ConsistentRead": false,
"IndexName": "gs1",
"ScanIndexForward": false
},
"op": "find",
"properties": {
"sk": {
"begins": "#route"
},
"model": "route",
"modelId": "Yi39uLPSRgNjymCcH",
"gs1pk": "model#route#Yi39uLPSRgNjymCcH",
"_type": "Event"
},
"params": {
"parse": true,
"high": true,
"index": "gs1",
"reverse": true,
"log": true,
"checked": true
}
}
}
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Can you post your schema or redacted portions?
from dynamodb-onetable.
bytekast
commented on July 21, 2024
const Schema = {
indexes: {
primary: { hash: 'pk', sort: 'sk' },
gs1: { hash: 'gs1pk', sort: 'sk' },
gs2: { hash: 'gs2pk', sort: 'sk' },
},
models: {
Event: {
pk: { type: String, value: 'event#${id}' },
sk: { type: String, value: '${detail-type}#${model}#${operation}#${time}' }, // Allows fast filter with begins_with
id: { type: String, required: true },
'detail-type': { type: String, required: true },
detail: { type: Object, required: true },
source: { type: String, required: true },
time: { type: String, required: true },
// Optional time to live
ttl: { type: Number, required: false },
// Flattened from event.detail
model: { type: String, required: true },
operation: { type: String, required: true },
// GSI - Search events by model + id. Eg Route#123, Order#abc, etc.
gs1pk: { type: String, value: 'model#${model}#${modelId}' },
// GSI - Search events by driverId when driver is not the event model
gs2pk: { type: String, value: 'driver#${driverId}' },
},
},
}
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Thanks. What is prefixFilter, model and modelId set to when you call the API.
from dynamodb-onetable.
bytekast
commented on July 21, 2024
model = route
modelId = Yi39uLPSRgNjymCcH
prefixFilter = null
from dynamodb-onetable.
mobsense
commented on July 21, 2024
So tunnel is null?
from dynamodb-onetable.
bytekast
commented on July 21, 2024
yep.
from dynamodb-onetable.
bytekast
commented on July 21, 2024
if I provide a valid prefixFilter it works fine
when omitted, it looks like onetable automatically adds a "begins": "#route" sk, which is why the query is return zero matches
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Okay, I think we need better doc on this one.
If you do not provide the variables to complete the sort key, i.e. some of the ${var} references are unresolved, then OneTable will take the resolved portion and implicitly use a begins with on that portion.
from dynamodb-onetable.
mobsense
commented on July 21, 2024
You are not providing: ${operation}#${time}
from dynamodb-onetable.
bytekast
commented on July 21, 2024
but i don't want to provide those prefixes. i want the query to return all of the items without filtering the sort key
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Do you want ZERO SK? i.e. only the PK?
from dynamodb-onetable.
bytekast
commented on July 21, 2024
yep. so if prefixFilter is null, don't add a sort key
from dynamodb-onetable.
mobsense
commented on July 21, 2024
When you use value templates, the PK and SK are created for you from the properties you provide. If you provide an explicit PK or SK, they are used instead.
I'm doing this from memory, so forgive me if I'm wrong here. But ...
If the sk property is undefined in the properties, the value template will be used. If it is null, then the value template will not be used. Let me confirm this.
So I think you need to include:
Model.find({sk: null, ....
Let me check this.
from dynamodb-onetable.
mobsense
commented on July 21, 2024
Just confirmed that. Set sk to null.
Normally users at least want a sk prefix of some kind. In single table / multi-tenant design, the PK may be the account ID and the leading SK prefix is used to differentiate between various entities.
from dynamodb-onetable.
bytekast
commented on July 21, 2024
ok, will try that. thanks!
from dynamodb-onetable.
Related Issues (20)
- Error on pagination query HOT 3
- count: true param is not returning count value HOT 7
- BatchWrite does not handle Unprocessed Items correctly and ignores them HOT 1
- "?boolean" in Table.d.ts triggers typescript build error HOT 2
- model.upsert({ ...data }) returns a data with encrypted values HOT 2
- Need release to resolve Typescript ESM Resolution issues HOT 2
- params.fields are not respected when using table.getItem() or table.queryItems()
- Metrics.d.ts should not include function interfaces declared as 'async' HOT 7
- Improve: Prohibit the removal of nested object required fields. HOT 1
- Feature: Across entities query HOT 3
- Empty strings are filtered out HOT 2
- Cannot resolve type when ESM HOT 3
- Compound global index not updated if only 1 of 2 properties passed to the update function HOT 1
- Updating unique template field does not remove unique record HOT 2
- "CredentialsProviderError: The SSO session associated with this profile has expired" in "overview" sample... HOT 2
- Create method does not create the sort key if one attribute is undefined HOT 4
- Connecting OneTable to a local dynamodb instance
- Support for Mapped Types HOT 3
- Please Help: Update Problem (referenced attribute) HOT 3
- OneSchema type for getCurrentSchema() not set HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dynamodb-onetable.