Comments (10)
ABusyHippie
commented on June 8, 2024
^I second this
from nft-api.
ABusyHippie
commented on June 8, 2024
@chanansh I found this documentation on calling API's with pagination and offset more generally that helped by allowing me to offset by the current page limit I'm getting of 500, but since I'm working with a data set of 10,000 it's still impractical to do this long term and would like to see a solution.
However in the meantime this may help you too: https://stackoverflow.com/questions/18314687/how-to-implement-cursors-for-pagination-in-an-api
from nft-api.
chanansh
commented on June 8, 2024
I just called the api with the cursor=previous_result['cursor] and if cursor is an empty string the stop. However, this method does not allow threading
from nft-api.
ABusyHippie
commented on June 8, 2024
I just called with &offset=500 to get the 2nd page of results and so on. It's very manual but works for now
from nft-api.
chanansh
commented on June 8, 2024
I just called with &offset=500 to get the 2nd page of results and so on. It's very manual but works for now
Can you please verify that offset =page size is equivalent to using the next cursor as the cursor?
from nft-api.
ABusyHippie
commented on June 8, 2024
Idk that for sure, I wasn't able to get cursor to work and settled for offset solution for time being. Interested to hear if anyone from the dev team has a better solution or documentation though
from nft-api.
AssFaceNFT
commented on June 8, 2024
How I paginated responses into a single array in javascript:
- Run an initial fetch to extract the data.total and data.page_size
- The linkforMoreFrens Math.ceil is where the magic happens. The Math.ceil rounds up the number of pages and then multiplies this by 500, to ensure that every offset is accounted for!
Extra for retrying failed attempts via a recursive function: it takes away 500 from the offset to go back to the previous query! Where n is the number of retries.
- In image 2 you can see that the 'listofResults'.json returned by my fetch_retry function is passed to outputData, another function I used to concatenate every result into a single array! I defined storageArray as an empty global array.
Then in output data, each result is applied to that array! Worked for me!
Hope this helps!
from nft-api.
chanansh
commented on June 8, 2024
Thanks. I tried something like this in python using 'joblib' threading. I even used several keys. However there was no speedup. The query speed was the same as if I did it serially.
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: AssFaceNFT ***@***.***>
Sent: Monday, March 21, 2022 11:26:39 PM
To: nft-api/nft-api ***@***.***>
Cc: Dr. Hanan Shteingart ***@***.***>; Mention ***@***.***>
Subject: Re: [nft-api/nft-api] multi-page queries, cursor, offset etc. (Issue #9)
How I paginated responses into a single array in javascript:
1. Run an initial fetch to extract the data.total and data.page_size
[image]<https://user-images.githubusercontent.com/93792440/159365488-b6b79728-5978-4d49-8f2c-9b9247dee894.png>
The linkforMoreFrens function I called at the end is an asynchronous for loop that works out how many offsets it needs to run!
1. The linkforMoreFrens Math.ceil is where the magic happens. The Math.ceil rounds up the number of pages and then multiplies this by 500, to ensure that every offset is accounted for!
[image]<https://user-images.githubusercontent.com/93792440/159364304-9bd430f7-5ea2-4a36-85df-f26b3cd16d59.png>
Extra for retrying failed attempts via a recursive function: it takes away 500 from the offset to go back to the previous query! Where n is the number of retries.
[image]<https://user-images.githubusercontent.com/93792440/159364540-7ade3627-b78c-4971-aba9-188eda289091.png>
1. In image 2 you can see that the 'listofResults'.json returned by my fetch_retry function is passed to outputData, another function I used to concatenate every result into a single array! I defined storageArray as an empty global array.
Then in output data, each result is applied to that array! Worked for me!
[image]<https://user-images.githubusercontent.com/93792440/159366232-fe58b39f-f179-462f-a6ef-71da659ad5f7.png>
Hope this helps!
—
Reply to this email directly, view it on GitHub<#9 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABNULTGKEO4RPFGNDLLEQODVBDSQ7ANCNFSM5PTVPSFQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
from nft-api.
chanansh
commented on June 8, 2024
Did it work faster for you?
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: Hanan Shteingart ***@***.***>
Sent: Tuesday, March 22, 2022 8:37:47 AM
To: nft-api/nft-api ***@***.***>; nft-api/nft-api ***@***.***>
Cc: Mention ***@***.***>
Subject: Re: [nft-api/nft-api] multi-page queries, cursor, offset etc. (Issue #9)
Thanks. I tried something like this in python using 'joblib' threading. I even used several keys. However there was no speedup. The query speed was the same as if I did it serially.
Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: AssFaceNFT ***@***.***>
Sent: Monday, March 21, 2022 11:26:39 PM
To: nft-api/nft-api ***@***.***>
Cc: Dr. Hanan Shteingart ***@***.***>; Mention ***@***.***>
Subject: Re: [nft-api/nft-api] multi-page queries, cursor, offset etc. (Issue #9)
How I paginated responses into a single array in javascript:
1. Run an initial fetch to extract the data.total and data.page_size
[image]<https://user-images.githubusercontent.com/93792440/159365488-b6b79728-5978-4d49-8f2c-9b9247dee894.png>
The linkforMoreFrens function I called at the end is an asynchronous for loop that works out how many offsets it needs to run!
1. The linkforMoreFrens Math.ceil is where the magic happens. The Math.ceil rounds up the number of pages and then multiplies this by 500, to ensure that every offset is accounted for!
[image]<https://user-images.githubusercontent.com/93792440/159364304-9bd430f7-5ea2-4a36-85df-f26b3cd16d59.png>
Extra for retrying failed attempts via a recursive function: it takes away 500 from the offset to go back to the previous query! Where n is the number of retries.
[image]<https://user-images.githubusercontent.com/93792440/159364540-7ade3627-b78c-4971-aba9-188eda289091.png>
1. In image 2 you can see that the 'listofResults'.json returned by my fetch_retry function is passed to outputData, another function I used to concatenate every result into a single array! I defined storageArray as an empty global array.
Then in output data, each result is applied to that array! Worked for me!
[image]<https://user-images.githubusercontent.com/93792440/159366232-fe58b39f-f179-462f-a6ef-71da659ad5f7.png>
Hope this helps!
—
Reply to this email directly, view it on GitHub<#9 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABNULTGKEO4RPFGNDLLEQODVBDSQ7ANCNFSM5PTVPSFQ>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
from nft-api.
ivan-liljeqvist
commented on June 8, 2024
Hey guys - please ask in https://forum.moralis.io and devs will check!
I know they are working on scaling the API to make the requests faster
from nft-api.
Related Issues (8)
- decide which server to use when using the API
- Increasing limit past 500?
- Typo on variable declaration HOT 1
- Integration with DCL (Decentraland) contracts
- GetNFTOwners，offset field is invalid
- Is Opensea api key necessary now? HOT 2
- GetNFTsForContract update is very slow and generate duplicated values
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nft-api.