Apps position is not the same showed in the iTunes Store about app-store-scraper HOT 23 CLOSED

Ops, I'm sorry. Got the wrong endpoint.

Check it:

curl -X GET \
  'https://itunes.apple.com/WebObjects/MZStore.woa/wa/search?submit=edit&mt=1&term=pandas' \
  -H 'accept-language: en-us' \
  -H 'x-apple-i-md: AAAABQAAABASYMVHFwrAt3wPSUwbTCqGAAAAAg==' \
  -H 'x-apple-i-md-m: hxpW6pCpiMIBZ+ch0fA6Hjke7zRLyqe5KdegwA9TzU2aDGPWCL1CX85fi+s4dFX4yDC58Owm+DJImUbI' \
  -H 'x-apple-store-front: 143441-1,32 ab:IMP0' \
  -H 'x-apple-tz: -7200'

It'll return a HTML file. A bit confusing.

But, search a for a <script> tag that has its.serverData. You'll notice that it's a JSON. Get it.

Access its property pageData.searchPageData.bubbles. It'll bring an array that aggregates data by its kind. The index 3 bubbles[3], whose name proprety is 'software', has a property results which contains 116 results.

Each item of array has an id property. its the app id.

The most important here: these results array in are "ordered correctly", which means that it brings the same apps order that I get when a search for something in my iPhone or iTunes APP, which is not true when using iTunes API.

Hope it'll help.

from app-store-scraper.

Sure,

here an example of url the app hits:

https://search.itunes.apple.com/WebObjects/MZSearchHints.woa/wa/hints?q=panda

Just sending a get request to it, you'll get a XML file.

You can set params to it through the header:

{"Accept-Language":"pt-br"}
{"X-Apple-Store-Front":"143503-15,32"}
{"X-Apple-I-MD-RINFO":"17106176"}
{"X-Apple-I-MD-M":"5qf7Ex9VvAxuH98/nDvbIHqW76fYyHTpVEhcWC2tBT8FKc7zqErTS9/kmJlqWXGSfPgcdl831PNkbddc"}
{"X-Apple-I-MD":"AAAABQAAABCDXFWHMZGdT6C1K2IFjB00AAAAAg=="}

These last two tokens represent language and country.

I'm quite busy this week, but if you can't check it yourself during the week, I could dig it deeper on the weekend.

But, all we need is to turn on the sniffer and watch the requests behavior to understand all the params sent on header.

I know it's a more verbose way to do it, but if really there's difference in the api, this way seems to be the most accurate way to retrieve data for some methods, like search for example.

from app-store-scraper.

@facundoolano did you find a way to filter by device (ipad/iphone)?

from app-store-scraper.

I haven't really compared both sets of search results, but if you've found differences it would definitely make sense trying to look into the actual APIs the iTunes app is using. I'd really be interested in seeing those in case you can take the time to dig around using a sniffer.

from app-store-scraper.

Mmm the hints endpoint you show is for search hints, not search itself. That endpoint is already being used for the suggest function (see this). I imagine the search itself uses a different endpoint.

from app-store-scraper.

Has this issue been worked on?

from app-store-scraper.

Nope, no updates.

from app-store-scraper.

@icaroponce Does your App Store Scraper use the endpoint you talk about here? So far it is in fact the only endpoint I can find that returns the same ordered results as the app.

from app-store-scraper.

Hey @cloforce,

Yes, it does.

We had to develop a small internal lib to use the endpoint pointed above, seeing that the real current position for a given app is very important to our platform we had to do that.

from app-store-scraper.

@icaroponce Awesome! Would you guys be open to sharing that internal library? Can email me privately about it.

from app-store-scraper.

To be clear, the endpoint you’re talking about is in use in this library: https://github.com/facundoolano/app-store-scraper

I’m still looking for the best way to get App Store search results that reflect the ones in iOS 11

from app-store-scraper.

That was a question, btw 😅

from app-store-scraper.

No @anater, It's not. Otherwise this issue would be already closed.

If you follow my instructions you will get the list that reflects the App Store. Pay attention to the headers set only, because you've to set it properly to a specific country and language.

from app-store-scraper.

@cloforce unfortunately I'm not working at that company anymore. That time we gave up opening it because we wanted update this lib or even add another endpoint here instead of creating a new lib to do almost the same thing.

I still can check with the guys if they would be up to share the lib though without problems.

from app-store-scraper.

@icaroponce Thanks for the reply. Any update from the guys? If not I think I'm going to begin building it out myself. Let me know, appreciate it.

from app-store-scraper.

Hey @cloforce,

UnfortunateIy I haven't got a reply yet.

You better start your own, I'll let you know in the case something changes.

from app-store-scraper.

@icaroponce can you refresh my memory a bit here? On May 4, 2017 you described a way to get the properly ordered app list. Does that list include all the app data or it only servers to get the ordered ids?

from app-store-scraper.

Yes, this url provides a correctly ordered appIds list. Then we used another endpoint, ( I guess that was the/lookup) to fetch app details. This endpoint allows us to send more than 100 appIds at once, then we basically could fetch everything in 3 requests (1 for all the ordered ids, 2 in order to fetch the details ). I also need to refresh my memory about some details, but as soon as a I have a free time I'll investigate it again and comment down here everything more elaborated and how to reproduce the complete flow.

from app-store-scraper.

I'm looking into this.

@icaroponce the current search endpoint allows to filter by device (ipad, mac or all), do you know if such filtering is available with the https://itunes.apple.com/WebObjects/MZStore.woa/wa/search endpoint?

from app-store-scraper.

This one seems to give the json directly

curl -X GET \
 'https://itunes.apple.com/WebObjects/MZStore.woa/wa/search?clientApplication=Software&media=software&term=panda' \
 -H 'accept-language: en-us' \
 -H 'X-Apple-Store-Front: 143465-19,24 t:native'

from app-store-scraper.

How can i unlock any iPhone Device i have mobile shop please help

from app-store-scraper.

The integer after the comma in the X-Apple-Store-Front header allows you to filer by iPad or iPhone. That value seems to change over time based on examples I've found across the web. Today, "143441,29" returns results for iPhone (aka "software") and "143441,30" returns results for iPad aka "iPadSoftware".

(I still haven't figured out the value for "macSoftware" for macOS.)

Also, you don't need the accept-language: en-us header nor the t:native as far as I can tell.

This works well

curl -H 'x-apple-store-front: 143441,29' 'https://search.itunes.apple.com/WebObjects/MZStore.woa/wa/search?clientApplication=Software&term=panda'

Finally, note that the search endpoint hostname is now search.itunes.apple.com, not itunes.apple.com.

from app-store-scraper.

The integer after the comma in the X-Apple-Store-Front header allows you to filer by iPad or iPhone. That value seems to change over time based on examples I've found across the web. Today, "143441,29" returns results for iPhone (aka "software") and "143441,30" returns results for iPad aka "iPadSoftware".

(I still haven't figured out the value for "macSoftware" for macOS.)

Also, you don't need the accept-language: en-us header nor the t:native as far as I can tell.

This works well

curl -H 'x-apple-store-front: 143441,29' 'https://search.itunes.apple.com/WebObjects/MZStore.woa/wa/search?clientApplication=Software&term=panda'

Finally, note that the search endpoint hostname is now search.itunes.apple.com, not itunes.apple.com.

I found this if ever it is helpful to anyone:
https://stackoverflow.com/questions/47406922/what-does-x-apple-store-front-means-in-apple-http-headers

from app-store-scraper.