facundoolano / app-store-scraper Goto Github PK

View Code? Open in Web Editor NEW

965.0 965.0 230.0 212 KB

scrape data from the itunes app store

License: MIT License

JavaScript 100.00%

app-store-scraper's People

Contributors

Stargazers

Watchers

Forkers

fjteam stevenwarren takaaptech foglio ekurutepe dahlgren piercus namnv04 mrcoles jacklee0223 davidgustys reactnativevn danieldram rchatterjee fchienvuhoang mailightkun dheerajbhaskar cgwyllie icaroponce hitendrasinh moremorefor peterromfeldhk jaydeepjivani hyunukjung caxap wooodhead hama ashalva chandon tracer8 kareemgrant throwjojo labco fredvanrijswijk vinicius-batista autocompare makkajai sm-github charimon joncursi pfdtk pdobrev zed7576 normancroan yura415 zpcooper byxucet samkaks filoozom mauryaakhilesh iplanwebsites xiaosan0318 minsoe ldthai joeynovak sdsuper koltclassic arkty grashupfer99 deze333 aceywli araratispiroglu solertis kittisak-phetrungnapha imerica giorgio-zamparelli harshvasudeva applysis scrxped yoshimaa lisakuai0827 publinchi christian-fei gerella profitcore kasnder aaa22223 wangrun webaifei johnmahugu ashishbijlani tiago-aguiar yurganov redstarxz quangvo09 sharkpunch wade1990 skelli-studios guozanhua afkcodes kartikhimself emovla neilshao rankmyapp zhyfff myukselen murphywuwu louisgv wlijie knorrkator

app-store-scraper's Issues

Question / Request: Number of downloads/installs

Hi.

Is it possible to get the number of times an app was downloaded / installed?

I'm only concered about getting this for my app, but I need an API for that, not viewing in the app portal.

store.reviews example returning 502 Bad Gateway instead of result

I've tried a few different appIds but all result in same

add unit tests (and travis CI)

like the ones in google-play-scraper

No device as IOS

store.device.IOS does not exist.

Check if reviews is a valid type array

If results.feed.entry exists but is not a valid type array (i.e. dictionary) in our case, this error occurs. The resulting json file returns a dictionary for this entry when there are no reviews in that page.

TypeError: reviews.slice is not a function
    at cleanList (.../node_modules/app-store-scraper/lib/reviews.js:10:18)
    at process._tickDomainCallback (internal/process/next_tick.js:129:7)

A simple check to see if it's a valid array would be sufficient for a fix in this case.

Crash when single review is returned

If there is only one review then the server returns a single object, not an array and the parser crashes with:

TypeError: reviews.map is not a function
    at cleanList (.../node_modules/app-store-scraper/lib/reviews.js:15:18)
    at process.internalTickCallback (internal/process/next_tick.js:77:7)

Try this call to duplicate:

store.reviews({id:` 1092841332, country: 'IL'}).then(console.log).catch(console.log);

Error is here:

function cleanList (results) {
  let reviews = results.feed.entry || [];
  return reviews.map((review) => ({
// ...

I managed to fix the problem with something like the code below but perhaps there could be a better way.

function cleanList (results) {
  let reviews = results.feed.entry || [];
  // Check if it's a single object and if so wrap it into array
  if (results.feed.entry && !Array.isArray(results.feed.entry)) {
    reviews = [results.feed.entry];
  }
  return reviews.map((review) => ({
// ...

Can't Get Reviews After Providing 'id' and 'appId'

Hi, Thanks for developing this scraper. I use 'app' to retrieve the full detail of an application with id: 1135011025 and find the appId. But after I providing both 'id' and 'appId' as options in 'reviews', there's nothing retrieved. The codes are as following:

var store = require('app-store-scraper');
store.reviews({
id: 1135011025,
appId: 'com.upside-travel.Winterfell',
sort: store.sort.HELPFUL,
page: 2
})
.then(console.log)
.catch(console.log);

Thanks for your help.

No Reviews?

Do you plan to add a review functionality?

country option in similar method being ignored

The country option (originally added by @piercus ) that's passed to itunes in the storefront header, started to be ignored recently, breaking the related test.

similar function assumes app always exists

The following invocation of app assumes that appId exists.
https://github.com/facundoolano/app-store-scraper/blob/master/lib/similar.js#L20

Getting UnhandledPromiseReject warning for this.

Returning old app info

After deploying a new version of the IOS app, app-store-scraper returns the previous version with its details.

Is there a reason that the updated details will take some time to propagate?

Test case: Opening the app store URL shows the update is live with new version, and the update is available to devices, although app-store-scraper returns the previous version with its details.

I need to check the version immediately when update is live.

App store version:

app-store-scraper version:

Retrieving app by "id" instead of "appId" seems to make a difference, why would this be the case?

Retrieve Reviews by Version

Is there any way to retrieve reviews by version or is there any planned support for this?

RESTfull Interface

We took your Google Play API and ported it to work for the app-store scraper.
If you're interested it's available here: https://github.com/lucky42890/App_Store_API-App-Info-

How to store this data in a file?

[Question] Languages supported

Hi @facundoolano, Thanks for sharing app-store-scraper! I'm wondering if it is possible to use it with languages other than English. I'm interested in obtaining search suggestions in Chinese/Japanese.

The "list" function returns an error.

The response body is output as shown below.

<HTML>
<HEAD>
<TITLE>Web Server Error</TITLE>
</HEAD>
<BODY BGCOLOR="white" FGCOLOR="black">
<H1>Web Server Error</H1>
<HR>

<FONT FACE="Helvetica,Arial"><B>
Description: The host did not return the document correctly.
</B></FONT>
<HR>
</BODY>

ENOBUFS error after many requests at once (403 error after many frequent requests)

I'm trying to export reviews by looping through 2 arrays of data (10 games and 38 countries) and iTunes reviews pages. Something like this (I'm not good at coding):

for( var i = 0; i < g.length; i++) {
	for( var j = 0; j < c.length; j++) {
		for( var p = 0; p < 10; p++) {
			// Parse reviews
			})
			.then(console.log)
			.catch(console.log);
		};
	};
}

I'm getting this in the output file:

{ Error: connect ENOBUFS x.xx.xxx.xxx:xxx - Local (undefined:undefined)
    at Object._errnoException (util.js:992:11)
    at _exceptionWithHostPort (util.js:1014:20)
    at internalConnect (net.js:960:16)
    at defaultTriggerAsyncIdScope (internal/async_hooks.js:284:19)
    at GetAddrInfoReqWrap.emitLookup [as callback] (net.js:1106:9)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:97:10)
  code: 'ENOBUFS',
  errno: 'ENOBUFS',
  syscall: 'connect',
  address: 'x.xx.xxx.xxx',
  port: xxx }

I've searched for workaround and tried to delay each request with setTimeout (in many various ways) with no success.

I would like to know if it's possible to add throttle option like in google-play-scraper to avoid this error?

List method broken

causing the build to fail: https://travis-ci.org/facundoolano/app-store-scraper/builds/335353628

the itunes rss feed api has changed, causing the method to break. url structure changed and seems like the category field has been removed as well.

Simple script does not terminate

Node v4.2.2
[email protected]

It appears that this library prevents a script from terminating. Here’s a super simple script that when I run from the command-line as node test.js it just hangs until I forcefully close it.

var store = require('app-store-scraper');

store.search({
    term: 'panda',
    num: 2,
    device: store.device.IOS,
    country : 'us'
})
.then(console.log)
.catch(console.log);

As a sanity check, the following does terminate on its own.

var store = require('app-store-scraper');

function foo() {
    return new Promise((resolve, reject) => {
	resolve('OK!');
    });
}

foo()
.then(console.log)
.catch(console.log);

Apps position is not the same showed in the iTunes Store

Hello there,

I'm using the module to search for some apps, but the result returned is not the same that I get on the iTunes app (Yes, I had set the country correctly) which works perfectly in the google-play-scraper module. As I'm using it for ASO purposes, it's critical to know the real apps positions in the iTunes. (For example, I want to know if the app which i'm optimizing in ranking better, if it's in the first position for a given word...)

Talking to a friend of mine who works with ASO daily, he told me that the iTunes API is not reliable when it comes to the apps positioning (maybe for a delay to update, idk, but it isn't the same list) and another solution would be necessary.

Had you guys have the same problems or notice something like this? Or am I the only one?

Actually, if it's really a problem, I'm considering a solution: I've found out using a Sniffer (like Wireshark), to monitor the network when I'm using the iTunes app, and the app do requests to another url, setting different headers to fetch the content to be presented on it...

Please, let me know what you guys think about it.

Compile to ES5

Hey,

Thanks a lot for your work here - we've been using app-store-scraper for some time with our Ionic app.

We have this issue with our app, the underlying cause being app-store-scraper. Until now it only affected development builds and we've just lived with it.

However, on Ionic's latest iteration, it's completely broken, so now our production builds will not run on iOS 9 at all and we need to get our release out ASAP.

After spending some time trying to monkey patch this morning, I've realised that the lesser of several evils was just to release a clone that's compiled to ES5: https://www.npmjs.com/package/app-store-scraper-compat.

I am just raising here for your information. I figured you might not be interested in supporting ES5 anyway.

If you end up releasing a backwards compatible version in the future I'll just kill off app-store-scraper-compat.

Thanks

Stephen

iTunes will return no data sometimes unless country code is provided.

Hi there,
Thank you so much for putting this package together, saves me a ton of time. It's really great but there is a small issue where with some appIds, unless the country_code is provided in the lookup url, it will return nothing back in the json response even though the app is live in the store.

I am thinking of making a pull request to add in a feature so you can add the country code similar to how you can add throttle to the play-store-scrapper. My goal is that if the country code is provided it tacks it on to the url, if not, everything runs normally as it already does. Let me know if this is okay and I'll push up the PR.

Again, thanks for all your work on this package :)
Cheers!

unable to make the scraper to work properly behind corporate proxy

at office we have an http proxy :

looks like the http_proxy & https_proxy environnement variables are not checked by the module which try direct http access.

Default value of page param on reviews method

Is there some special reason for the default value of the page param on reviews method to be 1 instead of 0?

My concerning is about the compatibility with the google-play-scraper module.

I think that if we could make it compatible and work the same way, it'd be great for both APIs.

100 items return on search

Hi,

Thanks for the scraper it works like a charm but the search function returns only 100 results and if there are some others it says "... 150 more items" and interrupt. Is there any method to see full search result?

Similar call not return any results

I have used the method Similar example, but it returned a null array. I have tested with other id but it was returned the same result. Can anyone help me?
I used the search method and it worked so i think i have installed it successfull.

Feature Request: Appstore Release Notes

Would love to have a list of release notes. Or even just the count would be amazing!

Thanks

Is there is a way to get watchOS apps ?

implement fullDetail for list

The new itunes rss feeds returns much less information than before, so a fullDetail option should be implemented as it is in google-play-scraper.

make sure memoization properly works for the full detail calls (dont make a request if the app has been called via the app method)

unskip (and fix as necessary) the full detail list test.

XML Feed for Reviews?

I've been working with this scraper to collect app reviews, but my use case requires review dates. I noticed the XML version of the reviews feed provides date info, and was going to work on adding that functionality to this scraper, but then I saw there was already an accepted PR for that feature in March 2017. Is there any reason the scraper is back to using the JSON feed for reviews?

Here's the PR in question:
#19

Get IPA

it's possible to get the IPA without login?

Unable to get reviews sorted as RECENT

I am trying to sort reviews as RECENT but it is giving me an error.

TypeError: reviews.slice is not a function

var store = require('app-store-scraper');
store.reviews({ id: 'XXXXXXXXX',sort: store.sort.RECENT,page: 2}).then(console.log).catch(console.log);

bug in extractSuggestions

In extractSuggestions function in app-store-scraper/lib/suggest.js assumes there is always suggestions. I think it should check before applying list.map.

TypeError: Cannot read property 'map' of undefined
    at extractSuggestions (/Users/rahul/appscraper/node_modules/app-store-scraper/lib/suggest.js:27:14)
    at process._tickCallback (internal/process/next_tick.js:109:7)

(I am very new to nodejs, and pardon me if it shouold be how it is.)

Is it possible to get retrieve all apps by a given developer name or ID?

You've the functionality to retrieve all apps by a given developer name in your Google Play Scraper script. Can you bring this feature to the app store scraper as well? Please, I'd be so damn happy.

[ Question ] Complete Review History

Hey @facundoolano ,
Thanks a lot for providing this API. Is it possible to extract the complete review history for an app instead of just 10 pages.

Get All Reviews?

Is there any way of knowing how many pages are able to be requested? is there a way of requesting all reviews?

Allow memoization for internal calls

the common.lookup helper is used by multiple methods, they should share the memoized results when possible. Pass it by parameter like it's done in google-play-scraper
the search method makes a first request that is common to all pages; this result should be reused when fetching subsequent pages.

App not found

App store Scrapper works pretty but on some app it shows error
that app is not found
for example
com.bestonlinepartners.udemy.udemy

Feature: Reviews call does not return last page, add pageDetail

When fetching all reviews from the app store the returned json does not tell me how many pages there are in total. While the xml returned by apple does include the last page.

For example: https://itunes.apple.com/gb/rss/customerreviews/page=1/id=585027354/sortby=mostrecent/xml
Has as 10th line the last page available for this app.

"currentVersionScore" value gone

I just noticed that reviews no longer contain the "currentVersionScore" value. Is this something Apple deprecated or could I be missing something. Thanks!

No Date in Reviews Results

Is there any way to add date data in the Reviews output?
I see that the google-play-scraper has this but not that app-store-scraper.

Thanks!

Lookup for country and language

I have seen that you can pass a language to the lookup API to retrieve the data for that language.

Something like this: https://itunes.apple.com/lookup?bundleId=com.you.app&country=ca&lang=fr

The app method doesn't seem to support it.

Error when summary field data not available

Summary field data available
https://itunes.apple.com/us/rss/topfreeipadapplications/limit=2/genre=7001/json

Summary field data not available
https://itunes.apple.com/us/rss/newfreeapplications/limit=2/genre=7001/json

Review date

Is it possible to get a date field for app reviews?

Feature request: add ratings

Can the ratings also be added to this library, like the android version?
They are a bit more complex to fetch, but very useful.
You can fetch them with the call below:
By using: GET http://ax.phobos.apple.com.edgesuite.net/WebObjects/MZStore.woa/wa/viewContentsUserReviews?id=&pageNumber=0&sortOrdering=2&type=Purple+Software
With headers:
User-Agent: iTunes/9.1.1
X-Apple-Store-Front: 143441-1

X-Apple-Store-Front tells it which apple store to talk to 143441-1 is usa.
This page includes all reviews and all ratings, so it includes way more data then the rss feed and is literally the call that itunes makes.

Bug in 'reviews' documentation

README.md - reviews - default page is 0 -- but in the code it defaults to 1

the method ".list" does not work

all other methods works well except the method ".list" does not work for me since yesterday knowing that it was functional...i can't get the app list from app store

Is it possible to find the Promotional Artwork image?

Hi,

Is it possbile to get the App Store Featuring Art images?

Read more about App Store Promotional Artwork here:
https://help.apple.com/itc/appspromoart/en.lproj/static.html

Research itunes endpoint for app lists

Instead of using the feeds endpoint, we could use whatever endpoint itunes uses internally. The feed is slow (see #55), has limited results and currently doesn't allow filtering by category.

List method working very slow

close to 5 seconds in travis, timing out locally. Make sure this is an external issue we can't control.