jpchip / giveaway-grabber Goto Github PK

Dockerize project. Want to be able to build image and run it, something like:

$ docker build -t giveaway-grabber .
$ docker run -it --rm giveaway-grabber 

The website here: [ https://giveawaylisting.com ] has a checkbox to filter books which is separate from the one to filter Kindle Editions. I believe the pertinent html code is as follows with the part that should be helpful in bold. [It was bold till I put it in a code block so you could actually SEE the code. Now it's surrounded in double asterisks. ]

<label><input type="checkbox" name="box" onclick="check(this,1,**/varies/**)">**Variable odds**</label>

<label><input type="checkbox" name="box" onclick="**check(this,5,/https:\/\/amzn\.com\/\d/**)">**Books**</label>

<label><input type="checkbox" name="box" onclick="**check(this,5,/ \(Kindle Edition\)(?:<\/a>\|-->)/)**">**Kindle Editions**</label></span>&emsp; &emsp;<span style="white-space: nowrap;">Requires:<label><input type="checkbox" name="box" onclick="check(this,0,/<!--v-->/)">Video</label><label><input type="checkbox" name="box" onclick="check(this,0,/<!--p-->/)">Poll</label><label><input type="checkbox" name="box" onclick="check(this,0,/<!--w-->/)">Tweet</label></span>&emsp; &emsp;<span style="white-space: nowrap;">Follow on:<label><input type="checkbox" name="box" onclick="check(this,0,/<!--t-->/)">Twitter</label><label><input type="checkbox" name="box" onclick="check(this,0,/<!--a-->/)">Amazon</label></span>

The code checks for a bunch of things such as:

Variable odds
Books
Kindle Editions   

Requires:Video Poll Tweet   Follow on:Twitter Amazon

So if I read it right all hardback books will have an url that begins https://Wamzyn.com/d/ ?

@jpchip You tired of me yet? ;)

Just figured out the 2FA situation a bit more...here is the specific situation where it did not wait. Your program tried to login and amazon returned the password AND capcha page. I entered both and clicked signin..I was then presented with 2FA page to enter code. From the Time the 2FA page came up to when I could enter the code the program attempted to enter 4 sweeps on page 1(Giveaway x not found). Now I just ran it as "normal" with the 2FA code and it waited as you mentioned so it's only in the previous circumstance.

A similar issue happens with the "normal" 2FA route is the user enter the wrong code or one that just expired.

Originally posted by @famewolf in #22 (comment)

One of the entries on page 1 was a kindle of some sort...there was no video or drop box...just a button that said "Enter". I clicked it manually and it went along with the other sweeps being as expected.

That's literally all the button said vs the normal "Continue to Enter".

Occasionally Amazon makes you re-enter your password. Add logic to handle that situation.

When the user wins a sweep they are given an url which takes them to a page showing the item they have won and a button saying "Confirm Address" with the address listed below. Once you are certain amazon has the right address I would think the preferred choice would be to have the software auto click the "Confirm Address" button to ensure the prize is locked in. Sure you might get a few things you didn't really want but it would prevent you from losing the win of a really nice cell phone because the power went out at your house and you lost the terminal window.

I was just wondering if there is something that can be done to keep the "Switch Users" screen from always being presented. I am not sure why that is happening. It is almost like the session cookie is getting corrupted or something like that. I see that there is a way to put puppeteer into a wait till the page is entirely loaded, but I am not sure that is the problem.

I get the feeling that it may just be my install.

If this is a one off or expected behavior due to chewing through all the pages, please feel free to close. Otherwise, I would be interested to hear if anyone has been able to eliminate the annoying "Switch Users" screen from happening.

Thanks!

I believe I have found a way to address the captchas that appear from time-to-time on entries. I am testing now, but it is looking good. It may be a somewhat bandwidth intensive solution though. Not exactly sure, but it makes those captchas a thing of the past.

I will test for a few days and do a pull request if everything works as expected.

On some sweeps (and I have no idea why) the box will drop, the program will click on the box..and then the box drops AGAIN instead of opening....in this scenario currently the program just sits until it timeout's and then continues to the next sweep. If you are watching you can manually click on the box again. I'm not sure if there is a programmatic way to detect this but it's annoying. Appears to be random.

Requested enhancement to include the total # of pages when showing page count so user can determine if program aborts early.

Multiple times the program for one reason or another (typically a brief lapse in my internet) will get the Giveaway x not found several times and determine it's done with all giveaways then exit. Unless it's obvious like it does it on page 4 you have to manually check to see what the last page is to know if you need to restart it. Showing the total pages in console will solve this issue.

A possible better way to handle this would be to have the program/macro check if page count is = to max page count and if not sleep for a determined period of time then retry the page/giveaway#.

Add a call to check the Keep Me Signed checkbox when logging in.

I almost have this completed and will issue a pull request shortly.

I'm shamelessly stealing this idea from a similar giveaway grabber that doesn't work anymore. I'm thinking the ability to specify to skip the kindle sweeps might be a good way to speed things up as typically the books are things I've never heard of. I'm also wondering if more of this could be specified in the .env vs adding flags?

Once some of the other issues are resolved, publish package on NPM. Want to be able to install it globally like npm install -g giveaway-grabber and run it like: $ gg

There are starting to be a lot of options, which are hard to enter via cli, and annoying to maintain as both cli and .env file options. Refactor app to run off a json config file instead, which would have all available options inside it.

1. add an init command, like gg init which would generate the config file (ie. ggrc.json)
2. Running gg without config file will ask you to run gg init first
3. Drop support for env variables and force everything to be in the single config file. Leave --page support.

Can do this all using the yargs library that's already part of the code base. Would be extra cool though if gg init prompted you for credentials and stuff, using Inquirer.js.

This would be a HUGE breaking change and would have to be a major release.

Not sure if I'll have the time to go through with this, but wanted to write it down while I was thinking about it.

i just need a tutorial on how to set this up, from installation to usage, THANK YOU

giveaway 18 did not exist?
giveaway 19 did not exist?
giveaway 20 did not exist?
giveaway 21 did not exist?
giveaway 22 did not exist?
giveaway 23 did not exist?
giveaway 24 did not exist?
{ TimeoutError: waiting for selector "ul.a-pagination li:last-child" failed: timeout 30000ms exceeded

This happens every time, no matter the page. Then the entire process stops.

Create a data system that will allow gg to be a little more selective in what pages it navigates into.

1. The URL of the sweep will be tracked.
2. The status of what happened with the sweep will be tracked. The statuses below will be implemented: Won, Lost, Already Entered, Is Black Listed, Giveaway Has Ended
3. The Date and Time of when the sweep was started.
4. The number of winners that can win before sweep is over.
5. The retail value of the sweep.
6. The Date and Time of when the sweep will end.
7. Win or Lost Date and Time

Most likely this will be an SQLite3 database. Infrastructure will be built so that DDL modifications can be rolled out as changes to Tables and Fields need to be introduced. The version that is stored in the database will be utilized to determine what modifications are needed for DDL processing on subsequent releases.

Need to introduce a throttle that is based on random seconds between page requests. That way the bot looks more like a very dedicated user working through the site.

I will create a pull request for this shortly. Just working through some testing.

I'm not sure what is expected to happen when the program gets to the end of the sweepstake pages (start over?) but what I see is the program crashing. I should note I've already increased the timeout to 60 seconds in a couple of places(index.js and giveaway.js) but this appears to be a different one.

Page 160 Start:
giveaway 1 already entered.
giveaway 2 cannot be entered.
giveaway 3 is ready!
waiting for video (~15 secs)...
Whoa, got to re-signin!
no submit button found, oh well. Moving on!
Whoa, got to re-enter password!
giveaway 4 cannot be entered.
giveaway 5 already entered.
giveaway 6 cannot be entered.
giveaway 7 cannot be entered.
giveaway 8 cannot be entered.
giveaway 9 cannot be entered.
giveaway 10 cannot be entered.
giveaway 11 already entered.
giveaway 12 already entered.
giveaway 13 already entered.
giveaway 14 did not exist?
giveaway 15 did not exist?
giveaway 16 did not exist?
giveaway 17 did not exist?
giveaway 18 did not exist?
giveaway 19 did not exist?
giveaway 20 did not exist?
giveaway 21 did not exist?
giveaway 22 did not exist?
giveaway 23 did not exist?
giveaway 24 did not exist?
(node:29820) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/FrameManager.js:1276:21)
at
-- ASYNC --
at Frame. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:144:27)
at Page.waitForNavigation (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/Page.js:644:49)
at Page. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:145:23)
at enterGiveaways (/home/famewolf/giveaway-grabber/src/giveaways.js:221:33)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:29820) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:29820) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The checkForPassword() function attempts to submit a password stored in process.env.AMAZON_PASSWORD.

However, if the password was supplied through a command line argument (--password=) rather than an environmental variable, process.env.AMAZON_PASSWORD will be null. This is causing the script the crash with the error UnhandledPromiseRejectionWarning: TypeError: text is not iterable

The videos do not need to be in the foreground while playing. Potentially you could have multiple tabs running waiting for the 15 second video to complete and switch back to them to click on the box.

https://amzn.to/2uO2QOw

Says the video is unsupported but I've seen that on other sweeps that worked fine. The program exits these and moves on to the next almost immediately. Not sure why. There are several sweeps with various wine glasses and they all act the same way...enter...then exit.

I checked one manually in my own browser and it also said the video was unsupported and I let it sit way longer than 15 seconds but the continue button at the bottom never became available.

In testing, I have discovered it stalls on finding the box to click on when running in headless mode. The same thing happens in the browser in the window is minimized. See if there is some way around that.

I need to enter the 6 digit code for two party authentication into chrome. Once it's in, it will be ok for 30 days....I just can't figure out where chrome is or how to get the script to pause so I can fill the page out.

Point me in the right direction?

Would it be possible to and an option to allow it to skip over any giveaway that requires a Captcha? I know it's an issue you just fixed, but unless it will ask on every time If you don't put the Captcha in, it would be good if your not there to check it.

In the same way the user is notified for capcha's notify the user if they should WIN a sweepstake.

Just had a sweep go by that according to my understanding should have been blocked? Do I need to add plurals since we went to whole words?

Should continue on to more pages if they exist. Got to catch 'em all!

I was going to select all of command prompt window and my mouse glitched out and closed prompt whatever i won is just lost now, I didn't get a notification. Is it a loss?

I was wondering whether the program could maintain it's OWN position so that when restarted the program would just continue where it was left off unless the user overrode it via a --page command.

After finishing the first page, it no longer "finds" giveaways. Error happens even after exiting and restarting from any page (e.g. --start=2)

After restarting the Node.js command prompt and starting the package from scratch, it works again. Not sure if the issue is with the package or with a possible Amazon rate-limiter?

Edit: Seems to happen after every page switch?

Normally when you hit the 2FA code screen you enter the 6 digit code and check a box that says "Don't ask anymore" which will be good for 30 days. I assume a cookie gets stored of some sort. Having to enter the code every time you start the program is not so bad. Having to enter it every single time the program crashes because your internet took longer than 30000ms or even 60000ms to answer a request can quickly become annoying when you are having to grab your phone and run the authenticator app for the 50th time that day because Sprint sometimes sucks.

I think the next think I'm going to do is update it to send an email on a win. I think the easiest way to do to this is to use send grid. Anyone can sign up for a free account, so I can allow setting the SENDGRID_API_KEY in the .env file, and if it exists, use sendgrid to shoot off an email to the email address used to log in when an entry wins.

I know I could use nodemailer, but then I'd have to ask for user's to put a bunch of stuff in .env, seems like a pain all around.

I really want to make it easier to not miss wins, so this will be my next priority.

Originally posted by @jpchip in #32 (comment)

Perhaps you could consider adding --mute-audio to the chromium launch config?

Maybe this could be an added config option that goes in the init file?

I wondered if the program could wait a bit and then retry on timeout errors such as these:

(node:15969) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/LifecycleWatcher.js:143:21)
at
-- ASYNC --
at Frame. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:108:27)
at Page.waitForNavigation (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/Page.js:676:49)
at Page. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:109:23)
at navigateToGiveaway (/home/famewolf/giveaway-grabber/src/giveaways.js:158:35)
at asyncForEach (/home/famewolf/giveaway-grabber/src/giveaways.js:302:10)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:15969) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:15969) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

It's a shame you are stripping the spaces...I would just separate by the command and grab the strings asis. Anyway I ended up having to pull "bra" and a few others that ended up blocking unexpected sweeps.

Originally posted by @famewolf in #22 (comment)

Multiple files are mentioned. I browsed Lifecyclewatcher.js but wasn't sure where to lengthen the timeout similar to how I had to do it in giveaways.js.

(node:15841) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/LifecycleWatcher.js:143:21)
at
-- ASYNC --
at Frame. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:108:27)
at Page.goto (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/Page.js:656:49)
at Page. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:109:23)
at checkForPassword (/home/famewolf/giveaway-grabber/src/giveaways.js:99:14)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:15841) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:15841) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I've noticed several times now when the program has finished going through all the pages it's not exiting. It's sitting there and requires a ctrl-c to finally exit to the command line. Not a major bug but needed to be pointed out.

giveaway 14 is blacklisted [kindle edition].
giveaway 15 is ready!
waiting for video (~15 secs)...
could not find video, oh well. Moving on!
giveaway 16 did not exist?
giveaway 17 is blacklisted [kindle edition].
giveaway 18 is blacklisted [kindle edition].
giveaway 19 is blacklisted [kindle edition].
giveaway 20 is blacklisted [kindle edition].
giveaway 21 did not exist?
giveaway 22 did not exist?
giveaway 23 did not exist?
giveaway 24 did not exist?
No more pages! Goodbye!
^C
famewolf@hp810-135qe ~/giveaway-grabber $

Given all the "could not find video, oh well. Moving on!" it was probably suffering the sign in problem when this occured.

Assess with an algorithm whether the giveaway is ripe to enter (meaning it has been listed long enough to be likely to be won given it's cost and number of required entries.) Then only enter the giveaway after it's reached a certain level of 'ripeness'.

From comment by @joshjenkinsAR on #32

So... amazon doesn't always take the giveaway down right away after you win it.
Sometimes that giveaway page hangs around for a while after you confirm your address.

When the script encounters these pages for items that have already been won and confirmed, it seems to just stop there.

Here's the html
alreadywonandconfirmed.zip
for one such page.

edit:
Here's the last bit of the console output:

giveaway 16 is ready!
waiting for box...

I get internet from a wifi hotspot in a rural area so it's not the fastest....the program is dying alot with the following error:

giveaway 13 cannot be entered.
(node:25497) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/FrameManager.js:1276:21)
at
-- ASYNC --
at Frame. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:144:27)
at Page.waitForNavigation (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/Page.js:644:49)
at Page. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:145:23)
at navigateToGiveaway (/home/famewolf/giveaway-grabber/src/giveaways.js:87:35)
at asyncForEach (/home/famewolf/giveaway-grabber/src/giveaways.js:176:10)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:25497) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:25497) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:25497) UnhandledPromiseRejectionWarning: TimeoutError: Navigation Timeout Exceeded: 30000ms exceeded
at Promise.then (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/FrameManager.js:1276:21)
at
-- ASYNC --
at Frame. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:144:27)
at Page.waitForNavigation (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/Page.js:644:49)
at Page. (/home/famewolf/giveaway-grabber/node_modules/puppeteer/lib/helper.js:145:23)
at navigateToGiveaway (/home/famewolf/giveaway-grabber/src/giveaways.js:87:35)
at asyncForEach (/home/famewolf/giveaway-grabber/src/giveaways.js:176:10)
at
at process._tickCallback (internal/process/next_tick.js:188:7)
(node:25497) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 3)
(node:25497) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)

I'm certain the wait time can be changed to 60 seconds instead of 30 if someone will point me to where?

I've noticed when amazon throws a capcha onto a sweepstake the program seems to go off the deep end. Perhaps the program should pause if a capcha is detected possibly by looking for the verbiage "Please enter the following letters below" or something similar. Otherwise the program just keeps doing this and will never stop till a capcha is finally entered...skipping all those sweeps.

could not find video, oh well. Moving on!
giveaway 18 is ready!
waiting for box...
could not find box?
could not get result, oh well. Moving on!
giveaway 19 cannot be entered.
giveaway 20 is ready!
waiting for box...
could not find box?
could not get result, oh well. Moving on!
giveaway 21 is ready!
waiting for box...
could not find box?
could not get result, oh well. Moving on!
giveaway 22 is ready!
waiting for box...
could not find box?
could not get result, oh well. Moving on!

My net is so slow that by the time I get to the last page it's pretty much time to start at the beginning again. (mainly due to missed capcha's and the aggravating necessity of sleep)

Perhaps the program could prompt in the console: Restart at Page 1 (Y/n):_

with no being the default. This cuts down on having to deal with the whole 2FA mess again and/or amazon asking for a capcha (if it's been asking for a capcha on the signin page somehow your program ignores it and continues so as soon as it dies on restart you get the guaranteed capcha you have to enter)

This would also work well in combination with the issue requesting to resume from where it left off since they might be towards the end of the pages.

Submit entries when there are videos

Add support for entering giveaways that require you to follow someone.

As suggested in #19, send an email notification when a captcha is presented. This would mean adding email server credentials to the .env file so the user could basically send themselves an email.

I'm on the latest version...noticed it was saying "can't find box, moving on" on every sweep....looked at screen to see it was showing a "sign in" button to right of the item picture. I clicked it and it took me to the switch account screen where I picked my name...it then went back to working. Happened again later. Had to downgrade to 2.7.0.

I received the normal amazon password and capcha screen while the program was running...I attempted to enter the password and capcha but puppeteer kept trying to enter the password screwing up what I was typing. The console showed:

Rudi, you didn't win
giveaway 5 did not exist?
giveaway 6 did not exist?
Whoa, got to re-enter password!
giveaway 7 did not exist?
Whoa, got to re-enter password!
giveaway 8 did not exist?
Whoa, got to re-enter password!
giveaway 9 did not exist?
Whoa, got to re-enter password!

When a giveaway is won, it asks for you to confirm your address. However, there is no way of knowing which items you won short of searching the nodejs terminal for "won" and looking up the item by page and giveaway number, not very practical.

How about implementing a simple write to file for the link of the items won?