licoffe / poe-stash-indexer-ng Goto Github PK

Lightning coil : http://poe-rates.com/index.php?league=Legacy&item=Lightning%20Coil&interval=1h
Windripper : http://poe-rates.com/index.php?league=Legacy&item=Windripper&interval=7d
Voidheart : http://poe-rates.com/index.php?league=Legacy&item=Voidheart&interval=7d

Shavronne Wrappings display the stats for 6Link normal version in addition to the foil version data : http://poe-rates.com/index.php?league=Legacy&item=Shavronne%27s%20Wrappings&interval=7d

Running on Debian 8.6 64-bit. MySQL user/db/privs all correct. Turns out the issue is you provide no empty 'data' dir in your skeleton, while your main.cpp sets the download_dir to ./data/. Might want to create an empty data dir in the repo or at least mention it in the README.

# Database credentials
DB_HOST     = tcp://127.0.0.1
DB_PORT     = 3306
DB_USER     = redacted
DB_PASS     = redacted
DB_NAME     = POE
DB_DATA_DIR = /var/lib/mysql

Segfaults in /var/log/messages:

Jan  5 13:24:10 REDACTED kernel: [  950.516414] indexer[13408]: segfault at 0 ip 00007f2e107ad9e0 sp 00007f2e067fb230 error 4 in libc-2.19.so[7f2e10743000+1a1000]
Jan  5 13:26:27 REDACTED kernel: [ 1088.254572] indexer[13813]: segfault at 0 ip 00007faec44209e0 sp 00007faeba5ef230 error 4 in libc-2.19.so[7faec43b6000+1a1000]
Jan  5 13:30:48 REDACTED kernel: [ 1348.780972] indexer[13903]: segfault at 0 ip 00007fca2f19a9e0 sp 00007fca293ef230 error 4 in libc-2.19.so[7fca2f130000+1a1000]

Backtrace from gdb, this is how I found out it had to do with a problem writing a file:

GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./indexer...done.
(gdb) run
Starting program: /root/POE-Stash-indexer-NG/indexer
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff2f44700 (LWP 13823)]
[Thread 0x7ffff2f44700 (LWP 13823) exited]
2017-01-05 13:26:38 > [main] Connected to DB in 0.036458 sec
2017-01-05 13:26:38 > [main] Checking last downloaded chunk
2017-01-05 13:26:38 > [main] New indexation:
[New Thread 0x7ffff2f44700 (LWP 13824)]
2017-01-05 13:26:38 > [mod_loop] Waiting for mods to insert
[New Thread 0x7ffff2537700 (LWP 13825)]
2017-01-05 13:26:38 > [requirement_loop] Waiting for requirements to insert
[New Thread 0x7ffff1d36700 (LWP 13826)]
2017-01-05 13:26:38 > [property_loop] Waiting for property to insert
[New Thread 0x7ffff1535700 (LWP 13827)]
2017-01-05 13:26:38 > [socket_loop] Waiting for socket to insert
[New Thread 0x7ffff0d34700 (LWP 13828)]
[New Thread 0x7fffe3fff700 (LWP 13829)]
[New Thread 0x7fffe37fe700 (LWP 13830)]
2017-01-05 13:26:38 > [processing_loop] Waiting for files to process
[Thread 0x7fffe37fe700 (LWP 13830) exited]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff0d34700 (LWP 13828)]
__GI__IO_fwrite (buf=0x7fffd8015a40, size=1, count=1694, fp=0x0) at iofwrite.c:41
41      iofwrite.c: No such file or directory.
(gdb) bt
#0  __GI__IO_fwrite (buf=0x7fffd8015a40, size=1, count=1694, fp=0x0) at iofwrite.c:41
#1  0x00007ffff79673c8 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#2  0x00007ffff79876cc in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#3  0x00007ffff7987a02 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#4  0x00007ffff7981ad3 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#5  0x00007ffff797c3df in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#6  0x00007ffff7986749 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#7  0x00007ffff7986f91 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#8  0x00007ffff797ddeb in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#9  0x000000000041b5de in download_JSON(std::string) ()
#10 0x0000000000423c98 in download_loop() ()
#11 0x00007ffff73ec970 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#12 0x00007ffff7bc70a4 in start_thread (arg=0x7ffff0d34700) at pthread_create.c:309
#13 0x00007ffff6b5c62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb) quit
A debugging session is active.

        Inferior 1 [process 13819] will be killed.

Quit anyway? (y or n) Y```

^^

Passwords with capital letters for MySQL access aren't properly parsed in the code. The code runs fine when used but MySQL returns a generic acces denied error. It took me quite some time to figure this out sadly as I didn't expect the tool to be the problem. The upside is however that I can confirm this behavior is present on Ubuntu 12.04, 14.04, 16.04 and Debian 8.

If this is intentional a mention in the readme or config file would be appreciated.

When the various loops write the temporary files it's done this way:

path << DB_DATA_DIR << "/mods_" << random_id(10) << ".txt";
mod_file.open( path.str());

However, when the parser tries to read them for DB insertion, this happens:

threaded_insert( "LOAD DATA CONCURRENT INFILE '" + mod_file + "' REPLACE INTO TABLE Mods FIELDS TERMINATED BY ',' ENCLOSED BY '\"' ESCAPED BY '\"' LINES TERMINATED BY '\n'" );
path << DB_DATA_DIR + "/" + mod_file;

This causes all insertions to fail as it will attempt to load the files from the same folder as the indexer itself, while the user might have specified another location.

When processing json, stashes with a NULL accountName are skipped (here).

    // If there is a valid accountName value
    if ( array.HasMember( "accountName" ) && !array["accountName"].IsNull()) {
        assert(array["accountName"].IsString());
        account_name = array["accountName"].GetString();
    } else {
        // If not, skip this stash tab
        continue;
    }

A NULL accountName indicates that the stash no longer exists/is removed, as Novynn mentioned here: https://www.reddit.com/r/pathofexiledev/comments/54b20b/empty_accountcharacter_names/

If I'm interpreting correctly, the relevant stash+item records should be removed from the database, because the items in that tab should also no longer exist.

On mobile, clicking on an element of the League and Time interval dropdown menus does not work. It clicks on the element behind because the menu gets closed.

Tested on Android 6.0.1, Chrome 56, Firefox 52

On the screenshot, i cannot select Hardcore Legacy, Legacy, Standard.
https://cloud.githubusercontent.com/assets/5748216/24043549/6a500170-0b17-11e7-9cd0-7e02f1e311c3.png

hey,

I currently installed everything under Ubuntu 16.04
But after a while (10hours+) the mod-loop is running slower and slower until somewhen it can not catch up anymore.
Which means that a big queue is there in the end.
If I end the program, the mod-queue is needing hours until it is sorted in.

I am currently running a server with 16gb ram and 4-core with 3.4ghz, which is never fully used.
But I am using only an HDD-Storage and not an SSD. Is that the whole problem, that Mysql-DB is to slow? or is there another reason?
Thanks for your help

Hi Licoffe,

Quick question. Is there a way to save my searches on poe-rates.com/live? What I mean is, if I were to clear my cache, browser history, etc I would have to recreate everything from scratch. Is there a way to export the searches I have built so far so that I wouldn't lose them in that case?

Thanks,
Tim

The modName column in the Mods table is too short, it is currently varchar(128).

This mod is over 128 characters (147?) and I don't know if it is even the longest mod name.

With at least # Intelligence Allocated in Radius, Fireball Projectiles gain Area of Effect Radius as they travel farther, up to #% increased Radius

SQL errors when trying to insert this because it is too long, I have changed my local table to varchar(256) for now with:

alter table Mods change column modName modName VARCHAR(256);

I have now been looking at getting this up and running on a windows machine using Cygwin. I've got all the dependencies installed but I encounter a strange error with cURL as seen below that I can't seem to fix.

In file included from /usr/include/curl/curl.h:2547:0,
from main.cpp:29:
/usr/include/curl/multi.h:155:40: error: ‘fd_set’ has not been declared
fd_set read_fd_set,
^~~~~~
/usr/include/curl/multi.h:156:40: error: ‘fd_set’ has not been declared
fd_set write_fd_set,
^~~~~~
/usr/include/curl/multi.h:157:40: error: ‘fd_set’ has not been declared
fd_set exc_fd_set,
^~~~~~
main.cpp: In function ‘int main(int, char)’:
main.cpp:1824:26: error: aggregate ‘main(int, char*)::sigaction sig_int_handle ’ has incomplete type and cannot be defined
struct sigaction sig_int_handler;
^~~~~~~~~~~~~~~
main.cpp:1827:47: error: ‘sigemptyset’ was not declared in this scope
sigemptyset( &sig_int_handler.sa_mask );
^
main.cpp:1829:51: error: invalid use of incomplete type ‘struct main(int, char**)::sigaction’
sigaction( SIGINT, &sig_int_handler, NULL );
^
main.cpp:1824:16: note: forward declaration of ‘struct main(int, char**)::sigaction’
struct sigaction sig_int_handler;
^~~~~~~~~
make: *** [Makefile:3: all] Error 1

(I left it out of the code tag as the format is then horrible to read)

The three Grand Spectrum jewels data is mixed up and they can't be separated : http://poe-rates.com/index.php?league=Legacy&item=Grand%20Spectrum&interval=12h

Hey there,

I noticed every time write_parsed_to_file happens the indexer displays "Could not write to file" and just continues. The speed of my inserts is down to an average of 41/s after half an hour running. Just in case, I tried to chmod all 4 folders it uses (which I created manually) to 777, then chowning them to the mysql user and group, but of course this doesn't look to be the right solution. What permissions exactly do I have to use or, if this is caused by something else, what do I have to change? All other tables are being populated correctly. (Also, my data dir is set do /var/lib/mysql which is the default for Linux - as far as I can see this is only used for these 4 tables, can I safely change it to something like the home directory or will that screw up the periodic database inserts?)

Thanks in advance.

By the way, really useful tool. Love it :)

When the user does a search and switches between desktop and mobile layouts by reducing/enlarging the window, the graphs aren't displayed.

It happens in both scenarios

• Load mobile layout -> enlarge window to desktop layout
• Load desktop layout -> reduced window to mobile layout

The screenshot shows the first scenario

https://cloud.githubusercontent.com/assets/5748216/25332618/695d537e-28e7-11e7-9919-e19750466f84.jpg

Hey,

Sadly this tool doesnt work anymore with a fresh database. I think it has something to do with the new introduced jewels or shaper/elder mods. Or is there another Problem? i didnt change anything and it worked fine for me. I think its only 2 lines of code for you.

greeting

MySQL Install has secure_file_priv in /var/lib/mysql-files. I added this to the config and was surprised my files ended up in /var/lib/mysql.

I'm not into regex at all, otherwise I would fix it myself

Running on Ubuntu 16.10 mysql 5.7.

I'm receiving error:

ERR: SQLException in main.cpp
(print_sql_error) on line 215
ERR: Data truncated for column 'modValue1' at row 19
(MySQL error code: 1265
, SQLState: 01000 )
LOAD DATA CONCURRENT INFILE 'mods_0.txt' REPLACE INTO TABLE Mods FIELDS TERMINATED BY ',' >ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '
'

Looking at the line causing the error, I notice it is the first line to have no value specified for modValue1:

"5e968f419af74b6c673927fd0f083ab3599573d2879d1f6e9c27085a761ea9fa","Immune to Curses during Flask effect;Removes Curses on use","","","","","EXPLICIT","5e968f419af74b6c673927fd0f083ab3599573d2879d1f6e9c27085a761ea9fa2"

Manually editing in a "0" for modValue1 and manually executing the SQL query resolves the error for that line, but pushes the error to the next instance of a blank valued modValue1.

It seems like LOAD DATA INFILE isn't handling empty values correctly, source

Ubuntu 16.10 MySQL5.7

ERR: SQLException in main.cpp
(print_sql_error) on line 215
ERR: Can't get stat of '/home/mysql/POE/mods_0.txt' (Errcode: 2 - No such file or directory)
(MySQL error code: 13
, SQLState: HY000 )
LOAD DATA CONCURRENT INFILE 'mods_0.txt' REPLACE INTO TABLE Mods FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY '
'

Before the error occurs, MySQL is executing query LOAD DATA CONCURRENT INFILE 'mods_0.txt'... and it seems like the moment the query completes the indexer throws the error. I think this is the very first 'LOAD DATA...' for mods on an empty table and it completes successfully (130k rows added, ~400s). The mod queue length is 2.

After the error my DB_DATA_DIR contains mods_1.txt and no mods_0.txt

Let me know what other info would be useful.