[1.0.1.0] Problem: batch downloading stucks for unknown reason about nijiedownloader HOT 18 CLOSED

can you provide step-by-step to replicate this?

fyi, when you start the job, then add new one, it won't be auto started.

from nijiedownloader.

From the start, with no settings file and database file:

• go to options, change root folder and filename format={memberId}\{imageId}_{page}_{imageTitle}_{serverFilename}, AutoSaveBat=on, Save.
• go to login, set a valid account and autologin=on, Login Success.
• go to Batch download, Add. Type=member, member=38, OK, Start.
• it saves the first image, says Saved to: filename / Running.
• nothing else happens.

Log file contains:
2014-03-08 13:01:14,467 INFO [ 1] - Upgrading configuration 2014-03-08 13:01:14,493 DEBUG [ 1] - Proxy= 2014-03-08 13:01:14,495 INFO [ 1] - Nijie Downloader v1.0.1.0 started. 2014-03-08 13:01:16,702 INFO [ 1] - Tracking 0 image(s) 2014-03-08 13:01:53,185 INFO [ 9] - Loggged Out 2014-03-08 13:01:57,413 INFO [ 9] - Logged In 2014-03-08 13:04:09,613 DEBUG [ 14] - Running Member Job: Member ID: 38 StartPage: 1 EndPage: 0 Limit: 0 2014-03-08 13:04:11,070 DEBUG [ 14] - Processing Image:75035 2014-03-08 13:04:12,449 DEBUG [ 14] - Downloading url: http://pic03.nijie.info/nijie_picture/38_20140301142431.png ==> <my root folder>\38\75035__陰毛の陰謀_38_20140301142431.png
EOF. After I close the app, this is added:
2014-03-08 13:07:49,434 INFO [ 1] - Nijie Downloader v1.0.1.0 closed.

Also tried v1.0.0.2 now and it can't even save any file anymore. Log says something about System.Net.WebException and System.IO.IOException.

from nijiedownloader.

Tried v1.0.2.0 - same thing.

• wipe all previous and current settings folders in AppData\Local\Nandaka.
• start the app, don't change any options, enter the login info and press Login.
• go to Batch download, Add, Job type = member, Member = 38, OK., Start.
• it saves the first image into D:\Temp\Nijie, says Saved to: filename / Running.
• nothing else happens.
  (see above, log reports all the same)

from nijiedownloader.

are you using windows xp? Try to update the .Net Framework 4 (repair).

from nijiedownloader.

I'm using Windows 8.1 x64. All .NET Frameworks are updated already, through the Update center. Control panel doesn't really show them in installed list so I see no option to repair anything. I'll test it under Windows XP VM later.

I wonder if this is caused by different deployement method. Could you do a test build with the default installation method?

from nijiedownloader.

I don't have the installer anymore, just extract the archive to any directory and ran it. Tested on Win7 64bit and XP SP3 32bit. I'll upload new version with more detailed logging later.

from nijiedownloader.

That's how I (try to) use it since 1.0.1.0, there is no other way.

Tried it under my XP VM machine and it can't even run:
System.Data.SqlServerCe.SqlCeException: The specified locale is not installed on this machine. Make sure you install the appropriate language pack. [ LCID = 1041 ]

I suppose it requires a Japanese language pack for .NET Framework which is not installed. Tried to install it from http://www.microsoft.com/ja-jp/download/details.aspx?id=23067 but it didn't help at all.

from nijiedownloader.

Try this patch: http://www.mediafire.com/download/44tl900u1azrh12/nijieDownloader.1.0.2.1.patch.7z
Just overwrite the old files.

Regarding XP: you need to set the regional settings -> Language for non-Unicode to Japanese, I think, or at least need to install japanese language pack (so it can show japanese text).

Fyi, my Win7 is running on English.

from nijiedownloader.

Thanks for this.

It now saves all the images, as far as I can see. Tested with member id 38.
Here are brief log details happening while the app was running:
Per item:
2014-03-17 16:19:41,486 DEBUG [ 4] - Downloading url: http://pic03.nijie.info/nijie_picture/38_20140312170650_0.png ==> D:\Temp\Nijie\38 - 76268 - チンポップ君の日常 公式漫画 殺し屋ンポ 暗殺.png 2014-03-17 16:19:44,550 ERROR [ 4] - Failed to save to DB: 76268 System.Data.Entity.Infrastructure.DbUpdateException: An error occurred while saving entities that do not expose foreign key properties for their relationships. The EntityEntries property will return null because a single entity cannot be identified as the source of the exception. Handling of exceptions while saving can be made easier by exposing foreign key properties in your entity types. See the InnerException for details. ---> System.Data.Entity.Core.UpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlServerCe.SqlCeException: An overflow occurred while converting to datetime.
On the last image:
2014-03-17 16:20:49,795 DEBUG [ 4] - Downloading url: http://pic04.nijie.info/nijie_picture/38_20130829200015.png ==> D:\Temp\Nijie\38 - 58951 - ニジエたん ずらし挿入.png 2014-03-17 16:20:52,314 ERROR [ 4] - Failed to save to DB: 58951 System.Data.Entity.Infrastructure.DbUpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.Entity.Core.UpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlServerCe.SqlCeException: A duplicate value cannot be inserted into a unique index. [ Table name = NijieTags,Constraint name = PK_dbo.NijieTags ]
Same things are being logged if all files are already exist (if I run the same job again).

My Windows is russian, but from my experience it seems most environment problems are happening on non-developer-language OS. Non-english OS in this case. Same goes for dates/sizes/numbers formats - they may be different for different regions, e.g. if you try to process "1.05" number under OS where decimals are supposed to look like "1,05", you're gonna have a bad time.

from nijiedownloader.

This should not be the case, as I pass the DateTime type and the SQL CE is supporting native .Net type as far as I know.

Anyway, can you set the Trace DB = true in the Download Settings and the clear the log files, then run it again. I want to see what is the data they being generated.

Also, try to set the concurrent job = 1 and see if this was the cause the race condition when inserting the Tags.

Btw, I tested on Win7 English (no changes on the non-unicode) and WinXP set to Japanese for non-unicode.

from nijiedownloader.

Concurrent job was already = 1. Like I said previously, with 1.0.2.0 I haven't changed any settings, even default download folder.

Enabled Trace DB and repeated member id = 38 batch. Here is the log file: http://pastebin.com/gKuMz1KT

from nijiedownloader.

checking the db trace, somehow the date is being parsed as min time.

Which is weird, on my Win7, it parsed correctly...
https://github.com/Nandaka/NijieDownloader/blob/master/NijieDownloader.Library/Nijie.Image.cs#L136

or because of the extra tailing ==> 2014-03-12 17:06:54に投稿.

I'll fix it later 😄

from nijiedownloader.

Getting similar exceptions in 1.0.3.0, except it does them only for some images, and there is no pattern of its happening:
System.Data.Entity.Infrastructure.DbUpdateException: An error occurred while saving entities that do not expose foreign key properties for their relationships. The EntityEntries property will return null because a single entity cannot be identified as the source of the exception. Handling of exceptions while saving can be made easier by exposing foreign key properties in your entity types. See the InnerException for details. ---> System.Data.Entity.Core.UpdateException: An error occurred while updating the entries. See the inner exception for details. ---> System.Data.SqlServerCe.SqlCeException: An overflow occurred while converting to datetime.
Had 250 of these exceptions after finishing the job list of ~400 members (~20000 images). After running the same list again (took far less time because most images already downloaded) got only 2 of these exceptions. It didn't prevent the files from being correctly downloaded though.

Tested with Concurrent job = 5.

from nijiedownloader.

An overflow occurred while converting to datetime.

This one I'm not sure why it keep happening if using Concurrent Job > 1, I've checked the log file and it can parse and store the work date to the object, but somehow it try to save the date as DateTime.MIN_VALUE to the DB...

from nijiedownloader.

May I ask what is the purpose of the database? It's under 200 kb for me, while my job list is huge. I though it's supposed to remember all downloaded images metadata so next time it knows what to skip. Even if you keep image ids as dwords, the file contains too many zero bytes.

Are you trying to remember server time for the images? In this case, are you sure this is needed? Since a server filename contains the actual upload date for each image, doesn't it mean that updated images would have different filenames too? Furthermore, I've checked and didn't find an option to edit published images. Member may delete an item or add tags only, it seems.

from nijiedownloader.

knows what to skip
Yap (future). For now, it will show image already downloaded in 50% opacity.

Even if you keep image ids as dwords, the file contains too many zero bytes.
it is the default format from MS SQL Compact, not mine. Both MemberId and ImageId is saved as Int.

server time for the images
More to the work time when image is uploaded (it is the time on the image page), so I can easy detect if there is changes on the image if possible. Also maybe another user will ask for this feature (at least what I get from PixivUtil).

anyway, there is an option to turn off the db save on the settings -> downloads tab.

from nijiedownloader.

Thanks for info.

Still, you may want to consider fixing the logic for detecting changes in images. If Nijie doesn't allow anyone to edit already uploaded files (that was my conclusion after checking), then it's useless.

I'll explain the database size thing I have:
My Database.sdf size is 196608 bytes now. My batch list results in downloading of ~19000 images.
After manually removing all zero [00] bytes from the file it reduced in size to 12682 effective bytes. This clearly means it doesn't hold the info about every downloaded image.

Checked with http://sourceforge.net/projects/compactview/ - my file only contains 1 image id and 1 member id with some info and urls, plus app-related info.

My guess is it isn't working as expected, or it only saves information and actual thumbnail data for tasks which I don't perform, like fetching images and member pages. I only use batch download for now.

from nijiedownloader.

I don't use the date for detect the changes for now, only the id.

Size thing
I suspect this is to avoid fragmentation if you have a lot of data, I won't fix this.
Also, I only save the info on db only when it successfully download the image from job.

Refer to:https://github.com/Nandaka/NijieDownloader/blob/master/NijieDownloader.UI/MainWindow.JobHelper.cs#L276

I still checking for why the value of DateTime is changed to DateTime.MinValue

from nijiedownloader.