nandaka / danboorudownloader Goto Github PK

In Main tab, when I change some provider settings like login@password in FormProvider and click on Save --- changes are not saved.
I had to edit DanbooruProviderList.xml manually.
Think this occurs because that new feature --- per-provider tags.xml, because in DanbooruProviderList.xml there are new string on each provider: HasPrivateTags & ProviderTagCollection; maybe it interferes (sorry, I know that you know it, just trying to describe the details of my point of view).

Anyway thank you for adding per-provider tags.xml!

Recently, I've been getting the following errors when using batch mode to download from SanCom:

[DoBatchJob] Error: The remote server returned an error: (522) Unassigned.
Stack Trace:
at System.Net.WebClient.DownloadFile(Uri address, String fileName)
at DanbooruDownloader3.FormMain.DoDownloadBatch(String targetUrl, DanbooruBatchJob job, DanbooruPost post, String filename) in C:\UserData\irawanad\Dropbox\DanbooruDownloader3\DanbooruDownloader3\DanbooruDownloader3\FormMain.cs:line 1166
at DanbooruDownloader3.FormMain.DoBatchJob(BindingList`1 batchJob) in C:\UserData\irawanad\Dropbox\DanbooruDownloader3\DanbooruDownloader3\DanbooruDownloader3\FormMain.cs:line 995

2014-07-13 02:38:44,248 ERROR - Input string was not in a correct format.
System.FormatException: Input string was not in a correct format.
at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
at DanbooruDownloader3.FormProvider.GetValues(Int32 index) in C:\UserData\irawanad\Dropbox\DanbooruDownloader3\DanbooruDownloader3\DanbooruDownloader3\FormProvider.cs:line 194

Error Resolving FileUrl (1 of 1): The remote server returned an error: (522) Origin Connection Time-out. Wait for 60s.
System.Net.WebException: The remote server returned an error: (522) Origin Connection Time-out.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at DanbooruDownloader3.FormMain.ResolveFileUrlBatch(ExtendedWebClient _clientPost, DanbooruPost post) in C:\UserData\irawanad\Dropbox\DanbooruDownloader3\DanbooruDownloader3\DanbooruDownloader3\FormMain.cs:line 1203

There are other instances of this error. Do you need to know all of them?

I keep getting

Download List
---------------------------
Der Remoteserver hat einen Fehler zurückgegeben: (403) Unzulässig.
---------------------------
OK   

errors when I try to search in danbooru. Here is a screenshot of the package in Wireshark.

The package seems all right. Clicking the link at the bottom or entering host and URI manually into the browser works just well.
I also compared the package with a one for konachan.com - it's almost similar (except for the host) but it works there.
The user agent can be activated when the checkbox is active but this effectively does nothing (except that the user agent is sent).

It seems you do not send the login or passwort information entered in "Edit". FireFox sends them in the Cookie field. I don't know if you already did this before and it's just a regression.
The request looks like this:

GET /post/index.xml?tags=yakumo_ran HTTP/1.1
Host: danbooru.donmai.us
Accept: */*
Cookie: login=FichteFoll; pass_hash=<some_x-digit_hash>;

I tried that on hurl (with the correct hash I read from the FireFox package) and it worked. But it didn't without.

As it seems that login information is required for danbooru you should probably show a message box warning about empty login information. Maybe introduce a new tag in the DanbooruProviderList.xml.

A progressbar for the total progress (all images) would be nice. Currently it only shows one for the current file.

Edit: And the display of the current download speed is also appreciated.

There already is a (clickable) direct link for the image but at times I like to open the image's post (e.g. for reference, mark as favorite, edit tags ...). It can be available via context menu for example.

I tried to download saved download list, program asks for a folder to save and then throws an exception.
Folder to download to is "c:\dd" (tried another ones without success).
OS - windows XP sp3 and windows 7 x64 russian system language.

Exception data:
http://pastebin.com/LRnPKC8c
Config:
http://imgur.com/SpGea6m
Saved download list XML:
http://pastebin.com/JJa88iee

Pretty much what the title says. Applies only to the "Url" column since there is no referrer.

Pretty much as the title says. Easily reproducible, at least on Sankaku.

Simply begin a batch download, and give it a different page number to start with; say, 5 or 10. When you begin the patch download, it will show "Getting List for Page 0" and then increment by 1 at every new page - but the logfile shows that the downloader is indeed pulling the right pages, so this is merely a display issue in the Full Batch Mode tab under the Status section.

http://www.pixiv.net/member_illust.php?id=20352

It downloads 10 images and says there is only 1 page but on website it says "31件" images and two pages

Log:
2013-01-19 00:13:46,079 - PixivUtil20121215 - INFO - ###############################################################
2013-01-19 00:13:46,079 - PixivUtil20121215 - INFO - Starting...
2013-01-19 00:13:46,079 - PixivUtil20121215 - INFO - Using Username: *******
2013-01-19 00:13:46,079 - PixivUtil20121215 - INFO - logging in with saved cookie
2013-01-19 00:13:46,079 - PixivUtil20121215 - INFO - Trying to log with saved cookie
2013-01-19 00:13:49,502 - PixivUtil20121215 - INFO - Logged in using cookie
2013-01-19 00:13:51,611 - PixivUtil20121215 - INFO - Member id mode.
2013-01-19 00:13:58,266 - PixivUtil20121215 - INFO - Processing Member Id: 20352
2013-01-19 00:13:58,282 - PixivUtil20121215 - INFO - Member Url: http://www.pixiv.net/member_illust.php?id=20352&p=1
2013-01-19 00:14:00,766 - PixivUtil20121215 - INFO - Member_id: 20352 complete, last image_id: 3095802

window direct log:
PixivDownloader2 version 20121215
https://nandaka.wordpress.com/tag/pixiv-downloader/
Reading V:\Program Files\PixivD\config.ini ...
done.
Creating database... done.
Using Username: *******
logging in with saved cookie
Trying to log with saved cookie
done.
PixivDownloader2 version 20121215
https://nandaka.wordpress.com/tag/pixiv-downloader/

1. Download by member_id
2. Download by image_id
3. Download by tags
4. Download from list
5. Download from online user bookmark
6. Download from online image bookmark
7. Download from tags list
8. Download new illust from bookmark
9. Download by Title/Caption
10. Download by Tag and Member Id

11. Download Member Bookmark

d. Manage database
e. Export online bookmark
x. Exit
Input: 1
Member id: 20352
Start Page (default=1):
End Page (default=0, 0 for no limit):
Processing Member Id: 20352
Reading V:\Program Files\PixivD\config.ini ...
done.
Page 1
Member Url: http://www.pixiv.net/member_illust.php?id=20352&p=1
Member Name : ???
Member Avatar: http://i1.pixiv.net/img03/profile/tateha/5646840.jpg
Member Token : tateha
#1

Already downloaded: 29155508
#1

Already downloaded: 29022590
#1

Already downloaded: 26879813
#1

Already downloaded: 20830865
#1

Already downloaded: 19781678
#1

Already downloaded: 16862443
#1

Already downloaded: 8160208
#1

Already downloaded: 6427420
#1

Already downloaded: 3362027
#1

Already downloaded: 3095802
Last Page
Done.

cfg file:
[Settings]
proxyaddress =
useproxy = False
useragent = Mozilla/5.0 (X11; U; Unix i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1
debughttp = False
userobots = False
filenameformat = %member_id% (%member_token%)%urlFilename% - %title%
filenamemangaformat = %member_id% (%member_token%)%image_id%%urlFilename%
timeout = 60
uselist = False
processfromdb = False
overwrite = False
tagsseparator = ,
daylastupdated = 7
rootdirectory = C:\DL Image Packs
retry = 3
retrywait = 5
createdownloadlists = False
downloadlistdirectory = .
irfanviewpath = C:\Program Files\IrfanView
startirfanview = False
startirfanslide = False
alwayscheckfilesize = False
checkupdatedlimit = 0
downloadavatar = False
createmangadir = False
usetagsasdir = False
useblacklisttags = False
usesuppresstags = False
tagslimit = -1
writeimageinfo = False

[Pixiv]
numberofpage = 0
r18mode = False

The 'Process Deleted Post' option under 'Application Settings' doesn't download deleted posts.

As the title says, this symbol, when used as a tag search, is not properly encoded in the search. As a result, this symbol (and possibly others) are skipped when this sort of search is done in either batch or manual search mode.

Test case:

1. Do either a regular booru search, or a batch download on Sankaku Complex, Danbooru, or Gelbooru. (Presumably any booru, but it’s confirmed busted for these three).
2. Search for the series “Blood+”.
3. Watch as the downloader pulls up any sort of image with blood in it, due to it ignoring the symbol - as a result, it pulls up all images that have "blood" as a tag, but not the expected "Blood+."

The title says it all. Both of them return a 403 error. There is a work around, but it might use quite a bit of bandwidth. Getting all of the tag pages from http://idol.sankakucomplex.com/tag/index and http://chan.sankakucomplex.com/tag/index . Then parse all of them to get the tags and reorganize them in the xml format.

Have added a batch download with a tag on sankaku and told it to start downloading and it only downloads the first 20 then stops. Have tried setting the limit to say 300 for testing and still on first 20.

A whitelist for tags would be good, so that certain tags are always in the filename. For example a picture has too many tags and so the filename will always be cutted: 1girl blue_hair aqua_eyes foresh.jpg but if we have foreshortening in the whitelist so name would be: 1girl blue_hair foreshortening aqua_e.jpg so danbooru downloader would just cut tags which are not on the whitelist. Instead of the ones which someone wants to keep. Excuse my bad english.

When tried to download flash animation from Sankaku Complex, got this (example):

[DoBatchJob] Download skipped, ID: 1875456 No file_url, probably deleted

Download in Main tab doesn`t solve this problem :(
Other boards works fine.

Sankaku supports https now! Can you integrate that in the next update? Right now I can only edit DanbooruProviderList.xml to get files from https addresses, but it will still download the full images from http addresses.

When writing a query the program shows the corresponsing query URL in the statusbar. However, the ampersands don't seem to be escaped (or the window style is not set) so that they are in fact just not displayed.

Steps to reproduce:

1. Chose a provider
2. Enter a tag to search for
3. Enter a limit or a page number (or both)

Thank for updating SankakuParser, but now rating shows like “safe”, “questionable” and “explicit”, not “s” & “q” & “e” as before and only on Sankaku (not other imageboards).
Could you add option to auto-rename “long” rating to one letter, because I use rating in filename, and it causes some problems with filename length like auto-crop character names (I use %character% tag at the end, including artist, copyright, id, rating and provider). Extended the filename helps, but it is uncomfortable to see “long” rating on Sankaku (only). Sure I can use batch-rename software, but it not fix "auto-crop".

Add e-shuushuu.net provider support

I had the "Illegal characters in path" error when downloading SWFs from Sankaku. Details below:

I was using Full batch mode, and I set the filename format to %id%+%artist%+%copyright%+%character%+%md5%+%tags%. I have ignored those tags like "4:3_aspect", so the problem was unlikely due to the illegal characters in these tags.

Someone else also had the problem, this is what he said:

The Sankaku Parser Seems to be broken again, However Only for .swf Files. Hopefully you can fix it soon =). This is what I have my current “Filename Format” set to “%searchtag%%id%-%md5%” and Im using Full Batch Mode.

Get the following error in batch mode. Only with Sankaku.

Version 3.2014.02.04

[DoBatchJob] Downloading list: http://chan.sankakucomplex.com/post/index?tags=buddy_complex+rating:safe&limit=500&page=1&login=username&password_hash=passwordhash
[DoBatchJob] Error: There is an error in XML document (86236, 61).
Inner: '�', hexadecimal value 0x0E, is an invalid character. Line 86236, position 61.
Stack Trace: 
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at DanbooruDownloader3.DAO.DanbooruTagsDao..ctor(String xmlTagFile)
   at DanbooruDownloader3.Entity.DanbooruProvider.LoadProviderTagCollection()
   at DanbooruDownloader3.Helper.ParseTags(String p, DanbooruProvider provider)
   at DanbooruDownloader3.Engine.SankakuComplexParser.Parse(String data, DanbooruSearchParam searchParam)
   at DanbooruDownloader3.DAO.DanbooruPostDao..ctor(Stream input, DanbooruPostDaoOption option)
   at DanbooruDownloader3.FormMain.GetBatchImageList(String url, String query, DanbooruBatchJob job)
   at DanbooruDownloader3.FormMain.DoBatchJob(BindingList`1 batchJob)

I can't attach anything but a picture but the batch file looks like below for Sankaku.

  <DanbooruBatchJob>
    <Provider>
      <Name>Sankaku Complex</Name>
      <DefaultLimit>20</DefaultLimit>
      <HardLimit>1000</HardLimit>
      <Preferred>Html</Preferred>
      <QueryStringXml />
      <QueryStringJson />
      <QueryStringHtml>/post/index?%_query%</QueryStringHtml>
      <Url>http://chan.sankakucomplex.com</Url>
      <UserName>username</UserName>
      <Password />
      <UseAuth>true</UseAuth>
      <PasswordSalt>choujin-steiner--%PASSWORD%--</PasswordSalt>
      <PasswordHash>passwordhash</PasswordHash>
      <BoardType>Danbooru</BoardType>
      <TagDownloadUseLoop>true</TagDownloadUseLoop>
    </Provider>
    <ProviderName>Sankaku Complex</ProviderName>
    <SaveFolder>%searchtag%\%character%\%md5%</SaveFolder>
    <TagQuery>buddy_complex</TagQuery>
    <Limit>500</Limit>
    <isCompleted>false</isCompleted>
    <isError>false</isError>
    <Rating>rating:safe</Rating>
    <StartPage>1</StartPage>
  </DanbooruBatchJob>

I don't know when or how you want to implement #5, but as it is now the column order is not saved when I re-arrange them. Would be pretty nice if it was. Though, completely irrelevant when you put all the information into one column.
#10 related.

Same type of error as Issue #25 - Error now is about "doctype".

'doctype' is an unexpected token. The expected token is 'DOCTYPE'. Line 1, position 3.

This still a Danbooru thing they need to change, or is this simply a slight fix needed to search for a case-insensitive phrase?

Rule34.paheal.net simply doesn't work, I have tried changing the DanbooruProviderList.xml and it either simply stops DanbooruDownloader from working or when I try to download images it leaves me with a bunch of dumplists from r34.

Then we have lolibooru which was change to allthefallen.org/gallery/

Well, this is kind of a design issue and maybe not possible with Win32API which you seem to be using mainly (I guess WPF would be capable of this):

Currently there are two large (in height) columns and 11 one-line columns in the list (including the checkbox). That's a huge waste of space.
I though of displaying the one-liners vertically while maintaining row of the actual image (if thumbnal is enabled).
The tags columns is approximately 3-6 lines with default columnsize. Most thumbnails are 9 lines tall.

I thought of a workaround where you would have to merge a few of the existing one-line columns and display them as one columns seperated by linebreaks (if the control supports that). I don't exactly know how this would look and feellike.

That allows a much smaller GUI in width while still being able to see the most relevant information (and without not noticing that there are even more columns which could be important - #4).
Let me know what you think.

File url doesn't auto-resolve when image added via right click in Main Tab (Sankaku)

Use Loop check box doesn't work on tag download.

Been having this problem for at long time on e621.net.
The error message "Error while saving pic" will display.

I'm attempting to blacklist several kinds of tags at once, but I can't get the program to ever ignore more than the first tag entered. If I swap the order of items around, it will always block whatever is first, but no others.

I have tried three different methods of entering tags to blacklist (one tag per line, tags separated by spaces, and tags separated by a comma, then a space), but none of them seem to work.

Well, on every other tab (window) this would open the jump list with at least "close window", "pin to the taskbar" and the applications name to restore it, but with Danbooru Downloader nothing happens at all.

Well, I tend to save my images to the same directory and I don't think I should have to enter that folder every time I start the program.

So now after the latest update I cannot download tags from any site including sankaku....

The auto completition of tags is kind of nice if you ask me, but sadly this only works for the first tag in the input field; the second and third ones just don't trigger it.

For example you can add an amout of spaces in the tags field, they will be parsed into ?tags=++++++yakumo_ran++++. Or when you enter characters into the page or limit field (?tags=yakumo_ran&limit=ss).

The API can handle that and just steps back to the default values but it's not really necessary.

Would be better if log files collected not in the parent folder, for example subfolder \logs would cope with this better :) .
And if unchecked “Enable Logging” it would be nice just do not create .txt files.
Thanks.

Let's say you do a batch download for a given tag... and then find that something you didn't want snuck in there. No problem, you add that to your blacklist, and you won't be getting anything else with that tag in the future.

The problem is, of course... those other images that you may not want are still in there. And you probably don't have it list every generic tag in the filename...

What I propose is simple. If there's a batch download running (it could work for single images too, I suppose, but it's most useful for batches), and the file with the blacklisted tag is already existing, have it check against the tags for the image anyway, and if there is a blacklisted tag for that image, have it prompt the user if they would like to delete the image, or possibly have it just keep track of which images are on the blacklist until after the batch job is complete, so that the user can see what images they already have that the program is asking to delete, and it specifies what tag(s) are causing it to ask "Can I delete these?"

This should make it easier on those who have rather large collections of rather generic tags to ensure it's as specific as possible to what they like.

how can I replace tags? If the copyright tag is not available, I’d like to use the artist tag, is that possible?

Would it be at all possible to add multiple tags to a full batch job but have each tag separated by some character so that they all get added as separate jobs but using the same provider(s) and folder/naming structure?

The GUI does not respond to messages (events) when you are fetching the image list (e.g. when you want to resize the window). This can take a while when you have a limit of about 100 (default for gelbooru). Also you can't cancel that process (and you can't cancel the load of preview images).

I don't know which part is lagging here but the application consumes 25% CPU (quad-core) while it's not responding.

Can you add a option to wait for a few seconds after every command (download/skip) in Batch Mode? This would be helpful in avoiding (429) Too Many Requests errors returned by some danbooru-based websites.

Usually a MsgBox on Windows already makes the window flash (if it's modal) but yours does not. Would be much more visible that the downloads have finished.

All in title, will be great to add this parser.

%width% & %height% for "original image" would be great to, to sort WP pictures directly.

429 error appears when I try download tags from Sankaku. Only ~200 pages loaded and then boom! 429 -- too many requests. Can you add retry ability in Download Tags form, like in main window? Or ability to choose from which page download start, and then checking filenames mask tags.xml.{number}.!tmp , then merging it. Think it harder doing that than try to understand my writings :)
Thank you.

This may potentially be tricky to implement due to the different styles of Booru software, but it'd be really nice to be able to search and download things by pool ID number.

An example syntax for this to make sure it's not accidentally downloading something else could be pool:nnnnn where n is a number. I don't think any booru out there has more than a five-digit pool (Sankaku Complex is at about 9200 as of the time of this post), but it could be good to have it able to take more digits than that for future growth.

the label "tag" on line 3 position 6 does not match the end tag of "user" line 3 position 86

Cannot deserialize type ‘DanbooruDownloader3.Entity.DanbooruProvider’ because it contains property ‘HasPrivateTags’ which has no public setter.

************** Exception Text **************
System.InvalidOperationException: Cannot deserialize type ‘DanbooruDownloader3.Entity.DanbooruProvider’ because it contains property ‘HasPrivateTags’ which has no public setter.
at System.Xml.Serialization.TypeScope.GetSettableMembers(StructMapping mapping, ArrayList list)
at System.Xml.Serialization.TypeScope.GetSettableMembers(StructMapping structMapping)
at System.Xml.Serialization.TypeScope.GetSettableMembers(StructMapping mapping, Dictionary`2 memberInfos)
at System.Xml.Serialization.XmlSerializationReaderILGen.WriteLiteralStructMethod(StructMapping structMapping)
at System.Xml.Serialization.XmlSerializationReaderILGen.GenerateMethod(TypeMapping mapping)
at System.Xml.Serialization.XmlSerializationILGen.GenerateReferencedMethods()
at System.Xml.Serialization.XmlSerializationReaderILGen.GenerateEnd(String[] methods, XmlMapping[] xmlMappings, Type[] types)
at System.Xml.Serialization.TempAssembly.GenerateRefEmitAssembly(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, Evidence evidence)
at System.Xml.Serialization.TempAssembly..ctor(XmlMapping[] xmlMappings, Type[] types, String defaultNamespace, String location, Evidence evidence)
at System.Xml.Serialization.XmlSerializer..ctor(Type type, String defaultNamespace)
at DanbooruDownloader3.FormMain.btnLoadList_Click(Object sender, EventArgs e)
at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.ButtonBase.WndProc(Message& m)
at System.Windows.Forms.Button.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

DownloadTagsForm doesn't use proxy settings.

I have a list of image IDs here and I want to batch-download them (rather I want to view them in the program with the thumbnails and select a few to download). I scanned the code a bit and it seems you really only support XML and JSON-formatted "lists" in a specific format that is built like a query and some additional stuff.

I would like to add a list list of simple IDs to the program (while selecting the provider manually or something) and the other information can be retrieved by the program. That would be really nice 'cause don't exactly want to determine which nodes are actually needed and there is no real documentation (at least I couldn't find one).

Getting this error when I try and download tags from Sankaku Complex, I can manually load that url though...