Comments (7)
Content-Length Filesize: -1
The application unable to get the filesize, so it cannot compare the filesize. The default action is download it anyway.
Is the downloaded file is the same? If yes, I can do post-checking.
from nijiedownloader.
Yes, they are exactly same. Like I've said previously elsewhere, the website doesn't allow members to modify uploaded pictures (afaik), meaning that changed files are only possible case for avatars.
A post-check would do the thing I guess, it would probably need to re-do the online filesize check if returned value is 0 or less.
Until this problem is addressed somehow, it seems a better idea for me would be to just disable overwriting completely, provided I don't care about updated avatars.
FYI, from the way how avatar changing works, you can see that updated image has a different filename on the server. The added "13xxxxxxxx" part is a timestamp in hex format. Even though the scheme is different for uploaded images, you can still see it serves the same purpose - a timestamp. This means it's possible to find changed/updated files by comparing filenames, rather than filesizes. In case a user chose to use {serverFilename} you could monitor the changes like this while still not using the database. This scheme used on the server may or may not be changed in future, but it may still provide valid results with proper use.
Another possibility is to write actual modified-time as creation time for all downloaded files from corresponding server responses, but this may create an issue with timezones changed on the user pc after the file was saved. This may also be more complex to implement.
from nijiedownloader.
try this one: http://www.mediafire.com/download/jsa9ftcezrx08dq/nijieDownloader.1.0.3.2-patch.7z
from nijiedownloader.
Thanks, will keep an eye on the log with this version and overwriting enabled.
While it's still running now, noticed the following things:
Content-Length Filesize: -1
is still happening, but it sometimes being followed byContent-Length Filesize:
with different size. Although I can't say if it refers to the same image (due to concurrency), but I don't seedifferent size
messages anymore in the log. There are now things like this happening:
2014-04-16 10:50:17,627 DEBUG [ 9] - Content-Length Filesize: -1 2014-04-16 10:50:17,681 DEBUG [ 8] - Content-Length Filesize: -1 2014-04-16 10:50:18,087 DEBUG [ 11] - Content-Length Filesize: 1805154
If numbers in [ ] refer to different threads, I find it a bit strange.- Things like this started happening:
Compression Enabled and Identical size: xxxxxx, deleting temp file...
It may be related to different changes and features, not sure. - #17 happened again and was fixed with Stop/Start sequence.
So far it looks like it doesn't try to do any redundant dupes.
By the way, I've noticed that my daily log size got decreased from ~20mb to ~7mb after disabling overwriting. I think it would really help to switch from filesize check to something else for regular images. Right now it looks like in case of changed/updated image under the same image id the app won't even be able to detect it as already downloaded, since the image file will have different name - then it will be downloaded as a new file and the old file under same id will never be touched again.
from nijiedownloader.
Content-Length Filesize: -1
I have no control for this one, as the values come from the server. If no Content-Length header, it will be filled to -1.Compression Enabled and Identical size: xxxxxx, deleting temp file...
yah, forgot to update the log message.#17
Still not sure what is the cause, doing refactoring to see the flow better.
For file size check, I need to depend on the filename and the file size, as the filename format is customize able, so I cannot depend on the server filename, unless I do save the image id to the db (only single image, no manga), and record the downloaded filesize (not done), and then refer to those to do the check.
from nijiedownloader.
I need to depend on the filename and the file size, as the filename format is customize able, so I cannot depend on the server filename, unless I do save the image id to the db
I forgot to tell that I was referring to the case if user-defined filename contains {serverFilename}. Since it's not included by default, this indeed would need to use the db.
and record the downloaded filesize (not done), and then refer to those to do the check.
I was trying to tell you that filesize check is useless if you take serverFilename into account, since with current scheme it'll be always same for same serverFilenames.
from nijiedownloader.
DB updated, now record both filesize and extracted filename on server (based on the url), no detection logic yet.
Technically, we can compare the url/server filename on the DB with the actual url parsed from the image page.
from nijiedownloader.
Related Issues (20)
- Cannot download image due to Buddhist Calendar HOT 9
- Issues with batch download: "The operation timed out", slow operation etc. HOT 5
- Unable to download mp4 files HOT 4
- Batch Job Downloads to Nonexistent Location HOT 2
- Batch download stops at each image due to DateTime error HOT 14
- Invalid URI HOT 2
- Can't run program HOT 1
- Error after updating to new version HOT 1
- unable to login
- Work date and description no longer retrieved
- Nijie Downloader 1.0.9.4 Batch Downloader Can't DL More than 1 page at a time
- 1.0.9.4 Object reference not set to an instance of an object HOT 8
- The search and batch download function no longer works. HOT 6
- How to download all image with Batch download HOT 1
- 1.0.9.5 when fetch images page or login, will show error about ssl/tls error HOT 5
- It still doesn't load mp4 files
- Download with artist name HOT 1
- Make it portable (CLI?)
- 226 ERROR [ 3] - Error when processing Search Job: Search Tags:
- search and download not working
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nijiedownloader.