Different Sizes of Data on Windows vs Ubuntu about moodle-dl HOT 12 CLOSED

Thank you very much :)

I develop mainly for Linux because I and many of my fellow students, who can give me direct feedback, use Linux. I only test under Windows from time to time. Because I haven't written any tests yet, there are often extra bugs.

Download errors are currently only written to the log file and if errors have occurred a warning is displayed in the console window at the end of the download.

Could you please look into your log file under Windows to see if there are any download errors listed there? If this is the case, please put them in here.

from moodle-dl.

I should change this, so that a short error message is also displayed in the console window.

from moodle-dl.

I think an issue with the filenames is likely.

https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#naming-conventions

from moodle-dl.

I filter special characters in the to_valid_name function.

Is there any missing?

from moodle-dl.

Do not end a file or directory name with a space or a period. Although the underlying file system may support such names, the Windows shell and user interface does not. However, it is acceptable to specify a period as the first character of a name. For example, ".temp".

This could still happen.

Another thing is case sensitivity (I think windows isn't case sensitive) and there exist path length limitations (or at least used to exist).

from moodle-dl.

Super good hint.
I will now implement the first half of it. I think the path length restriction is a minor problem.

And case sensitivity should not make a problem, because before a file gets downloaded it is checked if it already exists... if it exists it gets renamed.

from moodle-dl.

@NextBlaubeere could you please test if this fixed the problem

from moodle-dl.

Wow, what a great response to this issue. 👍

So, I checked the ModdleDownloader.log file on Windows but there are no errors present:

2020-03-19 00:11:38  INFO  {main}  --- main started ---------------------
2020-03-19 00:11:38  DEBUG  {main}  Loading config...
2020-03-19 00:11:38  DEBUG  {main}  Checking for changes for the configured Moodle-Account....
2020-03-19 00:11:38  DEBUG  {moodle_service}  Fetching current Moodle State...
2020-03-19 00:11:39  DEBUG  {results_handler}  Detected moodle version: 2019111802
2020-03-19 00:12:11  DEBUG  {moodle_service}  Checking for changes...
2020-03-19 00:12:11  DEBUG  {main}  Start downloading changed files...
2020-03-19 00:17:35  INFO  {main}  705 changes found for the configured Moodle-Account.
2020-03-19 00:17:37  DEBUG  {mail_service}  Mail-Notifications not configured, skipping.
2020-03-19 00:17:37  DEBUG  {main}  All done. Exiting...

On a sidenote:
There is a really weird issue going on with extracting the zip file I created of the files and then moved from Ubuntu to Windows10.
When I extract it via WinRAR i get an Error "The syntax for the file name, directory, or the volume label is wrong." on two minor files. (orig. ger.: "Die Syntax für den Dateinamen, Verzeichnisnamen oder die Datenträgerbezeichnung ist falsch")
I then renamed the files inside the zip-file and extract them. But renaming them again to the original file-name does not work, not because of special characters but because of file-explorer does not let me input more than a certain amount of characters.
At first i thought i hit the maximum characters of the NTFS Filesystem but HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled was already on 1.
Even stranger was the fact, that extracting the same ZIP-File with 7zip resulted without any error AND the same files were renameable with more characters then those extractet with WinRAR.

Back to topic
I am going to test the new revision and get back to you later that evening. Thanks for your work!

from moodle-dl.

Interestingly according to your log, 705 files were downloaded. Strange is really that there is no error in the log because at least two files were not downloaded, which is probably the same problem as with WinRAR. But only 624 files were created according to your first post.

I should really write tests.

Do you use Windows 10?

from moodle-dl.

Yes I am using Windows 10. :-)

I now tested the new revision and it looks very promising:

2020-03-19 16:55:37  INFO  {main}  --- main started ---------------------
2020-03-19 16:55:37  DEBUG  {main}  Loading config...
2020-03-19 16:55:37  DEBUG  {main}  Checking for changes for the configured Moodle-Account....
2020-03-19 16:55:37  DEBUG  {moodle_service}  Fetching current Moodle State...
2020-03-19 16:55:37  DEBUG  {results_handler}  Detected moodle version: 2019111802
2020-03-19 16:56:08  DEBUG  {moodle_service}  Checking for changes...
2020-03-19 16:56:08  DEBUG  {main}  Start downloading changed files...
2020-03-19 17:04:27  INFO  {main}  707 changes found for the configured Moodle-Account.
2020-03-19 17:04:29  DEBUG  {mail_service}  Mail-Notifications not configured, skipping.
2020-03-19 17:04:29  DEBUG  {main}  All done. Exiting...

even the two "special" files are there.

Windows: 2,70 GB (707 Files, 203 Folders)

👍 👍 👍

from moodle-dl.

Do not end a file or directory name with a space or a period. Although the underlying file system may support such names, the Windows shell and user interface does not. However, it is acceptable to specify a period as the first character of a name. For example, ".temp".

This could still happen.

Another thing is case sensitivity (I think windows isn't case sensitive) and there exist path length limitations (or at least used to exist).

Oh god. I don't know if WinRAR oder 7Zip, but one of those two created folders with empty/space names. So i can't remove them neither with explorer nor with the shell. :'D
Time to get a liveUSB Ubuntu ...

from moodle-dl.

I also changed the code to pathlib, which should theoretically avoid more bugs, but behaves a bit different than os.path. So I hope that everything still works ^^

Yes I know the bug :D from the time when I also used Windows. I hope you get it fixed.

I'll close the thread for now, if there's anything left, just reopen it.

from moodle-dl.