Using Chunk? about pysmb HOT 5 CLOSED

@Mannshoch : Can you provide an estimate on the file size before your app runs out of RAM?

from pysmb.

I have planned to backup a ~16GB ZIP form a WebDAV with as cron job once a month onto a NAS. My Router has 2GB RAM (around 1.4GB is unused). It is possible to ask WebDAV (webdavclient3) for the file size.

from pysmb.

The issue might be that your app is trying to read the entire zip into memory before uploading to your NAS.
If you can read your zip file in smaller chunks from your WebDAV server, you can use pysmb's storeFileFromOffset to upload each chunk of data at the last offset.

from pysmb.

that's my current code:

for filename in webdav_fs.ls(webdav_pfad, detail=False):
    webdav_file_path = os.path.join(filename)
    smb_file_path = os.path.join(smb_path, filename)
    file_size = webdav_fs.info(webdav_file_path)['size']
    offset = 0
    chunk_size = 8 * 1024  # 8KB
    f = io.BytesIO()
    webdav_client.download_fileobj(webdav_file_path, f)
    f.seek(0)  
    while True:
        chunk = f.read(chunk_size)
        if not chunk:
            break
        smb_conn.storeFileFromOffset(smb_service_name, smb_file_path, f, offset=offset)
        offset += len(chunk)

from pysmb.

webdav_client.download_fileobj(webdav_file_path, f)
This line will download the entire WebDAV file into the memory.
Please check if your client can download the file in chunks from the remote WebDAV server.

from pysmb.