Giter Club home page Giter Club logo

Comments (14)

mobizt avatar mobizt commented on June 16, 2024

Thanks for reporting this issue.

To fix this issue, please update the library to v4.3.14.

from firebase-esp8266.

b1n0-kun avatar b1n0-kun commented on June 16, 2024

Thank you. I'm waiting for the new patche comes online.
Will feedback to you when I finish.

from firebase-esp8266.

b1n0-kun avatar b1n0-kun commented on June 16, 2024

Hi Mobitz.
After I update the library to 4.3.14, the problem is still there.
To be more precise, after 1st attempt of backup, it said -> Download failed,
and then from 2nd attempt, it return -> file is still opened.
Is there any specific requirement for the backup file ?

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

it's not possible.

Uninstall library by delete from libraries folder and install the latest version from Library Manager.

from firebase-esp8266.

b1n0-kun avatar b1n0-kun commented on June 16, 2024

I tried that delete and re-installed library also, the issue still there.
Even after I manually deleted the backup file created by esp and tried to backup again, it still showed "file is still opened"

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

In ESP8266, you should reserve the BearSSL rx buffer large enough for your data at that node to download and upload.

As this is the test result of node data size up to 10k bytes when download.

The BearSSL rx buffer size should be 10k up too (12k in this case) as following.

fbdo.setBSSLBufferSize(12000/* Rx buffer size in bytes from 512 - 16384 */, 512 /* Tx buffer size in bytes from 512 - 16384 */);

Test result of 10k data backup (download) and restore (upload)..

Connecting to Wi-Fi...........
Connected with IP: 192.168.137.30

Firebase Client v4.3.14

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready
Downloaded 10%
Downloaded 19%
Downloaded 29%
Downloaded 39%
Downloaded 49%
Downloaded 59%
Downloaded 69%
Downloaded 79%
Downloaded 89%
Downloaded 99%
Downloaded 100%
Backup completed

backup file, /test.bak
file size, 10312

Restore...

Uploading file /test.bak (10312) to /test_backup
Uploaded 4%
Uploaded 9%
Uploaded 14%
Uploaded 19%
Uploaded 24%
Uploaded 29%
Uploaded 34%
Uploaded 39%
Uploaded 44%
Uploaded 49%
Uploaded 54%
Uploaded 59%
Uploaded 64%
Uploaded 69%
Uploaded 74%
Uploaded 79%
Uploaded 84%
Uploaded 89%
Uploaded 94%
Uploaded 99%
Uploaded 100%
Restore completed

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

If the BearSSL rx buffer is too small to handle the data, the invalid or truncated SSL record will be stored in rx buffer and cannot decrypt it to raw (text) data then no or empty data returns from BearSSL engine.

from firebase-esp8266.

b1n0-kun avatar b1n0-kun commented on June 16, 2024

I raised the BearSSL to 12000 as you did, and successfully backup, but only with maximum of 2 child nodes, when I raised to 3 child node, Download failed, and the maximum of the backup file I got here is 294B
image

Kinda surprise with your test succesfully with file size up to 10kb...

Here is content of my child node, nothing too heavy apparently
image

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

It's normal that library is currently supported large file download/upload (and OTA) in RTDB.

Megabytes of data download/upload tested.

You can test with this complete test code.

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <FirebaseESP8266.h>

#include <addons/TokenHelper.h>
#include <addons/RTDBHelper.h>

#define WIFI_SSID "WIFI_AP"
#define WIFI_PASSWORD "WIFI_PASSWORD"

#define API_KEY "API_KEY"

#define DATABASE_URL "URL"

#define USER_EMAIL "USER_EMAIL"
#define USER_PASSWORD "USER_PASSWORD"

FirebaseData fbdo;

FirebaseAuth auth;
FirebaseConfig config;

bool taskCompleted;

void setup()
{

  Serial.begin(115200);

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

  Serial.print("Connecting to Wi-Fi");
  while (WiFi.status() != WL_CONNECTED)
  {
    Serial.print(".");
    delay(300);
  }
  Serial.println();
  Serial.print("Connected with IP: ");
  Serial.println(WiFi.localIP());
  Serial.println();
  Serial.printf("Firebase Client v%s\n\n", FIREBASE_CLIENT_VERSION);

  config.api_key = API_KEY;
  auth.user.email = USER_EMAIL;
  auth.user.password = USER_PASSWORD;
  config.database_url = DATABASE_URL;

  config.token_status_callback = tokenStatusCallback;

  Firebase.begin(&config, &auth);

  Firebase.reconnectWiFi(true);

  fbdo.setBSSLBufferSize(12000, 512);
}

// The Firebase download callback function
void rtdbDownloadCallback(RTDB_DownloadStatusInfo info)
{
  if (info.status == fb_esp_rtdb_download_status_init)
  {
    Serial.printf("Downloading file %s (%d) to %s\n", info.remotePath.c_str(), info.size, info.localFileName.c_str());
  }
  else if (info.status == fb_esp_rtdb_download_status_download)
  {
    Serial.printf("Downloaded %d%s\n", (int)info.progress, "%");
  }
  else if (info.status == fb_esp_rtdb_download_status_complete)
  {
    Serial.println("Backup completed\n");
  }
  else if (info.status == fb_esp_rtdb_download_status_error)
  {
    Serial.printf("Download failed, %s\n", info.errorMsg.c_str());
  }
}

// The Firebase upload callback function
void rtdbUploadCallback(RTDB_UploadStatusInfo info)
{
  if (info.status == fb_esp_rtdb_upload_status_init)
  {
    Serial.printf("Uploading file %s (%d) to %s\n", info.localFileName.c_str(), info.size, info.remotePath.c_str());
  }
  else if (info.status == fb_esp_rtdb_upload_status_upload)
  {
    Serial.printf("Uploaded %d%s\n", (int)info.progress, "%");
  }
  else if (info.status == fb_esp_rtdb_upload_status_complete)
  {
    Serial.println("Restore completed\n");
  }
  else if (info.status == fb_esp_rtdb_upload_status_error)
  {
    Serial.printf("Upload failed, %s\n", info.errorMsg.c_str());
  }
}

void loop()
{

  if (Firebase.ready() && !taskCompleted)
  {
    taskCompleted = true;

    FirebaseJson data;

    // Prepare node data to test
    Serial.println("\nPrepare data... ");

    // Warning memory allocation error can be occurred
    for (int i = 0; i < 8; i++)
    {
      data.set("/data/group" + String(i) + "/a", "________________________________________________________");
      data.set("/data/group" + String(i) + "/b", "________________________________________________________");
      data.set("/data/group" + String(i) + "/b", "________________________________________________________");
      data.set("/data/group" + String(i) + "/c", "________________________________________________________");
      data.set("/data/group" + String(i) + "/d", "________________________________________________________");
      data.set("/data/group" + String(i) + "/e", "________________________________________________________");
      data.set("/data/group" + String(i) + "/f", "________________________________________________________");
      data.set("/data/group" + String(i) + "/g", "________________________________________________________");
      data.set("/data/group" + String(i) + "/h", "________________________________________________________");
      data.set("/data/group" + String(i) + "/i", "________________________________________________________");
      data.set("/data/group" + String(i) + "/j", "________________________________________________________");
      data.set("/data/group" + String(i) + "/k", "________________________________________________________");
      data.set("/data/group" + String(i) + "/l", "________________________________________________________");
      data.set("/data/group" + String(i) + "/m", "________________________________________________________");
    }

    Serial.println("\nSet data... ");
    if (!Firebase.setJSON(fbdo, "/test_backup", data))
      Serial.println(fbdo.errorReason());

    Serial.println("\nBackup test...\n");
    if (Firebase.backup(fbdo, StorageType::FLASH, "/test_backup", "/test.json", rtdbDownloadCallback))
    {
      Serial.printf("backup file, %s\n", fbdo.getBackupFilename().c_str());
      Serial.printf("file size, %d\n", fbdo.getBackupFileSize());
    }
    else
      Serial.println(fbdo.fileTransferError().c_str());

    Serial.println("\nRestore test... \n");

    if (!Firebase.restore(fbdo, StorageType::FLASH, "/test_backup", "/test.json", rtdbUploadCallback))
      Serial.println(fbdo.fileTransferError().c_str());
  }
}

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

Result of the test code.

Connecting to Wi-Fi...........
Connected with IP: 192.168.137.54

Firebase Client v4.3.14

Token info: type = id token (GITKit token), status = on request
Token info: type = id token (GITKit token), status = ready

Prepare data...

Set data...

Backup test...

Downloaded 15%
Downloaded 30%
Downloaded 46%
Downloaded 61%
Downloaded 77%
Downloaded 92%
Downloaded 100%
Backup completed

backup file, /test.json
file size, 6650

Restore test...

Uploading file /test.json (6650) to /test_backup
Uploaded 7%
Uploaded 15%
Uploaded 23%
Uploaded 30%
Uploaded 38%
Uploaded 46%
Uploaded 53%
Uploaded 61%
Uploaded 69%
Uploaded 76%
Uploaded 84%
Uploaded 92%
Uploaded 99%
Uploaded 100%
Restore completed

from firebase-esp8266.

b1n0-kun avatar b1n0-kun commented on June 16, 2024

Hi Mobitz.
I tried your code and yes, it can download up to 12kB content ( with your sample, up to group 13).
I will try again get it to work with my code.
Another irrelevant issue, the stream pause + stream.clear() you showed me in previous topic to gain memory, I couldn't get it resumed, for fbdo pause and resume, it worked.

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

I case of stream, the pointers to callback functions were removed after using stream.clear();.

To resume the callback, you can set the callback function again with following.

fbdo.pauseFirebase(false);
stream.pauseFirebase(false);
Firebase.setStreamCallback(stream, streamCallback, streamTimeoutCallback);

FYI
When ESP8266 core update is available (newer than current release v3.1.2), stream time out checking will be better (nearly real-time server connection tracking) as it relies on TCP Keepalive by calling stream.keepAlive as in the new stream examples.

from firebase-esp8266.

b1n0-kun avatar b1n0-kun commented on June 16, 2024

Sorry for a late feedback, I successfully set the backup with raising the BearSSL Buffersize.
My NodeMCU get overflowed when I tried to set those buffer size to12k-512b for fbdo and 5k-1024 for stream, then I lower the fbdo to 10k, seems OK now. Can these settings can be change schedulely for backup node ? (raise before and lower after task ?)

Thanks for the suggestion about stream resuming, will test it asap, my daily work get busier and can only touch the esp less than 1 hour/day

from firebase-esp8266.

mobizt avatar mobizt commented on June 16, 2024

Yes, you can change BearSSL buffer size to any value before opening the session (starting new server connection).
The new BearSSL buffer memory allocation with new size will take effect when current or last session was closed and re-open.

Note that, the session will keep alive (reuse) in this library to reduce the SSL handshake waiting time during connecting to server.

To close current session, use fbdo.clear().

from firebase-esp8266.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.