Controller Crash about keg-cop HOT 48 CLOSED

Well that's strange. I looked at mine and it's definitely reset. It sounds like mine is running longer:

Yay, library issues!

from keg-cop.

When I get done cooking I'll re-compile 1.0.0 and see if the problem persists. If it does, it will be an upstream change.

from keg-cop.

Thaaaaats weird.

Gimme a sec. :)

Wasn't me. (this time)

from keg-cop.

When was the last time you updated the espressif32 package in platformio?

I just installed PIO on here so not too long ago.

ETA: Great, an update. 😦

from keg-cop.

That happens - Windows is especially challenging with that.

You can try to manually delete the ./.pio/libdeps directory inside the project. That's effectively the same thing. The end result we are looking for is forcing the library dependencies to update.

In the case of that particular error, it's the fault of a forced dependency that is not even used in this project. Let me try to clean that up. I prefer letting the upstream authors manage their own code but sometimes they don't have the same focus.

from keg-cop.

Excellent! Try the current version in Brewflasher though, there have been a lot of platform changes which may help your stability issues.

from keg-cop.

Hmm. The panic isn't just on Serial connection either.

I just checked the web interface and it seems the temp is slowly going up and up because the esp32 keeps rebooting due to a panic.
Then the thermostat never triggers because it's in "minimum off time" (which is fine). But, basically it seems that the device is not staying on for longer than 5 min at a time

from keg-cop.

Here's another log dump on a freshly flashed esp32 and after setting up the wifi.
You can see that it connects, starts up, and then fails.

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10044
load:0x40080400,len:5872
entry 0x400806ac

1970-01-01T00:00:00Z N: Serial logging started at 115200.
1970-01-01T00:00:00Z V: Starting WiFi.
*WM: [1] Free heap: 246792
*WM: [1] ESP SDK version: v3.3.5-1-g85c43024c
*WM: [3] allocating params bytes: 20
*WM: [2] Added Parameter: name
*WM: [1] AutoConnect.
*WM: [2] esp_wifi_set_country: US
*WM: [2] ESP32 event handler enabled.
*WM: [2] Connecting as WiFi client.
*WM: [3] STA static IP:
*WM: [2] setSTAConfig static ip not set, skipping.
*WM: [1] Connecting to saved AP: REDACTED
*WM: [3] Using password: REDACTED
*WM: [3] WiFi_enableSTA: enabled.
*WM: [3] Mode after delay: STA
*WM: [2] 30000 ms conne▒Q▒▒▒▒▒сset.
*WM: [2] 30000 ms timeout,@waiting for connection....................................
*WM: [2] Connection result: WL_CONNECTED
*WM: [3] lastconxresult: WL_CONNECTED
*WM: [1] AutoConnect: SUCCESS
*WM: [1] STA IP address: 192.168.0.60
1970-01-01T00:00:04Z N: Connected. IP address: 192.168.0.60.
*WM: [3] Freeing allocated parameters.
*WM: [3] Unloading AsyncWiFiManager.
1970-01-01T00:00:04Z N: Entering blocking loop to get NTP time..
2022-06-26T19:45:58Z N: NTP time set.
2022-06-26T19:45:58Z N: Flowmeters loaded.
2022-06-f6T19:45:58Z V: No OTA pending.
2022-06-26T19:45:58Z N: mDNS responder started for devkegcop.local.
2022-06-26T19:45:58Z N: HTTP registered via mDNS on port 80.
2022-06-26T19:45:58Z N: Async HTTP server started on port 80.
2022-06-26T19:45:58Z V: Open: http://devkegcop.local to view application.
2022-06-26T19:45:58Z V: MQTT: Creating process.
2022-06-26T19:45:58Z N: Started Keg Cop version 1.0.1-Alpha.3/1.0.1-Alpha.3 (fix_kk_report) [6971746].
2022-06-26T19:46:03Z V: MQTT: No broker configured.
2022-06-26T19:46:08Z V: MQTT: No broker configured.
2022-06-26T19:46:08Z V: Keg Screen reporting not enabled, skipping.
ur MdiatonEror Cre anc'd Inerup wt imou o CU1)
C 000c71 S 000004 A0 : x80878 1 0fb1d40
2 0fb0058 3 0x3f41c8c 4 0x00000014 5 0x3ff0000
A6 0x3ff0048 7 0x00000001 8 : x0000001 9 : xf184d
A0 0x000000a5 11 0x3ff0060 12 0x8008b38f A3 0x3fbfd
14 : 0000008 15 : x0000001 AR : x000001f ECCUSE 0x00000006
ECVADR: x0000000 LBEG 0x400014fd LEND 0x4000150d LOUT 0xfffff

ELF fle SA256: 000000000000

Bactrac: 0x40c7:0x3ff1d40 040a7d50xffbd60 x408c3800xffbd80 x408c3360xffce30

Coe 0 rgisterdump
P : x0086e26 S 0x00060d34 0 : x80148ea A1 0x3ff5480
A2 : 0x3fc61 A 0x3ff54ac A4 0x0000000d A 0x000000ac
A6 0x00000000 7 0x00060423 A8 0x8014e585 9 0x3ff5450
A1 0xfffff A1 0x3ff54ac 12 0x0000000d A3 0x00003f68
A1 : 0x00000011 15 0x00000011 AR : x0000019 EXCCUSE 0x00000006
EXCVDDR 0x00000000 BEG : x4000c2e0 EN 0x4000c2f6 CONT 0xfffff

ELFfile SA256 0000000000000

Bactrac: 0x4086e260xffb40 0x048ea:03fb400x089bc:03fb40

ebooing..

from keg-cop.

Okay, I ran all night on 1.0.0. Back on the Alpha code now.

from keg-cop.

It ran for a few hours this morning before resetting due to: W: WiFi lost connection. So this was a legit restart. So far I can't reproduce the resets you are getting which have to be =<5 mins if your kegerator is not cooling.

I created a debug version with debug symbols. See what this does for you:

BrewFlasher: 1.0.1-Debug.1 Debug release

from keg-cop.

I'll try out this version after work!

I noticed that it does go for longer than 5 min; it has gotten up to an hour or so. But, I can't seem to zero in on what causes it.

It was risky, but I let it run over night. The temp was within 1° of target, so it is able to cool, but it's restarting constantly.

Here's the a screenshot as I'm writing this message to show that it can go over 5min but is still happening.

from keg-cop.

Well, I'm running for multiple hours now so I definitely can't repro (yet.)

from keg-cop.

It seems the firmware is not longer in your github?
The address that BrewFlasher looks for is 404.

I looked in the 'debug reset' branch and it seems that the compiled firmware in there is a few days old, so I'm not sure if they are correct, but I'll try them.

from keg-cop.

Thaaaaats weird.

Gimme a sec. :)

from keg-cop.

No, it was me. Apparently when I delete a tag, the release it was tied to breaks. Go figure.

from keg-cop.

My test board all of a sudden will not save WiFi credentials so I can't test at all - working on finding another ESP32 somewhere.

from keg-cop.

I had a really hard time finding the recommended D32 for this project (Lolin D32 v1.0.0).
I found other ones for cheaper on amazon and just had to do a breadboard to the PCB header.
I wonder if the alternative board is part of the cause for the panic? I tried on a few boards...but I can keep trying.

Did you re-release the debug build on github?

from keg-cop.

I did:

1.0.1-Debug.1

John broke BrewFlasher again so you will have to do it manually. Pay attention to the partition table changes, they are listed here.

from keg-cop.

Key (helpful) change: The system will log uptime and reboot reasons in uptime.csv. Go to the /edit page (un: admin, pw: p@ssword) and right click the file to download the csv:

from keg-cop.

John fixed BrewFlasher:

https://web.brewflasher.com/fw/78/

from keg-cop.

awesome. I was having trouble with the new offsets. I'll try brewflasher now

from keg-cop.

I got it connected to the network.
Web interface loaded fine.

Tried to connect via Serial and had the following.
It seems that on connect it reboots the device, then panics, then reboots, panics, and then hangs.
Each time it does connect to the network, so the issue is upstream of the network being established.

I'll keep digging.

ets Jun 8 2016 00:22:57

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:12784
load:0x40080400,len:3032
entry 0x400805e4

1970-01-01T00:00:00Z N: Serial logging started at 115200.
1970-01-01T00:00:00Z V: Starting WiFi.
*WM: [1] Free heap: 246888
*WM: [1] ESP SDK version: v3.3.5-1-g85c43024c
*WM: [3] allocating params bytes: 20
*WM: [2] Added Parameter: name
*WM: [1] AutoConnect.
*WM: [2] esp_wifi_set_country: US
*WM: [2] ESP32 event handler enabled.
*WM: [2] Connecting as WiFi client.
*WM: [3] STA static IP:
*WM: [2] setSTAConfig static ip not set, skipping.
*WM: [1] Connecting to saved AP: REDACTED
*WM: [3] Using password: REDACTED
*WM: [3] WiFi_enableSTA: enabled.
*WM: [3] Mode after delay: STA
*WM: ▒2] 30000 ms connectTimeout set.
*WM: [2] 30000 ms timeout, waiting for connection....GGuru Meditation Error: Cor e 1 panic'ed (Unhandled debug exception)
Debug exception reason: BREAK instr
Core 1 register dump:
PC : 0x400803c0 PS : 0x00060036 A0 : 0x4008c383 A1 : 0x3f fbd6c0
A2 : 0x00050023 A3 : 0x00000000 A4 : 0x00000001 A5 : 0x00 000001
A6 : 0x00000001 A7 : 0x00060c23 A8 : 0x0000000c A9 : 0x00 000000
A10 : 0x00000000 A11 : 0x00000000 A12 : 0x3ffc8e5c A13 : 0x3f fb796c
A14 : 0x00000001 A15 : 0x00000001 SAR : 0x00000016 EXCCAUSE: 0x00 000001
EXCVADDR: 0x00000000 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00 000000

ELF file SHA256: 0000000000000000

Backtrace: 0x400803c0:0x3ffbd6c0 0x4008c380:0x3ffbd6e0 0x4008c336:0xa5a5a5a5

Rebooting...
ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:12784
load:0x40080400,len:3032
entry 0x400805e4

1970-01-01T00:00:02Z N: Serial logging started at 115200.
1970-01-01T00:00:02Z V: Starting WiFi.
*WM: [1] Free heap: 246888
*WM: [1] ESP SDK version: v3.3.5-1-g85c43024c
*WM: [3] allocating params bytes: 20
*WM: [2] Added Parameter: name
*WM: [1] AutoConnect.
*WM: [2] esp_wifi_set_country: US
*WM: [2] ESP32 event handler enabled.
*WM: [2] Connecting as WiFi client.
*WM: [3] STA static IP:
*WM: [2] setSTAConfig static ip not set, skipping.
*WM: [1] Connecting to saved AP: REDACTED
*WM: [3] Using password: REDACTED
*WM: [3] WiFi_enableSTA: enabled.
*WM: [3] Mode after delay: STA
*WM: ▒▒] 30000 ms connectTimeout set.
*WM: [2]▒30000 ms timeout, waiting for connection.....Guru Meditation Error: Cor e panic'ed (Double exception)
Core 0 register dump:
PC 0x40089cf PS 0x00060036 A0 0x80085a6 A1 0x3fb9b
A2 0x00000000 A3 0x3fb96c A4 0x92901199 A5 0x00000001
A6 0x00060021 A7 0x00000400 A8 0x00000001 A9 0x000000 01
A10 0x00060023 A11 0x00060023 A12 0x00060021 A13 0x3fb96c
A14 0x00000006 A15 0x00000000 SAR 0x00000000 EXCCAUSE 0x000000 02
EXCVADDR 0x00000002 LBEG 0x4000ce LEND 0x4000cf LCOUNT 0xffff
Core was running in ISR context:
EPC1 : 0x40089cf EPC2 : 0x00000000 EPC3 : 0x00000000 EPC4 : 0x00000 000

ELF file SHA256: 000000000000000

Backtrace:0x40089cf0x3fb9b 0x40085a3:0x3fb9d 0x400841b:0x3fb9f 0x40081bd0x3fba00 x4008707d0x3fba00x4013814d0x3fb810 0x4012f83:0x3fb830 0x401434f:0x3fb850 0x40143 521:0x3fb870 0x4019cb:0x3fb890 0x4019c2b0x3fb8b 0x40089be0x3fb8d

Rebooting...

from keg-cop.

Core was running in ISR context:

Disconnect any of your flowmeters and see what happens.

from keg-cop.

That was a new esp32 with nothing connected.

But, I started to think when you said you had to find a new esp32.
I just flashed another brand ESP32 (this one only has 30 pins instead of 38) and it has been up for 20min with both web interface and serial connected...

So, I wonder if the issue was with an incompatibility with this specific board+ libraries?
It's weird that both boards of the same brand had the panic and now this new one doesn't.

I'll keep monitoring

from keg-cop.

That could be - it makes sense. WiFi and BLE use some pins on some boards and not others. This was written on Lolin D32 (regular, not pro) so that may guide you a bit.

For what its worth, I did not change anything having to do with pins.

from keg-cop.

Here's the pinout: https://www.wemos.cc/en/latest/d32/d32_pro.html

from keg-cop.

I just reflashed v1 on the board that was panicing.
No issues so far -- I can connect to both web and serial at the same time.
It's been up for 10min now.

So, it has to be some combo of new libraries (?) and the board.
Obviously I understand that you probably don't want to support or track down root causes for board-dependent issues.
So, I'm ok if you want to close this issue and I'll move forward with this other board while I try to get my hands on a Lolin D32.

But, I think this is good to have tracked for other users who may end up with similar issues.
So hopefully it wasn't fully a waste of time!

Here's the pinout of the board that panics on the new version:

And here's the pinout of the board that seems to be doing ok (so far) with the new version:

from keg-cop.

Hmm. I spoke too soon about this new board.

from keg-cop.

Here's where we need to get you running with PlatformIO and you can try a few things. Are you able to clone the repo and get this branch to compile? If so I can suggest a few things to rule in/out some of the pin sets to narrow it down.

from keg-cop.

Also: I have deleted the debug download and replaced it with 1.1.0-Alpha.1 to reflect the semantic versioning of what will be a release.

Keg Cop 1.1.0-Alpha.1

from keg-cop.

I'll try to download and compile on my own now.
I'll let you know how it goes!

I'm currently at 10 hours of uptime which I think is the longest stretch.
I was curious if the panics were related to interactivity or if they were random.
My keezer obviously doesn't get a ton of use during morning/day time, but I do have nitro coffee on tap that's being tracked, so it did get some use early this morning and didn't crash.

Also, I have it sending data to a server via the Keg Screen feature.

I just wanted to write out my full use case in case it has any helpful hints.

from keg-cop.

I was curious if the panics were related to interactivity or if they were random.

Honestly, we won't know till we know.

Once you get to where you can compile and upload your own firmware, we can disable functionality to see if we get some material change.

Offhand I would try commenting lines 82-86 (thus disabling flowmeter support) and see if that has a positive change. Not sure it will compile right offhand with that change, but that would be my first try at it.

from keg-cop.

When was the last time you updated the espressif32 package in platformio?

from keg-cop.

Ok, repo cloned, PIO setup.
Compile fails due to an issue in the AsyncWifiManager strings_en.h file:

Here's a snippet (I added the underscores to try to get the error code to line up correctly; it should be under the 'C' in 'CONFIG'):

.pio/libdeps/lolin_d32/AsyncWiFiManager/src/strings_en.h:342:49: error: 'CONFIG_ESP32_PHY_MAX_TX_POWER' was not declared in this scope
const wifi_country_t WM_COUNTRY_US{"US", 1, 11, CONFIG_ESP32_PHY_MAX_TX_POWER, WIFI_COUNTRY_POLICY_AUTO};
_______________________________________________________^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

from keg-cop.

Yeah ... I've got work to do because of the new platform packages.

from keg-cop.

Updated Keg Cop Alpha to 1.1.0-Alpha.2

You'll have to git fetch --all and git checkout taplist_io to get this branch. It might fix the crashes - check it.

from keg-cop.

It got SO much further this time, but still had an issue in the Onewire library file OneWire_direct_gpio.h

'rtc_gpio_desc' and 'esp32_gpioMux' not declared in this scope.

In file included from .pio/libdeps/lolin_d32/OneWire/OneWire.cpp:144:
.pio/libdeps/lolin_d32/OneWire/util/OneWire_direct_gpio.h: In function 'void directModeInput(uint32_t)':
.pio/libdeps/lolin_d32/OneWire/util/OneWire_direct_gpio.h:161:26: error: 'rtc_gpio_desc' was not declared in this scope
uint32_t rtc_reg(rtc_gpio_desc[pin].reg);

from keg-cop.

Click on the alien, drop down lolin_d32 under Project Tasks, General. "Clean all":

from keg-cop.

You should try the one I added to BrewFlasher as well.

from keg-cop.

I'm trying the new one from BrewFlasher.

I tried the 'clean all' task and it for some reason ended up with a 'permission denied' error. I tried to troubleshoot this and everything seemed to just blame antivirus. I tried disabling everything, but still got the same error.
I tried running in admin mode.
I forked the repo and tried that in admin mode -- always the same error.

But, it seemed to help regardless as it allowed the 'build' task to get further. Now, it's an issue with

Compiling .pio\build\lolin_d32\lib4d9\LittleFS_esp32\lfs.c.o
.pio/libdeps/lolin_d32/LittleFS_esp32/src/esp_littlefs.c:19:2: warning: #warning ("Use the built-in LITTLEFS library") [-Wcpp]
#warning("Use the built-in LITTLEFS library")
^~~~~~~
.pio/libdeps/lolin_d32/LittleFS_esp32/src/LittleFS.cpp: In member function 'virtual bool LITTLEFSImpl::exists(const char*)':
.pio/libdeps/lolin_d32/LittleFS_esp32/src/LittleFS.cpp:44:28: error: no matching function for call to 'LITTLEFSImpl::open(const char*&, const char [2])'
File f = open(path, "r");
______________________^
In file included from .pio/libdeps/lolin_d32/LittleFS_esp32/src/LittleFS.cpp:17:
C:/Users/Austin/.platformio/packages/framework-arduinoespressif32/libraries/FS/src/vfs_api.h:38:17: note: candidate: 'virtual fs::FileImplPtr VFSImpl::open(const char*, const char*, bool)'
FileImplPtr open(const char* path, const char* mode, const bool create) override;
____________^~~~
Archiving .pio\build\lolin_d32\lib88e\libESPTelnet.a

from keg-cop.

I know you're not here to provide platformIO support. I'll try to figure it out myself; I just wanted to update you on where I'm at.

from keg-cop.

Updated Keg Cop Alpha to 1.1.0-Alpha.3

git pull on the taplist_io branch to get the current changes. You'll probably want to do that clean command (or delete the ./.pio/libdeps directory) to be sure.

from keg-cop.

🥳 🍻

the 'clean all' task still ended up with an Access Denied for ArduinoJson, but deleting the lipdeps folder and running the 'clean' task seemed to clear everything up

from keg-cop.

Just got it installed and updated.
I've gotten a few:

Reason: ESP_RST_TASK_WDT, Description: Reset due to task watchdog

Any clue what that might be?

To be honest, I may have messed this board up. When I put it back in my breadboard I was off by one row. So, I'm not sure if I accidently put some voltage where I shouldn't have 🤷‍♂️
If I keep getting instability I'll flash a new board and swap 'em

from keg-cop.

WDT = Watchdog Timer. It means it locked up for whatever reason.

from keg-cop.

Updated Keg Cop Alpha to 1.1.0-Alpha.4

... how's the testing going?

from keg-cop.

It seemed good for as long as it was up (we just had a power surge).
I can't seem to download the uptime.csv anymore, though. The page loads, but right-click -> download doesn't result in anything.
I was checking the "about" page occasionally and I believe it was already at 16 hours or so before this power surge.

Anyways, the platform update seems to have resolved the crashing.
I'm flashing Alpha.4 right now and will keep you updated.
Feel free to mark this issue as resolved; I can always re-open if problems crop up again 😄

from keg-cop.

Yeah unfortunately the new platform libraries broke the SPIFFS editor. Now in theory I should not need it when I am done, but I use it a lot in development. Not sure what's up with that.

from keg-cop.