Airgradient rebooting itself?

#25

Okay… Thanks to some posts on the issues and PR sections of the esp8266 Arduino core github page a few days ago , I was able to workaround the bug preventing me from compiling with core v3.1.0 and above by manually replacing the bad mkbuildoptglobals.py file. This allowed me to compile the sketch with the newer Core versions in the older Arduino IDE v1.8.x , which finally allowed me to run the ESP Exception Decoder. And the results look much better:

It crashes so often, I have decoded dozens of exceptions and each one is always the same:

circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420

atomic_base.h is an “internal header file” and line 420 looks like this:

25  /** @file bits/atomic_base.h
26   *  This is an internal header file, included by other library headers.
27   *  Do not attempt to use it directly. @headername{atomic}
28   */
.
.
.
419      _GLIBCXX_ALWAYS_INLINE __int_type
420      load(memory_order __m = memory_order_seq_cst) const noexcept
421      {
422	memory_order __b = __m & __memory_order_mask;
423	__glibcxx_assert(__b != memory_order_release);
424	__glibcxx_assert(__b != memory_order_acq_rel);

Here are some decoded exceptions:

Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000

Decoding stack results
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x4010062b: ets_intr_unlock() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 226
0x40228c85: sys_timeout_abs at core/timeouts.c line 189
0x401000ab: app_entry_redefinable() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 386
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40223559: glue2esp_linkoutput at glue-esp/lwip-esp.c line 301
0x40223787: new_linkoutput at glue-lwip/lwip-git.c line 272
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x4022cb94: ip4_output_if_opt_src at core/ipv4/ip4.c line 1764
0x40223559: glue2esp_linkoutput at glue-esp/lwip-esp.c line 301
0x40223787: new_linkoutput at glue-lwip/lwip-git.c line 272
0x40223be2: ethernet_output at netif/ethernet.c line 312
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x402176d3: Twi::WAIT_CLOCK_STRETCH() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/PolledTimeout.h line 130
0x40217974: Twi::read_bit() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 336
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40212c64: u8x8_byte_arduino_hw_i2c(u8x8_t*, uint8_t, uint8_t, void*) at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\U8x8lib.cpp line 1316
0x40202606: u8x8_cad_ssd13xx_fast_i2c at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_cad.c line 582
0x40212c48: u8x8_byte_arduino_hw_i2c(u8x8_t*, uint8_t, uint8_t, void*) at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\U8x8lib.cpp line 1316
0x402035dc: u8x8_byte_EndTransfer at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_byte.c line 61
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100a4c: micros() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring.cpp line 181
0x40100110: SoftwareSerial::preciseDelay() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\SoftwareSerial\src/SoftwareSerial.h line 246
0x4021790d: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 325
0x4021787d: Twi::write_stop() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 304
0x40217b01: Twi::readFrom(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 433
0x4021278c: TwoWire::requestFrom(unsigned char, unsigned int, bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 129
0x402127c0: TwoWire::requestFrom(unsigned char, unsigned char, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 137
0x40212966: SensirionI2CCommunication::receiveFrame(unsigned char, unsigned int, SensirionI2CRxFrame&, TwoWire&, CrcPolynomial) at C:\Users\Ken\Documents\Arduino\libraries\Sensirion_Core\src\SensirionI2CCommunication.cpp line 101
0x40216c68: __esp_suspend() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 64
0x40216db9: __esp_delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 161
0x40216e2e: esp_try_delay(unsigned int, unsigned int, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 182
0x40217d90: __delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/coredecls.h line 69
0x402037a7: updateTVOC() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 184
0x40204a84: loop() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 149

Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000

Decoding stack results
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40228c85: sys_timeout_abs at core/timeouts.c line 189
0x401000ab: app_entry_redefinable() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 386
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40216e54: esp_yield() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 144
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40214744: Print::print(__FlashStringHelper const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 106
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40214854: Print::printNumber (unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 262
0x4021e804: utoa at /workdir/repo/newlib/newlib/libc/stdlib/utoa.c line 74
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40101400: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x40213661: IPAddress::printTo(Print&) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 165
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a35: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(WiFiManager::wm_debuglevel_t, __FlashStringHelper const*, IPAddress) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 88
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x402157a1: String::copy(char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 258
0x40215971: String::operator=(char const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 303
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x402159f2: String::String(unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 82
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a86: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(__FlashStringHelper const*, IPAddress) at C:\Users\Ken\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 3281
0x402064d4: String::operator!=(char const*) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/WString.h line 234
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x402178c8: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 313
0x402179cb: Twi::write_byte(unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 347
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x402127ec: TwoWire::beginTransmission(unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 158
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100110: SoftwareSerial::preciseDelay() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\SoftwareSerial\src/SoftwareSerial.h line 246
0x4021790d: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 325
0x4021787d: Twi::write_stop() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 304
0x40217b01: Twi::readFrom(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 433
0x4021278c: TwoWire::requestFrom(unsigned char, unsigned int, bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 129
0x402127c0: TwoWire::requestFrom(unsigned char, unsigned char, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 137
0x40212966: SensirionI2CCommunication::receiveFrame(unsigned char, unsigned int, SensirionI2CRxFrame&, TwoWire&, CrcPolynomial) at C:\Users\Ken\Documents\Arduino\libraries\Sensirion_Core\src\SensirionI2CCommunication.cpp line 101
0x40216c68: __esp_suspend() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 64
0x40216db9: __esp_delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 161
0x40216e2e: esp_try_delay(unsigned int, unsigned int, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 182
0x40217d90: __delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/coredecls.h line 69
0x402037a7: updateTVOC() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 184
0x40204a84: loop() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 149

Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000

Decoding stack results
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40228c85: sys_timeout_abs at core/timeouts.c line 189
0x40216bfa: loop_task(ETSEvent*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 273
0x401000ab: app_entry_redefinable() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 386
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x4020f304: ESP8266WiFiSTAClass::status() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\ESP8266WiFi\src\ESP8266WiFiSTA.cpp line 558
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40214744: Print::print(__FlashStringHelper const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 106
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40214854: Print::printNumber (unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 262
0x40101400: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x40213661: IPAddress::printTo(Print&) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 165
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a35: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(WiFiManager::wm_debuglevel_t, __FlashStringHelper const*, IPAddress) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 88
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x402157a1: String::copy(char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 258
0x40215971: String::operator=(char const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 303
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x402159f2: String::String(unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 82
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a86: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(__FlashStringHelper const*, IPAddress) at C:\Users\Ken\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 3281
0x402064d4: String::operator!=(char const*) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/WString.h line 234
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40217736: Twi::WAIT_CLOCK_STRETCH() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 167
0x4021790d: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 325
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x402176e2: Twi::WAIT_CLOCK_STRETCH() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 65
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40101403: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x40101400: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x4021787d: Twi::write_stop() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 304
0x40217bb1: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 393
0x40212662: TwoWire::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 209
0x40212834: TwoWire::endTransmission(unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 172
0x40212d09: u8x8_byte_arduino_hw_i2c(u8x8_t*, uint8_t, uint8_t, void*) at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\U8x8lib.cpp line 1354
0x40202619: u8x8_cad_ssd13xx_fast_i2c at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_cad.c line 586
0x40202780: u8x8_d_ssd1306_sh1106_generic at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_d_ssd1306_128x64_noname.c line 327
0x402027b4: u8x8_d_sh1106_128x64_noname at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_d_ssd1306_128x64_noname.c line 481
0x4020364c: u8x8_DrawTile at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_display.c line 89
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x402019b5: u8g2_send_buffer at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8g2_buffer.c line 89
0x40216c68: __esp_suspend() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 64
0x40216db9: __esp_delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 161
0x40216e2e: esp_try_delay(unsigned int, unsigned int, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 182
0x40217d90: __delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/coredecls.h line 69
0x40204f20: AirGradient::getCO2_Raw() at C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp line 691
0x402038f8: updateCo2() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 206
0x40204a8a: loop() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 151
#26

This is the esp8266 Arduino Core. It’s installed under Board Manager, not Library manager. If you look there, you should see the version v3.1.1 installed and you should be able to revert down to v3.0.2, recompile, and flash the board.

Thanks! I have a lot more to report, but it’s past 6am here and I need to get some sleep. Typing this on my phone in bed.

#27

NOTE: This is my 3rd post in a row, so someone reply to this so I can continue to post. :rofl:

I opened an issue in the Arduino Core repository and one of the collaborators there has been responding and has been very helpful and patient with me. Their first response was that it’s likely something to do with SoftwareSerial. I think because circular_queue is part of SoftwareSerial. And they pointed out that EXCVADDR is pointing at nil (0x00000000) instead of an object.

He/she quoted this from my exception:

 Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000
...
0x40100110: SoftwareSerial::preciseDelay() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\SoftwareSerial\src/SoftwareSerial.h line 246

So, that gave me a couple of leads to pursue. The first was the nil/NULL comment stuck me and I remember always see the warning about 2 NULL conversions whenever I compiled the code, but I noticed I didn’t get that warning when I built using <v3.0.2. Here are the build outputs from each.

Core v3.1.1

C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp: In member function 'TMP_RH AirGradient::returnError(TMP_RH_ErrorCode)':
C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp:548:14: warning: converting to non-pointer type 'float' from NULL [-Wconversion-null]
  548 |   result.t = NULL;
      |              ^~~~
C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp:549:15: warning: converting to non-pointer type 'int' from NULL [-Wconversion-null]
  549 |   result.rh = NULL;
      |               ^~~~
. Variables and constants in RAM (global, static), used 33372 / 80192 bytes (41%)
║   SEGMENT  BYTES    DESCRIPTION
╠══ DATA     1516     initialized variables
╠══ RODATA   3832     constants       
╚══ BSS      28024    zeroed variables
. Instruction RAM (IRAM_ATTR, ICACHE_RAM_ATTR), used 62695 / 65536 bytes (95%)
║   SEGMENT  BYTES    DESCRIPTION
╠══ ICACHE   32768    reserved space for flash instruction cache
╚══ IRAM     29927    code in IRAM    
. Code in flash (default, ICACHE_FLASH_ATTR), used 355008 / 1048576 bytes (33%)
║   SEGMENT  BYTES    DESCRIPTION
╚══ IROM     355008   code in flash   
esptool.py v3.0
Serial port COM4
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 84:f3:eb:24:4b:d0
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 394432 bytes to 280039...
Writing at 0x00000000... (5 %)
Writing at 0x00004000... (11 %)
Writing at 0x00008000... (16 %)
Writing at 0x0000c000... (22 %)
Writing at 0x00010000... (27 %)
Writing at 0x00014000... (33 %)
Writing at 0x00018000... (38 %)
Writing at 0x0001c000... (44 %)
Writing at 0x00020000... (50 %)
Writing at 0x00024000... (55 %)
Writing at 0x00028000... (61 %)
Writing at 0x0002c000... (66 %)
Writing at 0x00030000... (72 %)
Writing at 0x00034000... (77 %)
Writing at 0x00038000... (83 %)
Writing at 0x0003c000... (88 %)
Writing at 0x00040000... (94 %)
Writing at 0x00044000... (100 %)
Wrote 394432 bytes (280039 compressed) at 0x00000000 in 6.4 seconds (effective 495.1 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Core v3.0.2

Executable segment sizes:
ICACHE : 32768           - flash instruction cache 
IROM   : 353468          - code in flash         (default or ICACHE_FLASH_ATTR) 
IRAM   : 29585   / 32768 - code in IRAM          (IRAM_ATTR, ISRs...) 
DATA   : 1516  )         - initialized variables (global, static) in RAM/HEAP 
RODATA : 3840  ) / 81920 - constants             (global, static) in RAM/HEAP 
BSS    : 27904 )         - zeroed variables      (global, static) in RAM/HEAP 
Sketch uses 388409 bytes (37%) of program storage space. Maximum is 1044464 bytes.
Global variables use 33260 bytes (40%) of dynamic memory, leaving 48660 bytes for local variables. Maximum is 81920 bytes.
esptool.py v3.0
Serial port COM4
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 84:f3:eb:24:4b:d0
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 392560 bytes to 277859...
Writing at 0x00000000... (5 %)
Writing at 0x00004000... (11 %)
Writing at 0x00008000... (17 %)
Writing at 0x0000c000... (23 %)
Writing at 0x00010000... (29 %)
Writing at 0x00014000... (35 %)
Writing at 0x00018000... (41 %)
Writing at 0x0001c000... (47 %)
Writing at 0x00020000... (52 %)
Writing at 0x00024000... (58 %)
Writing at 0x00028000... (64 %)
Writing at 0x0002c000... (70 %)
Writing at 0x00030000... (76 %)
Writing at 0x00034000... (82 %)
Writing at 0x00038000... (88 %)
Writing at 0x0003c000... (94 %)
Writing at 0x00040000... (100 %)
Wrote 392560 bytes (277859 compressed) at 0x00000000 in 6.3 seconds (effective 497.5 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Here is the code snippet from AirGradient.cpp. This is part of the overall set of code that reads the temperature (TMP) and relative humidity (RH) from the sensor. I’m guessing this piece defines the values that are returned if the sensor reading results in an error.

546   TMP_RH AirGradient::returnError(TMP_RH_ErrorCode error) {
547     TMP_RH result;
548     result.t = NULL;
549     result.rh = NULL;
550   
551     result.t_char[0] = 'N';
552     result.t_char[1] = 'U';
553     result.t_char[2] = 'L';
554     result.t_char[3] = 'L';
555   
556     result.rh_char[0] = 'N';
557     result.rh_char[1] = 'U';
558     result.rh_char[2] = 'L';
559     result.rh_char[3] = 'L';
560   
561     result.error = error;
562     return result;
}

But this turns out to not likely be related, Tmp/RH is not on SoftwareSerial, the warning isn’t indicative of the exception error, and there was a change in compiler warnings since the #8495 PR that’s probably suppressing the still-existent warnings.

The second lead was that I remembered that in the PlatformIO decoder filter output (the ones that I said weren’t very useful), there was SoftwareSerial mentioned:

Exception (0):
epc1=0x402150d0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Illegal instruction
  epc1=0x402150d0 in circular_queue<unsigned int, SoftwareSerial*>::available() const at ??:?

>>>stack>>>

I posted this and mentioned that the output wasn’t very helpful and looked incomplete to which the response was that I needed -g in build_flags or build_type = debug (which does that for us) for the decoder to work. So, in an attempt to see if the PlatformIO decoder can give us more details/clues, I set the build_type = debug flag , re-compiled and ran it waiting for an exception… after waiting 15 minutes, I grew suspicious. So, I removed the flag, re-compiled, and ran it again. This time, it crashed within 5 minutes. I let it run for 15 minutes and it crashed a couple more times as expected. I then re-added the build_type = debug flag, re-compiled, and ran it. To my surprise, it didn’t crash for over an hour. I let it run and went to bed. I got up 6 hours later to find it was still running with no crashes/reboots! As I type this, it’s been over 7 hours and still going.

My assumption is building with the debug flag should only minimally affect code behavior, otherwise, it wouldn’t be very useful in debugging purposes. And so with that premise, I’m hopeful this clue can really narrow-in on possible root causes. I’m doing some reading or maybe someone with a better-suited skillset will be able to help solve this or point me in the right direction.

2 Likes
#28

Thanks ken830, just sent a PM about the uptime display code.

#29

I’m learning a lot about ESPHome right now and see what you indicated that behind the scenes, it is using the PlatformIO code, even if I don’t have to deal with any of the sketches with ESPHome.

I did try to change which version it was using behind the screens. Before the top of my ESPHome config started with:

esp8266:
  board: d1_mini

I now changed it to:

esp8266:
  board: d1_mini
  framework:
    platform_version: 3.1.0

which does look like it lowers the version of PlatformIO to 3.1.0, which looks to be based on Arduino Core v3.0.1

Without specifying a version, my install defaults to platformio/espressif8266 @ 3.2.0 which is based on Arduino Core v3.0.2. So neither in ESPhome look to be using Core v3.1 which was suspected as the culprit above.

#30

Update:

This is running for 10 hours with no crashes/reboots using the latest PlatformIO espressif8266 platform v4.1.0 released last week to include the latest Arduino Core v3.1.x! The only difference is compiling with the build_type = debug flag set.

PXL_20230126_233003096.PORTRAIT
PXL_20230126_233003096.PORTRAIT1920×1440 114 KB

@MallocArray : Super! Yeah, I’m just barely competent enough to poke/fumble around. Let me know how it goes running the older release. It would be interesting to see if it’s solved. By the way, are you able to monitor the serial output? I would like to see if it was the same exception 0 that I’m seeing when yours reboot. That is something no one has been able to confirm/deny for me yet.

BTW, I think you didn’t have to revert that far back. The previous-to-current release (platform-espressif8266 v4.0.1) is the release right before the move to Arduino Core v3.1.x, so it would’ve been Arduino Core v3.0.2.

OLED display "flair" (degrees, decimals)
#31

Without doing anything special in the latest ESPHome, it is building with this:
Processing ag-pro (board: d1_mini; framework: arduino; platform: platformio/espressif8266 @ 3.2.0)
The documentation says it defaults to the “recommended” release, which apparently is 3.2.0 but I may try bumping it up higher and see if anything changes.

Less than 2 hours after going back to platformio 3.1.0 it rebooted again, so I set it back to the defaults of 3.2.0 and am trying it with i2c at 100 kHz from another thread’s conversation and see if that changes anything.

I can do a webpage monitoring the output, but hard to catch it when it randomly reboots somewhere between 30 minutes and 16 hours for me.

#32

I wanted to see if you can plug-in a USB cable and run a serial monitor on your PC to watch the serial port output of the D1 Mini. Runtime Exceptions should be seen there. You just need a large enough screen buffer to scroll back to the time of reboot if it occurs when you’re away.

#33

Seems to be worse in ESPHome with i2c at 100 kHz, as it is rebooting much more frequently.

[19:22:22][D][pmsx003:234]: Got PM1.0 Concentration: 0 µg/m^3, PM2.5 Concentration 0 µg/m^3, PM10.0 Concentration: 0 µg/m^3
[19:22:22][D][sensor:126]: 'Particulate Matter <2.5µm Concentration': Sending state 0.00000 µg/m³ with 0 decimals of accuracy
[19:22:23][D][sgp30:282]: Got eCO2=502.0ppm TVOC=146.0ppb
[19:22:23][D][sgp30:156]: Baseline reading not available for: 43167s
INFO 192.168.2.140: Ping timed out!
INFO Disconnected from ESPHome API for 192.168.2.140
WARNING Disconnected from API
WARNING Can't connect to ESPHome API for 192.168.2.140: Timeout while connecting to ('192.168.2.140', 6053)
INFO Trying to reconnect to 192.168.2.140 in the background
INFO Successfully connected to 192.168.2.140
[19:23:13][D][sgp30:282]: Got eCO2=432.0ppm TVOC=83.0ppb
[19:23:13][D][sgp30:156]: Baseline reading not available for: 43179s
[19:23:14][D][pmsx003:234]: Got PM1.0 Concentration: 0 µg/m^3, PM2.5 Concentration 0 µg/m^3, PM10.0 Concentration: 0 µg/m^3
#34

What is your hardware configuration? I don’t know what the upgrade kit comes with, but I didn’t see you mention the OLED screen. You’ve checked your I2C pull-ups?

#35

It is the same board and came with the OLED already soldered on along with all of the headers and the case, I supplied the rest of the components. I do have the OLED running. I haven’t done anything with the pullup resistors. I did add an SGP30 that I sourced myself on the 3.3v header, but I have reboots even with it removed, so only the D1 mini, PMS, and OLED.

I thought the issue with the pullup resistors was more related to getting erratic readings or 0 for readings, so since I’m getting data, I haven’t read up on them much.

I did set putty to record the serial output from the D1 mini running ESPHome, but nothing good in the logs. It just stops the regular output, does the boot sequence, and then starts output again.

#36

Actually, I dug through a few more of the reboots and did find one large block of an Exception Decoder. Too much to put all of it here, but the highlights are:

Soft WDT reset

>>>stack>>>

ctx: sys
sp: 3fffed40 end: 3fffffb0 offset: 01a0
3fffeee0:  00000018 00012e0d 00000000 00000000  
3fffeef0:  00000000 00000000 00000001 401007a4  
3fffef00:  00000000 3ffeaa98 00000000 00000000  
3fffef10:  4025e908 3ffeaa98 3ffeff20 4022e614

…

3fffff90:  3fffdad0 00000000 3fff08f8 4021ed30  
3fffffa0:  3fffdad0 00000000 3fff08f8 4022d9c0  
<<<stack<<<

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00080030
~ld
[0;32m[I][logger:258]: Log initialized[0m
[0;35m[C][ota:469]: There have been 0 suspected unsuccessful boot attempts.[0m
[0;32m[I][app:029]: Running through setup()...[0m
[0;32m[I][i2c.arduino:175]: Performing I2C bus recovery[0m
[0;35m[C][uart.arduino_esp8266:059]: Setting up UART bus...[0m
[0;35m[C][sgp30:036]: Setting up SGP30...[0m
[sgp30:046]: Serial Number: 28065498[0m
[sgp30:066]: Product version: 0x22[0m
[0;35m[C][ssd1306_i2c:010]: Setting up I2C SSD1306...[0m
[0;35m[C][wifi:037]: Setting up WiFi...[0m
AirGradient Pro with ESPhome, display keeps turning back on
#37

If you only have D1 mini, PMS, and OLED, then I wouldn’t worry about your pull-ups. I assume the OLED will have the pull-ups covered and there are no other devices on the I2C bus.

You need to capture more of these reboots. That is not an exception. That is a watchdog timer reset. I have never touched an Arduino, ESP, or microcontroller before a few days ago, and I barely have any SW experience, so I can’t tell you much more than that without having to spend a LOT of time digging through documentation first. But maybe you can go down that rabbit hole in parallel.

But watch dog timers are kind of fail-safes in case a program gets stuck in an impossible loop that was not foreseen/intended in the design. In that case, the program doesn’t get a chance to reset the this timer and when the timer runs out, it means something has gone horribly wrong and so the watchdog timer will force a reset to get back to a known state. That’s the general idea behind them – something I learned nearly three decades. The specifics in this case and for this microcontroller, I don’t know as of now.

#38

Update: I’m going down a few paths. First of all, in my discussion with a couple of contributors on the esp8266 Arduino core repository, I’m getting a lot of little prompts for me to dive into pieces of the ESP8266 architecture, particularly about the memory map and how instructions are cached. It seems from the exception dumps, the crash is always at one instruction and the PC points to that instruction located in the flash range (0x4020_0000 - 0x403F_0000 ).

With my discussions with one of the Core contributors, the suspicion is some ISR is sitting in flash instead of in IRAM. So, when an interrupt comes in, the cache miss forces the processor to go out to SPI to fetch the instructions and the SPI Flash could be slow/busy with another task and so the instructions end up being all zeros. Apparently, an illegal instruction opcode is three bytes of zeros. It’s not clear if that’s the ONLY cause of Exception 0. I would imagine any undefined opcode would cause the same Exception.

For those interested but don’t already possess the background knowledge: The Wemos D1 Mini stores program instructions primarily in flash memory, but that flash memory is an IC that sits on the board, outside of the processor. The processor needs to access it through the SPI bus, which is very slow compared to internal RAM. Instructions are cached and some special instructions, like ISRs (Interrupt Service Routines) need to be in internal RAM. PC = Program Counter, which is basically a pointer to the current/next instruction (it’s storing the address of the instruction). In the ESP8266 memory map https://github.com/esp8266/Arduino/issues/url I found, we can see where the PC is pointing to when the exception occurred.

Anyway, there’s more for me dig through and more questions to ask. This kind of deeper understanding could help us identify some obscure issue if we ever cross paths with it. But I will get to that a bit later.

A few hours ago, I had an idea. Recall that in an early update of mine, I mentioned that SoftwareSerial was suspect because the circular_queue belongs to that library and SofwareSerial showed up in the failed exception decode from PlatformIO. A couple more clues include my own testing where I started with the near-stock FW on a bare D1 Mini board and let it run for 12+ hours and then I slowly added pieces to it and tested for a couple of hours – first the PCB with OLED, then the TVOC on I2C, then S8 and then the PMS… It finally rebooted at least once within 30 minutes when I added the PMS. Several others here on the forum also mentioned PMS feeling like it’s potentially contributing to the crashes. And the original response from the Arduino core contributor to my issue report was that SoftwareSerial was likely and that v3.0.2 shipped with SoftwareSerial 6.12.7 while the recent v3.1.x ships with 7.0.0.

So, tonight, after my test with the board running over 14 hours with Core v3.1.1 built with the debug flag on, I was satisfied enough to call it a success and was able to pursue to the SoftwareSerial rabbit hole. I’ve taken the Core v3.1.1 source in my Arduino library, and replaced SoftwareSerial with version 6.12.7, recompiled and started the test. So far, so good at over 3h12m now.

If this proves to be stable, then I will revert to 7.0.0 to confirm that it crashes. Hopefully, it does. Then, onward to the long march to narrow down the exact release that caused the crashing. There are 13 releases between the two versions. I’ll have to do a binary-search algorithm since computer science tells us it’s technically the fastest way to search a ordered list. Might be fewer updates tomorrow as I may go into the office for my real full-time gig, but stay tuned!

PXL_20230127_104519187
PXL_20230127_1045191871471×1103 127 KB

#39

Many thanks for this extensive investigation. We also suspected the software serial in the past creating all kinds of different issues.
At some point in the future we want to switch to an ESP32 and use several hw serials instead and I believe this will increase the stability in general.

2 Likes
#40

While @ken830 is doing the real work, my AG has been up for over 30 hours after reverting the board to earlier version v3.0.2, so I can recommend that as the good-for-now solution :clap: :clap:

#41

Yes, hardware serial port will be 100% rock solid.

Yay! I’m so glad that even if I never get down to the bottom of this, at least we have several work-arounds that are effective. And I don’t think you give up anything with any of the workarounds. As you can see below, you can have the latest Arduino core if you swap out to the previous-to-last version of SoftwareSerial. Or you can have everything if you build it with the debug flag. At least with my limited time testing it (~12-24h). We’ll have to run for weeks to see if that holds up.

Keep in mind that my make-shift uptime display/print-out is crude and is just based on the internal Arduino millis() function, which is an unsigned-long (32-bits) that counts up the milliseconds from the beginning of time (when it boots). When it reaches it’s maximum value of 0xFFFF_FFFF, it’s going to rollover to 0x0000_0000. That takes exactly 2^32 milliseconds = ~49.7102696 days (not counting clock accuracy tolerances and drift). I guess we can write a bit of code to detect the overflow condition and count them in an unsigned-int , which would give us a limit of 2^(32+16) milliseconds = ~8,919.59429 years.

Update for the day:

The Core v3.1.1 with the older version of SoftwareSerial (v6.12.7, included in Core v3.0.x) was up for 14+ hours. I then moved on to test all the combinations of Core and SoftwareSerial, up to 8-hours each for the passing ones. Even with 13 releases between them, I took a chance and skipped all the way to the last 6.x.x release, which happens to be the previous-to-current release. This validated my hunch that it was major version 7.0.0 of SoftwareSerial that is making it crash. Here’s a handy chart of where we’ve been and where we are:

Test Configuration Default? Result
Core v3.1.1 + SoftwareSerial v7.0.0 [X] Exception 0
Core v3.1.1 + SoftwareSerial v6.17.1 – Working
Core v3.1.1 + SoftwareSerial V6.12.7 – Working
Core v3.0.2 + SoftwareSerial v7.0.0 – Exception 0
Core v3.0.2 + SoftwareSerial v6.17.1 – Working
Core v3.0.2 + SoftwareSerial V6.12.7 [X] Working
Core v3.1.1 + build_type = release [X] Exception 0
Core v3.1.1 + build_type = debug – Working
Core v3.1.1 Exception 0
Core v3.0.2 Working
Core v3.0.0 Working

Next steps? Well, the Core contributors had some hunches of what could be sources of the problem (related to ISRs not in IRAM), so I’ve capture a set of ELF files that may help to tell if that is the case. There’s also a pre-processor directive macro that was suggested that could hopefully make the stack dump more informative and give us a hint as to which part of the code called the circular_queue::available() function that is resulting in an Exception 0.

I may also reach out to the SoftwareSerial contributors via an issue report to see if they could make something out of it.

1 Like
#42

I’ve still been testing on the ESPHome side. Since @ken830 was going down the path of the serial output being a culprit, I was looking at my config with OLED, PMS5003, and SPG30 and reboots often, but not regular. I tried removing the OLED config (soldered to the board so couldn’t remove it easily) with no luck, but if I unplug the PMS5003, then it was stable, but no readout.

Watching the logs, I was flooded with readings from both PMS and SGP, even though I’m only reporting the readings back to HomeAssistant every 30 seconds. So if the serial output is causing issues, I tried reducing the Update_Interval from the default of every 1 second, to every 120 seconds. This cuts down on the output tremendously and so far looks really good, as I’ve been up for 16 hours without a reboot. I still need to re-enable my OLED in this config and then I may try going back to regular updates from the SPG and see if it is only the PMS or the other way around that is causing an issue with the serial output. I could also change the logging from the default of DEBUG and maybe go to INFO instead. (SGP30 has a message that to be optimized it needs to be updated every seconds)

image

[11:04:18][W][sgp30:289]: Update interval for SGP30 sensor must be set to 1s for optimized readout
[11:04:18][I][sgp30:127]: Current eCO2 baseline: 0x930C, TVOC baseline: 0x98D6
[11:04:47][D][pmsx003:234]: Got PM1.0 Concentration: 0 µg/m^3, PM2.5 Concentration 0 µg/m^3, PM10.0 Concentration: 0 µg/m^3
[11:04:47][D][sensor:126]: 'Particulate Matter <2.5µm Concentration': Sending state 0.00000 µg/m³ with 0 decimals of accuracy
[11:05:06][D][sensor:126]: 'WiFi Signal': Sending state -59.00000 dBm with 0 decimals of accuracy
[11:05:09][D][sensor:126]: 'Uptime Sensor': Sending state 56092.21875 s with 0 decimals of accuracy
[11:06:06][D][sensor:126]: 'WiFi Signal': Sending state -57.00000 dBm with 0 decimals of accuracy
[11:06:09][D][sensor:126]: 'Uptime Sensor': Sending state 56152.21875 s with 0 decimals of accuracy
[11:06:18][D][sgp30:282]: Got eCO2=400.0ppm TVOC=0.0ppb
[11:06:18][W][sgp30:289]: Update interval for SGP30 sensor must be set to 1s for optimized readout
[11:06:18][I][sgp30:127]: Current eCO2 baseline: 0x930C, TVOC baseline: 0x98D6
[11:06:47][D][pmsx003:234]: Got PM1.0 Concentration: 0 µg/m^3, PM2.5 Concentration 0 µg/m^3, PM10.0 Concentration: 0 µg/m^3
[11:06:47][D][sensor:126]: 'Particulate Matter <2.5µm Concentration': Sending state 0.00000 µg/m³ with 0 decimals of accuracy
[11:07:06][D][sensor:126]: 'WiFi Signal': Sending state -58.00000 dBm with 0 decimals of accuracy
[11:07:09][D][sensor:126]: 'Uptime Sensor': Sending state 56212.21875 s with 0 decimals of accuracy
[11:08:06][D][sensor:126]: 'WiFi Signal': Sending state -57.00000 dBm with 0 decimals of accuracy
[11:08:09][D][sensor:126]: 'Uptime Sensor': Sending state 56272.21875 s with 0 decimals of accuracy
[11:08:18][D][sgp30:282]: Got eCO2=400.0ppm TVOC=0.0ppb
[11:08:18][W][sgp30:289]: Update interval for SGP30 sensor must be set to 1s for optimized readout
AirGradient Pro with ESPhome, display keeps turning back on
#43

Thanks a ton @ken830 for all the effort, was pretty convinced coming into this that I had broken something with my soldering, don’t know if I could have figure all this out myself. I have reflashed my board with the 3.0.2 version today and haven’t noticed any issues so far. I am going to try adding the uptime code when I can to keep track of it as well.

Otherwise I did start noticing a variety of other issues in the last few days, not sure if they were new or if I was just paying more attention. I had the PM/AQI reading go negative for about ten minutes once, and a few times the temperature and humidity went to zero for a few seconds, then the whole screen refreshes and its back to normal. Otherwise there were just many many reboots sometimes multiple in the span of a few minutes. Hopefully I won’t have any more issues to report but I might try to take a crack at the debug myself at some point. Thanks again all!

#44

Realized you can’t edit posts after a certain time. Does anyone know if there is a way to pin posts or make them easily visible to anyone new who find this thread having this issue? I want to make the workaround easily visible.

WORKAROUND AS OF JAN 28 2023

============================================

If you are having frequent random reboots the temporary solution @ken830 has found is to revert your esp8266 Arduino Core version to v3.0.2, as some updates in v3.1.0 seem to be causing this issue. Ken has several posts in this thread exploring this issue and narrowing down that fix, if you wan to know more about why it happens feel free to scroll back and read some of them, but if you just want a quick fix this should help:

If you are interested in tracking the uptime of your board or helping to track this issue here is Ken’s code to print the uptime to the screen:

If you are unsure how to implement this fix:

  • Follow these instructions from AirGradient to set up the Arduino IDE, but instead of installing the latest version of the ESP8266 platform, install version 3.0.2.
    Install the Arduino Software and D1 Mini
  • Follow the relevent build instructions for your board to manually flash the firmware (under “Manual Flashing With The Arduino Software”)
    Built instructions
1 Like