Airgradient rebooting itself?

[ken830]

EDIT: CAN SOMEONE PLEASE REPLY TO THIS SO THAT I CAN POST AN UPDATE? The forum won’t let me post 3 consecutive replies. And I can’t add my update here because it exceeds the maximum length limit per post.

ESP8266 Arduino Core:

• v3.1.1 (2023-01-14) <== CRASHES
• v3.1.0 (2023-01-06) <== CRASHES
• v3.0.2 (2021-07-26) <== WORKS
• v3.0.1 (2021-06-26) <== WORKS
• v3.0.0 (2021-05-15) <== WORKS

PlatformIO platform-espressif8266:
*v4.1.0 (2023-01-16), Arduino Core v3.1.0 <== CRASHES
*v4.0.1 (2022-01-01), Arduino Core v3.0.2
*v4.0.0 (2022-05-31), Arduino Core v3.0.2
*v3.2.0 (2021-08-13), Arduino Core v3.0.2

It’s very clear that the v3.1.0 release of the ESP8266 Arduino Core is what makes the difference here. So, I set-off trying to figure out a way to decode the exception stack trace/dump. PlatformIO has built-in exception decoder “filter” that I enabled and I captures a slew of exceptions. I guess they are somewhat decoded, but I can’t make enough sense out of it.

Here are a few from PlatformIO:

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (0):
epc1=0x402150d0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Illegal instruction
  epc1=0x402150d0 in circular_queue<unsigned int, SoftwareSerial*>::available() const at ??:?

>>>stack>>>

ctx: sys
sp: 3ffffbc0 end: 3fffffb0 offset: 0190
3ffffd50:  401030b3 3ffecfe0 3ffecfb8 056e5a89
3ffffd60:  00007fff 0585c417 3ffed8c4 40103290
3ffffd70:  3ffe9e5c 00000000 00004000 40103290
3ffffd80:  00007fff 0585c417 4010374a c0037035
3ffffd90:  00000000 00000001 0000000e 40100768
3ffffda0:  00000020 00004208 00000000 00000001
3ffffdb0:  3ffe9e5c 00004208 00000000 00000022
3ffffdc0:  3fffc200 401006a4 3fffc258 4000050c
3ffffdd0:  40004748 00000030 00000020 ffffffff
3ffffde0:  40105833 04000002 00000000 00000001
3ffffdf0:  fbf8ffff 04000002 3feffe00 00000100
3ffffe00:  60000200 00eaa101 fffffffe 00000008
3ffffe10:  00000000 00000000 60000600 00000030
3ffffe20:  0000000c 00000000 00000020 00000022
3ffffe30:  4000050c 401006a4 3fffc258 4000050c  
3ffffe40:  40240b26 00000030 00000011 ffffffff
3ffffe50:  00000002 00000002 3ffef446 ffffffff
3ffffe60:  3ffee748 40241018 00000000 bfffffff
3ffffe70:  ffffffff 3fffc6fc 3ffe93b4 00000008
3ffffe80:  3ffee770 058a5a57 60000600 00000030
3ffffe90:  40101ed7 2c9f0300 3fffc228 40106059
3ffffea0:  4000050c 401006a4 3fffc258 4000050c
3ffffeb0:  40000f68 00000030 00000011 ffffffff
3ffffec0:  40000f58 00000000 00000020 00000000
3ffffed0:  40101d76 40241018 3ffe93b4 bfffffff
3ffffee0:  4023dcf1 3ffef430 3ffe93b4 3fffdab0
3ffffef0:  00000000 3fffdcb0 3ffee790 00000030
3fffff00:  000000b0 00000016 3ffe85ec 40239084
3fffff10:  00000008 00000000 f7ffffff 60000600
3fffff20:  00000008 00000000 f7ffffff 40223721
3fffff30:  40240b54 00000008 3ffee770 058a5a57
3fffff40:  40240f66 058a2de7 60000600 4021efd9
3fffff50:  40243294 3ffe93b4 4024102b 3ffe92fc
3fffff60:  3ffee770 01bb1298 3ffee770 60000600
3fffff70:  40248211 3ffe92fc 3ffee770 058a2de7
3fffff80:  40248256 3fffdab0 00000000 3fffdcb0
3fffff90:  3ffee798 40000f65 3fffdab0 00000000
3fffffa0:  40000f49 40000f49 3fffdab0 40000f49
<<<stack<<<

0x401030b3 in rcReachRetryLimit at ??:?
0x40103290 in rcReachRetryLimit at ??:?
0x40103290 in rcReachRetryLimit at ??:?
0x4010374a in wDev_ProcessFiq at ??:?
0x40100768 in interrupt_handler at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40105833 in Cache_Read_Enable_New at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40240b26 in pm_set_sleep_time at ??:?
0x40241018 in pm_get_sleep_type at ??:?
0x40101ed7 in ppProcessTxQ at ??:?
0x40106059 in ets_timer_disarm at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40101d76 in fix_cache_bug at ??:?
0x40241018 in pm_get_sleep_type at ??:?
0x4023dcf1 in register_chipv6_phy at ??:?
0x40239084 in pm_wakeup_init at ??:?
0x40223721 in sys_timeout_abs at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c:189
0x40240b54 in pm_set_sleep_time at ??:?
0x40240f66 in pm_get_sleep_type at ??:?
0x4021efd9 in memp_free at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/memp.c:447
0x40243294 in pp_noise_test at ??:?
0x4024102b in pm_get_sleep_type at ??:?
0x40248211 in ets_timer_handler_isr at ??:?
0x40248256 in ets_timer_handler_isr at ??:?


--------------- CUT HERE FOR EXCEPTION DECODER ---------------

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (0):
epc1=0x402150d0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Illegal instruction
  epc1=0x402150d0 in circular_queue<unsigned int, SoftwareSerial*>::available() const at ??:?

>>>stack>>>

ctx: sys
sp: 3ffffbd0 end: 3fffffb0 offset: 0190
3ffffd60:  401030b3 3ffecfb8 3ffffed0 00000000
3ffffd70:  00007fff 03cb1ff9 3ffed8c4 40103290
3ffffd80:  3ffe9e44 00000000 00004000 401001a0
3ffffd90:  00007fff 03cb1ff9 4010374a c0033035
3ffffda0:  00000000 00000001 0000000e 40100768
3ffffdb0:  00000020 00006208 00000000 40102528
3ffffdc0:  3ffe95a0 401057a7 3ffece78 00000022
3ffffdd0:  3fffc200 401006a4 3fffc258 4000050c
3ffffde0:  40004855 00000030 00000020 ffffffff
3ffffdf0:  40101d70 00e8a101 3feffe00 60000200
3ffffe00:  fffdffff 04000001 fffffffe 04000001  
3ffffe10:  04000003 00000001 04000002 00000008
3ffffe20:  00000000 00000000 60000600 00000030
3ffffe30:  4022fed0 3ffffed0 3ffedf38 00000022
3ffffe40:  4000050c 401006a4 3fffc258 4000050c
3ffffe50:  40240f5b 00000030 00000010 ffffffff
3ffffe60:  4024102b 00000008 3ffef446 ffffffff
3ffffe70:  3ffee748 40241018 00000000 bfffffff
3ffffe80:  ffffffff 3fffc6fc 00000000 3ffe92fc
3ffffe90:  3ffee770 03d90c89 60000600 00000030
3ffffea0:  4000050c 401006a4 3fffc258 4000050c
3ffffeb0:  40000f68 00000030 00000010 ffffffff
3ffffec0:  40000f58 00000000 00000020 00000000
3ffffed0:  00000002 40241018 00000000 00000000
3ffffee0:  4023dcf1 3ffe93b4 00000000 3fffdab0
3ffffef0:  00000000 3fffdcb0 3ffee780 00000030
3fffff00:  000000b0 00000016 3ffe85ec 40239084
3fffff10:  00000008 00000000 f7ffffff 60000600
3fffff20:  00000008 00000000 f7ffffff 40223721
3fffff30:  40240b54 00000008 3ffee770 03d90c89
3fffff40:  40240f66 0133925b 60000600 4021efd9
3fffff50:  03d8c006 3ffee770 4024102b 3ffe92fc
3fffff60:  3ffee770 0133aa44 3ffee770 60000600
3fffff70:  40248211 3ffe92fc 3ffee770 03d8c006
3fffff80:  40248256 3fffdab0 00000000 3fffdcb0
3fffff90:  3ffee788 40000f65 3fffdab0 00000000
3fffffa0:  40000f49 3ffe9660 3fffdab0 40000f49  
<<<stack<<<

0x401030b3 in rcReachRetryLimit at ??:?
0x40103290 in rcReachRetryLimit at ??:?
0x401001a0 in ets_post at ??:?
0x4010374a in wDev_ProcessFiq at ??:?
0x40100768 in interrupt_handler at ??:?
0x40102528 in pp_post at ??:?
0x401057a7 in lmacTxFrame at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40101d70 in fix_cache_bug at ??:?
0x4022fed0 in scan_parse_beacon at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40240f5b in pm_get_sleep_type at ??:?
0x4024102b in pm_get_sleep_type at ??:?
0x40241018 in pm_get_sleep_type at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40241018 in pm_get_sleep_type at ??:?
0x4023dcf1 in register_chipv6_phy at ??:?
0x40239084 in pm_wakeup_init at ??:?
0x40223721 in sys_timeout_abs at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c:189
0x40240b54 in pm_set_sleep_time at ??:?
0x40240f66 in pm_get_sleep_type at ??:?
0x4021efd9 in memp_free at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/memp.c:447
0x4024102b in pm_get_sleep_type at ??:?
0x40248211 in ets_timer_handler_isr at ??:?
0x40248256 in ets_timer_handler_isr at ??:?


--------------- CUT HERE FOR EXCEPTION DECODER ---------------

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (0):
epc1=0x402150d0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Illegal instruction
  epc1=0x402150d0 in circular_queue<unsigned int, SoftwareSerial*>::available() const at ??:?

>>>stack>>>

ctx: sys
sp: 3ffffbc0 end: 3fffffb0 offset: 0190
3ffffd50:  0000ffff ffffffff 00000005 40102528
3ffffd60:  3ffe95a5 401057bf 3ffecfb8 3ffffed0
3ffffd70:  401030b3 3ffecfb8 00004000 00000030  
3ffffd80:  00007fff 0ceca541 3ffed8c4 c0033035
3ffffd90:  00000000 00000001 0000000e 40100768
3ffffda0:  00000020 00000000 00000020 401001a0
3ffffdb0:  3ffe9e2c 7fffffff 00000000 00000022
3ffffdc0:  3fffc200 401006a4 3fffc258 4000050c
3ffffdd0:  400046c9 00000030 00000020 ffffffff  
3ffffde0:  40105833 00000000 00000000 00000001
3ffffdf0:  04000002 00e8a101 3feffe00 00000100
3ffffe00:  60000200 00000001 fffffffe 00000008
3ffffe10:  00000000 00000000 60000600 00000030
3ffffe20:  3ffeda00 3ffffed0 3ffee090 00000022
3ffffe30:  4000050c 401006a4 3fffc258 4000050c
3ffffe40:  40240b51 00000030 00000010 ffffffff
3ffffe50:  40240b4d 00000008 00000000 3feffe00
3ffffe60:  fc7000f7 fffffff7 00000000 bfffffff
3ffffe70:  ffffffff 3fffc6fc 00000000 00000008
3ffffe80:  3ffee770 0cf5ea72 60000600 00000030
3ffffe90:  3ffee770 3fff0f94 3fffc228 40106059
3ffffea0:  4000050c 401006a4 3fffc258 4000050c
3ffffeb0:  40000f68 00000030 00000010 ffffffff
3ffffec0:  40000f58 00000000 00000020 00000000
3ffffed0:  40101d76 40241018 3ffee748 00000000  
3ffffee0:  4023dcf1 3ffe93b4 00000000 3fffdab0
3ffffef0:  00000000 3fffdcb0 3ffee790 00000030
3fffff00:  00000000 0003513c 5b645a1c 40239084
3fffff10:  00000008 00000000 f7ffffff 60000600
3fffff20:  00000008 00000000 f7ffffff 402237c8
3fffff30:  40240b54 00000008 3ffee770 0cf5ea72
3fffff40:  40240f66 040c9eff 3ffefbec 60000600
3fffff50:  0cf5b546 3ffee770 4024102b 3ffe92fc
3fffff60:  3ffee770 040caf9d 3ffee770 60000600
3fffff70:  40248211 3ffe92fc 3ffee770 0cf5b546
3fffff80:  40248256 3fffdab0 00000000 3fffdcb0
3fffff90:  3ffee798 3fffdad0 3ffef458 4020e6e2
3fffffa0:  40000f49 3ffe95f8 3fffdab0 40000f49
<<<stack<<<

0x40102528 in pp_post at ??:?
0x401057bf in lmacRxDone at ??:?
0x401030b3 in rcReachRetryLimit at ??:?
0x40100768 in interrupt_handler at ??:?
0x401001a0 in ets_post at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40105833 in Cache_Read_Enable_New at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40240b51 in pm_set_sleep_time at ??:?
0x40240b4d in pm_set_sleep_time at ??:?
0x40106059 in ets_timer_disarm at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40101d76 in fix_cache_bug at ??:?
0x40241018 in pm_get_sleep_type at ??:?
0x4023dcf1 in register_chipv6_phy at ??:?
0x40239084 in pm_wakeup_init at ??:?
0x402237c8 in sys_timeout_LWIP2 at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c:306
0x40240b54 in pm_set_sleep_time at ??:?
0x40240f66 in pm_get_sleep_type at ??:?
0x4024102b in pm_get_sleep_type at ??:?
0x40248211 in ets_timer_handler_isr at ??:?
0x40248256 in ets_timer_handler_isr at ??:?
0x4020e6e2 in loop_task(ETSEventTag*) at core_esp8266_main.cpp:?


--------------- CUT HERE FOR EXCEPTION DECODER ---------------

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Exception (0):
epc1=0x402150d0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Illegal instruction
  epc1=0x402150d0 in circular_queue<unsigned int, SoftwareSerial*>::available() const at ??:?

>>>stack>>>

ctx: sys
sp: 3ffffbd0 end: 3fffffb0 offset: 0190
3ffffd60:  00007fff 022ef061 3ffed8c4 40103290
3ffffd70:  3ffe9e44 00000000 00000000 ffffff60
3ffffd80:  00007fff 022ef061 00000004 00000100  
3ffffd90:  3ffe9e44 7fffffff 00000000 c0037031
3ffffda0:  00000000 00000001 00000002 40100768
3ffffdb0:  00000020 3ffed030 00000000 022ef061  
3ffffdc0:  000005e0 00000000 401045c5 00000022
3ffffdd0:  3fffc200 401006a4 3fffc258 4000050c
3ffffde0:  40004826 00000030 00000020 ffffffff  
3ffffdf0:  40101d70 00eaa101 3feffe00 60000200
3ffffe00:  fffdffff 04000002 fffffffe 00000008
3ffffe10:  0115006c 00000001 fe7fffff 00000008
3ffffe20:  00000000 00000000 60000600 00000030
3ffffe30:  40104df3 3ffffed0 0000000c 40102528
3ffffe40:  3ffe95ac 40101e72 04000000 00040000
3ffffe50:  0000001b 000000f1 00000004 00000000
3ffffe60:  00000000 00000000 00000000 c0037035
3ffffe70:  00000000 00000001 00000002 40100768  
3ffffe80:  00000000 00000000 0000001f 401001a0
3ffffe90:  402418bd 2c9f0300 3fffc228 40106059
3ffffea0:  4000050c 401006a4 3fffc258 4000050c
3ffffeb0:  40000f68 00000030 00000010 ffffffff
3ffffec0:  40000f58 00000000 00000020 00000000
3ffffed0:  00000002 40241018 3ffee748 00000000
3ffffee0:  4023dcf1 3ffe93b4 00000000 3fffdab0
3ffffef0:  00000000 3fffdcb0 3ffee780 00000030
3fffff00:  000000b0 00000016 3ffe85ec 40239084
3fffff10:  00000008 00000000 f7ffffff 60000600
3fffff20:  00000008 00000000 f7ffffff 40223721
3fffff30:  40240b54 00000008 3ffee770 023387cf
3fffff40:  40240f66 00afdbb7 60000600 60000600
3fffff50:  02335356 3ffee770 4024102b 3ffe92fc  
3fffff60:  3ffee770 00aff0ca 3ffee770 60000600
3fffff70:  40248211 3ffe92fc 3ffee770 02335356
3fffff80:  40248256 3fffdab0 00000000 3fffdcb0
3fffff90:  3ffee788 3fffdad0 3ffef458 4020e6e2
3fffffa0:  40000f49 40000f49 3fffdab0 40000f49
<<<stack<<<

0x40103290 in rcReachRetryLimit at ??:?
0x40100768 in interrupt_handler at ??:?
0x401045c5 in lmacIsIdle at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40101d70 in fix_cache_bug at ??:?
0x40104df3 in lmacRecycleMPDU at ??:?
0x40102528 in pp_post at ??:?
0x40101e72 in pp_soft_wdt_feed_local at ??:?
0x40100768 in interrupt_handler at ??:?
0x401001a0 in ets_post at ??:?
0x402418bd in LightSleepWakedCbSetFunc at ??:?
0x40106059 in ets_timer_disarm at ??:?
0x401006a4 in interrupt_handler at ??:?
0x40241018 in pm_get_sleep_type at ??:?
0x4023dcf1 in register_chipv6_phy at ??:?
0x40239084 in pm_wakeup_init at ??:?
0x40223721 in sys_timeout_abs at /local/users/gauchard/arduino/arduino_esp8266/origin/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c:189
0x40240b54 in pm_set_sleep_time at ??:?
0x40240f66 in pm_get_sleep_type at ??:?
0x4024102b in pm_get_sleep_type at ??:?
0x40248211 in ets_timer_handler_isr at ??:?
0x40248256 in ets_timer_handler_isr at ??:?
0x4020e6e2 in loop_task(ETSEventTag*) at core_esp8266_main.cpp:?


--------------- CUT HERE FOR EXCEPTION DECODER ---------------

I will now try to get Arduino IDE V1.8 working with the new esp8266 arduino core by fixing that stupid bug they introduced. Maybe that decoder works better??

[makingmark]

@ken830

If you want to play along, could you check what version of the ESP8266 library you have installed on your Arduino IDE? If it’s newer than v3.0.0, could you try to roll it back to v3.0.0 and re-compile and see if the reboots go away?

@ken830 I think this is how I got confused about whether others were running stock firmware. I don’t have any library with ESP in the title when I look in the Arduino Library manager. Just the AirGradient library version 2.2.0, the others as directed by the build instuctions, and the S8_UART library I used to modify the CO2 calibration.

However…when I look in the file manager of my Mac, I see a path like: /Users/me/Library/Arduino15/packages/esp8266/hardware/esp8266/3.1.1.

How would I change this version?

[makingmark]

@ken830

@makingmark : Oh, and one more thing, since you mentioned you had to modify the code to change to Fahrenheit – does that mean you have PCB v3.7?

Yes, I have PCB v3.7.

[makingmark]

@ken830

EDIT: CAN SOMEONE PLEASE REPLY TO THIS SO THAT I CAN POST AN UPDATE?

see also my responses below (or above now, I guess)

[ken830]

Okay… Thanks to some posts on the issues and PR sections of the esp8266 Arduino core github page a few days ago , I was able to workaround the bug preventing me from compiling with core v3.1.0 and above by manually replacing the bad mkbuildoptglobals.py file. This allowed me to compile the sketch with the newer Core versions in the older Arduino IDE v1.8.x , which finally allowed me to run the ESP Exception Decoder. And the results look much better:

It crashes so often, I have decoded dozens of exceptions and each one is always the same:

circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420

atomic_base.h is an “internal header file” and line 420 looks like this:

25  /** @file bits/atomic_base.h
26   *  This is an internal header file, included by other library headers.
27   *  Do not attempt to use it directly. @headername{atomic}
28   */
.
.
.
419      _GLIBCXX_ALWAYS_INLINE __int_type
420      load(memory_order __m = memory_order_seq_cst) const noexcept
421      {
422	memory_order __b = __m & __memory_order_mask;
423	__glibcxx_assert(__b != memory_order_release);
424	__glibcxx_assert(__b != memory_order_acq_rel);

Here are some decoded exceptions:

Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000

Decoding stack results
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x4010062b: ets_intr_unlock() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 226
0x40228c85: sys_timeout_abs at core/timeouts.c line 189
0x401000ab: app_entry_redefinable() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 386
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40223559: glue2esp_linkoutput at glue-esp/lwip-esp.c line 301
0x40223787: new_linkoutput at glue-lwip/lwip-git.c line 272
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x4022cb94: ip4_output_if_opt_src at core/ipv4/ip4.c line 1764
0x40223559: glue2esp_linkoutput at glue-esp/lwip-esp.c line 301
0x40223787: new_linkoutput at glue-lwip/lwip-git.c line 272
0x40223be2: ethernet_output at netif/ethernet.c line 312
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x402176d3: Twi::WAIT_CLOCK_STRETCH() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/PolledTimeout.h line 130
0x40217974: Twi::read_bit() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 336
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40212c64: u8x8_byte_arduino_hw_i2c(u8x8_t*, uint8_t, uint8_t, void*) at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\U8x8lib.cpp line 1316
0x40202606: u8x8_cad_ssd13xx_fast_i2c at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_cad.c line 582
0x40212c48: u8x8_byte_arduino_hw_i2c(u8x8_t*, uint8_t, uint8_t, void*) at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\U8x8lib.cpp line 1316
0x402035dc: u8x8_byte_EndTransfer at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_byte.c line 61
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100a4c: micros() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring.cpp line 181
0x40100110: SoftwareSerial::preciseDelay() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\SoftwareSerial\src/SoftwareSerial.h line 246
0x4021790d: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 325
0x4021787d: Twi::write_stop() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 304
0x40217b01: Twi::readFrom(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 433
0x4021278c: TwoWire::requestFrom(unsigned char, unsigned int, bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 129
0x402127c0: TwoWire::requestFrom(unsigned char, unsigned char, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 137
0x40212966: SensirionI2CCommunication::receiveFrame(unsigned char, unsigned int, SensirionI2CRxFrame&, TwoWire&, CrcPolynomial) at C:\Users\Ken\Documents\Arduino\libraries\Sensirion_Core\src\SensirionI2CCommunication.cpp line 101
0x40216c68: __esp_suspend() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 64
0x40216db9: __esp_delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 161
0x40216e2e: esp_try_delay(unsigned int, unsigned int, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 182
0x40217d90: __delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/coredecls.h line 69
0x402037a7: updateTVOC() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 184
0x40204a84: loop() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 149

Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000

Decoding stack results
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40228c85: sys_timeout_abs at core/timeouts.c line 189
0x401000ab: app_entry_redefinable() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 386
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40216e54: esp_yield() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 144
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40214744: Print::print(__FlashStringHelper const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 106
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40214854: Print::printNumber (unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 262
0x4021e804: utoa at /workdir/repo/newlib/newlib/libc/stdlib/utoa.c line 74
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40101400: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x40213661: IPAddress::printTo(Print&) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 165
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a35: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(WiFiManager::wm_debuglevel_t, __FlashStringHelper const*, IPAddress) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 88
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x402157a1: String::copy(char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 258
0x40215971: String::operator=(char const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 303
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x402159f2: String::String(unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 82
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a86: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(__FlashStringHelper const*, IPAddress) at C:\Users\Ken\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 3281
0x402064d4: String::operator!=(char const*) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/WString.h line 234
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x402178c8: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 313
0x402179cb: Twi::write_byte(unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 347
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x402127ec: TwoWire::beginTransmission(unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 158
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100110: SoftwareSerial::preciseDelay() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\SoftwareSerial\src/SoftwareSerial.h line 246
0x4021790d: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 325
0x4021787d: Twi::write_stop() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 304
0x40217b01: Twi::readFrom(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 433
0x4021278c: TwoWire::requestFrom(unsigned char, unsigned int, bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 129
0x402127c0: TwoWire::requestFrom(unsigned char, unsigned char, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 137
0x40212966: SensirionI2CCommunication::receiveFrame(unsigned char, unsigned int, SensirionI2CRxFrame&, TwoWire&, CrcPolynomial) at C:\Users\Ken\Documents\Arduino\libraries\Sensirion_Core\src\SensirionI2CCommunication.cpp line 101
0x40216c68: __esp_suspend() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 64
0x40216db9: __esp_delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 161
0x40216e2e: esp_try_delay(unsigned int, unsigned int, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 182
0x40217d90: __delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/coredecls.h line 69
0x402037a7: updateTVOC() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 184
0x40204a84: loop() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 149

Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000

Decoding stack results
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x40228c85: sys_timeout_abs at core/timeouts.c line 189
0x40216bfa: loop_task(ETSEvent*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 273
0x401000ab: app_entry_redefinable() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 386
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x4020f304: ESP8266WiFiSTAClass::status() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\ESP8266WiFi\src\ESP8266WiFiSTA.cpp line 558
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x40214744: Print::print(__FlashStringHelper const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 106
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40214854: Print::printNumber (unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\Print.cpp line 262
0x40101400: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x40213661: IPAddress::printTo(Print&) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 165
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a35: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(WiFiManager::wm_debuglevel_t, __FlashStringHelper const*, IPAddress) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/IPAddress.h line 88
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x402157a1: String::copy(char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 258
0x40215971: String::operator=(char const*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 303
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x402159f2: String::String(unsigned long, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\WString.cpp line 82
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40208a86: WiFiManager::DEBUG_WM__FlashStringHelper const*, IPAddress>(__FlashStringHelper const*, IPAddress) at C:\Users\Ken\Documents\Arduino\libraries\WiFiManager\WiFiManager.cpp line 3281
0x402064d4: String::operator!=(char const*) const at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/WString.h line 234
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x4010107e: umm_free_core(umm_heap_context_t*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 642
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40217736: Twi::WAIT_CLOCK_STRETCH() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 167
0x4021790d: Twi::write_bit(bool) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 325
0x40217b48: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 368
0x4021347c: HardwareSerial::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/HardwareSerial.h line 193
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40100c04: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 167
0x40100b40: interrupt_handler(void*, void*) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_wiring_digital.cpp line 138
0x402176e2: Twi::WAIT_CLOCK_STRETCH() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 65
0x40100648: ets_post(uint8, ETSSignal, ETSParam) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 238
0x40101403: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x40101400: malloc(size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\umm_malloc\umm_malloc.cpp line 912
0x4021787d: Twi::write_stop() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 304
0x40217bb1: Twi::writeTo(unsigned char, unsigned char*, unsigned int, unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_si2c.cpp line 393
0x40212662: TwoWire::write(unsigned char const*, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 209
0x40212834: TwoWire::endTransmission(unsigned char) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\Wire\Wire.cpp line 172
0x40212d09: u8x8_byte_arduino_hw_i2c(u8x8_t*, uint8_t, uint8_t, void*) at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\U8x8lib.cpp line 1354
0x40202619: u8x8_cad_ssd13xx_fast_i2c at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_cad.c line 586
0x40202780: u8x8_d_ssd1306_sh1106_generic at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_d_ssd1306_128x64_noname.c line 327
0x402027b4: u8x8_d_sh1106_128x64_noname at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_d_ssd1306_128x64_noname.c line 481
0x4020364c: u8x8_DrawTile at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8x8_display.c line 89
0x40218864: uart_write(uart_t*, char const*, size_t) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\uart.cpp line 547
0x402019b5: u8g2_send_buffer at C:\Users\Ken\Documents\Arduino\libraries\U8g2\src\clib\u8g2_buffer.c line 89
0x40216c68: __esp_suspend() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/core_esp8266_features.h line 64
0x40216db9: __esp_delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 161
0x40216e2e: esp_try_delay(unsigned int, unsigned int, unsigned int) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266\core_esp8266_main.cpp line 182
0x40217d90: __delay(unsigned long) at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\cores\esp8266/coredecls.h line 69
0x40204f20: AirGradient::getCO2_Raw() at C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp line 691
0x402038f8: updateCo2() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 206
0x40204a8a: loop() at C:\Users\Ken\Documents\Arduino\sketch_jan24a/sketch_jan24a.ino line 151

[ken830]

@makingmark

@ken830 I think this is how I got confused about whether others were running stock firmware. I don’t have any library with ESP in the title when I look in the Arduino Library manager. Just the AirGradient library version 2.2.0, the others as directed by the build instuctions, and the S8_UART library I used to modify the CO2 calibration.

However…when I look in the file manager of my Mac, I see a path like: /Users/me/Library/Arduino15/packages/esp8266/hardware/esp8266/3.1.1.

How would I change this version?

This is the esp8266 Arduino Core. It’s installed under Board Manager, not Library manager. If you look there, you should see the version v3.1.1 installed and you should be able to revert down to v3.0.2, recompile, and flash the board.

@makingmark

see also my responses below (or above now, I guess)

Thanks! I have a lot more to report, but it’s past 6am here and I need to get some sleep. Typing this on my phone in bed.

[ken830]

NOTE: This is my 3rd post in a row, so someone reply to this so I can continue to post.

I opened an issue in the Arduino Core repository and one of the collaborators there has been responding and has been very helpful and patient with me. Their first response was that it’s likely something to do with SoftwareSerial. I think because circular_queue is part of SoftwareSerial. And they pointed out that EXCVADDR is pointing at nil (0x00000000) instead of an object.

He/she quoted this from my exception:

 Exception 0: Illegal instruction
PC: 0x40219d04: circular_queue ::available() const at c:\users\ken\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/atomic_base.h line 420
EXCVADDR: 0x00000000
...
0x40100110: SoftwareSerial::preciseDelay() at C:\Users\Ken\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.1\libraries\SoftwareSerial\src/SoftwareSerial.h line 246

So, that gave me a couple of leads to pursue. The first was the nil/NULL comment stuck me and I remember always see the warning about 2 NULL conversions whenever I compiled the code, but I noticed I didn’t get that warning when I built using <v3.0.2. Here are the build outputs from each.

Core v3.1.1

C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp: In member function 'TMP_RH AirGradient::returnError(TMP_RH_ErrorCode)':
C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp:548:14: warning: converting to non-pointer type 'float' from NULL [-Wconversion-null]
  548 |   result.t = NULL;
      |              ^~~~
C:\Users\Ken\Documents\Arduino\libraries\AirGradient_Air_Quality_Sensor\AirGradient.cpp:549:15: warning: converting to non-pointer type 'int' from NULL [-Wconversion-null]
  549 |   result.rh = NULL;
      |               ^~~~
. Variables and constants in RAM (global, static), used 33372 / 80192 bytes (41%)
║   SEGMENT  BYTES    DESCRIPTION
╠══ DATA     1516     initialized variables
╠══ RODATA   3832     constants       
╚══ BSS      28024    zeroed variables
. Instruction RAM (IRAM_ATTR, ICACHE_RAM_ATTR), used 62695 / 65536 bytes (95%)
║   SEGMENT  BYTES    DESCRIPTION
╠══ ICACHE   32768    reserved space for flash instruction cache
╚══ IRAM     29927    code in IRAM    
. Code in flash (default, ICACHE_FLASH_ATTR), used 355008 / 1048576 bytes (33%)
║   SEGMENT  BYTES    DESCRIPTION
╚══ IROM     355008   code in flash   
esptool.py v3.0
Serial port COM4
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 84:f3:eb:24:4b:d0
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 394432 bytes to 280039...
Writing at 0x00000000... (5 %)
Writing at 0x00004000... (11 %)
Writing at 0x00008000... (16 %)
Writing at 0x0000c000... (22 %)
Writing at 0x00010000... (27 %)
Writing at 0x00014000... (33 %)
Writing at 0x00018000... (38 %)
Writing at 0x0001c000... (44 %)
Writing at 0x00020000... (50 %)
Writing at 0x00024000... (55 %)
Writing at 0x00028000... (61 %)
Writing at 0x0002c000... (66 %)
Writing at 0x00030000... (72 %)
Writing at 0x00034000... (77 %)
Writing at 0x00038000... (83 %)
Writing at 0x0003c000... (88 %)
Writing at 0x00040000... (94 %)
Writing at 0x00044000... (100 %)
Wrote 394432 bytes (280039 compressed) at 0x00000000 in 6.4 seconds (effective 495.1 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Core v3.0.2

Executable segment sizes:
ICACHE : 32768           - flash instruction cache 
IROM   : 353468          - code in flash         (default or ICACHE_FLASH_ATTR) 
IRAM   : 29585   / 32768 - code in IRAM          (IRAM_ATTR, ISRs...) 
DATA   : 1516  )         - initialized variables (global, static) in RAM/HEAP 
RODATA : 3840  ) / 81920 - constants             (global, static) in RAM/HEAP 
BSS    : 27904 )         - zeroed variables      (global, static) in RAM/HEAP 
Sketch uses 388409 bytes (37%) of program storage space. Maximum is 1044464 bytes.
Global variables use 33260 bytes (40%) of dynamic memory, leaving 48660 bytes for local variables. Maximum is 81920 bytes.
esptool.py v3.0
Serial port COM4
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 84:f3:eb:24:4b:d0
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 392560 bytes to 277859...
Writing at 0x00000000... (5 %)
Writing at 0x00004000... (11 %)
Writing at 0x00008000... (17 %)
Writing at 0x0000c000... (23 %)
Writing at 0x00010000... (29 %)
Writing at 0x00014000... (35 %)
Writing at 0x00018000... (41 %)
Writing at 0x0001c000... (47 %)
Writing at 0x00020000... (52 %)
Writing at 0x00024000... (58 %)
Writing at 0x00028000... (64 %)
Writing at 0x0002c000... (70 %)
Writing at 0x00030000... (76 %)
Writing at 0x00034000... (82 %)
Writing at 0x00038000... (88 %)
Writing at 0x0003c000... (94 %)
Writing at 0x00040000... (100 %)
Wrote 392560 bytes (277859 compressed) at 0x00000000 in 6.3 seconds (effective 497.5 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Here is the code snippet from AirGradient.cpp. This is part of the overall set of code that reads the temperature (TMP) and relative humidity (RH) from the sensor. I’m guessing this piece defines the values that are returned if the sensor reading results in an error.

546   TMP_RH AirGradient::returnError(TMP_RH_ErrorCode error) {
547     TMP_RH result;
548     result.t = NULL;
549     result.rh = NULL;
550   
551     result.t_char[0] = 'N';
552     result.t_char[1] = 'U';
553     result.t_char[2] = 'L';
554     result.t_char[3] = 'L';
555   
556     result.rh_char[0] = 'N';
557     result.rh_char[1] = 'U';
558     result.rh_char[2] = 'L';
559     result.rh_char[3] = 'L';
560   
561     result.error = error;
562     return result;
}

But this turns out to not likely be related, Tmp/RH is not on SoftwareSerial, the warning isn’t indicative of the exception error, and there was a change in compiler warnings since the #8495 PR that’s probably suppressing the still-existent warnings.

The second lead was that I remembered that in the PlatformIO decoder filter output (the ones that I said weren’t very useful), there was SoftwareSerial mentioned:

Exception (0):
epc1=0x402150d0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

Illegal instruction
  epc1=0x402150d0 in circular_queue<unsigned int, SoftwareSerial*>::available() const at ??:?

>>>stack>>>

I posted this and mentioned that the output wasn’t very helpful and looked incomplete to which the response was that I needed -g in build_flags or build_type = debug (which does that for us) for the decoder to work. So, in an attempt to see if the PlatformIO decoder can give us more details/clues, I set the build_type = debug flag , re-compiled and ran it waiting for an exception… after waiting 15 minutes, I grew suspicious. So, I removed the flag, re-compiled, and ran it again. This time, it crashed within 5 minutes. I let it run for 15 minutes and it crashed a couple more times as expected. I then re-added the build_type = debug flag, re-compiled, and ran it. To my surprise, it didn’t crash for over an hour. I let it run and went to bed. I got up 6 hours later to find it was still running with no crashes/reboots! As I type this, it’s been over 7 hours and still going.

My assumption is building with the debug flag should only minimally affect code behavior, otherwise, it wouldn’t be very useful in debugging purposes. And so with that premise, I’m hopeful this clue can really narrow-in on possible root causes. I’m doing some reading or maybe someone with a better-suited skillset will be able to help solve this or point me in the right direction.

[makingmark]

Thanks ken830, just sent a PM about the uptime display code.

[MallocArray]

I’m learning a lot about ESPHome right now and see what you indicated that behind the scenes, it is using the PlatformIO code, even if I don’t have to deal with any of the sketches with ESPHome.

I did try to change which version it was using behind the screens. Before the top of my ESPHome config started with:

esp8266:
  board: d1_mini

I now changed it to:

esp8266:
  board: d1_mini
  framework:
    platform_version: 3.1.0

which does look like it lowers the version of PlatformIO to 3.1.0, which looks to be based on Arduino Core v3.0.1

Without specifying a version, my install defaults to platformio/espressif8266 @ 3.2.0 which is based on Arduino Core v3.0.2. So neither in ESPhome look to be using Core v3.1 which was suspected as the culprit above.

[ken830]

Update:

This is running for 10 hours with no crashes/reboots using the latest PlatformIO espressif8266 platform v4.1.0 released last week to include the latest Arduino Core v3.1.x! The only difference is compiling with the build_type = debug flag set.

@MallocArray : Super! Yeah, I’m just barely competent enough to poke/fumble around. Let me know how it goes running the older release. It would be interesting to see if it’s solved. By the way, are you able to monitor the serial output? I would like to see if it was the same exception 0 that I’m seeing when yours reboot. That is something no one has been able to confirm/deny for me yet.

BTW, I think you didn’t have to revert that far back. The previous-to-current release (platform-espressif8266 v4.0.1) is the release right before the move to Arduino Core v3.1.x, so it would’ve been Arduino Core v3.0.2.

[MallocArray]

Without doing anything special in the latest ESPHome, it is building with this:
Processing ag-pro (board: d1_mini; framework: arduino; platform: platformio/espressif8266 @ 3.2.0)
The documentation says it defaults to the “recommended” release, which apparently is 3.2.0 but I may try bumping it up higher and see if anything changes.

Less than 2 hours after going back to platformio 3.1.0 it rebooted again, so I set it back to the defaults of 3.2.0 and am trying it with i2c at 100 kHz from another thread’s conversation and see if that changes anything.

I can do a webpage monitoring the output, but hard to catch it when it randomly reboots somewhere between 30 minutes and 16 hours for me.

[ken830]

I wanted to see if you can plug-in a USB cable and run a serial monitor on your PC to watch the serial port output of the D1 Mini. Runtime Exceptions should be seen there. You just need a large enough screen buffer to scroll back to the time of reboot if it occurs when you’re away.

[MallocArray]

Seems to be worse in ESPHome with i2c at 100 kHz, as it is rebooting much more frequently.

[19:22:22][D][pmsx003:234]: Got PM1.0 Concentration: 0 µg/m^3, PM2.5 Concentration 0 µg/m^3, PM10.0 Concentration: 0 µg/m^3
[19:22:22][D][sensor:126]: 'Particulate Matter <2.5µm Concentration': Sending state 0.00000 µg/m³ with 0 decimals of accuracy
[19:22:23][D][sgp30:282]: Got eCO2=502.0ppm TVOC=146.0ppb
[19:22:23][D][sgp30:156]: Baseline reading not available for: 43167s
INFO 192.168.2.140: Ping timed out!
INFO Disconnected from ESPHome API for 192.168.2.140
WARNING Disconnected from API
WARNING Can't connect to ESPHome API for 192.168.2.140: Timeout while connecting to ('192.168.2.140', 6053)
INFO Trying to reconnect to 192.168.2.140 in the background
INFO Successfully connected to 192.168.2.140
[19:23:13][D][sgp30:282]: Got eCO2=432.0ppm TVOC=83.0ppb
[19:23:13][D][sgp30:156]: Baseline reading not available for: 43179s
[19:23:14][D][pmsx003:234]: Got PM1.0 Concentration: 0 µg/m^3, PM2.5 Concentration 0 µg/m^3, PM10.0 Concentration: 0 µg/m^3

[ken830]

What is your hardware configuration? I don’t know what the upgrade kit comes with, but I didn’t see you mention the OLED screen. You’ve checked your I2C pull-ups?

[MallocArray]

It is the same board and came with the OLED already soldered on along with all of the headers and the case, I supplied the rest of the components. I do have the OLED running. I haven’t done anything with the pullup resistors. I did add an SGP30 that I sourced myself on the 3.3v header, but I have reboots even with it removed, so only the D1 mini, PMS, and OLED.

I thought the issue with the pullup resistors was more related to getting erratic readings or 0 for readings, so since I’m getting data, I haven’t read up on them much.

I did set putty to record the serial output from the D1 mini running ESPHome, but nothing good in the logs. It just stops the regular output, does the boot sequence, and then starts output again.

[MallocArray]

Actually, I dug through a few more of the reboots and did find one large block of an Exception Decoder. Too much to put all of it here, but the highlights are:

Soft WDT reset

>>>stack>>>

ctx: sys
sp: 3fffed40 end: 3fffffb0 offset: 01a0
3fffeee0:  00000018 00012e0d 00000000 00000000  
3fffeef0:  00000000 00000000 00000001 401007a4  
3fffef00:  00000000 3ffeaa98 00000000 00000000  
3fffef10:  4025e908 3ffeaa98 3ffeff20 4022e614  

…

3fffff90:  3fffdad0 00000000 3fff08f8 4021ed30  
3fffffa0:  3fffdad0 00000000 3fff08f8 4022d9c0  
<<<stack<<<

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 
tail 4
chksum 0xcc
load 0x3fff20b8, len 40, room 4 
tail 4
chksum 0xc9
csum 0xc9
v00080030
~ld
[0;32m[I][logger:258]: Log initialized[0m
[0;35m[C][ota:469]: There have been 0 suspected unsuccessful boot attempts.[0m
[0;32m[I][app:029]: Running through setup()...[0m
[0;32m[I][i2c.arduino:175]: Performing I2C bus recovery[0m
[0;35m[C][uart.arduino_esp8266:059]: Setting up UART bus...[0m
[0;35m[C][sgp30:036]: Setting up SGP30...[0m
[sgp30:046]: Serial Number: 28065498[0m
[sgp30:066]: Product version: 0x22[0m
[0;35m[C][ssd1306_i2c:010]: Setting up I2C SSD1306...[0m
[0;35m[C][wifi:037]: Setting up WiFi...[0m

[ken830]

If you only have D1 mini, PMS, and OLED, then I wouldn’t worry about your pull-ups. I assume the OLED will have the pull-ups covered and there are no other devices on the I2C bus.

You need to capture more of these reboots. That is not an exception. That is a watchdog timer reset. I have never touched an Arduino, ESP, or microcontroller before a few days ago, and I barely have any SW experience, so I can’t tell you much more than that without having to spend a LOT of time digging through documentation first. But maybe you can go down that rabbit hole in parallel.

But watch dog timers are kind of fail-safes in case a program gets stuck in an impossible loop that was not foreseen/intended in the design. In that case, the program doesn’t get a chance to reset the this timer and when the timer runs out, it means something has gone horribly wrong and so the watchdog timer will force a reset to get back to a known state. That’s the general idea behind them – something I learned nearly three decades. The specifics in this case and for this microcontroller, I don’t know as of now.

[ken830]

Update: I’m going down a few paths. First of all, in my discussion with a couple of contributors on the esp8266 Arduino core repository, I’m getting a lot of little prompts for me to dive into pieces of the ESP8266 architecture, particularly about the memory map and how instructions are cached. It seems from the exception dumps, the crash is always at one instruction and the PC points to that instruction located in the flash range (0x4020_0000 - 0x403F_0000 ).

With my discussions with one of the Core contributors, the suspicion is some ISR is sitting in flash instead of in IRAM. So, when an interrupt comes in, the cache miss forces the processor to go out to SPI to fetch the instructions and the SPI Flash could be slow/busy with another task and so the instructions end up being all zeros. Apparently, an illegal instruction opcode is three bytes of zeros. It’s not clear if that’s the ONLY cause of Exception 0. I would imagine any undefined opcode would cause the same Exception.

For those interested but don’t already possess the background knowledge: The Wemos D1 Mini stores program instructions primarily in flash memory, but that flash memory is an IC that sits on the board, outside of the processor. The processor needs to access it through the SPI bus, which is very slow compared to internal RAM. Instructions are cached and some special instructions, like ISRs (Interrupt Service Routines) need to be in internal RAM. PC = Program Counter, which is basically a pointer to the current/next instruction (it’s storing the address of the instruction). In the ESP8266 memory map https://github.com/esp8266/Arduino/issues/url I found, we can see where the PC is pointing to when the exception occurred.

Anyway, there’s more for me dig through and more questions to ask. This kind of deeper understanding could help us identify some obscure issue if we ever cross paths with it. But I will get to that a bit later.

A few hours ago, I had an idea. Recall that in an early update of mine, I mentioned that SoftwareSerial was suspect because the circular_queue belongs to that library and SofwareSerial showed up in the failed exception decode from PlatformIO. A couple more clues include my own testing where I started with the near-stock FW on a bare D1 Mini board and let it run for 12+ hours and then I slowly added pieces to it and tested for a couple of hours – first the PCB with OLED, then the TVOC on I2C, then S8 and then the PMS… It finally rebooted at least once within 30 minutes when I added the PMS. Several others here on the forum also mentioned PMS feeling like it’s potentially contributing to the crashes. And the original response from the Arduino core contributor to my issue report was that SoftwareSerial was likely and that v3.0.2 shipped with SoftwareSerial 6.12.7 while the recent v3.1.x ships with 7.0.0.

So, tonight, after my test with the board running over 14 hours with Core v3.1.1 built with the debug flag on, I was satisfied enough to call it a success and was able to pursue to the SoftwareSerial rabbit hole. I’ve taken the Core v3.1.1 source in my Arduino library, and replaced SoftwareSerial with version 6.12.7, recompiled and started the test. So far, so good at over 3h12m now.

If this proves to be stable, then I will revert to 7.0.0 to confirm that it crashes. Hopefully, it does. Then, onward to the long march to narrow down the exact release that caused the crashing. There are 13 releases between the two versions. I’ll have to do a binary-search algorithm since computer science tells us it’s technically the fastest way to search a ordered list. Might be fewer updates tomorrow as I may go into the office for my real full-time gig, but stay tuned!

[Achim_AirGradient]

Many thanks for this extensive investigation. We also suspected the software serial in the past creating all kinds of different issues.
At some point in the future we want to switch to an ESP32 and use several hw serials instead and I believe this will increase the stability in general.

[makingmark]

While @ken830 is doing the real work, my AG has been up for over 30 hours after reverting the board to earlier version v3.0.2, so I can recommend that as the good-for-now solution