>>> Join Us in the Fight Against Air Pollution

AirGradient Forum

AirGradient Pro with ESPhome, display keeps turning back on

Software integrations
#33

Touching 20 hours now, but I’ve had 48 hours before but eventually it restarts so lets wait.

Latest(and also recommended) pulls version 3.0.2 now. I tested with 3.1.0/3.1.1 but they changed dhcpsoftap and then esphome breaks.

I dived a little more into how the dependencies work. Arduino framework looks for the release of SoftwareSerial at the point that version is released. So the issue on Esphome could be mitigated with only pulling a different softwareserial version, I guess. So I was using 6.12.7 before(Arduino 3.0.2) and that one crashes and now 6.17.1. I never used 7.0.0 because I never have used any newer Arduino framework version.
In the future updating the Arduino framework(when they revert to the older softwareserial) should work if Esphome supports it. That will probably take a long time so meanwhile we only can pull a different softwareserial to mitigate the issue(s) on Esphome.

#34

Nope still irregular restarts after 20, 46 and 28 hours. So nothing changed for me using the softwareserial 6.17.1.

Ill test again when a new arduino release with changed softwareserial releases.

#35

If it still restarts, then it’s likely not related (or not 100% caused by) SoftwareSerial. That’s good to know. Eliminates a big piece.

#36

I tried adding the same libraries lines to mine and I haven’t had reboots on my AG Pro where I originally found this issue, but now that I’m watching uptimes, I’m not seeing reboots on my original AirGradient DYI or IKEA monitor with an SGP40 and D1. Seems like the whole house starts rebooting or being stable around the same times, so I’m wondering if it is related to the data returns, such as when I hit certain temperatures or TVOC levels or something.

So not certain it solved anything for me at the moment, since my untouched configs are stable right now as well, but I’ll keep monitoring.

#37

I follow this thread with great interest. I had similar problems. Most recently I had an uptime of 66 hours. I posted my esphome file in this thread: Esphome with graphs - #5 by argafal

If you try it I’d be most curious to hear if/how it works for you.

#38

@MallocArray You make an interesting observation here. I follow this thread since I have (had?) a similar issue with my own esphome yaml. My random reboots have also disappeared the last few days. I cannot understand why. I guess I should simply be happy. :wink:

At the off chance that some external condition could be a contributing factor I will document what I observed below:

  • In a post on Feb 28 I published an esphome yaml with graphs: Esphome with graphs - #4 by argafal
  • I commented that it worked well for me but randomly reboots every 18-24 hours. My only local change to the published file: I ran with mqtt enabled.
  • A day or two later, I added an uptime counter as an additional sensor.
  • My random reboots have disappeared.
  • In my post today Mar 5 (Esphome with graphs - #5 by argafal) I posted the most recent configuration I use. Given that I always had MQTT enabled, it is identical to the Feb 28 one except for the additional uptime counter (and some updated comments).
  • I do not use a TVOC sensor, I do not have one.
  • I looked at my graphs for temperature, humidity, particles density, and co2 for the last two weeks. I do not see any obvious change in trend.

@MallocArray Did you always have an uptime sensor in your definition or did you recently add it too, by any chance?

#39

I did not always have an uptime, but my SGP40 for TVOC needs 12 hours to self-calibrate and I couldn’t get that due to the reboots, which is when I started tracking uptime, but I only did it on my AG Pro. After fiddling with it for months, I added uptime sensors to my other two (AG DIY and IKEA) and discovered they were rebooting too, even the IKEA with a different PMS sensor and not an air gradient board at all.

I opened a Issue on the ESPHome github and they asked for logs, so I moved my AG Pro to my ESPHome server to collect decrypted logs and then everything went stable for me. I also had a power outage in my house that had rebooted everything, so wasn’t sure I was having a network issue, but after a few days, everything started rebooting again. That is what leads me to think maybe it is a particular reading, such as Ken mentioned certain ending conditions causing the serial library to force a reboot.

#40

My random reboots are back now. It was nice while it lasted :smiley:

I’ll be curious to try the new PCB and see if that gives any improvements. With my PCB v3.3 I had to change the i2c frequency depending on the MCU (D1 vs C3), otherwise the display doesn’t work. I’m also still getting occasional error messages from the particle sensor and also occasionally from the Co2 sensor. I have changed the version of SoftwareSerial already.

How is it going for you, @MallocArray ?

#41

About the same. I’m not getting as frequent of reboots, but they do happen sometimes, but I also see the unmodified ESPHome install with similar stability, so I don’t know what to think

1 Like
#42

Same here. Occasional reboots every other day. It would be best to find out where it is coming from for every crash but that requires logging everything from serial and possibly removing and adding every single sensor to reduce the possibilities. At this moment it isnt really convenient for me to do that and I don’t have a second board to test it. I don’t really want to stop logging data for too long.

Another way to look into these issues is to look here GitHub - nkitanov/iaq_board: IAQ Board is a DIY (Do-It-Yourself) device for measuring internal air quality
It looks like most of the hardware is the same and they encountered the sgp31 issues also.

Or just put all the sensors on a breadboard and see if the same issues arise after a while.
If someone has a good test strategy we can try to collectively find the main issue(s)

#43

It’s either an individual, hardware specific issue (not likely, since there are multiple of you), or there is an issue with ESPHome. I say that because I’ve been running mostly-stock AirGradient code with OLED, SHT, TVOC, S8, PM5003 on a modified v3.3 PCB with OLED on 5V and SHT on 3.3V, I2C at 100kHz, Arduino Core 3.1.0, Software Serial v6.12.7 and it has been running non-stop for 17 days, 21 hours, and 9 minutes with zero reboots, zero sensor dropouts/timeout. Rock solid.

#44

I’d concur that there is a problem with ESPHome. Having reduced the pair of devices I have to just literally the ESPHome WiFi stack and an uptime sensor, they still reboot every few hours.

It’s worth noting that ESPHome has several built-in reboots e.g. WiFi drops for a period, though detecting these is tricky when the presence of debug logging seems to sometimes be enough to cause more instability!

What I notice about @argafal’s approach is that the Home Assistant native API is not in use. I might have to try the MQTT approach. Despite the native one being ‘recommended’, I generally have more faith in MQTT.

Thanks all by the way for chipping in thoughts, this is both frustrating and interesting!

#45

Having trialed some changes for a while, I am fairly confident in saying that this set of ESP lines are causing the problem for me. No sensors needed, just enabling UART seems to start reboot cycles. Remove this and the device is rock solid (albeit doing very little!)

esp8266:
  board: d1_mini
  framework:
    version: recommended
    platform_version: 3.2.0

esphome:
  name: "${devicename}"
  libraries:
    - uart=https://github.com/plerup/espsoftwareserial.git#6.17.1

# Remove this, reboots solved
uart:
  - rx_pin: D5
    tx_pin: D6
    baud_rate: 9600
    id: uart1
  - rx_pin: D4
    tx_pin: D3
    baud_rate: 9600
    id: uart2

So this does seem to point back to some instability around the UART code again, but I was under the impression 6.17.1 did not have the major errors the new versions did. Very odd; I’ll see if I can carve out some time to see what the actual reboot failures are.

The graph downloaded confirms that, unless the build is really broken in dependency management somehow, that it has the right libraries

Dependency Graph
|-- ESPAsyncTCP-esphome @ 1.2.3
|-- EspSoftwareSerial @ 6.17.1+sha.12f8480
|-- ESPAsyncWebServer-esphome @ 2.1.0
|   |-- ESPAsyncTCP-esphome @ 1.2.3
|   |-- Hash @ 1.0
|   |-- ESP8266WiFi @ 1.0
|-- DNSServer @ 1.1.1
|-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|-- AsyncMqttClient-esphome @ 0.8.6
|   |-- ESPAsyncTCP-esphome @ 1.2.3
|-- ArduinoJson @ 6.18.5
1 Like
#46

Just following up, I got the following software WDT errors in two stack dumps

Stack 1 
WARNING Found stack trace! Trying to decode it                                                                                                                                                                                      
WARNING Decoded 0x402338f0: tcp_tmr at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/core/tcp.c:239                                                                                             
WARNING Decoded 0x402418b6: sta_input                                                                                                                                                                                               
WARNING Decoded 0x402321f3: tcpip_tcp_timer at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c:153                                                                                
WARNING Decoded 0x40253688: pm_keep_active_enable                                                                                                                                                                                   
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x40259669: ets_timer_handler_isr                                                                                                                                                                                   
WARNING Decoded 0x402596ae: ets_timer_handler_isr                                                                                                                                                                                   
WARNING Decoded 0x40222772: loop_task(ETSEventTag*) at core_esp8266_main.cpp                                                                                                                                                        
WARNING Decoded 0x40105a8d: call_user_start_local                                                                                                                                                                                   
WARNING Decoded 0x40105a93: call_user_start_local                                                                                                                                                                                   
WARNING Decoded 0x4010000d: call_user_start                                                                                                                                                                                         
WARNING Decoded 0x401000ab: app_entry_redefinable                                                                                                                                                                                   
WARNING Decoded 0x402498dc: cont_ret at cont.S.o                                                                                                                                                                                    
WARNING Decoded 0x4024988d: cont_continue at cont.S.o                                                                                                                                                                               
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x40106225: ets_timer_disarm                                                                                                                                                                                        
WARNING Decoded 0x40100aa5: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x40100a98: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x4010028c: esphome::ISRInternalGPIOPin::digital_read()                                                                                                                                                             
WARNING Decoded 0x401058d9: lmacTxFrame                                                                                                                                                                                             
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x40102b25: rcUpdateTxDone                                                                                                                                                                                          
WARNING Decoded 0x401026f4: pp_post                                                                                                                                                                                                 
WARNING Decoded 0x40105973: lmacTxFrame                                                                                                                                                                                             
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)                                                                                                                  
WARNING Decoded 0x4010345c: rcReachRetryLimit                                                                                                                                                                                       
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)                                                                                                                  
WARNING Decoded 0x401026f4: pp_post                                                                                                                                                                                                 
WARNING Decoded 0x40100a98: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x40104b4e: lmacRecycleMPDU                                                                                                                                                                                         
WARNING Decoded 0x40104fbf: lmacRecycleMPDU                                                                                                                                                                                         
WARNING Decoded 0x401009d4: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x402090f8: esphome::socket::LWIPRawImpl::accept(sockaddr*, unsigned int*)                                                                                                                                          
WARNING Decoded 0x40207a2b: esphome::ota::OTAComponent::handle_()                                                                                                                                                                   
WARNING Decoded 0x402090f8: esphome::socket::LWIPRawImpl::accept(sockaddr*, unsigned int*)                                                                                                                                          
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)                                                                                                                  
WARNING Decoded 0x40100a98: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401009d4: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x4022c66a: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:520                                                                                                                                           
WARNING Decoded 0x4022c65b: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:519                                                                                                                                           
WARNING Decoded 0x4022c65b: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:519                                                                                                                                  [120/230]
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x40207a2b: esphome::ota::OTAComponent::handle_()                                                                                                                                                                   
WARNING Decoded 0x40100a98: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)                                                                                                                  
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x4022df7e: __d2b at /workdir/repo/newlib/newlib/libc/stdlib/mprec.c:779                                                                                                                                            
WARNING Decoded 0x40100a98: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x4022ccd9: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854                                                                                                                                           
WARNING Decoded 0x4022df7e: __d2b at /workdir/repo/newlib/newlib/libc/stdlib/mprec.c:779                                                                                                                                            
WARNING Decoded 0x4022ccd9: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854                                                                                                                                           
WARNING Decoded 0x4022e254: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232                                                                                                                                
WARNING Decoded 0x402290e1: __cvt at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf_float.c:102                                                                                                                               
WARNING Decoded 0x4022e254: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232                                                                                                                                
WARNING Decoded 0x4022e190: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:179                                                                                                                                
WARNING Decoded 0x40230f47: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned int&, unsigned int) at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basi
c_string.tcc:156                                                                                                                                                                                                                    
WARNING Decoded 0x40229599: _printf_float at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf_float.c:326 (discriminator 4)                                                                                                     
WARNING Decoded 0x40101202: free                                                                                                                                                                                                    
WARNING Decoded 0x40231800: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::substr(unsigned int, unsigned int) const at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/ba
sic_string.h:2837                                                                                                                                                                                                                   
WARNING Decoded 0x4022e190: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:179                                                                                                                                
WARNING Decoded 0x402313a4: std::char_traits<char>::assign(char&, char const&) at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/char_traits.h:329                                                        
 (inlined by) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_set_length(unsigned int) at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basic_string.h:219           
 (inlined by) std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace_aux(unsigned int, unsigned int, unsigned int, char) at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include
/bits/basic_string.tcc:419                                                                                                                                                                                                          
WARNING Decoded 0x4022e690: _svfprintf_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:658                                                                                                                              
WARNING Decoded 0x40231834: operator delete(void*) at /workdir/repo/gcc-gnu/libstdc++-v3/libsupc++/del_op.cc:50                                                                                                                     
WARNING Decoded 0x40230f64: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basic_string.h:235            
WARNING Decoded 0x4021d1c4: AsyncMqttClient::publish(char const*, unsigned char, bool, char const*, unsigned int, bool, unsigned short)                                                                                             
WARNING Decoded 0x40230f64: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basic_string.h:235            
WARNING Decoded 0x4022a469: snprintf at /workdir/repo/newlib/newlib/libc/stdio/snprintf.c:87 (discriminator 4)                                                                                                                      
WARNING Decoded 0x402227f1: esp_yield                                                                                                                                                                                               
WARNING Decoded 0x40222f15: __delay                                                                                                                                                                                                 
WARNING Decoded 0x40101010: umm_free_core at umm_malloc.cpp                                                                                                                                                                         
WARNING Decoded 0x40101202: free                                                                                                                                                                                                    
WARNING Decoded 0x40231834: operator delete(void*) at /workdir/repo/gcc-gnu/libstdc++-v3/libsupc++/del_op.cc:50                                                                                                                     
WARNING Decoded 0x40230f64: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basic_string.h:235            
WARNING Decoded 0x4020555d: esphome::mqtt::MQTTClientComponent::publish(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*, unsigned int, unsigned char, bool)                     
WARNING Decoded 0x4021052b: void std::__push_heap<__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Sc
heduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, std::unique
_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp_val<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Schedule
r::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)> >(__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, s
td::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler
::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, int, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp
_val<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::Schedul
erItem> > const&)>&)                                                                                                                                                                                                                
WARNING Decoded 0x40101202: free                                                                                                                                                                                                    
WARNING Decoded 0x402108fc: void std::__adjust_heap<__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::
Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, std::uniq
ue_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp_iter<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Sched
uler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)> >(__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem
, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Schedu
ler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, int, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_c
omp_iter<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::Sch
edulerItem> > const&)>)                                                                                                                                                                                                             
WARNING Decoded 0x40210320: esphome::Scheduler::SchedulerItem::cmp(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::Scheduler
Item, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)
WARNING Decoded 0x40206c89: esphome::mqtt::MQTTSensorComponent::publish_state(float)                                                                                                                                        [60/230]
WARNING Decoded 0x40210320: esphome::Scheduler::SchedulerItem::cmp(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::Scheduler
Item, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)                                                                                                                                                              
WARNING Decoded 0x40218ef8: esp8266::polledTimeout::timeoutTemplate<false, esp8266::polledTimeout::YieldPolicy::DoNothing, esp8266::polledTimeout::TimePolicy::TimeUnit<esp8266::polledTimeout::TimePolicy::TimeSourceMillis, 1000ul
l> >::expiredOneShot() const                                                                                                                                                                                                        
WARNING Decoded 0x40106369: ets_timer_arm_new                                                                                                                                                                                       
WARNING Decoded 0x402227f1: esp_yield                                                                                                                                                                                               
WARNING Decoded 0x40222f15: __delay                                                                                                                                                                                                 
WARNING Decoded 0x4020e3fb: esphome::Application::loop()

(Stack 2 in next post due to post length)

which both look reliably inside ESP8266SoftwareSerial::read_bit

I’m not sure I’m much closer to understanding the problem, though I shall try again with SoftwareSerial 8.0.1 and see if anything changes. I did read on espsoftwareserial github that there can be issues with WiFi interrupts being higher priority than UART interrupts and thus WiFi can be a potential cause of timeouts, so I think that’s probably my next mole to try whacking…

It’s slightly frustrating to keep running into advice that could be summarised as ‘Don’t use ESP8266’ when the existing Pro hardware is already purchased & using it :sweat_smile:

#47
Stack 2 
WARNING Found stack trace! Trying to decode it                                                                                                                                                                                      
WARNING Decoded 0x40232123: lwip_cyclic_timer at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/core/timeouts.c:261                                                                              
WARNING Decoded 0x402601bc: etharp_output                                                                                                                                                                                           
WARNING Decoded 0x4023207d: memp_free at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/core/memp.c:447                                                                                          
WARNING Decoded 0x40253688: pm_keep_active_enable                                                                                                                                                                                   
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x40259669: ets_timer_handler_isr                                                                                                                                                                                   
WARNING Decoded 0x402596ae: ets_timer_handler_isr                                                                                                                                                                                   
WARNING Decoded 0x40222772: loop_task(ETSEventTag*) at core_esp8266_main.cpp                                                                                                                                                        
WARNING Decoded 0x40105a8d: call_user_start_local                                                                                                                                                                                   
WARNING Decoded 0x40105a93: call_user_start_local                                                                                                                                                                                   
WARNING Decoded 0x4010000d: call_user_start                                                                                                                                                                                         
WARNING Decoded 0x401000ab: app_entry_redefinable                                                                                                                                                                                   
WARNING Decoded 0x402498dc: cont_ret at cont.S.o                                                                                                                                                                                    
WARNING Decoded 0x4024988d: cont_continue at cont.S.o                                                                                                                                                                               
WARNING Decoded 0x401058d9: lmacTxFrame                                                                                                                                                                                             
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x40102b25: rcUpdateTxDone                                                                                                                                                                                          
WARNING Decoded 0x401026f4: pp_post                                                                                                                                                                                                 
WARNING Decoded 0x4010598b: lmacRxDone                                                                                                                                                                                              
WARNING Decoded 0x4010327f: rcReachRetryLimit                                                                                                                                                                                       
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x4010345c: rcReachRetryLimit                                                                                                                                                                                       
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)                                                                                                                  
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)                                                                                                                  
WARNING Decoded 0x40100a98: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x40207a2b: esphome::ota::OTAComponent::handle_()                                                                                                                                                                   
WARNING Decoded 0x401009d4: interrupt_handler                                                                                                                                                                                       
WARNING Decoded 0x40207a32: esphome::ota::OTAComponent::handle_()                                                                                                                                                                   
WARNING Decoded 0x40207a2b: esphome::ota::OTAComponent::handle_()                                                                                                                                                                   
WARNING Decoded 0x402090f8: esphome::socket::LWIPRawImpl::accept(sockaddr*, unsigned int*)                                                                                                                                          
WARNING Decoded 0x40104880: lmacProcessTXStartData                                                                                                                                                                                  
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)                                                                                                                     
WARNING Decoded 0x401037e7: wDev_ProcessFiq                                                                                                                                                                                         
WARNING Decoded 0x401054be: lmacMSDUAged                                                                                                                                                                                            
WARNING Decoded 0x40103638: wDev_ProcessFiq                                                                                                                                                                                         
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x4022c4b8: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:476                                                                                                                                           
WARNING Decoded 0x4022c4b8: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:476                                                                                                                                           
WARNING Decoded 0x40255e9f: pp_attach                                                                                                                                                                                               
WARNING Decoded 0x40255e9f: pp_attach                                                                                                                                                                                               
WARNING Decoded 0x40207a2b: esphome::ota::OTAComponent::handle_()                                                                                                                                                                   
WARNING Decoded 0x40255eee: pp_attach                                                                                                                                                                                               
WARNING Decoded 0x401007a4: ets_post                                                                                                                                                                                                
WARNING Decoded 0x401026f4: pp_post                                                                                                                                                                                                 
WARNING Decoded 0x40254faf: ppTxPkt                                                                                                                                                                                                 
WARNING Decoded 0x4023e1d3: ieee80211_output_pbuf                                                                                                                                                                                   
WARNING Decoded 0x4022df7e: __d2b at /workdir/repo/newlib/newlib/libc/stdlib/mprec.c:779                                                                                                                                            
WARNING Decoded 0x4022df7e: __d2b at /workdir/repo/newlib/newlib/libc/stdlib/mprec.c:779                                                                                                                                            
WARNING Decoded 0x4022ccd9: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854                                                                                                                                           
WARNING Decoded 0x4022df7e: __d2b at /workdir/repo/newlib/newlib/libc/stdlib/mprec.c:779
WARNING Decoded 0x4022ccd9: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854
WARNING Decoded 0x4022ccd9: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854
WARNING Decoded 0x4022df7e: __d2b at /workdir/repo/newlib/newlib/libc/stdlib/mprec.c:779
WARNING Decoded 0x4022ccd9: _dtoa_r at /workdir/repo/newlib/newlib/libc/stdlib/dtoa.c:854
WARNING Decoded 0x4022e254: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232
WARNING Decoded 0x402290e1: __cvt at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf_float.c:102
WARNING Decoded 0x4022e254: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:232
WARNING Decoded 0x4022e190: __ssputs_r at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf.c:179
WARNING Decoded 0x40230f47: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned int&, unsigned int) at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basic_string.tcc:156
WARNING Decoded 0x40229599: _printf_float at /workdir/repo/newlib/newlib/libc/stdio/nano-vfprintf_float.c:326 (discriminator 4)
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)
WARNING Decoded 0x40231800: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::substr(unsigned int, unsigned int) const at /workdir/arena.x86_64/gcc-gnu/xtensa-lx106-elf/libstdc++-v3/include/bits/basic_string.h:2837
WARNING Decoded 0x40100a98: interrupt_handler
WARNING Decoded 0x401009d4: interrupt_handler
WARNING Decoded 0x401007a4: ets_post
WARNING Decoded 0x402216c9: String::copy(char const*, unsigned int)
WARNING Decoded 0x40106225: ets_timer_disarm
WARNING Decoded 0x40100aa5: interrupt_handler
WARNING Decoded 0x40100a98: interrupt_handler
WARNING Decoded 0x4010028c: esphome::ISRInternalGPIOPin::digital_read()
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)
WARNING Decoded 0x40100a98: interrupt_handler
WARNING Decoded 0x40100a98: interrupt_handler
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)
WARNING Decoded 0x4010032c: esphome::uart::ESP8266SoftwareSerial::read_bit_(unsigned int*, unsigned int const&)
WARNING Decoded 0x401003aa: esphome::uart::ESP8266SoftwareSerial::gpio_intr(esphome::uart::ESP8266SoftwareSerial*)
WARNING Decoded 0x40100a98: interrupt_handler
WARNING Decoded 0x40100a98: interrupt_handler
WARNING Decoded 0x4025f8b0: chip_v6_unset_chanfreq
WARNING Decoded 0x401009d4: interrupt_handler
WARNING Decoded 0x4021052b: void std::__push_heap<__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp_val<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)> >(__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, int, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp_val<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)>&)
WARNING Decoded 0x4020d300: esphome::wifi::WiFiComponent::wifi_sta_ip()
WARNING Decoded 0x402108fc: void std::__adjust_heap<__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp_iter<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)> >(__gnu_cxx::__normal_iterator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >*, std::vector<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, std::allocator<std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > > > >, int, int, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> >, __gnu_cxx::__ops::_Iter_comp_iter<bool (*)(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)>)
WARNING Decoded 0x40210320: esphome::Scheduler::SchedulerItem::cmp(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)
WARNING Decoded 0x40210320: esphome::Scheduler::SchedulerItem::cmp(std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&, std::unique_ptr<esphome::Scheduler::SchedulerItem, std::default_delete<esphome::Scheduler::SchedulerItem> > const&)
WARNING Decoded 0x4020dc2c: esphome::wifi::WiFiComponent::wifi_ssid[abi:cxx11]()
WARNING Decoded 0x40218ef8: esp8266::polledTimeout::timeoutTemplate<false, esp8266::polledTimeout::YieldPolicy::DoNothing, esp8266::polledTimeout::TimePolicy::TimeUnit<esp8266::polledTimeout::TimePolicy::TimeSourceMillis, 1000ull> >::expiredOneShot() const
WARNING Decoded 0x40106369: ets_timer_arm_new
WARNING Decoded 0x402227f1: esp_yield
WARNING Decoded 0x40222f15: __delay
WARNING Decoded 0x4020e3fb: esphome::Application::loop()
#48

I’ve been away for a bit but I’ve seen my device mostly restart on hardware wdt and a occasional wdt 4 so software. No stack dumps saved sadly. My uptime is now around 5/6 days. And I also run espsoftwareserial 8.0.1 and all other the recommended versions.

I have about 20 esp devices with esphome and all run the native api. Only the airgradient has reset issues so I don’t want to blame the api at this point. Many have uptimes of months.

I’ve seen that esphome 2023.4.2 released today has a fix for i2c reset issue which can cause a hardware wdt. I updated immediatly and I will test again for next days/weeks. If it only resets on software error it makes it a lot easier to debug.

1 Like
#49

Some progress. I was able to narrow down the UART problem more. The specific problem area seems to be 9600 baud. If I init the board with this, I’ll get regular reboots (with zero sensors listed in ESPHome). With 115200 (which of course renders the associated sensors useless), the board appears to be stable.

1 Like
#50

I just found a Github issue that sounds almost identical to this problem. Frustratingly it was closed as theoretically fixed in ESPSoftwareSerial 5.3.0. It’s still interesting to see that this sort of thing has cropped before.

1 Like
#51

I was curious if anyone on this thread has had any luck proving that ESPHome actually pays attention to version information. I notice some really odd behaviour, when I select framework version & platform_version.

Changing version to e.g. 3.1.1 yields no change other than a warning about not using the recommended version.
Changing platform_version yields no messaging whatsoever.
Changing both to e.g. 3.1.1 & 3.2.0 respectively yields weird messages like Installing platformio/framework-arduinoespressif8266 @ ~3.30101.0, which bears no resemblance to 3.2.0, but does mention 3.1.1 in its documentation (3.1.1 of what, nobody bothered to document).

None of this behaviour seems to match what is written in ESPHome’s documentation.

Other than whichever bunch of devs responsible for this need training in versioning semantics and documentation, does anyone know whether we can prove we’re actually running a given version of these packages?

ESPHome’s build dependency graph looks to be semi-useless, because it doesn’t even mention any of these packages.

#52

I should add, to the above, the reason I’m trying to narrow this down is that with version: latest, platform_version: 4.2.0 and ESPSoftwareSerial:8.0.3 I’m suddenly getting hardware watchdog timeouts, whereas a board with just version: recommended and ESPSoftwareSerial:8.0.1 is still stable (if no UART sensors), but it’s currently impossible to prove anything as to what I’m changing!