Looks like I have been stable now for a bit as well.
Same boat here. Tried to post a similar topic about an hour ago. Power cycled, reconnected to wifi (all good, and 2.4ghz), and firmware is up to date, but still solid blue/purple in upper left corner.
I am seeing the same, also at about 05:40 PDT. There is no network issue locally, as I can ping the AG and retrieve data in Home Assistant.
There is also no issue locally with internet connectivity
Same here. Two devices at different locations went offline about three hours ago. Wifi and internet at both locations is fine.
Same, likely a server issue since the device itself is reporting readings on the console.
Hi all,
Thanks for the reports!
There was a technical incident on our data centre service provider’s side related to the load balancer. This is what caused monitors to appear offline or send data only intermittently to the AirGradient server, even though the devices themselves were still running and connected locally.
The issue has now been addressed, and monitors have recovered. We are continuing to monitor the situation to make sure the connection remains stable.
In most cases, no action should be needed and monitors should recover automatically.
5 Likes
whats the load like on your servers out of interest. no need to disclose secrets but is it a constant headache ?
How many reliable always on monitors report to the system daily ?
Hi @Pink_House, thanks for the question!
Let me find out with our development team, and I’ll let you know as soon as I get an answer. Thanks!
1 Like
Hi @Pink_House, I was informed that the server load was not that high. The limitation was related to the load balancer, which Hetzner didn’t let us monitor.
1 Like
Great, thank you for the information !
1 Like
My pleasure! We will consistently improve our infrastructure to reduce the chance of situations like this and minimize downtime as much as possible.
Please always feel free to let us know if you experience any issues.
My monitors keep dropping offline for the last two days. Is this issue known? They are online about half the time and have dropped about 40 times each the last two days. I do have a firewall rule, they are only allowed connection to the AirGradient server. Did you change your server configuration and should I whitelist more ip addresses?
Hi @Bart_Netherlands, thanks for reaching out to us. Please let me confirm this with our team, and I’ll give you an update. Could you let me know what IP addresses are currently whitelisted? Thanks!
91.98.9.143 and private networks: 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
@Bart_Netherlands Thanks for the info! Currently, we have two IP addresses for our hardware endpoints.
91.98.9.143
46.225.42.92
Could you please try adding to your whitelist and see if the issue is fixed?
Thanks! I’ve just added the IP and will check if this fixes the issue
Would it be possible to send an email notification when the server IP addresses change? I think a good location would be the users & permissions page. Here you can enable “Offline monitors notifications”. Maybe you can add a “Server IP change notification” option which is disabled by default?
1 Like
Adding 46.225.42.92 fixed it! My monitors haven’t dropped offline since I added the IP address
1 Like
@Bart_Netherlands I’m glad to hear that it works for you! Please let us know if you need any help.
Regarding your feature request, please let me add this to our list for discussion with my team.
To understand a little more context, would you mind describing your current setup and use case? (for example, you want to make sure about the IoT device security, etc.). This will help us a lot to understand our customers deeply.
Thanks!
Thanks for your quick replies.
My current setup for home is one outdoor monitor and two indoor monitors. I just think it is good practice to have firewall rules in place for IoT devices for improved security. I can imagine business customers (offices, schools etc.) definitely want to use firewall rules for good security.
1 Like