T10-G3 Inverter Status during non-solar hours

[talal.hassan01]

Hi All,

I have been experiencing a weird phenomenon since 29th January 2025. After the recent HACS update and HA Core updates, I started to see Load Power values populating even when the sun was set. I was unsure and double checked the "FoxESS Inverter" sensor value and was showing "Online" at 8PM.

The issue automatically got fixed yesterday, however, today the same thing happened again and I can see the Inverter status as On-line and the Last Cloud Sync value showing updated 1 min ago.

Looking forward to experts guidance around this issue.

ss.png (4.16 KiB) Viewed 48472 times

Thanks.
Talal.

Inverter: FoxESS T-10 G3 (Three phase, Ongrid Inverter)
Panels: Jinko N Type - 555w x 10.

[Dave Foster]

That's a weird one - can you post a graph of your load_power sensor over a few days please.

I heard a report yesterday that a user had seen his api_calls get reset - so it is possible that Fox may be doing something with the openapi - but i'm not seeing anything wrong with my test system and i'm running the latest core.

Do you have any error messages in your log that relate to 'foxess'

[talal.hassan01]

Hey Dave,

Thank you for your response. Here are some stats you requested for. I also saw a holding pattern for Grid Consumption and Load Power sensors where they would continue to report the same values as were being populated as the sun was setting, for the rest of the night, until the next morning.

Ohh and no errors reported against FoxESS in the logs.

[Dave Foster]

Ok, thanks - so now just trying to understand if it is the Fox cloud that is playing up - if you log in to the V2 app and look in ‘Device’ what status do you see reported for the inverter and the datalogger ?.

Are you running the latest version of the integration v0.43

[talal.hassan01]

Apologies for not mentioning that the integration I am running is HACS -> Fox Cloud.

App and Fox Cloud website, both showing Inverter and the Datalogger as Offline.

Fox integration with HA also running latest, i.e., 0.43

[talal.hassan01]

Just found this error while I was going through the Logs again, somehow missed it.

[Dave Foster]

Hmm ok there are a lot of errors in getting data from the cloud - is there anything special about your home assistant, is it on a slow wifi or does your broadband have a slow connection for example?

I think if this continues you’ll need to enable full debugging by adding the code below to your configuration.yaml and open an issue on github on the integration page to post the logs (there will be a lot of data) - I am one of the developers for the cloud app so when you do that it’ll be me that answers.

Code: Select all

logger:
  default: warning
  logs:
    custom_components.foxess: debug

[talal.hassan01]

So my integration involves a laptop that was lying around in the house and is above the minimum specifications required to run Home Assistant on a Windows based machine.

For the internet part, the same laptop is on ethernet connection with a thought to keep connectivity on point always.

And then the datalogger also connects to the same network via router wirelessly.

So I would say speedtest and other tools show a decent bandwidth when I test on the same laptop. I do have this advanced "logger" code in my configuration.yaml.

However, I am happy to dive into the details of what could potentially be the root cause of this issue and looking forward to your step by step debug as well

[Dave Foster]

When you have a transition between day and night and the inverter has shut down, if home assistant is reporting the inverter on-line, can you please copy and past the foxess logs for the time period just before and just after sunset.

Looking at your previous image, it seems the integration is losing it's internet connection to the cloud at or around sunset.

At the same time - see if you can access the internet from the laptop (i.e. ping google 8.8.8.8)

[talal.hassan01]

Hey Dave,

So the inverter status today at sunset changed to Offline. I am sharing some actions I performed after you inquired about potential internet / bandwidth / connectivity issues:

1. Switched the laptop from static IP to dynamic. HA however is still on a static IP address.
2. Removed DuckDNS addon as I am behind a double NAT and, until I purchase a static public IP address from the ISP, I believe I will not be able to access HA outside my home network and if I decide not to use HA Cloud.
3. Was running Microsoft Edge on the laptop, running HA. Closed that too, just to lessen out any load contributing to performance issues.

Here is an extract from the logs as well (removed confidential details):

2025-02-05 18:23:09.069 DEBUG (MainThread) [custom_components.foxess.sensor] {'report': {'deviceType': 'T10-G3', 'masterVersion': '1.49', 'afciVersion': '0.00', 'hasPV': True, 'slaveVersion': '1.02', 'hasBattery': False, 'function': {'scheduler': False}, 'hardwareVersion': '--', 'managerVersion': '1.24', 'stationName': 'Talal-Hassan', 'productType': 'T3', 'status': 1, 'plantName': 'Talal-Hassan'}, 'online': True}

2025-02-05 18:24:11.856 DEBUG (MainThread) [custom_components.foxess.sensor] Inverter off-line set online flag false for SN:
2025-02-05 18:24:11.856 WARNING (MainThread) [custom_components.foxess.sensor] FoxESS has Cloud timeout or the Inverter is off-line, connection will be retried in 1 minute

2025-02-05 18:24:11.859 DEBUG (MainThread) [custom_components.foxess.sensor] {'report': {'deviceType': 'T10-G3', 'masterVersion': '1.49', 'afciVersion': '0.00', 'hasPV': True, 'slaveVersion': '1.02', 'hasBattery': False, 'function': {'scheduler': False}, 'hardwareVersion': '--', 'managerVersion': '1.24', 'stationName': 'Talal-Hassan', 'productType': 'T3', 'status': 3, 'plantName': 'Talal-Hassan'}, 'online': False}

I am still not convinced that the actions I performed above may have resulted in resolution of my issue, so I will keep on monitoring for the next couple of days and keep this thread posted with updates.

I appreciate your guidance and support in helping me understand / resolve the issue.

Thanks.

[Dave Foster]

The logs show the succesful transition from on-line to off-line so that’s working today - there is a fox cloud timeout which only happens where an api call doesn’t get a response within 75 seconds which I wouldn’t expect - i’m not sure it’s a performance issue, timeouts are more indicative of a loss of network connection?

[talal.hassan01]

Yes, you have a point here which I would further like to play around and test with. From the router that was provided by the ISP, there are wired connections leading to the laptop HA is running on, however, I also have a TP Link Mesh Router network of 3 nodes and that is also connected by the same source, providing WIFI to MQTT, personal handheld devices and laptops etc.

The data logger is also connected wirelessly to the TP Link Router. I believe if I connect it directly with the ISP router (via WIFI) or if there exists a LAN based data logger for my inverter series, I can strike out the connectivity / bandwidth root cause, by potentially reducing 1 hop and being connected directly to the source but on WiFi.

I will keep you posted on tomorrow's progress as well.

Thanks!

[talal.hassan01]

Hey Dave,

Unfortunately, the inverter was again stuck on On-line status today. It continues to be Online as we speak. Here are the stats from yesterday to today:

I also wanted to inquire if I can somehow share the log file with you in private, if possible, to debug this issue.

Much appreciated as always.

Talal.

[Dave Foster]

Sure if you email me on b5djf@yahoo.com I will work through the logs and see what it is doing - please leave your apikey and SN details visible as to eliminate your install I might try running a test system and see if I get similar results.

[talal.hassan01]

Awesome, thanks alot. Let me share the logs!

[Dave Foster]

Awesome, thanks alot. Let me share the logs!

Got them thanks, leave it with me and i’ll trawl through the logs tomorrow

[talal.hassan01]

Hey Dave,

Sorry for chiming in again. I just noticed that my system was commissioned sometime late 2023. And while going through the FoxCloud x Home Assistant documentation, I noticed that I should be using device ID instead of device SN.

In this case:

Q1. Should I change Device ID in configuration.yaml to the actual ID from the foxesscloud.com URL instead of the Serial Number?

Q2. What will happen to the Solar readings? If I change my sum up variables to the ones pointing at ID instead of serial number as of tomorrow for e.g., will it let me maintain history and start functioning as expected ?

Sorry for creating a bit of a confusion but I just realized this.

TIA

[Dave Foster]

No don’t change anything, leave it using the SN - that is the correct format - it’s not when you installed the system it is when you first installed the integration (which used to use DeviceID) and it used that as the entity unique_id so changing it will break all your history.

On your logs here is what i’ve found - a normal solar inverter shuts down at sunset and restarts at dawn and the datalogger goes off-line when it does that.

However for whatever reason your datalogger is not shutting down and sometimes stays powered up, sending data samples which keeps the site status as on-line, because the inverter isn’t actually sending new data samples they stay the same as the last known good sample and repeat like that all night. - the only good news that I can see is that ‘running state’ is changing from 163 ‘on-grid’ to ‘161’ waiting.

I’ve never seen that behaviour before - is there anything special about your system - old inverter firmware, or very latest firmware - is the datalogger firmware up to date ?.

[talal.hassan01]

Oh okay acknowledged. Thanks for the confirmation.

Yes you are spot on. Whilst going through the documentation yesterday, I also saw that the state was changing, but then the datalogger kept receiving data with an Auxiliary tag (Auxilliary TimeSlice).

And when you point out that the data logger acts like it is still receiving data, if I log on to FoxCloud website or App, I see the status across all as Offline as well as the Inverter.

For the firmware part, I believe everything I have may be running the latest. Here are the stats:

datalogger.JPG (9.18 KiB) Viewed 41906 times

Thanks.

[Dave Foster]

Leave it with me, i’ll need to do some testing over the weekend and see if I can intercept the running state as being in the checking status and use that to switch the inverter state flag off-line (I guess the Fox cloud must be doing something similar?)

[talal.hassan01]

Sure thing, will await your expert advice after the testing

[Dave Foster]

Typical... yesterday the inverter shutdown gracefully and went off-line normally

I'm running some test software here using your api details, and just waiting for the inverter to stay on-line and switch the running state to 161 (off-grid)

[talal.hassan01]

Haha yeah, just as we investigate, it goes back to normal behavior. Seeing the inverter go offline for 2 days in a row now.

[Dave Foster]

Ok good news of a kind, it did it tonight - in fact it's still showing as online, whereas the runningstate has gone to off-grid.

Now when that happens, the runningstate (on a solar only inverter) will trigger an off-line event that overrides what is being reported in the site details.

The logs -

2025-02-09 13:20:33.316 DEBUG (MainThread) [custom_components.foxess.sensor] Var: generation, SN: 60xxxxx set to 2909.3
2025-02-09 13:20:33.316 DEBUG (MainThread) [custom_components.foxess.sensor] Var: runningState, SN: 60xxxxx set to 161
2025-02-09 13:20:33.316 DEBUG (MainThread) [custom_components.foxess.sensor] TestState: 161, hasBat: False online: True
2025-02-09 13:20:33.316 DEBUG (MainThread) [custom_components.foxess.sensor] Waiting so set off-line state, TestState: 161, hasBat: False online: False
2025-02-09 13:20:33.316 DEBUG (MainThread) [custom_components.foxess.sensor] Var: currentFault, SN: 60xxxxx set to
2025-02-09 13:20:33.316 DEBUG (MainThread) [custom_components.foxess.sensor] Var: currentFaultCount, SN: 60xxxxx set to 0
2025-02-09 13:20:33.316 WARNING (MainThread) [custom_components.foxess.sensor] Fox1 Inverter is off-line, waiting, will retry in 5 minutes


I'll keep testing for another day or two and then i'll do a pre-release so you can download a beta version and test it on your system

[talal.hassan01]

This is good news indeed. I was also about to share that the inverter didn't go off-line today.

Really appreciate your efforts, looking forward to it!