HA Database corrupting on almost daily basis

Post Reply
Ched
Posts: 64
Joined: Fri Feb 03, 2023 11:02 pm

I am running HA on an old RPi2 and it was working fine at first but now it seems to corrupt the database on an almost daily basis.
Any ideas?
Could it be an iffy micro sd card or would I be better off running HA on an old laptop with Ubuntu on?

Thanks for any suggestions.
calum
Posts: 389
Joined: Fri Feb 24, 2023 11:00 am
Location: Stockport

HA is very hard on its storage, I'm guessing your SD card is on its way out.

I'd look for something else to run HA on, it's outgrown the RPi platform in my opinion.
H1-3.7 / 6xHV2600 / 14x400W / RS485 Modbus->HA
FoxESS Modbus HA Integration
Contact Fox here
Ched
Posts: 64
Joined: Fri Feb 03, 2023 11:02 pm

Cheers, I will have to look at the micro sd as I can't remember what one I used.
I just remembered its a RPi3 but still a bit weak but it uses such low power. Only other things I have kicking about are old laptops. Might give one of those a go.
Thanks.
calum
Posts: 389
Joined: Fri Feb 24, 2023 11:00 am
Location: Stockport

Just make sure you're not running it off a laptop hard drive! :o
H1-3.7 / 6xHV2600 / 14x400W / RS485 Modbus->HA
FoxESS Modbus HA Integration
Contact Fox here
Ched
Posts: 64
Joined: Fri Feb 03, 2023 11:02 pm

calum wrote: Sun May 14, 2023 11:04 am Just make sure you're not running it off a laptop hard drive! :o
Cheers. I was thinking of using the whole laptop which has a small SSD drive in. Think I will try an alternative microsd card first as a laptop will consume lots more power than a rpi3.
Cheers
calum
Posts: 389
Joined: Fri Feb 24, 2023 11:00 am
Location: Stockport

There's lots of stuff in forums about how to tune HA to be nicer to the SD card, mainly revolving around how much gets written to the database and how often. Worth a look if you want to persist with the platform.
H1-3.7 / 6xHV2600 / 14x400W / RS485 Modbus->HA
FoxESS Modbus HA Integration
Contact Fox here
Ched
Posts: 64
Joined: Fri Feb 03, 2023 11:02 pm

Cheers.
I've ordered a new microsd card - a Samsung PRO Endurance which is supposed to support cctv type recording so HA shouldn't be too bad with it. If that doesn't help I will look at cutting down the write frequency but it did seem a bit complex in that I have to decide what sensors to actually log so that might be a challenge.
There does also seem to be a fair bit of info on using ssd drives. So looks like I have a few options.
Thanks.
User avatar
APfox
Posts: 20
Joined: Tue Mar 28, 2023 9:40 am

calum wrote: Sat May 13, 2023 9:17 pm it's outgrown the RPi platform in my opinion.
Why exactly?
And what do you propose as an alternative then?

At the moment HA runs as a docker container on a rpi4 8GB with an 128GB Samsung 850 Pro. (Ubuntu 20.04 LTS)
I'm now on the look for something else as I want to run HAOS so i can make use of the Supervisor and the add-ons. I'm also thinking on moving from the default sqlite to MariaDB (my db exploded since using the foxess-modbus and foxess-em integrations)

I was thinking to use a RPi4 4GB with a new SSD for it , but now you say it's outgrown the RPi platform 🤔
H1-5.0-E1 inverter
ECS2900-H3
4400wp (SW) + 2000wp (NE) PV Array
HomeAssistant via WaveShare RS485 PoE to ETH (b)
no wi-fi (no stable connection)

goal: disconnect the FoxEss system from the internet completely and manage it from HA.
canton7
Posts: 42
Joined: Sun Oct 23, 2022 12:17 pm

> (my db exploded since using the foxess-modbus and foxess-em integrations)

There's a new experimental option for foxess_modbus under "CONFIGURE" to round and filter sensors slightly, which should help a lot with database size issues.
User avatar
APfox
Posts: 20
Joined: Tue Mar 28, 2023 9:40 am

canton7 wrote: Thu May 18, 2023 1:55 pm > (my db exploded since using the foxess-modbus and foxess-em integrations)

There's a new experimental option for foxess_modbus under "CONFIGURE" to round and filter sensors slightly, which should help a lot with database size issues.
yeah saw that one and installed it already.
I've also started an analysis on which data actually is important enough to keep and how to set update intervals depending on the needs
H1-5.0-E1 inverter
ECS2900-H3
4400wp (SW) + 2000wp (NE) PV Array
HomeAssistant via WaveShare RS485 PoE to ETH (b)
no wi-fi (no stable connection)

goal: disconnect the FoxEss system from the internet completely and manage it from HA.
calum
Posts: 389
Joined: Fri Feb 24, 2023 11:00 am
Location: Stockport

APfox wrote: Thu May 18, 2023 11:33 am
calum wrote: Sat May 13, 2023 9:17 pm it's outgrown the RPi platform in my opinion.
Why exactly?
And what do you propose as an alternative then?

At the moment HA runs as a docker container on a rpi4 8GB with an 128GB Samsung 850 Pro. (Ubuntu 20.04 LTS)
I'm now on the look for something else as I want to run HAOS so i can make use of the Supervisor and the add-ons. I'm also thinking on moving from the default sqlite to MariaDB (my db exploded since using the foxess-modbus and foxess-em integrations)

I was thinking to use a RPi4 4GB with a new SSD for it , but now you say it's outgrown the RPi platform 🤔
My feeling is that for something that is monitoring a key part of my house's infrastructure (namely, power), I want to be running something that is more performant, reliable and scaleable. Especially when you take into account things like long term analysis, logging to InfluxDB, running Grafana and other plugins - if you are using HA's capabilities, you end up having to deal with weaknesses of the platform like storage performance, which is really key when you get into substantial databases. They also don't work well with the small SSDs and SD cards that tend to be used with RPis.

So sure, it can be made to work, but constantly having to be wary of how hard I'm beating on the storage, and not being able to use what I see as the full capabilities of the device in a performant way, is my reason for saying that. As I said at the time, it's a personal opinion, not claiming it to be a universal truth :)

Ex-business uSFF PCs with low power x86 CPUs are a better bet IMO, although I acknowledge they come at a cost of somewhat higher power usage vs a RPi.
H1-3.7 / 6xHV2600 / 14x400W / RS485 Modbus->HA
FoxESS Modbus HA Integration
Contact Fox here
Ched
Posts: 64
Joined: Fri Feb 03, 2023 11:02 pm

Just to update:
The new Samsung MicroSD card has been running for a week and it's been fine. Fingers crossed it will stay OK for a few months at least.
Cheers for your help.
Post Reply