Exclude data dir from backup during update process

Talks about metering
Post Reply
sovking
Posts: 80
Joined: Fri Feb 27, 2015 12:02 am

Exclude data dir from backup during update process

Post by sovking » Tue Oct 16, 2018 4:09 pm

This apply to 123solar too.
When data collected is enough, the backup process takes long time on small devices like raspberry, orangePI and so on.
Part of this time is due to the creation of tar.gz archive of 123solar or metern directory that are growing over the time with the collected data.
In one installation I have almost 10 years with 8 inverters... so.. it can be a pain waiting this backup hoping to avoid gateway timeout. :)

To shorten this time, an idea is to exclude the data directory from this backup: I usually make this backup in other ways (rsync and others...).
I think the update is only for code, data backup is out of scope of such script.

What do you think Jeanmarc ?

jeanmarc
Posts: 1600
Joined: Thu Aug 29, 2013 7:16 am

Re: Exclude data dir from backup during update process

Post by jeanmarc » Tue Oct 16, 2018 5:25 pm

Hi,
You mean the backup than it's done during the update ?

sovking
Posts: 80
Joined: Fri Feb 27, 2015 12:02 am

Re: Exclude data dir from backup during update process

Post by sovking » Tue Oct 16, 2018 10:14 pm

Yes, I've checked: every backup done during update include data directory.

jeanmarc
Posts: 1600
Joined: Thu Aug 29, 2013 7:16 am

Re: Exclude data dir from backup during update process

Post by jeanmarc » Wed Oct 17, 2018 6:42 am

Well, I think it is wise to backup all data. My mN repertory is only 8MB but i only keep 365 detailled days.
What's takes time is the tar.gz process, maybe i should add an option to skip that ? That say the backup will be bigger... :roll:

jeanmarc
Posts: 1600
Joined: Thu Aug 29, 2013 7:16 am

Re: Exclude data dir from backup during update process

Post by jeanmarc » Wed Oct 17, 2018 1:38 pm

In latest updater i put a bigger time limit. I think it should override the default one in php.ini.
ini_set(max_execution_time, 600);

sovking
Posts: 80
Joined: Fri Feb 27, 2015 12:02 am

Re: Exclude data dir from backup during update process

Post by sovking » Wed Oct 17, 2018 1:47 pm

Yes is wise to backup data, but that would be done by backup process, not by update process.
If you concern about doing backup because something wrong would happen during backup process, maybe data directory could be lost... in that case backup data should be safer, but in that case, I would prefer do not compress data or better have the choice in general setting panel, if I want to backup data during update, and in that case if I want compress it. That's because, these machine running 123solar are so CPU limited, and often connected remotely so timing factor could be more important than space on SD card.

I've done the following test on my bigger installation (data directory is 88MB:

Code: Select all

root@lime:/home/www# time cp -Rp  123solar /home/testdata3

real    0m2.426s
user    0m0.249s
sys     0m1.995s
while:

Code: Select all

time tar cfz /home/test.tgz 123solar        

real    0m47.752s
user    0m46.954s
sys     0m1.852s
So there tar with default gzip compression on, takes 45 seconds more: it' s quite a lot :)

jeanmarc
Posts: 1600
Joined: Thu Aug 29, 2013 7:16 am

Re: Exclude data dir from backup during update process

Post by jeanmarc » Wed Oct 17, 2018 3:39 pm

Yes, i am afraid to loose data but not mine, i do daily backup ;)
Maybe i could implement an option to skip compressing, it's easy to do so.

Could you do one more test ? Change the version in mN scripts/version.php, then do a new update (it have a 10min time limit). Does it succeed ?

jeanmarc
Posts: 1600
Joined: Thu Aug 29, 2013 7:16 am

Re: Exclude data dir from backup during update process

Post by jeanmarc » Fri Oct 19, 2018 3:40 pm

i've done some testing, the archiving will be done in background in next release.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest