/var/log/hspc out of control

[dlenn845]

Is it normal to see this number of files per day created in /var/log/hspc?
-bash-3.00# ls *05-12* | wc -l
1971
-bash-3.00# ls *05-11* | wc -l
4079
-bash-3.00# ls *05-10* | wc -l
3981
-bash-3.00# ls *05-09* | wc -l
4178
-bash-3.00# ls *05-08* | wc -l
4210
-bash-3.00# ls *05-07* | wc -l
4379
-bash-3.00# ls *05-06* | wc -l
4205
-bash-3.00# date
Tue May 12 12:22:39 EDT 2009
-bash-3.00# du -h
4.0K ./profiler
4.3G .
-bash-3.00# pwd
/var/log/hspc

[Anatoliy Pylypchuk]

Hello,

/var/log/hspc directory contains all PBAS log files. If you have set the LogLevel to DEBUG in the Top > Configuration Director > Logging and Errors > Log Settings and also enable all options there, then such size of log directory is normal.

Regarding the files amount:

PBAS generate the problem reports for each failed action and put it to the separate file. So, in a case you have a lot of such actions, you will have a log of files in this directory.

[dlenn845]

I guess I need a trouble ticket opened for this issue. I guess you are telling me that it is NOT normal.

[AlexandrT]

If you take a look at "Action log" in your PBAS interface I suppose you will see a lot of actions with error status. Problem reports links should be attached there. If you fix the problem described in error reports then logs wont be grow up so catastrophically.

[dlenn845]

There are 8 total events that have a status of failed in June. and here is the number of files in that log directory for the same time period... can you please explain this to me. Also within those 8 errors are failed invoice processing due to declines in payment methods, and errors renewing domains for the same reason.

-bash-3.00# ls *2009-06-* | wc -l
39018

[AlexandrT]

Seems I was unclear, I meant "Top > Configuration Director > Logging and Errors > Action Log > Tasks Queue tab"
Filter by status "Error"

[philippe henneau]

was about to open a ticket til I found this: post: I get warning every week: disk space exceeds 80% of disk space, I keep increasing the VPS disk space every day, this can only be a temporary solutiion, the space just explodes. We are small: 100 customers, 500 domains, look at the space, it is more than what all my customers use:

[root@admin hspc]# du /var/log -h
8.1M /var/log/httpd
4.0K /var/log/hspc/profiler
39G /var/log/hspc
8.0K /var/log/mail
40M /var/log/hspcd
4.0K /var/log/samba
40G /var/log


Any advis?

Flupke

[AlexandrT]

Did you try to apply these Parallels KBs: http://kb.parallels.com/en/2178 and http://kb.parallels.com/en/2177 ?

[philippe henneau]

yes, nothing changed, /var/log/hspc uses 55Gb!
Tons of files there, see attachment below for example.

Can I just delete them?

---------- atachment -----------------

-rw-r--r-- 1 apache apache 41649 Mar 15 18:47 p2009-03-15_18-47-08.gz
-rw-r--r-- 1 apache apache 62628 Mar 15 18:47 p2009-03-15_18-47-12.1.gz
-rw-r--r-- 1 apache apache 62342 Mar 15 18:47 p2009-03-15_18-47-12.gz
-rw-r--r-- 1 apache apache 83357 Mar 15 18:47 p2009-03-15_18-47-16.1.gz
-rw-r--r-- 1 apache apache 83080 Mar 15 18:47 p2009-03-15_18-47-16.gz
-rw-r--r-- 1 apache apache 102261 Mar 15 18:47 p2009-03-15_18-47-19.1.gz
-rw-r--r-- 1 apache apache 102050 Mar 15 18:47 p2009-03-15_18-47-19.gz
-rw-r--r-- 1 apache apache 122636 Mar 15 18:47 p2009-03-15_18-47-23.1.gz
-rw-r--r-- 1 apache apache 122364 Mar 15 18:47 p2009-03-15_18-47-23.gz
-rw-r--r-- 1 apache apache 142992 Mar 15 18:47 p2009-03-15_18-47-27.1.gz
-rw-r--r-- 1 apache apache 142715 Mar 15 18:47 p2009-03-15_18-47-27.gz
-rw-r--r-- 1 apache apache 163268 Mar 15 18:47 p2009-03-15_18-47-32.1.gz
-rw-r--r-- 1 apache apache 163009 Mar 15 18:47 p2009-03-15_18-47-32.gz
-rw-r--r-- 1 apache apache 183775 Mar 15 18:47 p2009-03-15_18-47-36.1.gz
-rw-r--r-- 1 apache apache 183526 Mar 15 18:47 p2009-03-15_18-47-36.gz
-rw-r--r-- 1 apache apache 203005 Mar 15 18:47 p2009-03-15_18-47-39.1.gz
-rw-r--r-- 1 apache apache 202791 Mar 15 18:47 p2009-03-15_18-47-39.gz
-rw-r--r-- 1 apache apache 221659 Mar 15 18:47 p2009-03-15_18-47-41.gz
-rw-r--r-- 1 apache apache 234811 Mar 15 18:47 p2009-03-15_18-47-42.1.gz
-rw-r--r-- 1 apache apache 221878 Mar 15 18:47 p2009-03-15_18-47-42.gz
-rw-r--r-- 1 apache apache 21444 Mar 15 18:57 p2009-03-15_18-57-05.1.gz
-rw-r--r-- 1 apache apache 21220 Mar 15 18:57 p2009-03-15_18-57-05.gz
-rw-r--r-- 1 apache apache 42344 Mar 15 18:57 p2009-03-15_18-57-10.1.gz
-rw-r--r-- 1 apache apache 42056 Mar 15 18:57 p2009-03-15_18-57-10.gz

[AlexandrT]

Yes, you may just delete them with no doubts.
However I would advice you to find the reason of "problem reports" generating first. It is not normal work of the system!
What does your "action log" menu say? What is the most frequent error there?

[flupke]

this is from the first months after the installation of PBA, everything has been fixed,no new error in the log for the time being.

[ОлеÑ�Ñ�Ð]

Here are needed settings to be checked in this case

1) Check that log rotation for main PBAS logs is set correctly on the server:

/etc/logrotate.d/hspc file (as example):

## HSPc iteself logs rotating
/var/log/hspc/hspc.log /var/log/taskman.log {
missingok
copytruncate
compress
rotate 5
size=16M
}

2) Log level should be decreased from DEBUG, at least following options could be disabled at Top > Configuration Director > Logging and Errors > Log Settings:

Enable SQL Log in Problem Reports
Enable Debug Log in Problem Reports
Enable Extended Problem Reports

In this case hspc.log itself and problem report will be detailed maximally which is useful for any troubleshooting but log files are growing very fast and have rather big size.

3)To remove problem report files following commands can be useful:

[root@svr11 ~]# find /var/log/hspc/rep*.gz -mtime +30 -exec rm {} \;

------------------------------------

[root@svr11 ~]# find /var/log/hspc/p*.gz -mtime +30 -exec rm {} \;

or

[root@svr11 ~]# rm -f /var/log/hspc/p2012-03-0*.gz

rm -f /var/log/hspc/p2012-03-1*.gz
rm -f /var/log/hspc/p2012-03-2*.gz
rm -f /var/log/hspc/p2012-03-3*.gz
rm -f /var/log/hspc/p2012-02-0*.gz
rm -f /var/log/hspc/p2012-02-1*.gz
rm -f /var/log/hspc/p2012-02-2*.gz
...

These are commands for removing March and February logs, the same should be done for January, December, November, October p*.gz files - you just need to replace month and year values.

4) There is a special task that could be used to remove extended reports, "Remove old report files" , it is a pending one, executed once a day, but could be started manually at Top > Configuration Director > Logging and Errors > Action Log > Task queue > Remove old report files > Run now.

5) Another way to remove old data could be found at Top > Configuration Director > Miscellaneous Settings > Data Retention > Tasks > check all needed options and press "Start the process".

6) Retention settings could be set to 1 month at Top > Configuration Director > Miscellaneous Settings >Data Retention > Settings in PBAS panel.