[system log] System Log Aggregator
We've added System Log in
Consider following scenario:
- developer missed 1 notice message on a page, while he was developing it (or maybe that 1 notice only happens on production server)
- page has 1000 visits per day
Since each notice is logged and 1000 new records are added to system log every day. System log records about errors, warnings and notices have backtrace information included to ease up debugging and it adds extra to overall database table size.
On one of a projects we've faced system log table grown to 370MB and 12490 records in it. After simplified duplicate removal (matching by "error message", "page url" and "error backtrace") this table had only 2480 records left. This can be even smaller number if we'll be processing url more intellectually (e.g. first remove irrelevant parameters from url, like "page number") and compare resulting urls.
I'm proposing to:
- add "LogLastRecurrenceOn" (date/time) and "LogRecurrenceCount" (int) columns
- from time to time (e.g. once a day) do system log record grouping (aggregation)
This way we still have all the info that was in each error, but also save space (on disk) and searching time needed to find right error along many similar ones.