...
On one of a projects we've faced system log table grown to 370MB and 12490 records in it. After simplified duplicate removal (matching by "error message", "page url" and "error backtrace") this table had only 2480 records left. This can be even smaller number if we'll be processing url more intellectually (e.g. first remove irrelevant parameters from url, like "page number") and compare resulting urls.
I'm proposing to:
- add "LastOccurrenceOnLogLastRecurrenceOn" (date/time) and "OccurrenceCountLogRecurrenceCount" (int) columns
- from time to time (e.g. once a day) do system log record grouping (aggregation)
...