Compress audit logs

[pgomulka]

audit logs should be compressed when rolling over due to size based
triggering policy breaching 1GB.
Files are not being deleted.

closes #63843

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?
• If submitting code, have you built your formula locally prior to submission with gradle check?
• If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
• If submitting code, have you checked that your submission is for an OS and architecture that we support?
• If you are submitting this code for a class then read our policy for that.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Logging)

[elasticmachine]

Pinging @elastic/es-security (:Security/Audit)

[pgomulka]

@elasticmachine update branch

[tvernum]

I'll leave this review in @albertzaharovits's hands.

[albertzaharovits]

I think we need to explicitly set the nomax file index attribute.

appender.audit_rolling.strategy.type = DefaultRolloverStrategy
appender.audit_rolling.strategy.fileIndex = nomax

[pgomulka]

that's right, good find.

[albertzaharovits]

@pgomulka Sorry for the huge delay on this PR.

Because the log file name changes, some folks might consider it breaking,
though only past rolled-over names are changed which I think is key. This change
brings the audit logs more in line with the other logs which is also worth pointing out.
But folks might have automated tools set up for all the logs and we might
get into arguments. To be safe, I would drop a breaking change notice, on 7.x, on the lines:

diff --git a/docs/reference/migration/migrate_7_11.asciidoc b/docs/reference/migration/migrate_7_11.asciidoc
index 20ab1574350..76a9ccbb6f0 100644
--- a/docs/reference/migration/migrate_7_11.asciidoc
+++ b/docs/reference/migration/migrate_7_11.asciidoc
@@ -32,3 +32,26 @@ will be applied to the values of a keyword field; for example, a
 field configured with a lowercase normalizer will return highlighted
 snippets in lower case.
 ====
+
+[discrete]
+[[breaking_711_security_changes]]
+=== Security changes
+
+[[audit-logs-are-rolled-over-and-archived-by-size]]
+.Audit logs are rolled-over and archived by size.
+[%collapsible]
+====
+*Details* +
+In addition to the existing daily rollover, the security audit logs are
+now rolled-over by disk size limit as well. Moreover, the rolled-over logs
+are also gzip compressed.
+
+*Impact* +
+The names of rolled over audit logfiles (but not the name of the current log)
+have changed.
+If you've setup automated tools to consume these files, you must configure them
+to use the new names and to possibly account for gzip archives instead of plaintext.
+The Docker build of Elasticsearch is not affected since it logs on stdout where
+rollover is not performed.
+
+====

Similarly for docs/reference/migration/migrate_8_0.asciidoc on master.

Also for safety I would label the PR as >breaking .

Also I would work a bit on the description of the PR, on the same lines with the deprecation notice.

[albertzaharovits]

After writing the above I noticed you've removed the 7.11 label.
I'm curious for the motivation.

[pgomulka]

I removed the 7.11 label as the logs compression would be only enabled in 8.0 exactly for the reason you described #63843 (comment)
This is a breaking change and we should deprecate in 7.x

[pgomulka]

@elasticmachine update branch

[albertzaharovits]

Alright, then. It's cautious to merge to master only and we've been not compressing for a long time and few have complained about it.

Thank you for solving!

[albertzaharovits]

LGTM