Translog size is larger than the default index.translog.flush_threshold_size of 512mb

[ialidzhikov]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): 6.5.4

Plugins installed: []

JVM version (java -version): jdk-11.0.1

OS version (uname -a if on a Unix-like system): Linux 4.14.88-coreos #1 SMP Mon Jan 7 23:48:02 -00 2019 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

GET logstash-2019.02.06/_settings
{
  "logstash-2019.02.06" : {
    "settings" : {
      "index" : {
        "refresh_interval" : "60s",
        "number_of_shards" : "3",
        "translog" : {
          "durability" : "async"
        },
        "provided_name" : "logstash-2019.02.06",
        "creation_date" : "1549411253895",
        "number_of_replicas" : "0",
        "uuid" : "ozXK8lszSbmQn_Qbn3oiGQ",
        "version" : {
          "created" : "6050499"
        }
      }
    }
  }
}

$ pwd
/data/nodes/0/indices/ozXK8lszSbmQn_Qbn3oiGQ
$ du -mh .
567M	./0/translog
1.4G	./0/index
8.0K	./0/_state
2.0G	./0
571M	./2/translog
1.4G	./2/index
8.0K	./2/_state
2.0G	./2
569M	./1/translog
1.4G	./1/index
8.0K	./1/_state
2.0G	./1
8.0K	./_state
5.8G	.

As you can see the translog files' sizes are more than the default 512mb. I expect the translog files to be flushed when their total size per shard reaches 521mb. In the snippet you can see than the total sizes are 567mb, 571mb and 569mb.

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

Provide logs (if relevant):

[jasontedor]

Flushing no longer means that the translog is truncated to be below 512 MB. Rather, flushing is now about reducing the amount of uncommitted data on disk, so that that data is now also permanently stored in Lucene. However, we continue to retain additional translog storage to assist with operation-based peer recoveries. You can see in the docs we have additional settings related to the retention size and age. We are in the process of updating our docs on flushing (#38503) to make this clearer.

[ialidzhikov]

@jasontedor , I cannot understand your explanation. The docs says that index.translog.retention.size is The total size of translog files to keep and Defaults to 512mb, in my case the translog files are going beyond the limit.

[ialidzhikov]

Also the translog file sizes are not being calculated as part of the total index size:

GET _cat/indices/logstash-2019.02.06?v
health status index               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   logstash-2019.02.06 ozXK8lszSbmQn_Qbn3oiGQ   3   0    8268485            0      4.2gb          4.2gb

The index size is 4.2G but on the fs it takes 5.8G because of the translogs.
This makes impossible to use the curator's filtertype_space. I want to rely that the index size_in_bytes will be approximately the same as the size on the fs.
The thing I want to achieve is to delete the oldest index/indices when 98% of the volume data is occupied.