[Meta] Upgrading from 7.x to 8.0

[roaksoax]

This issue tracks all the items that need to be handled for the 8.0 release.

• Support Breaking Changes in Stack Components:
  • Elasticsearch finalizing Types Removal (upstream: Types removal in 8.0 elasticsearch#41059)
    • Elasticsearch Output no longer sends type when target ES >= 8.0 Only add _type if ES version < 8 logstash-plugins/logstash-output-elasticsearch#892
    • Elasticsearch Output no longer sends type when target ES >= 7.0 and user don't customize it with document_type Remove type when connecting to ES 7 and user doesn't specify it logstash-plugins/logstash-output-elasticsearch#994
    • X-Pack Internal Monitoring uses type, but is possibly slated for removal in 8.0
    • Investigate _type usage in elasticsearch output Specifying types in bulk requests is deprecated logstash-plugins/logstash-output-elasticsearch#915
  • [kaise] Elasticsearch migration to composable templates and removal of _template API (upstream: Composable Templates elasticsearch#53101)
    • Implement Composable Templates to support Elasticsearch 8.0 logstash-plugins/logstash-output-elasticsearch#944
  • Switch ES plugins to Logstash specific APIs (Following System Indices [Meta] Logstash use system indices API  #12291)
  • Drop _xpack API endpoint usage (and settings) Switch license checking from _xpack to _license #11382
  • Agent Migration
  • GeoIP transition to using service.
• Logstash Breaking Changes
  • Internal Collection - Deprecation in favor of agent (or drop sending data directly to ES cluster) ([Stack Monitoring] Remove internal collectors #11169)
  • [kaise] Execution Engine - Remove Ruby Execution Engine (remove ruby execution #11236)
  • ECS - Pipelines in ECS-Compatibility mode as the default (meta: [META] A Path to ECS-by-default in Logstash Plugins #11623)
  • DOC - Document breaking changes
  • [kaise] Settings - Remove "field-reference-parse" Remove "field-reference-parser" setting #12355: INPROGRESS
  • [andsel] Modules - Remove modules infrastructure and existing modules (Azure/Netflow)
  • Elasticsearch Output under default distribution requires licensed Elasticsearch (check if ES is licensed before pushing out events to it logstash-plugins/logstash-output-elasticsearch#875)
  • Operating System support changes in 8.0.0 Drop support for older Operating Systems for 8.0.0 #12511
  • Auditing for terminology: master/slave, *list, and others Auditing for terminology: master/slave, *list, and others #12052
  • [andsel] Use LS_JAVA_HOME instead of JAVA_HOME [Meta ] Use LS_JAVA_HOME in favor of JAVA_HOME #12725 (See Remove support for JAVA_HOME elasticsearch#69149 for references in ES)
  • Review/evaluate other breaking changes ( breaking change )
    and open issues targeted for 8.0.0
• Other (e.g. Stack Features Logstash should Support)
  • Elasticsearch Introduction of Hidden Indices (upstream: Implement hidden indices elasticsearch#50452)
  • System Indices - Clean-up old implementation ([Meta] Logstash use system indices API  #12291)
  • Runtime Fields
  • Logstash Breaking Changes
    • avoid Ruby Event wrapper - just rely on the Java org.logstash.Event in Ruby land
  • ? stack update JRuby to 9.3 (would resolve loading LS might (rarely) fail due JRuby auto-load bug #12077)
    • Java collection patches revisit LS' Java (collection) integration patches #12800
    • actively (CI) testing against jruby-head
  • New 'major' release of Ruby filter (Breaking: restrict unintended user constant usage logstash-plugins/logstash-filter-ruby#56)
  • Document upgrade instructions

[HenkPoley]

Is the 'migration to composable templates' scripting still planned? (for setups without clashing inheritance)

Or otherwise, some kind of guidance how to move parts around. Documentation on v2 and legacy templates is currently quite separate.

Is it literally: If you don't have inheritance, then ignore PUT _component_template and "composed_of": [..] ? Then it seems pretty straightforward.

---

E.g. for the no-inheritance case, with jq on a directory of legacy template json files:

jq '{index_patterns: [."index_patterns"], template: {settings: ."settings", mappings: { properties: ."mappings"."properties"}, aliases: ."aliases"}, priority: 49, composed_of: [], version: ."version"|0}' *

It doesn't transfer "_source": { "enabled": false },. Maybe somebody knows more jq-foo than I do.

Potentially add , _meta: { description: "TODO: customize"} before the last } if you want to have a description field metadata, etc.

[yaauie]

@HenkPoley while Logstash does persist the templates it is given and it will need to change how we compose our own templates internally, more likely your question would be better addressed by the Elasticsearch folks. I'd recommend starting a thread in the Elasticsearch forum.

[yaauie]

@roaksoax I do not believe that the "Hidden Indices" feature of Elasticsearch should be classified with "Breaking Changes". It is a new feature, by which a user can now explicitly configure an index to not be available to wildcard queries. Logstash does not use this feature when creating indices (yet?) and any wildcard queries that are performed in the Elasticsearch Input and Filter plugins will continue to work as-is unless a user explicitly configures an index to not be returned.

[HenkPoley]

Ah oops, I just followed a link from the huge Elasticsearch-project Composable Templates meta ticket to here, and didn't want to spew in there.

But now I see this is a different, but related, project.

[jsvd]

Adding a terminology audit task that will result in breaking changes #12052

[karenzone]

@roaksoax From the output-elasticsearch doc: "Using the transport protocol to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and will be removed in 8.0.0." Should that effort be on this list?

[yaauie]

@karenzone neither Logstash core not any of our plugins use the deprecated Elasticsearch Transport Protocol -- we use the HTTP APIs.

[karenzone]

In that case, I'll pull that line out. Thanks, @yaauie.