[Fleet] Add UI toggle for synthetic `_source` to data streams

[kpollich]

Summary

Add a UI toggle that enabled synthetic _source for a given data stream to the policy editor.

Parent issue #132818
Ref #139862

Implementation

• Add an Indexing Options (Experimental) section to the policy editor at the end of each data stream's settings section
  • Include a switch element for Synthetic source
  • Ensure the Synthetic source switch also includes a tooltip with additional information/references
  • When the switch is enabled, update the component template for the given data stream to enable synthetic source
  • When the switch is disabled, update the component template for the given data stream to disable synthetic source
• Update the create/edit package policy API endpoints to accept data stream synthetic source opt-in status
  • Ensure open API spec is updated to align with this change
• Update policy upgrade API to honor existing experimental feature opt-in values for corresponding data streams

Proposed Request Addition to Package Policy API

Show current example payload

{
  "name": "nginx-51",
  "description": "",
  "namespace": "default",
  "policy_id": "a1c33c30-2de9-11ed-b772-1b55494b0e06",
  "enabled": true,
  "inputs": [
    {
      "type": "logfile",
      "policy_template": "nginx",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "logs",
            "dataset": "nginx.access"
          },
          "vars": {
            "paths": {
              "value": [
                "/var/log/nginx/access.log*"
              ],
              "type": "text"
            },
            "tags": {
              "value": [
                "nginx-access"
              ],
              "type": "text"
            },
            "preserve_original_event": {
              "value": false,
              "type": "bool"
            },
            "processors": {
              "type": "yaml"
            }
          }
        },
        {
          "enabled": true,
          "data_stream": {
            "type": "logs",
            "dataset": "nginx.error"
          },
          "vars": {
            "paths": {
              "value": [
                "/var/log/nginx/error.log*"
              ],
              "type": "text"
            },
            "tags": {
              "value": [
                "nginx-error"
              ],
              "type": "text"
            },
            "preserve_original_event": {
              "value": false,
              "type": "bool"
            },
            "processors": {
              "type": "yaml"
            }
          }
        }
      ]
    },
    {
      "type": "httpjson",
      "policy_template": "nginx",
      "enabled": false,
      "streams": [
        {
          "enabled": false,
          "data_stream": {
            "type": "logs",
            "dataset": "nginx.access"
          },
          "vars": {
            "interval": {
              "value": "10s",
              "type": "text"
            },
            "search": {
              "value": "search sourcetype=nginx:plus:access",
              "type": "text"
            },
            "tags": {
              "value": [
                "forwarded",
                "nginx-access"
              ],
              "type": "text"
            },
            "preserve_original_event": {
              "value": false,
              "type": "bool"
            },
            "processors": {
              "type": "yaml"
            }
          }
        },
        {
          "enabled": false,
          "data_stream": {
            "type": "logs",
            "dataset": "nginx.error"
          },
          "vars": {
            "interval": {
              "value": "10s",
              "type": "text"
            },
            "search": {
              "value": "search sourcetype=nginx:plus:error",
              "type": "text"
            },
            "tags": {
              "value": [
                "forwarded",
                "nginx-error"
              ],
              "type": "text"
            },
            "preserve_original_event": {
              "value": false,
              "type": "bool"
            },
            "processors": {
              "type": "yaml"
            }
          }
        }
      ],
      "vars": {
        "url": {
          "value": "https://server.example.com:8089",
          "type": "text"
        },
        "username": {
          "type": "text"
        },
        "password": {
          "type": "password"
        },
        "token": {
          "type": "password"
        },
        "ssl": {
          "value": "#certificate_authorities:\n#  - |\n#    -----BEGIN CERTIFICATE-----\n#    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF\n#    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2\n#    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB\n#    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n\n#    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl\n#    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t\n#    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP\n#    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41\n#    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O\n#    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux\n#    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D\n#    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw\n#    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA\n#    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu\n#    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0\n#    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk\n#    sxSmbIUfc2SGJGCJD4I=\n#    -----END CERTIFICATE-----\n",
          "type": "yaml"
        }
      }
    },
    {
      "type": "nginx/metrics",
      "policy_template": "nginx",
      "enabled": true,
      "streams": [
        {
          "enabled": true,
          "data_stream": {
            "type": "metrics",
            "dataset": "nginx.stubstatus"
          },
          "vars": {
            "period": {
              "value": "10s",
              "type": "text"
            },
            "server_status_path": {
              "value": "/nginx_status",
              "type": "text"
            }
          }
        }
      ],
      "vars": {
        "hosts": {
          "value": [
            "http://127.0.0.1:80"
          ],
          "type": "text"
        }
      }
    }
  ],
  "package": {
    "name": "nginx",
    "title": "Nginx",
    "version": "1.4.1"
  }
}

"inputs": [
  {
    "type": "logfile",
    "policy_template": "nginx",
    "enabled": true,
    "streams": [
      {
        "enabled": true,
        "data_stream": {
          "type": "logs",
          "dataset": "nginx.access"
        },
        "vars": {
          "paths": {
            "value": [
              "/var/log/nginx/access.log*"
            ],
            "type": "text"
          },
          "tags": {
            "value": [
              "nginx-access"
            ],
            "type": "text"
          },
          "preserve_original_event": {
            "value": false,
            "type": "bool"
          },
          "processors": {
            "type": "yaml"
          },
          "experimental_features": {
             "synthetic_source": true
           }
        }
      }

Enabling synthetic source

Reference https://github.com/elastic/integrations/blob/main/docs/how_to_test_new_indexing_features.md

We'll need to make an update to the appropriate component template equivalent to:

PUT /_component_template/<type>-<package>.<dataset>@package
{
  "template": {
    "settings": {},
    "mappings": {
      "_source": {
        "mode": "synthetic"
      }
    }
  }
}

Designs

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

Making some pretty solid progress on this but ran into another mapping incompatibility when trying to enable synthetic source for the logs-nginx.access data stream

Not sure how to resolve this one off the top of my head. Will have to investigate further.

[kpollich]

Another example of an error from the akamai integration

I wonder if we should only allow opt-in to synthetic source on metrics data streams for now. It seems like it'll take a decent chunk of time to track down every possible mapping conflict w/ synthetic source.

Alternatively we can just let these errors surface as they do currently. It's an experiment feature after all, and the errors are actually fairly successful in conveying that the given data stream does not support synthetic source.

@andresrc @joshdover curious for your thoughts here.

[andresrc]

@kpollich I agree we should add this initially just on metrics datasets. We don't expect this to be very useful initially for logs integrations as event.original is only in the _source. /cc @ruflin

[ruflin]

As this is an experimental feature, I think the errors above are fine and I would not put artificial limitation in places to just show it for metrics-*. Part of being able to enable the feature is finding all these limitations. My take is, lets just bubble it up.