Skip to content

[Doc]Add section for conceptual info#11715

Merged
karenzone merged 5 commits intoelastic:masterfrom
karenzone:processing-info
Jun 9, 2020
Merged

[Doc]Add section for conceptual info#11715
karenzone merged 5 commits intoelastic:masterfrom
karenzone:processing-info

Conversation

@karenzone
Copy link
Copy Markdown
Contributor

@karenzone karenzone commented Mar 24, 2020

Important info that customers need can be buried way down in the details. This PR adds a section for bubbling up and curating that information.

Replaces #11581

PREVIEW: http://logstash_11715.docs-preview.app.elstc.co/guide/en/logstash/master/processing.html

@karenzone karenzone changed the title Add section for conceptual info [Doc]Add section for conceptual info Mar 24, 2020
@karenzone
Copy link
Copy Markdown
Contributor Author

@colinsurprenant FYI. From discussions in #11175. I believe that we have a good structure for moving forward. I'm ready to sync about new content for event ordering guaranties with multiple worker versus single worker.

This one replaces #11581

@colinsurprenant
Copy link
Copy Markdown
Contributor

Great! We could start with something around the lines of:

  • historically, logstash events ordering between their creation by an input plugin and output would be preserved when a single worker was used to execute a pipeline.
  • Note that this behaviour was never officially documented but it was a feature that many users relied upon.
  • That ordering was never guaranteed when multiple workers were used across threads because the time to finish processing events could vary between workers and the fastest to completion would be output before the slower ones.
  • When the Java Execution engine was introduced, it broke the single worker ordering preservation. This was fixed with the introduction of the pipeline.ordered ...

@karenzone
Copy link
Copy Markdown
Contributor Author

@colinsurprenant, your comments on event ordering were very helpful! I used some of them in this PR, and all of them in the release notes. Thanks for taking time to add them.

We also discussed that new section might be a good place to talk about Java pipeline initialization time. If somebody noticed a slowdown and googled "Java pipeline initialization time", what kind of guidance can we offer them? (Carried over from discussion in #logstash).

Note: I'm leaving this PR in DRAFT status until we flesh out a few more details. Your comments are always welcomed and appreciated.

@karenzone
Copy link
Copy Markdown
Contributor Author

PENDING:

https://github.com/elastic/logstash/pull/11715/files#diff-fb62b697603cee2152850ac6aee1bec6R32

What should we advise people to do if they notice a slowdown?

@karenzone karenzone marked this pull request as ready for review April 14, 2020 19:16
@karenzone
Copy link
Copy Markdown
Contributor Author

@colinsurprenant Please let me know what you think.
PREVIEW: http://logstash_11715.docs-preview.app.elstc.co/guide/en/logstash/master/processing.html

@colinsurprenant
Copy link
Copy Markdown
Contributor

Thanks @karenzone! I left a few comments for what I thought was not clear to me, but I trust you to figure the best phrasing!

@karenzone
Copy link
Copy Markdown
Contributor Author

@elasticmachine, run elasticsearch-ci/docs

@karenzone
Copy link
Copy Markdown
Contributor Author

@colinsurprenant Will you please take a look to see if I captured your comments correctly? Thanks!


When maintaining event order is important, use a single worker.
This approach ensures that batches are computed one-after-the-other, and
that events maintain their order within the batch.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this might be slightly misleading to think that "When maintaining event order is important, use a single worker" and using a single worker is all you need.

Maybe something like:

"When maintaining event order is important, use a single worker and set 'pipeline.ordered => true'.

I think this will allow the this doc to live longer if we ever change the "auto" setting behaviour... WDYT?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice clarification! Thank you for suggesting it.

@colinsurprenant
Copy link
Copy Markdown
Contributor

Thanks @karenzone, looking good, left a minor comment!

@karenzone karenzone requested a review from colinsurprenant June 9, 2020 13:44
Copy link
Copy Markdown
Contributor

@colinsurprenant colinsurprenant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@karenzone karenzone merged commit 34ee17b into elastic:master Jun 9, 2020
@karenzone karenzone deleted the processing-info branch June 9, 2020 21:16
@elasticsearch-bot
Copy link
Copy Markdown

Karen Metts backported this into the following branches!

Branch Commits
7.x 9aec969, 9597ab1, 335032d, 610abaa, 9bb792a
7.8 ee90e80, 77ad368, 7478c1c, 6c110a8, db7262e
7.7 ba2e6ec, 5e3310f, 5f8d318, d14143d, b7ddc30

elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Backport of #11715 to 7.8
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Backport of #11715 to 7.8
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Backport of #11715 to 7.7
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Backport of #11715 to 7.7
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Backport of #11715 to 7.x
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Backport of #11715 to 7.x
elasticsearch-bot pushed a commit that referenced this pull request Jun 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants