[Data] - Tabular form for Streaming Executor Op Metrics

[goutamvenkat-anyscale]

Description

Instead of rendering a large json blob for Operator metrics, render the log in a tabular form for better readability.

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request improves the readability of operator metrics in logs by rendering them in a tabular format instead of a JSON blob. The implementation is clean and includes a comprehensive test case. I've suggested a minor refactoring to the new formatting function to improve its clarity and efficiency.

[goutamvenkat-anyscale]

/gemini summary

[gemini-code-assist]

Summary of Changes

This pull request significantly improves the user experience when inspecting Streaming Executor Operator Metrics by transforming the previously unstructured JSON log output into a clear, categorized, and tabular format. This change aims to make debugging and monitoring more efficient by presenting critical performance data in an easily digestible manner, allowing users to quickly grasp the state and performance of operators.

Highlights

• Improved Metrics Logging: Operator metrics are now logged in a tabular format for enhanced readability, replacing the previous raw JSON blob output.
• New Utility Function: A new private utility function, _format_metrics_table, has been introduced to categorize, sort, and format metric dictionaries into a visually organized pivot table.
• Updated Logging Calls: Existing logging statements for completed operator metrics and general operator metrics have been updated to utilize the new tabular formatting function.
• New Test Case: A dedicated test case, test_format_metrics_table, has been added to ensure the correct functionality and output of the new metrics formatting utility.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• python/ray/data/_internal/execution/streaming_executor.py
  • Modified _scheduling_loop_step to use _format_metrics_table for logging completed operator metrics.
  • Added _format_metrics_table function to format metric dictionaries into a categorized pivot table.
  • Updated _log_op_metrics to also leverage _format_metrics_table for logging operator metrics.
• python/ray/data/tests/test_streaming_executor.py
  • Imported _format_metrics_table for testing purposes.
  • Added test_format_metrics_table to validate the output of the new metrics formatting function, including an empty dictionary scenario.

Activity

• The author, goutamvenkat-anyscale, requested a summary of the pull request.
• A bot provided a review comment suggesting a refactoring of the _format_metrics_table function for improved clarity and conciseness, including a code example.

[raulchen]

use from ray._private.thirdparty.tabulate.tabulate import tabulate for tabulation