[BUG] Allow user to control react agent max_interations value to prevent empty response by pyek-bot · Pull Request #3756 · opensearch-project/ml-commons

pyek-bot · 2025-04-23T00:03:32Z

Description

Allows user of PlanExecute Agent to control react agent max_iterations. This is important as the default value of max_iterations is set to 10. However, when this limit is exhausted then the model returns an empty response.
Modifies the system_prompts to be more appropriate for PlanExecuteReflect.
Returns react agent memory details to user in response

Related Issues

Resolves #3753

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ylwu-amzn · 2025-04-24T22:55:02Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

-    private static final String DEFAULT_SYSTEM_PROMPT = "Always respond in JSON format.";
-    private static final String DEFAULT_REACT_SYSTEM_PROMPT = "You are a helpful assistant.";
+    private static final String DEFAULT_SYSTEM_PROMPT =
+        "You are part of an OpenSearch cluster. When you deliver your final result, include a comprehensive report. This report MUST:\\n1. List every analysis or step you performed.\\n2. Summarize the inputs, methods, tools, and data used at each step.\\n3. Include key findings from all intermediate steps — do NOT omit them.\\n4. Clearly explain how the steps led to your final conclusion.\\n5. Return the full analysis and conclusion in the 'result' field, even if some of this was mentioned earlier.\\n\\nThe final response should be fully self-contained and detailed, allowing a user to understand the full investigation without needing to reference prior messages. Always respond in JSON format.";


Curious why add You are part of an OpenSearch cluster ?

It is important to give the model context about how it is being used. Without this, it relies on the tool descriptions which may not be detailed enough to guide the model that it is being used within OpenSearch.

I modified the react_system_prompt in the latest commit. I removed some information that I was using to bypass a bug. So please check that again as well!

bug: #3755

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

…step result Signed-off-by: Pavan Yekbote <pybot@amazon.com>

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

dhrubo-os · 2025-04-25T00:59:50Z

Don't we have any unit test for this class?

codecov · 2025-04-25T01:43:43Z

Codecov Report

Attention: Patch coverage is 0% with 35 lines in your changes missing coverage. Please review.

Project coverage is 78.08%. Comparing base (9e417e6) to head (a84e1c6).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...thms/agent/MLPlanExecuteAndReflectAgentRunner.java	0.00%	33 Missing ⚠️
...ml/action/agents/TransportRegisterAgentAction.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3756      +/-   ##
============================================
- Coverage     78.12%   78.08%   -0.04%     
  Complexity     7156     7156              
============================================
  Files           631      631              
  Lines         32211    32227      +16     
  Branches       3660     3661       +1     
============================================
  Hits          25164    25164              
- Misses         5492     5508      +16     
  Partials       1555     1555

Flag	Coverage Δ
ml-commons	`78.08% <0.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pyek-bot · 2025-04-25T02:23:02Z

Don't we have any unit test for this class?

No we don't. Tracking it in this issue: #3750

I am working on it parallelly. However, trying to keep this PR specific to the fix since we want to backport it to 3.0 for GA.

pyek-bot · 2025-04-25T23:21:23Z

Please add 3.0 backport label!

…ent empty response (#3756) * fix: expose max_iteration for react Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: defaults for agent execution and differentiate between step and step result Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: return react agent id in agent response to expose more details Signed-off-by: Pavan Yekbote <pybot@amazon.com> * spotless Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: remove test prompt from react system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> * refactor: rename parameters exposed to user to executor Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: give user complete control over planner system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> --------- Signed-off-by: Pavan Yekbote <pybot@amazon.com> (cherry picked from commit 8339bcf)

…ent empty response (#3756) (#3784) * fix: expose max_iteration for react Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: defaults for agent execution and differentiate between step and step result Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: return react agent id in agent response to expose more details Signed-off-by: Pavan Yekbote <pybot@amazon.com> * spotless Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: remove test prompt from react system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> * refactor: rename parameters exposed to user to executor Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: give user complete control over planner system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> --------- Signed-off-by: Pavan Yekbote <pybot@amazon.com> (cherry picked from commit 8339bcf) Co-authored-by: Pavan Yekbote <pybot@amazon.com>

* [BUG] Agent Framework: Handle model response when toolUse is not accompanied by text (#3755) * fix: handle model response when toolUse is not accompanied by text Signed-off-by: Pavan Yekbote <pybot@amazon.com> * feat: add test case for parseLLMOutput Signed-off-by: Pavan Yekbote <pybot@amazon.com> --------- Signed-off-by: Pavan Yekbote <pybot@amazon.com> * [BUG] Allow user to control react agent max_interations value to prevent empty response (#3756) * fix: expose max_iteration for react Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: defaults for agent execution and differentiate between step and step result Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: return react agent id in agent response to expose more details Signed-off-by: Pavan Yekbote <pybot@amazon.com> * spotless Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: remove test prompt from react system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> * refactor: rename parameters exposed to user to executor Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: give user complete control over planner system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> --------- Signed-off-by: Pavan Yekbote <pybot@amazon.com> * Clean up JSM from MCP (#3773) Signed-off-by: rithin-pullela-aws <rithinp@amazon.com> * [Bug] ListTools call does not return tool attributes (#3785) * initial commit for MCP server in OpenSearch (#3781) * initial commit for MCP server in OpenSearch Signed-off-by: zane-neo <zaniu@amazon.com> * Make change to support register or remove tools across cluster Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * fix UT failure caused by code change Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * format code Signed-off-by: zane-neo <zaniu@amazon.com> * add license header Signed-off-by: zane-neo <zaniu@amazon.com> * fix notifications initialized not respond issue Signed-off-by: zane-neo <zaniu@amazon.com> * fix minor issues and add UTs Signed-off-by: zane-neo <zaniu@amazon.com> * Add more UTs Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com> * Remove beta1 qualifier (#3794) (#3795) (cherry picked from commit 3f503f1) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> * [AUTO] Increment version to 3.1.0-SNAPSHOT (#3789) * Increment version to 3.1.0-SNAPSHOT Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> * Update build.gradle Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> --------- Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> * add release note for 3.0 (#3792) Signed-off-by: Mingshi Liu <mingshl@amazon.com> * support MCP session management (#3803) * support MCP session management Signed-off-by: zane-neo <zaniu@amazon.com> * Addressing comments Signed-off-by: zane-neo <zaniu@amazon.com> * add feature flag for mcp server and renaming mcp connector feature flag Signed-off-by: zane-neo <zaniu@amazon.com> * Address critical comments in #3781 Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com> * upgrade http client to version align with core (#3809) * upgrade http client to versoin align with core Signed-off-by: zane-neo <zaniu@amazon.com> * upgrade httpclient-h2 to correct versiono Signed-off-by: zane-neo <zaniu@amazon.com> * use placeholder approach Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com> * support customized message endpoint and addressing comments (#3810) * support customized message endpoint and addressing comments Signed-off-by: zane-neo <zaniu@amazon.com> * fix UT failures Signed-off-by: zane-neo <zaniu@amazon.com> * add files to jacoco exception Signed-off-by: zane-neo <zaniu@amazon.com> * fix tool name issue and optimize register tool api Signed-off-by: zane-neo <zaniu@amazon.com> * fix schema not parsed correctly issue and NPE when parameters is null Signed-off-by: zane-neo <zaniu@amazon.com> * fix failure UT Signed-off-by: zane-neo <zaniu@amazon.com> --------- Signed-off-by: zane-neo <zaniu@amazon.com> * excluding circuit breaker for Agent (#3814) Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * change release note (#3811) * change release note Signed-off-by: zane-neo <zaniu@amazon.com> * Update opensearch-ml-common.release-notes-3.0.0.0.md * Update opensearch-ml-common.release-notes-3.0.0.0.md * Update opensearch-ml-common.release-notes-3.0.0.0.md --------- Signed-off-by: zane-neo <zaniu@amazon.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> * Downgrade MCP version to 0.9 (#3821) Signed-off-by: rithin-pullela-aws <rithinp@amazon.com> * remove libs folder (#3824) Signed-off-by: Yaliang Wu <ylwu@amazon.com> * add more logging to deploy/undeploy flows for better debugging (#3825) * add more logging to deploy/undeploy flows for better debugging Signed-off-by: Bhavana Goud Ramaram <rbhavna@amazon.com> * Fix python client not able to connect to MCP server issue (#3822) Signed-off-by: zane-neo <zaniu@amazon.com> Co-authored-by: Dhrubo Saha <dhrubo@amazon.com> * exclude trusted connector check for hidden model (#3838) Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * adding tenantId to the connector executor when this is inline connector (#3837) * adding tenantId to the connector executor when this is inline connector Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> * added more unit tests Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> --------- Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> --------- Signed-off-by: Pavan Yekbote <pybot@amazon.com> Signed-off-by: rithin-pullela-aws <rithinp@amazon.com> Signed-off-by: zane-neo <zaniu@amazon.com> Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> Signed-off-by: opensearch-ci-bot <opensearch-infra@amazon.com> Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Dhrubo Saha <dhrubo@amazon.com> Signed-off-by: Yaliang Wu <ylwu@amazon.com> Signed-off-by: Bhavana Goud Ramaram <rbhavna@amazon.com> Co-authored-by: Pavan Yekbote <pybot@amazon.com> Co-authored-by: Rithin Pullela <rithinp@amazon.com> Co-authored-by: zane-neo <zaniu@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: opensearch-ci-bot <opensearch-infra@amazon.com> Co-authored-by: Mingshi Liu <mingshl@amazon.com> Co-authored-by: Yaliang Wu <ylwu@amazon.com> Co-authored-by: Bhavana Goud Ramaram <rbhavna@amazon.com>

…ent empty response (opensearch-project#3756) * fix: expose max_iteration for react Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: defaults for agent execution and differentiate between step and step result Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: return react agent id in agent response to expose more details Signed-off-by: Pavan Yekbote <pybot@amazon.com> * spotless Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: remove test prompt from react system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> * refactor: rename parameters exposed to user to executor Signed-off-by: Pavan Yekbote <pybot@amazon.com> * fix: give user complete control over planner system prompt Signed-off-by: Pavan Yekbote <pybot@amazon.com> --------- Signed-off-by: Pavan Yekbote <pybot@amazon.com> Signed-off-by: Abdul Muneer Kolarkunnu <muneer.kolarkunnu@netapp.com>

pyek-bot requested review from HenryL27, Zhangxunmt, austintlee, b4sjoo, dhrubo-os, jngz-es, mingshl, model-collapse, rbhavna, xinyual, ylwu-amzn and zane-neo as code owners April 23, 2025 00:03

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 23, 2025 00:04 — with GitHub Actions Error

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 23, 2025 00:04 — with GitHub Actions Failure

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 23, 2025 03:22 — with GitHub Actions Error

pyek-bot changed the title ~~[BUG/Enhancement] Allow user to control react agent max_interations value to prevent empty response~~ [BUG] Allow user to control react agent max_interations value to prevent empty response Apr 24, 2025

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 22:53 — with GitHub Actions Error

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 22:53 — with GitHub Actions Failure

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 22:53 — with GitHub Actions Error

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 22:53 — with GitHub Actions Failure

ylwu-amzn reviewed Apr 24, 2025

View reviewed changes

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 23:10 — with GitHub Actions Failure

ylwu-amzn reviewed Apr 24, 2025

View reviewed changes

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java Outdated Show resolved Hide resolved

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 23:50 — with GitHub Actions Failure

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 24, 2025 23:56 — with GitHub Actions Failure

pyek-bot added 7 commits April 24, 2025 17:36

fix: expose max_iteration for react

eaf4d97

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

fix: defaults for agent execution and differentiate between step and …

de3acc5

…step result Signed-off-by: Pavan Yekbote <pybot@amazon.com>

fix: return react agent id in agent response to expose more details

620e878

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

spotless

7266f24

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

fix: remove test prompt from react system prompt

c91b17b

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

refactor: rename parameters exposed to user to executor

d5dfa8a

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

fix: give user complete control over planner system prompt

a84e1c6

Signed-off-by: Pavan Yekbote <pybot@amazon.com>

pyek-bot force-pushed the bug_fix_expose_react_max_iterations branch from 954f012 to a84e1c6 Compare April 25, 2025 00:38

pyek-bot temporarily deployed to ml-commons-cicd-env-require-approval April 25, 2025 00:39 — with GitHub Actions Inactive

pyek-bot temporarily deployed to ml-commons-cicd-env-require-approval April 25, 2025 01:44 — with GitHub Actions Inactive

pyek-bot had a problem deploying to ml-commons-cicd-env-require-approval April 25, 2025 01:44 — with GitHub Actions Failure

pyek-bot temporarily deployed to ml-commons-cicd-env-require-approval April 25, 2025 23:19 — with GitHub Actions Inactive

ylwu-amzn approved these changes Apr 25, 2025

View reviewed changes

jngz-es approved these changes Apr 26, 2025

View reviewed changes

ylwu-amzn merged commit 8339bcf into opensearch-project:main Apr 26, 2025
11 of 13 checks passed

mingshl added the backport 3.0 label Apr 26, 2025

opensearch-trigger-bot bot mentioned this pull request Apr 26, 2025

[Backport 3.0] [BUG] Allow user to control react agent max_interations value to prevent empty response #3784

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Allow user to control react agent max_interations value to prevent empty response#3756

[BUG] Allow user to control react agent max_interations value to prevent empty response#3756
ylwu-amzn merged 7 commits intoopensearch-project:mainfrom
pyek-bot:bug_fix_expose_react_max_iterations

pyek-bot commented Apr 23, 2025 •

edited

Loading

Uh oh!

ylwu-amzn Apr 24, 2025

Uh oh!

pyek-bot Apr 24, 2025

Uh oh!

pyek-bot Apr 24, 2025

Uh oh!

Uh oh!

dhrubo-os commented Apr 25, 2025

Uh oh!

codecov bot commented Apr 25, 2025 •

edited

Loading

Uh oh!

pyek-bot commented Apr 25, 2025

Uh oh!

pyek-bot commented Apr 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

pyek-bot commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

ylwu-amzn Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dhrubo-os commented Apr 25, 2025

Uh oh!

codecov bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pyek-bot commented Apr 25, 2025

Uh oh!

pyek-bot commented Apr 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pyek-bot commented Apr 23, 2025 •

edited

Loading

codecov bot commented Apr 25, 2025 •

edited

Loading