Globus cli client config and configurable ingress pathType#900
Merged
jasonb5 merged 7 commits intov1.5.4-releasefrom Mar 19, 2026
Merged
Globus cli client config and configurable ingress pathType#900jasonb5 merged 7 commits intov1.5.4-releasefrom
jasonb5 merged 7 commits intov1.5.4-releasefrom
Conversation
- Add GLOBUS_CLI_CLIENT_ID and GLOBUS_CLI_CLIENT_SECRET to backend secret - Make ingress pathType configurable via values.yaml - Add validation for Globus credentials when wget.url is not set - Update test snapshots and test cases
…ecret.yaml Cleaned up unnecessary trailing blank lines to improve file formatting consistency.
downiec
added a commit
that referenced
this pull request
Mar 24, 2026
* Updated version number to 1.3.0 and added changelog file for 1.3.0, to be updated as needed. * Staticbuild v1.3.0 update (#690) * Harden images and reorg * fix timezone with package (#680) * Joyride tutorials update (#679) * Merged in latest v1.2.1 * Updates to the joyride tutorials and some minor bugfixes. * Revert Markdown update to fix current jest testing error * Updated React Markdown library and fixed the jest test errors. * Some modifications to remove the build and linting errors that came up after the merge with latest branch. * Load frontend config before rendering App Stop tests from leaking calls to globus.org * Some fixes to get the react front-end to build and get the postgres to start up. There are still errors with django session store. * Fixed some linting errors so frontend should build correctly. * Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved. * Switch from CRA to Vite and clean up tests (#692) * Switch from CRA to Vite and clean up tests * Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working. --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> * Staticbuild v1.3 fixes (#697) * Fix proxy and redirects for globus auth on docker compose * Various lint and test fixes * Update yarn.lock * Some changes made to try and get globus transfers working. * Update docs to match new config (#707) * Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved. * Switch from CRA to Vite and clean up tests (#692) * Switch from CRA to Vite and clean up tests * Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working. --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> * Fix proxy and redirects for globus auth on docker compose * Various lint and test fixes * Update yarn.lock * Some changes made to try and get globus transfers working. * Update docs to match new config Auto generate configuration documentation with pre-commit hooks * Revert yarn.lock, fix more tests * Includes some updates and minor fixes so that everything builds and runs correctly. Removed the GLOBUS_REDIRECT setting and did some cleanup of the manage_metagrid.sh functions. * Updated the test files to no longer use the act() functions, as they are deprecated. Cleaned up and fixed some failing tests. * Fix esgsearch json tests * Minor test fixes to see if tests will pass on github (they pass locally). --------- Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> Co-authored-by: Zach Price <pricezt@ornl.gov> --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com> --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com> Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com> * Fixed some linting errors related to unused imports * Some minor modifications to the pre-commit files, including removing repeated settings-doc-markdown config, to see if that resolves the pre-commit failure. * Fix broken link in Globus transfer panel (#719) * Fix broken link in Globus transfer panel * Globus link display update (#3) * Added a warning case for when a transfer request returns no failures or successes in the backend response. No successes indicates no transfer occurred although there wasn't any specified errors provided. Added new test for the added warning case. * Fix pre-commit prerequisites --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: Carlos Downie <42552189+downiec@users.noreply.github.com> * V1.3.0 additional fixes (#721) * Update version number, add version changelog (to be updated with the changes that will be included in this update) * Updated documentation references to config variables (#706) * Updated documentation, specifically references to the configuration parameters which have changed over time, to use the latest variable names currently used in configuration. Updated requirements to use the latest readthedocs them of Sphinx * Update version number, add version changelog (to be updated with the changes that will be included in this update) * fixed the pagination so that it will reset to page one when doing a new search (#705) * update for obs4MIPs (#711) Co-authored-by: Sasha Ames <ames4@llnl.gov> * Updated the settings file to autoformat on save to hopefully reduce linting errors. Fixed linting errors for the initial_projects_data update. Fixed the status tooltip to correctly display the node's status. Updated test fixtures to accurately reflect the node status response (the node names included the https:// in front and thats why status tooltip wasn't finding them. (#714) * Added alert for the case that a collection search fails in the manage collections popup (#710) * Added alert for the case that a collection search fails in the collection management popup. Added a test to bring the coverage up to 100% for the search table component. Removed some unused code, the get endpoint function is not used. * Minor linting error fixes. * Fixed some backend tests based on the removed/unused functions * Small fix to test. * Minor update to update summary. * Made it so the globus ready column does not appear if no globus ready nodes are configured (as it was before). Changed ActiveFacets type to be more strict in order to avoid the issue of calling join on a string which was causing errors. Fixed and updated some tests related to the url to string and string to url functions in the utils file. * Fixed failing test. Updated the manage_metagrid.sh helper scripts to include commands for pre-commits, backend and frontend tests. Updated setup.cfg to allow backend tests to run correctly. Removed unneeded commands in backend README.md. Removed the autoformat setting on save. --------- Co-authored-by: Sasha Ames <ames4@llnl.gov> * Dark mode (#718) * Add dark mode * Persist theme choice * Add tests for Dark mode feature * Merged in the latest changes from v1.3.0-release. Updated the styling to fix issues with the dark-mode rendering of navbar and certain colors. Added custom colors for dark and light version of status icons. Modified how colors are applied to tooltips and node status pages. I'd like to update/improve the way custom colors and styles are applied to make future updates smoother. (#4) --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> * Custom footer (#727) * Add dark mode * Persist theme choice * Add tests for Dark mode feature * Merged in the latest changes from v1.3.0-release. Updated the styling to fix issues with the dark-mode rendering of navbar and certain colors. Added custom colors for dark and light version of status icons. Modified how colors are applied to tooltips and node status pages. I'd like to update/improve the way custom colors and styles are applied to make future updates smoother. * Added a new configuration option for setting the footer of the metagrid frontend so that links and text can be displayed as desired. Updated the manage_metagrid script to include more developer actions, including run migrations, and installing packages for local development. Modified styling to better fit the current footer. Will need to add option for adjusting the footer height based on the height of the footer text. Updated tests and documentation for the custom footer options. * Final updates for the v1.3.0 release. Includes updates to the joyride tutorial to add the theme switch. Updated the manage_metagrid script and created a new configHelper script that sets the proper initial configuration settings for first time deployment or developers. Added tests for the theme buttons in the app. --------- Co-authored-by: Zach Price <pricezt@ornl.gov> * V1.3.1 release * Traefik for prod (#734) Co-authored-by: Zach Price <pricezt@ornl.gov> * Helm deployment fixes (#735) * Include helm chart and healthchecks * Modified the traefik.yml config to separate between local and production, in order to attempt to get production traefik deployment to function as it did in another branch. Updated docker compose files to separate keycloak into a profile and remove docs service from production. Updated manage script to utilize new compose files for deployment and testing. * Updated some docs and the manage scripts * Merged in some minor changes from traefik-deployment-fixes branch to update app css and include other minor fixes. * Delete docker-compose.prod-overlay.yml * Pre-comfig fixes, update to scripts. * Some updates to docker deployment to test if they will work in production * Separated trafik.yml into one for production and one for local, to avoid using the conditional statements that were being used which seemed to not be working for traefik deployment. Updated the deployment script to make sure local and production custm overlays are applied last so their variables aren't overriden. * Removed unnecessary prod_traefik file. * Updated prod overlay to name the containers with prod to distinguish deployment container types. Updated the config helper script to generate production overlay * Updated changelog. Added dark theme styling to the React Joyride tutorials. * Implemented some suggestions from PR comments. Brightened the tutorial background in dark theme. * Minor fix to compose file. * Reverted container names to leave them be. * Updated obs4mips url and modified the display button. Remove obsolete updateProjects.sh script and updated the README's regarding updating project data. Added the update functions to the manage_metagrid.sh scripts instead. Still need to test in production, but should work. * Fix minor lint error. Added build step to project migration function. --------- Co-authored-by: Zach Price <pricezt@ornl.gov> * Minor fix that was missed for changing duplicate Originals and Replicas label, to Replicas Only. * Initial commit for version 1.3.2. Includes a refactoring of how messages are updated and how the changelog is updated. The data for messages and changes is now located in a messageData.json file, removing the need to modify the messageDisplayData.ts file in multiple locations every time a new version and changelog entry is created. * Fixed minor issues with helm and wget downloads. * Updated management script to properly quit if tests fail. Updated do_request to add option for obtaining object using request.body, this fixes the wget download which expects data in a different format than other requests. Updated the changelog. * Initial v1.3.3 release version update, includes changes to News display, to make version history collapsed as it was getting long. * Added a dev action that will automatically generate necessary changes to increment/create a new version. It will create a new changelog file, update the package.json and update the messageData.json file to add the new version. * Initial v1.3.4 commit includes fix for path updating to proper value. * Fixed linting error, tests regard changes to path and added some extra save steps to see if that will resolve globus transfer token auth loop issue in Firefox. * Fixed linting error, tests regard changes to path and added some extra save steps to see if that will resolve globus transfer token auth loop issue in Firefox. * Updated the way paths are set so that an auth step is included if no globus permissions were requested. This seems to allow the path set redirect to request the appropriate permissions for allowing transfers to occur. * Updated change log to include mention of how path set functionality will behave. * Add hash annotation to Deployments to ensure restart when config changes, fix typos in ingress template (#743) Co-authored-by: Zach Price <pricezt@ornl.gov> * Removed unnecessary settings from the config helper. * Updated app version number for the Helm chart, added step in updateVersion script so helm chart version is automatically updated when the version number is updated. * Fixes helm & containers (#744) * Fixes hash for automatic deployment rolls * Fixes container trigger and pushing images * Fixes workflow tags * Fixes default filename for configHelper.sh * Fixes frontend makefile * Adds minimal frontend-config.js * Revert "Adds minimal frontend-config.js" This reverts commit 8856c9df1e8754241e1e50f443887c32a83ccef5. * Adds helm chart testing job * Fixes working directory * Updates workflow and adds test values * Fixes building dependency * Adds readme and helmfile to test locally * Fixes NOTES rendering * Adds defaults to support no-config install * Removes test.yaml since defaults were fixed * Fixes spelling error * Fixes workflow * Updates readme * Feature/v1.3.4 minor updates (#747) * Some minor updates to config helper and manage_metagrid scripts. * Added option to cancel endpoint path selection from globus side and it will return user to metagrid properly. * Updated to the manage script for updating the version values in helm files. Fixes some linting errors. * Added a tooltip to the download button to hint to user why it's disabled. Updated download button behavior ro make sure user can't start a Globus download without selecting a collection or start a wget download without selecting a cart item. * V1.3.4 patch (#749) * Created a DataBundlePersister class which will bundle key value pairs into a single object, stringify and then save to session storage as a single bundle. This will reduce requests to session storage by consolidating into a single request when values need to be persisted or loaded. * Minor updates to changelogs for consistent punctuation. * Some minor updates to persistent class * Updated the way session storage is used by the datasetdownload component to improve globus transfer reliability. Includes extensive refactoring of tests and test components to pass using the new DataBundlePersister class. Updated the cart summary to display info on selected cart items not just on the entire cart totals. * Some minor changes to the documentation to update the frontend project structure. Moved the reset tokens button to a menu in the globus transfer button. Globus transfer and wget download are separated into their own buttons using download type to select which is displayed. Updated tests as needed and added new ones for the reset token feature. Updated react joyride tour to mention changes on the summary table. * Made some minor updates, refactors and added new tests to increase test coverage * Update version used in README.md for helm * Initial update to version numbers. * Improve cart items and selections synchronization (#752) * Updated the App.tsx to make sure the selected items list is properly updated when new items are added to the cart or when items are removed from the cart. Fixes the issue where items were removed from the cart but the selection summary had the old selections there, which caused an issue with cart selections. Made it so that items added to cart are automatically selected for download, to make downloads more intuitive. * Fixed some broken tests, updated the changelog. * Fix the issue with duplicate saved searches and deleting saved searches. * Update v1.3.5.md * Fixes the JSON buttons by updating the clickableRoute function (#757) * Fixes the JSON buttons by removing th clickableRoute function which is no longer needed and was causing the button to fail (do nothing). Removed some commented out code that has been there a while. Update version in helm README. * Updated the JSON output links so that the url updates the location origin to match the current origin. Otherwise some older saved search cards will point to an older origin and the JSON feature doesn't work. * Added some tests for the createSearchRouteURL function and fixed some fixtures so other tests will pass. * Updated changelog * Minor change to the docker-compose to hide internal ports to only work with localhost. * Fix Helm Chart (#759) * Fixes version format in Chart.yaml * Fixes updating version in chart readme * Adds new job to link helm chart * Updates the chart version and script * Fixes to trigger on all pull requests * Removes updating the helm chart version from update script * App recoil refactoring (#760) * Significant refactoring to reduce the complexity and dependency among various components upon the main App component. Updated tests and added new recoil wrapper helper class that can set recoil state for tests. * Refactored some components to utilize recoil. Fixed tests to use the new recoil method. Updated customRender so that it can apply initial recoil state. * Another round or refactoring to simplify more components by using recoil state. Updated and modified the recoilWrapper to make it a singleton and then have it set defaults at the start of all the tests. Hopefully updating the test suite after refactoring will be more straight forward however still need to fix some broken tests. * Some updates to the test suite to fix broken tests and fix how the recoil state is initialized for each test. * Updated docker compose to close off internal ports to use only localhost * Performed another round of refactoring and updated utils, test functions and components to improve readability and testability. Fixed some minor issue with custom-render function not properly setting the auth state. Added coverage to bring test coverage up over 94%. * Updated the FacetsForm to correctly handle globus ready status * Fix minor linting error for recoil tests. * Fixed issue with joyride tutorials, where the add/minus buttons weren't being selected. Updated tutorial to work for the case where user is logged in. * Increase the testing timeout. Tests pass locally but seemed to fail/timeout on github. * Updated the timeout value for globus tests only, reverting the overall timeout for other tests. * More changes to see why test fails in github but not locally * Test with additional troubleshooting info. * Increase timeout time again * Increase timeout time again for specific failing test. * Skipping test for now, it passes locally but consistently fails in github, will need to troubleshoot further. * Initial commit with incremented version. * Migration to Jotai from Recoil (#765) * Initial migration to Jotai, includes updates test suite and modified test functions. Decided to centralize the atoms in a single atom file located in the common folder. 5 out of 400 tests were broken, will need to troubleshoot tests further. * Some cleanup of unused code and tests. Updated and fixed some tests so that all tests now pass. * Updated yarn.lock file. * Fixed a minor bug caused by having the stored atoms not initialize with their stored values. Now all the atoms with storage are set to load their stored value on initialization. Did some minor refactoring for a couple components to use some stored atoms rather than passing in props. Fixed tests that were affected by refactorings. * Did some code cleanup and updated changelog. * Added some tests and did some minor refactoring to increase coverage. Brought all component coverage to 100% except for the globus dataset download component. That will require some new tests and refactoring. * Fixed linter complaint * Feature/sortable table columns (#767) * Initial commit, added column sorting for both search and files tables. Updated the changelog, fixed all tests so they pass. * Added tests and updates to bring coverage back to 100% for all components except globus dataset downloads. * Changed version from 1.3.5 to 1.5.0 * Minor linter error fix. * Update metagrid_messages.md * Update metagrid_messages.md * Update v1.5.0.md - on News * Bump vite from 6.0.7 to 6.1.6 in /frontend (#771) Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.0.7 to 6.1.6. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v6.1.6/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v6.1.6/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.1.6 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update metagrid_messages.md * Add a banner to top of page (#774) * Initial commit containing a banner component which will show if the new BANNER_TEXT setting is configured. The Banner can be closed/hidden, but will re-appear upon page refresh. Behavior can be updated so that it will never show up again, if needed. * Updated package.json to remove unused Recoil dependency. Updated some minor packages. Updated the Banner component to stay hidden after uses clicks the close button, even after a refresh, by using localstorage variable. Added tests to covere the banner component feature. * Testing updates to github backend actions to better match the build steps used locally. Manually updated the docs to include BANNER_TEXT config data, since the auto-generation of the docs seems to not be working currently. * Added a hook to generate frontend settings (it seems it was missing) so that docs are updated properly. * Feature/optional node status (#773) * Removed references to SOLR URL as it is no longer used. Updated the configHelper to remove SOLR URL. Updated the STATUS_URL so that it is optional and then updated frontend navigation menu to hide node status page link if node status url is unset. Also updated table to only include node status indicators if node status url is configured. * Fixed the config helper to have node status url as optional. Updated the table so that when node status isn't available, the nodes name/data will still be shown with a tooltip. * Updated some tests to cover the changes to node status link in the navbar and the node status icon in the search table. * Remove E3SM project * Initial commit changing references of v1.5.0 to v1.4.0. * update message * Feature/cached searches (#778) * Implemented several updates to the api and search related components so that a search isn't constantly performed even when search facets and project were not changed. This allows users to switch pages in the app without clearing up their existing search they had made on the search page. Page size is saved and restored based on the length of the cached search results. Added an expiration time of 1 hour to the cached results. Cached results are also updated whenever the search is changed. Save the active search query to the local storage via Jotai atom with storage. Fixed some broken and failing tests. * Updated some components related to the search so that pagination is correctly displayed. If user selects a page other than 1, then when they return to search page, it will load page 1 again. It was more trouble to preserve pagination in the cache, but possible improvement to be done in future. * Updates to the saved searches cards, so that their search count is set immediately using the current result count (since it's already known). Then it is cached to avoid having to run an extra fetch, just to obtain the result count which was already known at time of saving the search. Also added an expiration time of 1 hour, so that saved searches can be updated after an hour, rather than all the time. Added logic that will make a savedSearch expired and therefore get updated whenever the search is used. Also added a date/time indicator on the search cards to know when the search was last updated or used. * Minor updates to remove unneeded if condition. Updates to the changelog to highlight changes regarding new caching behavior. * Feature/cached pagination (#779) * Implemented several updates to the api and search related components so that a search isn't constantly performed even when search facets and project were not changed. This allows users to switch pages in the app without clearing up their existing search they had made on the search page. Page size is saved and restored based on the length of the cached search results. Added an expiration time of 1 hour to the cached results. Cached results are also updated whenever the search is changed. Save the active search query to the local storage via Jotai atom with storage. Fixed some broken and failing tests. * Updated some components related to the search so that pagination is correctly displayed. If user selects a page other than 1, then when they return to search page, it will load page 1 again. It was more trouble to preserve pagination in the cache, but possible improvement to be done in future. * Updates to the saved searches cards, so that their search count is set immediately using the current result count (since it's already known). Then it is cached to avoid having to run an extra fetch, just to obtain the result count which was already known at time of saving the search. Also added an expiration time of 1 hour, so that saved searches can be updated after an hour, rather than all the time. Added logic that will make a savedSearch expired and therefore get updated whenever the search is used. Also added a date/time indicator on the search cards to know when the search was last updated or used. * Minor updates to remove unneeded if condition. Updates to the changelog to highlight changes regarding new caching behavior. * Updated caching to include pagination, regardless of which page is loaded. Only issue is that when loading a new search, pagination is preserved so the results may be of a page other than one. Will need to update so that page is reset to one with every new search. * Added modifications to the pagination caching behavior so that default page is 1 whenever a new search is performed (except in the case where pagination was the only change). * Updating the version number back to v1.5.0 and restored v1.5.0 news message. * Incremented version number to v1.4.1 for small patch updates. * verify=False as workaround for egress issue at LLNL (#782) * verify=False as workaround for egress issue at LLNL * Fixes PR triggers for building containers --------- Co-authored-by: Jason Boutte <boutte3@llnl.gov> * Fixes helm chart (#781) * Fixes postgresql password for bitnami charts >= 16.0.0 * Moves immutable config values out of values.yaml * Disable postgresql backup by default * Changelog and bugfix (#783) * verify=False as workaround for egress issue at LLNL (#782) * verify=False as workaround for egress issue at LLNL * Fixes PR triggers for building containers --------- Co-authored-by: Jason Boutte <boutte3@llnl.gov> * Includes incrementing helm chart version, adding changelog entry for 1.4.1 and a minor bugfix for banners. * Updated version value in the metagrind_message. --------- Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com> Co-authored-by: Jason Boutte <boutte3@llnl.gov> * Fixes triggers on PR's against master or push to master (#785) * Minor pre-commit fix * Updates to the metagrid_messages. * Fix pre-commit * Feature/storage update (#786) * Updates to how local storage is used, including compression of larger session storage data related to caching search results, so that local storage isn't filled. Also incorporate the use of session storage for values that do not need to persist beyond open tabs or windows. * Updates to the message text to introduce changes made in v1.4.0 * Updates to tests and using a different compression library so that all tests are passing. Updates to some util functions and improved test coverage. * Reverting to using local storage for search results, as using duplicate tabs can lead to erratic behavior when session storage is used. * Updated how the messages are displayed so that the news isn't inside a collapsible component. * Updated how keys are generated and set default to 0 (the first item in messages list). * Initial commit with version increment. * change facets list (#795) * Added error handling for failed fetch search results. (#791) * Added error handling for failed fetch search results. If search fails, and cached results are available it will use cached results and notify user there's a problem. Resets the pagination and current request URL. If issue occurs in rednering facets, (sometimes caused by old cache) we reset search cache. Cached results will still expire after an hour and therefore the fetch error will be shown if the fetching problems persist for long enough. * Fixed minor lint error. * Added eagle at anl as globus ready by default. Fixed issue with globus download process looping, the transfer goal state was getting lost when using session storage, so reverted to using local storage. * Minor fix to pre-commit errors, update to changelog. * Bugfix/globus multi file Globus download (#797) * Modified the get request in the search_files function, so that it will return file lists from multiple datasets. Would like to convert this to a POST request later on. * Fixed tests and added new test to use dataset_id parameter. * Reverted the changes made to handle the failed search results. There was some unstable looping caused by changes made during error handling. (#800) * Fixes disabling node status (#799) * Fixed some linting errors and incremented helm chart version * Initial commit, version incremented. * V1.5.1 package updates (#776) * Updated some packages to latest version. Updated the changelog for v1.5.0. * Updated axios and removed unused lib/axios (it has typescript support now). Updated react-markdown and react-hotjar, while addressing breaking changes. Updated a few other packages which didn't seem to need any further fixes. * Updated the js-pkce package and made necessary updates to the DatasetDownloads functions. * Updated antd library, migrated react-router-dom 7.5 to react-router 7.6, uuid to 11, eslint configs, and prettier. * Fixed linter errors and failing tests. * Updated some linting rules and packages. Fixed minor linting errors. * Some more updates to package.json for eslint and linting fixes. * Add speicifc timeout for test which is failing due to timeout error. * Update timeout for test which is failing due to timeout error. * Minor update to changlog. * Change the GET into a POST request for the search_files function used to create globus transfer tasks. * Updated tests for backend changes. * Updated the Search Table to have a results limit of 10000, to avoid e… (#812) * Updated the Search Table to have a results limit of 10000, to avoid errors when attempting to view a page containing results beyond 10000. Added a handler that will reset cached page to 1 if there is a 422 error (the error caused by requesting a page beyond supported limit). * Increased coverage, added test for new 422 error handling. Updated coverage in other places. * Update metagrid_messages.md - pyesgf message (#813) * Feature/update to json link (#814) * Updated the Search Table to have a results limit of 10000, to avoid errors when attempting to view a page containing results beyond 10000. Added a handler that will reset cached page to 1 if there is a 422 error (the error caused by requesting a page beyond supported limit). * Increased coverage, added test for new 422 error handling. Updated coverage in other places. * Modified the createSearchRouteURL function so that the configured search setting is used for the JSON link, rather than the proxy path. * Made trivial change to the URL used as an example for the search url setting. * Initial commit with version incremented to v1.5.3. * Updates to Globus to utilize backend for all Globus operations (#827) * Change heading to Site Administrators, clarify purpose in docs, DRCDP data_node (#816) * Update mkdocs.yml * clarify purpose of docs * DRCDP corrections add data_node correct facet group name * Hotfix/globus transfer (#819) * Hot fix for current globus transfer issues. Reverted to using get rather than post for the search_files helper function used when submitting transfers. Added some additional logic to help improve debugging. * Updated tests and added two more to cover the new test cases and logic that was added. Updated changelog * Rewrote entire backend globus login flow to eliminate the need for a nativeclient on the frontend, instead utilizing a confidential client on the backend. Simplifies the download process while improving reliability. Removed the need to persistently store the globus related tokens on the frontend. Need to do some cleanup to remove obsolete code. * Removed commented out code and deleted experimental functions that are no longer needed. Basic cleanup. * Removed obsolete GLOBUS_CLIENT_ID settings related code, as we only need the client key and secret now. Removed broken tests from backend, will create new ones later to bring coverage back to 100%. Removed more unused code. * Added some try/except statements on the globus api to provide more information if an error occurs. Fixed a minor bug that was causing the auth url not to be set when it was needed. * Due to the added testing complexity for the backend globus functions, reduced the 100% coverage requirements to 95% (will set it back to 100% when tests can be fully covered). Brought the test coverage for the backend up to 98%, will need to cleanup and write more tests to get the last 2% of coverage. Fixed all broken tests and removed obsolete tests for the frontend globus related features. Removed fixtures and code that are no longer needed in the frontend, now that globus transfer is mostly handled by the backend. * Updated the changelog with current changes in this branch. Switched from access tokens to refresh tokens and updated tests accordingly. Added function to reset stored tokens on the backend so that consents can be refreshed should the need ever arrive or for testing purposes. Added tests to cover new api function for token reset. * Minor update to make sure the url is reset when errors occur in globus transfer attempts. --------- Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com> * Update do_citation to support a whitelist of multiple domains (#838) * Change heading to Site Administrators, clarify purpose in docs, DRCDP data_node (#816) * Update mkdocs.yml * clarify purpose of docs * DRCDP corrections add data_node correct facet group name * Hotfix/globus transfer (#819) * Hot fix for current globus transfer issues. Reverted to using get rather than post for the search_files helper function used when submitting transfers. Added some additional logic to help improve debugging. * Updated tests and added two more to cover the new test cases and logic that was added. Updated changelog * support multiple domains for citation API * Update test_views.py --------- Co-authored-by: Carlos Downie <42552189+downiec@users.noreply.github.com> * Feature/integrate esgpull (#837) * Modified the search page copy and save search buttons to be a menu instead, where the copy search feature will be in a drop-down. Then added 2 new copy features related to esgpull, one for search and one for download. * Added Intake ESGF search option and created a function to generate the python code to run the search. The command assumes you have intake installed and ready to use. * cleaned up some code and fixed broken tests. * Added tootltips and updated the joyride tutorials to show the search copy menu items. Updated the changelog with notes about the update. Added several tests to increase coverage and verify the copy search options give the appropriate output. * Updates to esgpull search and download feature so that unnecessary facets are removed from valid list and missing ones were added. Added custom comments on esgpull commands to highlight any facets that were not applied. Other minor updates. * Adding an esgpull download options for a single dataset using the master_id attribute. * Fix some tests and increase coverage. * Minor changes to fix precommit issues. * Updated the backend.yml postgres to match the postgres version used in Dockerfile * Testing helm workflow timeout increase to see if it is merely a timing issue. * Reverting changes for postgres image version and timeout changes made for helm. * Update initial_projects_data.py to change E3SM (#843) * Update initial_projects_data.py to change E3SM * change E3SM logic * Initial commit for v1.5.4-release. * Add --build flag to docker compose command (#855) * messages update (#844) remove extraneous and clarify * Feature/stac integration 1.5 (#811) * Initial testing commit, merging in most significant changes from the previous STAC integration branches. * linting fixes. * Merged in the latest changes from v1.5.1 branch, fixed some broken code cuased by the STAC changes. Updated tests so that they pass. * Updated the tables to include file sizes and checksums for STAC results if the data is available. Removed the numResults requirement as it seems to be missing sometimes, instead counting the results features array directly. Updated the filename to utilize assets key when there is no id, title or name attribute in the asset object. * Added objects to map differences between CMIP6 facets and facets url. Created some functions to generate stac filters for and, or and equals operations. Updated search to utilize generated filters. Fixed minor issue with metadata values that weren't being displayed. Updated the tabs to utilize cmip6:further_info_url data for es-docs. * Fixed minor issue with filtering so that OR filters work properly. Updated search limit to 9999 rather than only 10. Still need to make page size work properly. Updated the query string to display the generated STAC filter string if project is a STAC project. * Removed the 'All except CMIP6' project as it seems to be broken now. Updated SearchesCards to handle the STAC Project searches. Modified STAC request url to include the project name and made it so future STAC projects can be added and will work. Adde some error checking for STAC search results so that status is returned after a search. * Modified the search page copy and save search buttons to be a menu instead, where the copy search feature will be in a drop-down. Then added 2 new copy features related to esgpull, one for search and one for download. * Added Intake ESGF search option and created a function to generate the python code to run the search. The command assumes you have intake installed and ready to use. * cleaned up some code and fixed broken tests. * Added tootltips and updated the joyride tutorials to show the search copy menu items. Updated the changelog with notes about the update. Added several tests to increase coverage and verify the copy search options give the appropriate output. * Implemented aggregations for STAC so that result counts are shown for specific facets. Added STAC icon and fixed table to display the STAC icon instead of dataset icon, for STAC datasets. Other changes. * Small fixes after v1.5.3-release merge. * Updates to esgpull search and download feature so that unnecessary facets are removed from valid list and missing ones were added. Added custom comments on esgpull commands to highlight any facets that were not applied. Other minor updates. * Adding an esgpull download options for a single dataset using the master_id attribute. * Updated postgres version to 16-alpine to remove the critical vulnerability in postgres-16. Minor fixes. * Fix some tests and increase coverage. * Removed the default filter for STAC * Removed the All (except CMIP6) project option since it appears to no longer work. Added logic to hide STAC projects if STAC URL is not configured. * Updated authentication options in the script and settings so that 'none' is now a deployment option. When none is set, there will not be a sign in button in the navigation bar. Updated keycloak deployment so that a basic keycloak instance is created which can be connected to for configuring it as an admin. The keycloak authentication doesn't work, so there are still issues with how the authentication is handled in the frontend. Updated some tests and added keycloak overlays to handle keycloak deployment while skipping it when none or globus options are selected. * Updated the cart, and saved searches to filter out STAC items when STAC is not active. Also updated the navbar to show the correct filtered count. Updated the search cards to display the STAC filter rather than the regular filter, when showing the query string. * Added several tests on various components and removed the facet route in order to improve overall test coverage. * Fixed issues with pagination and page size. * Fixed edge case where a logged in user can't perform globus search or transfer if the deployment has authentication method set to none. Minor updates to linter config settings to remove deprecated settings. * Updates database (#854) * Updates backend database settings to be configurable * Allows helm testing to fail * Removes pydantic database settings, replaces with environ.db_url to parse DATABASE_URL * Updates chart to use new DATABASE_URL for backend and adds examples * Updates docker-compose to use DATABASE_URL * Moves helm chart testing to helm-chart workflow, adds workflow trigger * Updates backend workflow to use DATABASE_URL * Removes job requirements * Fixes workflow trigger * Removes pull_request trigger * Merges containers and helm-chart workflows * Removes bitnami repo * Fixes running migrate job on upgrade * Updates default search/wget urls * Adds job to test upgrade * Fixes liveness/readiness probe indentation, fixes duplication port labels * Fix some issues with precommit and startup (#858) * Fix some issues with precommit and startup * Fixes pre-commit workflow to run on all pull requests --------- Co-authored-by: Jason Boutte <boutte3@llnl.gov> * Feature/wget integration (#859) * Initial integration of wget functions. Settings and config still need to be resolved * Finished adding files, and fixing issues with the wget downloads, downloads are now working properly. Need to add a few tests to increase test coverage. * Fix some issues with precommit and startup * Updated the feature to allow using a dedicated wget url (as it was before) so that the integrated wget is only active if the WGET_URL setting is None (default). Fixed tests to handle changes, the WGET_URL external test is only run if the WGET_URL is set, otherwise the test is skipped. * Added STAC wget downloads, where a script is automatically generated when STAC items are selected for wget download in the cart. The wget download can handle both stac and non-stac items being downloaded. Added some tests for the newly added functions and updated the wget success messages to account for STAC and non-STAC wget script results. Still need to add wget download for individual STAC records within the search Table. * Updated the Table to handle wget downloads for STAC datasets * The project name is supposed to be lowercase according to the STAC standards, therefore checking uppercase is not necessary. * Updated backend coverage to ignore unused utility function. * Feature/custom support info (#863) * Initial integration of wget functions. Settings and config still need to be resolved * Finished adding files, and fixing issues with the wget downloads, downloads are now working properly. Need to add a few tests to increase test coverage. * Fix some issues with precommit and startup * Updated the feature to allow using a dedicated wget url (as it was before) so that the integrated wget is only active if the WGET_URL setting is None (default). Fixed tests to handle changes, the WGET_URL external test is only run if the WGET_URL is set, otherwise the test is skipped. * Added STAC wget downloads, where a script is automatically generated when STAC items are selected for wget download in the cart. The wget download can handle both stac and non-stac items being downloaded. Added some tests for the newly added functions and updated the wget success messages to account for STAC and non-STAC wget script results. Still need to add wget download for individual STAC records within the search Table. * Updated the Table to handle wget downloads for STAC datasets * The project name is supposed to be lowercase according to the STAC standards, therefore checking uppercase is not necessary. * Updated backend coverage to ignore unused utility function. * Initial update to include new METAGRID_SUPPORT_INFO optional setting that allows admin support information to be included by site admins for users to reach out. * Fixes globus transfer issues when authenticated (#866) * Found an issue with authenticated user transfer client not having the scope updated causing transfers to fail. Updated the frontend to display a maintenance/error page when the backend fails to provide settings values. * Minor update to increase test coverage. * Feature/production deployment fixes (#867) * Updated a few packages to remove django backend warnings. Updated docker overlays to see if production deployment wil run successfully with default configuration. * Updated the configHelper to include newly added and changed settings. Added a refresh collation and reindex function for the postgres database, for cases where postgres container is updated and there is a collation version mismatch error during deployment. * Fixed a globus transfer backend test. * Modified backend test to fix issue with non-matching client id * Fix pre-commit issues. * add WrPMIP and obs4MIPs update (#874) * Fixed minor pre-commit issues. * Feature/relocate keyword search (#871) * Made visual and functional updates to the Facets Filter component. Filename search is removed, additional properties panel is hidden for STAC search, added Keyword Search panel and removed keyword search from the top nav bar, updated the sizing of the filter option drop-downs to improve the readability of the listed items, updated the antd library to latest version of 5.x, updated the changelog to include latest changes for v1.5.4. Still need to fix the tests that were broken by the changes to the U.I. * Fixed more than half of the failing tests. Restored the deleted filename search and commented it out, in case we decide to bring it back or users miss it. * Fixed remaining tests, added 2 for the keyword search feature and skipped the tests associated with filename search. * Added more tests and updated some components to improve test suite coverage. * Updated axios package to latest, minor updates to the changelog and fixed a bug that occurs where setting the path to a newly added collection wasn't redirecting to that specific collection. * Updated joyride tutorial for main page to remove the top navbar keyword search and added it to the facets tutorial. * Increased test coverage and made the keyword search hidden for STAC projects (for the time being, until keyword search is implemented). * Update helm chart for v1.5.4 (#872) * Adds/enables postgresql persistence * Updates readme with v1.5.4 database upgrade instructions * Exposes wget feature * Removes old bitnami repo * Fixes appending DJANGO_ALLOWED_HOSTS * Updates README * Config helper fixes (#876) * Added fixes to the configHelper script to allow setting the password for a postgres database. Updated the default prod overlay to include values for user and password, updated the variable names for postgres and password to match the latest version of postgres. Created a template for settings, in the case the config helper isn't used, the template can be used instead. Fixed a minor issue with metagrid manage script starting up production for globus, and added a provider to the error page render. * Fix minor pre-commit issue * Update helm chart (#885) * Disables node status feature by default * Fixes only creating node status resources if enabled * Updates helm chart readme with FAQ * Fixes helm chart readme toc * Fix indenting FAQ answer * Updates helm chart readme faq * Updates node status * Change node_status secret to configmap, split backend secret into configmap/secret * Another helm update (#888) * Fixes wget globus public index spelling * Adds initial helm unittest for backend * Adds initial helm unittests for frontend * Adds initial helm unittests for nodestatus * Adds initial helm unittests for postgresql * Adds initial helm unittests for ingress * Fixes settings DJANGO_ALLOWED_HOSTS * Fixes allowing additional hosts for ingress TLS and django * Updates readme * Tests disabling database * Updates backend deployment checksum to include configmap * Adds documentation for testing chart. * Updates workflow to unittest helm chart * Fixs pre-commit issues * Pins the helm version installed for unittesting * Chart updates (#893) * Expose tls secretName * Fixes handling multiple ingress hosts * Fixes adding config/secret to backend * Defines DJANGO_SOCIAL_AUTH_REDIRECT_IS_HTTPS when ingress tls is enabled * Globus cli client config and configurable ingress pathType (#900) * feat(helm): add globus client config and configurable ingress pathType - Add GLOBUS_CLI_CLIENT_ID and GLOBUS_CLI_CLIENT_SECRET to backend secret - Make ingress pathType configurable via values.yaml - Add validation for Globus credentials when wget.url is not set - Update test snapshots and test cases * docs: update README to match current values.yaml * chore(helm): remove trailing blank lines from README.md and backend secret.yaml Cleaned up unnecessary trailing blank lines to improve file formatting consistency. * fix: update snapshots * fix: required settings for install/upgrade in GH workflow * fix: convert to strings * fix: ensure values are quoted --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com> Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com> Co-authored-by: Sasha Ames <ames4@llnl.gov> Co-authored-by: Jason Boutte <jasonb5@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jason Boutte <boutte3@llnl.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes # (issue)
Type of change
How Has This Been Tested?
Checklist