Merged
Conversation
…o be updated as needed.
* Harden images and reorg * fix timezone with package (#680) * Joyride tutorials update (#679) * Merged in latest v1.2.1 * Updates to the joyride tutorials and some minor bugfixes. * Revert Markdown update to fix current jest testing error * Updated React Markdown library and fixed the jest test errors. * Some modifications to remove the build and linting errors that came up after the merge with latest branch. * Load frontend config before rendering App Stop tests from leaking calls to globus.org * Some fixes to get the react front-end to build and get the postgres to start up. There are still errors with django session store. * Fixed some linting errors so frontend should build correctly. * Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved. * Switch from CRA to Vite and clean up tests (#692) * Switch from CRA to Vite and clean up tests * Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working. --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> * Staticbuild v1.3 fixes (#697) * Fix proxy and redirects for globus auth on docker compose * Various lint and test fixes * Update yarn.lock * Some changes made to try and get globus transfers working. * Update docs to match new config (#707) * Updated the local config file used for local testing. The new file .local_config, contains the configuration used when running Metagrid locally. Updated the manage_metagrid script to run Metagrid from the single docker-compose file. Updated some settings to use llnl instead of ornl for local development and testing. Removed some unused environment variable files after after testing that they aren't used. Left the default variables in place because Metagrid fails to build when several of the defaults aren't set, still working to troubeshoot why. Globus transfer steps in the front-end are working, however transfers seem to fail and need to be resolved. * Switch from CRA to Vite and clean up tests (#692) * Switch from CRA to Vite and clean up tests * Fixed issue with loading screen continuing even after cancel button click. Updated local test config. Globus transfers still not working. --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> * Fix proxy and redirects for globus auth on docker compose * Various lint and test fixes * Update yarn.lock * Some changes made to try and get globus transfers working. * Update docs to match new config Auto generate configuration documentation with pre-commit hooks * Revert yarn.lock, fix more tests * Includes some updates and minor fixes so that everything builds and runs correctly. Removed the GLOBUS_REDIRECT setting and did some cleanup of the manage_metagrid.sh functions. * Updated the test files to no longer use the act() functions, as they are deprecated. Cleaned up and fixed some failing tests. * Fix esgsearch json tests * Minor test fixes to see if tests will pass on github (they pass locally). --------- Co-authored-by: downiec <42552189+downiec@users.noreply.github.com> Co-authored-by: Zach Price <pricezt@ornl.gov> --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com> --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: Sasha Ames <sashakames@users.noreply.github.com> Co-authored-by: bstrdsmkr <bstrdsmkr@gmail.com>
…repeated settings-doc-markdown config, to see if that resolves the pre-commit failure.
* Fix broken link in Globus transfer panel * Globus link display update (#3) * Added a warning case for when a transfer request returns no failures or successes in the backend response. No successes indicates no transfer occurred although there wasn't any specified errors provided. Added new test for the added warning case. * Fix pre-commit prerequisites --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: Carlos Downie <42552189+downiec@users.noreply.github.com>
* Update version number, add version changelog (to be updated with the changes that will be included in this update) * Updated documentation references to config variables (#706) * Updated documentation, specifically references to the configuration parameters which have changed over time, to use the latest variable names currently used in configuration. Updated requirements to use the latest readthedocs them of Sphinx * Update version number, add version changelog (to be updated with the changes that will be included in this update) * fixed the pagination so that it will reset to page one when doing a new search (#705) * update for obs4MIPs (#711) Co-authored-by: Sasha Ames <ames4@llnl.gov> * Updated the settings file to autoformat on save to hopefully reduce linting errors. Fixed linting errors for the initial_projects_data update. Fixed the status tooltip to correctly display the node's status. Updated test fixtures to accurately reflect the node status response (the node names included the https:// in front and thats why status tooltip wasn't finding them. (#714) * Added alert for the case that a collection search fails in the manage collections popup (#710) * Added alert for the case that a collection search fails in the collection management popup. Added a test to bring the coverage up to 100% for the search table component. Removed some unused code, the get endpoint function is not used. * Minor linting error fixes. * Fixed some backend tests based on the removed/unused functions * Small fix to test. * Minor update to update summary. * Made it so the globus ready column does not appear if no globus ready nodes are configured (as it was before). Changed ActiveFacets type to be more strict in order to avoid the issue of calling join on a string which was causing errors. Fixed and updated some tests related to the url to string and string to url functions in the utils file. * Fixed failing test. Updated the manage_metagrid.sh helper scripts to include commands for pre-commits, backend and frontend tests. Updated setup.cfg to allow backend tests to run correctly. Removed unneeded commands in backend README.md. Removed the autoformat setting on save. --------- Co-authored-by: Sasha Ames <ames4@llnl.gov>
* Add dark mode * Persist theme choice * Add tests for Dark mode feature * Merged in the latest changes from v1.3.0-release. Updated the styling to fix issues with the dark-mode rendering of navbar and certain colors. Added custom colors for dark and light version of status icons. Modified how colors are applied to tooltips and node status pages. I'd like to update/improve the way custom colors and styles are applied to make future updates smoother. (#4) --------- Co-authored-by: Zach Price <pricezt@ornl.gov> Co-authored-by: downiec <42552189+downiec@users.noreply.github.com>
* Add dark mode * Persist theme choice * Add tests for Dark mode feature * Merged in the latest changes from v1.3.0-release. Updated the styling to fix issues with the dark-mode rendering of navbar and certain colors. Added custom colors for dark and light version of status icons. Modified how colors are applied to tooltips and node status pages. I'd like to update/improve the way custom colors and styles are applied to make future updates smoother. * Added a new configuration option for setting the footer of the metagrid frontend so that links and text can be displayed as desired. Updated the manage_metagrid script to include more developer actions, including run migrations, and installing packages for local development. Modified styling to better fit the current footer. Will need to add option for adjusting the footer height based on the height of the footer text. Updated tests and documentation for the custom footer options. * Final updates for the v1.3.0 release. Includes updates to the joyride tutorial to add the theme switch. Updated the manage_metagrid script and created a new configHelper script that sets the proper initial configuration settings for first time deployment or developers. Added tests for the theme buttons in the app. --------- Co-authored-by: Zach Price <pricezt@ornl.gov>
Co-authored-by: Zach Price <pricezt@ornl.gov>
* Include helm chart and healthchecks * Modified the traefik.yml config to separate between local and production, in order to attempt to get production traefik deployment to function as it did in another branch. Updated docker compose files to separate keycloak into a profile and remove docs service from production. Updated manage script to utilize new compose files for deployment and testing. * Updated some docs and the manage scripts * Merged in some minor changes from traefik-deployment-fixes branch to update app css and include other minor fixes. * Delete docker-compose.prod-overlay.yml * Pre-comfig fixes, update to scripts. * Some updates to docker deployment to test if they will work in production * Separated trafik.yml into one for production and one for local, to avoid using the conditional statements that were being used which seemed to not be working for traefik deployment. Updated the deployment script to make sure local and production custm overlays are applied last so their variables aren't overriden. * Removed unnecessary prod_traefik file. * Updated prod overlay to name the containers with prod to distinguish deployment container types. Updated the config helper script to generate production overlay * Updated changelog. Added dark theme styling to the React Joyride tutorials. * Implemented some suggestions from PR comments. Brightened the tutorial background in dark theme. * Minor fix to compose file. * Reverted container names to leave them be. * Updated obs4mips url and modified the display button. Remove obsolete updateProjects.sh script and updated the README's regarding updating project data. Added the update functions to the manage_metagrid.sh scripts instead. Still need to test in production, but should work. * Fix minor lint error. Added build step to project migration function. --------- Co-authored-by: Zach Price <pricezt@ornl.gov>
…as label, to Replicas Only.
…ges are updated and how the changelog is updated. The data for messages and changes is now located in a messageData.json file, removing the need to modify the messageDisplayData.ts file in multiple locations every time a new version and changelog entry is created.
…request to add option for obtaining object using request.body, this fixes the wget download which expects data in a different format than other requests. Updated the changelog.
…ay, to make version history collapsed as it was getting long.
… to increment/create a new version. It will create a new changelog file, update the package.json and update the messageData.json file to add the new version.
…a save steps to see if that will resolve globus transfer token auth loop issue in Firefox.
…a save steps to see if that will resolve globus transfer token auth loop issue in Firefox.
…globus permissions were requested. This seems to allow the path set redirect to request the appropriate permissions for allowing transfers to occur.
…ges, fix typos in ingress template (#743) Co-authored-by: Zach Price <pricezt@ornl.gov>
…rsion script so helm chart version is automatically updated when the version number is updated.
* Fixes hash for automatic deployment rolls * Fixes container trigger and pushing images * Fixes workflow tags * Fixes default filename for configHelper.sh * Fixes frontend makefile * Adds minimal frontend-config.js * Revert "Adds minimal frontend-config.js" This reverts commit 8856c9d. * Adds helm chart testing job * Fixes working directory * Updates workflow and adds test values * Fixes building dependency * Adds readme and helmfile to test locally * Fixes NOTES rendering * Adds defaults to support no-config install * Removes test.yaml since defaults were fixed * Fixes spelling error * Fixes workflow * Updates readme
* Some minor updates to config helper and manage_metagrid scripts. * Added option to cancel endpoint path selection from globus side and it will return user to metagrid properly. * Updated to the manage script for updating the version values in helm files. Fixes some linting errors. * Added a tooltip to the download button to hint to user why it's disabled. Updated download button behavior ro make sure user can't start a Globus download without selecting a collection or start a wget download without selecting a cart item.
* Created a DataBundlePersister class which will bundle key value pairs into a single object, stringify and then save to session storage as a single bundle. This will reduce requests to session storage by consolidating into a single request when values need to be persisted or loaded. * Minor updates to changelogs for consistent punctuation. * Some minor updates to persistent class * Updated the way session storage is used by the datasetdownload component to improve globus transfer reliability. Includes extensive refactoring of tests and test components to pass using the new DataBundlePersister class. Updated the cart summary to display info on selected cart items not just on the entire cart totals. * Some minor changes to the documentation to update the frontend project structure. Moved the reset tokens button to a menu in the globus transfer button. Globus transfer and wget download are separated into their own buttons using download type to select which is displayed. Updated tests as needed and added new ones for the reset token feature. Updated react joyride tour to mention changes on the summary table. * Made some minor updates, refactors and added new tests to increase test coverage
remove extraneous and clarify
* Initial testing commit, merging in most significant changes from the previous STAC integration branches. * linting fixes. * Merged in the latest changes from v1.5.1 branch, fixed some broken code cuased by the STAC changes. Updated tests so that they pass. * Updated the tables to include file sizes and checksums for STAC results if the data is available. Removed the numResults requirement as it seems to be missing sometimes, instead counting the results features array directly. Updated the filename to utilize assets key when there is no id, title or name attribute in the asset object. * Added objects to map differences between CMIP6 facets and facets url. Created some functions to generate stac filters for and, or and equals operations. Updated search to utilize generated filters. Fixed minor issue with metadata values that weren't being displayed. Updated the tabs to utilize cmip6:further_info_url data for es-docs. * Fixed minor issue with filtering so that OR filters work properly. Updated search limit to 9999 rather than only 10. Still need to make page size work properly. Updated the query string to display the generated STAC filter string if project is a STAC project. * Removed the 'All except CMIP6' project as it seems to be broken now. Updated SearchesCards to handle the STAC Project searches. Modified STAC request url to include the project name and made it so future STAC projects can be added and will work. Adde some error checking for STAC search results so that status is returned after a search. * Modified the search page copy and save search buttons to be a menu instead, where the copy search feature will be in a drop-down. Then added 2 new copy features related to esgpull, one for search and one for download. * Added Intake ESGF search option and created a function to generate the python code to run the search. The command assumes you have intake installed and ready to use. * cleaned up some code and fixed broken tests. * Added tootltips and updated the joyride tutorials to show the search copy menu items. Updated the changelog with notes about the update. Added several tests to increase coverage and verify the copy search options give the appropriate output. * Implemented aggregations for STAC so that result counts are shown for specific facets. Added STAC icon and fixed table to display the STAC icon instead of dataset icon, for STAC datasets. Other changes. * Small fixes after v1.5.3-release merge. * Updates to esgpull search and download feature so that unnecessary facets are removed from valid list and missing ones were added. Added custom comments on esgpull commands to highlight any facets that were not applied. Other minor updates. * Adding an esgpull download options for a single dataset using the master_id attribute. * Updated postgres version to 16-alpine to remove the critical vulnerability in postgres-16. Minor fixes. * Fix some tests and increase coverage. * Removed the default filter for STAC * Removed the All (except CMIP6) project option since it appears to no longer work. Added logic to hide STAC projects if STAC URL is not configured. * Updated authentication options in the script and settings so that 'none' is now a deployment option. When none is set, there will not be a sign in button in the navigation bar. Updated keycloak deployment so that a basic keycloak instance is created which can be connected to for configuring it as an admin. The keycloak authentication doesn't work, so there are still issues with how the authentication is handled in the frontend. Updated some tests and added keycloak overlays to handle keycloak deployment while skipping it when none or globus options are selected. * Updated the cart, and saved searches to filter out STAC items when STAC is not active. Also updated the navbar to show the correct filtered count. Updated the search cards to display the STAC filter rather than the regular filter, when showing the query string. * Added several tests on various components and removed the facet route in order to improve overall test coverage. * Fixed issues with pagination and page size.
… transfer if the deployment has authentication method set to none. Minor updates to linter config settings to remove deprecated settings.
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (88.74%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## master #857 +/- ##
==========================================
- Coverage 97.75% 94.98% -2.77%
==========================================
Files 46 50 +4
Lines 1823 2952 +1129
Branches 415 654 +239
==========================================
+ Hits 1782 2804 +1022
- Misses 41 148 +107
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
* Updates backend database settings to be configurable * Allows helm testing to fail * Removes pydantic database settings, replaces with environ.db_url to parse DATABASE_URL * Updates chart to use new DATABASE_URL for backend and adds examples * Updates docker-compose to use DATABASE_URL * Moves helm chart testing to helm-chart workflow, adds workflow trigger * Updates backend workflow to use DATABASE_URL * Removes job requirements * Fixes workflow trigger * Removes pull_request trigger * Merges containers and helm-chart workflows * Removes bitnami repo * Fixes running migrate job on upgrade * Updates default search/wget urls * Adds job to test upgrade * Fixes liveness/readiness probe indentation, fixes duplication port labels
…xes for the helm deployment
* Fix some issues with precommit and startup * Fixes pre-commit workflow to run on all pull requests --------- Co-authored-by: Jason Boutte <boutte3@llnl.gov>
* Initial integration of wget functions. Settings and config still need to be resolved * Finished adding files, and fixing issues with the wget downloads, downloads are now working properly. Need to add a few tests to increase test coverage. * Fix some issues with precommit and startup * Updated the feature to allow using a dedicated wget url (as it was before) so that the integrated wget is only active if the WGET_URL setting is None (default). Fixed tests to handle changes, the WGET_URL external test is only run if the WGET_URL is set, otherwise the test is skipped. * Added STAC wget downloads, where a script is automatically generated when STAC items are selected for wget download in the cart. The wget download can handle both stac and non-stac items being downloaded. Added some tests for the newly added functions and updated the wget success messages to account for STAC and non-STAC wget script results. Still need to add wget download for individual STAC records within the search Table. * Updated the Table to handle wget downloads for STAC datasets * The project name is supposed to be lowercase according to the STAC standards, therefore checking uppercase is not necessary. * Updated backend coverage to ignore unused utility function.
* Initial integration of wget functions. Settings and config still need to be resolved * Finished adding files, and fixing issues with the wget downloads, downloads are now working properly. Need to add a few tests to increase test coverage. * Fix some issues with precommit and startup * Updated the feature to allow using a dedicated wget url (as it was before) so that the integrated wget is only active if the WGET_URL setting is None (default). Fixed tests to handle changes, the WGET_URL external test is only run if the WGET_URL is set, otherwise the test is skipped. * Added STAC wget downloads, where a script is automatically generated when STAC items are selected for wget download in the cart. The wget download can handle both stac and non-stac items being downloaded. Added some tests for the newly added functions and updated the wget success messages to account for STAC and non-STAC wget script results. Still need to add wget download for individual STAC records within the search Table. * Updated the Table to handle wget downloads for STAC datasets * The project name is supposed to be lowercase according to the STAC standards, therefore checking uppercase is not necessary. * Updated backend coverage to ignore unused utility function. * Initial update to include new METAGRID_SUPPORT_INFO optional setting that allows admin support information to be included by site admins for users to reach out.
* Found an issue with authenticated user transfer client not having the scope updated causing transfers to fail. Updated the frontend to display a maintenance/error page when the backend fails to provide settings values. * Minor update to increase test coverage.
* Updated a few packages to remove django backend warnings. Updated docker overlays to see if production deployment wil run successfully with default configuration. * Updated the configHelper to include newly added and changed settings. Added a refresh collation and reindex function for the postgres database, for cases where postgres container is updated and there is a collation version mismatch error during deployment. * Fixed a globus transfer backend test. * Modified backend test to fix issue with non-matching client id * Fix pre-commit issues.
* Made visual and functional updates to the Facets Filter component. Filename search is removed, additional properties panel is hidden for STAC search, added Keyword Search panel and removed keyword search from the top nav bar, updated the sizing of the filter option drop-downs to improve the readability of the listed items, updated the antd library to latest version of 5.x, updated the changelog to include latest changes for v1.5.4. Still need to fix the tests that were broken by the changes to the U.I. * Fixed more than half of the failing tests. Restored the deleted filename search and commented it out, in case we decide to bring it back or users miss it. * Fixed remaining tests, added 2 for the keyword search feature and skipped the tests associated with filename search. * Added more tests and updated some components to improve test suite coverage. * Updated axios package to latest, minor updates to the changelog and fixed a bug that occurs where setting the path to a newly added collection wasn't redirecting to that specific collection. * Updated joyride tutorial for main page to remove the top navbar keyword search and added it to the facets tutorial. * Increased test coverage and made the keyword search hidden for STAC projects (for the time being, until keyword search is implemented).
* Adds/enables postgresql persistence * Updates readme with v1.5.4 database upgrade instructions * Exposes wget feature * Removes old bitnami repo * Fixes appending DJANGO_ALLOWED_HOSTS * Updates README
* Added fixes to the configHelper script to allow setting the password for a postgres database. Updated the default prod overlay to include values for user and password, updated the variable names for postgres and password to match the latest version of postgres. Created a template for settings, in the case the config helper isn't used, the template can be used instead. Fixed a minor issue with metagrid manage script starting up production for globus, and added a provider to the error page render. * Fix minor pre-commit issue
* Disables node status feature by default * Fixes only creating node status resources if enabled * Updates helm chart readme with FAQ * Fixes helm chart readme toc * Fix indenting FAQ answer * Updates helm chart readme faq * Updates node status * Change node_status secret to configmap, split backend secret into configmap/secret
* Fixes wget globus public index spelling * Adds initial helm unittest for backend * Adds initial helm unittests for frontend * Adds initial helm unittests for nodestatus * Adds initial helm unittests for postgresql * Adds initial helm unittests for ingress * Fixes settings DJANGO_ALLOWED_HOSTS * Fixes allowing additional hosts for ingress TLS and django * Updates readme * Tests disabling database * Updates backend deployment checksum to include configmap * Adds documentation for testing chart. * Updates workflow to unittest helm chart * Fixs pre-commit issues * Pins the helm version installed for unittesting
* Expose tls secretName * Fixes handling multiple ingress hosts * Fixes adding config/secret to backend * Defines DJANGO_SOCIAL_AUTH_REDIRECT_IS_HTTPS when ingress tls is enabled
Collaborator
|
@downiec @sashakames Heads up, tagging non-valid semver e.g. (v1.5.4.1) will prevent the helm chart from being published. Tagging with v1.5.4-1 would be valid. |
* feat(helm): add globus client config and configurable ingress pathType - Add GLOBUS_CLI_CLIENT_ID and GLOBUS_CLI_CLIENT_SECRET to backend secret - Make ingress pathType configurable via values.yaml - Add validation for Globus credentials when wget.url is not set - Update test snapshots and test cases * docs: update README to match current values.yaml * chore(helm): remove trailing blank lines from README.md and backend secret.yaml Cleaned up unnecessary trailing blank lines to improve file formatting consistency. * fix: update snapshots * fix: required settings for install/upgrade in GH workflow * fix: convert to strings * fix: ensure values are quoted
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #291, #677, #831, #835, #840, #869
Type of change
How Has This Been Tested?
Checklist