Support for FIPS compliance mode

[beanuwave]

This PR has been abandoned in favor of a sequence of smaller, more manageable PRs. For the time being it remains open just as a refererence. See this comment for details.

Description

This PR makes FIPS mode available through the OPENSEARCH_CRYPTO_STANDARD=FIPS-140-3 environmental parameter instead of the tests.fips.enabled setting. It provides FIPS 140-3 support by replacing all BC dependencies with BCFIPS dependencies and making FIPS approved-only mode configurable at launch. Running this mode restricts the BCFIPS provider to rely solely on FIPS-certified ciphers.

• The fips.gradle build script is removed in order to support a single-build solution.
• All BC dependencies are replaced by BCFIPS.
• Fixed creation of password protected internal keystore for valuable settings at node start. The add-string command was made with additional --stdin option, which interferes with password input.
• The Password Matcher inside Identity-Shiro that relies on BC to check if hashed passwords match with OpenBSDBCrypt is replaced by the password4j implementation.
• Adds full support for the BCFKS format (*.bcfks) for key and trust stores, also making it the default.
• KeyStore instantiation was added to forbidden-apis in favour of KeyStoreFactory.
• Google's truststore is converted to the BCFKS format.
• Makes the best guess of which store type is provided based on the filename extension.
• Store types are strictly limited to JKS, PKCS12, PKCS11, and BCFKS.
• Refactors PemUtils to parse private keys in formats EC, PKCS8, PKCS1, and DSA, with or without encryption, and with or without parameters.
• dependency ':libs:opensearch-common' was added to rest-client build, to support strict keystore types. It's also the reason for JVM bump JAVA8 to JAVA11.
• allow ingest-attachment plugin to run in FIPS mode, since BC dependencies find no use.
• The java.security file is added to the build to distinguish between FIPS and non-FIPS environments.
• The fips_java.security file is altered due to evolving security standards.
• The security.policy file is altered to grant necessary security permissions.
• Increased security standards in KeyTabs and algorithms for Kerberos.
• SecureRandom gets instantiated in to different way, depending on if it runs with FIPS or not.
• Uninstalls SunJCE provider from security providers list at runtime when FIPS mode is enabled.

Runtime limitations (known so far) that come with enabling FIPS mode:

Admins can continue to manage their systems without being impacted by this change. However, for those keen on FIPS compliance, the most common obstacle will likely be the requirement to set a stronger password for the internal keystore and also convert key and truststores to *.bcfks format.

• Does not allow empty passwords for keystores or private keys (they need to be at least 112 bits in strength).
• The ssl.verification_mode=NONE setting is not permitted.
• JKS and PKCS12 key and trust store types are not supported at all.
• The internal keystore cannot be auto-migrated from versions 1 or 2.
• Azure Classic Discovery plugin -> deprecated.
• SQL-CLI client.
• HDFS plugin won't connect since it's using RC4 cipher for token authentication.

Reasons for refactoring PemUtils, which is used by the Reindex API in cases of migrating data from a remote cluster that is TLS protected:

• Lack of support for evolving standards like PKCS#8.
• Password-Based Key Derivation Functions such as PBKDF-OPENSSL are not supported in FIPS mode in favor of the PBKDF2 standard.
• Java type safety.
• It is generally a good idea to let ASN1 annotation parsing be done by external security libraries.

Related Issues

opensearch-project/security#3420

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8e8ed47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

Could use some help maybe from @cwperks or @peternied reviewing this, please.

[github-actions]

❌ Gradle check result for 1da3c64: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9a88327: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[prakashsatpathy]

Hi @beanuwave ,
We found that, this has been waiting for quite a while, is there any update on moving with this PR?
We are also looking for same FIPS compliance for OpenSearch.
And if there are any challenges, can you please share those.

[github-actions]

❌ Gradle check result for 9a88327: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[aaronmcohen]

I want to echo @prakashsatpathy #14912 (comment) above. What are the challenges and how can those of us interested in this help?

[kaimst]

I want to echo @prakashsatpathy #14912 (comment) above. What are the challenges and how can those of us interested in this help?

Thanks for offereing your support @aaronmcohen.

This PR has been abandoned in favor of a sequence of smaller, more manageable PRs. For the time being it remains open just as a refererence.

We are in the final stages of another PR that fixes various FIPS related tests. After that, one more PR which is already developed and part of this original PR will follow. Then OS core should be FIPS compliant.

[andrross]

@kaimst I updated the description and converted this to draft to hopefully avoid confusion in the future. Let me or any other maintainer know if/when it should be closed.

[cwperks]

@beanuwave thanks for working on this, I went through the large chunk of changes and there is one major issue that bothers me (or I do not understand something). The FIPS mode compliance is optional (at least at the moment) but its leaks everywhere as a mandatory prerequisite now:

• through BC fips dependencies
• through BC fips providers
• through java security policy

We have to make it truly optional, otherwise it is becoming a major breaking change (in my opinion).

@reta @terryquigleysas We've now had a few minor distributions where all BC jars have swapped to using BCFIPS versions of the jars.

Due to the optionality built in, the lib/ folder does not contain BCFIPS jars in the vanilla distribution, but does include the BCFIPS jars when building with -Pcrypto.standard=FIPS-140-3.

I'd like to revisit this because its causing problems when the jars are included in the lib/ directory in the FIPS distribution and not the vanilla distribution. There are problems with downstream plugins (Terry to elaborate) causing jar hell.

Can we include the BCFIPS jars in the lib/ folder by default?

[terryquigleysas]

@beanuwave thanks for working on this, I went through the large chunk of changes and there is one major issue that bothers me (or I do not understand something). The FIPS mode compliance is optional (at least at the moment) but its leaks everywhere as a mandatory prerequisite now:

• through BC fips dependencies
• through BC fips providers
• through java security policy

We have to make it truly optional, otherwise it is becoming a major breaking change (in my opinion).

@reta @terryquigleysas We've now had a few minor distributions where all BC jars have swapped to using BCFIPS versions of the jars.

Due to the optionality built in, the lib/ folder does not contain BCFIPS jars in the vanilla distribution, but does include the BCFIPS jars when building with -Pcrypto.standard=FIPS-140-3.

I'd like to revisit this because its causing problems when the jars are included in the lib/ directory in the FIPS distribution and not the vanilla distribution. There are problems with downstream plugins (Terry to elaborate) causing jar hell.

Can we include the BCFIPS jars in the lib/ folder by default?

Yes, I think there is a good case for shipping these in the lib dir now as the chance of introducing a breaking change has lessened, if not gone away altogether. For example, it is only the FIPS variants of the BC libs that are on the standard distribution now

./plugins/opensearch-security/bcpkix-fips-2.1.9.jar
./plugins/opensearch-security/bc-fips-2.1.2.jar
./plugins/opensearch-security/bcutil-fips-2.1.4.jar
./plugins/opensearch-flow-framework/bc-fips-2.1.2.jar
./plugins/opensearch-performance-analyzer/bcpkix-fips-2.1.9.jar
./plugins/opensearch-performance-analyzer/bc-fips-2.1.2.jar
./plugins/opensearch-ml/bc-fips-2.1.2.jar
./lib/tools/plugin-cli/bc-fips-2.1.2.jar
./lib/tools/plugin-cli/bcpg-fips-2.1.11.jar

@beanuwave is also now seeing the jar hell that ensues as he brings them into the libs dir for the work he is doing, and then uses any of the plugins listed above.

Having them only in the lib dir should also reduce the size of the distros a bit and would be preferable to the work involved in making a FIPS-only distro or asking customers to make manual changes.

[beanuwave]

Can we include the BCFIPS jars in the lib/ folder by default?

IIRC the major concern was that any third-party plugin relying on non-FIPS BC libs would cause a breaking change. @cwperks @reta @terryquigleysas

[cwperks]

Can we include the BCFIPS jars in the lib/ folder by default?

IIRC the major concern was that any third-party plugin relying on non-FIPS BC libs would cause a breaking change. @cwperks @reta @terryquigleysas

Speaking for the plugins in the default distribution, it was possible to swap the BC jars to BCFIPS jars so 3P plugins would also have the option to make a similar swap if they are affected.

I know its a pain to write code that has to check if a class is available at runtime and then use reflection to instantiate/call methods etc so advantage of having it in the lib is to have it globally available.

[beanuwave]

Every split-up PR that originated from this one has been merged, so keeping this PR in draft has served its purpose. However, that doesn’t mean we’re finished - the FIPS work is still ongoing, just in other areas.

Shoutout to everyone who has participated so far and helped keep the progress going. Thank you!
@kaimst @reta @cwperks @andrross @terryquigleysas @peternied @KarstenSchnitter