bug: invalid number of arguments in "env" directive in /usr/local/apisix/conf/nginx.conf:29

[BFergerson]

Current Behavior

I'm trying to dynamically set upstreams.tls.client_cert/client_key in stand-alone mode. I can't find a way to set the value via file lookup so I'm trying to set it via environment value. Whenever I set the value to the certificate value I get:

2022-06-09T07:06:59.109311603Z /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
2022-06-09T07:06:59.143984084Z /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
2022-06-09T07:06:59.164148394Z 2022/06/09 07:06:59 [emerg] 1#1: invalid number of arguments in "env" directive in /usr/local/apisix/conf/nginx.conf:29
2022-06-09T07:06:59.164159537Z nginx: [emerg] invalid number of arguments in "env" directive in /usr/local/apisix/conf/nginx.conf:29

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

1. Use config.yaml:

apisix:
  node_listen:
    - port: 80
  enable_admin: false
  config_center: yaml
  enable_ipv6: false

2. Use apisix.yaml:

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org:80": 1
    type: roundrobin
    scheme: grpcs
    tls:
      client_cert: ${{CCERT}}
      client_key: ${{CKEY}}
#END

3. Use Dockerfile:

FROM apache/apisix:2.14.1-alpine

ENV CCERT="-----BEGIN CERTIFICATE-----\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n-----END CERTIFICATE-----\n"
ENV CKEY="-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n-----END RSA PRIVATE KEY-----\n"

ADD ./config.yaml /usr/local/apisix/conf/config.yaml
ADD ./apisix.yaml /usr/local/apisix/conf/apisix.yaml

CCERT and CKEY are taken from

apisix/t/node/grpc-proxy-mtls.t

Lines 88 to 89 in 1b0c182

client_cert: "-----BEGIN CERTIFICATE-----\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n-----END CERTIFICATE-----\n",

client_key: "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n-----END RSA PRIVATE KEY-----\n"

Environment

• APISIX version (run apisix version): apache/apisix:2.14.1-alpine

[tokers]

@spacewander @tzssangglass Please take a look when you have time.

[tzssangglass]

I reproduced it, and the problem was with this paragraph in nginx.conf:

env APISIX_PROFILE;
env PATH; # for searching external plugin runner's binary

env CKEY=-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n-----END RSA PRIVATE KEY-----\n;
env CCERT=-----BEGIN CERTIFICATE-----\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n-----END CERTIFICATE-----\n;


http {
    # put extra_lua_path in front of the builtin path
……

I will check it.

[tzssangglass]

APISIX startup phase gets the environment variables(CCERT , CCERT ) in apisix.yaml here:

apisix/apisix/cli/ops.lua

Line 646 in 975038e

local exported_vars = file.get_exported_vars()

and the environment variable is used here when generating nginx.conf:

apisix/apisix/cli/ngx_tpl.lua

Lines 54 to 58 in 975038e

	{% if envs then %}
	{% for _, name in ipairs(envs) do %}
	env {*name*};
	{% end %}
	{% end %}

[tokers]

APISIX startup phase gets the environment variables(CCERT , CCERT ) in apisix.yaml here:

apisix/apisix/cli/ops.lua

Line 646 in 975038e

local exported_vars = file.get_exported_vars()

and the environment variable is used here when generating nginx.conf:

apisix/apisix/cli/ngx_tpl.lua

Lines 54 to 58 in 975038e

	{% if envs then %}
	{% for _, name in ipairs(envs) do %}
	env {*name*};
	{% end %}
	{% end %}

Looks like the name is empty so the env directive is incomplete.

[tzssangglass]

First I did some verification that passing a normal string variable with env works.

apisix.yaml

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org": 1
    type: roundrobin
    scheme: ${{UP_SCHEME}}
routes:
  -
    uri: /*
    upstream_id: 1
#END

config.yaml

apisix:
  node_listen:
    - port: 80
  enable_admin: false
  config_center: yaml
  enable_ipv6: false

test 1
set env like

export UP_SCHEME=http

UP_SCHEME in nginx.conf like

env UP_SCHEME=http;

curl http://127.0.0.1:80/get -i
HTTP/1.1 200 OK

test 2
set env like

export UP_SCHEME=grpc

UP_SCHEME in nginx.conf like

env UP_SCHEME=grpc;

curl http://127.0.0.1:80/get -i
HTTP/1.1 502 Bad Gateway

Passing the certificate through the environment variable failed, it should be because you passed the certificate with a newline character \n.

[BFergerson]

@tzssangglass, thanks for looking into this but are you sure it's the \n? My original code does not contain the \n. I only added them after I found the certificates embedded in APISIX using them.

Here are certificates without \n that cause the same issue:

ENV CCERT="-----BEGIN CERTIFICATE-----MIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAGA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcxDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAxMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlHdWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UEAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SCLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1CX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iqE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKnHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMgaRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e57wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjwfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs+lDqn1MobTXzDpuCFXA=-----END CERTIFICATE-----"
ENV CKEY="-----BEGIN RSA PRIVATE KEY-----MIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5NaUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0DyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+D+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqvdh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3zfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7EuiSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xBkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiEYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7CfURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpcCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8vf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhPonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhMDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjxiu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzHZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcfzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGHK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrOVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHlJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18zRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==-----END RSA PRIVATE KEY-----"

[BFergerson]

Looks like it's actually the -----BEGIN CERTIFICATE----- part.

[tzssangglass]

Passing the certificate through the environment variable failed, it should be because you passed the certificate with a newline character \n.

To verify it, I did the following tests:

set env var like

export CCERT="-----BEGIN CERTIFICATE-----\nMIIEojCCAwqgAwIBAgIJAK253pMhgCkxMA0GCSqGSIb3DQEBCwUAMFYxCzAJBgNV\nBAYTAkNOMRIwEAYDVQQIDAlHdWFuZ0RvbmcxDzANBgNVBAcMBlpodUhhaTEPMA0G\nA1UECgwGaXJlc3R5MREwDwYDVQQDDAh0ZXN0LmNvbTAgFw0xOTA2MjQyMjE4MDVa\nGA8yMTE5MDUzMTIyMTgwNVowVjELMAkGA1UEBhMCQ04xEjAQBgNVBAgMCUd1YW5n\nRG9uZzEPMA0GA1UEBwwGWmh1SGFpMQ8wDQYDVQQKDAZpcmVzdHkxETAPBgNVBAMM\nCHRlc3QuY29tMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAyCM0rqJe\ncvgnCfOw4fATotPwk5Ba0gC2YvIrO+gSbQkyxXF5jhZB3W6BkWUWR4oNFLLSqcVb\nVDPitz/Mt46Mo8amuS6zTbQetGnBARzPLtmVhJfoeLj0efMiOepOSZflj9Ob4yKR\n2bGdEFOdHPjm+4ggXU9jMKeLqdVvxll/JiVFBW5smPtW1Oc/BV5terhscJdOgmRr\nabf9xiIis9/qVYfyGn52u9452V0owUuwP7nZ01jt6iMWEGeQU6mwPENgvj1olji2\nWjdG2UwpUVp3jp3l7j1ekQ6mI0F7yI+LeHzfUwiyVt1TmtMWn1ztk6FfLRqwJWR/\nEvm95vnfS3Le4S2ky3XAgn2UnCMyej3wDN6qHR1onpRVeXhrBajbCRDRBMwaNw/1\n/3Uvza8QKK10PzQR6OcQ0xo9psMkd9j9ts/dTuo2fzaqpIfyUbPST4GdqNG9NyIh\n/B9g26/0EWcjyO7mYVkaycrtLMaXm1u9jyRmcQQI1cGrGwyXbrieNp63AgMBAAGj\ncTBvMB0GA1UdDgQWBBSZtSvV8mBwl0bpkvFtgyiOUUcbszAfBgNVHSMEGDAWgBSZ\ntSvV8mBwl0bpkvFtgyiOUUcbszAMBgNVHRMEBTADAQH/MB8GA1UdEQQYMBaCCHRl\nc3QuY29tggoqLnRlc3QuY29tMA0GCSqGSIb3DQEBCwUAA4IBgQAHGEul/x7ViVgC\ntC8CbXEslYEkj1XVr2Y4hXZXAXKd3W7V3TC8rqWWBbr6L/tsSVFt126V5WyRmOaY\n1A5pju8VhnkhYxYfZALQxJN2tZPFVeME9iGJ9BE1wPtpMgITX8Rt9kbNlENfAgOl\nPYzrUZN1YUQjX+X8t8/1VkSmyZysr6ngJ46/M8F16gfYXc9zFj846Z9VST0zCKob\nrJs3GtHOkS9zGGldqKKCj+Awl0jvTstI4qtS1ED92tcnJh5j/SSXCAB5FgnpKZWy\nhme45nBQj86rJ8FhN+/aQ9H9/2Ib6Q4wbpaIvf4lQdLUEcWAeZGW6Rk0JURwEog1\n7/mMgkapDglgeFx9f/XztSTrkHTaX4Obr+nYrZ2V4KOB4llZnK5GeNjDrOOJDk2y\nIJFgBOZJWyS93dQfuKEj42hA79MuX64lMSCVQSjX+ipR289GQZqFrIhiJxLyA+Ve\nU/OOcSRr39Kuis/JJ+DkgHYa/PWHZhnJQBxcqXXk1bJGw9BNbhM=\n-----END CERTIFICATE-----\n"
export CKEY="HrMHUvE9Esvn7GnZ+vAynaIg/8wlB3r0zm0htmnwofYLp1VhtLeU1EmMJkPLUkcn2+v6Uav9bOQMkPdSpUMcEpRplLSXs+miu+B07CCUnsMrXkfQawRMIoePJZSLH5+PfDAlWIK2Q+ruYnjtnpNziiAtXf/HRRwHHMelnfedXqD8kn3Toe46ZYyBir99o/r/do5ludez5oY7qhOgNSWKCfnZE8Ip82g7t7n7jsAf5tTdRulUGBQ4ITV2zM3cxpD0PWnWMbOfygZIDxR8QU9wj8ihuFL1s1NM8PplcKbUxC4QlrSN+ZNkr6mxy+akPmXlABwcFIiSK7c/xvU1NjoILnhPpL6aRpbhmQX/a1XUCl+2INlQ5QbXbTN+JmDBhrU9NiYecRJMfmA1N/lhwgt01tUnxMoAhfpUVgEbZNalCJt+wn8TC+Xp3DZ0bCpXrfzqsprGKan9qC3mCN03jj50JyGFL+xt8wX8D0uaIsu4cVk4et7kbTIj9rvucsh0cfKn8va8/cdjw5QhFSRBkW5Vuz9NwvzVQ6DHWs1a8VZbN/hERxcbWNk/p1VgGLHioqZZTOd5CYdN4dGjnksjXa0Z77mTSoNx3U79FQPAgUMEA1phnO/jdryM3g5M+UvESXA/75we435xg5tLRDvNwJw2NlosQsGY7fzUi2+HFo436htydRFv8ChHezs2v99mjfCUijrWYoeJ5OB2+KO9XiOIz7gpqhTef9atajSYRhxhcwdCVupC1PrPGn9MzhdQLeqQCJj3kyazPfO3xPkNpMAqd2lXnLR4HGd9SBHe75Sik3jW9W1sUqrn2fDjyWd0jz57pl4qyHjbzjd3uE5qbH/QuYZBIzI9tEn7tj12brWrwHsMt+/4M7zp8Opsia64V3Y7ICLIi7fiYfr70RujXyn8Ik5TB1QC98JrnDjgQlTPDhHLk1r8XhZXqIIg6DmaN7UUjIuZhKxARTs8b5WMPvVV4GownlPN28sHIMAX84BNbP0597Fxipwp2oTMFKTzvxm+QUtbWvIPzF3n25L4sPCyUx5PRIRCJ5kDNQfhiN6o3Y/fAY0PyxI06PWYoNvSn3uO24XNXbF3RkpwKtV8n/iNo5dyM1VqFPWDuKRSLHY7E4lQTdqx4/n+rrnoH6SlmQ0zwxwxBeAz/TvkmiW7WLe3C5cUDKF9yYwvAe8ek4oTR3GxaiDWjNFsu7DUoDjpH5f3IxrX2IN4FyzE47hMeg4muPov7h74WwosqgnfmwoAEFV4+ldmzpdSjghZoF2M9EZI24Xa9rVdd6j2t6IjX20oL+SLQL/9HppMi1nC+3Zby1WOvuTR4g8K1QP75OeY4xTD1iEAXpd0WOX7C3ndceVF4THLCI4Imcf9FH9MBrE55FPMEsAk54HiAoyMd6tgqv/akRqmuAmnSsrWALhqiCnAVh2uzk644gSzmsFbh7zF33qrcafPpU4PxUEvpqbLz7asoNUDf4YB4gCcgZx30eK/w9FpMaLveiNq77EW7qcvJQPcjZ4uLaKkQVODJsd+1CbZF6370aiLxouXLFT3eQI7Ovu6be8D3MmazRPgCV36qzMwONqrXE/JbMFMKe5l1e4Y6avMejrj43BMgGo2u8LimCWkBeNwqIjH7plwbpDKo4OKZVbrzSZ0hplUDd/jMrb6Ulbc04uMeEigehrhSsZ0ZwoDiZcf/fDIclaTGNMl40N2wBiqdnw9uKTqD1YxzqDQ7vgiXG55ae31lvevPTgk/lLvpwzlyitjGs+6LJPu/wSCKA2VIyhJfK+8EnItEKjBUrXdOklBdOmTpUpdQ+zfd2NCrFRDJZKl26Uh412adFEkqY37O/0FbSCpAIsUCvaItcqK7qh5Rq26hVR0nS1MRs+MjGBzGqudXPQZHy+Yp7AlAa5UgJUaAwn2b/id6kNdv6hNWqSzHvOAVKdgC9/j0yN1VJD92+IoJTTiXsMQELcgm1Ehj2GZpTHu+GPuaOovHBnZMq/Kg4nUS+ig86X01jV28uGGtglERf1HqVQpdZwbrXtUqH0cbjlvUwQ1j7zp9yhs+0ta87v0I+elAZhXzqvehMiLJu2o9/k2+4dPvkEscduHOU6jZqe8ndNEMQWiaZEYJKxNWPTaQ6nZSlFTsT7GlENeJlFzlw8QkyRJPMBWkXuaymQUcu43Pm+gAjinHSAGUeaSaIdL2Yb0M88qNwG+UlNEslx/J37pA1oMJyxb7XOeySxkP7dXi5JvygLIfkEA3ENC4NHU9nsUvTvp5AZidZCxxtYCNYfjY6xyrlfnE+V+us31LA9Wc/tKa4y3Ldj30IT2sssUrdZ0l7UbwfcZT42ZeJpxDofpZ2rjgswTs0Upr72VuOCzjpKa1CJwxhVVtPVJJovcXp4bsNPJers+yIYfTl1aqaf4qSzU5OL/cze2e6qAh7622zEa/q6klpUx9b1f8YGlQhjQcy3++JnwwsHR71Ofh9woXq57LDCHFA6f95zdkadDDhwgRcvWVnbA2Szps8iJv7h2m25qZPFtN6puJj3RlmT6hnfBeYCjpfy/2TxyCqm6bG3HZxGuhzWs2ZGxzsjBJ3ueO1pAOjtDhkRqzoWt/v2o367IYP7iTcp4pi+qJHIWCN1ElDI0BVoZ+Xq9iLfKmjrjcxQ7EYGHfQDE52QaCQ3nMB7oiqncZ1Q5n/ICDHha9RkPP9V9vWiJIZwgOJtPfGzsGQ9AigH6po65IJyxmY5upuhg7DTmsLQnKC/fwjkBF9So/4cdZuqDbxGrDDOgpL7uvWXANRNMrqYoMFUG7M90QJHj7NgSL+B6mSNwa9ctTua7Estkoyvavda3Bl3qHQ0Hva5gjSg6elL6PQ4ksqhESvjztuy58qk9aZHsQB8ZKRu8VSay40a/3ueX6bnd0hwsYy42aWJR1z+uie3yTWPuG2JZ7DjkgDduWdC+cxfvTVTG58E5luafy5j/t85UVoB2nr46VHlt/vg4M9G8/4F0d0Y6ThI4/XTfg6l1vq5ouzhQxd+SRwnuXieZy+4/2XKJnrV6t+JbNAvwdGR1V9VPLlnb+IqpvOCYyL1YLYSlNubb9HU0wxVPppGSpJLmi+njQzl71PBgMm6QV9j889wPUo387fRbJjXbSSVLon61xk/4dNvjsgfv9rF+/qEML0q4tXBJVOJ1iwKjn84Nk6vdHM3Hu8knp0hYFa4AECYKInSTVXajWAKFx4SOq8G8MA/0YlIN872LBjUm2GKs17wsJuWID+mSyVE5pV5gQ+r92YvPcC+yIvB8hTTaRclAP/KyJesDTA=="

this cert and key from: https://github.com/apache/apisix/blob/master/t/cli/test_upstream_mtls.sh#L57-L74

it is the right configuration, and it is important to note that

1. CCERT has line break
2. CKEY has no line break

config.yaml

apisix:
  node_listen:
    - port: 80
  enable_admin: false
  config_center: yaml
  enable_ipv6: false

test 1

apisix.yaml only use CKEY

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org": 1
    type: roundrobin
    scheme: http
    tls:
      client_key: ${{CKEY}}
routes:
  -
    uri: /*
    upstream_id: 1
#END

Can be started normally, no errors are reported in the configuration.

test 2

apisix.yaml use CKEY and CCERT

upstreams:
  -
    id: 1
    nodes:
      "httpbin.org": 1
    type: roundrobin
    scheme: http
    tls:
      client_cert: ${{CCERT}}
      client_key: ${{CKEY}}
routes:
  -
    uri: /*
    upstream_id: 1
#END

invalid number of arguments in "env" directive

This can prove to be related to line breaks.

[tzssangglass]

I only added them after I found the certificates embedded in APISIX using them.

Here are certificates without \n that cause the same issue:

I will try to find the available configuration forms.

[BFergerson]

@tzssangglass, the following gets pass the error I reported:

ENV CCERT="\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH/ifZikSnRa3/E2B4ohVWRwjo/IxyDEWomgR4tLk1pSJhP/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl/7Zkjf4P7mJkJHtM\n/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B/u2LVbn4l0adamqQGepOAjD/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk/nL2Sg5UB84OoJe5XPgNBvVMKN0c/NAPKVIPninvUcG/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV/INL0sxj/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n\n"
ENV CKEY="\nMIIEpAIBAAKCAQEAnyiPLohB/4n2YpEp0Wt/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax/DER5b6//UvpfkSoGy/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS/Eo/OXKYFLR/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7/v6l7AbybcVYEXLTNJtrGEEH6/aOL8V9ogwwZuIvb/TEidCkfcf\nzg/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm/NjjvslL8F69p/I3cDJ2/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl/VGmc/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n\n"

I just removed -----BEGIN CERTIFICATE-----, -----END CERTIFICATE-----, -----BEGIN RSA PRIVATE KEY-----, -----END RSA PRIVATE KEY-----.

Now it complains:

2022-06-10T06:02:27.632844455Z 2022/06/10 06:02:27 [error] 56#56: *14 [lua] config_yaml.lua:222: failed to check item data of [upstreams] err:failed to parse cert: PEM_read_bio_X509_AUX() failed ,val: {"hash_on":"vars","scheme":"grpcs","id":1,"pass_host":"pass","type":"roundrobin","nodes":{"httpbin.org:80":1},"tls":{"client_key":"\nMIIEpAIBAAKCAQEAnyiPLohB\/4n2YpEp0Wt\/xNgeKIVVkcI6PyMcgxFqJoEeLS5N\naUiYT\/+Egi1rjEExVk20qrdSBWMuGW1Uhx8hqDT5iRqx0SQhhPrIKX859Bo1eJV0\nDyQ9JOhkdQl9jKNU1nRakifx3OQEMHEBZz36CwMLAcuMqq5Zn\/emQFMrZf+2ZI3+\nD+5iZCR7TP1oEkdKRoY2UwBkX+QyVqDudZIt4tySPteZjoZp5wqZ5LJxlT35fdqv\ndh8\/+L94qhPlE4T2EaXBm53XczLz2gl4mr4PAf7ti1W5+JdGnWpqkBnqTgIw\/9ca\n2jlT9lttltD87Q1yeKy3vnGqTUuKBrklH5CBDQIDAQABAoIBAHDe5bPdQ9jCcW3z\nfpGax\/DER5b6\/\/UvpfkSoGy\/E+Wcmdb2yEVLC2FoVwOuzF+Z+DA5SU\/sVAmoDZBQ\nvapZxJeygejeeo5ULkVNSFhNdr8LOzJ54uW+EHK1MFDj2xq61jaEK5sNIvRA7Eui\nSJl8FXBrxwmN3gNJRBwzF770fImHUfZt0YU3rWKw5Qin7QnlUzW2KPUltnSEq\/xB\nkIzyWpuj7iAm9wTjH9Vy06sWCmxj1lzTTXlanjPb1jOTaOhbQMpyaAzRgQN8PZiE\nYKCarzVj7BJr7\/vZYpnQtQDY12UL5n33BEqMP0VNHVqv+ZO3bktfvlwBru5ZJ7Cf\nURLsSc0CgYEAyz7FzV7cZYgjfUFD67MIS1HtVk7SX0UiYCsrGy8zA19tkhe3XVpc\nCZSwkjzjdEk0zEwiNAtawrDlR1m2kverbhhCHqXUOHwEpujMBjeJCNUVEh3OABr8\nvf2WJ6D1IRh8FA5CYLZP7aZ41fcxAnvIPAEThemLQL3C4H5H5NG2WFsCgYEAyHhP\nonpS\/Eo\/OXKYFLR\/mvjizRVSomz1lVVL+GWMUYQsmgsPyBJgyAOX3Pqt9catgxhM\nDbEr7EWTxth3YeVzamiJPNVK0HvCax9gQ0KkOmtbrfN54zBHOJ+ieYhsieZLMgjx\niu7Ieo6LDGV39HkvekzutZpypiCpKlMaFlCFiLcCgYEAmAgRsEj4Nh665VPvuZzH\nZIgZMAlwBgHR7\/v6l7AbybcVYEXLTNJtrGEEH6\/aOL8V9ogwwZuIvb\/TEidCkfcf\nzg\/pTcGf2My0MiJLk47xO6EgzNdso9mMG5ZYPraBBsuo7NupvWxCp7NyCiOJDqGH\nK5NmhjInjzsjTghIQRq5+qcCgYEAxnm\/NjjvslL8F69p\/I3cDJ2\/RpaG0sMXvbrO\nVWaMryQyWGz9OfNgGIbeMu2Jj90dar6ChcfUmb8lGOi2AZl\/VGmc\/jqaMKFnElHl\nJ5JyMFicUzPMiG8DBH+gB71W4Iy+BBKwugHBQP2hkytewQ++PtKuP+RjADEz6vCN\n0mv0WS8CgYBnbMRP8wIOLJPRMw\/iL9BdMf606X4xbmNn9HWVp2mH9D3D51kDFvls\n7y2vEaYkFv3XoYgVN9ZHDUbM\/YTUozKjcAcvz0syLQb8wRwKeo+XSmo09+360r18\nzRugoE7bPl39WdGWaW3td0qf1r9z3sE2iWUTJPRQ3DYpsLOYIgyKmw==\n\n","client_cert":"\nMIIDOjCCAiICAwD6zzANBgkqhkiG9w0BAQsFADBnMQswCQYDVQQGEwJjbjESMBAG\nA1UECAwJR3VhbmdEb25nMQ8wDQYDVQQHDAZaaHVIYWkxDTALBgNVBAoMBGFwaTcx\nDDAKBgNVBAsMA29wczEWMBQGA1UEAwwNY2EuYXBpc2l4LmRldjAeFw0yMDA2MjAx\nMzE1MDBaFw0zMDA3MDgxMzE1MDBaMF0xCzAJBgNVBAYTAmNuMRIwEAYDVQQIDAlH\ndWFuZ0RvbmcxDTALBgNVBAoMBGFwaTcxDzANBgNVBAcMBlpodUhhaTEaMBgGA1UE\nAwwRY2xpZW50LmFwaXNpeC5kZXYwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQCfKI8uiEH\/ifZikSnRa3\/E2B4ohVWRwjo\/IxyDEWomgR4tLk1pSJhP\/4SC\nLWuMQTFWTbSqt1IFYy4ZbVSHHyGoNPmJGrHRJCGE+sgpfzn0GjV4lXQPJD0k6GR1\nCX2Mo1TWdFqSJ\/Hc5AQwcQFnPfoLAwsBy4yqrlmf96ZAUytl\/7Zkjf4P7mJkJHtM\n\/WgSR0pGhjZTAGRf5DJWoO51ki3i3JI+15mOhmnnCpnksnGVPfl92q92Hz\/4v3iq\nE+UThPYRpcGbnddzMvPaCXiavg8B\/u2LVbn4l0adamqQGepOAjD\/1xraOVP2W22W\n0PztDXJ4rLe+capNS4oGuSUfkIENAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAHKn\nHxUhuk\/nL2Sg5UB84OoJe5XPgNBvVMKN0c\/NAPKVIPninvUcG\/mHeKexPzE0sMga\nRNos75N2199EXydqUcsJ8jL0cNtQ2k5JQXXg0ntNC4tuCgIKAOnO879y5hSG36e5\n7wmAoVKnabgjej09zG1kkXvAmpgqoxeVCu7h7fK+AurLbsGCTaHoA5pG1tcHDxJQ\nfpVcbBfwQDSBW3SQjiRqX453\/01nw6kbOeLKYraJysaG8ZU2K8+WpW6JDubciHjw\nfQnpU2U16XKivhxeuKYrV\/INL0sxj\/fZraNYErvJWzh5llvIdNLmeSPmvb50JUIs\n+lDqn1MobTXzDpuCFXA=\n\n"}}, context: init_worker_by_lua*

[tzssangglass]

Now it complains:

Yes, I know, it's expected exception (because of formatting issues) and I'm looking for available ways to configure it, let me think about it.

[tzssangglass]

hi @BFergerson , verified and currently unable to do so. The reasons are as follows

1. if CCERT has -----BEGIN CERTIFICATE----- then it would be env CCERT=-----BEGIN CERTIFICATE----- in nginx.conf, this format is considered incorrect by nginx.
2. if CCERT doesn't have -----BEGIN CERTIFICATE-----, this format is considered incorrect by https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/ssl.md#parse_pem_cert.

it looks like that you can only set cert and key in apisix.yaml

[BFergerson]

@tzssangglass, what do you think the level of effort would be to support being able to set these values via files? It's been brought up before #5562.

I think it would be a useful feature and I'm willing to give it a shot. Right now it looks like you have to either hardcode certificates or use an external scripting solution.

Any direction you could offer would be helpful. My plan would be to add the config keys:

• upstreams[].tls.client_cert_file
• upstreams[].tls.client_key_file
• ssl[].cert_file
• ssl[].key_file

[tzssangglass]

I think it would be a useful feature and I'm willing to give it a shot. Right now it looks like you have to either hardcode certificates or use an external scripting solution.

I tried to implement it with some hacking, but didn't find a usable way. Do you have any ideas? cc @spacewander @tokers @membphis

Maybe we can add functionality to support reading cert and key in standalone mode.

[tokers]

We are adding a way to fetch TLS bundles in Upstream objects through the SSL objects. I think adjusting SSL object will be more appropriate.