IPA_SERVER_IP not respected in replica setup, causing failures when containers are not on the same docker network

[tkent]

The IPA_SERVER_IP environment variable is not respected when used with ipa-replica-install. This causes a range of failure behaviors when adding a replica in domain-level 1 environments.

Steps to reproduce

There are a variety of failure behaviors that show up. I ran into a lot of them chasing this down :-). I've included two easy-to-test scenarios below.

For all scenarios, put the master and replica containers on different hosts or VMs. That way they are not on the same docker network. Obviously, the docker networks shouldn't be connected via swarm

Scenario: Master has no knowledge of replica, replica tries to join master - per the docs.

Test

On the master

• Just have a vanilla master container setup. Nothing special.

On the replica

• Configure replica options that you would expect to in a non-docker environment. For example:

--principal=admin
--admin-password={{ password }}
--domain={{ domain }}
--server={{ freepia_master_server_fqdn }}
--setup-ca
--setup-dns
--no-host-dns
--no-reverse
--forwarder=8.8.8.8
--forwarder=8.8.4.4
--no-ntp
--no-ssh
--no-sshd
--unattended

Outcome

• The client configuration will step will create a DNS entry for the hostname associated with the container. That entry will have an A record with the docker container's IP address.
• The replica configuration process will hang indefinitely at the conncheck step

Note, repeating this exact test with the --skip-conncheck option produces a LDAP authentication error after client setup. This is a particularly confusing error and sent us down the wrong path for a while.

Scenario: Master has a pre-existing A record for the replica, replica skips conncheck

Test

On the master

• Setup an A record for the replica's hostname, using the IPA_SERVER_IP

On the replica

• Use installation options that include --skip-conncheck, but not --no-reverse or --no-host-dns entries.

Outcome

• A warning about being unable to have a reverse IP entry will be shown. The reverse entry was for the docker container's address.
• The existing DNS entry in the master will be updated with an additional A record using the docker container's address.
• An error will be displayed at the end of the configuration in the form of

incorrect section name: <IP_ADDRESS> 

(This is the error referenced in #92 that couldn't be reproduced.)

Workaround

To work around the issue requires some prep work on the master:

1. Create a DNS entry matching the replica's hostname on the master, with an A record for the IPA_SERVER_IP.
2. (Optional, but recommended) Create a reverse DNS entry for the IPA_SERVER_IP and point it back to the DNS entry in 1.
3. Add the --skip-conncheck ipa-replica-install option. If using --setup-dns, also add the --no-host-dns, and --no-reverse options.

[adelton]

Reading Scenario: Master has a pre-existing A record for the replica, replica skips conncheck, I believe I've found the source of misunderstanding:

On the master

    Setup an A record for the replica's hostname, using the IPA_SERVER_IP

The IPA_SERVER_IP is used by the server (master or replica) to set its own IP address in the DNS, for its own A record. It is not used to set A record for other machines like the (future) replica. The goal is primarily to be able to pass IP address of the host on which the container runs into the container as the container does not have a way to find out the (public) IP address of the host (unless setups like --net=host are used).

[adelton]

The workarounds listed are actually the correct approach and boil down to: make sure you have correct networking and DNS records.

Overall, I don't think there is any issue to somehow fix in code.

[tkent]

I might not have been clear on the issue. The IPA_SERVER_IP value set in the replica's container is not used to set the IP address in the DNS during replica setup and this causes the problem.

Having the replica setup code use the IPA_SERVER_IP value set in the container for it's address in DNS would fix this issue.

[adelton]

It is not set before the replica setup starts, that true. It's setup after the new replica was created.

The problem with setting it before starting ipa-replica-install is, at that point the container is not IPA-enrolled so it is not authorized to make the DNS record changes. We've tried hard to make the replica setup process a "single command" operation, if only to simplify the parameter passing to the container.

Having said that -- ipa-replica-install has --ip-address option. Wouldn't that do the right thing in this situation? And if it would not, would it make sense to file it as a generic FreeIPA request for enhancement?

[tkent]

I should have mentioned this in my bug report, but using the --ip-address option in ipa-replica-install results in the following error:

--ip-address: invalid IP address X.X.X.X: no network interface matches the IP address and netmask X.X.X.X

This is because the docker container only see's it's docker interface. This particular error was obviously on purpose to help sysadmin's who enter an unusable address. However, now there is a use case this error check prevents.

Changing this in the main FreeIPA source would probably be best. I'm just not 100% sure that will result in solving this issue, but it's certainly a good place to start. I'll file an issue in freeipa/freeipa and reference this one.

Between now and when the --ip-address works in this scenario, a documentation blurb about the issue would probably save folks a bit of time. I know it took me a while to chase down. I'll see about sending a pull request for that if you'd like.

Thanks!

[tkent]

Just to close the loop, the issue has been filed with freeipa, under ticket 6879

[adelton]

Hmm, looking at https://bugzilla.redhat.com/show_bug.cgi?id=1377973, I wonder if we can do something like https://bugzilla.redhat.com/show_bug.cgi?id=1377973#c8 in the ipa-replica-install code as well, for containers?

[tkent]

If I read the suggestion in comment 8 correctly, it's to allow the loopback address to be specified as the --ip-address value in the ipa-server-install script. If that's the case, I wouldn't want to go down that path. While that might allow the installation script to complete correctly, I assume any DNS entries created during the installation would point to the loopback which would cause problems.

Instead, I would want the ability to skip the checks that ensure the IP address specified using --ip-address is assigned to any interface on the host. I assume an environment variable such as "CONTAINER_INSTALLATION=1" could be used to control factors like that.

Did I follow the suggestion correctly, or miss the boat?

[adelton]

allow the loopback address

Not the loopback address. Address which cannot be found on any of the local interfaces.

[tkent]

Ah, sorry, I misread the example. I see now, 127.0.0.2 - I missed the 2.

In that case, it seems like comment 8 is the same thing I would want to see. That is, the ability to specify an address for the replica that is not bound on any local interfaces. Between docker containers and cloud-based 1-1 NAT, the situation where your advertised IP is not bound to a local interface is fairly common these days.

[adelton]

@tkent, with https://bugzilla.redhat.com/show_bug.cgi?id=1377973 CLOSED/ERRATA, have things actually improved in containers as well?

[tkent]

@adelton The situation is changed now, but not really improved.

If you specify both --ip-address=X.X.X.X and --skip-conncheck during the replica setup, you get past the errors I mentioned before. However, the replica setup still fails because it can't connect to itself during the "setting up initial replication" step.

For what it's worth, this issue is no longer relevant to us. We stopped using the DNS component of the FreeIPA containers and just manually manage our SRV records. I'd be quite happy to just see it closed since I don't think it impacts many people.

[adelton]

Thanks for the info. I'll close the issue now but if you still have the setup around to investigate, it's be good to get to the bottom of this. What IP address does the replica try to connect to when it fails during that "setting up initial replication" step? Would pre-creating A record of the replica, pointing to its host's IP address, help?