intermittent server.Stop() error

**Describe the bug**
Occasionally, we see the following error on shutdown:

`2023-05-08T12:14:01.550-0400 [ERROR] testdirectory-default-logger: error stopping directory: %s: op="testdirectory.(Directory).Stop" err="gldap.(Server).Stop: no listener: invalid state"`

**Expected behavior**

If the listener is already closed or nil, we shouldn't try to close it.