Use Amass for (sub)domain collection.

[cedws]

Most domains currently in the repository were collected by hand from logs. I'd like to use OWASP's Amass to search a list of root domains and collect a large list of subdomains.

I've already got something together for this on a branch, but I need to run it and give it a test. There's a huge number of domains that Apple owns that will probably take days or even weeks to find. I'll be running this on an AWS instance for now to see if it ever finishes the search.

Three things come to mind that could go wrong:

• The instance crashes and the entire process will have to be redone
• The instance uses massive amounts of bandwidth and bankrupts me
• The configured DNS server (1.1.1.1) blacklists me

[cedws]

Update on this: the AWS instance ran out of memory shortly after starting the Amass job. This seems to be a problem with Amass rather than a matter of limited memory.