update: rename bcp47 to lang

[leondz]

bcp47 is a type; it describes format not function

renamed to lang or lang_spec for more descriptive, precise, intuitive code

learnings & positions taken here:

• Probes are monolingual. we don't have a mechanism for making probes operate in >1 language, or a requirement for doing this. Thus for probes, bcp47 -> lang.
• Detectors are optionally multilingual. we already have detectors implementing this, and it's intuitive that content returned by an llm can be in more than one language, and that detectors support >1 language - this is zero extra lift. thus for detectors, bcp47 -> lang_spec
• Attempt language semantics are unclear. Attempts should be in one language, especially after unanimous decisions made during implementation of multilingual, but attempt bcp47 is occasionally populated from detector or probe bcp47. This PR takes the position that attempts are monolingual. This will be unravelled precisely when Turn+Conversation lands Migrate string output/input to Turn objects #1089 , but it's something to watch for. there are a few assignments left, e.g. in detectors.base.Detector.detect, that violate this.
• "xx,*" is not a valid lang spec, it's equivalent to "*"
• we will follow IANA BCP47 strictly
• langcodes provides some normalisation functions that might be useful in the language code format mapping

Resolves #1139

[leondz]

NB this added 1-3 dependencies for iirc ~7MB. Wanted to just query the IANA reference data file and do the checks manually but this was slightly too much work. I will do penance in removing other dependencies later to offset this.

[jmartin-tech]

This looks good to me.

One item I think needs a little documentation is that detectors changed bcp47 -> lang_spec while probes moved to lang. I believe I understand the reasoning for this however it would be helpful to have some documentation to reference for reasoning.

[jmartin-tech]

Note in current usage lang passed here is now the detector lang_spec. I don't think we need to change anything for this PR however in a future iteration the callers will need to ensure to only pass a single lang from the spec list.

[leondz]

yeah, this is the phenomenon described in bullet 3

[leondz]

This suggests detector lang_spec should really be a set later, to make these kinds of comparisons - or that language compatibility gets factored out to its own method as suggested during multilingual review #943. Will log issue for post-Turn #1089.

[leondz]

reasoning present in:

• PR

expected behavior expressed in:

• code comments
• variable names
• test logic
• test comments
• test failure msgs

reasoning added to:

• Probe rst docs
• Detector rst docs