Moving formatting into a delegate

[hgrecco]

The purpose of this issue is to have a central place to discuss everything related to formatting in 0.24

In the next version of Pint, all formatting operations are going to be moved into a Delegate. As expressed in #1895"

string formatting is becoming more and more complex as users request different (reasonable) means to represent quantities (both magnitude and units). Right now, formatting is intertwined with the registry. Adding more configuration settings increases the registry API surface and makes it harder to maintain and upgrade. The plan is to create a Formatter Delegate. All formatting settings will work directly on the delegate (for a yet unspecified number of versions, we will map this to existing registry functions for backward compatibility.). Also, users will be able to build their own formatter and replace the existing one.

Part of this work has been merged into master, and the rest is being tested in develop.

A few important things:

1. Pint version 0.24 will be backwards compatible in relation to formatting.
2. Old expected behavior should not change.
3. New behavior might be introduced. e.g. Implement sorting units by dimension order #1864
4. Clear bugs can be fixed. e.g. format_babel doesn't use comma seperator from babel locale #1849, Format doesn't respect width #1798
5. Existing functions/properties have been rewriting to point to the new API (and marked deprecated).

Please add other issues, or PRs

As of 9e0789c

1. All test are passing (except doc building).
2. Docs are not building. (maybe due to 3)
3. The separate_format_defaults is not implemented.

Please provide your insights or comments about the API and overall organization.

[hgrecco]

@keewis I would really like your feedback on this, particularly into the separate_format_defaults.

[keewis]

I think I need to understand your plans a bit better to be able to comment on them. In particular, I'd like to understand how you imagine the delegates to be used (the implementation I can slowly try to understand once that's done), and how this impacts the use of the register_unit_format decorator. (As an aside, I like to think that starting with the way it is used helps a lot with API design).

I'd also recommend thinking about the different ways to customize the format string now: this would make it much easier to make room for features like the "custom modifiers" I had a PR for (which unfortunately is stale now).

The separate_format_defaults was a way to gracefully transition from one way of interpreting the spec to a different one, and as with any deprecation it would be possible to remove it at some point. So I wouldn't worry about that for now, we can put it back later if we think it would break too many people's code.

[hgrecco]

My idea is to start pushing composition over inheritance as much as I can in Pint. (I also would like to discuss the idea of adopting protocols, but that is another discussion!)

For the end users, there would not be much of a change. In the next versions, no change at all. Later, certain configurations are going to be within ureg.formatter (i.e. ureg.set_fmt_locale -> ureg.formatter.set_locale).

For the advanced users, it would allow them to swap the formatter for their own by anything that implements format_quantity and format_unit (and maybe format_measurement and maybe format_magnitude). See here This allows them to implement the desired format in an isolated way.

class MyFormatter:
    """Here goes the code
    """ 

ureg.formatter = MyFormatter()

For pint developers, it allows them to better organize the code but having all formatting things in a single place. Also, it could simplify testing because we could in certain tests replace the more complex formatter by something simpler.

We can make this work with register_unit_format by hooking this decorator with the default Formatter object.

[hgrecco]

By the way @MichaelTiemannOSC was also trying to implement certain formatting improvements that I think should be easier to implement with this delegate.

[keewis]

If I understand correctly, then, you're planning to move (and refactor) the code that is currently in the registry to the default formatter object. With the effect that instead of hard-coding the format parsing and the call of the appropriate formatting function, you'd even be able to customize those?

[hgrecco]

If I understand correctly, then, you're planning to move (and refactor) the code that is currently in the registry to the default formatter object.

Exactly, this is partially done in develop. The Formatting facet is gone, and the formatter delegate is in place.

If you desire a specific output format, you can simply exchange the formatter object easily by a custom without creating a new formatting string.

If you want to create a new spec, you still can.

[hgrecco]

I was testing a few ideas. A few comments:

1. The idea of the formatter is quite nice. I allow you to a lot of things wihout hacking pint source code. Example:

import pint

from pint.delegates.formatter.base_formatter import BaseFormatter

class NewFormatter(BaseFormatter):

    def format_quantity(self, quantity, spec: str = "") -> str:
          return "Magnitude: {}\nUnits: {}".format(quantity.m, quantity.u)

ureg = pint.UnitRegistry()
ureg.formatter = NewFormatter()

print(str(3 * ureg.meter))

returns

Magnitude: 3
Units: meter

2. Some units formatters modify not only the units but also the magnitudes. For example, pretty printing will change * by ×.
3. format_magnitude is a nice addition.

[MichaelTiemannOSC]

The failure of the docs build is not due to #1864. I think it is due to things changing in Sphinx that break various pins within pint.

The changes to allow sorting by dimension order can easily be overriden in child classes as well (dim_order now being just an instance variable of BaseFormatter).

[hgrecco]

This is just to show some of the progress made with the new formatter. The new organization not only is cleaner, but some things are fixed automatically. For example:

Old version

 	 en_US 	 3.14 meters²
 	 es_ES 	 3.14 metros²
C 	 en_US 	 3.14 meters²
C 	 es_ES 	 3.14 metros²
P 	 en_US 	 3.14 meters²
P 	 es_ES 	 3.14 metros²
L 	 en_US 	 3.14 \mathrm{meter}^{2}
L 	 es_ES 	 3.14 \mathrm{meter}^{2}
Lx 	 en_US 	 3.14 \si[]{\meter\squared}
Lx 	 es_ES 	 3.14 \si[]{\meter\squared}

New version

 	 en_US 	 3.14 meters ** 2
 	 es_ES 	 3,14 metros ** 2
C 	 en_US 	 3.14 meters**2
C 	 es_ES 	 3,14 metros**2
P 	 en_US 	 3.14 meters²
P 	 es_ES 	 3,14 metros²
L 	 en_US 	 3.14\ \mathrm{meters}^{2}
L 	 es_ES 	 3,14\ \mathrm{metros}^{2}
Lx 	 en_US 	 3.14\ \si[]{\meters\squared}
Lx 	 es_ES 	 3,14\ \si[]{\metros\squared}

1. Notice how L and Lx now work properly in relation with unit names (they were not handled well before).
2. Notice also how all numbers have the right decimal separator!
3. Another thing I noticed after trying different architectures, that locale awareness cannot be just and addition. If you have a babel and non-babel classes, you end up with too many of them. That is why babel related keywords are now in all methods. Formatters are still very easy to build, the example above will work. babel will still be optional, but locale keywords will be needed. Without babel, number formatting will be affected by locale but not the unit names.

Take a look at develop ... (test are failing but it is shaping nicely)

[hgrecco]

Fewer and fewer tests are failing, but now there is a decision to be made. While I have been trying to make everything backwards compatible, it is becoming hard to keep the same string representation for numbers. As I mentioned before, I am fixing the localized version (that is a change, but a valuable one). The previous behavior was broken.

But there is one more: number of digits.

Are we ok with changing how numbers are formatted?

Until now, numbers were formatted into strings sometimes with str(value) except when a spec was provided (or a numpy array was being formatted). But this leads to unexpected differences in the way numbers are shown. Also, this is not localizable. The way to make numbers localizable with format is using the n type.

But:

>>> str(24.0)
24.0
>>> format(24.0, "n")
24
>>> str(1.234567890)
'1.23456789'
>>> format(1.234567890, "n")
'1.23457'

In other words, using the n format changes the way magnitudes (and quantities) are printed. We could create a function to handle all the cases, but is a pain in the neck. Also, using n requires changing the locale which is not thread safe AFAIK.

Another way would be to use babel format_decimal. On very nice thing about this is that it does not require changing the locale to make it work and therefore is fully thread safe. But AFAIK it uses the Locale Data Markup Language specification and not Python's Format Specification Mini-Language. Users will require to learn yet another thing.

Options:

1. Use format with n as the default.
   🟢 works for all python numerical types out of the box
   🟢 localizable,
   🟢 builtin, less code to maintain
   🔴 changes current representation
   🔴 not thread safe
   🟢 does not require learning a new mini language
2. Use format with distinct defaults for float and int
   🟡 works for all numerical python types, with distinct code paths.
   🟢 localizable,
   🔴 requires building a specific function
   🟡 some representations will work, others not (corner cases)
   🔴 not thread safe
   🟢 does not require learning a new mini language
3. Use babel's format_decimal
   🟢 works for all numerical python types out of the box
   🟢 localizable,
   🟢 in babel, less code to maintain
   🟡 maybe changes current representation
   🟢 not thread safe
   🔴 require learning a new mini language

Another option will be exploring translating python mini language to babel (is something available?). Or use a package that implements a localizable with arguments format if available.

Ps.- @MichaelTiemannOSC I have a great way to sort units in mind. Simple, easy to implement and that will available in all formatters out of the box.

[MichaelTiemannOSC]

I read through you questions and your recent commits. I don't have the knowledge myself to answer them, but perhaps a release candidate with the option you think is cleanest (#3?) can get enough user feedback to provide more data.

I look forward to seeing the new sorting idea you have in mind!

[dalito]

I am slightly in favor of option 3 but I haven't used babel localization much. So there may be implications that I don't foresee.

[hgrecco]

@dalito I think it might be possible to generate UCUM notation! Let's stabilize the API and see.
@MichaelTiemannOSC good idea. I think we sshould release an alpha or beta version and invite people to try it

[hgrecco]

🎊 All test are passing

Disclaimer: I had to changed a few tests

• Proper localization of number 24.1 in localized fr_FR is 24,1
• es_AR was changed to the more common es_ES
• In the empty format vatios por centímetros should be vatios / centímetros
• everything related to default_format and missing specific unit format was labeled as xfail until the behavior is clear

There is still work to do:

• docstring
• documenation
• clean up and organize code
• implement sorting
• solve/decide number related issue (see above)

I will merge this into main tomorrow and continue working.