Add parameter to call which allows passing configuration to the underlying ArgumentFormatter

[plazer1]

This PR adds the possibility to use a custom help formatter via plac.call(..., formatter_class=MyCustomFormatter). By using a custom formatter, users can e.g. preserve newlines in help strings of arguments (as well as the main program description, which is currently already possible).

This PR also adds __pycache__ to .gitignore.

Code:

import plac

def main(
    foo:
      (r'''
        Lorem ipsum dolor *sit amet* -- consectetur adipiscing elit!
        Maecenas hendrerit quam nisl (vitae lacinia dui ullamcorper vulputate).
      ''', 'option', 'f', str)
      = 'bar'
):
  ...

if __name__ == '__main__':
  plac.call(main, formatter_class=plac.MultilineFormatter)

---

Current output with RawDescriptionHelpFormatter:

usage: a.py [-h] [-f bar]

options:
  -h, --help     show this help message and exit
  -f, --foo bar  Lorem ipsum dolor *sit amet* -- consectetur adipiscing elit! Maecenas hendrerit quam nisl (vitae lacinia dui ullamcorper vulputate).

Example output with RawTextHelpFormatter: it preserves the text verbatim -- does not strip and de-dent the description text, nor replace newlines with spaces:

usage: a.py [-h] [-f bar]

options:
  -h, --help     show this help message and exit
  -f, --foo bar
                         Lorem ipsum dolor *sit amet* -- consectetur adipiscing elit!
                         Maecenas hendrerit quam nisl (vitae lacinia dui ullamcorper vulputate).

[ialbert]

The problem is Hyrum’s Law - we really don't want to alter how plac works, we don't know what might break down the line:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.”

https://www.hyrumslaw.com/

[plazer1]

I would be happy if I could pass my custom formatter to call, which is currently impossible. I'll adjust my PR to keep the default formatter and allow using a custom one. Better yet – if you're not opposed – I'd also like to add the custom formatter implementation to plac so that users don't have to implement it themselves (I'll also add a mention of this new formatter to the readme).

[ialbert]

yes, that might work,

that being said we have to keep backward and python 2.7(!) compatibility

[plazer1]

that being said we have to keep backward and python 2.7(!) compatibility

Hmmm.... According to the readme the lowest supported version is 2.6; however, according to the source code it's actually 2.3. So which is it: 2.3, 2.6 or 2.7?

[plazer1]

My implementation of the custom formatter has gotten quite complex and fragile as I kept finding more and more issues with it. After (hopefully) solving them all and working around the quirks of textwrap in inelegant ways, I don't think the code is reliable enough to merge into upstream, besides being Python2-incompatible. So I decided to remove the new formatter implementation and only keep the parameter which allows passing a formatter to the parser.

[ialbert]

Thanks for following up and all the efforts and being transparent about the challenges.

It may be that the tradeoffs here do warrant a merge into the main library. Adding a new feature seems attractive, but it can also lead to headaches down the line.

So perhaps it would be best to keep this on the backburner for a while, let you explore this funcionality for a while, then revisit if it ends up being so useful.

The best think about plac is that it is so easy to hack and vendor your private version of it.

[plazer1]

To be frank I do not understand what are you trying to tell me 😅 Are you saying you do not want to merge the PR as it currently is? Just to be clear, I already removed the new functionality. All that's left in this PR right now is the addition of keyword arguments to call which get passed on to argparse.ArgumentParser, such as (but not limited to) formatter_class.

If you're worried about potential future clashes between arguments of call and arguments of ArgumentParser, then the solution is quite simple -- we will pass the ArgumentParser config as a single argument of type dict instead of kwargs **parser_confparams.

[ialbert]

What I mean is that adding just the option doesn't seem to warrant a new release.

It is an interesting feature, but since we can't make a new formatter the default without potentially breaking tests, and you also noted that formatting is not that easy to implement correctly, thus making this customization relevant to only a small group, it may be best to err on the side of caution and leave things as they are.

[plazer1]

err on the side of caution and leave things as they are

Not to sound spiteful, but the changes I propose here pose no risk -- there is virtually zero chance of breaking anything -- unlike the changes of 1.4.6 which completely broke plac on Windows yet nobody seemed to have noticed for 3 weeks... So I'd say there can't be that many users to begin with, since you're mentioning this functionality will only be useful to "a small group". Speaking of which, would you care to guess how many users of plac besides @micheles actually use the advanced functionality implemented in plac_ext? I would say it's less than those who keep scratching their head about how to add a linebreak to argument description 🙂

I understand the position you're in -- maintaining a project you did not author. But if you don't intend to make any improvements, sooner or later somebody is going to fork and do it for you, and your project will become irrelevant. (And I had another improvement idea in mind but seeing as reluctant you are to make even such a simple, non-invasive change, I don't think I'll invest my time in opening another PR.)

---

Addendum to the above:

thus making this customization relevant to only a small group

I understand that. But you have to understand that this trivial code change, which allows the user to customize the behavior, is the difference between me having to maintain my fork of plac (which includes all the related chores like merging from upstream, resolving conflicts, making releases, etc.), and just using the upstream without any of the pain of having my own fork. I'd much prefer the latter option.

[ialbert]

For me backward compatibility is very important, as is keeping code as simple as possible.

I do agree that the entire plac_ext might be rarely used; I have never needed it myself. But one of my greatest annoyances is when an upstream package makes gratuitous changes that break my code for reasons like cleanup, formatting, or adhering to new conventions. These shouldn't be necessary because plac isn't a library meant to gain new functionality and features.

I'd like to keep plac as close to the original vision as possible. Minimal features, clean documentation; for those who need fancier features, another option may be better.

The current codebase is meant to be kept functional. Over the past five years, all changes were necessary primarily because of changes in Python itself. We always respond promptly to valid problems; we fixed the one you experienced within hours. But changing things up is a much bigger ask.

[plazer1]

for those who need fancier features, another option may be better

Generating parser automatically from code is a fancy feature. Possibly the fanciest feature of any argument parser, so I don't intend to switch to any other; I'd rather fork, add my functionality, get rid of "junk" I don't need (plac_ext and plac_runner), and drop the Python 2 compatibility.

For your reference, here is my improved interface which I made on top of mainline plac without requiring any modifications to the upstream code. IMO it's much superior to the decorators which are unsightly and error-prone (you can misspell the param name), at the cost of being Python2-incompatible and abusing the type hints -- but this is already a feature of plac (although currently undocumented?); I just made it more user-friendly:

def main(
    include_init_files:
        OPT('i', str, 'Include `__init__.py` files: [A]lways, when non-[E]mpty, [N]ever',
            choices='AEN')
        = 'E',
    open_in_editor:
        FLAG('o', 'Open files in editor; pause after each file')
        = False,
    *fpaths:
        POS(pathlib.Path, 'Paths to files or directories (default: current directory)'),
):
  ...

[ialbert]

I think these are all interesting ideas, and perhaps the best home for them is a new package or fork that builds on the prior work.

It sounds like you have a clear vision for how plac could evolve, but that vision doesn't fully align with what I'm trying to do as its maintainer. My goal is deliberately conservative: preserve backwards compatibility, keep the implementation simple, and keep plac as close as practical to its original scope and design. That means the bar for adding new functionality is intentionally quite high, even when a particular change is small or unlikely to cause breakage.

That's not a judgment on the usefulness of the features you're proposing. In fact, a fork may give you much more freedom to pursue these ideas without having to repeatedly reconcile them with the constraints I've chosen for this package.

I appreciate the time you've put into the proposal and the discussion, but given that difference in direction, I don't think this is something I want to merge into plac. I'm going to close this PR, and I wish you well if you decide to develop these ideas further in a fork or a separate package.

[plazer1]

After perusing through an ancient version of the documentation (is there any current documentation besides the very limited README?) I realized the functionality I am asking for has already been implemented pretty much from the beginning, albeit in a rather unconventional and currently undocumented way:

def main(...):
    ...

main.formatter_class = MyCustomHelpFormatter

if __name__ == '__main__':
    plac.call(main)

It's a shame the old documentation was (seemingly?) lost to time, because it contained lots of information that is still relevant to the current version but not present in the README. Quoting from the (14 years old!) documentation:

Moreover, it is possible to pass options to the underlying argparse.ArgumentParser object (currently it accepts the default arguments description, epilog, prog, usage, add_help, argument_default, parents, prefix_chars, fromfile_prefix_chars, conflict_handler, formatter_class). It is enough to set such attributes on the main function.

Another currently-undocumented cool feature, used by my OPT, FLAG and POS wrappers in the comment above, is using parameter annotations instead of decorators:

In plac the parser is inferred by the function annotations. Here is an example:

# example8.py
def main(command: ("SQL query", 'option', 'q'), dsn):
    if command:
        print('executing %s on %s' % (command, dsn))
        # ...