🌐 Mailman 3 August 2021 - Python-list - python.org

Defining a Python enum in a C extension - am I doing this right?
by Bartosz Golaszewski Aug. 8, 2021

Aug. 8, 2021

Hi! I'm working on a Python C extension and I would like to expose a custom enum (as in: a class inheriting from enum.Enum) that would be entirely defined in C. It turned out to not be a trivial task and the regular mechanism for inheritance using .tp_base doesn't work - most likely due to the Enum's meta class not being pulled in. Basically I'm trying to do this: import enum class FooBar(enum.Enum): FOO = 1 BAR = 2 in C. After a lot of digging into cpython's internals, this is what I came up with, wrapped in an example buildable module: #include <Python.h> PyDoc_STRVAR(module_doc, "C extension module defining a class inheriting from enum.Enum."); static PyModuleDef module_def = { PyModuleDef_HEAD_INIT, .m_name = "pycenum", .m_doc = module_doc, .m_size = -1, }; struct enum_descr { const char *name; long value; }; static const struct enum_descr foobar_descr[] = { { .name = "FOO", .value = 1, }, { .name = "BAR", .value = 2, }, { } }; static PyObject *make_bases(PyObject *enum_mod) { PyObject *enum_type, *bases; enum_type = PyObject_GetAttrString(enum_mod, "Enum"); if (!enum_type) return NULL; bases = PyTuple_Pack(1, enum_type); /* Steals reference. */ if (!bases) Py_DECREF(enum_type); return bases; } static PyObject *make_classdict(PyObject *enum_mod, PyObject *bases) { PyObject *enum_meta_type, *classdict; enum_meta_type = PyObject_GetAttrString(enum_mod, "EnumMeta"); if (!enum_meta_type) return NULL; classdict = PyObject_CallMethod(enum_meta_type, "__prepare__", "sO", "FooBarEnum", bases); Py_DECREF(enum_meta_type); return classdict; } static int fill_classdict(PyObject *classdict, PyObject *modname, const struct enum_descr *descr) { const struct enum_descr *entry; PyObject *key, *val; int ret; key = PyUnicode_FromString("__module__"); if (!key) return -1; ret = PyObject_SetItem(classdict, key, modname); Py_DECREF(key); if (ret < 0) return -1; for (entry = descr; entry->name; entry++) { key = PyUnicode_FromString(entry->name); if (!key) return -1; val = PyLong_FromLong(entry->value); if (!val) { Py_DECREF(key); return -1; } ret = PyObject_SetItem(classdict, key, val); Py_DECREF(key); Py_DECREF(val); if (ret < 0) return -1; } return 0; } static PyObject *make_new_type(PyObject *classdict, PyObject *bases, const char *enum_name) { PyObject *name, *args, *new_type; int ret; name = PyUnicode_FromString(enum_name); if (!name) return NULL; args = PyTuple_Pack(3, name, bases, classdict); if (!args) { Py_DECREF(name); return NULL; } Py_INCREF(bases); Py_INCREF(classdict); /* * Reference to name was stolen by PyTuple_Pack(), no need to * increase it here. */ new_type = PyObject_CallObject((PyObject *)&PyType_Type, args); Py_DECREF(args); if (!new_type) return NULL; ret = PyType_Ready((PyTypeObject *)new_type); if (ret < 0) { Py_DECREF(new_type); return NULL; } return new_type; } static PyObject *make_enum_type(PyObject *modname, const char *enum_name, const struct enum_descr *descr) { PyObject *enum_mod, *bases, *classdict, *new_type; int ret; enum_mod = PyImport_ImportModule("enum"); if (!enum_mod) return NULL; bases = make_bases(enum_mod); if (!bases) { Py_DECREF(enum_mod); return NULL; } classdict = make_classdict(enum_mod, bases); if (!classdict) { Py_DECREF(bases); Py_DECREF(enum_mod); return NULL; } ret = fill_classdict(classdict, modname, descr); if (ret < 0) { Py_DECREF(bases); Py_DECREF(enum_mod); Py_DECREF(classdict); return NULL; } new_type = make_new_type(classdict, bases, enum_name); Py_DECREF(bases); Py_DECREF(enum_mod); Py_DECREF(classdict); return new_type; } PyMODINIT_FUNC PyInit_pycenum(void) { PyObject *module, *modname, *sub_enum_type; int ret; module = PyModule_Create(&module_def); if (!module) return NULL; ret = PyModule_AddStringConstant(module, "__version__", "0.0.1"); if (ret < 0) { Py_DECREF(module); return NULL; } modname = PyModule_GetNameObject(module); if (!modname) { Py_DECREF(module); return NULL; } sub_enum_type = make_enum_type(modname, "FooBar", foobar_descr); Py_DECREF(modname); if (!sub_enum_type) { Py_DECREF(module); return NULL; } ret = PyModule_AddObject(module, "FooBar", sub_enum_type); if (ret < 0) { Py_DECREF(sub_enum_type); Py_DECREF(module); return NULL; } return module; } Basically I'm calling the EnumMeta's __prepare__ method directly to create a correct classdict and then I call the PyType_Type object too to create the sub-type. This works and AFAICT results in a class that behaves exactly as expected, but... am I doing this right? Any feedback is appreciated. I want to use this in a real project, namely the v2 of libgpiod python bindings[1] so it's important to me to get this right. Best regards, Bartosz Golaszewski [1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/

5 10

Ask for help on using re
by Jach Feng Aug. 7, 2021

Aug. 7, 2021

I want to distinguish between numbers with/without a dot attached: >>> text = 'ch 1. is\nch 23. is\nch 4 is\nch 56 is\n' >>> re.compile(r'ch \d{1,}[.]').findall(text) ['ch 1.', 'ch 23.'] >>> re.compile(r'ch \d{1,}[^.]').findall(text) ['ch 23', 'ch 4 ', 'ch 56 '] I can guess why the 'ch 23' appears in the second list. But how to get rid of it? --Jach

6 16

error from pypa build
by Robin Becker Aug. 5, 2021

Aug. 5, 2021

I'm building a pure python wheel in a python3.9 virtualenv. If I use python setup.py bdist_wheel I do get a wheel named rlextra-3.6.0-py3-none-any.whl I'm trying out building a modern python 3 only wheel so I followed instructions here https://packaging.python.org/tutorials/packaging-projects/#creating-the-pac… and see the error below which is not very informative. Any ideas of what's wrong? I still get the error even after upgrading pip so I don't think that is the problem. > (.py39) user@pc:~/devel/reportlab/REPOS/rlextra > $ pip install build > Collecting build > Downloading build-0.5.1-py2.py3-none-any.whl (15 kB) > Collecting pep517>=0.9.1 > .......... > Installing collected packages: tomli, pyparsing, toml, pep517, packaging, build > Successfully installed build-0.5.1 packaging-21.0 pep517-0.11.0 pyparsing-2.4.7 toml-0.10.2 tomli-1.2.0 > WARNING: You are using pip version 21.1.3; however, version 21.2.2 is available. > You should consider upgrading via the '/home/user/devel/reportlab/.py39/bin/python39 -m pip install --upgrade pip' command. > (.py39) user@pc:~/devel/reportlab/REPOS/rlextra > $ ls > build dist docs examples LICENSE.txt README.txt setup.cfg setup.py src tests tmp > (.py39) user@pc:~/devel/reportlab/REPOS/rlextra > $ python -m build --wheel > Traceback (most recent call last): > File "/home/user/devel/reportlab/.py39/lib/python3.9/site-packages/build/__main__.py", line 302, in main > build_call(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependency_check) > File "/home/user/devel/reportlab/.py39/lib/python3.9/site-packages/build/__main__.py", line 145, in build_package > _build(isolation, builder, outdir, distribution, config_settings, skip_dependency_check) > File "/home/user/devel/reportlab/.py39/lib/python3.9/site-packages/build/__main__.py", line 101, in _build > return _build_in_isolated_env(builder, outdir, distribution, config_settings) > File "/home/user/devel/reportlab/.py39/lib/python3.9/site-packages/build/__main__.py", line 77, in _build_in_isolated_env > with IsolatedEnvBuilder() as env: > File "/home/user/devel/reportlab/.py39/lib/python3.9/site-packages/build/env.py", line 92, in __enter__ > executable, scripts_dir = _create_isolated_env_venv(self._path) > File "/home/user/devel/reportlab/.py39/lib/python3.9/site-packages/build/env.py", line 221, in _create_isolated_env_venv > pip_distribution = next(iter(metadata.distributions(name='pip', path=[purelib]))) > StopIteration > > ERROR > (.py39) user@pc:~/devel/reportlab/REPOS/rlextra thanks for any assistance -- Robin Becker

1 0

SQLALchemy: update with in clause from kwargs
by Larry Martell Aug. 4, 2021

Aug. 4, 2021

I am trying to write a function that takes kwargs as a param and generates an update statement where the rows to be updated are specified in an in clause. Something like this: def update_by_in(self, **kwargs): filter_group = [] for col in kwargs['query_params']: # obviously this line does not work as col is a string, but this is the intent filter_group.append(col.in_(tuple(kwargs['query_params'][col]))) self._session.query(self.model_class).filter(*filter_group).update(kwargs['values']) self.update_by_in( **{'query_params': {'companyCode': ['A', 'B', 'C']}, 'values': {'portfolioName': 'test'}} ) Is there a way to do this? I think I need to use setattr in building up the filter_group list, but I'm not quite sure how to do it.

3 3

[RELEASE] Python 3.10.0rc1 is available
by Pablo Galindo Salgado Aug. 4, 2021

Aug. 4, 2021

Python 3.10.0 is almost ready. This release, 3.10.0rc1, is the penultimate release preview. You can get it here: https://www.python.org/downloads/release/python-3100rc1/ *This is the first release candidate of Python 3.10* This release, **3.10.0rc1**, is the penultimate release preview. Entering the release candidate phase, only reviewed code changes which are clear bug fixes are allowed between this release candidate and the final release. The second candidate and the last planned release preview is currently planned for 2021-09-06 while the official release is planned for 2021-10-04. There will be no ABI changes from this point forward in the 3.10 series and the goal is that there will be as few code changes as possible. *Call to action* Core developers: all eyes on the docs now - Are all your changes properly documented? - Did you notice other changes you know of to have insufficient documentation? Community members We strongly encourage maintainers of third-party Python projects to prepare their projects for 3.10 compatibilities during this phase. As always, report any issues to the Python bug tracker <https://bugs.python.org/>. Please keep in mind that this is a preview release and its use is **not** recommended for production environments. *And now for something completely different* In theoretical physics, quantum chromodynamics (QCD) is the theory of the strong interaction between quarks and gluons, the fundamental particles that make up composite hadrons such as the proton, neutron, and pion. The QCD analog of electric charge is a property called color. Gluons are the force carrier of the theory, just as photons are for the electromagnetic force in quantum electrodynamics. There are three kinds of charge in QCD (as opposed to one in quantum electrodynamics or QED) that are usually referred to as "color charge" by loose analogy to the three kinds of color (red, green and blue) perceived by humans. Other than this nomenclature, the quantum parameter "color" is completely unrelated to the everyday, familiar phenomenon of color. *We hope you enjoy those new releases!* Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. Regards from cloudy London, Your friendly release team, Pablo Galindo @pablogsal Ned Deily @nad Steve Dower @steve.dower

1 1

Re: Defining a Python enum in a C extension - am I doing this right?
by Sean DiZazzo Aug. 4, 2021

Aug. 4, 2021

On Tuesday, August 3, 2021 at 3:04:19 AM UTC-7, Bartosz Golaszewski wrote: > On Sat, Jul 31, 2021 at 3:01 PM Bartosz Golaszewski <br...(a)bgdev.pl> wrote: > > > > On Fri, Jul 30, 2021 at 2:41 PM Serhiy Storchaka <stor...(a)gmail.com> wrote: > > > > > > 23.07.21 11:20, Bartosz Golaszewski пише: > > > > I'm working on a Python C extension and I would like to expose a > > > > custom enum (as in: a class inheriting from enum.Enum) that would be > > > > entirely defined in C. > > > > > > I think that it would be much easier to define it in Python, and then > > > either import a Python module in your C code, or exec a Python code as a > > > string. > > > > > > > You mean: evaluate a string like this: > > > > ''' > > import enum > > > > class FooBar(enum.Enum): > > FOO = 1 > > BAR = 2 > > BAZ = 3 > > ''' > > > > And then pull in the FooBar type from the resulting dictionary into my > > C code? Sounds good actually. I think I'll be able to add the FooBar > > type to another type's tp_dict too - because some enums I want to > > create will be nested in other classes. > > > > Bart > Just a follow-up: this is how I did it eventually: > > ``` > #include <Python.h> > > typedef struct { > PyObject_HEAD; > } dummy_object; > > PyDoc_STRVAR(dummy_type_doc, "Dummy type in which the enum will be nested."); > > static PyTypeObject dummy_type = { > PyVarObject_HEAD_INIT(NULL, 0) > .tp_name = "pycenum.DummyType", > .tp_basicsize = sizeof(dummy_object), > .tp_flags = Py_TPFLAGS_DEFAULT, > .tp_doc = dummy_type_doc, > .tp_new = PyType_GenericNew, > .tp_dealloc = (destructor)PyObject_Del, > }; > PyDoc_STRVAR(module_doc, > "C extension module defining a class inheriting from enum.Enum."); > > static PyModuleDef module_def = { > PyModuleDef_HEAD_INIT, > .m_name = "pycenum", > .m_doc = module_doc, > .m_size = -1, > }; > static int add_foobar_enum(PyObject *module) > { > static const char *foobar_src = > "class FooBar(enum.Enum):\n" > " FOO = 1\n" > " BAR = 2\n" > " BAZ = 3\n"; > > PyObject *main_mod, *main_dict, *enum_mod, *result, *foobar_type; > int ret; > > main_mod = PyImport_AddModule("__main__"); > if (!main_mod) > return -1; > > main_dict = PyModule_GetDict(main_mod); > if (!main_dict) { > Py_DECREF(main_mod); > return -1; > } > > enum_mod = PyImport_ImportModule("enum"); > if (!enum_mod) { > Py_DECREF(main_mod); > return -1; > } > > ret = PyDict_SetItemString(main_dict, "enum", enum_mod); > Py_DECREF(enum_mod); > if (ret) { > Py_DECREF(main_mod); > return -1; > } > > result = PyRun_String(foobar_src, Py_single_input, > main_dict, main_dict); > if (!result) { > Py_DECREF(main_mod); > return -1; > } > > foobar_type = PyDict_GetItemString(main_dict, "FooBar"); > if (!foobar_type) { > Py_DECREF(main_mod); > return -1; > } > > ret = PyDict_SetItemString(dummy_type.tp_dict, "FooBar", foobar_type); > Py_DECREF(foobar_type); > Py_DECREF(main_mod); > if (ret) > return -1; > > PyType_Modified(&dummy_type); > > return ret; > } > > PyMODINIT_FUNC PyInit_pycenum(void) > { > PyObject *module; > int ret; > > module = PyModule_Create(&module_def); > if (!module) > return NULL; > > ret = PyModule_AddStringConstant(module, "__version__", "0.0.1"); > if (ret) { > Py_DECREF(module); > return NULL; > } > > ret = PyType_Ready(&dummy_type); > if (ret) { > Py_DECREF(module); > return NULL; > } > > ret = add_foobar_enum(module); > if (ret) { > Py_DECREF(module); > return NULL; > } > > Py_INCREF(&dummy_type); > ret = PyModule_AddObject(module, "DummyType", (PyObject *)&dummy_type); > if (ret) { > Py_DECREF(&dummy_type); > Py_DECREF(module); > return NULL; > } > > return module; > } > ``` > > Bart No

1 0

Re: argparse: delimiter for argparse list arguments
by Jon Ribbens Aug. 3, 2021

Aug. 3, 2021

On 2021-08-03, Roel Schroeven <roel(a)roelschroeven.net> wrote: > Jon Ribbens via Python-list schreef op 3/08/2021 om 17:48: >> On 2021-08-03, Michael Torrie <torriem(a)gmail.com> wrote: >> > On 8/2/21 1:43 PM, Sven R. Kunze wrote: >> >> maybe, I am missing something here but is it possible to specify a >> >> delimiter for list arguments in argparse: >> >> >> >> https://docs.python.org/3/library/argparse.html >> >> >> >> Usually, '--' is used to separate two lists (cf. git). >> > >> > I've not seen this syntax in git. Are you referring the the double and >> > triple dot notation git uses? Can you give me an example of how git >> > uses -- as a list separator? >> >> Loads of git commands do this. e.g. commit, diff, log, status, etc. >> It's not completely unlike what you're describing above, which is >> already supported automatically by argparse. > > Commands like git commit do not use '--' to separate two lists, but as > Dan and Sven said to separate options from arguments, That is not quite correct. Check out 'git diff' for example - it takes as position arguments zero, one, or two commit hashes followed by zero or more pathnames. You can use '--' to separate these two lists, because of course sometimes a pathname can be indistinguishable from a commit hash.

2 1

argparse: delimiter for argparse list arguments
by Sven R. Kunze Aug. 3, 2021

Aug. 3, 2021

Hi everyone, maybe, I am missing something here but is it possible to specify a delimiter for list arguments in argparse: https://docs.python.org/3/library/argparse.html Usually, '--' is used to separate two lists (cf. git). Cheers, Sven

4 4

Re: argparse: delimiter for argparse list arguments
by Jon Ribbens Aug. 3, 2021

Aug. 3, 2021

On 2021-08-03, Michael Torrie <torriem(a)gmail.com> wrote: > On 8/2/21 1:43 PM, Sven R. Kunze wrote: >> maybe, I am missing something here but is it possible to specify a >> delimiter for list arguments in argparse: >> >> https://docs.python.org/3/library/argparse.html >> >> Usually, '--' is used to separate two lists (cf. git). > > I've not seen this syntax in git. Are you referring the the double and > triple dot notation git uses? Can you give me an example of how git > uses -- as a list separator? Loads of git commands do this. e.g. commit, diff, log, status, etc. It's not completely unlike what you're describing above, which is already supported automatically by argparse.

2 1