Python 3.8 woes

February 05, 2020

I’m not as deep into Python these days as I was twenty years ago. Twenty years ago, Python was small language with clear documentation and clear, consistent ways of doing things. One thing was hard, though, and that was packaging your python application so people on different systems could use it.

These days, Python is big, has lots of computer-sciency features that I don’t grok, and packaging Python is still hard. And the documentation is, for the most part, not very useful. I didn’t care a lot about that, though, since we only use Python as Krita’s extension language together with PyQt. And we had a nice and working setup for that.

Well, nice… It’s a bit hacky, especially for Windows. Especially since we need to build Krita with mingw, because msvc has problems compiling the Vc library. And Python has problems getting built with mingw-gcc on Windows.

We have three related parts: python, sip, which creates Python libraries out of special hand-written header-like files, and PyQt, which binds Pyton and Qt.

So, we start with a system-wide install of Python. This is used to configure Qt and build sip and PyQt. Then we download an embeddable Python of exactly the same version as the system-wide install, and install that with Krita’s other dependencies.

But last week the KDE binary factory windows images got updated to Python 3.8.1 (from 3.6.0) so we had to update the references to Python in Krita’s build system. There are a couple of places where the exact version is hard-coded, not just in the build system, but also in the code that setups Python for actual usage when Krita runs.

And at that point, our house of cards fell apart. Python 3.8 has a new, improved, more consistent way of finding dll libraries on Windows. At least, that was the idea. Following discussion in Python’s bug tracker, a Python developer declared victory:

Python 3.8 on Windows is getting a new fix for DLL Hell. If you bundle precompiled DLLs with your wheels or use ctypes, you should read https://t.co/tjNwZPII3X and the os.add_dll_directory() function. pic.twitter.com/NwesjRgWCN

— Zooba (@zooba) March 31, 2019

Given that the actual error we had was

c:\dev\krita>python
Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64
bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import PyQt5
>>> import PyQt5.Qt
>>> import PyQt5.QtCore
Traceback (most recent call last):
File "", line 1, in
ImportError: DLL load failed while importing QtCore: The specified module
could not be found.

This seemed related: the PYQt pyd files link to the Qt dll’s. The pyd files are in lib/krita-python-plugins, the Qt dll’s in the bin folder, and it looks like the QtCore pyd, even though dependency walker shows it can find the QtCore.dll, Python cannot find it anymore.

So, on to the changelog. This says:

DLL dependencies for extension modules and DLLs loaded with ctypes on Windows are now resolved more securely. Only the system paths, the directory containing the DLL or PYD file, and directories added with add_dll_directory() are searched for load-time dependencies. Specifically, PATH and the current working directory are no longer used, and modifications to these will no longer have any effect on normal DLL resolution. If your application relies on these mechanisms, you should check for add_dll_directory() and if it exists, use it to add your DLLs directory while loading your library. Note that Windows 7 users will need to ensure that Windows Update KB2533623 has been installed (this is also verified by the installer). (Contributed by Steve Dower in bpo-36085.)

Well, that sounds relevant, so let’s check the documentation and see if there are examples of using this add_dll_directory…

os.add_dll_directory(path)

Add a path to the DLL search path.

This search path is used when resolving dependencies for imported extension modules (the module itself is resolved through sys.path), and also by ctypes.

Remove the directory by calling close() on the returned object or using it in a with statement.

See the Microsoft documentation for more information about how DLLs are loaded.

Availability: Windows.

New in version 3.8: Previous versions of CPython would resolve DLLs using the default behavior for the current process. This led to inconsistencies, such as only sometimes searching PATH or the current working directory, and OS functions such as AddDllDirectory having no effect.

In 3.8, the two primary ways DLLs are loaded now explicitly override the process-wide behavior to ensure consistency. See the porting notes for information on updating libraries.

Well, no… So let’s google for it. That didn’t find a lot of useful links. The most useful was a patch for VTK that uses this method from the C++ wrapper The examples in that tweet would’ve been a good addition to the documentation.

So I started experimenting myself. And failed. Our code that detects whether PyQt5 is available isn’t very complicated, but no matter what I did — even copying the Qt5 dll’s to the PyQt5 folder, using add_dll_directory in various ways, I would always get the same error.

And now I’m stuck. Downgrading to Python 3.6 makes everything work again, another hint that this dll finding change is the problem, but that’s not the way forward, of course.

For now, the beta of Krita 4.2.9 is delayed until we find a solution.