From 6e49c314cd5050763f983759d156f26ffbc59368 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?= Date: Sat, 7 May 2022 23:38:31 +0200 Subject: [PATCH] Allow referencing thread identifier in `compiledir_format` Add a new `thread_id` key that permits referencing the current thread ID (`threading.get_ident()`) in `compiledir_format`. This makes it possible to use per-thread compiledirs in multithreaded programs, and therefore make it possible for aesara to run multiple compiler instances in parallel. Combined with pytest-xdist, this change makes it possible to run aesara's tests in parallel and better utilize multiprocessor (multicore) systems. On my Ryzen 5, this provides around 4x speedup (down from 82 minutes to 20 minutes per test run). Given that we are running the test suite multiple times to test different Python versions, this literally saves hours at a time. Prior to this change, xdist would not be able to fully utilize the available resources since the compiledir lock would prevent aesara from starting multiple compiler instances in parallel, and therefore practically serialize tests on compiler invocations. --- aesara/configdefaults.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/aesara/configdefaults.py b/aesara/configdefaults.py index 8121919958..60eb4f997e 100644 --- a/aesara/configdefaults.py +++ b/aesara/configdefaults.py @@ -7,6 +7,7 @@ import socket import sys import textwrap +import threading import numpy as np @@ -1345,6 +1346,7 @@ def _get_home_dir(): "numpy_version": np.__version__, "gxx_version": "xxx", "hostname": socket.gethostname(), + "thread_id": threading.get_ident(), }