The Python programming language releases new versions yearly, with a feature-locked beta release in the first half of the year and the final release toward the end of the year.
Python 3.12 has just been released. Developers are encouraged to try out this latest version on non-production code, both to verify that it works with your programs and to get an idea of whether your code will benefit from the new feature sets and performance enhancements in this latest version.
Here’s a rundown of the most significant new features in Python 3.12 and what they mean for Python developers.
Better error messages
Error messages have been getting more precise (exact positions in lines) and more detailed (better suggestions about what might be wrong) in recent Python versions. Python 3.12 brings additional enhancements:
- Missing module suggestions (“did you forget to import X?”) now include modules from the standard library.
- Better error suggestions for a common syntax error with imports; for example,
import p from mreturns an error asking if you meantfrom m import p. - Import errors for a given module now include suggestions from the namespace of the module imported from. For instance, if you try
from thismodule import thisclasswhen you meanThisClass, you’ll get a suggestion forThisClass. NameErrorsuggestions now also includeself.prepended to the name when raised inside a class instance (e.g.,name 'speed' is not defined. Did you mean 'self.speed'?). This is because omittingselffor instance variables is a common source of errors in class instances.
Fewer restrictions on f-string formatting
F-strings, Python’s convenient system for performing string formatting, used to be heavily restricted in terms of how they could be formatted. Python 3.12 removes many of these limitations. The changes:
- F-string expressions can now be any valid Python expression.
- F-string expressions can now contain the same kinds of quotes as those used to set off the f-string itself. For instance,
f"The shopping list, {", ".join(groceries)}"is now a valid f-string. - F-string expressions can now be multiline expressions, as long as they follow the same rules for other multiline expressions (e.g., using parentheses to allow expressions to span multiple lines).
- Backslashes and Unicode character definitions are now allowed in f-strings. You can use everything from simple control characters (
\n) to references to the Unicode namespace (\N{POUND SIGN}). - Errors within f-string expressions now yield the exact location of the error within the enclosing statement, not just within the expression itself. This makes f-string errors easier to track down and troubleshoot.
Support for the Linux perf profiler
The widely used Linux profiler tool perf works with Python, but only returns information about what’s happening at the C level in the Python runtime. Information about actual Python program functions doesn’t show up.
Python 3.12 enables an opt-in mode to allow perf to harvest details about Python programs, not just the runtime. The opt-in can be done at the environment level or inside a Python program with the sys.activate_stack_trampoline function.
Faster debug/profile monitoring
Running a profiler or attaching a debugger to a Python program gives you visibility and insight into what the program’s doing. It also comes at a performance cost. Programs can run as much as an order of magnitude slower when run through a debugger or profiler.
PEP 669 provides hooks for code object events that profilers and debuggers can attach to, such as the start or end of a function. A callback function could be registered by a tool to fire whenever such an event is triggered. There will still be a performance hit for profiling or debugging, but it’ll be greatly reduced.
Buffer protocol dunders
Python’s buffer protocol provides a way to get access to the raw region of memory wrapped by many Python objects, like bytes or bytearray. But most interactions with the buffer protocol happen through C extensions. Up till now, it hasn’t been possible for Python code to know whether a given object supports the buffer protocol, or to type-annotate code as being compatible with the protocol.
PEP 688 implements new dunder methods for objects that allow Python code to work with the buffer protocol. This makes it easier to write objects in Python that expose their data buffers, instead of having to write those objects in C. The __buffer__ method can be used for code that allocates new memory or simply accesses existing memory; it returns a memoryview object. The __release_buffer__ method is used to release the memory used for the buffer.
Right now the PEP 688 methods don’t have a way to indicate if a given buffer is read-only or not—which is useful if you’re dealing with data for an immutable object like bytes. But the door is open to add that feature if it’s needed.
Typing improvements
Python’s type-hinting syntax, added in Python 3.5, allows linting tools to catch a wide variety of errors ahead of time. With each new version, typing in Python gains features to cover a broader and more granular range of use cases.
TypedDict
In Python 3.12, you can use a TypedDict as source of types to hint keyword arguments used in a function. The Unpack variadic generic, introduced in version 3.11, is used for this. Here’s an example from the relevant PEP:
class Movie(TypedDict):
name: str
year: int
def foo(**kwargs: Unpack[Movie]) -> None: ...
Here, foo can take in keyword arguments of names and types that match the contents of Movie—name:str and year:int. One scenario where this is useful is type-hinting functions that take optional keyword-only arguments with no default values.
Type parameter syntax
The type parameter syntax provides a cleaner way to specify types in a generic class, function, or type alias. Here’s an example taken from the PEP:
# the old method
from typing import TypeVar
_T = TypeVar("_T")
def func(a: _T, b: _T) -> _T:
...
# the new type parameter method
def func[T](a: T, b: T) -> T:
...
With the new method, one doesn’t need to import TypeVar. One can just use the func[T] syntax to indicate generic type references. It’s also possible to specify type bounds, such as whether a given type is one of a group of types, although such types can’t themselves be generic. An example is func[T: (str,int)].
Finally, the new @override decorator can be used to flag methods that override methods in a parent, as a way to ensure any changes made to the parent during refactoring (renaming or deleting) also are reflected in its children.
Performance improvements
With Python 3.11, a number of allied projects got underway to improve Python’s performance by leaps and bounds with each new version. The performance improvements in Python 3.12 aren’t as dramatic, but they’re still noteworthy.
Comprehension inlining
Comprehensions, a syntax that lets you quickly construct lists, dictionaries, and sets, are now constructed “inline” rather than by way of temporary objects. The speedup for this has been clocked at around 11% for a real-world case and up to twice as fast for a micro-benchmark.
Immortal objects
Every object in Python has a reference count that tracks how many times other objects refer to it, including built-in objects like None. PEP 683 allows objects to be treated as “immortal,” so that they never have their reference count changed.
Making objects immortal has other powerful implications for Python in the long run. It makes it easier to implement multicore scaling, and to implement other optimizations (like avoiding copy-on-write) that would have been hard to implement before.
Smaller object sizes
With earlier versions of Python, the base size of an object was 208 bytes. Objects have been refactored multiple times over the last few versions of Python to make them smaller, which doesn’t just allow more objects to live in memory but helps with cache locality. As of Python 3.12, the base size of an object is now 96 bytes—less than half of what it used to be.
Subinterpreters
A long-awaited feature for Python is subinterpreters—the ability to have multiple instances of an interpreter, each with its own GIL, running side-by-side within a single Python process. This would be a big step toward better parallelism in Python.
However, version 3.12 only includes the CPython internals to make this possible. There’s still no end-user interface to subinterpreters. A standard library module, interpreters, is intended to do this, but it’s now slated to appear in Python 3.13.
Additional changes
Python 3.12 rolls out countless other little changes in addition to the big ones discussed so far. Here’s a quick look.
Unstable API
A key ongoing project has been the refactoring of CPython’s internals, especially its API sets, so that fewer of CPython’s low-level functions need to be exposed. Python 3.12 introduced the unstable API tier, an API set marked specifically as being likely to change between versions. It’s not intended to be used by most C extensions, but by low-level tools such as debuggers or JIT compilers.
Standard library deprecations and removals
With version 3.11, a number of standard library modules long known to be obsolete (so-called dead batteries) got flagged for removal as of Python 3.12 and 3.13. In version 3.12, one of the biggest removals was distutils, which has long been obviated by setuptools. Other modules removed in this version were asynchat, asyncore (both replaced by asyncio), and smtpd.
Garbage collection
Python’s garbage collection mechanism (GC) used to be able to run whenever an object was allocated. As of Python 3.12, the GC runs only on the “eval breaker” mechanism in the Python bytecode loop—that is, between executing one bytecode and another. It also runs whenever CPython’s signal-handler-checking mechanism is invoked. This makes it possible to run GC periodically on a long-running call to a C extension outside the runtime.
Copyright © 2023 IDG Communications, Inc.
