Support for pickling sentinel objects as singletons

[HexDecimal]

My attempt at implementing PEP 661 since I was unhappy with #594. I'm hoping that this is a PR of decent quality and not just me desperately wanting pickle support.

Changes made to Sentinel:

• Added module_name parameter following PEP implementation and tweaking that to use the local _caller helper function. Breaking change due to repr becoming a keyword parameter. module_name removed due to changes in PEP 661.
• name requires qualified names to support pickle singletons, this is tested and documented.
• Added __reduce__ to track Sentinel by name and module_name.
• Added copy and pickle tests. There were several other use-cases for pickle from the PEP 661 discussion. The PEP requires that the sentinel identity is preserved during these operations.
• Updated documentation for Sentinel.

For a while I still supported the repr parameter and after following the PEP 661 discussion I ended up implementing the truthiness tweaks mentioned there, but then I ended up scrapping all of that so that I could follow the PEP more closely, but they would be easy to reintroduce later if desired.

I'm unsure of how to mention version changes in the docs. Replacing repr with module_name is technically a breaking change.

Closes #720
Closes #742

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[Viicos]

Replacing repr with module_name is technically a breaking change.

I think breaking changes are expected for draft PEPs, so imo no deprecation process should be used. Users should be aware that changes are expected. However, I think we can:

• Improve the documentation about draft PEPs, stating that implementation can change.
• Explicitly state that Sentinels are from a draft PEP in the Sentinel class documentation, and isn't stable yet.

[Viicos]

cc @taleinat

[JelleZijlstra]

I'm not willing to remove the repr parameter from typing_extensions.Sentinel without a deprecation. However, I'm OK to re-export the builtin Sentinel class in 3.15 (if it makes it in), even if it doesn't have this parameter.

[HexDecimal]

I'm not willing to remove the repr parameter from typing_extensions.Sentinel without a deprecation. However, I'm OK to re-export the builtin Sentinel class in 3.15 (if it makes it in), even if it doesn't have this parameter.

Then I will change repr into a keyword parameter.

If compatibility is important enough then I can also have the code assume module_name is actually repr if it does not refer to a existing module. Would this be necessary?

Configuration like repr would be defined only with the initial Sentinel object. It wouldn't get stored in the reduce function. This allows changes in repr to be reflected in unpickled sentinels. This means that removing repr later will not break serialization.

[HexDecimal]

I've replaced the __reduce__ method with a version using pickle's singleton support. This is the most conservative and inoffensive option for a new reduce function since it doesn't add a custom unpickle function, is forward compatible with any future method of pickling, is the standard method of handling singletons via pickle, and is generally strict. This does not support pickling anonymous sentinels which will now raise pickle.PicklingError instead of TypeError. I can revert this if the previous behavior was more desired, but I personally only need to pickle sentinels which have a top-level definition, and the more I work with them the more that anonymous sentinels feel counter intuitive.

I've attempted to add to the discussion of PEP 661. Anonymous sentinels bring a lot of issues which need to be addressed.
https://discuss.python.org/t/pep-661-sentinel-values/9126/251

Before this is merged I'd like to add a strict keyword parameter to Sentinel which prevents sentinels from being accidentally named after existing top-level objects which are not Sentinel as well as ensuring that repr is never given conflicting values. I'll need feedback on if this is appropriate.

[Viicos]

Thanks @HexDecimal. I just tested your implementation in Pydantic to implement an UNSET sentinel, and pickling seems to work as expected.

[JelleZijlstra]

Which of these makes the most sense for the above code?

I feel strongly that 2 is the right behavior. Constructing a class shouldn't look around in the globals for other stuff that might be using the same name.

[HexDecimal]

Constructing a class shouldn't look around in the globals for other stuff that might be using the same name.

That was necessary to handle unpickling until I finally implemented the correct reduce function, but at this point I can remove that code now and revert to the old behavior. I'd accept that the other options are unnecessarily handholdy.

[codecov]

Codecov Report

❌ Patch coverage is 98.79518% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 97.38%. Comparing base (83caa59) to head (cf7f6f2).

Files with missing lines	Patch %	Lines
src/typing_extensions.py	97.82%	1 Missing ⚠️

@@           Coverage Diff           @@
##             main     #617   +/-   ##
=======================================
  Coverage   97.38%   97.38%           
=======================================
  Files           3        3           
  Lines        7690     7731   +41     
=======================================
+ Hits         7489     7529   +40     
- Misses        201      202    +1     

Flag	Coverage Δ
3.10	89.03% <98.79%> (+0.04%)	⬆️
3.10.4	89.03% <98.79%> (+0.04%)	⬆️
3.11	88.26% <92.77%> (+0.04%)	⬆️
3.11.0	87.50% <92.77%> (+0.05%)	⬆️
3.12	88.21% <87.95%> (+0.04%)	⬆️
3.12.0	88.20% <87.95%> (+0.04%)	⬆️
3.13	81.74% <87.95%> (+0.08%)	⬆️
3.13.0	82.47% <87.95%> (+0.08%)	⬆️
3.14	78.19% <87.95%> (+0.10%)	⬆️
3.9	89.73% <87.95%> (+0.04%)	⬆️
3.9.12	89.73% <87.95%> (+0.04%)	⬆️
pypy3.10	88.86% <98.79%> (+0.04%)	⬆️
pypy3.11	88.12% <92.77%> (+0.05%)	⬆️
pypy3.9	89.57% <87.95%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/test_typing_extensions.py	98.40% <100.00%> (+<0.01%)	⬆️
src/typing_extensions.py	93.96% <97.82%> (+<0.01%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[HexDecimal]

I've narrowed the scope of this PR down to only what's needed to support pickling sentinels. Unfortunately this still means that repr needs to be touched to allow for the module_name parameter. Thankfully a lot of other changes were no longer needed after this latest rebase.

Has anyone mentioned that _marker is a terribly obfuscated name for a sentinel?

[JelleZijlstra]

This is mostly good but we shouldn't add a module_name argument since PEP-661 doesn't have that; see #742.

@HexDecimal thanks for the persistent work here! Are you interested in the remaining work of making Sentinel forward-compatible with the accepted version of PEP 661, as outlined in #742?

[HexDecimal]

I attempted to deprecate the Sentinel class using a @deprecated decorator but this broke Python 3.11.15 specifically, not 3.11.0 or 3.12.x or any other version.

[HexDecimal]

• Renamed Sentinel to sentinel, Old name kept, I wanted to deprecate the old name but was prevented due to this error in Python 3.11.15, this specific version does not handle PEP 702 deprecation of classes.
• Deprecated the repr parameter and removed it from the documentation. Parameter deprecations are enforced via PEP 702 style deprecations, including enforcing name as a positional-only parameter.
• Deprecated subclassing of sentinel
• Renamed _name attribute to __name__
• Strongly discouraged assigning attributes or using __weakref__ via the documentation. I'm assuming this will switch to __slots__ later.

Hopefully the code quality here is acceptable. Skipping deprecations would've cleaned this up a lot. Accounting for PEP 702 ahead of time would've prevented code from being shuffled around.

[JelleZijlstra]

We should keep it named as Sentinel, I don't think it's worth trying to rename it. We'll just have the slightly odd case that typing_extensions.Sentinel == builtins.sentinel.

[HexDecimal]

Using the correct name is important. Switching from typing_extensions to native Python and backporting to older versions should be a simple addition or removal of from typing_extensions import sentinel. Using Sentinel with the wrong case needs to be actively discouraged or else things will get worse for code migration.

[JelleZijlstra]

Fair, I'd like to hear more opinions on this though. I'll post in the issue.