Page MenuHomeSolus

Update Python to 3.7.0
Open, WishlistPublic

Tokens
"Yellow Medal" token, awarded by ermo."Party Time" token, awarded by Jacalz."Party Time" token, awarded by davidjharder."Like" token, awarded by algent."Love" token, awarded by maverick1."Yellow Medal" token, awarded by kyrios123.
Assigned To
Authored By
yaomtc, Aug 22 2018

Description

https://www.python.org/downloads/release/python-370/

Among the major new features in Python 3.7 are:

  • PEP 539, new C API for thread-local storage
  • PEP 545, Python documentation translations
  • New documentation translations: Japanese, French, and Korean.
  • PEP 552, Deterministic pyc files
  • PEP 553, Built-in breakpoint()
  • PEP 557, Data Classes
  • PEP 560, Core support for typing module and generic types
  • PEP 562, Customization of access to module attributes PEP 563, Postponed evaluation of annotations PEP 564, Time functions with nanosecond resolution
  • PEP 565, Improved DeprecationWarning handling PEP 567, Context Variables
  • Avoiding the use of ASCII as a default text encoding (PEP 538, legacy C locale coercion and PEP 540, forced UTF-8 runtime mode)
  • The insertion-order preservation nature of dict objects is now an official part of the Python language spec.
  • Notable performance improvements in many areas.

Please see What’s New In Python 3.7 for more information.

Source tarball: https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
MD5: eb8c2a6b1447d50813c02714af4681f3

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Note that python 3.7 is not backwards compatible with 3.6

In particular PEP 479 changes will compile and fail at runtime. Other changes I believe will cause it to fail to build/install (which would only apply to those packages using python setup.py).

Note that python 3.7 is not backwards compatible with 3.6

Would that necessitate a separate package, then, until some indeterminate future date?

Not usually. We will likely be updating python3 once it has been determined that all of our python3 packages are unaffected by the changes.

JoshStrobl triaged this task as Wishlist priority.Oct 16 2018, 11:30 PM
JoshStrobl moved this task from Backlog to Improvement on the Software board.

How is this coming along? It's not that I need 3.7, but a couple of months have passed now, so I was just curious to see if there was any update on this :)

I guess this will become a priority soon as from next release the branch 3.6 won't be supported anymore.

dlukes added a subscriber: dlukes.Apr 14 2019, 7:30 AM

It will be quite funny if Debian has python 3.7 stable before Solus 😛 (Buster should be shipping with 3.7 in the next few months)

Updating python3 is not a small task. Over 100 packages need to be tested.

Security fixes on Python 3.6 will be shipped until december 2021 (as per PEP 494).

@DataDrake That's … a lot. Is there any (easyish) way for me to help?

Updating python3 is not a small task. Over 100 packages need to be tested.

I can help too!

Can't we reduce the set of python packages to test by running their unit tests?

kyrios123 added a comment.EditedMay 21 2019, 11:40 AM

One of the reason for which I don't even try is that many packages depends on each other; like package A depends on package B and package B depends on package A (at least for the unit tests)... So it's not obvious to determine the rebuild order. I guess some unit tests have to be temporarily disabled during such exercices.

Repeat after me kids: "Unit tests are not integration tests!"

I'm not expecting anything earth-shattering, but upgrading to 3.7 will require 100% rebuilds and unit tests won't cover everything.

Btw, when do we start removing python2 support? I mean, I wanted to remove python2 support on some packages but packages like spyder are stopping it. When do we start nuking them since nobody use spyder instead of spyder3?

People do use spyder. We get reminded every time it breaks. It'll be among the last thing to go, along with ypkg2, eopkg, and the solus-sc.

Tzigamm added a subscriber: Tzigamm.Aug 1 2019, 2:58 PM

Blender 2.80 (out yesterday) requires Python >=3.7, what can be done to update? I need the latest version of blender ASAP but I have a lot of free time to compile and test, should we create a metatask with every package that needs testing?

JoshStrobl added a subscriber: JoshStrobl.EditedAug 1 2019, 3:07 PM

If it needs the latest version of python 3, then you're not going to get it ASAP. Upgrading to the latest release of Python (2 and 3) will require us to update OpenSSL. Updating OpenSSL involves also keeping the existing version around, both in the same upgrade (so shipping both at the same time, in the same package, to ensure no libs are removed and ABIs pulled out from underneath python's feet). This has to be done to ensure the package manager continues to function, so we can perform the necessary package transactions to get the dependencies to build (and even rebuild) Python 2/3 in the first place. And all of that will be done at the same time I'm updating systemd.

So, don't expect it in the immediate future.

Sure, ASAP means As Soon As Possible, meaning if the soonest possible is 3 months it will do, it's just a way to say I'll do my maximum to help.

I confirm Blender now expects python 3.7 see here.

It seems just updating the package.yml file is enough to compile python, but some tests don't pass, probably due to it not being backward compatible so we need to adapt the tests.

After eopkg installing it, python3 -V returns the expected value (Python 3.7.4), is there a way to get a list of every package that need python3 and start compiling+testing?

ermo added a subscriber: ermo.Aug 2 2019, 11:41 AM

If it needs the latest version of python 3, then you're not going to get it ASAP. Upgrading to the latest release of Python (2 and 3) will require us to update OpenSSL. Updating OpenSSL involves also keeping the existing version around, both in the same upgrade (so shipping both at the same time, in the same package, to ensure no libs are removed and ABIs pulled out from underneath python's feet). This has to be done to ensure the package manager continues to function, so we can perform the necessary package transactions to get the dependencies to build (and even rebuild) Python 2/3 in the first place. And all of that will be done at the same time I'm updating systemd.
So, don't expect it in the immediate future.

Holy ...

That's a lot of potential breakage right there. It sort of sounds like an all-hands-on-deck scenario?

Would it make sense to plan some sort of py-3.7.0 hackfest where people can chip in and confirm whether their pet package works once you land (the unholy trifecta of) systemd, openssl and py3.7 in -unstable?

Updating python3 itself will require mass rebuilds to update the site-packages path and yea, to test for breakage related to that specific upgrade. However I intend on having the OpenSSL and systemd upgrades occur much more prior to a Python 3.7 upgrade and so those will be less of a concern, rebuilds of the existing python 2 and 3 versions will occur in respect to those updates.

In general, Python 3.7 introduces no breaking changes compared to Python 3.6. So, any Python code which works on Python 3.6 expected to work on 3.7 as well. So, IMHO, upgrading Python to 3.7 is pretty much safe operation. I also don't see the need of rebuilding _all_ the packages of python-* series - they should continue to work.

Thank you for your opinion on the matter, however yes we will need to perform rebuilds of all python packages which reference /usr/lib/python3.6/*, because we will want and need to move those over to /usr/lib/python3.7/ when the upgrade does occur. I'm not satisfied with "should" when it comes to stack upgrades, I deal with absolutes.

Oh, I see. In fact, this is completely true - library path changes, so all packages must be rebuilt.

tribals added a comment.EditedAug 2 2019, 1:17 PM

I'm very interested in upgrading Python to 3.7, so I'm willing offer any help I can. Probably "Python 3.7" hackfest is worth considering to happen :)

While I appreciate the offer, this is something that myself, Core Team, and Global Maintainers should handle.

dlukes removed a subscriber: dlukes.Aug 2 2019, 1:23 PM

@tribals if you really want to help here is what you can do :

Search for python 3 packages and update them to their latest version : some package in the repo might be too old to build with 3.7 so having updated packages will help making the rebuilds smoother. But be careful, when a python package is updated the other packages that uses it must be tested because in the python magic world, it is not rare that developers put strict version check and not just on minimal version but also on maximum version or on some specific versions.

@kyrios123 you mean python-* packages or packages that list python3 as a build dep ?

@kyrios123 you mean python-* packages or packages that list python3 as a build dep ?

He means packages that contains %python3_install, that is, python packages that use Python3.

Is there a way to parse all the package.yml files in the repo to generate an actual list so we can organize everything too?

Also, what should we do for each package? Create a new task where everything is documented, just submit a patch?

The only way I know is cloning all the repo locally then parse them.
You can already start with the component programming.python you'll miss lot of packages and you'll get quite many python2 only stuffs but it's a start and it prevents you from cloning everything.
You can also have a look at the reverse dependencies of python3 to complete your list

Maybe someone from the core team with ssh access to the actual repo could give us a hand? In theory it should be just some greping, I can create a script to do it if everyone is already busy doing systemd and OpenSSL.

Nobody is doing openssl and systemd. These are just things that will have to be done before updating python to 3.7+

ermo added a comment.Aug 2 2019, 9:16 PM

@Tzigamm:

Also note that Josh is currently swamped with IRL stuff, so don't expect him to land the OpenSSL and systemd stuff right away.

Just trying to manage your expectations here -- your enthusiasm and offer to help out with the transition is much appreciated!

Yeah don't worry about my expectations, I'm guessing the test of every python-based package will probably take a while so I know it's not gonna be included in next sync, or probably in a while! It's okay though because apparently the only package that actually requires it (atm) seems to be Blender, and it can be downloaded from their website. But since I have 3-4 weeks of free time I thought I'd give back to the community!

ermo added a comment.Aug 2 2019, 9:44 PM

@Tzigamm

I just noticed this: https://flathub.org/apps/details/org.blender.Blender

Maybe it'd make sense for you to try that out in the meantime?

Sure it's possible, but blender's update schedule is not fast enough the next update will be out before we get python working (I'm talking probably 6 months) so don't worry about me, the gziped binaries are good for me, let's try and focus on python ;)

Okay so if someone with access to the repo (or a mirror of it) could run grep -iRl "%python3_install" ./*/package.yml in the repo's .solus folder (or wherever the packages' folders are located) it would help us a lot, it will output a list of every package that needs to be rebuilt and tested.

Also, I understood that after a successful solbuild, the new python version goes through a series of tests, and due to some changes in 3.7 some of the tests don't pass anymore. Did the Solus team create the tests?

You'd better search for python3 and you should probably search in both package.yml and abi_used_libs

Okay, this should work, and only print each package once: dirname $(grep -iRl "python3" ./*/{package.yml,abi_used_libs}) | sort | uniq

Tzigamm added a subscriber: joebonrichie.EditedAug 5 2019, 1:33 PM

thanks to @joebonrichie we have have this list, which contain every package that need testing with python 3.7: https://pastebin.com/raw/HG80U5Qq

EDIT: I tried to format a list from it to cross each line after testing but... It's A LOT: 589 packages. We need a way to cross them without having a message in the tracker so big you need 10 seconds to scroll through it, probably a Google Sheets, or an alternative to it

Here is the link to a ODS file I formatted: for each package, anyone can specify if they tested the package with the new python version. It's running on my personal server which is rocking an Intel Atom so don't ask too much of it.

https://cloud.be-ta.art/s/DKJP9jF5odeF2P6

Due to dependency hell I've started working on this. Currently at ~100/600 rebuilds.

go @joebonrichie go !!!
I never dared to even try.

@joebonrichie thanks for claiming it!

algent awarded a token.Sep 6 2019, 1:08 PM

@joebonrichie You absolute madman. Godspeed.

Rebuilds are mostly done with the exception of things I can't build locally like tensorflow and graph-tool (582/588).

TODO:
proj.4 stack upgrade to 6.2.0 (requires rebuilds and rename/replaces)
opencv required an update to 3.4.7 (requires safety rebuilds)

Other misc packages that required updating. I'll push these updates ahead of the rebuild

BLOCKERS:
libopenshot and openshot-qt are waiting on a new release of libopenshot-audio so it can build against gcc9.

Blockers are resolved. Rebasing and rebuilding against updates in unstable and starting more thorough testing.

Jacalz awarded a token.Thu, Oct 3, 5:57 PM
ermo awarded a token.Mon, Oct 7, 12:08 AM

New rebuild list. Now over 600 packages. https://pastebin.com/raw/WEbQgVjx. Rebased everything against the GNOME update and have had python3.7 and the rebuilds installed on my system for a few days now without any problems noticed, just trying my best to look for any breakages and missing packages currently.

I am thinking it is probably wise to wait a couple of weeks before landing this as small issues with the GNOME update are still coming out of the woodwork. Also, Josh will probably still want to land openssh and systemd upgrades before landing this.

Jacalz added a subscriber: Jacalz.Mon, Oct 7, 11:54 AM

@joebonrichie , your rebuild list doesn't include cutter? It depends on Python 3 too (see abi_use_libs), and it will need to be rebuilt after stack upgrade lands.

@aleksvor thanks for the catch. It didn't get pulled in when i ran make clone on the repo for some reason. I'll do some digging on why make clone isn't pulling down all the packages.

Thanks, I updated the package list per rCOMN7649dd796c13 and added the missing ones to the list.

Just an update on this, working on updating everything in eopkg info -c programming.python - especially core python items like python-setuptools, python-wheel, python-pytest to make sure everything is fully 3.7 ready. Validating things locally and pushing things one by one. Progress is slower than hoped for, as my hatred for python is growing by the minute.