It will be quite funny if Debian has python 3.7 stable before Solus ? (Buster should be shipping with 3.7 in the next few months)

Updating python3 is not a small task. Over 100 packages need to be tested.

Security fixes on Python 3.6 will be shipped until december 2021 (as per PEP 494).

@DataDrake That's … a lot. Is there any (easyish) way for me to help?

In T6817#151149, @DataDrake wrote:

Updating python3 is not a small task. Over 100 packages need to be tested.

I can help too!

Can't we reduce the set of python packages to test by running their unit tests?

One of the reason for which I don't even try is that many packages depends on each other; like package A depends on package B and package B depends on package A (at least for the unit tests)... So it's not obvious to determine the rebuild order. I guess some unit tests have to be temporarily disabled during such exercices.

Repeat after me kids: "Unit tests are not integration tests!"

I'm not expecting anything earth-shattering, but upgrading to 3.7 will require 100% rebuilds and unit tests won't cover everything.

Btw, when do we start removing python2 support? I mean, I wanted to remove python2 support on some packages but packages like spyder are stopping it. When do we start nuking them since nobody use spyder instead of spyder3?

People do use spyder. We get reminded every time it breaks. It'll be among the last thing to go, along with ypkg2, eopkg, and the solus-sc.

Blender 2.80 (out yesterday) requires Python >=3.7, what can be done to update? I need the latest version of blender ASAP but I have a lot of free time to compile and test, should we create a metatask with every package that needs testing?

If it needs the latest version of python 3, then you're not going to get it ASAP. Upgrading to the latest release of Python (2 and 3) will require us to update OpenSSL. Updating OpenSSL involves also keeping the existing version around, both in the same upgrade (so shipping both at the same time, in the same package, to ensure no libs are removed and ABIs pulled out from underneath python's feet). This has to be done to ensure the package manager continues to function, so we can perform the necessary package transactions to get the dependencies to build (and even rebuild) Python 2/3 in the first place. And all of that will be done at the same time I'm updating systemd.

So, don't expect it in the immediate future.

Sure, ASAP means As Soon As Possible, meaning if the soonest possible is 3 months it will do, it's just a way to say I'll do my maximum to help.

I confirm Blender now expects python 3.7 see here.

It seems just updating the package.yml file is enough to compile python, but some tests don't pass, probably due to it not being backward compatible so we need to adapt the tests.

After eopkg installing it, python3 -V returns the expected value (Python 3.7.4), is there a way to get a list of every package that need python3 and start compiling+testing?

In T6817#155282, @JoshStrobl wrote:

If it needs the latest version of python 3, then you're not going to get it ASAP. Upgrading to the latest release of Python (2 and 3) will require us to update OpenSSL. Updating OpenSSL involves also keeping the existing version around, both in the same upgrade (so shipping both at the same time, in the same package, to ensure no libs are removed and ABIs pulled out from underneath python's feet). This has to be done to ensure the package manager continues to function, so we can perform the necessary package transactions to get the dependencies to build (and even rebuild) Python 2/3 in the first place. And all of that will be done at the same time I'm updating systemd.

So, don't expect it in the immediate future.

Holy ...

That's a lot of potential breakage right there. It sort of sounds like an all-hands-on-deck scenario?

Would it make sense to plan some sort of py-3.7.0 hackfest where people can chip in and confirm whether their pet package works once you land (the unholy trifecta of) systemd, openssl and py3.7 in -unstable?

Updating python3 itself will require mass rebuilds to update the site-packages path and yea, to test for breakage related to that specific upgrade. However I intend on having the OpenSSL and systemd upgrades occur much more prior to a Python 3.7 upgrade and so those will be less of a concern, rebuilds of the existing python 2 and 3 versions will occur in respect to those updates.

In general, Python 3.7 introduces no breaking changes compared to Python 3.6. So, any Python code which works on Python 3.6 expected to work on 3.7 as well. So, IMHO, upgrading Python to 3.7 is pretty much safe operation. I also don't see the need of rebuilding _all_ the packages of python-* series - they should continue to work.

Thank you for your opinion on the matter, however yes we will need to perform rebuilds of all python packages which reference /usr/lib/python3.6/*, because we will want and need to move those over to /usr/lib/python3.7/ when the upgrade does occur. I'm not satisfied with "should" when it comes to stack upgrades, I deal with absolutes.

Oh, I see. In fact, this is completely true - library path changes, so all packages must be rebuilt.

I'm very interested in upgrading Python to 3.7, so I'm willing offer any help I can. Probably "Python 3.7" hackfest is worth considering to happen :)

While I appreciate the offer, this is something that myself, Core Team, and Global Maintainers should handle.

@tribals if you really want to help here is what you can do :

Search for python 3 packages and update them to their latest version : some package in the repo might be too old to build with 3.7 so having updated packages will help making the rebuilds smoother. But be careful, when a python package is updated the other packages that uses it must be tested because in the python magic world, it is not rare that developers put strict version check and not just on minimal version but also on maximum version or on some specific versions.

@kyrios123 you mean python-* packages or packages that list python3 as a build dep ?

In T6817#155413, @Tzigamm wrote:

@kyrios123 you mean python-* packages or packages that list python3 as a build dep ?

He means packages that contains %python3_install, that is, python packages that use Python3.

Yep like specified in the doc :)

Is there a way to parse all the package.yml files in the repo to generate an actual list so we can organize everything too?

Also, what should we do for each package? Create a new task where everything is documented, just submit a patch?

The only way I know is cloning all the repo locally then parse them.
You can already start with the component programming.python you'll miss lot of packages and you'll get quite many python2 only stuffs but it's a start and it prevents you from cloning everything.
You can also have a look at the reverse dependencies of python3 to complete your list

Maybe someone from the core team with ssh access to the actual repo could give us a hand? In theory it should be just some greping, I can create a script to do it if everyone is already busy doing systemd and OpenSSL.

Nobody is doing openssl and systemd. These are just things that will have to be done before updating python to 3.7+

@Tzigamm:

Also note that Josh is currently swamped with IRL stuff, so don't expect him to land the OpenSSL and systemd stuff right away.

Just trying to manage your expectations here -- your enthusiasm and offer to help out with the transition is much appreciated!

Yeah don't worry about my expectations, I'm guessing the test of every python-based package will probably take a while so I know it's not gonna be included in next sync, or probably in a while! It's okay though because apparently the only package that actually requires it (atm) seems to be Blender, and it can be downloaded from their website. But since I have 3-4 weeks of free time I thought I'd give back to the community!

@Tzigamm

I just noticed this: https://flathub.org/apps/details/org.blender.Blender

Maybe it'd make sense for you to try that out in the meantime?

Sure it's possible, but blender's update schedule is not fast enough the next update will be out before we get python working (I'm talking probably 6 months) so don't worry about me, the gziped binaries are good for me, let's try and focus on python ;)

Okay so if someone with access to the repo (or a mirror of it) could run grep -iRl "%python3_install" ./*/package.yml in the repo's .solus folder (or wherever the packages' folders are located) it would help us a lot, it will output a list of every package that needs to be rebuilt and tested.

Also, I understood that after a successful solbuild, the new python version goes through a series of tests, and due to some changes in 3.7 some of the tests don't pass anymore. Did the Solus team create the tests?

You'd better search for python3 and you should probably search in both package.yml and abi_used_libs

Okay, this should work, and only print each package once: dirname $(grep -iRl "python3" ./*/{package.yml,abi_used_libs}) | sort | uniq

thanks to @joebonrichie we have have this list, which contain every package that need testing with python 3.7: https://pastebin.com/raw/HG80U5Qq

EDIT: I tried to format a list from it to cross each line after testing but... It's A LOT: 589 packages. We need a way to cross them without having a message in the tracker so big you need 10 seconds to scroll through it, probably a Google Sheets, or an alternative to it

Here is the link to a ODS file I formatted: for each package, anyone can specify if they tested the package with the new python version. It's running on my personal server which is rocking an Intel Atom so don't ask too much of it.

https://cloud.be-ta.art/s/DKJP9jF5odeF2P6

no pushy :p

go @joebonrichie go !!!
I never dared to even try.

@joebonrichie thanks for claiming it!

Our hero!

@joebonrichie You absolute madman. Godspeed.

Rebuilds are mostly done with the exception of things I can't build locally like tensorflow and graph-tool (582/588).

TODO:
proj.4 stack upgrade to 6.2.0 (requires rebuilds and rename/replaces)
opencv required an update to 3.4.7 (requires safety rebuilds)

Other misc packages that required updating. I'll push these updates ahead of the rebuild

BLOCKERS:
libopenshot and openshot-qt are waiting on a new release of libopenshot-audio so it can build against gcc9.

Blockers are resolved. Rebasing and rebuilding against updates in unstable and starting more thorough testing.

New rebuild list. Now over 600 packages. https://pastebin.com/raw/WEbQgVjx. Rebased everything against the GNOME update and have had python3.7 and the rebuilds installed on my system for a few days now without any problems noticed, just trying my best to look for any breakages and missing packages currently.

I am thinking it is probably wise to wait a couple of weeks before landing this as small issues with the GNOME update are still coming out of the woodwork. Also, Josh will probably still want to land openssh and systemd upgrades before landing this.

@joebonrichie , your rebuild list doesn't include cutter? It depends on Python 3 too (see abi_use_libs), and it will need to be rebuilt after stack upgrade lands.

@aleksvor thanks for the catch. It didn't get pulled in when i ran make clone on the repo for some reason. I'll do some digging on why make clone isn't pulling down all the packages.

Looks like python-pyliblo is missing too. https://dev.getsol.us/source/python-pyliblo/browse/master/abi_used_libs

Thanks, I updated the package list per rCOMN7649dd796c13 and added the missing ones to the list.

Just an update on this, working on updating everything in eopkg info -c programming.python - especially core python items like python-setuptools, python-wheel, python-pytest to make sure everything is fully 3.7 ready. Validating things locally and pushing things one by one. Progress is slower than hoped for, as my hatred for python is growing by the minute.

@joebonrichie

How are things progressing?

Now that Python 3.8.1 has been released for a couple of weeks, should we aim to update to the new version? Or is the plan to update to 3.7.x and then consider 3.8.x?

Given how much time @joebonrichie has already put into this, I'd rather we land 3.7 first. It'll also make 3.8 testing less of a rush and will be easier to find any breakages.

Got it, let's get 3.7 in the repos! Since @joebonrichie's last activity was 2.5 months ago should someone else/a group of people start from scratch or should we wait for him to come back?

@Tzigamm its around 600 packages and he has been working on it right along ever since, even if he hasn't reported that here. These things take time and lots of testing. Started from scratch is a huge waste of time. Be patient please.

Back from the dead. Everything is rebased and rebuilt. Plan is to push it next next Sunday (touches wood). Will re-validate everything over the coming week.

Round of applause for @joebonrichie! ? ? ? ? ?

Will delay by one more day. IRL commitments.

I guess you can close this now... so I can open a new task for python 3.8 ?

technically graph-tool and onionshare still need to be built, but w/e :D