Difficulty building from source, my RPMs

laurent · 2021-08-18T21:59:52.904Z

We use node 16 and npm 7 now so you might want to try that.

james-carroll · 2021-08-18T22:04:06.046Z

npm ERR! gyp ERR! stack Error: Command failed: /usr/bin/python3

As far as I'm aware, Joplin actually requires specifically Python 2 to build (or for me it has anyway).

For my builds, setting Python 3 modules via $PYTHONPATH broke it, and my build packages specifically mentions Python 2, so I assume past me had reasons for it.

t0dd · 2021-08-18T22:16:47.392Z

Laurent: I will attempt to force node16. Thanks.

James: I had to force python3 on RHEL in the past for Joplin to build. Hmm... I will experiment.
-todd

james-carroll · 2021-08-18T22:21:02.849Z

I wonder if there's some confusion caused by having a python binary directly.

I think Debian/Ubuntu decided to ship without a python binary and use exclusively python3 in the default installation, leaving python implicitly Python 2. I'm unsure with FedorySuseCentEnterprise editions, but I could image if anything relied on #!/usr/bin/env python hashbangs or similar, it could cause differing results if python == python3

t0dd · 2021-08-18T22:42:21.300Z

Indeed. Some scripts call, for example, python ... no version specified. On most modern OSes, that will be python 3. On RHEL 8 that was python 2. For the SQLite portion of the build, it called python, but expected python 3. So, for RHEL 8, I symlinked /usr/bin/python3 to $HOME/.local/bin/python. Kludgey and terrible, but worked. Now, there seems to be some other novelty going on within the builds that ... well ... ugh. Python 2 is dead and buried. It's been end-of-lifed for a good long while now. So, if this other script is requiring python 2 and then yet another wants python 3. I dunno. We'll see. I haven't been this stymied in a very long time.

james-carroll · 2021-08-18T22:53:49.401Z

So I've just experimented on my end, based on Ubuntu 20.04, without the Python 2 environment (and hence no python binary in $PATH), I got the following

npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
npm ERR! gyp ERR! stack     at PythonFinder.failNoPython (/root/parts/joplin/build/packages/tools/node_modules/node-gyp/lib/configure.js:484:19)
npm ERR! gyp ERR! stack     at PythonFinder.<anonymous> (/root/parts/joplin/build/packages/tools/node_modules/node-gyp/lib/configure.js:406:16)
npm ERR! gyp ERR! stack     at F (/root/parts/joplin/build/packages/tools/node_modules/node-gyp/node_modules/which/which.js:68:16)
npm ERR! gyp ERR! stack     at E (/root/parts/joplin/build/packages/tools/node_modules/node-gyp/node_modules/which/which.js:80:29)
npm ERR! gyp ERR! stack     at /root/parts/joplin/build/packages/tools/node_modules/node-gyp/node_modules/which/which.js:89:16
npm ERR! gyp ERR! stack     at /root/parts/joplin/build/packages/tools/node_modules/isexe/index.js:42:5
npm ERR! gyp ERR! stack     at /root/parts/joplin/build/packages/tools/node_modules/isexe/mode.js:8:5
npm ERR! gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:198:21)
npm ERR! gyp ERR! System Linux 5.8.0-63-generic
npm ERR! gyp ERR! command "/snap/node/5252/bin/node" "/root/parts/joplin/build/packages/tools/node_modules/.bin/node-gyp" "rebuild"
npm ERR! gyp ERR! cwd /root/parts/joplin/build/packages/tools/node_modules/sharp
npm ERR! gyp ERR! node -v v16.7.0
npm ERR! gyp ERR! node-gyp -v v3.8.0
npm ERR! gyp ERR! not ok

This was a pretty huge hint, so I set $PYTHON = python3, and then I got the same error as you did above. I tried again with both Python3 and Python2 in the environment, but `PYTHON=python", and in this situation the build worked properly.

So it seems to me, Joplin does have a strict build dependency on Python 2 which might be going un-noticed generally for the time being, but probably wants looking at given as you say, it's well past being EOL.

For the time being, it looks like you could set the PYTHON variable to point to a sideloaded Python2 installation and maybe it'll fix this for you, or potentially modify $PATH so that python == python2 for the Joplin build specifically. It's not the mix of Py2 and 3 generally that is breaking it, but probably just an odd instance of something having a specific dependency on Py2 and assuming it's simply python.

t0dd · 2021-08-19T01:42:44.837Z

There is some deep magic going on the the RPM build environment. /usr/bin/python3 gets forced by "something" no matter what I do. Outside of RPM, sure ... no problem. But within, no. And Red Hat and Fedora spout in 1-zillion places: "Thou shalt not build with python2 anymore." Grr. Python 2 hasn't been used in any of the Joplin builds for a long time. Maybe a year or more. Ugh.

This may be the end of my maintenance of this RPM build set. A lot of folks use it, unfortunately. I'm tired of fighting nuances to language versions, and that one script called by a script within a script within some node tree of depends ... and twitchy build environments. Maybe for a short while I will just repackage the upstream binaries into RPMs. I hate to do that (not native built) but ... whatever.

I'm tired and grumpy. Maybe I will feel differently tomorrow. G'night.

tessus · 2021-08-19T07:28:13.407Z

Ok, so I probably should update Toolchain issues: npm 7 causes problems

Can you please tell me from which Joplin version npm 7 will work?

laurent · 2021-08-19T09:17:38.205Z

It should be from 2.3 I believe, that's when all the package-lock files were upgraded to v2.

t0dd · 2021-08-19T16:19:27.080Z

More that, python 2 causes issues. Heck, OS venders don't even ship the full stack for that version. It's node module sharp (at least, I think that is the cuprit) that is triggering this. When I get back to a computer, I'm going to see if there is a newer version or some such that can be used.

UPDATE1: No ... sharp-libvips seems to not be where that python2 call is coming from.

tessus · 2021-08-19T18:32:01.192Z

afaik it is a module, not Joplin itself. I think we had such a topic before. I'll search for it a bit later tonight.

t0dd · 2021-08-20T15:20:40.695Z

The emails are starting to roll in noting an incompatibility between desktop and mobile. I will probably have to emergency package with the upstream binaries and then try to solve this another time.

laurent · 2021-08-20T17:23:28.725Z

This issue makes me realise that sync target upgrades will always be quite tricky. Even if I release all the apps at the same time, there are plenty of non-official channels that are updated with a delay.

I wonder if there would be a way to notify all package maintainers in advance, and then wait for them to be ready before doing the release. I guess just a thread on the forum wouldn't quite work since not everyone will read it. @t0dd, @james-carroll, do you have any suggestion on how I could let you know about these upgrades?

sttrebo · 2021-08-20T17:31:06.439Z

I use (at work) s/w versioning of major.minor.patch.githash (x.y.x.a) scheme. for me, the simplest approach would be to consider any sync target upgrades as a major change so the major version (x) would be incremented.

james-carroll · 2021-08-20T18:18:08.382Z

I don't think it's unreasonable to ask all the third party maintainers to make forum accounts, put them all in a group, and ping them once a year. If they're unwilling to put in the effort there, IMO they're treating the package manager responsibility poorly.

I know generally speaking the updates aren't usually incompatible in the way sync-target upgrades have been, but especially with Joplin Cloud (hopefully?) becoming one of the main ways of funding the project, it becomes more serious to consider that you're actually representing someone elses brand and work, and potentially, damaging someone elses brand and work.

I'm perfectly aware the majority of users of the snap for example can't usually tell it's a third party package, which is why I actually specifically come on and try to root out mistakes or problems it's creating uniquely, because it actually presents itself as Joplin Desktop. If they don't want to make themselves available like that, they should call themselves Joplisn't instead.

Personally however, I don't find it too hard to go through every project I'm partially responsible for and keep myself up to date with what's going on, so I don't think I've ever been more than a few hours out of sync except where I might be particularly busy on the day. It isn't that much to ask for, and CI systems exist for a reason anyway.

t0dd · 2021-08-20T19:30:48.056Z

I'm pretty good about keeping up myself, but sometimes even I am AFK for a month occasionally (I once took 4 months off from computers, but that was years ago).

I package element (chat app) as well. They have a private chat room just for package maintainers. I suppose, like James said, you could identify those who do stuff like I do and have us all in one chat room, or private forum channel, or something, and just ping us when we need to know something particularly important. I have the github release page RSS feed tied to a chat channel and I watch for every release. But that is just me.

Maybe just gather our email addresses and send us an IMPORTANT!!! subject-lined email? Dunno.

t0dd · 2021-08-21T00:33:01.845Z

UPDATE: I successfully built packages using the upstream binary. I can't seem to solve this particular issue, but at least my userbase is not dead in the water.

laurent · 2021-08-25T09:59:57.124Z

Thanks for the feedback. I think as a first step I'll make a list of third-party packages with point of contact so that if such situation happens it will be easier to reach out. I think the F-Droid and Chocolatey packages are fully automated but not sure about the Homebrew one in particular.

@t0dd, @james-carroll, how about your package? Is it published automatically when a new version is released or do you need to manually do it?

james-carroll · 2021-08-25T10:46:51.048Z

I could automate it the entire way but I like seeing the patch notes and doing some manual testing first, so it's been manual the past few months. It is probably overkill though, the builds have been reliable basically since the start aside from some odd changes to the build environment.

But I am considering arm64 builds and as long as GitHub doesn't have free runners for arm64, that adds some restrictions on my ability to automate the process.

t0dd · 2021-08-25T13:09:26.649Z

@laurent Similarly, I am sure it could be largely automated, but the current process is not egregious. This is the general process:

Note, COPR is a Fedora Project -sponsored build system that properly isolates package builds (the build user doesn't have access to root, the system is a clean chroot, etc). You can even turn off your build's access to the internet, which if it were a truly proper build, that is what I would do, but ... nodejs makes that very challenging. OpenSUSE also has a similar build system available to the public which is very good.

If minor update: e.g., 2.2.6 to 2.2.7

Bump the release in joplin.spec, but mark it as a test build.
Update the release log in contributed org.joplinapp.joplin.metainfo.xml file. Reference my contribs here.
Rebundle and commit the joplin-2.2-contrib.tar.gz to include that updated .metainfo.xml file.
Upload .spec to COPR build system TEST repo, and press a button to build for CentOS (8, 9), RHEL (8), Fedora (33, 34, rawhide, and the upcoming 35), and then OpenSUSE (Leap 15.2, 15.3, and Tumbleweed).
Resolve any failure ... there haven't been many until 2.3.3 when we now see a Python2 regressive requirement.
Test drive Joplin test build on my own machine. I have reached out to the community to test on OpenSUSE in the past, but I haven't done that in awhile because ... it didn't need to be done.
Passes testing, then I flip the testing bit to off in the .spec file.
Upload .spec to COPR build system PRODUCTION repo and build to all those platforms (but a reduced set).
Wait for any complaints from the community in case I broke something. This is rare but it happens quickly since the COPR repos automate delivery to everyone's desktops using their package managers. This is the whole reason I do this, so that folks can update their desktops and Joplin just shows up all properly packaged and tidy.

For major releases, e.g., 2.2.7 to 2.3.3, I only add more testing. And in this case (2.3.3 ad 2.3.5) I had to revamp the spec file to use the upstream binary pre-built AppImage as the "source", etc.

And it should be noted that I don't make unblessed tagged releases available to the general public, but I do like to test some pre-release builds and whatnot. Especially if it is a .Y release coming up.

If the upstream source is hunky-dorey, this process takes 15 minutes of manual labor, not including my testing. And probably an hour of actual time (waiting for the builds to complete, downloading, testing, and then once accepted, waiting for production builds to complete).

So ... can all this be automated? Hands-free? With the last year or so of builds, mostly. The last version of course not unless I accept pre-builds as the norm moving forward (I don't want to do that). The actual building process is not hard if running smoothly. But I do like to test before I hand packages to strangers. I have had ... "incidents" ... in my dark past where I broke people (only a couple minor moments with Joplin). A lot of people. I don't want to ever do that again.

Just added this process to my github readme. And it should be noted that I am only building and packaging Joplin desktop. I don't test or build the CLI or server.