Help Packaging the Joplin for ArchLinux

alfredopalhares · 2022-01-08T17:40:34.126Z

Hello,

I am the Arch Linux Joplin packager, and I am trying to building Joplin 2.6.10 to Arch Linux.

I cannot build the desktop on the 2.6.10 in any shape or form, so I have tried to port some of your Yarn 3 migrations I got it to build with several modifications, I went this way, since its the next build woflow that you guys are using.

I can build the cli just fine (although it needs improvements), but the desktop I just get a blank screen. am getting this error on the console with the debug mode:

Uncaught Error: Cannot find module '@joplin/fork-uslug'
Require stack:
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/node_modules/@joplin/renderer/MdToHtml.js
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/node_modules/@joplin/renderer/MarkupToHtml.js
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/node_modules/@joplin/renderer/index.js
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/markdownUtils.js
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/models/BaseItem.js
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/models/Resource.js
- /usr/share/joplin-desktop/resources/app.asar/node_modules/@joplin/lib/services/ResourceEditWatcher/index.js
- /usr/share/joplin-desktop/resources/app.asar/app.js
- /usr/share/joplin-desktop/resources/app.asar/index.html
    at Module._resolveFilename (internal/modules/cjs/loader.js:892)
    at Function.o._resolveFilename (electron/js2c/renderer_init.js:29)
    at Module._load (internal/modules/cjs/loader.js:737)
    at Function.f._load (electron/js2c/asar_bundle.js:5)
    at Function.o._load (electron/js2c/renderer_init.js:29)
    at Module.require (internal/modules/cjs/loader.js:964)
    at require (internal/modules/cjs/helpers.js:88)
    at Object.<anonymous> (MdToHtml.ts:70)
    at Object.<anonymous> (MdToHtml.ts:610)
    at Module._compile (internal/modules/cjs/loader.js:1083)

This is the build that I am doing:

github.com

alfredopalhares/arch-pkgbuilds/blob/feature/joplin-on-yarn3/joplin/PKGBUILD

# Maintainer: Alfredo Palhares <alfredo at palhares dot me>
# Contributor: Mark Wagie <mark dot wagie at tutanota dot com>
# Contributor:  Matteo Parolari
# Contributor: gardar <aur@gardar.net>

# Please contribute to:
# https://github.com/alfredopalhares/arch-pkgbuilds

pkgbase="joplin"
pkgname=('joplin' 'joplin-desktop')
pkgver=2.6.10
groups=('joplin')
pkgrel=5
install="joplin.install"
depends=('electron' 'gtk3' 'libexif' 'libgsf' 'libjpeg-turbo' 'libwebp' 'libxss' 'nodejs>=17.3'
         'nss' 'orc' 'rsync' 'libvips')
optdepends=('libappindicator-gtk3: for tray icon')
arch=('x86_64' 'i686')
makedepends=('git' 'npm' 'yarn' 'python2' 'rsync' 'jq' 'yq' 'electron' 'libgsf' 'node-gyp>=8.4.1' 'libvips')
url="https://joplinapp.org/"

This file has been truncated. show original

I know that you do not support any other systems other than the AppImage and I understand and respect that, but if you can shed some light with your knowlegde, because I have been at this for several days.

Regards,
Alfredo Palhares

laurent · 2022-01-08T17:46:51.626Z

The "fork-uslug" repo was recently added under /packages so if you have some script that copy specific folders, make sure you copy that one. Also this particular package is not published so you need to get it from the repo.

alfredopalhares · 2022-01-08T17:53:56.783Z

Hello Laurent.

Is it under packages, I applied a patch of this commit Tools: Use Yarn 3 to manage monorepo

And package its build form the yarn command as its listed on the workspaces. But still it seens that I am missing something. Do you have any idea on where should I look for it.

Thank you for your help!
Regards

laurent · 2022-01-08T18:01:22.679Z

Not really unfortunately because your script does a lot of things and patch stuff, so I don't really know where the problem can come from. It says that this particular fork-slug package is missing, so you'd need to check why it's not being included.

Actually you have a line that lists packages there and fork-uslug is not being included, so maybe that's where? arch-pkgbuilds/PKGBUILD at 60c7def2cb97251bd06a521a010c0aec8dc7fd3c · alfredopalhares/arch-pkgbuilds · GitHub

james-carroll · 2022-01-08T20:29:10.946Z

I'm adapting to the yarn changes too, (independent of the AUR) and uslug gives me some errors as well. For me, namely:

Error: uslug@https://github.com/laurent22/uslug.git#commit=8c12bc7678eaefa752e673ea9cfbc0b1a14d7237: EINVAL: invalid argument, rename '/tmp/xfs-5aa2219e' -> '/tmp/xfs-5aa2219e/package.tgz'

There's someone with the same issue on another package here; I'll try to find a resolution and maybe by lucky co-incidence it might help OP

laurent · 2022-01-08T21:40:19.136Z

This package used to be on a separate GitHub repo, but I've moved it to the monorepo now so it's strange you have this reference. Is it a matter of running npm run clean from the root?

james-carroll · 2022-01-08T21:55:36.282Z

It looks like my issue is probably unrelated, though it might cast some doubt on reliability of the node packages from various sources.

I'd been using Node installed as a Snap (which is actually an official release from the NodeJS foundation), and basically it just doesn't work in there. I can't figure it out yet, but frankly it's not the first time I've had extremely "minor" but crippling differences with the Snap package vs the other distributions of Node.

It worked absolutely fine for me when I'd installed Node from NodeSource · GitHub. I used to also just download the tarball release from the NodeJS site directly in the past.

It makes me wonder if there also be some subtle differences in the Arch Node package that just makes it different to other releases, for the purposes of the AUR build script, maybe downloading Node specifically from Download | Node.js - building with it, and discarding it after might produce different results.

(OFC I don't know AUR packaging guidelines so maybe they'd disapprove this)

leematos · 2022-01-09T23:58:11.871Z

I'm actually hitting a similar problem right now just trying to build dev branch on Mac for iOS debugging.

error: Error: Unable to resolve module @joplin/fork-uslug from /Users/l/joplin/packages/renderer/MdToHtml.js: @joplin/fork-uslug could not be found within the project or in these directories:
  ../renderer/node_modules
  ../../node_modules
  /Users/l/joplin/packages/app-mobile/node_modules/@joplin/fork-uslug

If you are sure the module exists, try these steps:
 1. Clear watchman watches: watchman watch-del-all
 2. Delete node_modules and run yarn install
 3. Reset Metro's cache: yarn start --reset-cache
 4. Remove the cache: rm -rf /tmp/metro-*
  34 |     source_map: require('./MdToHtml/rules/source_map').default,
  35 | };
> 36 | const uslug = require('@joplin/fork-uslug');
     |                        ^
  37 | const markdownItAnchor = require('markdown-it-anchor');
  38 | // The keys must match the corresponding entry in Setting.js
  39 | const plugins = {

For my app-mobile/node_modules/[at]joplin dir uslug isn't there:

node_modules % cd [at]joplin 
l@lees-mbp @joplin % ls -lha
total 0
drwxr-xr-x    11 l  staff   352B Jan  9 17:45 .
drwxr-xr-x  1033 l  staff    32K Jan  9 18:38 ..
lrwxr-xr-x     1 l  staff     5B Jan  9 17:45 app-mobile -> ../..
lrwxr-xr-x     1 l  staff    25B Jan  9 17:44 fork-htmlparser2 -> ../../../fork-htmlparser2
lrwxr-xr-x     1 l  staff    17B Jan  9 17:44 fork-sax -> ../../../fork-sax
lrwxr-xr-x     1 l  staff    17B Jan  9 17:44 htmlpack -> ../../../htmlpack
drwxr-xr-x   259 l  staff   8.1K Jan  9 17:45 lib
drwxr-xr-x    48 l  staff   1.5K Jan  9 17:45 renderer
drwxr-xr-x    83 l staff   2.6K Jan  9 17:45 tools
lrwxr-xr-x     1 l  staff    17B Jan  9 17:44 turndown -> ../../../turndown
lrwxr-xr-x     1 l  staff    28B Jan  9 17:44 turndown-plugin-gfm -> ../../../turndown-plugin-gfm

I looked at app-desktop/node_modules and I don't see it there either, but the desktop app builds for me now.

as a node/node packaging newb I assume there is magic here to build this type of package? (gonna go do some reading now)

laurent · 2022-01-10T01:04:08.075Z

npm run clean && yarn install from the root doesn't fix it?

leematos · 2022-01-10T02:48:13.806Z

Just tried upgrading node, making sure I'm on npm 7, and did:

npm clean
yarn install

still complained. Cleaned my iOS build dir and then

pod install

still erroring.

I'm gonna attempt a clean repo pull in a new directory tomorrow to see how that goes.

alfredopalhares · 2022-01-10T11:22:09.689Z

Cleaning the worksapce still produces the same error for me.

Is it possible that there is a reference somewhere in the code GitHub - laurent22/uslug: A permissive slug generator that works with unicode. Instead of the one in the monorepo?

Regards,
Alfredo Palhares

Daeraxa · 2022-01-11T12:12:53.992Z

Interestingly somebody raised an issue trying to build joplin-cli in a dockerfile which is also complaining about uslug using npm install-g joplin : npm install need a ssh private key · Issue #6003 · laurent22/joplin · GitHub

#6 45.39 npm ERR! code 128
#6 45.39 npm ERR! An unknown git error occurred
#6 45.39 npm ERR! command git --no-replace-objects ls-remote ssh://git@github.com/laurent22/uslug.git
#6 45.39 npm ERR! Warning: Permanently added 'github.com,20.205.243.166' (ECDSA) to the list of known hosts.
#6 45.39 npm ERR! git@github.com: Permission denied (publickey).
#6 45.39 npm ERR! fatal: Could not read from remote repository.
#6 45.39 npm ERR!
#6 45.39 npm ERR! Please make sure you have the correct access rights
#6 45.39 npm ERR! and the repository exists.

laurent · 2022-01-11T12:25:34.355Z

It's a big mystery this uslug package. How can a URL that's not referenced anywhere in the repository can still popup here and there.

roman_r_m · 2022-01-11T21:13:10.876Z

I am also seeing this trying to run Android app:

error: Error: Unable to resolve module @joplin/fork-uslug from /home/roman/dev/joplin/packages/renderer/MdToHtml.js: @joplin/fork-uslug could not be found within the project or in these directories:
  ../renderer/node_modules
  ../../node_modules
  /home/roman/dev/joplin/packages/app-mobile/node_modules/@joplin/fork-uslug

Daeraxa · 2022-01-11T21:39:56.935Z

laurent:

How can a URL that's not referenced anywhere in the repository can still popup here and there.

This might be just igorance and lack of knowledge about yarn or the project in general but there are a bunch of references to https://github.com/laurent22/uslug.git in the yarn.lock file.

jabraun · 2022-01-19T15:14:19.259Z

I am, unfortunately, not a developer, but I wanted to bump this thread in hopes that folks will continue looking for a solution. @alfredopalhares has done a lot of often-thankless work to package Joplin for the Arch ecosystem, and many of us on Arch/Manjaro depend on his efforts. Not to mention it sounds as though the issue is affecting packages for other systems, too.

laurent · 2022-01-19T15:20:06.895Z

Daeraxa:

there are a bunch of references to https://github.com/laurent22/uslug.git in the yarn.lock file.

I guess that's where it comes from then, although it's strange it goes get it from there in some cases and not others.

Daeraxa · 2022-02-04T10:39:05.747Z

Did you resolve it?
Got the same issue after I just tried building & running the android mobile app, fresh clone after an upstream fetch:

info Reloading app...
 BUNDLE  ./index.js 

error: Error: Unable to resolve module @joplin/fork-uslug from /home/dae/.local/share/Trash/files/joplin/packages/renderer/MdToHtml.js: @joplin/fork-uslug could not be found within the project or in these directories:
  ../renderer/node_modules
  ../../node_modules
  ../../../../../../../node_modules
  /home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/@joplin/fork-uslug

If you are sure the module exists, try these steps:
 1. Clear watchman watches: watchman watch-del-all
 2. Delete node_modules and run yarn install
 3. Reset Metro's cache: yarn start --reset-cache
 4. Remove the cache: rm -rf /tmp/metro-*
  34 |     source_map: require('./MdToHtml/rules/source_map').default,
  35 | };
> 36 | const uslug = require('@joplin/fork-uslug');
     |                        ^
  37 | const markdownItAnchor = require('markdown-it-anchor');
  38 | // The keys must match the corresponding entry in Setting.js
  39 | const plugins = {
    at ModuleResolver.resolveDependency (/home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/node-haste/DependencyGraph/ModuleResolution.js:129:15)
    at DependencyGraph.resolveDependency (/home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/node-haste/DependencyGraph.js:288:43)
    at Object.resolve (/home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/lib/transformHelpers.js:129:24)
    at resolve (/home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/DeltaBundler/traverseDependencies.js:396:33)
    at /home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/DeltaBundler/traverseDependencies.js:412:26
    at Array.reduce (<anonymous>)
    at resolveDependencies (/home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/DeltaBundler/traverseDependencies.js:411:33)
    at processModule (/home/dae/.local/share/Trash/files/joplin/packages/app-mobile/node_modules/metro/src/DeltaBundler/traverseDependencies.js:140:31)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)

roman_r_m · 2022-02-04T10:46:47.797Z

Nope, I gave up on trying to run it.

laurent · 2022-02-04T11:15:01.347Z

Maybe try to delete the uslug entry from yarn.lock, then run yarn install again? That should get rid of any old reference and hopefully fix the issue.