Troubleshooting FAQ and collecting topic for contributing to Joplin codebase

PackElend · 2020-03-02T20:28:52.901Z

this is a living topic what shall collect and describe nicely all traps and hurdles when trying to contribute to codebase of Joplin.

For the time being anyone is welcome to edit this wiki.

If you feel that there is something that should be mentioned here, just reply in this topic and quote (discourse allows to quote across different topic) or reference to it, we will take from there.

development environment

build

contribute

basics about the PR

as there are not that many reviewers around here keep your PR clean to safe us time, you should be familiar with Use git diff or git difftool and know why to use them.
Moreover, some editors reformat the code when saving, or you might have added test code that shouldn’t end up in the PR.

Pull requests that include changes unrelated to your issue will not be merged . Moreover it does not reflect well on your work when mentors have to remind you to clean up your pull request.

Once you have implemented changes requested by a mentor, please do not mark the comment as “resolved”. The mentor will need this comment to find out what still needs to be checked, and will mark the comment as “resolved” themselves.

Effectively using Github

After forking Joplin, clone the repository to build it locally
git clone git@github.com:<your-userid>/joplin.git
Then add another remote to pull changes from upstream.
git remote add upstream https://github.com/laurent22/joplin.git
Github does not automatically sync your forked repository with the main Joplin repo, you need to do it manually here’s how to.
There is another niche way of doing the same automatically too.
When you are making a pull request make sure you do not commit changes to your master branch (master branch of the forked-repo should be the one which is constantly synced with the main Joplin repository), create a new branch, push changes to it, and create a Pull-Request from that branch, it will save you a lot of time, in the case, your PR is not accepted you don’t need to reconfigure master branch. This is a really good video which will refresh your concepts about branches.
Use Github-keywords like fixes #issue-number and closes, if your PR would fix an issue.
Use git diff or git difftool to see exactly what’s changed and remove any change that’s not relevant to your issue. Consider using git add -p to add only the relevant changes.

app architecture

PackElend · 2020-03-02T20:51:16.585Z

laurent · 2020-03-11T15:05:17.006Z

@PackElend, I've noticed that several applicants commit changes unrelated to their PR. It's been a time sink in some PR as we have to remind them over and over again to remove these changes.

So I think we should suggest they check their changes carefully before committing them. Something like:

Before committing your changes, make sure you check that what you are going to commit. For example, some editors reformat the code when saving, or you might have added test code that shouldn't end up in the PR.

Use git diff or git difftool to see exactly what's changed and remove any change that's not relevant to your issue. Consider using git add -p to add only the relevant changes.

Pull requests that include changes unrelated to your issue will not be merged. Moreover it does not reflect well on your work when mentors have to remind you to clean up your pull request.

PackElend · 2020-03-11T15:21:29.927Z

~~agree will update the quick start guide~~
done in in the live blog and here

PackElend · 2020-03-23T08:19:44.163Z

Where to install a new dependency that will be used by all the platform apps?

You should add the dependency in ElectronClient/package.json, CliClient/package.json, and ReactNativeClient/package.json.

PackElend · 2020-04-08T08:06:29.609Z

unit test description

github.com/laurent22/joplin

Desktop: Use system keychain for storing passwords

laurent22:master ← jyuvaraj03:keytar

opened 05:57AM - 23 Mar 20 UTC

jyuvaraj03

+123 -8

PackElend · 2020-04-10T07:06:53.929Z

important info

github.com/laurent22/joplin

[Desktop] Add option to preview note body in note list

laurent22:master ← anihm136:feature/note-list-preview-snippet

opened 05:57AM - 24 Mar 20 UTC

anihm136

+157 -8

PackElend · 2020-04-11T12:30:47.343Z

important remark on imported notes by wrapper

github.com/laurent22/joplin

DESKTOP : Added a Markdown-it plugin to support embedded youtube videos

laurent22:master ← Rishgod:embedVideoII

opened 04:20PM - 13 Mar 20 UTC

Rishgod

+88 -0

PackElend · 2020-04-14T12:04:00.591Z

restart travis CI

github.com/laurent22/joplin

Desktop: Change Toolbar Layout

laurent22:master ← rabeehrz:issue-1944

opened 09:39AM - 06 Apr 20 UTC

rabeehrz

+45 -34

elsiehupp · 2020-10-17T17:25:51.373Z

PackElend:

development environment

I would recommend that you should recommend VS Code, since it's consistent across Linux, macOS, and Windows, and it's an Electron app itself. It supports complexity without being too complex.

I also discovered through trial and error that you need to install npm. npm is available as a module in VS Code, but it is also available as its own installer package. I don't know which is preferable.

VS Code includes native Git support, but for relative newbs the GitHub GUI client is nice and easy to use.

Also FYI this command doesn't work "out of the box":
git clone git@github.com:<your-userid>/joplin.git
since it requires the GitHub user to have SSH keys. The HTTPS version of the command is:
git clone https://github.com/<your-userid>/joplin.git
which works perfectly fine and doesn't require the user to figure out how to set up GitHub SSH.

PackElend:

After forking Joplin...

You might want to explain how to do a fork and how to create a branch, since it's better not to assume the user has a lot of familiarity with Git. Most of these things can be accomplished using the GitHub web interface (which also includes a direct link to open in GitHub Desktop), and consolidating as many steps as possible into as few applications as possible could help streamline the instructions.

PackElend:

build

I haven't gotten to this, yet, myself, but you could probably start with the instructions in joplin / BUILD.md. It would probably be worth seeing how many of the build dependencies can be installed through VS Code. I'm sure I will see for myself soon enough!

PackElend:

contribute

Again, joplin / CONTRIBUTING.md would probably be the place to start for this. Obviously the GitHub document should remain canonical—and indeed you should link to it, as with BUILD.md, but you could summarize the salient bits here. Considering the Joplin Discourse forum is substantially more newb-oriented than the GitHub is, the instructions here should be much more straightforward and explicit (though clarity in writing is equally important on GitHub, as well.)

elsiehupp · 2020-10-18T19:57:36.864Z

I'm trying to work my way through BUILD.md, and I'm still running into problems, probably due to the fact that I'm running the macOS 11 beta. I'm currently waiting for the latest Xcode GM to download, but in the meantime, here are my thoughts on BUILD.md...

I think the "Required dependencies" should be titled "Install Build Dependencies".
It should have a section for each build platform.
It should explain explain how to install each dependency if it can be installed with a single command-line command.
In the case of macOS, since Cocoapods is already installed with brew, it would be simplest if everything else was also installed with brew.
There should also be explicit instructions to download an appropriate IDE. In the case of macOS and iOS, it seems like it is also necessary to download Xcode, so that should be spelled out, as well.

Here's a sample set of instructions for macOS:

1. Set up your workspace

Install Xcode from the Mac App Store.
Install Visual Studio Code.
Install GitHub Desktop.
Create a fork of Joplin on GitHub by clicking Fork in the upper right-hand corner:

Note: you will need a GitHub account in order to work on Joplin. A GitHub account is very useful: you can even use it to sign up for the Joplin Forums, as well!

Install wei/pull and enable it for your new fork. If this is the only thing you're working on on GitHub, it should be safe to enable it for all future branches, as well.
Navigate to the page for your new fork on GitHub (https://github.com/<your_username>/Joplin), click the green Code button:

and select Open with GitHub Desktop:

Follow the instructions to create a local copy (clone) of the repository on your Mac.

Note: you should not put a cloned repository inside any folder that syncs with another cloud service, as the cloud sync may confuse GitHub. A safe place to clone your Joplin repository is <home_folder>/Repositories/Joplin

2. Install the Build Dependencies

Open Applications / Utilities / Terminal for the following steps. Copy and paste each command into the terminal window and press ⏎ to continue:
Install Homebrew:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
Homebrew will let you install the rest of the dependencies more easily.
Install Yarn:
brew install yarn
Install Node.js:
brew install node
Install Cocoapods:
brew install cocoapods

3. Finishing Your Setup and Testing Your Workspace

Launch Visual Studio Code, select File > Open... (⌘O), navigate to the folder where GitHub cloned your Joplin repository, and click Open.
Open a new terminal window in VS Code with Terminal > New Terminal (⌃⇧`) to enter the following commands.
Finish your setup by installing the rest of Joplin's dependencies using npm:
npm install
Test that your workspace is correctly set up by building and running the existing code for Joplin for Mac as-is:
cd ElectronClient
npm start

The above command should launch a working copy of Joplin. If it doesn't, copy any feedback from the terminal window into your clipboard and search for it on the Joplin Development Forums. If you don't find a solution to your problem, start a new topic, and paste the terminal output in full. Your report will help us make sure these instructions are up-to-date and functional!

4. Working On Code

In order to minimize conflicts with existing code, you use a separate branch of your repository for each feature or fix you are working on.
- To open a new branch, in GitHub Desktop, click Current Branch:
  
  and select New Branch:
  
  Enter a descriptive name for the feature you plan on working on, with hyphens instead of spaces, and click Create Branch:
- To use a branch you have already made, in GitHub Desktop, click Current Branch:
  
  start typing the name of the branch and click on it to switch.
  If you have unsaved work in an existing branch, GitHub Desktop will offer to stash it on your computer until you switch back to that branch.
When you are done working, you can commit your changes to a branch by selecting the files in GitHub desktop, entering a summary of your changes, and clicking Commit to <name of branch>:

If the Commit button shows the wrong branch, you should be able to switch branches (see above) and commit your changes to the correct branch.

Note: because commits create a history of your work that allows you to revisit any problems later on, it can be useful to err on the side of creating commits as often as possible when you have made changes worth summarizing.

...
...
...

How's this to start? Can you tell I've read a lot of Dummies books in the past? (I actually really would appreciate feedback on this.)

PackElend · 2020-10-18T20:13:34.567Z

elsiehupp:

How's this to start?

that is greater, I hope @laurent could spare a minute to pick it up

elsiehupp · 2020-10-18T21:39:45.476Z

Some initial ideas for the "Architecture" subheading:

Joplin is primarily written in TypeScript, a more robust form of JavaScript, and runs in Node.js, which allows JavaScript to be used outside of a browser environment. Joplin's codebase uses Yarn, npm, and Cocoapods to manage its dependencies. Its graphical user interface is written using React, a cross-platform framework that allows the much of the same code to be used on all of the platforms on which Joplin currently runs. On Android and iOS, Joplin uses ReactNative, and on macOS and Windows, it uses React to build an HTML interface that runs in Electron.

After this, there could be a list of the folders in the root directory of the repository, with a short explanation of what each of them contains.

The Joplin repository is organized as follows:

joplin / .github contains templates used by the the GitHub website when managing the Joplin repository.

joplin / Assets contains images (primarily icons) used by Joplin's GUI clients.

Note: the organization of this folder leaves something tobe desired. It might make sense to make high-level divisions between application icons, toolbar icons, assets used by the Joplin website, and assets used in App Store listings.

joplin / CliClient contains code used in the Joplin command-line client.

Question: do the GUI clients use anything from the CliClient directory? If so, it might be better to name it something like "JoplinFramework" and separate out the files solely pertaining to the command-line interface.

joplin / Clipper contains code used by the Joplin web-clipper browser extensions.

joplin / ElectronClient contains code specific to platform integration on macOS and Windows.

joplin / Modules contains... a mish-mosh?

"Modules" seems like it would contain code shared between the different clients, ie "JoplinFramework" as described above.

joplin / ReactNativeClient contains code specific to platform integration on Android and iOS.

joplin / Tools seems like another mish-mosh?

joplin / docs looks like it's literally the website.

If your intention is for the documentation in full to be mirrrored between the GitHub repository and the website, it might be worth looking into a way to parse Markdown files and automatically publish them as HTML. It looks like someone has developed a way to do this, albeit somewhat manually.

joplin / patches ...I don't understand the contents of this folder.

joplin / readme seems to contain a somewhat assemblage of Markdown files, some of which seem to be drafts or components of the main README.md and which generally don't seem interlinked.

Honestly it feels like this folder should be called "Documentation"; it should be built with aggressive interlinking; and much of it should be built automatically as the website. If website and documentation assets (ie images) are stored in this directory, one way to keep things organized could for each page to have its own folder, with any assets stored alongside a nested README.md file (since my understanding is that GitHub automatically renders Markdown files with this name (and this name only) as HTML below the directory listing.

However, in the long run, it could work better to organize the documentation into the same directories as the code to the greatest extent possible. Some Markdown pages might not make sense to file this way, and those pages could be organized under "Website" or "Meta", though as much as possible the Markdown documentation should remain inline.

FWIW, there is an nmp package called TypeDoc that can automatically generate documentation directly from TypeDoc comments, and, yes, someone has made a Markdown plugin for it. It could make sense for the Joplin repository's build process to automatically run TypeDoc just like it currently runs unit tests, so that the documentation is always up to date.

elsiehupp · 2020-10-18T22:15:28.058Z

Oooh, there's also a UML generator for Visual Studio Code that looks really cool!

GitHub

myxvisual/vscode-ts-uml

💻 Dynamic generates TypeScript UML diagrams to Visual Studio Code. - myxvisual/vscode-ts-uml

elsiehupp · 2020-10-19T01:06:53.763Z

I got npm start to work. It looks like the problems I had been running into were:

I had installed Node.js from the website rather than using brew. It was a huge pain in the ass to uninstall Node.js and reinstall it from scratch. One of the advantages with brew is that it doesn't use sudo, so it's much less likely to create permissions errors.
I'm not 100% sure if this was a breaking problem, but I didn't have an up-to-date version of the Xcode Command Line Tools installed. I downloaded the installer from developer.apple.com, and it required me to log in with my Apple ID. If this is indeed a prerequisite, I should add it to the list.

Here's a draft "prerequisites" paragraph:

These instructions assume that you are using a Mac with an up-to-date version of macOS and that you are using an administrator account as the sole or primary user of the computer. If either of these is not the case, you may need to use specific older versions of the development tools, and you may run into permissions errors and/or conflicts. Feel free to ask for help in the Joplin Development Forums if you need it, but these instructions exist to remain as streamlined as possible.

You will need an Apple ID in order to build Joplin for iOS [??? and also for Mac?], and you will need a GitHub account in order to upload changes you make to the code. You can download the code without a GitHub account if you just want to play around with it, but you will not be able to contribute to the project without one. You can use your GitHub account to sign up for the Joplin Forums, as well!

This is starting to get redundant and unwieldy, so once I get some feedback I will go back and edit my initial draft rather than an entire new one.

elsiehupp · 2020-10-21T04:40:13.237Z

I've been doing some work in my own branch, and while I've been able to build, run and commit, it looks like npm install is not installing all the dependencies:

elsiehupp@Elsies-MacBook joplin % npm install      

> joplin@1.0.0 postinstall /Users/elsiehupp/Repositories/joplin
> cd Tools && npm i && cd .. && cd ReactNativeClient && npm i && cd .. && cd ElectronClient && npm i && cd .. && cd CliClient && npm i && cd .. && gulp build

audited 186 packages in 1.477s

4 packages are looking for funding
  run `npm fund` for details

found 2 vulnerabilities (1 low, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details

> Joplin@0.8.0 postinstall /Users/elsiehupp/Repositories/joplin/ReactNativeClient
> patch-package --patch-dir ../patches/shared && jetify && npm run build

patch-package 6.2.2
Applying patches...
htmlparser2@4.1.0 ✔
Jetifier found 1332 file(s) to forward-jetify. Using 4 workers...

> Joplin@0.8.0 build /Users/elsiehupp/Repositories/joplin/ReactNativeClient
> gulp build

[00:34:48] Working directory changed to ~/Repositories/joplin
[00:34:48] Using gulpfile ~/Repositories/joplin/gulpfile.js
[00:34:48] Starting 'build'...
[00:34:48] Starting 'copyLib'...
[00:34:49] Finished 'copyLib' after 564 ms
[00:34:49] Starting 'tsc'...
[00:35:03] Finished 'tsc' after 14 s
[00:35:03] Starting 'updateIgnoredTypeScriptBuild'...
[00:35:03] Finished 'updateIgnoredTypeScriptBuild' after 797 ms
[00:35:03] Finished 'build' after 15 s
npm WARN @react-native-community/datetimepicker@3.0.3 requires a peer of react-native-windows@>=0.62 but none is installed. You must install peer dependencies yourself.
npm WARN babel-eslint@10.1.0 requires a peer of eslint@>= 4.12.1 but none is installed. You must install peer dependencies yourself.
npm WARN react-native-fs@2.16.6 requires a peer of react-native@^0.59.5 but none is installed. You must install peer dependencies yourself.
npm WARN react-native-fs@2.16.6 requires a peer of react-native-windows@^0.57.2 but none is installed. You must install peer dependencies yourself.

audited 1204 packages in 24.832s

27 packages are looking for funding
  run `npm fund` for details

found 5 vulnerabilities (4 low, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details

> Joplin@1.3.3 postinstall /Users/elsiehupp/Repositories/joplin/ElectronClient
> npm run build && gulp electronRebuild


> Joplin@1.3.3 build /Users/elsiehupp/Repositories/joplin/ElectronClient
> patch-package --patch-dir ../patches/shared && patch-package --patch-dir ../patches/node && gulp build

patch-package 6.2.2
Applying patches...
htmlparser2@4.1.0 ✔
patch-package 6.2.2
Applying patches...
sax@1.2.4 ✔
[00:35:12] Using gulpfile ~/Repositories/joplin/ElectronClient/gulpfile.js
[00:35:12] Starting 'build'...
[00:35:12] Starting 'compileScripts'...
[00:35:12] Starting 'compilePackageInfo'...
[00:35:12] Starting 'copyPluginAssets'...
[00:35:12] Starting 'copyTinyMceLangs'...
[00:35:12] Starting 'updateIgnoredTypeScriptBuild'...
[00:35:12] Starting 'compileExtensions'...
Copying to /Users/elsiehupp/Repositories/joplin/ElectronClient/tools/../gui/note-viewer/pluginAssets
Copying /Users/elsiehupp/Repositories/joplin/ElectronClient/tools/../../Modules/TinyMCE/langs => /Users/elsiehupp/Repositories/joplin/ElectronClient/tools/../node_modules/tinymce/langs
[00:35:13] Finished 'compileScripts' after 991 ms
[00:35:13] Finished 'compilePackageInfo' after 991 ms
[00:35:13] Finished 'updateIgnoredTypeScriptBuild' after 1.01 s
[00:35:13] Finished 'compileExtensions' after 1.02 s
[00:35:13] Starting 'copyLib'...
Copying to /Users/elsiehupp/Repositories/joplin/ElectronClient/tools/../pluginAssets
[00:35:14] Finished 'copyTinyMceLangs' after 1.11 s
[00:35:14] Finished 'copyPluginAssets' after 1.19 s
[00:35:14] Finished 'copyLib' after 694 ms
[00:35:14] Finished 'build' after 1.72 s
[00:35:15] Using gulpfile ~/Repositories/joplin/ElectronClient/gulpfile.js
[00:35:15] Starting 'electronRebuild'...
Running: "/Users/elsiehupp/Repositories/joplin/ElectronClient/tools/../node_modules/.bin/electron-rebuild"

[00:35:16] Finished 'electronRebuild' after 1.47 s
npm WARN ajv-keywords@3.4.1 requires a peer of ajv@^6.9.1 but none is installed. You must install peer dependencies yourself.
npm WARN babel-eslint@10.1.0 requires a peer of eslint@>= 4.12.1 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: 7zip-bin-linux@1.3.1 (node_modules/7zip-bin-linux):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for 7zip-bin-linux@1.3.1: wanted {"os":"linux","arch":"any"} (current: {"os":"darwin","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: 7zip-bin-win@2.2.0 (node_modules/7zip-bin-win):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for 7zip-bin-win@2.2.0: wanted {"os":"win32","arch":"any"} (current: {"os":"darwin","arch":"x64"})

audited 1481 packages in 11.708s

16 packages are looking for funding
  run `npm fund` for details

found 9 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details

> joplin@1.3.0 postinstall /Users/elsiehupp/Repositories/joplin/CliClient
> npm run build && patch-package --patch-dir ../patches/shared && patch-package --patch-dir ../patches/node


> joplin@1.3.0 build /Users/elsiehupp/Repositories/joplin/CliClient
> gulp build

[00:35:23] Using gulpfile ~/Repositories/joplin/CliClient/gulpfile.js
[00:35:23] Starting 'build'...
[00:35:23] Starting 'prepareBuild'...
[00:35:23] Finished 'prepareBuild' after 724 ms
[00:35:23] Starting 'compileExtensions'...
[00:35:23] Finished 'compileExtensions' after 4.18 ms
[00:35:23] Starting 'copyLib'...
[00:35:23] Finished 'copyLib' after 92 ms
[00:35:23] Finished 'build' after 824 ms
patch-package 6.2.2
Applying patches...
htmlparser2@4.1.0 ✔
patch-package 6.2.2
Applying patches...
sax@1.2.4 ✔
audited 906 packages in 6.316s

4 packages are looking for funding
  run `npm fund` for details

found 10 vulnerabilities (8 low, 2 high)
  run `npm audit fix` to fix them, or `npm audit` for details
[00:35:26] Using gulpfile ~/Repositories/joplin/gulpfile.js
[00:35:26] Starting 'build'...
[00:35:26] Starting 'copyLib'...
[00:35:26] Finished 'copyLib' after 97 ms
[00:35:26] Starting 'tsc'...
[00:35:39] Finished 'tsc' after 13 s
[00:35:39] Starting 'updateIgnoredTypeScriptBuild'...
[00:35:39] Finished 'updateIgnoredTypeScriptBuild' after 652 ms
[00:35:39] Finished 'build' after 14 s
npm WARN @typescript-eslint/eslint-plugin@2.10.0 requires a peer of eslint@^5.0.0 || ^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN @typescript-eslint/parser@2.10.0 requires a peer of eslint@^5.0.0 || ^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN eslint-plugin-import@2.20.2 requires a peer of eslint@2.x - 6.x but none is installed. You must install peer dependencies yourself.
npm WARN eslint-plugin-react@7.18.0 requires a peer of eslint@^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN eslint-plugin-react-hooks@2.4.0 requires a peer of eslint@^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 but none is installed. You must install peer dependencies yourself.

audited 852 packages in 67.763s

26 packages are looking for funding
  run `npm fund` for details

found 4 low severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details
elsiehupp@Elsies-MacBook joplin %

tessus · 2021-01-27T01:34:48.705Z