Mobile: Plugin support

[personalizedrefriger]

Note: Plugin support has been merged. It should be available in the Android Joplin pre-release.

See this GitHub issue for more information about plans for plugin support on iOS.

---

Over the past few weeks, I've been working on adding plugin support to mobile (see the relevant pull request on GitHub).

This post summarizes the status of the pull request.

To-do

• Better development experience
  • Currently, it's possible to inspect plugin WebViews in Google Chrome, but only in a development build of Joplin. Assuming most plugin authors won't be running a development build of Joplin, a way to enable plugin WebView debugging should be added.
• Refactoring & automated tests (where possible)
• Performance improvements
  • Fix: Large renderer content scripts (like that included in ABC Sheet Music) significantly slow down the renderer.
  • Fix: Plugin background WebView is created even if there are no plugins to load.
• Support more of the plugin API (see below).
• Add a way to start the mobile app in safe mode (i.e. without running plugins).

Plugins known to work

ABC sheet music

The ABC sheet music plugin mostly works.

Known issues:

• In dark mode, the music is difficult to read.

An error message is briefly shown before music renders.

ABC sheet music registers a renderer content script. This content script tries to access the plugin's settings.

With the plugin API pull request, the renderer is run inside the note viewer WebView for security and to provide renderer plugins access to DOM APIs. This does mean, however, that communication with renderer scripts is asynchronous.

The settings API is synchronous, however (and so is markdown-it rendering). Thus, attempting to access a setting returns undefined, loads the setting, then triggers a re-render (with the setting loaded).

After the first render, an error message is shown by the ABC Sheet Music plugin because it doesn't expect the setting to be undefined. The error message disappears because the setting is not undefined in the second render.

The example CodeMirror 6 plugin

The mobile editor is based on CodeMirror 6. As such, the demo CodeMirror 6 plugin for desktop works.

Because the existing CodeMirror 6 plugin API includes a compatibility layer, plugins that target CodeMirror 5 may work, depending on the features they use.

Note: For testing purposes, this plugin is included with the demo Android APK and enabled by default.

Function plot

Math mode

Known issue: Clicking on the same line as a math expression does not move the cursor to that line.

Spoilers

Known issue: Some content is missing a show/hide arrow (renders a box instead).

Plugins known not to work

• Plugins that require file system access (e.g. Simple Backup, Outline)

Plugin API

Supported

• Markdown-it and CodeMirror 6 content scripts
• joplin.data
• joplin.views.toolbarButtons.create
  • Currently only works for ToolbarButtonLocation.EditorToolbar
• joplin.clipboard.readText, joplin.clipboard.writeText
  • Missing support for readHtml, writeHtml
• Most of joplin.views.dialogs
  • Missing support for showOpenDialog
• joplin.commands
  • Only a very small set of built-in commands are currently supported.
• joplin.settings
• joplin.views.panels

Unsupported

• joplin.imaging
• joplin.views.noteList
• joplin.views.menus
• joplin.window
• joplin.interop
• joplin.require

API Notes

• There are almost certainly bugs in the APIs listed as "supported" above.
• joplin.versionInfo() has been extended to include a platform key, which is set to mobile when running on mobile. This API is not finalized — it might be replaced with something similar to joplin.isMobile() or joplin.platform.isMobile(). (Example usage).

Development notes

Plugins can be loaded from .jpl files by clicking the "Install from file" button in plugin settings.

If a plugin has uncaught errors/promise rejections, an "Error" button should be visible on that plugin's card in settings. It may be necessary to close and re-open settings for the button to be visible just after installing a plugin:

Clicking that button should show entries in the application log that include the plugin's ID.

Android APK

Release Mobile Plugins — Prerelease 3 · personalizedrefrigerator/joplin

This APK is built from the plugin support pull request and published under the "releases" page of my fork of Joplin on GitHub.

Before building, I first changed the application ID from net.cozic.joplin to net.personalizedrefrigerator.joplin (and renamed the app to Joplin (Beta)). Thus, the application can be installed alongside an existing version of Joplin without overwriting it.

Feedback

Suggestions and feedback are welcome and can be posted in the comments below.

[whitewall]

So far I have never needed plugins on mobile. But I highly appreciate your work on this. Thanks!

[shikuz]

Looking forward to this! Thank you!

Will be interesting to see if the Quick Links Plugin works on mobile.

[roman_r_m]

I'm pretty sure it won't because of this

@personalizedrefriger

Because the existing CodeMirror 6 plugin API includes a compatibility layer, plugins that target CodeMirror 5 may work, depending on the features they use.

[personalizedrefriger]

I've created a pull request that should add support for CodeMirror 6 to quick links:

Support CodeMirror 6

[Wladefant]

Same goes for the anchor link plugin. Does it work? Or as an alternative it could just be built into the default app experience

It apparently does work as far as the ui goes, and even displays the copied when pressed, but does not copy unfortunately

[personalizedrefriger]

That should now be fixed! (I plan to release a new prerelease version later today.)

Information about the fix

A debug statement attempted to log plugin messages with logger.debug. That debug statement tried to convert a message sent by the anchor link plugin to a string, which failed with the error TypeError: Cannot determine default value of object.

Changing the log statement from

	logger.debug(`Handling message from content script: ${contentScriptId}: ${message}`);

to

	logger.debug(`Handling message from content script: ${contentScriptId}:`, message);

seems to fix the issue.

Edit: APK with the fix.

[shikuz]

Thanks so much for your efforts. This will be an incredible upgrade to the mobile experience. I love the bundled line numbers (mostly thanks to the folding sections).

I understand that plugin panels may be a tricky thing to fully support, but perhaps in the future registered panels can add an icon to the top header, next to the the list of notebooks button. Then, when pressed, the panel may be rendered as a floating element, like the list of notebooks. I don't know what's the technical complexity of this, but if this is achievable at some point, it could make plugins almost as powerful as on the desktop. Features provided by the outline plugin, backlinks and such will benefit from this (much easier to navigate than via the rendered preview sections of toc or backlinks).

[personalizedrefriger]

@shikuz

registered panels can add an icon to the top header, next to the the list of notebooks button

I like this idea!

Because phone screens can be small, it might make sense to show just one icon, then show a tabbed dialog (one tab per panel). For example,

[shikuz]

This would be totally amazing!

[Wladefant]

First of all would love a pin button on the notebooks

Apparently obsidian does that too. Something similar would be needed

When talking about the panels, I would go for the easier solution that would make the same thing for desktop and mobile. I mean the editing bar is the same.

And most plugins are integrated there either way, so I would just say, make the same on mobile

Even better (especially for small screens to have a button to change between the plugin buttons and editing buttons.

Edit: I now understand that we are talking about different things and they are both compatible and should be working quite well together.
I agree with the panel views and that would help for consistency between desktop and mobile.
The top bar should be fully hidden when the panels are activated and only when they are closed should the bar be visible once again. Another thing for example for the oultine plugin that you should keep in mind that when a link is clicked it should jump to the note to the heading

[Wladefant]

Just saw the collapse button, quite nice, should be integrated into the Viewer too

[Wladefant]

As expected the note tabs plugins does not work, but would be quite nice, if it could (at the bottom for example). Dont know what the complexity here is.

I can see that all the viewer plugins work good:

Anchor Links works
Admonition works

[personalizedrefriger]

This has been implemented!

The plugin panels dialog can be opened by clicking "show plugin panels" in the actions menu for a note. Currently, a note needs to be open to show or hide the plugin panels dialog.

[shikuz]

This is very very cool, thank you @personalizedrefriger !!

I can confirm that the Automatic Backlinks panel (and preview section) seem to work on mobile out of the box, which is a great addition.

One thought regarding the implementation, is that currently access to panels requires at least 2 clicks (actions --> show plugin panels), while a more optimized solution will require just 1 click to open the panel. Is this something that you can envision? I suggested a button on the header for this reason, but I'll be very grateful for any one-click solution that anyone comes up with.

I can also understand if at this point plugins on the mobile are not a priority feature (i.e., that requires quick access), but I do expect them to gain popularity and perhaps even become indispensable as they are currently on the desktop.

[Medullitus]

A fantastic project!

Thank you very much @personalizedrefriger

[cwhittl]

If you need an IOS beta, let me know. I have a Dev account so I can deploy just might need some links on how to for this project.

[personalizedrefriger]

The iOS AppStore guidelines has restrictions on 3rd-party code that we'll need to make sure we comply with. At the time of this writing, the guidelines were:

Guidelines

See App Review Guidelines - Apple Developer

4.7 Mini apps, mini games, streaming games, chatbots, and plug-ins

Apps may offer certain software that is not embedded in the binary, specifically mini apps, mini games, streaming games, chatbots, and plug-ins. You are responsible for all such software offered in your app, including ensuring that such software complies with these Guidelines and all applicable laws. Software that does not comply with one or more guidelines will lead to the rejection of your app. You must also ensure that the software adheres to the additional rules that follow in 4.7.1 and 4.7.5. These additional rules are important to preserve the experience that App Store customers expect, and to help ensure user safety.

4.7.1 Software offered in apps under this rule must:

• follow all privacy guidelines, including but not limited to the rules set forth in Guideline 5.1 concerning collection, use, and sharing of data, and sensitive data (such as health and personal data from kids);
• include a method for filtering objectionable material, a mechanism to report content and timely responses to concerns, and the ability to block abusive users; and
• use in-app purchase in order to offer digital goods or services to end users.

4.7.2 Your app may not extend or expose native platform APIs to the software without prior permission from Apple.
4.7.3 Your app may not share data or privacy permissions to any individual software offered in your app without explicit user consent in each instance.
4.7.4 You must provide an index of software and metadata available in your app. It must include universal links that lead to all of the software offered in your app.
4.7.5 Your app must share the age rating of the highest age-rated content available in your app.

As such, before creating a version that can be distributed for iOS, I suspect that we'll need to allow only a specific set of plugins and versions (though perhaps the restrictions are different for TestFlight?). We may also need to get permission from Apple to allow plugins to access the clipboard (which currently uses a native API) and Joplin's database (this includes access to notes).

[personalizedrefriger]

Mobile plugin support has been merged and should be present in the Joplin v3.0.1 pre-release for Android:

Release android-v3.0.1 · laurent22/joplin-android

See the following GitHub issue for more information about plans for plugin support on iOS:

iOS: Enable plugin support

[crtkugrrvjy]

Amazing work thank you!

Embedded Search works!