Plugin: Released Omnivore Sync

rino · 2024-08-28T06:02:49.273Z

Omnivore is a read-it-later app. This plugin fetch new articles and highlights via Omnivore API, manually or periodically. Articles are converted to Markdown. Highlights are stored in the notes by their dates.
It is my first plugin, and development is very early stage, but it should do some jobs.

Omnivore Sync

image644×1734 89.5 KB

How to use

Install from Plugins section in Joplin's settings.
In Omnivore Sync plugin settings, set "Omnivore API Key" obtained from Omnivore's site.
Choose "Sync Type," "Target Notebook (created if not exist)," "Highlight Note Title Prefix."
For manual sync, run "Tools > Sync Omnivore Articles" from the Joplin's menu. It takes a minute depends on the number of articles/highlights in your Omnivore library.
I could not figure out how to show logs or an indicator, so nothing is printed out. Please stay patient!

eagle · 2024-08-28T11:35:21.178Z

Hi Rino and welcome to the forum

Thanks a lot for providing an Omnivore plugin. I did a test and it seems very promising!

One thing I would wish for is some method of selecting which Omnivore articles that syncs to Joplin. The Logsec plugin solves it by defining a custom label in Ominvore, and have a setting in the plugin where we can define the label. Then only articles with that label gets synced.

Thanks again!

rino · 2024-08-28T13:11:16.227Z

Thank you for using, and giving important feature request!

I've implemented selected label sync feature (0.1.5).
Note that the given list of labels is converted to Omnivore's search,
and each sync process checks the label-matched articles/highlights using the search.

I published to npm but takes some time to be cached by official plugin;
if you want to test v0.1.5, please use GitHub's release version.

GitHub Releases

Have a good day!

eagle · 2024-08-28T13:52:45.065Z

Hi Rino

Thanks a lot for the updated version. Now I can add the label "joplinsync" to articles in Omnivore and only those articles get synced to Joplin. Great!

Another question:
It seems that all highlights and annotations created on the same day in Omnivore is copied into 1 note in Joplin, and in creation order.

So in my Joplin note "Omnivore Highlights 2024-08-28" i have the following:

Annotation from article A
Highlight form article B
Annotation from article A
etc.

Is this intended? Would it better if highlights and annotations are grouped by article? Or perhaps one Joplin note per article (containing highlights AND annotations from that article)?

Great work!

rino · 2024-08-28T15:54:34.947Z

Hi eagle

I hope it works well!

The "By Date" grouping with anti-chronological order sorting for highlights/notes is currently intended, but not concrete decision.
Considering Logseq's Journal way and how SNS treat posts, I first implemented "By Date" behavior.
I wanted to know the real user's usecase. That's why I published here!

So, I've just implemented the new grouping, "By Article," alongside a new template "Highlight and Note."
Could you test it?

GitHub Release v0.1.6

Highlight Syncing Behavior

By Date

Highlights are grouped by the date they were created
Within each date group, highlights are sorted chronologically (older on top)
Highlights are further grouped by article within each date
The note title is "{Prefix} {Date}"

By Article

All highlights from the same article within the sync period are grouped together
Highlights are sorted by their position in the article
The note title is "{Prefix} - {Article Title}"

Cashie99 · 2024-08-29T05:28:39.890Z

I've been looking for something like this for a while. Congrats.

However, when I'm using it on my Mac, and trying to define the folder articles are synced into, the plugin creates a new folder with that name for every article synced. I wind up with multiple "!!!Omnivore" folders, each with one article.

rino · 2024-08-29T05:56:34.136Z

Oh, sorry for the bug!
I'm on a Mac as well (macOS 15 Beta) with Joplin Server on my LAN,
but could not figure out how to reproduce the multiple folders bug in the normal and dev mode.
Could you provide more details?

Cashie99 · 2024-08-29T08:52:26.541Z

Oh, totally not an issue. I appreciate you contributing to Joplin.

Ok. I'm on a Mac Mini running 14.6.1.

I'm using Joplin Cloud. This is on the desktop v3.1.3.

I just added two articles to Omnivore. I've specified a folder called "!!!Omnivore". As you can see, they're put into individual folders named this.

Screenshot 2024-08-29 at 6.51.48 PM

eagle · 2024-08-29T09:16:41.758Z

I can not reproduce this on my Joplin 3.0.15 running on Macbook Air 14.6.1 syncing to Dropbox.

I tested by specifying "!!!Omivore" folder and it works fine - no multiple folders are created.

eagle · 2024-08-29T09:29:17.177Z

The new version works exactly as I expected now

Hopefully you will get feedback from more users with other workflow suggestions. I think the plugin works brilliant for my use now.

I have not thought this completely through but perhaps this could be useful:

I usually want only annotations and highlights to be synced to Joplin.
So I select "Sync type: Highlights and annotations only" in plugin preferences.
But sometimes I also want to sync the complete article .... so how about changing the behavior to this:

If there are any label(s) defined in "Article labels" AND "Sync type" = "Highlights and annotations only", any article with that label is still synced (as an article). (Overrides "Sync type").

Thanks again.

rino · 2024-08-29T10:13:32.922Z

Thank you for sharing the details.

I'm having trouble reproducing this duplication behavior.

Fetching functions are called using await, so getting each article/highlight is done one by one.
So far, after reviewing the code, I have not figured out the cause.
However, Joplin Cloud possibly returns the list of folders without including the recently created folders (due to cache?),
while Joplin Desktop and Joplin Server include the recent folders.
I'm thinking of a procedure to manage this, resolving this duplication and reducing the number of API calls.

rino · 2024-08-29T10:29:16.708Z

I'm happy to hear this plugin meets your expectations!
Thank you for the suggestion!

In that case, how about these settings?

Sync Type: Articles, highlights and annotations
Article Labels: Specify the unique label, such as "joplinsync," so only articles with the label are synced as the entire article.
Highlight Labels: Leave empty to sync every highlight and annotation.

With this configuration, sync articles if they have the specific label, and leave every article not synced
while all the highlights/annotations are always synced.

eagle · 2024-08-29T12:15:06.083Z

Thanks for your suggestions. I was clearly overthinking this. No changes necessary

Cashie99 · 2024-08-29T20:21:00.771Z

See how you go. If it's a problem limited to my install, or I'm being a feckless idiot, I have some workarounds I've developed. But I'll keep an eye on this space.

rino · 2024-08-31T06:22:48.778Z

Released v0.1.7. GitHub Releases: v0.1.7
This commit addresses the issue of multiple folder creations and improves
the reliability of the sync process across different environments.

I'm unsure of the cause but adjusted the folder checking and creation process.
It reduces the number of API calls (although all Joplin API calls are local).

rino · 2024-08-31T18:20:54.058Z

I made a mistake.
v0.2.0: Re-enabled "As Article" settings.
I already published it to npm; the Plugin will notify you of an update when the Joplin crawler detects it.

rino · 2024-09-07T03:05:52.150Z

v0.2.1 and v0.2.2 introduce more readability for highlights and annotations.
v0.2.1: Convert encoded text to be decoded in highlights
v0.2.2: Preserve Markdown formatting

Now, I'm working on storing remote-linked images as attachments.

rino · 2024-09-08T08:47:05.491Z

v0.2.3: Get remote-linked images and store them in Joplin note's attachments.
v0.2.4: Add support for the article and highlight sub-notebooks*
*Before v0.2.4, you could specify "Target Notebook." Now, it is called "Main Notebook."
Every note goes to the Main Notebook if the Article or Highlight sub-notebook is empty.
Sub-notebooks are created as children of the Main Notebook.

rino · 2024-10-29T16:27:34.481Z

Sadly, Omnivore's free hosted version will be shut down on Nov 15, 2024, according to their blog post. Although Omnivore's code base is fully open-source, the rigid procedure for self-hosting, running the iOS app, and web clipper is still unclear. I'll keep my eyes on any updates on the self-hosted Omnivore solution.
Thank you for using this plugin!