I’d Like Your Opinions on Pen Support Implementation

[cnburke]

Hello everybody,

I’m going to take a crack at adding some pen support to the Joplin app. I’d like your thoughts and opinions on how you guys think this would support your needs.

Quick background on myself: I’m a software engineer, and I have a lot of experience working on WPF Windows desktop applications with material design and MVVM. I currently work for a company developing medical devices. I’m no electron expert, and it will take me a bit to ramp up to how I need to integrate features into Joplin, but I’ll do my best. I’ve been using Joplin for a while now, so I want to give back some.

Ok, on to the fun stuff. I’d first like to focus first on pen integration with the iPad. IMO, this tablet would have the largest initial ROI on feature distribution.

There’s a couple things I want to keep at the forefront of my mind while developing this feature. Pen development (at least in mind) can either be awesome, or just really terrible depending on the implementation. I’ve used some pen implementations on different devices in the past, and because the experience was so bad, I never ended up using it. Because of this, I think I’d like to try something like the following:

Based on the underlying OS (iPad OS/iOS or Android), rather than building/integrating drawing features based on a drawing library or standing one up from scratch, I’d like to integrate the OS’ native Pen feature set into Joplin. My two paths forward for the iPad below:

• Have the ability to take notes directly from the notes app and display in Joplin, with the ability to edit them and have the resulting edit reflected in Joplin. This could look something like the current PDF display, with the note attachment displayed as a file link, and the preview beneath. To do the back and forth transfer of the attached note file in Joplin to the Notes app, I’d assume the iPad OS/iOS sharing feature will need to be utilized.
• I’d also like to see the integration of Notability into Joplin. There are obviously a ton of note taking apps across the appleverse, but Notability has a ton of followers and usage. They also maintain a huge feature set. There’s some 3rd party work on open-sourcing the .note file that Notability uses under the hood, and Notability already has their own file syncing capability for redundant backups. That capability could then be used to update a note in Notability, back it up to a server (like google drive), and have an applet push that note to the joplin backup sync instance to automagically grab the latest resource for an attachment and update it for you (assuming that’s what you want, I’m sure there’d need to be some configurability there).

Past that, I’d not sure what’d need to happen for the desktop apps and android apps. For desktop, I’m assuming that the biggest users on the windows desktop would be wacom/drawing tablet owners, surface owners (and by extension, windows tablet mode users), and maybe those wanting to draw with a mouse (you interesting people). For the mac side, sidecare is now a big thing, so I’d assume that iPad support for the mac desktop app using sidecar would be desired. Android tablet owners, there’s so many options, feedback here would be good as I don’t do anything there.

And finally that leave mobile (small) devices. Android and iPhones. I’m really not sure how appropriate drawing tools would be on these smaller screens, but I’m open to opinions.

Thanks for reading, looking forward to hearing your thoughts!

[laurent]

As I mentioned in PM, for drawing one possible library would be Fabric.js

The advantage of it is that it's JavaScript, so the same implementation could be used in all platforms. So if you get it working for iPad, without extra effort it will also work on Android, but also on Windows, macOS and Linux. Of course that's assuming it can work at all - in particular I'm not sure how the library works in mobile devices, with pens. Could be worth giving it a try anyway.

[zblesk]

I've recently acquired the most basic Wacom tablet, so I'm suddenly very interested in this topic on Windows.

Best of luck!

[kauffy]

This is a big "missing" for me from Evernote-- I use a Samsung Note 10+ and I will fairly often use the stylus to take a quick note, which can automatically be saved to Evernote without extra steps.

I also used to LOVE using OneNote on my tablet PC back in the day-- with "continuous graph paper" I could write and draw forever.

For my ideal implementation:

1. I can start a note as handwriting/drawing and immediately begin writing/drawing. I can use differing line widths and colors.
2. I can have a different background for these notes-- graph paper & dots at different sizes, solid color, paper texture, etc.
3. I can grab a screenshot, annotate it with the stylus, and then grab that into Joplin automatically (I have a million annotated screenshots that are currently "trapped" on my phone as photos).

Just my 8 cents! Good luck!

[cnburke]

Thanks for the thoughts @kauffy. To your point on the "continuous" paper - that's actually one thing I hated about OneNote. Specifically because when trying to export to a standard A4 (portrait or landscape) PDF, the drawing would always be cut off. I'd like to implement this with some page boundaries, so that way you'd always know that a specific drawing would be contained within a standard page size. There'd still be the option to have different backgrounds and such, like graph paper or colored paper.

I'm looking into all the options suggested this far! I think there's value to the Fabric.js solution that @laurent suggested, as it would probably be the quickest way to get some level of drawing support across all platforms. Still doing more research.

[kauffy]

Porque no los dos?!

I really do like the continuous paper, but I also get your point-- why not allow the note to be created one way or the other? Personally, I rarely export or PDF the "oversized" diagrams, but I could easily see doing so if I were taking conventional notes, or similar.

The reason I like it is, back in the days of old, when I wanted to work out a problem, or to design a system (think flowcharts and mindmap type things), I'd use a giant pad of butcher block paper (like 3' x 4' or 91 cm x 1.2m), and start near the center (or top-left corner), and work outward (or down & to the right).

It's the single-biggest feature I've missed since moving off of OneNote.

[cnburke]

Yeah, to your point, no reason you couldn't do both

[mccurdyc]

Ooh! I'm excited to see there is discussion on this topic because this feature is also something that became important to me after acquiring an iPad.

I would love to have to types of drawing support:

1. basically one that ends up embedding the drawing as an image or something that could be exported alongside the markdown file. If you've used Bear or Evernote, think that editing experience, but I don't know how they handle exporting.

2. (really nice to have, but less critical) - annotations on top of plaintext for being able to highlight and circle things and write handwritten notes on top of the plaintext that could be stripped when exporting the markdown as plaintext, but would somehow still remain when you opened in Joplin. Think Apple notes or OneNote experience for writing plaintext and then being able to draw on top of it.

Again, 2 is a nice-to-have and less "I need this feature".

Thanks for thinking about implementing this feature!!! Best of luck!

[Yajo]

I think this should have these steps:

1. Find an open handwritting standard format. If there's none, (ab)use one of SVG, PNG, WEBP, PDF or XCF (Gimp's). It would be nice if you could edit that section, close, and later, open and still select ink areas, move them around, delete them, undo what you did last time, etc.
2. Find an editor for that format.
3. Allow including a new handwritten section in the note.
4. When clicked on that section, handwritting mode starts. Maybe full-screen would be the easiest to start with. Just like any external editor in any attachment.
5. When closing that editor, the section is included just as an image, in the resulting markdown file.

I guess that if you share some mockups that will help getting feedback beforehand.

Great addition, thank you very much!!

[chenlijun99]

The point 2 can be particularly tricky to get right if you take into account the fact that the text could be subsequently edited. Algorithms must implemented that appropriately anchor pen strokes with characters/words/paragraphs/etc.
AFAIK even OneNote doesn't support such feature. See [1] and [2].

[trymeout]

This will be cool for tablets and will be more user frieldly on tablets.

I wonder if it is possible to write letters abd words and to have then be converted to text and to be able to erase text easiky with a stylus?

Maybe the ability to add jpg into the note for drawing which will be a drawing mode and have a mode where you can write with a stylus.

[cnburke]

These are great comments. Thanks! I’m looking into Apple’s recent feature (believe it’s called “scribble” - that turns most textboxes into a writable area for the apple pencil. So, we might get that feature for free. Android tablets will be more challenging.

[trymeout]

Is scribble open source, if not then something like is needed to make it into Joplin

[kartoo]

I would love to be able to use pen support for taking notes and making sketches and drawings inside my notes. It would be great for shot planning (animation/movies) for creating storyboards. Also support for pressure sensitivity would be awesome too.