iOS 26 - iOS Shortcuts Rocketbook 'integration' and Apple Intelligence

Apps
1

Like many of you, I use a physical notebook, in my case a Rocketbook due to the re-usability and originally the app integrations. However, I don’t use the integrations anymore, partly due to the stagnant development of the Rocketbook app/integrations, partly due to privacy concerns (because I haven’t read the Privacy Policy, and so don’t know how my data is processed, this could be unfounded.)

I was excited to see Project 3: Official Rocketbook Integration ( Project 3: Rocketbook integration with Joplin ) but then realised I likely need some way to power OCR, on my self-hosted set up, and currently that’s a no-go to running what is most likely a local AI model. :cry:

With the release of iOS 26, the ability to directly interact with Apple Intelligence is available within iOS Shortcuts, so I’ve looked at how I might be able to accompany this myself. I have created an iOS Shortcut that:

  1. Take Photo / Select Photo
  2. Create AI Prompt (see below)
  3. Passes to AI Model (see below)
  4. Copy’s response to Clipboard
  5. Open’s Joplin
  6. Paste Text into Joplin (manual step)

AI Model

There are now 3 options:

  • ChatGPT
    If you aren’t signed in to the Extension (in Settings), it appears that your data is secure and not used for training, but this option may not be for the more privacy focused. This option certainly produces the most reliable and consistent, high quality results from my ‘scribbles’ but with a tiny bit for effort on my part, I can use another more privacy focused option.

  • On-Device
    This model is somewhat limited but if your writing style is consistent and clear, the results are very good.

  • Cloud (Private Cloud Compute) - learn more at: https://youtu.be/546ufMY7488?si=wvWbsbhKX5s16jXt&t=138
    This option works brilliant for my use case, is both consistent and private (if you choose to believe Apple). My notes can be quite complex, with a legend showing several types of items on my ‘list’ and then corresponding symbols on list items, with sub-items too. This is the option I am using.

AI Prompt

This is my prompt, but you can tweak it to your requirements:

{variable to image from step 1}
Extract text from this image of my notebook, format all the text using extended markdown syntax.
I tend to use squares to represent tasks and these should be markdown checkboxes in the returned text.

Do not add any comments or further questions to your response.

Conclusion and Improvements

This is a great improvement to my process, and much more reliable OCR than the default Rocketbook integration. This process could be drastically improved if Joplin supported the ability to pass Markdown/Notebook/Note Name via Shortcuts, instead of Copy and Paste, but I’m very happy with this.

For those who want to copy my Shortcut, verbatim, here is the link:
https://www.icloud.com/shortcuts/f92658ba4a0b4f9bbe5dcd4674aa3917

Very happy for any contributions, or ideas of how to improve this process would be greatly received.

2

This topic definitely needs some TLDR in beginning.

1 Like