Kanban Plugin Weekly Report

[mablin7]

I'll be posting weekly updates of the kanban plugin project in this topic.

Check out the main project thread for general info: Kanban Board Project

Feedback is always welcome!

[mablin7]

Week 1

June 06. - 13.

Progress

This week I've been working on the background logic of the plugin, so there's isn't much to show for now. I finished the configuration parser, and I'm done with most of the logic that will be responsible for sorting the notes into columns and updating them as they're dragged around. For now, only the tag rule works, but it will be easy to add new ones, because the system is extensible.

I decided to use Joplin's search functionality to implement the sorting: each rule will create a search filter (for tags, it would be tag:tagName eg.), so each column will have a search query, potentially composed of multiple terms. Then, just use the data API to execute the query of each column and we have the sorted notes. It's really simple, but it involves a lot more back and forth between the plugin and joplin than necessary, so performance testing is on my todo list.

Achieved

• Set up the project
• Wrote config parser
• Added TS types
• Wrote system for creating rules

Planned for next week

• Finish and reorganize the sorting code.
  I want to wrap it all the logic for keeping the board state and sorted notes into a factory, so we can handle multiple open boards
• Scaffold frontend side
  Create basic structure, set up store and hooks which receive the sorted notes from the backend.

[laurent]

@mablin7

I decided to use Joplin's search functionality to implement the sorting: each rule will create a search filter (for tags, it would be tag:tagName eg.), so each column will have a search query, potentially composed of multiple terms. Then, just use the data API to execute the query of each column and we have the sorted notes. It's really simple, but it involves a lot more back and forth between the plugin and joplin than necessary, so performance testing is on my todo list.

Good idea. I assume it should be fine it terms of performance but if you notice any issue, perhaps we can improve things by supporting a "batch" API endpoint, that would take multiple queries and return the results.

[LightLifePhotography]

Kool stuff, Check out nextcloud deck for some inspiration

[mablin7]

Week 2

June 14. - 20.

Progress

This week was still all about the backend of the plugin: I've reorganized the code, hooked it up to the joplin api, so that when a board note is opened the config is detected and parsed and I've added the missing rules. Now almost all of the sorting functionality is working. There are two major problems currently:

1. The way I implemented the sorting of notes, by using joplin's existing search filters, doesn't allow to distinguish notebooks of the same name, but different path. So here, if there was a rule like notebookPath:folder 1/nested folder , both note 1 and note 2 would be matched.

├─ folder 1/
│  ├─ nested folder/
│  │  ├─ note 1
├─ folder 2/
│  ├─ nested folder/
│  │  ├─ note 2

Right now, I have a workaround, by filtering notes based on their parent_id, which is the unique id of their parent notebook, but this way we lose recursive search. I'm not yet sure how to resolve this issue.

2. The completedTag and completedNotebook rules can't work, because right now all rules are very simple functions, which have no knowledge of the notes they're operating on, while these would need to know whether their target note is a todo. Maybe I'll change the way the rules work, or come up with another way around this, but for now I'm putting this off.

I've spent most of the time reworking the way the backend is organized so that it's cleaner and more testable. I've decided on redux-like architecture, but the store would be the joplin database itself. The frontend will create actions, which are sent to the backend of the plugin. The backend then acts as a sort of reducer, turning the action into a data API requests, which are sent to joplin, changing the notes as necessary. Then the state is queried again, by running the search query of each column, and the returned notes are finally sent back to the frontend to display. Overall, I think this will keep each component of the plugin nice, simple and testable.

Achieved

• Reorganized sorting logic to use pure functions
• Wrapped sorting logic into board objects
• Created basic handle which checks if the last opened note is a board and parses the config if yes

Planned for next week

• Test cases for the sorting logic
  This was planned for this week, but unfortunately I didn't get it done on time. But this should be done on Monday.
• Scaffold frontend side
  Create a react project, create the basic component tree.
• Get started with the visual layout
  Set up the columns and so on, but just static for now. I think I'll only get to drag'n'drop the week after this.

[laurent]

@mablin7

I've spent most of the time reworking the way the backend is organized so that it's cleaner and more testable. I've decided on redux-like architecture, but the store would be the joplin database itself. The frontend will create actions, which are sent to the backend of the plugin. The backend then acts as a sort of reducer, turning the action into a data API requests, which are sent to joplin, changing the notes as necessary.

Since it's redux-like, I assume it's not the actual Redux lib. Any reason for that?

[mablin7]

It's because, the way I have it now, there's no need for a store, which redux would provide. The frontend dispatches an action, like "move note", to the backend, which then chooses some data api query(ies) (via a reducer-like function, but maybe it's more like a map, because it maps actions to queries) and executes them through the plugin api. Finally it runs the search, created from the column sorting rules, and returns the updated sorted notes to the frontend.

Like this, the only state the plugin keeps is the parsed configuration, but that's assumed to be static (if it changes the board would be removed and recreated), so I think there's no need for a state management library.

[laurent]

Thanks for clarifying and yes it looks like Redux wouldn't be needed in that case. It also makes sense to keep the app data as the single source of truth, as you've done, as that should make your code simpler and more reliable.

[mablin7]

Week 3

June 21. - 27.

Progress

This week was mostly about university exams for me, so I apologize for not being able to show much, but I've still managed to make some progress. Most notably, here's the first screenshot of an acutal UI (no mockup)!

I was planning on releasing a working test version with this progress update, that's why it got delayed to Monday, but unfortunately a roadblock came up which I haven't managed to resolve.

The way the plugin is constructed, the state of the board (which column each note belongs to) is not kept by the plugin. It fetches the data directly from Joplin, using its native search feature. When a note is dragged between columns, the plugin first updates Joplin's database, then fetches the board state again.

This is really neat, because it keeps the plugin code simple, but there's one major issue: getting the updated board state after a drag is asynchronous, because it involves sending a query to the main Joplin process and waiting for a reply. Of course, this happens very fast, for a user the delay is not noticeable. The problem is that React, the framework I use for the UI, doesn't really know what to render in this short time. If I tell it to render nothing, the UI would flicker. If I don't tell it what to render, it would render the previous last known state, so the dragged note would just snap back for a split second.

Ideally, I'd like to tell it to hold on, and keep the dragged card "floating" in place just for a few more milliseconds, while the state is updated in the background. Sadly, even though I've been trying for the past 1,5 days, I just couldn't make it work with the drag and drop library (react-beautiful-dnd) I was planning to use (which is a really great lib, would be awesome if I could get it to work).

I will keep trying, I'm thinking about switching to a different, lower-level drag&drop library, which hopefully would be more flexible with async updates. If all else fails, I can replicate the board state in the plugin, though I'd prefer to avoid this.

Achieved

• Got a basic UI (drag&drop not working yet)
• Wrote full test suite for backend
• Smaller fixes to backend

Planned for next week

• Fix drag&drop
• Release first working test version
• Fix recursive notebook search
• (if there's time) Get started on the config UI

Thanks for following along, as always feedback is appreciated!

[uxamanda]

@mablin7

Achieved

• Got a basic UI

So excited! It is so cool to see the first version. Congrats on juggling all of this with your exams!

As for the drag and drop itself, had a random idea – can you duplicate the card in the meantime, so the user is dragging a static version temporarily and the ones in the columns are the current state? Seems tricky to get the logic right, not sure it helps.

[mablin7]

I got the drag&drop working! What I had to do in the end was to duplicate the two columns involved and move the note on the cloned columns. Not as nice and simple as I had imagined but it works!

[mablin7]

Week 4

June 28. - July 4.

Progress

This week I've been making incremental improvements to the plugin in order to release a test version along with the progress report. So without further ado, here it is, the first test (super-early, super-buggy) release of the kanban plugin :
com.mablin7.test.kanban.jpl (1.6 MB)

(Please read further for notes for testers)

Most significant of the changes I've made this week, is that I finally gave up trying to use the Joplin search to sort notes into columns. I thought it's clever, but it created more problems than it solved, so I ended up writing my own sorting algorithm. It fetches all viable notes, then matches them against the rules defined in the config. This gives way more control over how the rules can work and also solves the 2 roadblocks mentioned in the Week 2 report:

@mablin7

1. The way I implemented the sorting of notes, by using joplin's existing search filters, doesn't allow to distinguish notebooks of the same name, but different path. [...]

2. The completedTag and completedNotebook rules can't work, because right now all rules are very simple functions, which have no knowledge of the notes they're operating on [...]

Both of these are now solvable, though the completedTag and completedNotebook rules are not yet implemented.

I've also improved the UI, bringing it closer to my mockup, but not quite there yet. The board opening should also work more reliably now, and some basic error messages are also shown, if the config is invalid.

Achieved

• Refactored the sorting logic (again), squished multiple bugs
• Improved visuals
• Basic error messages
• Improved board opening

Planned for next week

From now on, I'm hoping to release a new version every week!

• Better error messages
• Automatic reloading
• completedTag and completedNotebook rules
• More forgiving YAML parsing

Notes for testers

First of all, thank you if you're willing to give this plugin a try, user testing truly helps a lot in development.

1. Though the plugin doesn't touch the contents of your notes, it does reorganize them, and if there are unexpected bugs it can certainly mess up your collection. Make sure to have a backup of your notes before trying out the plugin!
2. You're going to need a board config. To start, try out this one, I've been using for testing, but please do not be afraid to experiment with it, since that's the point of user testing:

columns: 
  - 
    backlog: true
    name: Backlog
  - 
    name: "Ready for review"
    tag: ready
  - 
    name: Working
    notebookPath: working
  - 
    completed: true
    tag: done
    name: Done
filters: 
  rootNotebookPath: "test/nested test"
  tag: task

3. The config parsing is currently quite fragile, for example tabs are not allowed for indentation. If you're getting a YAML parse error, try this online validator, it should give you more details than the plugin. I am working on more informative error messages and making the parser more forgiving.
4. The board currently only gets updated if you reopen it. This means that if you changed the config, or added/edited the notes on the board, you need to open an unrelated note to close the board, the open it again to reload. The next release will surely include a better solution.
5. If you find any bugs, or have any suggestions, post on the main thread please.
6. Enjoy!

[mablin7]

There was a bug in the test release, which prevented the board from opening. Please use this fixed version:
com.mablin7.test.kanban.jpl (1.6 MB)

[bepolymathe]

Hi @mablin7

Thanks for that. I try to do a quick test but the tasks do not appear (I took exactly your test configuration). In the "nested test" folder you insert tasks with a "ready" tag for example? And that's all. Is this right?

[mablin7]

Hey all! Here's another hotfix which allows not to specify rootNotebookPath
com.mablin7.test.kanban.jpl (1.6 MB)

@bepolymathe Sorry, I wrote the progress report in a hurry and forgot to actually include usage instructions. So here it is:

Installation

1. Download the .jpl file
2. Open Joplin > Tools > Options > Plugins > Gear icon > Install from file
3. Select the file you downloaded
4. Restart Joplin

Usage

1. Create a new note anywhere. This will be the "board note"
2. Paste this basic config into your board note

```kanban
columns: 
  - 
    name: Backlog    
    backlog: true
  - 
    name: Working
    tag: wip
  - 
    name: Done
    completed: true
    tag: done
```​

3. Switch to another note and switch back, or restart Joplin. (this will be improved soon!)
4. Now you should see the board open on the right side with all your notes, which were in the same notebook as the board.
5. Drag the notes around and see their tags update.

You can also try notebook-based columns instead of tag based ones. Replace the config with this:

```kanban
columns: 
  - 
    name: Backlog    
    backlog: true
  - 
    name: Working
    notebookPath: wip
  - 
    name: Done
    completed: true
    notebookPath: done
```​

Make sure that the notebooks you specify here exists in the same notebook where the board is!

Example notebook structure:

📁 board
├─ 🗒️ board note
├─ 🗒️ backlog notes/tasks...
├─ 📁 wip
│ ├─ 🗒️ wip notes/tasks...
├─ 📁 done
│ ├─ 🗒️ done notes/tasks...

[bepolymathe]

Thank you for this clarification. After a first try following your instructions, it seems that the tags are not updated if I move a note or a task.

[mablin7]

Hm, that's bad. If it's not private, could you send me a screenshot of your setup? I just want to see your board note, and the notebook where it's in.

Also, could you try enabling debug mode, as described here:
https://joplinapp.org/debugging/#desktop-application

and send me the logs you see? You should see two consoled pop up, one in the main joplin window and one in a separate window. I need logs from both

[bepolymathe]

Yes of course.

[mablin7]

That's strange... Could you also share the logs?

[uxamanda]

Fun to play with! The drag and drop feels nice and responsive. I know you are not done, so my apologies if some of my "bugs" are just TBD!

I tested this on Mac, Joplin 2.2.1 (prod, darwin).

Some things I noticed with the Tag-based Example:

• the board wouldn't load notes if the name of the notebook has a space in it (e.g. notebook Testing Kanban)
• the first time i moved a note to the "working" column it went back to the backlog without "sticking". This seems to be because I didn't have the WIP tag created yet. once i moved it again, it worked properly adding and removing the tag
• have to switch to a new notebook (not just a new note) to get the board to re-render (because clicking notes in the same notebook keeps the board on the screen I think)
• sometimes moving a note caused another note to move. for example, in this case I move the note called "test" to the working column and the note "test note" moved too. This doesnt seem to happen every time, and as you can see "test note" is still marked as done. They are only semi-linked and don't always do the same thing when I move them, maybe a bug because they are named similarly?
  
  
• moving a note (not a todo) to done moves it visually, but no tag is added, so it doesnt "stick" when you reload
• errors i see in the console (could be from other issues)

Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in a useEffect cleanup function.
    in NoteEditor (created by Connect(NoteEditor))
    in Connect(NoteEditor) (created by ResizableLayout)
    in div (created by styled.div)
    in styled.div (created by ResizableLayout)
    in div (created by Resizable)
    in Resizable (created by ResizableLayout)

UserWebviewIndex.html:1 Uncaught (in promise) Error: Not Found
    at Object.<anonymous> (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/rest/routes/tags.js:25)
    at Generator.next (<anonymous>)
    at fulfilled (/Applications/Joplin.app/Contents/Resources/app.asar/node_modules/@joplin/lib/services/rest/routes/tags.js:5)

• Testing the notebook path example didn't seem to work for me. Only the backlog notes loaded, and moving notes into a different column just disappeared them from the board (although it did move them to the correct folder.)
  
  
  
• For the tag based example, the notes with the tag "WIP" needed to be in the same folder as the board note to appear, notes with that tag in different folders were ignored.

@mablin7

You should see two consoled pop up, one in the main joplin window and one in a separate window. I need logs from both

FYI, I only see the dev console when I run in debug, not sure where to see the separate window.