It was not related to the keyboard software, because the software was completely uninstalled at that point (this is why I wanted to delete the folder). So Joplin was the only one accessing that folder.
It wasn't uninstalled properly since it was still locked by Joplin and maybe other processes. We don't lock any files expect Joplin's own config files so that's why I'm saying it's probably that keyboard that's doing something weird.
I checked using the resource monitor the locks on the folder path, and Joplin was the only executable locking it.
I have no idea how the locking system works, but once I exited Joplin, I could delete the folder.
I don't know how important this is and if it poses any security threats, but I did find it unsettling that somehow Joplin had access to that folder.