This is likely related to some other Joplin performance issues. The fact that it only happens when switching notebooks sounds like a different kind of slowdown. It could be that all plugins reload during the switch.
Do you have any other plugins installed? If so which ones?
It might be worth disabling all plugins except for this one, to see where the performance issues are coming from.