[Update] No that's not it. I remove the scheduled tasks and the disk IO is still high.
Now it still seems like it's a general issue of docker itself.
@laurent I have a theory now.
A task maybe scheduled by setInterval, which accumulates if the callback is too slow.
To avoid overlapping tasks, a new run is canceled if the previous one is still running.
However, the task state is loaded from disk, and maybe worse, it seems that it's loaded twice?.
This disk loading part can be slow, e.g. when the system is under high disk IO. In this case we'll spawn a new disk loader (i.e. database connection) every few seconds.
[Speculated, Not fully verified] Later, even if there is no longer external disk IO, the pending database connections themselves may still produce high disk IO, blocking new connections & scheduled tasks.
This matches with my observations:
- joplin server only has high disk read, not write
lsofandstraceonly shows files about sqlite- This high-disk-io issue only happenes when my other matenance scripts (backup, indexing) are running.
To reproduce the issue, I just start multiple processed that read the disk simutaneouly. (I use rclone md5sum /some/dir myself)
I plan to verify my theory by disabling tasks #11 and #12.