Maybe this is due to the version of electron/electron-builder we are using.
I am also using macOS Monterey Version 12.3.1 with M1 chip, and its working fine for me.
Try to remove the APP and /Users/<your_username>/Library/Application Support/Joplin and Install again after a restart.