Basically I want to know if it possible to use Markdown links across Notebooks?
It seems that this option only allows to use this link while inside the same notebook not to a different one.
No. A target note's markdown link will take you to the target note regardless of what notebook the note with the originating markdown link is in.
If you look on the left you will see that as I follow the markdown links for the two notes the notebook changes. (The gif has been recorded using a pre-release of Joplin but the same happens with v2.9.17)
Don't use the "Insert/Edit Link" function. That is for adding URLs, as in Internet addresses. Just paste the Markdown code [ESXi](:/e603a28fee5046a1a85716ce7d49afa3) directly into the note.
There also appears to be a bug. When using the Rich-Text editor you have to paste the Markdown Link into the note using CTRL+V. If you right-click the note and select Paste the code is "escaped" i.e automatically modified to not display as a link.