RFC: Architecture for a Secure Plugin Ecosystem

[laurent]

@akshajrawat

CodeQL and SEMGREP are the most closest competitor.
Comparing Semgrep and CodeQL · Doyensec's Blog : states how codeQL perform better than semgrep in standard benchmark testing, though also making a lot of false positives + higher execution time. Meanwhile, on a real world code both semgrep and codeql gave 100% detection result, though codeql still generated 2% false positive (things which it detected as threats but were actually not).

The main tradeoff of using CodeQL here would also be how hard it is to write custom rule sets in it. To write custom rule CodeQL untilized QL language which is way more complex than semgrep's YAML based rulesets.
CodeQL also generated a lot of false positved (noise).

Semgrep is chosen here because its syntax for writing rules is very high level and closely mirrors standard source code while also performing significantly equal to other tools.
This makes it significantly easier for the Joplin's core team to write and maintain custom security rules specific to the Joplin API.

I feel this analysis is again based on an unverified assumption: that writing rules is hard. But first are we even going to write any rule? I see the example on their blog: User input appears to be compared in an insecure manner that allows for side-channel timing attacks.. We are definitely not going to manually write down hundreds of rules to cover every single case out there (and miss hundreds of edge cases and possible exploits while doing this). Surely there's a way to automate this or existing rules we can use?

And if we do have to write certain rules, this is now trivial thanks AI. I'm not saying, because of that we should use CodeQL, but the whole analysis seems to be based on something we won't do (writing rules), or something that is assumed is hard but is not (thanks to AI).

@akshajrawat

Because LLMs are prone to hallucinations, they will play a strictly supporting role in the review pipeline acting as a post-analysis assistant for human reviewers, not an authoritative decision-maker.

As far as I can see hallucinations are rare in code reviews these days and it's only going to get better, so that shouldn't be a reason not to use them.

Before using them anyway we need to determine: do we need to use them? Perhaps you can start evaluating the code review tools on some plugins and see what comes out?

I've added the current workflow and the workflow that we'll acheive after this system is live at the first and last section of the proposal.

Thanks for adding this, but the proposal shouldn't start with this. Please review its structure perhaps based on https://discourse.joplinapp.org/t/gsoc-2026-how-to-submit-your-proposal-draft/49137

But if this is something which overcomplicate the process we can drop it and the pipeline would still work fine.

I think what confuses me is that you talk about multiple workflows and some of them "waking up", etc. Please get yourself familiar with GitHub Actions - what you describe could be just one workflow with sequential jobs, and each job can have its own secure context.

I've shifted the /approve based CI to label based execution, so now only user with Triage access, Write access and Maintainer/Admin access can trigger the approval flow using status : approved label.

So does that seem like a better approach to you? You don't need to agree with me automatically, I can be completely wrong sometimes too. So please keep looking at it critically.