Started working on the final scanning flow using CodeQl as a main scanner. Deeply analyzed the whole threat model, made additions and updated flows and finalized a set of 21 custom rules.
Wrote mock tests for these rules and made sure all of them passes.
Fork test branch : GitHub - akshajrawat/plugins-test · GitHub
Furthermore, tested all of these rules in top recommended joplin plugin to minimize as much false positives as possible.
Tested the scanning workflow directly in the github and made sure the scan result are shown in the issue comments in a reviewer friendly manner!
This is the repository where I am doing the testings. Not everything is finalized yet though, I'll complete it this week and update the plugins-test repository. Also, the test on 20 repositories were done locally so I dont have data for it. After the final changes I will do those again and provide you the data.
/.github/codeql contains all the rules and an AI generated rules.md for now.
/scripts/tests contains all the test setup. I am using a setup where /mocks will have all the code that intentionally violates the rules and than we can run codeql on it and parse the result for testing.
Thanks for the update. Please see my comments below:
/scripts/tests contains all the test setup
CodeQL has codeql test run - is that what you are using for testing? If not please use this as we want to use standard tools.
AI generated rules.md for now.
This is probably a good use of AI, however what is generated needs to be reviewed, including rules.md. There's a chance it made some mistakes, and very high chance it added a bunch of useless and duplicate rules, and that will make it hard to get scan results that makes sense. For example, a given plugin might fail two different rules for very similar reasons thus adding noise to the report.
We need something clean and properly reviewed.
Updating the plan
As a few things have changed since you started the project, could you please provide an update plan?
I think the goal eventually should be to have the plugin-test repository setup so that the whole review flow can be done:
developer publishes a plugin via the new CLI
a submission issue is created on a test registry repo
CI scans it with CodeQL using a published ruleset and threat model
posts a structured report
a maintainer applies a label
and the publish job uploads the .jpl artifact to a Releases page
And there should be this requirements:
Being able to process the top 20 plugins without any false positive (avoiding false positive is very important for this project)
And 5-10 examples that shows that actual threats are being detected. So doing the above steps with a test maliscious plugins, and demonstrating that the threats are flagged.
Please create a PLAN.md file at the root of the repository with your schedule and what you plan to deliver and let me know when it's done.
I will now start making few test plugins that can be used to verify if the rules are doing what they are supposed to do.
I will be using the test mock code in : plugins-test/.github/codeql/tests at master · akshajrawat/plugins-test · GitHub , for making the test plugins, so we can also confirm that the test code are also checking their respective rules.
The update registry workflow, that will work after the label is added will depend on the code we have to get merged on plugin-repo-cli, so should i open a pr for it in parallel with the testings?