BibTeX Plugin

Introduction

Hi everyone,
I'm Abdallah Ahmed, a freshman CS student from Egypt. I will be developing a BibTeX Plugin for Joplin to help people who use the application for research purposes.

The development lifecycle of the plugin will be carefully monitored here alongside other discussions about the implementation process.

Project goals (Deliverables)

According to the spec provided by Mr. @Laurent22,

Adding support for BibTex references should be possible via the upcoming plugin API. It would require hooking into the config screen, note editor, and note viewer.

Three main tasks need to be done in the plugin, namely:

  • Add a way in the config screen that allows the user to import a local BibTeX file.
  • Create a new command that displays a pop-up dialog enabling the user to insert a reference in the current place in the note.
  • Enhancing the markdown parser so that it can distinguish references from the note content and renders them in a specified format.

In addition to the above tasks, other crucial aspects must be integrated into the implementation process:

  • User-friendly UI.
  • Unit tests and Continuous Integration (Using Travis CI).
  • Documentation for developers and users alike.
  • Bi-weekly reports (or blog posts) containing a log of everything that has been done and what's next.

Implementation

The implementation relies heavily on Joplin's plugin API to do all sorts of things: showing dialogs, accessing settings, and modifying content. Below are the exact modifications that will be carried out by the plugin:

Config Screen

As required by the spec, the plugin should have a settings section in the config screen to allow the user to set the .bib file path. This can be added dynamically by calling the following methods whenever the plugin starts:

// To add a new section in the settings page
await joplin.settings().registerSection();

// To add a new key/value settings field
await joplin.settings().registerSettings();

Also, the section should have a meaningful name (like 'citation') and an expressive user-friendly icon.

Upon setting the file path of the citation source, the data of the file should be loaded into memory and then parsed by the Parser class, whose only purpose is to validate and parse .bib files. After validation and parsing, the Parser should return an Array of Reference objects. References returned by the parser should then be stored in the DataStore, which is responsible for storing citation data and providing it when needed. The Parser class must also report errors to the data store when encountering them (like when there is a parsing error or the file path is invalid). Regarding the implementation of the Parser class, there are various pre-built options out there (such as BibTeX-JS, BibTeX.js, Citation.js); A from-scratch implementation is also feasible. Moreover, the Parser should just act as a thin wrapper that hides most of the implementation details and the underlying libraries used.

Note Editor

The modifications applied to the Note Editor will involve registering a new addBibTexReference command by utilizing the Commands API as follows:

await joplin.commands.register({
    name: 'addBibTexReference',
    label: 'Add BibTex Reference',
    iconName: 'ICON',
    execute: () => {}
});

This command will be executed by either one of the following methods:

When the command is invoked, a popup dialog will be displayed allowing the user to search through locally stored references and insert whichever one into the note content. This behavior can be implemented with the aid of Dialogs Views API as follows:

await joplin.views.dialogs.create("dialog_id");

To provide the search feature, a SearchUtility class will be built so that it can request citation data from the DataStore and filter the results based on the search query.

Note Viewer

The last bit of work will be directed towards the Note Viewer as the plugin is required to register a markdown-it content script that will parse the note content and render the references in a distinguishable style. Besides, the full content of the references must automatically appear at the bottom of the note viewer.

The following diagram describes the whole process in a high-level fashion:

File Watchers

There is a possibility the .bib file changes during the running time of the plugin. To account for this, there ought to be a file watcher in place that listens for changes and reports them to the Parser class to start the parsing process again. Available options are:

Testing

Writing tests is a crucial part of the software development lifecycle. Most software products should have some kind of automated testing process whose mission is to ensure the delivery of correct functionality, the BibTeX plugin is no Exception. Most components in the plugin will be accompanied by test cases to assert the component is properly working. Unit tests, for example, will be applied to pure isolated components that don't have many side effects (like Parser class). On the other hand, components that depend mainly on side effects will have lower chances of being unit-tested; they will probably be tested using integration or manual testing (like UI components).

Documentation

Like testing, documentation is an important part of any software product. The plan for documentation includes an API reference for every component so that future developers will have an easier time maintaining the existing codebase. As for users, if the situation demands it, there will be a step-by-step guide (or tutorial) explaining the usage of the plugin.

UI/UX

UI/UX is equally important as the other sections of the project. Before the coding work commences, a suitable amount of time should be assigned to the design of UI/UX. There are several software products out there that are used for UI design and prototyping, such as:

The following Figma sketch demonstrates the basic appearance of the pop-up dialog:

16 Likes

I love that you are working on this!

I have a question regarding your project: Some reference mangers, e.g. Zotero, offer the functionality to export BibTex files that are updated automatically. Other applications like RMarkdown or Zettlr (nice documentation) import these, which makes integrating references a really smooth process. The BibTex file is used as a „database“ for references which is always up to date because it’s directly connected to the reference manager.
Is this out of scope for your plugin or would it support this behaviour ?

5 Likes

If I understand correctly, your question is: Would the plugin be able to detect changes affecting the .bib file in real-time?
If this is your question, then the answer is yes. As I stated above,

There is a possibility the .bib file changes during the running time of the plugin. To account for this, there ought to be a file watcher in place that listens for changes and reports them to the Parser class to start the parsing process again.

4 Likes

Thats what I meant. Sorry, I just looked at the graphic and missed explanation on file watchers.

1 Like

No, you are right. I should probably include also the file watcher in the main diagram. Thanks for the hint :smiling_face_with_three_hearts:

1 Like

Hey, I've been hoping for this plugin for a long time. Really excited!

Will the bibtex file also be synced?

That's a very good idea, but I cannot guarantee that I will have time to work on that within the period of Google Summer of Code. After delivering the core functionality, I guess I can start working on that.

1 Like

Cool.

I also wanted to share my thoughts on another approach on reference management in Joplin.

Every note on Joplin could also get an optional bibtex entry as metadata, containing fields like author, year, etc.

A note would then contain, well, notes to a publication. The document could be attached to the note and the library metadata is stored with that note instead of a central place. Would also simplify copying/sharing that note.

Then there would not be a bibtex file, but we would just query all notes containing that metadata.
Ideally it would be possible to export all this metadata combined in a bibtex file.

I also understand that this would mean that all entries would require a dedicated note and that might not be desired.

This is a good idea, but I guess it is a bit out of the scope because, at the end of the day, Joplin is not reference management software. Instead, it is a note-taking app with synchronization capabilities and lots of other features.

I understand. Good luck with everything and let me know if you need someone to test stuff. :slight_smile:

1 Like

Of course, no worries.

@xUser5000 just it case you haven't seen, there are some suggestions in this comment on Github

1 Like

This comment is of huge help to me. It described precisely how to work with in-line citations. Thanks

1 Like

Hi, I just want to wholeheartedly second the idea of trying to make the Markdown syntax the plugin uses (and creates) as similar as possible to that used by Pandoc with citeproc, as Klemet suggested on the GitHub issue. Besides Pandoc, this syntax is also used by other tools like RMarkdown (which uses Pandoc internally) and Zettlr, and is the closest thing to a "standard" way to extend Markdown to handle citations that I'm aware of.

I can see the sense in the spec suggested previously of using Markdown links, e.g. [@SmithStatisticalMechanics2015](http://dx.doi.org/10.1002/andp.19053221004), since this would degrade somewhat gracefully in any Markdown editor/renderer. However, being able to render the citations fully with Pandoc or other tools seems like potentially a more useful form of graceful degradation for a lot of people (and way to avoid vendor lock-in). For me lack of interoperability with Pandoc would be a blocker to porting my research note library to Joplin.

I don't know how set in stone the original spec is on this point but I hope you can take this into consideration!

2 Likes

Unfortunately, I have to stick to the specification for now. But this can change later in the future.

Hi, I am really looking forward to see this integration. Maybe having a look at this plugin for obsidian is also helpful: obsidian-citation-plugin. I would highly appreciate to be able to set specific BibTex fields myself. I don't personally like to only being able to look a paper up online. So, an option to set a link to Zotero instead of the DOI or an URL would be amazing. Maybe adding this to the options of the Plugin could be an idea?

1 Like

Hey there !

I'm just posting here following @xUser5000 response on a comment I made on GitHub, concerning the way that markdown-it should ideally handle quotes. The idea was that it should replicate, as best as possible, the behaviour of the citeproc plugin of Pandoc, that handles citations. This is because citeproc gives a very flexible syntax that allows citations in the text to take very different forms, which I find really awesome !

I made a small test file to test the different aspects of the syntax to see what Pandoc did with it, using the markdown-preview-plus plugin of the text editor Atom, which allows to use Pandoc with custom arguments to preview the output of a markdown file.

Here, I'll paste the outputs of Pandoc so that one can see how it works. The syntax's rules are summarized on this page.


Testing citations with Pandoc

@dijkstraNoteTwoProblems1959

This is shown by Dijkstra (1959).

[@dijkstraNoteTwoProblems1959]

This is shown by (Dijkstra 1959).

-@dijkstraNoteTwoProblems1959

This is shown by -Dijkstra (1959).

[-@dijkstraNoteTwoProblems1959]

This is shown by (1959).

-[@dijkstraNoteTwoProblems1959]

This is shown by -(Dijkstra 1959).

@dijkstraNoteTwoProblems1959 and @dijkstraNoteTwoProblems1959

This is shown by Dijkstra (1959) and Dean (1997).

[@dijkstraNoteTwoProblems1959 and @deanFindingOptimalRoutes1997]

This is shown by Dean (1997).

[@dijkstraNoteTwoProblems1959; and @deanFindingOptimalRoutes1997]

This is shown by (Dijkstra 1959; and Dean 1997).

[@dijkstraNoteTwoProblems1959; and -@deanFindingOptimalRoutes1997]

This is shown by (Dijkstra 1959; and 1997).

@dijkstraNoteTwoProblems1959 [p. 33]

This is shown by Dijkstra (1959, 33).

@dijkstraNoteTwoProblems1959 [pp. 33-35, 38-39]

This is shown by Dijkstra (1959, 33–35, 38–39).

@dijkstraNoteTwoProblems1959 [chap. 1 pp. 33-35]

This is shown by Dijkstra (1959, chap. 1 pp. 33-35).

Dean, D J. 1997. “Finding Optimal Routes for Networks of Harvest Site Access Roads Using GIS-Based Techniques” 27: 12.
Dijkstra, E. W. 1959. “A Note on Two Problems in Connexion with Graphs.” Numerische Mathematik 1 (1): 269–71. https://doi.org/10.1007/BF01386390.

I hope this will be useful !

3 Likes

This looks pretty good.

But again, Joplin is a note-taking application with synchronization capabilities; it is not a reference management tool like Zotero. This plugin just serves as middleware between reference management software and Joplin by importing and listening to changes affecting local BibTeX files. If you want to edit the BibTeX fields, do it with Zotero.

1 Like

But again, Joplin is a note-taking application with synchronization capabilities

This is what I like it for. Zotero has URIs for each element, making it possible to link to these elements from somewhere. These URIs can also be exported with the BibTex files.

Maybe adding this to the options of the Plugin could be an idea?

Here I meant the config screen: BibTeX Plugin: UI/UX Document

My intention was to add the URI to the citation (as it is accessible in the BibTex file) instead of the DOI or an URL.

As I don't use Joplin to publish my notes, I rather prefer to link to my files than having a link to a website. If I collaborate on a note it could be handy to have the DOI. This is why I suggested a field in your config screen to make it possible to choose. If you have a closer look at that Obsidian Plugin it might get more clear what I mean.

Thank you a lot of taking care of this Plugin anyways.

2 Likes

Excellent! One thing worthy of note is how Zettlr (repo: GitHub - Zettlr/Zettlr: A Markdown Editor for the 21st century.) uses the bibTex plugin from Zotero. Bear in mind that many here are using Zotero in parallel with Joplin, and that this plugin will be wonderful for importing any hot citations via the Zotero BibTex export option.

5 Likes