This is week 6 (zero-based) in the journey of developing a BibTeX Plugin for Joplin. In this week, I'm gonna work on the following features:
- Enable importing multiple
.bibfiles. - Automate the release process.
1 Like
Hey, @xUser5000 ! Thank you for your hard work, it already works wonderfully : ) !
I'm a bit surprised, however, that the format of the reference seem to be fixed, and not very pretty to read:
It seems like changing the format/text of the reference from something else than the citekey (i.e. changing bettsForestDegradationNot2021 into something like Betts et al. 2021) breaks the reference.
I'm doubly surprised that this aspect of the plugin does not appear in the trello, or on the GitHub page. From my point of view, an important aspect of references is that they should be included nicely in the text, and especially follow the format of academic literature (e.g. "(Betts et al. 2021)", or [1], etc.) rather than a code or a citekey that is rather strange to read.
A potential solution that I'd see to this issue would be to replace the DOI link in the current system with the citekey, and to let the user input the text that they want for the link. The plugin could then "read" the citekey contained in the link. What I mean, is that instead of having:
[@bettsForestDegradationNot2021](http://dx.doi.org/10.21203%2Frs.3.rs-686817%2Fv1)
We would have:
[(Betts et al. 2021)](@bettsForestDegradationNot2021)
However, such a solution would present another problem when compared to how references are currently handled in most current software: that as soon as the user have chosen the reference(s) to add, the rest of the formatting (both in the end section of the document and in the place of the text where the reference is inserted) should be completely automatic. Therefore, the user should not even have to input "Betts et al. 2021"; from my point of view, this should be done for them, as is done in Zettlr and Pandoc.
That's why I still think that the best implementation of this might be the current functioning of Pandoc, which just reads citekeys and other syntax to format the citations automatically, very quickly (no need to write a full link), and with total and complete customization from the user in how they ultimatly look like. For that, I refer to post I made previously: BibTeX Plugin - #18 by Klemet .
Still, I really don't want to sound either judgmental, or ungrateful about what you've done. I'm really impressed that the plugin have already come so far, and it's still a wonderful new option that will make Joplin incredible for use in academia : ) ! I'm just afraid that as is, it's not going to be very popular as the formatting is pretty strict and strange from my point of view.
1 Like
Thanks @Klemet, your suggestions are really helpful and I'm grateful that you're testing the plugin and providing constructive feedback.
Regarding your suggestions, it was in my plan initially, but I got carried away with other things throughout the past weeks and completely forgot about it. Don't get me wrong, I still consider this to be a very important feature and chances are that I'm gonna work on it very soon (either this week or the next). Anyway, the way pandoc renders in-line references is pretty impressive and, as you said, makes it more customizable. However, the syntax used to render references cannot be fully supported because, as stated in the spec, references should be written as a valid markdown URL in order to be supported by any markdown tool.
With that said, I will try to figure out something that has the best of both worlds. More specifically, I want to devise a not-so-new syntax that can be parsed as a valid markdown URL and can be used to customize rendering references.
What do you think? The syntax does not have to be present right now, but what about the idea itself?
Thank you for your answer ! I did not see this detail in the spec.
I guess that the need to adapt to any Markdown tool makes sense; but at the same time, using this syntax also means that Pandoc won't be able to parse it properly, since this syntax would become unique to your plugin. Maybe the question is about either using the tools that the Markdown language already have (e.g. URLs); or to go along what Pandoc has done, which is starting to be used in other software too (like Zettlr).
I guess that I'm currently way into the Pandoc way right now, as I think that the syntax is very easy to read (respecting the philosophy of Markdown from what I understand of it), and also very flexible. I don't really see the importance of sticking to a basic functionalities of Markdown, especially since Joplin uses many extensions of the language, if I'm not mistaken (and for good reasons, I believe !).
Thus, as it stands, I don't think that I'd be using the plugin as it is, as I don't see much advantages other keeping my references in Pandoc format for future outputs. If I start using it, I'll have to ultimately change it again everytime I want to use Pandoc on Joplin notes; or if I don't like the formatting made by the plugin, which is the case for me right now.
Now, I am not writing that as a way to pressure you, or to "punish" your work in any way; if anything, I'm feeling quite ungrateful for writing it ! But I guess that pleasing every user is an impossible thing : ). However, I believe that it's important to please most users; and I'm really not sure that the current syntax will.
I've been thinking, and wondering if proposing the option to users might not be best if you want to respect the spec. Like, the "Options" menu could have a "Pandoc-style citations" button. If enabled, the plugin would insert and read references using the Pandoc syntax. If not, something resembling the current syntax would be used. Apart from that, I've got trouble seeing how the current link syntax could be kept without sacrificing the rest of the advantages of the Pandoc syntax.
1 Like
I like this idea very much. By adding an option for rendering pandoc-style citations, users will have the freedom of choosing whether to render references in the default way or to use pandoc style. I'm gonna add this to Trello and work on it starting from the upcoming week.
1 Like
Hi @xUser5000, thanks for your outstanding work! This was a sorely missed feature, in my opinion.
I have a bibtex file managed via BibDesk, with about 800 entries. When I reference your plug-in to it. I get this
Error while parsing BibTeX file:
expected "text", got EOF at line 20 col 1:
^ (String->Text->MathString->Text)
These are the first 31 lines (actually the headers and the two entries, the 20th line is the blank line between them).
I must say, however, that the error keeps throwing a reference to "line 20 col 1", no matter of the actual content of the line. I already tried to delete the first references, the headers and so on, and the error is the same. Is there some hidden cache that your plug-in keeps reading, overlooking changes in the *.bib files?
Update
I tried to export the *bib file from BibDesk as "minimal BibTex", and it seems to work.
Is anybody else having problems with BibDesk-generated *.bib files?
Thank you in advance
Francesco
%% This BibTeX bibliography file was created using BibDesk.
%% http://bibdesk.sourceforge.net/
%% Saved with string encoding Unicode (UTF-8)
@electronic{CheshireHilberSchoni2021,
author = {Paul Cheshire and Christian Hilber and Olivier Schöni},
date-added = {2021-06-30 18:08:09 +0200},
date-modified = {2021-06-30 18:08:14 +0200},
generator = {WordPress 4.9.18},
keywords = {NDS},
lastchecked = {06/06/2021},
msapplication-tileimage = {https://blogsmedia.lse.ac.uk/blogs.dir/123/files/2020/04/cropped-lse-logo-blogs-footer-270x270.jpg},
title = {Why central London has seen the biggest rises in house prices, despite COVID},
twitter:text:title = {Why central London has seen the biggest rises in house prices, despite COVID},
url = {https://blogs.lse.ac.uk/covid19/2021/03/31/why-central-london-has-seen-the-biggest-rises-in-house-prices-despite-covid/},
urldate = {March 31st, 2021},
viewport = {width=device-width, initial-scale=1, shrink-to-fit=no},
year = {2021},
Bdsk-Url-1 = {https://blogs.lse.ac.uk/covid19/2021/03/31/why-central-london-has-seen-the-biggest-rises-in-house-prices-despite-covid/}}
@article{Kahn1973,
author = {Kahn, Louis I},
date-added = {2021-06-28 17:49:02 +0200},
date-modified = {2021-06-28 17:49:41 +0200},
journal = {A+ U},
keywords = {NDS, home, house},
pages = {5--222},
title = {The Room, the Street, and Human Agreement},
volume = {1},
year = {1973},
Bdsk-Url-1 = {https://www.douban.com/note/115159310/}}
1 Like
Can you post a sample of both? What's the difference?
Right!
Thanks for your attention, Laurent!
BibDesk *.bib, first 31 lines:
%% This BibTeX bibliography file was created using BibDesk.
%% http://bibdesk.sourceforge.net/
%% Saved with string encoding Unicode (UTF-8)
@electronic{CheshireHilberSchoni2021,
author = {Paul Cheshire and Christian Hilber and Olivier Schöni},
date-added = {2021-06-30 18:08:09 +0200},
date-modified = {2021-06-30 18:08:14 +0200},
generator = {WordPress 4.9.18},
keywords = {NDS},
lastchecked = {06/06/2021},
msapplication-tileimage = {https://blogsmedia.lse.ac.uk/blogs.dir/123/files/2020/04/cropped-lse-logo-blogs-footer-270x270.jpg},
title = {Why central London has seen the biggest rises in house prices, despite COVID},
twitter:text:title = {Why central London has seen the biggest rises in house prices, despite COVID},
url = {https://blogs.lse.ac.uk/covid19/2021/03/31/why-central-london-has-seen-the-biggest-rises-in-house-prices-despite-covid/},
urldate = {March 31st, 2021},
viewport = {width=device-width, initial-scale=1, shrink-to-fit=no},
year = {2021},
Bdsk-Url-1 = {https://blogs.lse.ac.uk/covid19/2021/03/31/why-central-london-has-seen-the-biggest-rises-in-house-prices-despite-covid/}}
@article{Kahn1973,
author = {Kahn, Louis I},
date-added = {2021-06-28 17:49:02 +0200},
date-modified = {2021-06-28 17:49:41 +0200},
journal = {A+ U},
keywords = {NDS, home, house},
pages = {5--222},
title = {The Room, the Street, and Human Agreement},
volume = {1},
year = {1973},
Bdsk-Url-1 = {https://www.douban.com/note/115159310/}}
Minimal Bibtex exported from BibDesk, always a *.bib file, first 33 lines:
%% This BibTeX bibliography file was created using BibDesk.
%% http://bibdesk.sourceforge.net/
%% Saved with string encoding Unicode (UTF-8)
@electronic{CheshireHilberSchoni2021,
author = {Paul Cheshire and Christian Hilber and Olivier Sch{\"o}ni},
title = {Why central London has seen the biggest rises in house prices, despite COVID},
urldate = {March 31st, 2021}}
@article{Kahn1973,
author = {Kahn, Louis I},
journal = {A+ U},
pages = {5--222},
title = {The Room, the Street, and Human Agreement},
volume = {1},
year = {1973}}
@electronic{Neate2020,
author = {Neate, R.},
title = {Super-rich jet off to disaster bunkers amid coronavirus outbreak},
urldate = {03/11/2020}}
@book{Eurostat2018,
address = {Luxembourg},
author = {European Commission. Statistical Office of the European Union.},
publisher = {Publications Office},
title = {Living conditions in Europe: 2018 edition},
year = {2018}}
@webpage{OECD2021,
lastchecked = {17/06/21},
url = {https://www.oecd.org/els/family/HM1-1-Housing-stock-and-construction.pdf},
year = {2021}}
1 Like
I tried adding both to the plugin and it works fine. You said that however you change the file, the message keeps showing up, maybe because you edit a file other than the one you are referencing?
I also discovered some sort of a bug. When some references don't have a year field, the year appears as -1 in the citation popup. This will be easy to fix, I guess.
Hi xUser5000!
No, I'm not editing any other file.
I simply tried to delete the entry which contains the line 20, and the error keeps popping up mentioning line 20, even if line 20 pertains to an other entry, to another field, or it is simply blank.
I see that you are right: the part of the BibDesk file I gave as an example actually works with the plug-in, in spite of being the part containing the line 20!
Therefore, I don't really know which part of the file is really producing the error. Should I post the whole file?
Can you try to narrow it down to a minimum example? Or perhaps you could share the whole bibtext file?
Ok, I'll try to split the file in parts to see which one actually produces the error.
At this point I guess that line 20 isn't the culprit.
I could finally find the real culprit.
The "$" sign, which appears in bibtex entries generated by BibDesk from the Library of Congress and in an abstract.
The first of them was located at line 1173 of the file, quite far from the line 20 mentioned in the error message.
At first I thought that the reason for the error was in the way Library of Congress records are stored by BibDesk in an "annote" field. I got rid of all these records, and the error appeared again, this time reporting a text from an abstract. I could isolate the faulty part of the abstract and I noticed that it contained the "$" signs, this time not as code, but as currency.
Just replacing "$" with USD, even leaving the Library of Congress records otherwise intact, produces an usable file.
@laurent and @xUser5000, thanks a lot for Joplin, for this plugin, and for helping me in finding a solution!
This seems to be a bug in BibDesk then as the $ sign should be escaped, or replaced by USD as you've done. If possible I'd suggest reporting this on their repo as they might not be aware of it.
Ok I'll do.
I must say that I never encountered this problem before, the same bibtex file works well with a smiliar plug-in in Atom.
Actually could you share just the bibtex entries that cause the bug, in case it's indeed something that can be fixed in our side, perhaps with different parsing parameters or something
1 Like
Here they are!
BibDesk entry imported from the Library of Congress
As you see, plenty of "$" signs here. This record is imported via BibDesk's "Searches > Library of Congress" menu command, we can therefore expect that BibDesk *.bib files have lots of entries like these. The "annote" field is the original LoC record. It is irrelevant in my workflow, but could be important for others researchers.
@book{Wolfe1981,
address = {New York},
annote = {LDR 00814pam 2200253 a 4500
001 1842835
005 20080821180115.0
008 810810s1981 nyua 000 0 eng
906 $a7$bcbc$corignew$d1$eocip$f19$gy-gencatlg
010 $a 81012589
020 $a0374158924 :$c$10.95
020 $a0374158932 (lim. ed.) :$c$50.00
035 $9(DLC) 81012589
040 $aDLC$cDLC$dDLC
043 $an-us---$ae-gx---
050 00 $aNA712$b.W6 1981
082 00 $a720/.973$219
100 1 $aWolfe, Tom.
245 10 $aFrom Bauhaus to our house /$cTom Wolfe.
260 $aNew York :$bFarrar Straus Giroux,$cc1981.
300 $a143 p. :$bill. ;$c20 cm.
650 0 $aArchitecture$zUnited States$xHistory$y20th century.
610 20 $aBauhaus$xInfluence.
991 $bc-GenColl$hNA712$i.W6 1981$p00058219071$tCopy 1$wBOOKS
},
author = {Wolfe, Tom},
call-number = {NA712},
date-added = {2020-05-31 18:37:41 +0200},
date-modified = {2020-05-31 18:37:41 +0200},
dewey-call-number = {720/.973},
genre = {Architecture},
isbn = {0374158924},
library-id = {81012589},
publisher = {Farrar Straus Giroux},
title = {From Bauhaus to our house},
year = {1981}}
BibDesk entry with "$" sign in the abstract
I don't remember if this entry was generated by me or by importing a *.bib file downloaded from the Journal website.
@article{Akbari2002,
abstract = {Urban shade trees offer significant benefits in reducing building air-conditioning demand and improving urban air quality by reducing smog.
The savings associated with these benefits vary by climate region and can be up to $200 per tree. The cost of planting trees and maintaining them can vary from $10 to $500 per tree. Tree-planting programs can be designed to have lower costs so that they offer potential savings to communities that plant trees. Our calculations suggest that urban trees play a major role in sequestering CO2 and thereby delay global warming. We estimate that a tree planted in Los Angeles avoids the combustion of 18 kg of carbon annually, even though it sequesters only 4.5?11 kg (as it would if growing in a forest). In this sense, one shade tree in Los Angeles is equivalent to three to five forest trees. In a recent analysis for Baton Rouge, Sacramento, and Salt Lake City, we estimated that planting an average of four shade trees per house (each with a top view cross section of 50 m2) would lead to an annual reduction in carbon emissions from power plants of 16,000, 41,000, and 9000 t, respectively (the per-tree reduction in carbon emissions is about 10?11 kg per year). These reductions only account for the direct reduction in the net cooling- and heating-energy use of buildings. Once the impact of the community cooling is included, these savings are increased by at least 25%.},
author = {Akbari, H.},
date-modified = {2017-10-31 08:30:33 +0000},
journal = {Environmental Pollution},
keywords = {Shade trees; Heating and cooling energy use; Carbon sequestration; Smog; Cost-benefit analysis; Heat islands, landscape ; energy saving ; energy efficiency ; outdoor ; passive cooling ;},
note = {LEB},
owner = {franz},
pages = {119--126},
publisher = {Elsevier},
review = {Some pragmatical insights about tree planting in urban areas. Very useful data for saving existmations are provided.},
timestamp = {2008.01.02},
title = {Shade trees reduce building energy use and CO2 emissions from power plants},
volume = {116},
year = {2002}}
As stated before, the reason of the error seems to be just the "$" sign.