41

Excellent! I should document the notebook thing a little better / put it in the cli output or something.

Great to hear your feedback, thanks!

1 Like
42

I just tried to set get a headless setup going and was able to do so using @steve28’s approach. However, when I set the monitoring path to the mountpoint of a webdav mount the script failed to detect new files when I added them on the webdav server. Adding them locally (directly to the mountpoint) worked just fine. Not sure what’s causing this.

43

Yeah, I had that same problem. I was able to get it working with an NFS mount, however. For me it was an SMB mount that wouldn’t detect new files.

44

@Shamp0o @steve28 watchdog is used to monitor changes on the filesystem. Depending on the OS and the type of the mount point, it’s possible that it can fail to detect changes. But these are problems that have to be fixed in the watchdog code.

e.g. on Linux one usually monitors the change of the inode data. However, (faulty) implementations of nfs/smb might not propagate that info on the fs level or certain components might not even have support for that info at all.

45

I see. Well, I guess it’s time to look into other solutions then.

@steve28 did you change any settings in your terminal joplin app to automatically sync whenever there’s a change? I have to manually sync in order to make notes created by the rest uploader appear. Running a regular instance of joplin alongside the server like you described in your post doesn’t work for me.

update

46

I released a new version. It adds automatic tagging functionality based on OCR/body text.

Changes:

  • Ignore .part files (caused app crashes on KDE systems)
  • Added logic to automatically assign tags to newly created notes
  • Added click option to turn off automatic tag generation

The tag matching logic is not overly sophisticated, so if you don’t like it, just turn it off using the command line option --autotag=no

Thanks!

2 Likes
47

Trying to get this work with JoplinPortable in Windows.

I have installed Poppler and tesseract-OCR and ensured environment variables work for both.

I have downloaded rest_uploader-1.5.0.tar.gz, extracted with 7Zip and placed the tree in my Progs folder.

I have installed WinPyhon and confirmed in a Command Prompt window typing “python -V” returns “Python 3.7.7”

When I type

python -m rest_uploader.cli /path/to/directory

it returns

python.exe: Error while finding module specification for 'rest_uploader.cli' (ModuleNotFoundError: No module named 'rest_uploader')

I have tried adding the path to rest_uploader and rest_uploader-1.5.0 to PATH but get the same error message.

I’m guessing I’m missing the step where I ‘tell’ WinPython where rest_uploader is? This is the first time I’ve met Python and researching it has me massively confused as there seems so many different ways to use it.

I also notice there isn’t a file called rest_uploader.cli

Any help would be appreciated.

1 Like
48

Ah, packaging in python - not always the most straightforward. I think you’re most of the way there!
Try this from a powershell prompt (you might need to run powershell as admin if you don’t have a virtual environment set up):
pip install rest_uploader
Then you should be able to call rest_uploader.exe from the command line.

49

Thanks for replying so quickly

PS D:\Progs\wpy64-3770\python-3.7.7.amd64> PIP install rest_uploader
PIP : The term 'PIP' is not recognized as the name of a cmdlet, function, script file, or operable program

both in PowerShell as admin or not

Interesting that when I type python -V in powershell nothing is returned but in a Command Prompt window I get the version no. Bizarre.

So it looks like my download of WinPython doesn’t include PIP (although it says it does)

WinPython has a program called WinPython Command Prompt.exe so ran it and voila:

D:\Progs\WPy64-3770\scripts>pip install rest_uploader
Collecting rest_uploader
  Downloading rest_uploader-1.5.0-py2.py3-none-any.whl (9.3 kB)
Requirement already satisfied: requests in d:\progs\wpy64-3770\python-3.7.7.amd64\lib\site-packages (from rest_uploader) (2.23.0)
Collecting Click>=6.0
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 6.1 MB/s
Collecting tabulate
  Downloading tabulate-0.8.7-py3-none-any.whl (24 kB)
Collecting PyPDF2
  Downloading PyPDF2-1.26.0.tar.gz (77 kB)
     |████████████████████████████████| 77 kB 5.1 MB/s
Collecting pytesseract
  Downloading pytesseract-0.3.4.tar.gz (13 kB)
Collecting pdf2image
  Downloading pdf2image-1.13.1-py3-none-any.whl (10.0 kB)
Collecting watchdog
  Downloading watchdog-0.10.2.tar.gz (95 kB)
     |████████████████████████████████| 95 kB 6.4 MB/s
Requirement already satisfied: chardet<4,>=3.0.2 in d:\progs\wpy64-3770\python-3.7.7.amd64\lib\site-packages (from requests->rest_uploader) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in d:\progs\wpy64-3770\python-3.7.7.amd64\lib\site-packages (from requests->rest_uploader) (1.25.9)
Requirement already satisfied: certifi>=2017.4.17 in d:\progs\wpy64-3770\python-3.7.7.amd64\lib\site-packages (from requests->rest_uploader) (2020.4.5.1)
Requirement already satisfied: idna<3,>=2.5 in d:\progs\wpy64-3770\python-3.7.7.amd64\lib\site-packages (from requests->rest_uploader) (2.9)
Collecting Pillow
  Downloading Pillow-7.1.2-cp37-cp37m-win_amd64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 6.4 MB/s
Collecting pathtools>=0.1.1
  Downloading pathtools-0.1.2.tar.gz (11 kB)
Building wheels for collected packages: PyPDF2, pytesseract, watchdog, pathtools
  Building wheel for PyPDF2 (setup.py) ... done
  Created wheel for PyPDF2: filename=PyPDF2-1.26.0-py3-none-any.whl size=61087 sha256=f6cbc86401db9268147b8a3e5e5c83d5a981427bd999923d50f89e109a9c51f8
  Stored in directory: c:\users\roger\appdata\local\pip\cache\wheels\80\1a\24\648467ade3a77ed20f35cfd2badd32134e96dd25ca811e64b3
  Building wheel for pytesseract (setup.py) ... done
  Created wheel for pytesseract: filename=pytesseract-0.3.4-py2.py3-none-any.whl size=13439 sha256=46280639c984c8545fa5ecdee583c90bfae3fe6e49de19feec5d683e5955722c
  Stored in directory: c:\users\roger\appdata\local\pip\cache\wheels\51\71\e6\07d988150d601c7c4ec08665d9c048a6a4c1c57f25c2c34e53
  Building wheel for watchdog (setup.py) ... done
  Created wheel for watchdog: filename=watchdog-0.10.2-py3-none-any.whl size=73609 sha256=1b0317709c21be3ab74a566715865df324e78d554a3df10e68ed84bfe057a5aa
  Stored in directory: c:\users\roger\appdata\local\pip\cache\wheels\36\93\24\29e375ee74e0f178889e6906cb73e693e9f06a5f589dcee6b9
  Building wheel for pathtools (setup.py) ... done
  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8790 sha256=c1c7e6babf79f35b7dc1a0305f8af9a83953b3ce2b1281959c4d08091e1c81fd
  Stored in directory: c:\users\roger\appdata\local\pip\cache\wheels\3e\31\09\fa59cef12cdcfecc627b3d24273699f390e71828921b2cbba2
Successfully built PyPDF2 pytesseract watchdog pathtools
Installing collected packages: Click, tabulate, PyPDF2, Pillow, pytesseract, pdf2image, pathtools, watchdog, rest-uploader
Successfully installed Click-7.1.2 Pillow-7.1.2 PyPDF2-1.26.0 pathtools-0.1.2 pdf2image-1.13.1 pytesseract-0.3.4 rest-uploader-1.5.0 tabulate-0.8.7 watchdog-0.10.2
WARNING: You are using pip version 20.0.2; however, version 20.1 is available.
You should consider upgrading via the 'D:\Progs\WPy64-3770\python-3.7.7.amd64\python.exe -m pip install --upgrade pip' command.
D:\Progs\WPy64-3770\scripts>

This prompts some questions

  • is that what you expected? is it all OK?
  • it says it downloaded rest_uploader…whl - so I needn’t have downloaded the .gz version?

Now I get:

D:\Progs\WPy64-3770>python -m rest_uploader d:\Temp\Scans
D:\Progs\WPy64-3770\python-3.7.7.amd64\python.exe: No module named  rest_uploader.__main__; 'rest_uploader' is a package and cannot be directly executed

I have found that rest_uploader.exe is present in D:\Progs\WPy64-3770\python-3.7.7.amd64\Scripts

Any ideas?

50

having installed via pip, you don’t need to call it as a python module… (although I would have thought that would work?) – just call it using rest_uploader.exe.
rest_uploader.exe d:\Temp\Scans
evidently the .exe is optional, at least on my Windows box.

51 
D:\Progs\WPy64-3770\python-3.7.7.amd64\Scripts>rest_uploader.exe d:\Temp\Scans  --autotag=no
Launching Application rest_uploader.cli.main
Language: eng
Automatically Tag Notes? no
Paste your Joplin API    Token:
Monitoring directory d:\Temp\Scans for files
Exception in thread Thread-1:
	Traceback (most recent call last):
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\threading.py", line 926, in _bootstrap_inner
	self.run()
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\watchdog\observers\api.py", line 196, in run
	self.dispatch_events(self.event_queue, self.timeout)
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\watchdog\observers\api.py", line 369, in dispatch_events
	handler.dispatch(event)
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\watchdog\events.py", line 336, in dispatch
	}[event.event_type](event)
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\rest_uploader\rest_uploader.py", line 70, in on_created
	self._event_handler(event.src_path)
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\rest_uploader\rest_uploader.py", line 53, in _event_handler
	if filesize < 1 or (ext == ".pdf" and not pdf_valid(path)):
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\rest_uploader\img_process.py", line 26, in pdf_valid
	if open_pdf(filename) is None:
  File "D:\Progs\WPy64-3770\python-3.7.7.amd64\lib\site-packages\rest_uploader\img_process.py", line 17, in open_pdf
	pdfFileObject = open(filename, "rb")
PermissionError: [Errno 13] Permission denied: 'd:\\Temp\\Scans\\ESET problem.pdf'
52

The app is working, yay! Looks like a permissions thing now. You might need to be more permissive on the directory it’s monitoring, or run the program using elevated permissions.

53

Tried running it in a Command Prompt window as administrator and nothing happens, get the prompt immediately upon pressing Enter.

Went back to the normal Command Prompt Window and similarly nothing happens when I type rest_uploader.exe anything. No idea what’s going on or not! rest_uploader.exe isn’t happy.

54

Try monitoring a different directory, or giving full read/write access to Everyone on the directory you wish to monitor.

55

Done both and no change. Typing just rest_uploader.exe with no folderPath should complain? but it doesn’t.

56

How did you install Python? The defaults aren’t great direct from Python, (doesn’t create an environment path var by default, installs just for local user, not all users) and I’ve never tried from the Windows Store.
rest_uploader appears to be working in your WinPython Command Prompt, though, if that’s where you’re getting the PermissionError – it just doesn’t like the permissions on the folder/file you’re dropping in the folder for whatever reason.

57

Really appreciate your help with this. Have to give it a rest for now - wife calling! Will experiment tomorrow

58

Right on - let me know if you get it working - it has really helped my workflow!

59

Not working yet. New attempt to get rest_uploader working

  • d/l Winpython64-3.7.7.0dot.exe - a Portable Windows Python in a 7zip auto-extractor from https://github.com/winpython/winpython/releases/tag/2.3.20200319
    (got to from https://winpython.github.io/)
  • ran it, extracted to D:\Progs, it created it’s root as WPy64-3770
    (there is no ‘install’ as it is a portable application)
  • ran D:\Progs\WPy64-3770\WinPython Command Prompt.exe
  • typed pip install rest_uploader
    this resulted in a d/l of 645 files 9.7 MB added to
    D:\Progs\WPy64-3770\python-3.7.7.amd64\Lib\site-packages
    plus some exe files in D:\Progs\WPy64-3770\python-3.7.7.amd64\Scripts
    one of which is rest_uploader.exe
  • checked Everyone had Full control access of D:\Temp\Scans and one test pdf was present
  • checked PATH includes D:\Progs\Poppler-0.68.0\bin;D:\Progs\Tesseract-OCR
  • ran D:\Progs\WPy64-3770\WinPython Command Prompt.exe
  • typed rest_uploader.exe d:\Temp\Scans --autotag=no
  • started OK, asked for token the first time, cursor sat blinking nothing arrived in inbox notebook

No idea what to do now, any suggestions?

60

It won’t do anything with existing files in the folder, only new ones. Try dropping a new file in the folder while it is running…