integration_ShowAllNotes test failing randomly

This is strange. The error says:

Suite error: services_SearchEngine

but there are no tests in that suite that match the filter.

Thanks for your patience @naviji.

At this point, I am suspecting the problem is two. The first as identified by Laurent above, which results in un-initialised errors.

The second being that your computer is slower or faster, to write the database than expected. So the test scripts may not be giving enough time for the app to complete the actions.

I have updated the patch to address both these issues. If you would be so kind to apply this patch and try again, I would be grateful. This patch should be applied without the earlier patch.

test-harness-fixes-2.patch (20.2 KB)

I am unable to reproduce the problem so your assistance has been very helpful.

2 Likes

Thank you too for taking time to help me out :slight_smile:

I ran npm run test with and without the patch.

It seems the Unhandled promise rejection: Error: Accessing database before it has been initialised went away though tests are still failing.

@naviji this is promising. I was waiting to see how the patch went before I fix the other integration tests, so I expect them to fail at the moment.

Before I move on to them, could you run just the integration_ShowAllNotes tests a few times as you were doing earlier and confirm if you think they are good now?

https://pastebin.com/2pZZvsCy
I ran it for 50 times. It was looking good but it failed on the 47th run.

So I ran it again for 50 times. Now it didn’t fail.

Update: ran it again 100 times.
Failed on the 80th run

Randomized with seed 57118 (jasmine --random=true --seed=57118)
Test number 80

> joplin@1.0.161 test /home/naviji/Projects/RoughWork/joplin/CliClient
> gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=integration_ShowAllNotes"

Testing with sync target: memory
Randomized with seed 98887
Started
F.

Failures:
1) integration_ShowAllNotes should show all notes
  Message:
    Expected $.length = 0 to equal 9.
    Expected $[0] = undefined to equal '087317389c0f42b4acf6a57dffe3c7ea'.
    Expected $[1] = undefined to equal '127a0ca73c114f57a66801267e7b269a'.
    Expected $[2] = undefined to equal '3c787692ec644f55b91f5649b9b561b1'.
    Expected $[3] = undefined to equal '9c80cc341b144b058aa3c1f608c2273a'.
    Expected $[4] = undefined to equal '9f7b33de018a48faa6e8e6eacc1397c4'.
    Expected $[5] = undefined to equal 'd801e0930b7f4041ad7c40c6770979af'.
    Expected $[6] = undefined to equal 'edf6565b4a254d878c44b1bfa6a28308'.
    Expected $[7] = undefined to equal 'fb1036788e424e4fa4c22c91bfd5888d'.
    Expected $[8] = undefined to equal 'fde98bcd59ee4e6ca927ef3b4a85235d'.
  Stack:
    Error: Expected $.length = 0 to equal 9.
    Expected $[0] = undefined to equal '087317389c0f42b4acf6a57dffe3c7ea'.
    Expected $[1] = undefined to equal '127a0ca73c114f57a66801267e7b269a'.
    Expected $[2] = undefined to equal '3c787692ec644f55b91f5649b9b561b1'.
    Expected $[3] = undefined to equal '9c80cc341b144b058aa3c1f608c2273a'.
    Expected $[4] = undefined to equal '9f7b33de018a48faa6e8e6eacc1397c4'.
    Expected $[5] = undefined to equal 'd801e0930b7f4041ad7c40c6770979af'.
    Expected $[6] = undefined to equal 'edf6565b4a254d878c44b1bfa6a28308'.
    Expected $[7] = undefined to equal 'fb1036788e424e4fa4c22c91bfd5888d'.
    Expected $[8] = undefined to equal 'fde98bcd59ee4e6ca927ef3b4a85235d'.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/integration_ShowAllNotes.js:63:34
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

Ran 2 of 240 specs
2 specs, 1 failure
Finished in 17.656 seconds
Randomized with seed 98887 (jasmine --random=true --seed=98887)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! joplin@1.0.161 test: `gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=integration_ShowAllNotes"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the joplin@1.0.161 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/naviji/.npm/_logs/2020-03-10T12_41_24_581Z-debug.log
2 Likes

@laurent

The integration tests use sleeps to give time for async actions to complete before checking the response but although we gave generous durations, it is not reliable for all machines. I am looking if there is a way we can confirm the actions are complete before continuing in the test case.

It may be the problem is not only in integration_ tests. It has been reported twice that services_Revisions tests fail (above and here). On first glance, they also seem to use sleeps to give background processing time to complete.

Anyway, I’ll look further into it, just posting in case you had any suggestions.

@naviji Thanks

BTW what platform are you running this on? WIndows?

Ubuntu 18.04.4 LTS

1 Like

Here is a new patch to try. This patch should better stop the random failures in the integration_ShowAllNotes and integration_TagList tests because it waits for processing to complete before continuing the test, rather than just waiting for a fixed amount of time.

test-harness-fixes-3.patch (28.2 KB)

It also helps with “Database is not initialized” errors, but I suspect they will still occur (rarely).

With the Revision test errors, I’m not familiar with those tests. Could you please confirm if they fail when run alone:

npm run test -- --filter="services_Revision"

Thanks

naviji@naviji-HP-Notebook:~/Projects/RoughWork/joplin/CliClient$ npm test

> joplin@1.0.161 test /home/naviji/Projects/RoughWork/joplin/CliClient
> gulp buildTests -L && jasmine --config=tests/support/jasmine.json

Testing with sync target: memory
Randomized with seed 19683
Started
...................................................................................................................F......F.....................................................................................................................

Failures:
1) services_Revision should not delete old revisions if one of them is still encrypted (3)
  Message:
    Expected 0 to be 1.
  Stack:
    Error: Expected 0 to be 1.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:371:41
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:97:5)

2) services_Revision should delete old revisions (1 note, 3 rev)
  Message:
    Expected 1 to be 2.
  Stack:
    Error: Expected 1 to be 2.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:95:29
        at <Jasmine>
        at processTicksAndRejections (internal/process/task_queues.js:97:5)

240 specs, 2 failures
Finished in 527.766 seconds
Randomized with seed 19683 (jasmine --random=true --seed=19683)
npm ERR! Test failed.  See above for more details.



naviji@naviji-HP-Notebook:~/Projects/RoughWork/joplin/CliClient$ npm run test -- --filter="services_Revision"

> joplin@1.0.161 test /home/naviji/Projects/RoughWork/joplin/CliClient
> gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"

Testing with sync target: memory
Randomized with seed 45079
Started
...F..........F.

Failures:
1) services_Revision should not delete old revisions if one of them is still encrypted (3)
  Message:
    Expected 0 to be 1.
  Stack:
    Error: Expected 0 to be 1.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:371:41
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

2) services_Revision should delete old revisions (1 note, 3 rev)
  Message:
    Expected 1 to be 2.
  Stack:
    Error: Expected 1 to be 2.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:95:29
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

Ran 16 of 240 specs
16 specs, 2 failures
Finished in 37.058 seconds
Randomized with seed 45079 (jasmine --random=true --seed=45079)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! joplin@1.0.161 test: `gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the joplin@1.0.161 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/naviji/.npm/_logs/2020-03-11T09_11_53_827Z-debug.log
1 Like

Ok, so the status here is:

  • the patch may have resolved the integration test issues for you (based on results from 101 runs on your machine)
  • the Revision tests fail consistently for you, even when run stand alone

Edit: updated result

I also ran

npm run test -- --filter="integration_ShowAllNotes"

100 times.

It didn't fail :+1: So I guess that's fixed.

1 Like

@naviji I think I’ve identified the problem with Revision tests. I’ll post another patch for you to test later today.

This patch adds a fix for issues found in the services_Revision tests.

The tests always passed for me even with the issue, so I don’t know if it fixes the problem you saw, but I think so.

test-harness-fixes-4.patch (43.8 KB)

It’s working. No failures yet.

Thanks @naviji that’s a good start. When you are happy, please let me know so I can submit a PR for the test fixes and get it into the code base.

Update: It failed when I ran it for 50 times.
Here are the logs.

Test 9

> joplin@1.0.161 test /home/naviji/Projects/RoughWork/joplin/CliClient
> gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"

Testing with sync target: memory
Randomized with seed 79575
Started
...........F....

Failures:
1) services_Revision should delete old revisions (1 note, 2 rev)
  Message:
    Expected 0 to be 1.
  Stack:
    Error: Expected 0 to be 1.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:75:28
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

Ran 16 of 241 specs
16 specs, 1 failure
Finished in 47.892 seconds
Randomized with seed 79575 (jasmine --random=true --seed=79575)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! joplin@1.0.161 test: `gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the joplin@1.0.161 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/naviji/.npm/_logs/2020-03-12T21_26_42_522Z-debug.log
Test 10

> joplin@1.0.161 test /home/naviji/Projects/RoughWork/joplin/CliClient
> gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"

Testing with sync target: memory
Randomized with seed 99366
Started
....F.....F.....

Failures:
1) services_Revision should delete old revisions (1 note, 2 rev)
  Message:
    Expected 0 to be 1.
  Stack:
    Error: Expected 0 to be 1.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:75:28
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

2) services_Revision should not create a revision if there is already a recent one
  Message:
    Expected 3 to be 2.
  Stack:
    Error: Expected 3 to be 2.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:431:41
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

Ran 16 of 241 specs
16 specs, 2 failures
Finished in 68.711 seconds
Randomized with seed 99366 (jasmine --random=true --seed=99366)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! joplin@1.0.161 test: `gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the joplin@1.0.161 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/naviji/.npm/_logs/2020-03-12T21_27_53_116Z-debug.log

Test 12

> joplin@1.0.161 test /home/naviji/Projects/RoughWork/joplin/CliClient
> gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"

Testing with sync target: memory
Randomized with seed 47013
Started
.......F........

Failures:
1) services_Revision should not create a revision if there is already a recent one
  Message:
    Expected 3 to be 2.
  Stack:
    Error: Expected 3 to be 2.
        at <Jasmine>
        at /home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/services_Revision.js:431:41
        at processTicksAndRejections (internal/process/task_queues.js:97:5)
        at async UserContext.<anonymous> (/home/naviji/Projects/RoughWork/joplin/CliClient/tests-build/test-utils.js:373:4)

Ran 16 of 241 specs
16 specs, 1 failure
Finished in 75.171 seconds
Randomized with seed 47013 (jasmine --random=true --seed=47013)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! joplin@1.0.161 test: `gulp buildTests -L && jasmine --config=tests/support/jasmine.json "--filter=services_Revision"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the joplin@1.0.161 test script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/naviji/.npm/_logs/2020-03-12T21_30_18_036Z-debug.log


Very sorry, for some reason I overlooked that test case. In the code I can see it has exactly the same problem as the others in that suite. I will fix it (and look for any others I might have missed) but won’t be until tonight.

Reattempt to fix services_Revision tests:

test-harness-fixes-6.patch (51.3 KB)

I think I remember seeing a models_Revision test fail too, but I can’t find it above. So I reviewed those tests too, but couldn’t find any clock dependencies, so left them as is.

If you don’t mind saying, what CPU are you running these tests on?

just a side note, you guys doing an excellent job here solving this issue!

1 Like