# Pre-release 1.4 is now available for testing (Updated 23/11/20)

**URL:** <https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247>\
**Category:** Beta Testing\
**Created:** [5 November 2020 22:46 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247 "2020-11-05T22:46:39Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [8 November 2020 13:46 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/21 "2020-11-08T13:46:23Z")

</div>

There is a simple conversion method, I believe python should be able to do similar things

> <https://github.com/rxliuli/joplin-api/blob/e146e3098de8302d4216840eb9aa8fc26bc95ea6/src/util/PageUtil.ts#L20>

Usage example

> <https://github.com/rxliuli/joplin-api/blob/e146e3098de8302d4216840eb9aa8fc26bc95ea6/test/util/PageUtil.test.ts#L4>

> Although the pagination added by v1.4.\* is a bit sudden, I don’t think there is any problem with the acquisition of large amounts of data (though about `/notes/${id}/tags` and `/notes/${id}/resources` I still don’t agree with the use of paging for the two APIs...)

---

<div class="post-metadata">

**Author:** ![foxmask](https://avatars.discourse-cdn.com/v4/letter/f/c68b51/32.png) [@foxmask](https://discourse.joplinapp.org/u/foxmask)\
**Post date:** [8 November 2020 13:49 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/22 "2020-11-08T13:49:12Z")

</div>

may be 😉 but I don't understand anything of TS.

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [8 November 2020 13:52 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/23 "2020-11-08T13:52:09Z")

</div>

To put it simply, this is a high-order function. You need to pass an original paging query function and the required query parameters, and then call the passed function in the function loop until there is no next page (here, `cousor` is When empty), return the list data accumulated in the process

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [8 November 2020 15:10 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/24 "2020-11-08T15:10:54Z")

</div>

> [@foxmask](#):
>
> can't we bypass the pagination ?  
> As I already deal with that, it's annoying ☹

I can understand this is a bit annoying, I also don't like having to update perfectly working apps just because an API changed, but in this case it's hard to avoid.

If I allow retrieving everything in one go, people will test their plugins with their own notes, maybe a few hundred of them, not realising that some users have 100,000 notes or more. No only their plugin is likely to break but it might also freeze or crash the main app. So it's best to develop external apps or plugins with pagination in mind.

rxliuli method would work and I've also put a pseudo algorithm there to retrieve everything if that can help: [https://joplinapp.org/api/references/rest\_api/#pagination](https://joplinapp.org/api/references/rest_api/#pagination)

---

<div class="post-metadata">

**Author:** ![foxmask](https://avatars.discourse-cdn.com/v4/letter/f/c68b51/32.png) [@foxmask](https://discourse.joplinapp.org/u/foxmask)\
**Post date:** [8 November 2020 16:58 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/25 "2020-11-08T16:58:22Z")

</div>

I understand, but how do you know on which page you are ? if you just know the current cursor, how do you know the previous / next one ?

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [8 November 2020 17:50 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/26 "2020-11-08T17:50:22Z")

</div>

You can't, so for now you could probably just retrieve everything and do the pagination yourself like you were previously doing.

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [8 November 2020 23:46 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/27 "2020-11-08T23:46:46Z")

</div>

Using the design of `cousor` does not support the scene of obtaining the specified page, it is most suitable for the scene of infinite scrolling

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [9 November 2020 14:52 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/28 "2020-11-09T14:52:02Z")

</div>

This `cousor` looks very strange now, it not only contains the last one for comparison, it even contains query parameters, making the set `fields` field invalid @laurent

Reproducible example:

> <https://github.com/rxliuli/joplin-api/blob/09d92d7eb4c8dcae0f1df685fd58f288a2b80259/test/api/NoteApi.test.ts#L89>

> Note: If the performance cannot be improved through concurrent requests, I can't think of how to quickly get the list of all notes on the client. . .(I have a scenario where you need to get all the note data to display some kind of chart, such as note relationship diagram)

[https://rxliuli.com/joplin-charts/](https://rxliuli.com/joplin-charts/)

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [9 November 2020 15:15 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/29 "2020-11-09T15:15:20Z")

</div>

I don't understand the issue. Do you have an example query that fails?

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [9 November 2020 15:21 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/30 "2020-11-09T15:21:07Z")

</div>

sample graph

 ![image](https://canada1.discourse-cdn.com/flex028/uploads/cozic/original/2X/4/43213d6a539d4aa85d722b1d92eb37e0a720d679.png)

> Note that you only need to pass the cursor to the next request, as it will continue the fetching process using the same parameters you initially provided.

I think it is unreasonable for `cousor` to include query parameters, especially query parameters that do not affect the number of returned results. Generally speaking, it should be the last id of the current page, right? Why not?

This is an example of query failure, when you run this unit test, it will fail(The number of notes must be at least 101)

> [@Pre-release 1.4 is now available for testing (Updated 23/11/20)](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing/12247/28):
>
> This cousor looks very strange now, it not only contains the last one for comparison, it even contains query parameters, making the set fields field invalid @laurent Reproducible example: [https://github.com/rxliuli/joplin-api/blob/09d92d7eb4c8dcae0f1df685fd58f288a2b80259/test/api/NoteApi.test.ts#L89](https://github.com/rxliuli/joplin-api/blob/09d92d7eb4c8dcae0f1df685fd58f288a2b80259/test/api/NoteApi.test.ts#L89) Note: If the performance cannot be improved through concurrent requests, I can't think of how to quickly get the list of all notes on the client. . .(I have a scenario where you need to get all t…

You can modify the token here for testing: [joplin-api/test/util/setupTestEnv.ts at dc44a50f4d222c27789dfcdec2ab66ebd07150ae · rxliuli/joplin-api · GitHub](https://github.com/rxliuli/joplin-api/blob/dc44a50f4d222c27789dfcdec2ab66ebd07150ae/test/util/setupTestEnv.ts#L6)

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [9 November 2020 15:29 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/31 "2020-11-09T15:29:48Z")

</div>

Do you mean that it adds extra parameters like updated\_time and so on?

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [9 November 2020 15:31 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/32 "2020-11-09T15:31:33Z")

</div>

No, I mean the title and parent\_id fields are missing from the results

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [9 November 2020 15:38 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/33 "2020-11-09T15:38:51Z")

</div>

In general, I hope to improve the efficiency of obtaining data through concurrent calls, so I made an attempt, but the current API cannot support this. I believe @foxmask also likes to have simple ways to improve the performance of obtaining full data

> <https://github.com/rxliuli/joplin-api/blob/09d92d7eb4c8dcae0f1df685fd58f288a2b80259/src/util/PageUtil.ts#L80>

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [9 November 2020 15:44 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/34 "2020-11-09T15:44:24Z")

</div>

> [@rxliuli](#):
>
> No, I mean the title and parent\_id fields are missing from the results

Ok there's probably a bug there then. What API end point is it? just `/notes`?

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [9 November 2020 15:46 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/35 "2020-11-09T15:46:15Z")

</div>

I'm not sure, but I guess all paging queries have this problem. The core reason is that `cousor` contains the query parameter `fields/limit/more...`

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [9 November 2020 16:29 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/36 "2020-11-09T16:29:25Z")

</div>

No it should return the fields you've requested in the initial query. Any field you specify when you do a subsequent cursor query will be ignored, as documented:

> Note that you only need to pass the cursor to the next request, as it will continue the fetching process using the same parameters you initially provided.

[https://joplinapp.org/api/references/rest\_api/#pagination](https://joplinapp.org/api/references/rest_api/#pagination)

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [9 November 2020 17:23 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/37 "2020-11-09T17:23:24Z")

</div>

I can't replicate this field issue although I've added [a test](https://github.com/laurent22/joplin/commit/7cb60aaaced511abb3b3d4676b0f32092f5b5279#diff-5b75aa92c06a97ab21aac744f4631ac0fb4da33441387e554bafa05b3f5bcc6cR601) for good mesure. If there's an issue, I'll need curl calls or something I can use to replicate without installing a whole application.

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [9 November 2020 23:20 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/38 "2020-11-09T23:20:14Z")

</div>

> [@laurent](#):
>
> No it should return the fields you've requested in the initial query. Any field you specify when you do a subsequent cursor query will be ignored, as documented:

This is the problem, I think this is a wrong design

The cursor of pre query ad-hoc is simply a clever approach. . .

---

<div class="post-metadata">

**Author:** ![laurent](https://avatars.discourse-cdn.com/v4/letter/l/ce7236/32.png) [@laurent](https://discourse.joplinapp.org/u/laurent)\
**Post date:** [10 November 2020 00:29 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/39 "2020-11-10T00:29:47Z")

</div>

This is the right approach to iterate over a feed, and there’s info on the doc on how to retrieve all the data like before. Field param is also working so at this point I still don’t know what the issue is.

Best would be to provide a minimal reproducible example if you think there’s a problem: [https://stackoverflow.com/help/minimal-reproducible-example](https://stackoverflow.com/help/minimal-reproducible-example)

---

<div class="post-metadata">

**Author:** ![rxliuli](https://yyz2.discourse-cdn.com/flex028/user_avatar/discourse.joplinapp.org/rxliuli/32/4083_2.png) [@rxliuli](https://discourse.joplinapp.org/u/rxliuli)\
**Post date:** [10 November 2020 00:42 UTC](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247/40 "2020-11-10T00:42:18Z")

</div>

In short, I plan to get all the `cousor` lists first, and then concurrently (currently 10) to get the real data

```auto
import { PageParam, PageRes } from '../modal/PageData'
import { asyncLimiting } from './asyncLimiting'
import { AsyncArray } from './AsyncArray'

type PageResValueType<T extends Promise<PageRes<any>>> = T extends Promise<
  PageRes<infer U>
>
  ? U
  : never

export class PageUtil {
  /**
   * Maximum number of pages
   * @private
   */
  private static readonly MaxLimit = 100

  /**
   * Retrieve all paged data in a loop (concurrent, currently set the concurrent number to 10)
   * Suitable for all-at-a-time acquisition of large amounts of data, for example, all notes need to be acquired to display a certain chart
   * Get the maximum number of pages each time to minimize the number of requests
   */
  static async pageToAllListForParallel<
    F extends (
      pageParam: PageParam<any> & Record<string, any>,
    ) => Promise<PageRes<any>>
  >(
    fn: F,
    pageParam?: Omit<Parameters<F>[0], 'cursor' | 'limit'>,
    options?: { limit: number },
  ): Promise<PageResValueType<ReturnType<F>>[]> {
    let cursor: string | undefined
    const cursorList: (string | undefined)[] = []
    do {
      // noinspection JSUnusedAssignment
      const res = await fn({
        ...pageParam,
        fields: ['id'],
        cursor,
        limit: this.MaxLimit,
      })
      cursor = res.cursor
      if (res.cursor) {
        cursorList.push(res.cursor!)
      }
    } while (cursor)
    console.log('cursorList: ', cursorList)
    cursorList.unshift(undefined)
    const callback = asyncLimiting(async (cursor: string | undefined) => {
      const res = await fn({
        ...pageParam,
        cursor,
        limit: this.MaxLimit,
      })
      return res.items
    }, options?.limit || 10)
    return new AsyncArray(cursorList).parallel().flatMap(callback)
  }
}

    it('test and get all notes concurrently', async () => {
      const res = await PageUtil.pageToAllListForParallel(noteApi.list, {
        fields: ['id', 'title', 'parent_id'],
      })
      console.log('first page and second page: ', res[0], res[100])
    })

```

Why not try to run this unit test?

> <https://github.com/rxliuli/joplin-api/blob/8429d266fd7b64336f39906802506a36eea78941/test/api/NoteApi.test.ts#L102>

[Previous page](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247.md?page=1)

[Next page](https://discourse.joplinapp.org/t/pre-release-1-4-is-now-available-for-testing-updated-23-11-20/12247.md?page=3)