There is a simple conversion method, I believe python should be able to do similar things
Usage example
Although the pagination added by v1.4.* is a bit sudden, I don’t think there is any problem with the acquisition of large amounts of data (though about
/notes/${id}/tagsand/notes/${id}/resourcesI still don’t agree with the use of paging for the two APIs...)
may be 
but I don't understand anything of TS.
To put it simply, this is a high-order function. You need to pass an original paging query function and the required query parameters, and then call the passed function in the function loop until there is no next page (here, cousor is When empty), return the list data accumulated in the process
I can understand this is a bit annoying, I also don't like having to update perfectly working apps just because an API changed, but in this case it's hard to avoid.
If I allow retrieving everything in one go, people will test their plugins with their own notes, maybe a few hundred of them, not realising that some users have 100,000 notes or more. No only their plugin is likely to break but it might also freeze or crash the main app. So it's best to develop external apps or plugins with pagination in mind.
rxliuli method would work and I've also put a pseudo algorithm there to retrieve everything if that can help: https://joplinapp.org/api/references/rest_api/#pagination
I understand, but how do you know on which page you are ? if you just know the current cursor, how do you know the previous / next one ?
You can't, so for now you could probably just retrieve everything and do the pagination yourself like you were previously doing.
Using the design of cousor does not support the scene of obtaining the specified page, it is most suitable for the scene of infinite scrolling
This cousor looks very strange now, it not only contains the last one for comparison, it even contains query parameters, making the set fields field invalid @laurent
Reproducible example:
Note: If the performance cannot be improved through concurrent requests, I can't think of how to quickly get the list of all notes on the client. . .(I have a scenario where you need to get all the note data to display some kind of chart, such as note relationship diagram)
I don't understand the issue. Do you have an example query that fails?
sample graph
Note that you only need to pass the cursor to the next request, as it will continue the fetching process using the same parameters you initially provided.
I think it is unreasonable for cousor to include query parameters, especially query parameters that do not affect the number of returned results. Generally speaking, it should be the last id of the current page, right? Why not?
This is an example of query failure, when you run this unit test, it will fail(The number of notes must be at least 101)
You can modify the token here for testing: joplin-api/test/util/setupTestEnv.ts at dc44a50f4d222c27789dfcdec2ab66ebd07150ae · rxliuli/joplin-api · GitHub
Do you mean that it adds extra parameters like updated_time and so on?
No, I mean the title and parent_id fields are missing from the results
In general, I hope to improve the efficiency of obtaining data through concurrent calls, so I made an attempt, but the current API cannot support this. I believe @foxmask also likes to have simple ways to improve the performance of obtaining full data
Ok there's probably a bug there then. What API end point is it? just /notes?
I'm not sure, but I guess all paging queries have this problem. The core reason is that cousor contains the query parameter fields/limit/more...
No it should return the fields you've requested in the initial query. Any field you specify when you do a subsequent cursor query will be ignored, as documented:
Note that you only need to pass the cursor to the next request, as it will continue the fetching process using the same parameters you initially provided.
I can't replicate this field issue although I've added a test for good mesure. If there's an issue, I'll need curl calls or something I can use to replicate without installing a whole application.
This is the problem, I think this is a wrong design
The cursor of pre query ad-hoc is simply a clever approach. . .
This is the right approach to iterate over a feed, and there’s info on the doc on how to retrieve all the data like before. Field param is also working so at this point I still don’t know what the issue is.
Best would be to provide a minimal reproducible example if you think there’s a problem: https://stackoverflow.com/help/minimal-reproducible-example
In short, I plan to get all the cousor lists first, and then concurrently (currently 10) to get the real data
import { PageParam, PageRes } from '../modal/PageData'
import { asyncLimiting } from './asyncLimiting'
import { AsyncArray } from './AsyncArray'
type PageResValueType<T extends Promise<PageRes<any>>> = T extends Promise<
PageRes<infer U>
>
? U
: never
export class PageUtil {
/**
* Maximum number of pages
* @private
*/
private static readonly MaxLimit = 100
/**
* Retrieve all paged data in a loop (concurrent, currently set the concurrent number to 10)
* Suitable for all-at-a-time acquisition of large amounts of data, for example, all notes need to be acquired to display a certain chart
* Get the maximum number of pages each time to minimize the number of requests
*/
static async pageToAllListForParallel<
F extends (
pageParam: PageParam<any> & Record<string, any>,
) => Promise<PageRes<any>>
>(
fn: F,
pageParam?: Omit<Parameters<F>[0], 'cursor' | 'limit'>,
options?: { limit: number },
): Promise<PageResValueType<ReturnType<F>>[]> {
let cursor: string | undefined
const cursorList: (string | undefined)[] = []
do {
// noinspection JSUnusedAssignment
const res = await fn({
...pageParam,
fields: ['id'],
cursor,
limit: this.MaxLimit,
})
cursor = res.cursor
if (res.cursor) {
cursorList.push(res.cursor!)
}
} while (cursor)
console.log('cursorList: ', cursorList)
cursorList.unshift(undefined)
const callback = asyncLimiting(async (cursor: string | undefined) => {
const res = await fn({
...pageParam,
cursor,
limit: this.MaxLimit,
})
return res.items
}, options?.limit || 10)
return new AsyncArray(cursorList).parallel().flatMap(callback)
}
}
it('test and get all notes concurrently', async () => {
const res = await PageUtil.pageToAllListForParallel(noteApi.list, {
fields: ['id', 'title', 'parent_id'],
})
console.log('first page and second page: ', res[0], res[100])
})
Why not try to run this unit test?