Pagination offsets

[ramimoghrabi]

Hello bro,

I was upgrading my project with the latest changes you have done on the api, i can see that you changed how pagination works, now i use pages(count, offset) lets say i specified one page as count, and i want to add another page when i reach the last post only, how can i accomplish this? i tried incrementing pages , but posts will duplicate in this case. Can you help me out on this ? Thanks in advance

[sbertix]

Hey @ramimoghrabi,

Let's assume, for simplicity, you're fetching the first page of followers like this.

let secret: Secret = /* a valid `Secret` */
var bin: Set<AnyCancellable> = /* a **retained** `Set` */

Endpoint.user(secret.identifier)
   .followers
   .unlock(with: secret)
   .session(.instagram)
   .pages(1)
   .catch { _ in Empty() }
   .sink {
      // Do whatever you need to do with your response here…

      guard let nextOffset = $0.offset else { return }
      // If next offset is non-`nil`, you can just cache it or
      // persist it somewhere.
   }
   .store(in: &bin)

Now assuming you saved nextOffset somewhere, when you need to load the second page of followers, you can just do:

let nextOffset: String = /* a valid `String`, obtained like above */
let secret: Secret = /* a valid `Secret` */
var bin: Set<AnyCancellable> = /* a **retained** `Set` */

Endpoint.user(secret.identifier)
   .followers
   .unlock(with: secret)
   .session(.instagram)
   .pages(1, offset: nextOffset)
   .catch { _ in Empty() }
   .sink { _ in }
   .store(in: &bin)

[sbertix]

(didn't notice at first : there is no all after the collectionId in grids)

grids:

https://i.instagram.com/api/v1/feed/collection/17853802247011558/?max_id=QVFCSTV1NkVwVFpINTJjUUNpVG9RUzdVOHVTSmRYek1tZzBfaDVXbjc2OXJnQUR1bnNURmxBQnlfOEFSeDJmR2gtRWhZUThxWTR3SFRvSWctUnNXdl96Mw==

swiftagram:

https://i.instagram.com/api/v1/feed/collection/17907088288931602/all/?show_igtv_first=false&max_id=QVFCeHdMS1hEaF9oLUhRRjdXbkJwb2tpNHozZGVZVnFGd3NoVE1qU09vRkNlSENHWFdhdjZDUThMMFRlUkpsNFpoR2ZDUnJsRHpXVTFWRklJSk1hSHFSRg%3D%3D&include_igtv_preview=true

I am installing a proxy on an Android to check what the native app is doing.

I guess it makes more sense for the all to be what's causing the issue (assuming it's not just the wrong offset being returned by Swiftagram's model). I doubt it's about escaping query items (I don't really know what Grids is but they would still need to escape it to have the request "work", it's probably just applied after the "logging phase", and for sure Insta unescapes it correctly)…

[diligiant]

You're right, that's not the issue (Grids is a MacOS insta client).

I changed .path(appending: "all/") to .path(appending: "/") in Endpoint.Group.Saved.Collection.summary and was able to "page". The returned structure is a little different that current Unit (the elements of self["saveMediaResponse"] are top level.)

When I compare all/ and /, the former returns a lot of information helping the official client to layout its user interface. As such, it seems to me that it should be kept ( all/ => without paging) and another api (/ => with paging) should appear. It that the spirit for you?

[sbertix]

You're right, that's not the issue (Grids is a MacOS insta client).

I changed .path(appending: "all/") to .path(appending: "/") in Endpoint.Group.Saved.Collection.summary and was able to "page". The returned structure is a little different that current Unit (the elements of self["saveMediaResponse"] are top level.)

Great 💪 thx man.

When I compare all/ and /, the former returns a lot of information helping the official client to layout its user interface. As such, it seems to me that it should be kept ( all/ => without paging) and another api (/ => with paging) should appear. It that the spirit for you?

I'm personally not a big fan of duplication.
I'd rather fix /all to have it paginate, especially if / is just legacy and never actually called in the client (mainly for spam detection). Were you checking that too? Is the app actually ever requesting the non-/all endpoint?

[diligiant]

I might have badly explained:

all/ (your current one) returns in saveMediaResponse what / returns top-level. / is the one (new) we need to be able to page (therefore all/ returning me the first elements indefinitely ;)

BUT all/ is not "deprecated". It just returns more elements that are mainly of use by people willing to emulate the Instagram UI. I guess the official application first calls all/ and then subsequently pages with /.

If it were me (but it's not), I'd keep the existing all/ and create a new / call (The most difficult part is coming with good names…)

[sbertix]

Sorry, I think I messed up with submitting the replying cause I don't see it anymore.
What I meant, as you mentioned inspecting the Android app's requests, was wether calling / and /all was a behaviour existing in the app or not. Cause I'm sure /all was called when I implemented it (as I got it from there), so I was wondering if perhaps / was a deprecated endpoint, never called in the client, meaning we shouldn't implement it in Swiftagram and that there's probably a way to make /all work with offsets instead.

I guess the official application first calls all/ and then subsequently pages with /.

So if it's just an educated guess I'll try and actually check the requests. No promises on when though.

If it were me (but it's not), I'd keep the existing all/ and create a new / call (The most difficult part is coming with good names…)

I'd rather mimic the design of the timeline Endpoint, as I don't expect users to actually value them separately: calling /all with a nil offset and / in any other case, even on "automatic" pagination.

[ramimoghrabi]

one more thing i wanted to know about timeline, that
Endpoint.posts.recent
.unlock(with: secret!)
.session(.instagram)
.pages(1)
.sink(receiveCompletion: {print ("completion: ($0)")}, receiveValue: { (wrap) in
guard let nextOffset1 = wrap.offset else { return } ---> Initializer for conditional binding must have Optional type, not 'Wrapper'
})

does not work as described above ? does it have another approach ?

[sbertix]

The timeline endpoint works a bit differently cause it encompasses many exceptions.

You can check offset computation directly from its definition in code, but I'll just add it here for you.

let output: Wrapper = /* a valid `Wrapper` you get as `Endpoint.recent.posts` output */

let nextOffset: String?
switch output.nextMaxId.string(converting: true) {
case .none:
    nextOffset = nil
case "feed_recs_head_load":
    nextOffset = output.feedItems
                .array()?
                .last?
                .endOfFeedDemarcator
                .groupSet
                .groups
                .array()?
                .first(where: { $0.id.string(converting: true) == "past_posts" })?
                .nextMaxId
                .string()
case let cursor?:
    nextOffset = cursor
}

[ramimoghrabi]

again i wanna thank you for giving me part of your valuable time.

[ramimoghrabi]

i think there might be a bug , in comments offset, setting the offset on first fetch is always nil, thus it won't load since i used the above return method if not found : guard let nextOffset = $0.offset else { return }

[sbertix]

Unfortunately I can't check it tonight. I'm definitely gonna take a look at it and fix it, if it is a bug, tomorrow 💪

[ramimoghrabi]

sure no problem, thanks

[ramimoghrabi]

Endpoint.users(matching: query)
.unlock(with: secret)
.session(.instagram)
.pages(1)
.sink(receiveCompletion: {_ in }, receiveValue: { users in
guard let nextOffset1 = users.offset else { return }
})

same issue here offset is nil. i am just testing and mentioning them

[sbertix]

Sorry I didn't manage to come back to this: busy week.
I'm planning on checking this asap and definitely fix it, if needs to, by the end of the week at the latest.
Apologies @ramimoghrabi

[ramimoghrabi]

no problem brother, just let me know when it is fixed so that i can test it again.

[ramimoghrabi]

hello again @sbertix ,i was wondering if you had time to check the comments pagination, as the wrapper.offset is always nil.