Pagination not doing anything

[MattWilcox]

Description

I have a feed from Sage API, which contains pagination.

The feed itself is set up correctly in FeedMe and imports data into Entries just great.
The correct node is set for the Pagination parameter.

Processing the feed, it does not paginate. It imports only the first batch.

This is the data in the Queue Job where we can see "paginationNode": "$next", and Sage's node for the next page is $next (yes, they use a dollar symbol for their keys).

{
    "description": null,
    "feed": {
        "id": 1,
        "name": "Protected Parks - Sage API",
        "feedUrl": "http://fit.loriacrm.com/sdata/crmj/sagecrm2/-/Site",
        "feedType": "json",
        "primaryElement": "$resources",
        "elementType": "craft\\elements\\Entry",
        "elementGroup": {
            "craft\\elements\\Asset": "",
            "craft\\elements\\Category": "",
            "craft\\elements\\Entry": {
                "section": "13",
                "entryType": "25"
            },
            "craft\\elements\\Tag": "",
            "craft\\elements\\GlobalSet": {
                "globalSet": ""
            },
            "verbb\\formie\\elements\\Submission": "",
            "verbb\\navigation\\elements\\Node": ""
        },
        "siteId": 2,
        "sortOrder": 1,
        "singleton": false,
        "duplicateHandle": [
            "add",
            "update",
            "delete"
        ],
        "updateSearchIndexes": false,
        "paginationNode": "$next",
        "fieldMapping": {
            "title": {
                "attribute": "true",
                "node": "$title",
                "default": "Default Import Title"
            },
            "slug": {
                "attribute": "true",
                "node": "noimport",
                "default": ""
            },
            "parent": {
                "attribute": "true",
                "node": "noimport",
                "options": {
                    "match": "title"
                },
                "default": ""
            },
            "postDate": {
                "attribute": "true",
                "node": "noimport",
                "options": {
                    "match": "auto"
                },
                "default": {
                    "date": "",
                    "locale": "en-GB",
                    "timezone": "Europe/London",
                    "time": ""
                }
            },
            "expiryDate": {
                "attribute": "true",
                "node": "noimport",
                "options": {
                    "match": "auto"
                },
                "default": {
                    "date": "",
                    "locale": "en-GB",
                    "timezone": "Europe/London",
                    "time": ""
                }
            },
            "enabled": {
                "attribute": "true",
                "node": "noimport",
                "default": ""
            },
            "authorId": {
                "attribute": "true",
                "node": "noimport",
                "options": {
                    "match": "email"
                },
                "default": [
                    "1573"
                ]
            },
            "id": {
                "attribute": "true",
                "node": "noimport",
                "default": ""
            },
            "sageSiteId": {
                "field": "craft\\fields\\PlainText",
                "node": "site_SiteID",
                "default": ""
            },
            "protectedSince": {
                "field": "craft\\fields\\PlainText",
                "node": "site_date_deed_signed",
                "default": ""
            },
            "trivialId": {
                "field": "craft\\fields\\PlainText",
                "node": "noimport",
                "default": ""
            },
            "googleMap": {
                "field": "doublesecretagency\\googlemaps\\fields\\AddressField",
                "fields": {
                    "name": {
                        "node": "noimport",
                        "default": ""
                    },
                    "street1": {
                        "node": "noimport",
                        "default": ""
                    },
                    "street2": {
                        "node": "noimport",
                        "default": ""
                    },
                    "city": {
                        "node": "noimport",
                        "default": ""
                    },
                    "state": {
                        "node": "noimport",
                        "default": ""
                    },
                    "zip": {
                        "node": "noimport",
                        "default": ""
                    },
                    "county": {
                        "node": "noimport",
                        "default": ""
                    },
                    "country": {
                        "node": "noimport",
                        "default": ""
                    },
                    "placeId": {
                        "node": "noimport",
                        "default": ""
                    },
                    "lat": {
                        "node": "vca_lat",
                        "default": ""
                    },
                    "lng": {
                        "node": "vca_lng",
                        "default": ""
                    },
                    "zoom": {
                        "node": "noimport",
                        "default": ""
                    }
                }
            }
        },
        "fieldUnique": {
            "title": "1",
            "slug": "",
            "parent": "",
            "enabled": "",
            "id": "",
            "sageSiteId": "1",
            "protectedSince": "",
            "trivialId": ""
        },
        "passkey": "rupynxcxbh",
        "backup": false,
        "setEmptyValues": 0,
        "dateCreated": "2023-12-05T13:42:04+00:00",
        "dateUpdated": "2023-12-19T18:46:56+00:00",
        "uid": "86420f8b-7b28-4630-a28a-8193edbaecb9",
        "debug": null,
        "paginationUrl": null
    },
    "limit": null,
    "offset": null,
    "processedElementIds": [],
    "continueOnError": true
}

If I do the same query using Consume rather than FeedMe you can see what sort of data we get back from their API:

Note the incorrect domain. That's Sage and we can't do anything about that, so that's actually being "fixed" via a module as follows, by listening to the After Fetch Feed and substituting the proper domain in:

Event::on(
	DataTypes::class,
	DataTypes::EVENT_AFTER_FETCH_FEED,
	function(FeedDataEvent $event) {
		$theRawResponse  = $event->response["data"];
		$replaceWithThis = str_replace("http://fit-live", "http://fit.loriacrm.com", $theRawResponse);
		$event->response["data"] = $replaceWithThis;
	}
);

And does seem to be working (you can see the replaced domain):

I can not work out why pagination is not triggering.

Additional info

• Craft version: 4.5.13
• PHP version: 8.1.26
• FeedMe: 5.3.0

[MattWilcox]

Update: we have managed to get the client to correct the main URL on Sage's end, so we are now getting "real" URLs returned, and have thrown away our edit in out Module.

It still does not paginate, and the logs don't show any attempt to paginate.

{"date":"2024-01-24 09:27:32","type":"info","message":"Protected Parks - Sage API: Preparing for feed processing."}
{"date":"2024-01-24 09:27:32","type":"info","message":"Protected Parks - Sage API: Finished preparing for feed processing."}
{"date":"2024-01-24 09:27:32","type":"info","message":"Protected Parks - Sage API: Starting processing of node `#1`.","key":"vyjozztouhyyxyfbuxea"}
{"date":"2024-01-24 09:27:32","type":"info","message":"Protected Parks - Sage API: Match existing element with data `{\"title\":\"Battersea Fields Millgrove Street Open Space\",\"sageSiteId\":\"5001\"}`.","key":"vyjozztouhyyxyfbuxea"}
...SKIPPED 10 IMPORTS...
{"date":"2024-01-24 09:27:32","type":"info","message":"Protected Parks - Sage API: The following elements have been deleted: {\"10\":244229,\"11\":244230,\"12\":244231,\"13\":244232,\"14\":244233,\"15\":244234,\"16\":244235,\"17\":244236,\"18\":244237,\"19\":244238}.","key":"ljoqblqqgeefcwwglcaw"}
{"date":"2024-01-24 09:27:32","type":"info","message":"Protected Parks - Sage API: Processing 10 elements finished in 0.38s"}

[MattWilcox]

Digging with my basic knowledge of the debugger in PHP, it looks as though FeedMe/Craft is rejecting the supplied pagination URL as being invalid?

At this point, it seems as though either the Sage API is using/returning URLs that are non-standard, or that there's some edge case here which Craft is identifying as an invalid URL when it's a valid URL?

I'm at the edge of my knowledge here, and I don't know what about this is invalid.