Describe the bug

The ZAP spider provides a parameter spider.handleParameters that when set to USE_ALL makes the spider search for all URLS of a site including/counting both the name and value of query parameter. However, the spider only adds one node to the spider tab "Added Nodes" per address + parameter name combination, whereas it adds every url to spider tab "URLs". This is sort of fine. Each individual SiteNode can represent multiple URLs.

The problem occur when querying the API for a list of explored urls. The API will return a list of URLS that is incomplete: it builds that list by adding one URL from each site node. Thus if we have two urls that only differ by the value of their query parameter, only one of them will be added to the list of URLS.

Steps to reproduce the behavior

1. Download application Vulnerable Django Play git clone git@github.com:kaakaww/vuln_django_play.git
2. Start Vulnerable Django Play application with : docker-compose build and then docker-compose up -d
3. Spider a website (e.g. Vulnerable Django Play) that has a URL that can be reached with different query parameter values.
4. Look and record list of URLS in spider tab
5. Look at list of added nodes in spider tab.
6. Query list of urls via JSON/core/view/urls/
7. Verify that list of urls from step 6 does not include all urls of step 4.

Expected behavior

I expected that list of URLS from step 5 is equivalent to list of URLS to step 2. I also expect that list of URLS from step 5 respects parameter spider.handleParameters and includes urls for different values of query parameters.

Based on the example given, I would expect the list of URLS to include both parameter values for query parameter sql in address http://localhost:8020/polls/sql/ :
{
"urls": [
"http://localhost:8020",
"http://localhost:8020/",
"http://localhost:8020/admin",
"http://localhost:8020/admin/",
"http://localhost:8020/polls",
"http://localhost:8020/polls/",
"http://localhost:8020/polls/1",
"http://localhost:8020/polls/1/",
"http://localhost:8020/polls/2",
"http://localhost:8020/polls/2/",
"http://localhost:8020/polls/3",
"http://localhost:8020/polls/3/",
"http://localhost:8020/polls/4",
"http://localhost:8020/polls/4/",
"http://localhost:8020/polls/5",
"http://localhost:8020/polls/5/",
"http://localhost:8020/polls/putstuffhere",
"http://localhost:8020/polls/putstuffhere/",
"http://localhost:8020/polls/rando_stuff",
"http://localhost:8020/polls/rando_stuff/",
"http://localhost:8020/polls/sql",
"http://localhost:8020/polls/sql/?sql=SELECT%20name%20FROM%20sqlite_master%20WHERE%20type='table'",
"http://localhost:8020/polls/sql/?sql=select%201%20where%201=1",
"http://localhost:8020/polls/sql/?sql=",
"http://localhost:8020/polls/sql/?sql=10687542265599578988667963401921049838451035",
"http://localhost:8020/static",
"http://localhost:8020/static/polls",
"http://localhost:8020/static/polls/style.css"
]
}

Software versions

Latest Version but confident it is not version specific.

Screenshots

This is a list of URLs obtained by spidering a vuln_django_play_api . Note that it includes :

http://localhost:8020/polls/sql/?sq%7C=SELECT%2520name%2520FROM%2520sqlite_master%2520WHER
and http://localhost:8020/polls/sql/?sql=select%201%20where%201=1 in the URL tab but only one Site Node representing them in the Added Nodes tab.

When accessing the list of URLS for this scan via API, I get only get url http://localhost:8020/polls/sql/?sql=SELECT%20name%20FROM%20sqlite_master%20WHERE%20type='table'", but not url http://localhost:8020/polls/sql/?sql=select%201%20where%201=1 I :
"urls":[
"http://localhost:8020",
"http://localhost:8020/",
"http://localhost:8020/admin",
"http://localhost:8020/admin/",
"http://localhost:8020/polls",
"http://localhost:8020/polls/",
"http://localhost:8020/polls/1",
"http://localhost:8020/polls/1/",
"http://localhost:8020/polls/2",
"http://localhost:8020/polls/2/",
"http://localhost:8020/polls/3",
"http://localhost:8020/polls/3/",
"http://localhost:8020/polls/4",
"http://localhost:8020/polls/4/",
"http://localhost:8020/polls/5",
"http://localhost:8020/polls/5/",
"http://localhost:8020/polls/putstuffhere",
"http://localhost:8020/polls/putstuffhere/",
"http://localhost:8020/polls/rando_stuff",
"http://localhost:8020/polls/rando_stuff/",
"http://localhost:8020/polls/sql",
"http://localhost:8020/polls/sql/?sql=SELECT%20name%20FROM%20sqlite_master%20WHERE%20type='table'",
"http://localhost:8020/static",
"http://localhost:8020/static/polls",
"http://localhost:8020/static/polls/style.css"
]

Errors from the zap.log file

No response

Additional context

I have a solution to the problem: Change method addUrlsToList in zap/src/main/java/org/zaproxy/zap/extension/api/CoreAPI.java so that when traversing the site nodes it adds all the URLS associated with a SiteNode to the list of urls to return.

diff --git a/zap/src/main/java/org/zaproxy/zap/extension/api/CoreAPI.java b/zap/src/main/java/org/zaproxy/zap/extension/api/CoreAPI.java
index c41c2ff73..21fb46fdb 100644
--- a/zap/src/main/java/org/zaproxy/zap/extension/api/CoreAPI.java
+++ b/zap/src/main/java/org/zaproxy/zap/extension/api/CoreAPI.java
@@ -1711,6 +1711,16 @@ public class CoreAPI extends ApiImplementor implements SessionListener {
addedUrls.add(uri);
}

+ // Each site node can correspond to multiple URLS because of query parameter values.
+ // Hence have to add to potentially add list of URLS per each site node.
+ for (HistoryReference aPastHistoryReference : child.getPastHistoryReference()) {
+ String otherUriOfChild = aPastHistoryReference.getURI().toString();
+ if (!addedUrls.contains(otherUriOfChild) && (baseUrl.isEmpty() || otherUriOfChild.startsWith(baseUrl))) {
+ list.addItem(new ApiResponseElement("url", otherUriOfChild));
+ addedUrls.add(otherUriOfChild);
+ }
+ }
+
addUrlsToList(baseUrl, child, addedUrls, list);
}
}

An alternative solution is to change the way names are given to SiteNodes to include the values of parameters and not just their name. This would require changing method getQueryParamString in class java.org.zaproxy.zap.model.SessionStructure

@@ -775,6 +804,10 @@ public class SessionStructure {
name = name.substring(0, 40) + "...";
}
sb.append(name);
+ String value = entry.getValue();
+ if (value != null) {
+ sb.append("=" + value);
+ }
}
});

Would you like to help fix this issue?

• Yes