Automatic Filtering Logic Doesn't Retain Multiple Status Codes with Same Content-Length

[p1g3]

Recently, I encountered an issue. When the automatic filtering is enabled and the status code matched is set to "ALL", if the Content-Length of both 404 and 403 are the same, the 403 won't be retained, which doesn't match my expectations.

After spending some time debugging, I realized that when the automatic filtering is enabled, not all matching filters are added with an "AND" connection. Instead, only the first matching filter is added. I believe this is unexpected behavior. A more reasonable approach would be to add all of them to the "Filters" and use "AND" for filtering decisions.

Below is the modified code I've come up with, just for your reference:

func (j *Job) calibrateFilters(responses []Response, perHost bool) error {
	// Work down from the most specific common denominator
	if len(responses) > 0 {
		// Content length
		baselineSize := responses[0].ContentLength
		sizeMatch := true
		for _, r := range responses {
			if baselineSize != r.ContentLength {
				sizeMatch = false
			}
		}
		if sizeMatch {
			if perHost {
				// Check if already filtered
				for _, f := range j.Config.MatcherManager.FiltersForDomain(HostURLFromRequest(*responses[0].Request)) {
					match, _ := f.Filter(&responses[0])
					if match {
						// Already filtered
						return nil
					}
				}
				_ = j.Config.MatcherManager.AddPerDomainFilter(HostURLFromRequest(*responses[0].Request), "size", strconv.FormatInt(baselineSize, 10))
			} else {
				// Check if already filtered
				for _, f := range j.Config.MatcherManager.GetFilters() {
					match, _ := f.Filter(&responses[0])
					if match {
						// Already filtered
						return nil
					}
				}
				_ = j.Config.MatcherManager.AddFilter("size", strconv.FormatInt(baselineSize, 10), false)
			}
		}

		// Content words
		baselineWords := responses[0].ContentWords
		wordsMatch := true
		for _, r := range responses {
			if baselineWords != r.ContentWords {
				wordsMatch = false
			}
		}
		if wordsMatch {
			if perHost {
				_ = j.Config.MatcherManager.AddPerDomainFilter(HostURLFromRequest(*responses[0].Request), "word", strconv.FormatInt(baselineWords, 10))
				return nil
			} else {
				_ = j.Config.MatcherManager.AddFilter("word", strconv.FormatInt(baselineWords, 10), false)
				return nil
			}
		}

		// Content lines
		baselineLines := responses[0].ContentLines
		linesMatch := true
		for _, r := range responses {
			if baselineLines != r.ContentLines {
				linesMatch = false
			}
		}
		if linesMatch {
			if perHost {
				_ = j.Config.MatcherManager.AddPerDomainFilter(HostURLFromRequest(*responses[0].Request), "line", strconv.FormatInt(baselineLines, 10))
				return nil
			} else {
				_ = j.Config.MatcherManager.AddFilter("line", strconv.FormatInt(baselineLines, 10), false)
				return nil
			}
		}
	}
	return fmt.Errorf("No common filtering values found")
}

[0xo7]

Hi, for directory scanning, if there are 5 or 10 files of the same size under the same domain name, then this size will be filtered out. User can customize 5 or 10 or other values. Can this function achieve similar needs ?

[p1g3]

When using AND to connect several filters, if the size is the same, it will not be filtered out. Only all filters that meet the conditions will be filtered out. ayu ***@***.***> 于2023年10月16日周一 09:37写道：

…

Hi, for directory scanning, if there are 5 or 10 files of the same size under the same domain name, then this size will be filtered out. User can customize 5 or 10 or other values. Can this function achieve similar needs ? — Reply to this email directly, view it on GitHub <#738 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALZC53USEPWEZGA4YE2CZ3DX7SFW7AVCNFSM6AAAAAA5YMLL6KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRTGU4TMMBXGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>