Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NS watch/blacklist sometimes fails #6822

Open
tripleee opened this issue Mar 1, 2022 · 64 comments
Open

NS watch/blacklist sometimes fails #6822

tripleee opened this issue Mar 1, 2022 · 64 comments
Assignees
Labels
status: confirmed Confirmed as something that needs working on.

Comments

@tripleee
Copy link
Member

tripleee commented Mar 1, 2022

What problem has occurred? What issues has it caused?

Domains with a subdomain bypass NS checks (originally, I thought anything with www. before the server name, but it seems to be more complex actually).

Recent example, www.eduauraa.com should trigger watched NS but doesn't.
https://metasmoke.erwaysoftware.com/post/352164

What would you like to happen/not happen?

NS watches and blacklists should trigger predictably.

@tripleee tripleee self-assigned this Mar 1, 2022
@tripleee
Copy link
Member Author

tripleee commented Mar 1, 2022

So far, unable to repro. This is a pattern I have observed multiple times in the past but the basic logic is already working correctly.

@tripleee
Copy link
Member Author

tripleee commented Mar 2, 2022

@teward Do you see DNS errors around these posts? Another one just now https://metasmoke.erwaysoftware.com/post/352301

@tripleee
Copy link
Member Author

tripleee commented Mar 3, 2022

@tripleee
Copy link
Member Author

tripleee commented Mar 9, 2022

@stale stale bot added the status: stale label Apr 16, 2022
@stale
Copy link

stale bot commented Apr 17, 2022

This issue has been closed because it has had no recent activity. If this is still important, please add another comment and find someone with write permissions to reopen the issue. Thank you for your contributions.

@stale stale bot closed this as completed Apr 17, 2022
@makyen makyen added the status: confirmed Confirmed as something that needs working on. label Apr 18, 2022
@makyen makyen reopened this Apr 18, 2022
@stale stale bot removed the status: stale label Apr 18, 2022
@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

Yet still another: https://metasmoke.erwaysoftware.com/post/368235

@tripleee
Copy link
Member Author

Another, I guess? https://metasmoke.erwaysoftware.com/post/368913

@tripleee
Copy link
Member Author

IDNA trouble: https://metasmoke.erwaysoftware.com/post/369464 should have triggered on watched NS mihanwebhost.com

@tripleee
Copy link
Member Author

tripleee commented Jun 3, 2022

@tripleee
Copy link
Member Author

tripleee commented Jun 15, 2022

Another: https://metasmoke.erwaysoftware.com/post/372088 (vaguely at the same time as Metasmoke went down briefly, but I don't think it's related to that; should have matched on watched IP, too).

@tripleee
Copy link
Member Author

Yet still another: https://metasmoke.erwaysoftware.com/post/373871

@tripleee
Copy link
Member Author

tripleee commented Aug 2, 2022

@tripleee
Copy link
Member Author

tripleee commented Aug 4, 2022

@tripleee
Copy link
Member Author

tripleee commented Aug 8, 2022

https://metasmoke.erwaysoftware.com/post/381157 unrelated reasons?

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

Something really weird going on with outlookindia.com, the site www.outlookindia.com has a separate set of NSes but I can't match on that either. https://metasmoke.erwaysoftware.com/post/382637

@tripleee
Copy link
Member Author

Ditto for caramellaapp.com in e.g. https://metasmoke.erwaysoftware.com/post/383062

@tripleee
Copy link
Member Author

tripleee commented Oct 5, 2022

@teward
Copy link
Member

teward commented Oct 5, 2022

@teward Do you see DNS errors around these posts? Another one just now https://metasmoke.erwaysoftware.com/post/352301

I have never seen DNS errors in the system on this. However, what needs to be known is that to do forced subdomain stuff and picking up proper subdomain detections to base TLD and such is "what is the base tld?" and I mention this because things like .co.uk are actually secondary level domains despite being TLDs.

If you can suggest a proper way to extract the base domain and then do stuff with that for subdomain queries then it's a simple call to the resolver libraries we're using for the base domain. That's not something that I'm going to write though, I don't have the spare cycles for it.

@teward
Copy link
Member

teward commented Oct 5, 2022

https://metasmoke.erwaysoftware.com/post/391301

Are you sure that's an instance? Specified domain's NS records are Cloudflare, are we flagging Cloudflare as suspicious now?

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@teward Cloudflare specifies a particular NS pair for each individual client, the NS watches and blacklists we have in place target a large number of these particular pairs (and in fact the collection of Cloudflare pairs dominate both of these files). This domain has the NS pair chance.ns.cloudflare.com. ullis.ns.cloudflare.comwhich is inwatched_nses.yml` since a while back.

@tripleee
Copy link
Member Author

tripleee commented Dec 2, 2022

https://metasmoke.erwaysoftware.com/post/401016 - weirdly the previous one https://metasmoke.erwaysoftware.com/post/401012 had "potentially bad NS"

@tripleee
Copy link
Member Author

Tangentially, https://metasmoke.erwaysoftware.com/post/402479 should have matched both IP address and name server, but bypassed those checks apparently because of the link obfuscation.

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

tripleee commented Feb 7, 2023

https://metasmoke.erwaysoftware.com/post/411601 is more straightforward and should be easy to fix.

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

Weirdly, IP lookup failed on https://metasmoke.erwaysoftware.com/post/412865

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

tripleee commented Mar 7, 2023

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

tripleee commented Apr 8, 2023

@tripleee tripleee changed the title NS lookup should extract TLD NS watch/blacklist sometimes fails Apr 8, 2023
@tripleee
Copy link
Member Author

https://metasmoke.erwaysoftware.com/post/422997 bare IP addresses are blacklisted but still not detected

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

tripleee commented May 3, 2023

@tripleee
Copy link
Member Author

tripleee commented May 3, 2023

@tripleee
Copy link
Member Author

tripleee commented May 8, 2023

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

Blacklisted IP not reported: https://m.erwaysoftware.com/posts/uid/meta/389525

@tripleee
Copy link
Member Author

https://metasmoke.erwaysoftware.com/post/431658 didn't trigger for Brenda+Theo (tutuapp.uno)

@tripleee
Copy link
Member Author

https://metasmoke.erwaysoftware.com/post/463575 weirdly didn't trigger even though both www.nimbleappgenie.com and nimbleappgenie.com resolve to 148.66.136.188 (which has been watched for a long time, but which I am now promoting to blacklisted)

@tripleee
Copy link
Member Author

@tripleee
Copy link
Member Author

tripleee commented May 6, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: confirmed Confirmed as something that needs working on.
Development

No branches or pull requests

3 participants