[Remove Request] - Large list (VERY)

[DOWRIGHTTV]

I came across these lists while trying to find some open-source data to use as signatures for an open-source security appliance project I am working on. ( https://github.com/DOWRIGHTTV/dnxfirewall )

URL you wish to be removed: do you guys currently have a system to validate whether an active domain still resolves to a valid IP address?

Why you believe this to be a false positive: failed to resolve name.

List it is on: working on pornography at the moment.

Other info you think we should know: if you guys don't already have a system, I can provide a python script that can do it for you. the current state takes a while for large lists (I think this is more to being throttled by public resolver though because my process was running less than 5%). I have currently processed 50k of the names and ~16k of them did not return a valid (A) record. I need to do a few more spot checks to make sure there wasn't any failures due to too many retries though. my current settings took 40 minutes to get through the 50k.

anyways, let me know if you would like me to provide you the lists I have so far and how i can get them to you if so. they also have the successfully resolved IP addresses since they could be useful in determining whether hosts can be combined/merged or if any are sub-domains to another domain on the list.

[gap579137]

I came across these lists while trying to find some open-source data to use as signatures for an open-source security appliance project I am working on. ( https://github.com/DOWRIGHTTV/dnxfirewall )

URL you wish to be removed: do you guys currently have a system to validate whether an active domain still resolves to a valid IP address?

Why you believe this to be a false positive: failed to resolve name.

List it is on: working on pornography at the moment.

Other info you think we should know: if you guys don't already have a system, I can provide a python script that can do it for you. the current state takes a while for large lists (I think this is more to being throttled by public resolver though because my process was running less than 5%). I have currently processed 50k of the names and ~16k of them did not return a valid (A) record. I need to do a few more spot checks to make sure there wasn't any failures due to too many retries though. my current settings took 40 minutes to get through the 50k.

anyways, let me know if you would like me to provide you the lists I have so far and how i can get them to you if so. they also have the successfully resolved IP addresses since they could be useful in determining whether hosts can be combined/merged or if any are sub-domains to another domain on the list.

That would be great if you could provide that script. We had a contributor that was working on scripts for us but IRL got busy and he has not been able to help as much. I have been learning python so I could take over but, I am a ways off from where I need to be to create it.

Thank you for your offer!

[DOWRIGHTTV]

i decided to rewrite it because it was not performing as well as i would like. the new version can check 10k domains in around 45 seconds with only one process.

see: https://gist.github.com/DOWRIGHTTV/7dfb9f639e3b1d02e347b15f100a4d1b

i am handling the dns stuff manually so this version doesnt include the ip address of host is available and only shows the number a (A) and (NS) records returned. if there are (0) (A) records it would indicate the domain is not active. here is the code. hopefully it helps and let me know if there is anything i could add that would be helpful.

[DOWRIGHTTV]

It's the least I could do to give back for even having this awesome resource to begin with. I just need to tidy up the script, make the outputs a little more readable, then add some arguments so settings can be set via shell.

I was thinking I could put the code in a gist with an unlicense [and dedicate it to your group] then just put the link here. Does that work with your contribution requirements?

[gap579137]

That works great!

[gap579137]

@DOWRIGHTTV I am using the python script, starting on some smaller lists. I am looking at the output and unless I missed it somewhere I am not sure what I am looking at.

0.0.0.0 0452securityserver854errordetected.xyz,0,1

Can you help me understand the 0 and 1

[DOWRIGHTTV]

@DOWRIGHTTV I am using the python script, starting on some smaller lists. I am looking at the output and unless I missed it somewhere I am not sure what I am looking at.

0.0.0.0 0452securityserver854errordetected.xyz,0,1

Can you help me understand the 0 and 1

I noticed a problem, but ill circle back on that in a sec. im not sure how much you know about dns so for the sake of this ill pretend like its nothing. the results are in this format: domain name, (A) record count, (NS) record count

the (A) records gives you the ip address of the server you need to connect to, so this has a minimum requirement of 1. if no (A) records are given, then the person who owns that domain has not set up a physical server in the public dns system (or has removed it from record). based on my experience in cyber security, this is a good indication they have "abandoned" that name, their ISP revoked their service due to abuse, or the authoritative server deleted the record due to abuse.

the (NS) record gives you the public dns server responsible for a given record. this isnt very useful within .com, but could be when resolving other top level domain like .xxx for example, because it could tell you if the TLD is itself valid. this would need require to log the returned authoritative server and see if it matches or something along those lines, which i dont do. if we think it is worth doing that, then i can implement that.

the way i figured to handle this would be to open the results in excel, add a filter to columns, then remove any names with a "0" in the second column. this would essentially be "cleaned" list so you could copy the first column into a separate file and put it into host file format.... it also occured to me that maybe i should write in a converter argument in the script so it would automate the cleaning for you if you wanted. let me know if that is something that would be useful.

now back to the problem. i forgot that your files are set up in a host file format, where it has the override ip of "0.0.0.0". essentially it was trying to resolve that entire string you see there, which is definitely not the one you intended.

for now, you need to do a notepad replace on all instances of the "0.0.0.0 " string in the file, which includes the space in between. what i can do is add another command line argument to specify host file formatting to handle this automatically. ill tag you when i get that updated.

sorry for the novel btw.

[gap579137]

@DOWRIGHTTV I am using the python script, starting on some smaller lists. I am looking at the output and unless I missed it somewhere I am not sure what I am looking at.
0.0.0.0 0452securityserver854errordetected.xyz,0,1
Can you help me understand the 0 and 1

I noticed a problem, but ill circle back on that in a sec. im not sure how much you know about dns so for the sake of this ill pretend like its nothing. the results are in this format: domain name, (A) record count, (NS) record count

the (A) records gives you the ip address of the server you need to connect to, so this has a minimum requirement of 1. if no (A) records are given, then the person who owns that domain has not set up a physical server in the public dns system (or has removed it from record). based on my experience in cyber security, this is a good indication they have "abandoned" that name, their ISP revoked their service due to abuse, or the authoritative server deleted the record due to abuse.

the (NS) record gives you the public dns server responsible for a given record. this isnt very useful within .com, but could be when resolving other top level domain like .xxx for example, because it could tell you if the TLD is itself valid. this would need require to log the returned authoritative server and see if it matches or something along those lines, which i dont do. if we think it is worth doing that, then i can implement that.

the way i figured to handle this would be to open the results in excel, add a filter to columns, then remove any names with a "0" in the second column. this would essentially be "cleaned" list so you could copy the first column into a separate file and put it into host file format.... it also occured to me that maybe i should write in a converter argument in the script so it would automate the cleaning for you if you wanted. let me know if that is something that would be useful.

now back to the problem. i forgot that your files are set up in a host file format, where it has the override ip of "0.0.0.0". essentially it was trying to resolve that entire string you see there, which is definitely not the one you intended.

for now, you need to do a notepad replace on all instances of the "0.0.0.0 " string in the file, which includes the space in between. what i can do is add another command line argument to specify host file formatting to handle this automatically. ill tag you when i get that updated.

sorry for the novel btw.

I appreciate the very indepth answer! I know a fair amount about DNS but your answer was very detailed and again I appreciate that.

As far as the "problem" you found. I can use a version of our lists that are auto generated without the leading IP.

IF you have time to create an update to the script that auto generates a clean list that would be amazing, but please do not feel like you need to do it right away. Knowing now what the end numbers "mean" I will be able to somewhat automate that process and manually update the lists.

Thank you again for taking the time to create this!

[DOWRIGHTTV]

@CryptoGap

i updated the gist to make it a little more automated.

it will now automatically detect the input format on a per line basis so the input file can be in host file or name only format or a mix of both. the output will still be the results in csv format, but you can use "-c" to also generate a cleaned up file that lists only domains that had a valid (A) record. if you use "-cH" it will prepend "0.0.0.0 " to all domains in the clean output.

i did some testing to make sure it worked, but not sure how well the coverage was, so let me know if you run into any issues.