Use AI to Detect Exposed Secrets

[domwhewell-sage]

The module for detecting secrets is great, if they conform to the regex. However whilst researching the a postman module for #898 I have found quite a few secrets that are generic "X-API-Token", "token" or "password" for internal services that would not match against any of them regex's.

I have put these into ChatGPT and it has managed to identify it is a secret using this simple prompt
"Can you take a look at this source code and see if you can identify any plaintext secrets"

I can see another module has been suggested with an OpenAI API key. It might be an idea to use this API key with the secrets detection module.

For example if none of the regex's match in the secrets detection module then make a request to the AI with something similar to the above prompt.

OFC this would need more investigation to see how many false positives this produces, but it could be useful

[TheTechromancer]

Using AI for secrets detection is a good idea and definitely something openai's models are capable of. I think the main issue is that most don't feel comfortable sending this kind of data to OpenAI. (I know they probably have scraped it all already, but it's just a matter of principle lol)

The most interesting project I've seen in this vicinity is Nosey Parker by prateorian. They trained a small AI model that can run locally and does not require sending the secrets over the wire. The tool is open source but sadly not the AI part of it. However I would very much like to train our own model and open source it.

This would require some work and would probably be a separate project which BBOT pulled down as a dependency. But I think it would be fun.

[domwhewell-sage]

Ah thats a great idea I had not considered on-device models.

Another downside that occurred to me is OpenAI may decide to put a filter on that blocks this query, it looks like Google Bard already does.

[TheTechromancer]

This seems like a good start to a discussion about using AI for secrets detection. Converting to discussion.

[domwhewell-sage]

This might be useful https://github.com/GhostPack/DeepPass. Currently it takes full documents but could be modified so a string could be sent directly to this function and the model returns potential passwords

[TheTechromancer]

Wow interesting. I suppose we could even send the RAW_DATA events to excavate for url / hostname extraction.