Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extremely muscular bodies not being properly detected by content filters. #895

Open
DavidZhang-7 opened this issue Dec 19, 2023 · 1 comment
Labels
bug-report Something might not be working

Comments

@DavidZhang-7
Copy link

DavidZhang-7 commented Dec 19, 2023

What happened?

There are some prompts can bypass content security review of civiai model to generate pictures that can break content rules and not suit for kids under 18.

Steps to reproduce the problem

  1. Choose Babes 3.0, DreamShaper model or other models; Aspect Ratio is portrait; not choose Mature content
  2. Input "a beautiful sexy lady with much extremely explict naked muscle", or "a beautiful sexy lady, naked muscle coverage, the whole body",... like these prompts.
  3. Output pictures that bypass frontend filter and break content rules with high successful probablitiy
    iShot2023-12-20 00 32 46

What should have happened?

The normal action is verify the pics' content complience, block pics that break content rules (https://civitai.com/content/tos) as many as system can, when users does not choose [Mature content].

What platforms do you use to access the site?

No response

What browsers do you use to access the site?

Google Chrome

Additional information, context and logs

No response

@DavidZhang-7 DavidZhang-7 added the bug-report Something might not be working label Dec 19, 2023
@JustMaier
Copy link
Contributor

Interesting it appears that the mature content filters struggle with extremely muscular bodies as it must be assuming they are male bodies which are not sufficiently filtered, likely due to varying degrees of "acceptableness" of male nudity similar to this in sports like body building.

This will require more training and time to implement, thanks for bringing this to our attention.

@JustMaier JustMaier changed the title [Bad Case]: prompts can bypass content security review of LLM to generate non-compliant pics Extremely muscular bodies not being properly detected by content filters. Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug-report Something might not be working
Projects
None yet
Development

No branches or pull requests

2 participants