OpenAI 发布图像分类器与 ChatGPT 警告机制打击儿童性虐待材料
OpenAI releases a new image classifier and ChatGPT warnings to combat CSAM
- 1OpenAI 发布可检测儿童性虐待材料(含 AI 生成图像)的图像分类器,并通过 API 向部分公司免费开放
- 2官方早期测试称该分类器标记图像减少 48% 仍找到同样数量滥用图像,被标记图像为滥用的几率翻倍
- 3ChatGPT 新增警告屏,相关请求将弹窗提示并引导求助,账号暂停 24 小时,屡犯可能被永久移除
- 4OpenAI 同步发布开发者指南与执法工具,早期测试中一批开发者已借此接入儿童安全工具并向 NCMEC 报备
在诉讼与安全批评压力下,OpenAI 把儿童安全从自家内容封禁扩展为对外输出的检测能力和执法工具,安全能力正在变成平台的对外产品线。
- 影响:接入基础模型的企业需承担儿童安全方面的连带责任;免费分类器与开发者指南降低了自建内容审核的门槛;平台对违规账号处置趋严,把 ChatGPT 嵌入业务流程的企业要预设账号被暂停的影响。
- 参考:可考虑评估 OpenAI 免费分类器 API 能否接入自有内容审核流程,并按自身场景核对误报口径;值得对照新发布的开发者指南,盘点产品是否负有向 NCMEC 报告 CSAM 的义务。
- 依据:检测效果数据为 OpenAI 官方口径的早期合作测试,暂无独立验证;威慑提示消息的试验由塔斯马尼亚大学研究者独立评估;背景是多起诉讼和 Common Sense Media 对青少年风险的批评。
OpenAI today announced new tools to combat child sexual abuse material (CSAM) amid growing safety concerns around the use of AI tools. These features include a new image classifier to detect CSAM, warnings and temporary blocks to prevent users from creating or accessing content that sexualizes children, a new guideline for developers, and a set of tools for law enforcement.
The company unveiled a new image classifier that can detect CSAM images, including the ones generated by AI. OpenAI is giving select companies access to its classifier through its API at no cost. The tool looks for visual signs of abuse in images not seen before and assigns a score that can give an indicative direction to reviewers about dealing with CSAM.
OpenAI said that it developed this classifier with the National Center for Missing and Exploited Children (NCMEC), and it is working with Tech Coalition and ROOST to share access to the tool. In its early testing with partners, the AI company found that the classifier flags 48% fewer images while finding the same number of abusive images, reducing the work of reviewers. Plus, images flagged by the classifier are twice as likely to be abusive images.
What’s more, the company is adding new warning screens in ChatGPT to prevent users from accessing material that sexualizes children. ChatGPT will show an alert to users attempting such prompts, direct users towards specialized support to stop harmful behaviour, and also suspend the account for 24 hours as a warning. The app already blocks CSAM content and requests related to it. It said that repeated offences could lead to the account being removed permanently.
OpenAI noted that it formed the message and guidelines around this behavior with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of deterrent messages.
The company is also publishing guidelines for developers using OpenAI tools to prevent, detect, and help users report CSAM content. It said that in early testing, the guidelines have helped a number of developers adopt child safety tooling and register to report CSAM to NCMEC. OpenAI is also equipping law enforcement dealing with child safety with AI-powered features like dedicated plugins, investigative tools, and complex data analysis, with access managed by program administrators. It is also providing training to investigators along with a field guide on how to use AI.
OpenAI has had several lawsuits against the company blaming its technology for harming users, especially children. Saftey-focused nonprofit Common Sense Media labeled OpenAI’s tech as “unacceptable risk” for teens. Other AI chatbots — specifically Grok — have been used to create nonconsensual sexualized imagery, including of children.
The AI startup had made changes in the last 12 months to its technology including releasing a safety blueprint, updating content guidelines to bar inappropritate content, retuning its model to prevent them from generating harmful content, and releasing a special version of ChatGPT for teens.
When you purchase through links in our articles, we may earn a small commission. This doesn’t affect our editorial independence.
Ivan covers global consumer tech developments at TechCrunch. He is based out of India and has previously worked at publications including Huffington Post and The Next Web.
You can contact or verify outreach from Ivan by emailing [email protected] or via encrypted message at ivan.42 on Signal.
来源:TechCrunch · AI · techcrunch.com