Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·19h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models
TL;DR
Large Language Models (LLMs) face challenges in transparency regarding their pretraining data, raising ethical concerns. MC-PDD, or Masked Corpus-Level Pretraining Data Detection, was developed to identify datasets used in LLM pretraining without needing access to model probability distributions.
✦ Why It Matters
Engineers can use MC-PDD to assess the ethical implications of LLMs by identifying their pretraining datasets.
Key Takeaways
- Use MC-PDD for dataset verification in LLMs.
- Apply masked language modeling techniques for data detection.
- Conduct model audits without needing internal access.
- Ensure compliance with data copyright regulations.
How It Works
MC-PDD masks specific tokens in a text and prompts the LLM to predict the missing content. It then compares the prediction hit rates between a candidate corpus and a reference corpus to determine if the candidate was included in the pretraining data.