Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·12d ago·5 min read
PILLAR: Private Inverted-Index Lexical Lookup for Augmented Retrieval
TL;DR
PILLAR is a Privacy-Preserving Retrieval-Augmented Generation (PPRAG) system that enables users to query a server-held corpus without revealing their query terms or access patterns. It combines sparse and dense retrieval methods to enhance privacy while maintaining low latency and high retrieval quality.
✦ Why It Matters
Consider implementing PILLAR for secure document retrieval in your applications today.
Key Takeaways
- Utilize PILLAR for secure document retrieval in applications.
- Implement sparse and dense retrieval stages for efficiency.
- Choose between PILLAR-Bin and PILLAR-Tree based on latency needs.
- Enhance user privacy without compromising retrieval quality.
How It Works
PILLAR operates in two stages to ensure privacy and efficiency. The first stage uses a small number of Private Information Retrieval (PIR) queries against a specially designed index of BM25 scores to filter the corpus for relevant documents without revealing the query terms to the server.
In the second stage, it retrieves the document embeddings of the filtered candidates and re-ranks them locally, minimizing the number of expensive PIR queries typically required in dense retrieval.