Blooming Onion: Efficient Deduplication through Approximate Membership Testing
| Authors | |
|---|---|
| Year of publication | 2022 |
| Type | Article in Proceedings |
| Conference | Proceedings of the Sixteenth Workshop on Recent Advances in Slavonic Natural Languages Processing, RASLAN 2022 |
| MU Faculty or unit | |
| Citation | |
| Web | |
| Keywords | deduplication; text corpora; Bloom filter |
| Description | Deduplication of source text is an important step in corpus building. Maximum corpus sizes have been grown significantly, along with the requirements for computing resources required for processing them. This article explores reducing the cost of deduplication by applying approximate membership testing using Bloom filtering. |
| Related projects: |