Design a duplicate-file removal algorithm
Problem: Design a Duplicate File Removal Algorithm
Your task is to design an efficient algorithm to detect and delete byte-identical duplicate files within a filesystem containing millions of photos. This system will operate on a single machine.
Definition of a Duplicate:
- A duplicate is strictly defined as a byte-identical file. Machine learning or computer vision similarity is explicitly out of scope.
Deliverables and Considerations:
- Signature Computation and Storage: Detail your strategy for computing and storing per-file signatures. Consider options such as full cryptographic hashes, a multi-stage approach (e.g., size + partial hash + full hash), and the implications of streaming I/O for large files.
- In-Memory Key-Value Store: Describe how you will utilize an in-memory key-value store to map these computed signatures to their canonical file paths.
- Collision Handling and Verification: Explain your approach to handling potential hash collisions and the necessary verification steps before any file deletion occurs.
- File Metadata: Address how your algorithm will treat files that may have identical names but differ in content, permissions, or timestamps.
- Complexity Analysis: Provide a Big-O analysis for both time and space complexity, with a particular focus on I/O considerations.
- Pseudocode: Furnish pseudocode for a function that returns the set of file paths that are safe to delete.
AI interview prep
Core dilemma
Balancing I/O efficiency for reading millions of files with memory constraints for storing signatures, while ensuring absolute correctness in duplicate identification and safe deletion.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Design a duplicate-file removal algorithm
Problem: Design a Duplicate File Removal Algorithm
Your task is to design an efficient algorithm to detect and delete byte-identical duplicate files within a filesystem containing millions of photos. This system will operate on a single machine.
Definition of a Duplicate:
- A duplicate is strictly defined as a byte-identical file. Machine learning or computer vision similarity is explicitly out of scope.
Deliverables and Considerations:
- Signature Computation and Storage: Detail your strategy for computing and storing per-file signatures. Consider options such as full cryptographic hashes, a multi-stage approach (e.g., size + partial hash + full hash), and the implications of streaming I/O for large files.
- In-Memory Key-Value Store: Describe how you will utilize an in-memory key-value store to map these computed signatures to their canonical file paths.
- Collision Handling and Verification: Explain your approach to handling potential hash collisions and the necessary verification steps before any file deletion occurs.
- File Metadata: Address how your algorithm will treat files that may have identical names but differ in content, permissions, or timestamps.
- Complexity Analysis: Provide a Big-O analysis for both time and space complexity, with a particular focus on I/O considerations.
- Pseudocode: Furnish pseudocode for a function that returns the set of file paths that are safe to delete.