Research-Stack/4-Infrastructure/NoDupeLabs

NoDupeLabs

NoDupeLabs is a professional-grade file deduplication system with a plugin-based architecture, comprehensive test coverage, and enterprise-ready features.

---

🎯 Project Status: 100% Complete

As of February 22, 2026, NoDupeLabs has achieved 100% project completion:

• ✅ All 26 tools implemented and tested
• ✅ 6,500+ tests passing (0 failing)
• ✅ 93.30% line coverage / 86.17% branch coverage
• ✅ 50+ files at 100% coverage
• ✅ Comprehensive documentation consolidated and updated

---

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/allaunthefox/NoDupeLabs.git
cd NoDupeLabs

# Install dependencies
pip install -e .

# Verify installation
python -m nodupe --version

Basic Usage

# Scan a directory for duplicates
nodupe scan /path/to/directory

# View scan results
nodupe plan

# Apply deduplication
nodupe apply

Programmatic Access

NoDupeLabs provides a Unix Domain Socket interface for programmatic access:

import socket
import json

# Connect to NoDuPeLabs IPC socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.connect('/tmp/nodupe.sock')

# Send request
request = {
    "method": "scan",
    "params": {"path": "/path/to/directory"}
}
sock.send(json.dumps(request).encode())

# Receive response
response = json.loads(sock.recv(4096).decode())

---

🛠️ Tool Categories

Core Tools

• Archive Tools - ZIP, TAR, and compressed archive handling
• Commands - CLI commands (scan, plan, apply, verify)
• Database - SQLite-based deduplication database
• Hashing - Multiple hash algorithm support (SHA256, BLAKE2, etc.)

Specialized Tools

• GPU/ML - GPU-accelerated and machine learning plugins
• MIME Detection - File type detection via magic numbers
• Parallel Processing - Multi-threaded and multi-process execution
• Time Sync - NTP time synchronization
• Verify - File integrity verification

---

📊 Test Coverage

Metric	Value	Status
Line Coverage	93.30%	✅ Excellent
Branch Coverage	86.17%	✅ Excellent
Total Tests	6,500+	✅ All Passing
Files at 100%	50+	✅ Complete

Running Tests

# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ --cov=nodupe --cov-report=html

# Run specific test category
pytest tests/parallel/ -v

---

📖 Documentation

Quick Reference

• Wiki Home - Main documentation hub
• CLI Reference - Command-line interface guide
• API Reference - Programmatic API
• Testing Guide - Testing best practices

Detailed Guides

• Parallel Testing Guide - Comprehensive testing guide
• Backup Recovery Guide - Backup and recovery procedures
• Development Setup - Development environment setup

---

🏗️ Architecture

NoDupeLabs uses a minimal-core, plugin-first architecture based on Aspect-Oriented Design:

┌─────────────────────────────────────────────────────────┐
│                    NoDuPeLabs Core                       │
├─────────────────────────────────────────────────────────┤
│  Loader  │  Registry  │  Discovery  │  Lifecycle Mgmt  │
└─────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────┐
│                     Plugin System                        │
├──────────┬──────────┬──────────┬──────────┬────────────┤
│ Archive  │ Database │ Hashing  │   MIME   │  Parallel  │
│  Tools   │  Tools   │  Tools   │  Tools   │   Tools    │
└──────────┴──────────┴──────────┴──────────┴────────────┘

Key Design Principles

1. Minimal Core - Lean, portable core engine
2. Plugin-First - All specialized logic in swappable plugins
3. Aspect-Oriented - Clean separation of concerns
4. ISO Compliant - ISO-8000 compliant action codes

---

🔒 Security

• Content-Addressable Storage - SHA256-based deduplication
• Secure Rollback - Transaction logging with snapshot support
• Access Control - Unix socket permissions for IPC
• Security Scanning - Integrated Semgrep and TruffleHog

---

📦 Project Structure

NoDuPeLabs/
├── nodupe/              # Source code
│   ├── core/            # Core engine
│   └── tools/           # Tool implementations
├── tests/               # Test suite (6,500+ tests)
├── docs/                # Documentation
├── wiki/                # Wiki documentation
├── .github/             # GitHub Actions workflows
└── pyproject.toml       # Project configuration

---

🤝 Contributing

Development Setup

# Create virtual environment
python -m venv .venv
source .venv/bin/activate

# Install development dependencies
pip install -e ".[dev]"

# Run pre-commit hooks
pre-commit install

Code Quality

# Run linters
ruff check nodupe/
black --check nodupe/
mypy nodupe/

# Run security scans
semgrep --config auto nodupe/
trufflehog filesystem nodupe/

---

📄 License

Apache License 2.0 - See LICENSE for details.

---

🎉 Recent Achievements

February 2026 - Project 100% Complete

• Parallel Testing Remediation - 70+ new process tests, 50:50 thread:process ratio
• VerifyTool Implementation - All 3 abstract methods implemented, 13/13 tests passing
• MIME Interface Modernization - Proper ABC implementation, 2 bugs fixed
• Documentation Consolidation - 7 files → 2 files (-77% reduction)
• CodeQL Compliance - All 19 alerts fixed or dismissed

---

📞 Support

• Issues: https://github.com/allaunthefox/NoDupeLabs/issues
• Discussions: https://github.com/allaunthefox/NoDuPeLabs/discussions
• Documentation: https://github.com/allaunthefox/NoDuPeLabs/wiki

---

Last Updated: February 22, 2026
Maintainer: NoDuPeLabs Development Team
Status: ✅ Production Ready