Research-Stack/5-Applications/nodupe/wiki/Testing/Guide.md

# Testing Guide

**Last Updated:** 2026-02-22
**Status:** ✅ Complete - 100% Project Completeness

---

## Quick Start

### Running Tests

```bash
# Run all tests
pytest tests/ -v

# Run with coverage
pytest --cov=nodupe tests/

# Run specific test file
pytest tests/parallel/test_parallel_thread_vs_process.py -v
```

### Coverage Reports

```bash
# HTML report
pytest --cov=nodupe --cov-report=html tests/
xdg-open htmlcov/index.html

# Terminal report with missing lines
pytest --cov=nodupe --cov-report=term-missing tests/
```

---

## Test Organization

```
tests/
├── archive/              # Archive tool tests
├── commands/             # Command tool tests
├── core/                 # Core module tests
├── database/             # Database tests
├── hashing/              # Hashing tests
├── leap_year/            # Leap year tool tests
├── maintenance/          # Maintenance tool tests
├── mime/                 # MIME detection tests
├── ml/                   # ML plugin tests
├── parallel/             # Parallel processing tests
├── plugins/              # Plugin system tests
├── scanner_engine/       # Scanner engine tests
├── security_audit/       # Security audit tests
├── time_sync/            # Time sync tests
└── test_*.py             # Cross-module tests
```

---

## Test Statistics

| Metric | Value | Status |
|--------|-------|--------|
| **Total Tests** | 6,500+ | ✅ Active |
| **Passing Tests** | 6,500+ | ✅ 100% |
| **Failing Tests** | 0 | ✅ None |
| **Line Coverage** | 93.30% | ✅ Excellent |
| **Branch Coverage** | 86.17% | ✅ Excellent |
| **Files at 100%** | 50+ | ✅ Complete |

---

## Parallel Testing

### Overview

The NoDupeLabs project uses a **pickle-safe test helpers** approach for testing parallel code that utilizes `ProcessPoolExecutor`.

**Why Pickle-Safe Helpers?**
- ✅ Tests REAL `ProcessPoolExecutor` (not mocks)
- ✅ More direct and explicit
- ✅ Better performance (no DI overhead)
- ✅ Already working perfectly (70+ tests passing)
- ✅ More maintainable long-term

**Why NOT Dependency Injection?**
- ❌ Static methods - Parallel is a utility class
- ❌ Direct executor instantiation - Clearer than abstraction
- ❌ Tests verify reality - Mocks don't test actual multiprocessing
- ❌ Performance - DI adds unnecessary overhead
- ❌ Complexity - Over-engineers simple utilities

**See:** [PARALLEL_TESTING_GUIDE.md](../docs/PARALLEL_TESTING_GUIDE.md) for complete details.

### Test Helpers

Location: `tests/parallel/test_helpers.py`

```python
from tests.parallel.test_helpers import (
    double_number,    # x * 2
    square_number,    # x * x
    add_one,          # x + 1
    identity,         # x (unchanged)
    is_even,          # x % 2 == 0
    maybe_raise,      # Raises if x == -1
    slow_square,      # Delays then squares
    MEDIUM_INT_RANGE, # list(range(100))
)
```

### Testing Patterns

#### Pattern 1: Test Both Thread and Process Paths

```python
from tests.parallel.test_helpers import double_number

def test_both_paths(self):
    """Verify thread and process results match."""
    items = [1, 2, 3, 4, 5]

    # Thread path (lambdas OK)
    thread_result = Parallel.process_in_parallel(
        lambda x: x*2, items, use_processes=False
    )

    # Process path (pickle-safe required)
    process_result = Parallel.process_in_parallel(
        double_number, items, use_processes=True
    )

    # Both should produce same results
    assert thread_result == process_result == [2, 4, 6, 8, 10]
```

#### Pattern 2: Error Handling

```python
from tests.parallel.test_helpers import maybe_raise

def test_error_handling(self):
    """Test exception handling with processes."""
    items = [1, -1, 3]  # -1 triggers error

    with pytest.raises(ParallelError) as exc_info:
        Parallel.process_in_parallel(
            maybe_raise,      # ✅ Pickle-safe
            items,
            workers=2,
            use_processes=True
        )

    assert 'Error for value -1' in str(exc_info.value)
```

#### Pattern 3: Timeout Testing

```python
from tests.parallel.test_helpers import slow_operation

def test_timeout_handling(self):
    """Test timeout with processes."""
    with pytest.raises(ParallelError):
        Parallel.process_in_parallel(
            slow_operation,   # ✅ Pickle-safe with delay
            [1],
            timeout=0.01,     # Very short timeout
            use_processes=False  # Threads for timeout
        )
```

### Test Files

| File | Tests | Focus |
|------|-------|-------|
| `test_parallel_thread_vs_process.py` | 31 | Both threads and processes |
| `test_parallel_logic.py` | 9+ | Process-specific tests |
| `test_parallel_logic_comprehensive.py` | 5+ | Process coverage |
| `test_helpers.py` | 20+ helpers | Pickle-safe functions |

---

## Writing Tests

### Basic Structure

```python
"""Tests for module_name - Brief description."""

from unittest.mock import MagicMock, patch

import pytest

from nodupe.module_name import ModuleClass

class TestModuleClass:
    """Test ModuleClass functionality."""

    def test_method_name(self):
        """Test description."""
        # Arrange
        obj = ModuleClass()

        # Act
        result = obj.method()

        # Assert
        assert result == expected
```

### Best Practices

1. **Use descriptive test names**
   ```python
   def test_process_in_parallel_with_processes_basic(self):
       """Clear what is being tested."""
   ```

2. **Test both success and error paths**
   ```python
   def test_success_case(self):
       # Test normal operation

   def test_error_case(self):
       # Test error handling
   ```

3. **Use fixtures for common setup**
   ```python
   @pytest.fixture
   def sample_data():
       return [1, 2, 3, 4, 5]

   def test_with_fixture(self, sample_data):
       # Use sample_data
   ```

4. **Mock external dependencies**
   ```python
   @patch('module.external_service')
   def test_with_mock(self, mock_service):
       # Test with mocked service
   ```

5. **Document thread vs process choice**
   ```python
   # ✅ Pickle-safe - can be used with processes
   double_number

   # ❌ Can't pickle - threads only
   lambda x: x * 2
   ```

---

## Test Coverage by Module

### ✅ Complete Modules (95%+ Coverage)

| Module | Coverage | Tests |
|--------|----------|-------|
| core/api/ | 100% | 400+ |
| database/ | 98.5% | 289 |
| maintenance | 99.5% | 265 |
| time_sync | 92.2% | 318 |
| hashing | 92.5% | 141 |
| scanner_engine | 96.5% | 226 |
| ml/embedding_cache | 99% | 57 |
| mime | 100% | 238 |
| telemetry | 100% | 16 |

### 🟡 Nearly Complete (80-95% Coverage)

| Module | Coverage | Tests | Remaining |
|--------|----------|-------|-----------|
| scanner_engine/processor | 88% | 32 | ~12 lines |
| scanner_engine/walker | 86% | 21 | ~14 lines |
| commands | 94% | 191 | Edge cases |
| leap_year | 60% | 45 | ~100 lines |

---

## Troubleshooting

### Common Issues

#### "Can't pickle local object"

**Cause:** Using local functions or lambdas with `use_processes=True`

**Solution:** Use pickle-safe helpers
```python
# ❌ WRONG
def test_something(self):
    def local_func(x):
        return x * 2
    Parallel.process_in_parallel(local_func, items, use_processes=True)

# ✅ RIGHT
from tests.parallel.test_helpers import double_number
Parallel.process_in_parallel(double_number, items, use_processes=True)
```

#### "Test hangs indefinitely"

**Cause:** Process pool not cleaning up properly

**Solution:** Add timeout
```python
@pytest.mark.timeout(30)  # 30 second timeout
def test_parallel_operation(self):
    Parallel.process_in_parallel(
        double_number, items, timeout=10.0
    )
```

#### "Different results from threads vs processes"

**Cause:** Race condition or shared state

**Solution:** Ensure functions are pure (no side effects)
```python
# ❌ WRONG - Has side effects
counter = 0
def increment(x):
    global counter
    counter += 1
    return x + counter

# ✅ RIGHT - Pure function
def double_number(x):
    return x * 2  # No side effects
```

---

## Resources

### Documentation
- [Parallel Testing Guide](../docs/PARALLEL_TESTING_GUIDE.md) - Complete guide
- [Testing Index](../docs/TESTING_INDEX.md) - Quick reference
- [VerifyTool Completion](../docs/VERIFY_TOOL_COMPLETION.md) - Tool implementation

### Test Files
- `tests/parallel/test_helpers.py` - Pickle-safe helpers
- `tests/parallel/test_parallel_thread_vs_process.py` - Thread vs process tests

### Tools
- pytest - Test framework
- coverage.py - Coverage measurement
- pytest-cov - Coverage plugin
- pytest-timeout - Timeout support

---

**Maintainer:** NoDupeLabs Development Team
**Status:** ✅ Complete - All Tests Passing