NoDupeLabs Backup - Content-Addressable Storage

Overview

This directory contains backups created by NoDupeLabs using Content-Addressable Storage (CAS).

Industry Standard References

This system implements Content-Addressable Storage (CAS) per:

ISO/IEC 15836 - Information technology - File management
ISO/IEC 21320-1 - Archive file format (ZIP, JAR)
NIST SP 800-111 - Storage Encryption Guidelines

CAS is also used by:

Git - content-addressed by SHA-1 hash
Docker - image layers by digest
IPFS - content-addressed filesystem
S3 - etag/content-hash versioning

How It Works

Files are hashed using sha256
Content is stored at: content/<hash>
Snapshots are stored at: snapshots/<id>.json
Same content = same hash = stored once (idempotent)

Recovery (Without NoDupeLabs)

If NoDupeLabs is lost, you can recover files manually:

Option 1: Using Snapshots

Find snapshot in snapshots/ directory
Read the JSON file - it contains file paths and their hashes
Copy from content/<hash> to the original path

Example recovery script:

import json
import shutil
from pathlib import Path

backup_dir = Path(".nodupe/backups")
snapshot_file = backup_dir / "snapshots" / "<snapshot_id>.json"

with open(snapshot_file) as f:
    data = json.load(f)

for file_data in data["files"]:
    src = Path(file_data["backup_path"])
    dst = file_data["path"]
    if src.exists():
        shutil.copy2(src, dst)
        print(f"Restored: {dst}")

Option 2: Direct Content Access

All backup content is stored in content/ directory by hash:

# Find a specific file
ls -la content/

# Copy a file by hash
cp content/<hash> /path/to/restore/file

Supported Hash Algorithms

sha256 (default)
sha384, sha512
sha3_256, sha3_384, sha3_512
blake2b, blake2s

Verification

To verify file integrity:

import hashlib

def verify_file(path, expected_hash, algorithm="sha256"):
    hasher = hashlib.new(algorithm)
    with open(path, "rb") as f:
        while chunk := f.read(8192):
            hasher.update(chunk)
    return hasher.hexdigest() == expected_hash

Generated by NoDupeLabs v1.0.0

2.2 KiB Raw Blame History