mirror of
https://github.com/allaunthefox/Research-Stack.git
synced 2026-07-31 03:05:21 +00:00
96 lines
2.2 KiB
Markdown
96 lines
2.2 KiB
Markdown
# NoDupeLabs Backup - Content-Addressable Storage
|
|
|
|
## Overview
|
|
|
|
This directory contains backups created by NoDupeLabs using
|
|
**Content-Addressable Storage (CAS)**.
|
|
|
|
## Industry Standard References
|
|
|
|
This system implements Content-Addressable Storage (CAS) per:
|
|
|
|
- **ISO/IEC 15836** - Information technology - File management
|
|
- **ISO/IEC 21320-1** - Archive file format (ZIP, JAR)
|
|
- **NIST SP 800-111** - Storage Encryption Guidelines
|
|
|
|
CAS is also used by:
|
|
|
|
- **Git** - content-addressed by SHA-1 hash
|
|
- **Docker** - image layers by digest
|
|
- **IPFS** - content-addressed filesystem
|
|
- **S3** - etag/content-hash versioning
|
|
|
|
## How It Works
|
|
|
|
1. Files are hashed using sha256
|
|
2. Content is stored at: `content/<hash>`
|
|
3. Snapshots are stored at: `snapshots/<id>.json`
|
|
4. Same content = same hash = stored once (idempotent)
|
|
|
|
## Recovery (Without NoDupeLabs)
|
|
|
|
If NoDupeLabs is lost, you can recover files manually:
|
|
|
|
### Option 1: Using Snapshots
|
|
|
|
1. Find snapshot in `snapshots/` directory
|
|
2. Read the JSON file - it contains file paths and their hashes
|
|
3. Copy from `content/<hash>` to the original path
|
|
|
|
Example recovery script:
|
|
|
|
```python
|
|
import json
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
backup_dir = Path(".nodupe/backups")
|
|
snapshot_file = backup_dir / "snapshots" / "<snapshot_id>.json"
|
|
|
|
with open(snapshot_file) as f:
|
|
data = json.load(f)
|
|
|
|
for file_data in data["files"]:
|
|
src = Path(file_data["backup_path"])
|
|
dst = file_data["path"]
|
|
if src.exists():
|
|
shutil.copy2(src, dst)
|
|
print(f"Restored: {dst}")
|
|
```
|
|
|
|
### Option 2: Direct Content Access
|
|
|
|
All backup content is stored in `content/` directory by hash:
|
|
|
|
```bash
|
|
# Find a specific file
|
|
ls -la content/
|
|
|
|
# Copy a file by hash
|
|
cp content/<hash> /path/to/restore/file
|
|
```
|
|
|
|
## Supported Hash Algorithms
|
|
|
|
- sha256 (default)
|
|
- sha384, sha512
|
|
- sha3_256, sha3_384, sha3_512
|
|
- blake2b, blake2s
|
|
|
|
## Verification
|
|
|
|
To verify file integrity:
|
|
|
|
```python
|
|
import hashlib
|
|
|
|
def verify_file(path, expected_hash, algorithm="sha256"):
|
|
hasher = hashlib.new(algorithm)
|
|
with open(path, "rb") as f:
|
|
while chunk := f.read(8192):
|
|
hasher.update(chunk)
|
|
return hasher.hexdigest() == expected_hash
|
|
```
|
|
|
|
---
|
|
Generated by NoDupeLabs v1.0.0
|