GitHub - 19-84/redd-archiver: A PostgreSQL-backed archive generator that creates browsable HTML archives from link aggregator platforms including Reddit, Voat, and Ruqqus. · GitHub

Source: original

Redd-Archiver

License: Unlicense Python 3.7+ PostgreSQL Required Version 1.0.0 Multi-Platform MCP Server Codeberg Mirror GitGud Mirror

⭐ If you find this project useful, please star the repo! It helps others discover the tool and motivates continued development.

Transform compressed data dumps into browsable HTML archives with flexible deployment options. Redd-Archiver supports offline browsing via sorted index pages OR full-text search with Docker deployment. Features mobile-first design, multi-platform support, and enterprise-grade performance with PostgreSQL full-text indexing.

Supported Platforms :

Platform | Format | Status | Available Posts | Data

Reddit | .zst JSON Lines (Pushshift) | ✅ Full support | 2.38B posts (40,029 subreddits, through Dec 31 2025) | Download Voat | SQL dumps | ✅ Full support | 3.81M posts, 24.1M comments (22,637 subverses, complete archive) | Download Ruqqus | .7z JSON Lines | ✅ Full support | 500K posts (6,217 guilds, complete archive) | Download

Tracked content:2.384 billion posts across 68,883 communities (Reddit full Pushshift dataset through Dec 31 2025, Voat/Ruqqus complete archives)

Version 1.0 features multi-platform archiving, REST API with 30+ endpoints, MCP server for AI integration, and PostgreSQL-backed architecture for large-scale processing.

🚀 Quick Start

Archive internet history before it disappears - Deploy in 2 minutes, no domain required.

Try the live demo: Browse Example Archive →

→QUICKSTART.md - Step-by-step deployment:

Why now? Communities get banned, platforms shut down, discussions vanish. Start preserving today.

Documentation

→ First time here? QUICKSTART.md - Deploy in 2-15 minutes

→ Quick answers? FAQ - Common questions answered in 30 seconds

→ Need help? Troubleshooting - Fix common issues

→ Using the API? API Reference - 30+ REST endpoints

→ How it works? Architecture - Technical deep-dive

→ Deployment guides:

→ Advanced:

🎯 Key Features

🌐 Multi-Platform Support

Archive content from multiple link aggregator platforms in a single unified archive:

Platform | Format | CLI Flag | URL Prefix

Reddit | .zst JSON Lines | --subreddit | /r/ Voat | SQL dumps | --subverse | /v/ Ruqqus | .7z JSON Lines | --guild | /g/

🤖 MCP Server (AI Integration)

29 MCP tools auto-generated from OpenAPI for AI assistants:

{ "mcpServers": { "reddarchiver": { "command": "uv", "args": ["--directory", "/path/to/mcp_server", "run", "python", "server.py"], "env": { "REDDARCHIVER_API_URL": "http://localhost:5000" } } } }

See MCP Server Documentation for complete setup guide.

Core Functionality

Technical Excellence

Deployment Options

📸 Screenshots

Dashboard

Dashboard

Main landing page showing archive overview with statistics for 9,592 posts across Reddit, Voat, and Ruqqus. Features customizable branding (site name, project URL), responsive cards, activity metrics, and content statistics. (Works offline)

Subreddit Index

Subreddit Index

Post listing with sorting options (score, comments, date), pagination, and badge coloring. Includes navigation and theme toggle. (Works offline - sorted by score/comments/date)

Post Page with Comments

Post Page

Individual post displaying nested comment threads with collapsible UI, user flair, and timestamps. Comments include anchor links for direct navigation from user pages. (Works offline)

Mobile Responsive Design

Mobile Dashboard

Fully optimized for mobile devices with touch-friendly navigation and responsive layout.

Search Interface

Search Form

PostgreSQL full-text search with Google-style operators. Supports filtering by subreddit, author, date range, and score. (Requires Docker deployment)

Search Results

Search results with highlighted excerpts using PostgreSQL ts_headline(). Sub-second response times with GIN indexing. (Server-based, Tor-compatible)

Sample Archive : Multi-platform archive featuring programming and technology communities from Reddit, Voat, and Ruqqus · See all screenshots →

🛠️ Installation

Prerequisites : Python 3.7+, PostgreSQL 12+, 4GB+ RAM

Quick Install (Docker):

git clone https://github.com/19-84/redd-archiver.git cd redd-archiver

Create required directories

mkdir -p data output/.postgres-data logs tor-public

Configure environment (IMPORTANT: change passwords!)

cp .env.example .env nano .env # Edit POSTGRES_PASSWORD and DATABASE_URL

Start services

docker compose up -d

Generate archive (after downloading .zst files to data/)

python reddarc.py data/ \ --subreddit privacy \ --comments-file data/privacy_comments.zst \ --submissions-file data/privacy_submissions.zst \ --output output/

Detailed installation procedures (Docker, Ubuntu/Debian, macOS, Windows WSL2):

📊 Usage

Quick workflow : Download data → Run archive generator → Deploy

Basic Example

Generate archive (assumes .zst files in data/ directory)

python reddarc.py data/ \ --subreddit privacy \ --comments-file data/privacy_comments.zst \ --submissions-file data/privacy_submissions.zst \ --output output/

Deploy with Docker

docker compose up -d

Access at http://localhost

Multi-Platform Support

Detailed Guides

CLI options and advanced workflows: See QUICKSTART.md for complete reference.

Environment Variables:

Required

DATABASE_URL=postgresql://user:pass@host:5432/reddarchiver

Optional Performance Tuning (auto-detected if not set)

REDDARCHIVER_MAX_DB_CONNECTIONS=8 # Connection pool size REDDARCHIVER_MAX_PARALLEL_WORKERS=4 # Parallel processing workers REDDARCHIVER_USER_BATCH_SIZE=2000 # User page batch size REDDARCHIVER_QUEUE_MAX_BATCHES=10 # Queue backpressure control REDDARCHIVER_CHECKPOINT_INTERVAL=10 # Progress save frequency REDDARCHIVER_USER_PAGE_WORKERS=4 # User page generation workers

🏗️ Architecture

Modular PostgreSQL-backed design with 18 specialized HTML modules and multi-platform import support:

Core Components:

Key Features:

Learn more : ARCHITECTURE.md - Complete technical deep-dive

🌐 REST API & Search

30+ REST API Endpoints for programmatic access with MCP/AI optimization:

AI-Optimized Features : Field selection, truncation controls, export formats (CSV/NDJSON), batch endpoints, context endpoints. Rate limited to 100 req/min.

PostgreSQL Full-Text Search : Lightning-fast GIN-indexed search with relevance ranking, highlighted excerpts, and advanced filters (subreddit, author, date, score). Sub-second results for large datasets.

Instance Registry : Distributed leaderboard system for tracking archive instances. Configure metadata, automate scoring, group teams for coordinated archiving.

Learn more : API Documentation · Search Setup · Registry Guide

🎯 Use Cases

Research & Academia

Community Archiving

Investigation & Analysis

🚨 Get Involved: Help Preserve Internet History

Internet content disappears every day. Communities get banned, platforms shut down, and valuable discussions vanish. You can help prevent this.

📥 Download & Mirror Data Now

Don't wait for content to disappear. Download these datasets today:

Platform | Size | Posts | Download

Reddit | 3.28TB | 2.38B posts | Academic Torrents · Magnet Link Voat | ~15GB | 3.8M posts | Archive.org † Ruqqus | ~752MB | 500K posts | Archive.org ‡

† Voat Performance Tip : Use pre-split files for 1000x faster imports (2-5 min vs 30+ min per subverse) ‡ Ruqqus : Docker image includes p7zip for automatic .7z decompression

Every mirror matters. Store locally, seed torrents, share with researchers. Be part of the preservation network.

🌐 Join the Registry: Deploy Your Instance

Already running an archive? Register it on our public leaderboard:

  1. Deploy your instance (Quick Start - 2-15 minutes)
  2. Submit via Registry Template
  3. Join coordinated preservation efforts with other teams

Benefits :

👉 Register Your Instance Now →

🆕 Submit New Data Sources

Found a new platform dataset? Help expand the archive network:

👉 Submit Data Source →Why submit?

📚 Additional Resources

Deployment Guides

Technical Guides

API & Integration

Project Documentation

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for development guidelines, code structure, and testing procedures.

Key areas for contribution:

See our modular architecture (18 specialized modules) for easy entry points to contribute.

📝 License

This is free and unencumbered software released into the public domain. See the LICENSE file (Unlicense) for details.

Anyone is free to copy, modify, publish, use, compile, sell, or distribute this software for any purpose, commercial or non-commercial, and by any means.

📦 Data Sources

This project leverages public datasets from the following sources:

🙏 Acknowledgments

This project builds upon the work of several excellent archival projects:

📧 Contact

💰 Support the Project

Redd-Archiver was built by one person over 6 months as a labor of love to preserve internet history before it disappears forever.

This isn't backed by a company or institution—just an individual committed to keeping valuable discussions accessible. Your support helps:

Every donation, no matter the size, helps keep this preservation effort alive.

Bitcoin (BTC)

bc1q8wpdldnfqt3n9jh2n9qqmhg9awx20hxtz6qdl7

Bitcoin QR Code Scan to donate Bitcoin

Monero (XMR)

42zJZJCqxyW8xhhWngXHjhYftaTXhPdXd9iJ2cMp9kiGGhKPmtHV746EknriN4TNqYR2e8hoaDwrMLfv7h1wXzizMzhkeQi

Monero QR Code Scan to donate Monero

Thank you for supporting internet archival efforts! Every contribution helps maintain and improve this project.

Star History

Star History Chart