Light Novel Tracker Notifier
A comprehensive light novel release tracking system that scrapes multiple sources concurrently, deduplicates results, and notifies users when new chapters are available. Built with a modular scraper architecture that supports adding new sources with minimal configuration.
Key Highlights
- Multi-source concurrent scraping with smart deduplication
- Celery-based scheduler with Redis backend for reliable task queuing
- PostgreSQL with SQLAlchemy ORM for persistent state management
- Dockerized deployment on Railway with zero-downtime updates
- Configurable polling intervals with adaptive rate limiting
- Real-time notifications via webhook and email integrations
Architecture
The system uses a producer-consumer pattern where a Celery beat scheduler dispatches scraping tasks to worker nodes. Each source adapter implements a common interface, allowing new sources to be added without modifying core logic. Scraped data flows through a pipeline: raw HTML → parser → deduplication → storage → notification dispatch.
The Problem
Light novel readers had to manually check multiple websites daily for new chapter releases. No centralized tracking system existed that could aggregate releases from disparate sources with different update schedules.
Engineering Challenges
- Handling diverse HTML structures across different novel hosting sites
- Implementing intelligent rate limiting to avoid IP bans while maintaining freshness
- Building a deduplication system that handles partial content overlaps
- Designing a notification system that doesn't spam users on high-activity days
Lessons Learned
- Observer pattern simplifies adding new data sources significantly
- Redis as a message broker provides excellent throughput for task queues
- Docker containerization eliminates environment-specific scraping bugs
- Graceful degradation strategies are essential for external dependency reliability
Future Improvements
- Add ML-based content classification for better deduplication
- Implement WebSocket for real-time push notifications
- Build a React frontend dashboard with analytics
- Add support for manga and webtoon sources
Other Projects
AI Code Review Assistant
An automated code review system powered by local LLMs via Ollama, providing intelligent feedback on pull requests without sending code to external APIs.
Flask SaaS Boilerplate
Production-ready Flask starter kit with authentication, billing, team management, and deployment configuration for Railway.