RCA News Automation: AI-Assisted Editorial and Publishing Pipeline
01 The Challenge
Publishing timely Romanian motor-insurance news requires monitoring multiple insurer and regulatory websites, extracting useful content, checking relevance, rewriting articles, and publishing them in Joomla.
The manual process creates several problems:
• Fragmented sources with different page structures and publishing patterns.
• Unreliable extraction caused by navigation elements, legal footers, and inconsistent HTML.
• Repetitive editorial work involving relevance, tone, structure, tags, and SEO data.
• Variable AI responses, including malformed JSON or incomplete fields.
• API rate limits and temporary provider failures.
• The need for human approval before publication.
• Complex Joomla article and image publishing.
• Concurrent updates from scraping, AI processing, Telegram, and the CMS.
The project therefore required a resilient editorial pipeline that could normalize content, preserve article states, recover from failures, and publish only explicitly approved material.
02 The Solution
I designed and developed a Python-based RCA news aggregation, review, and publishing platform connecting insurance sources, AI providers, Telegram, and Joomla.
The system includes:
• Multi-source scrapers built with requests, BeautifulSoup, and a shared BaseScraper.
• Layered extraction that removes navigation, scripts, footers, and boilerplate.
• Persistent SQLite states such as new, pending_review, published, rejected, skipped, and ai_failed.
• WAL mode, timeouts, and indexed FIFO queries for safe multithreaded processing.
• AI relevance filtering and editorial rewriting through Groq, with Mistral fallback.
• Defensive JSON recovery for malformed or incomplete model responses.
• Rate limiting based on minute and daily API quotas.
• Telegram controls for editing, categorization, approval, rejection, and manual submissions.
• Separate scheduler and polling threads with controlled shutdown and per-run limits.
• Two-stage Joomla REST publishing for articles and images.
• A dry-run mode for testing without publishing live content.
AI prepares and organizes the material, but every final publication remains under human editorial control.
03 Business Impact
The platform transforms a fragmented editorial process into one traceable workflow connecting every source article, AI decision, draft, edit, and publication action.
Its main benefits are:
• Broader automated monitoring of insurance and regulatory sources.
• Cleaner source content with less navigation and boilerplate.
• Reduced repetitive work for relevance checks, rewriting, formatting, tags, and SEO.
• Explicit human approval before publication.
• Recoverable handling of malformed AI responses and provider failures.
• Transparent article states throughout the workflow.
• Faster review through FIFO processing and automatic candidate loading.
• Support for both scraped and manually submitted content.
• Safer Joomla testing through dry-run mode.
• An extensible foundation for new sources, AI models, commands, and publishing rules.
The result is a human-supervised editorial automation platform that improves speed and consistency while preserving responsibility for the final published article.
Project Gallery
Use the arrows, dots, keyboard, or swipe to explore.
