DLE Parser PRO

DLE Parser PRO is a complete DataLife Engine solution for automatically importing, processing, translating and publishing content from external sources. It brings together HTML parsing, RSS/Atom feeds, a hybrid RSS + HTML mode, Sitemap import, image processing, DLE additional fields, duplicate protection, AI rewriting and integration with DLE Multi-Language.

It is built for owners of news sites, niche portals, blogs, aggregators and content projects who need more than a copied article — they need a ready-to-publish post with clean HTML, local images, categories, tags, additional fields, SEO-friendly URLs, translations and diagnostics for every processing step.

Highlights

• Four modes: HTML pages, RSS/Atom, hybrid RSS + HTML and Sitemap — almost any site can be connected
• Selectors made easy: AI suggests them, and if a selector misses, the module finds the article on the page by itself — without menus, comments or ads
• Your choice of AI: DeepSeek, OpenAI, Claude, Gemini, OpenRouter and any service with an OpenAI-compatible API — rewriting, translation, tags and meta descriptions
• Up-to-date models automatically: the model list with prices is loaded from the provider as soon as you enter a key
• Ready-made news post: images on your server, categories, tags, SEO-friendly URLs, additional fields and SEO — just like adding it through the DLE admin panel
• Under control: filters, replacements, per-source limits, duplicate protection and a test run with preview before publishing

Architecture: four parsing modes

HTML Parser — classic web scraping

• Extracts items straight from the HTML structure of the source site
• Handles item lists correctly: the parser walks through every container it finds and looks for the link inside each block
• URL-based progress: tracks the last processed item by its URL rather than by its position on the page
• Processing direction of your choice: newest to oldest or oldest to newest
• Configurable pagination: /page/{page}/, ?page={page}, /p/{page}, /offset/{page} and custom patterns
• Title, full text, categories, main image and additional fields are extracted with CSS selectors
• Content is cleaned locally inside the selected block, without removing elements from the whole document DOM
• Images from the article text are downloaded automatically and external links are replaced with local ones
• Supports DLE additional fields: files, images, videos, galleries, HTML blocks and text values

RSS/Atom Parser — working with news feeds

• Supports RSS 2.0, RSS 1.0 (RDF) and Atom 1.0
• Reads content from content:encoded, description and Atom content
• Handles namespaces: media, content, dc, atom
• Picks up images from enclosure, media:content, media:thumbnail, media:group and the HTML content
• Also looks for the main image on the article page through Open Graph and Twitter meta tags
• Extracts categories from the feed and maps them to DLE categories
• Separate cursor strategies for new_to_old and old_to_new, so new items at the top of the RSS feed are never skipped
• Removes ads and service blocks from RSS content with exclusion selectors

Hybrid Parser — RSS and HTML combined

• RSS is used to quickly get the list of fresh items
• The HTML page is used to extract the full text, images, categories and additional fields
• HTML category selector with a category merge policy: RSS first, HTML fallback or HTML priority
• Never gets stuck on a problematic item: item-level failure tracking, fail cursor advancement and a retry limit
• Merges RSS and HTML data, giving priority to the more complete source
• Downloads media and additional fields from the HTML version of the article
• The best mode for sites where RSS carries only a teaser and the full article is available on the page

Sitemap Parser — importing content from sitemap.xml

• Supports regular sitemap.xml files and sitemap indexes with nested sitemap files
• Automatically retrieves the list of article URLs from the Sitemap
• Caches the sitemap inventory to speed up repeat runs and reduce load on the source site
• Processing direction of your choice: newest to oldest or oldest to newest
• Title, full text, categories, images and additional fields are extracted with CSS selectors
• Tracks the total number of URLs and processing progress
• Protects against republishing items that have already been imported

AI rewriting, translation and DLE Multi-Language

AI processing: six provider options

• DeepSeek, OpenAI, Anthropic Claude, Google Gemini, OpenRouter and any service with an OpenAI-compatible API: Groq, Mistral, xAI, Together, Fireworks, Moonshot, Z.ai, Qwen, YandexGPT, plus local Ollama and LM Studio
• Up-to-date models are loaded from the provider automatically as soon as you enter a key: a simple searchable list with the price per 1M tokens; you can also type in any model manually
• “Test connection” button — a short request to the model showing token usage and cost
• Output language: the title, text, description and tags are written in the language you choose — handy for translating foreign sources
• AI-generated meta description for each news post (140–160 characters) and automatic meta tags
• Monthly spending limit with a warning or a stop to AI processing; spending per model is shown on the “Statistics” tab
• Retries when the service is overloaded and automatic parameter adjustment for models that no longer accept deprecated settings
• Rewrites the title, short description and full text of the article
• Chunk-based processing of long articles, split into safe fragments
• Keeps the HTML structure intact during rewriting and translation: paragraphs, lists, headings, blockquote, code/pre
• Protects media elements before AI processing: images, figure, picture, iframe, video and pre/code blocks are temporarily replaced with markers
• Media elements are restored automatically after AI processing
• Customizable prompts for the title, short description, full text and tags
• Re-processes suspicious fragments if the AI left part of the text in the source language
• Extra HTML normalization after the AI response to prevent broken lists and unclosed tags

DLE Multi-Language integration

• Automatically detects the active languages from the DLE Multi-Language configuration
• Fills in the language fields: title, short and full description, tags, news URL (SEO-friendly URL), meta description and keywords for each language
• If translation into a language fails, that language version is not saved — source-language text never ends up in another language’s field
• Translates the full article, short description, title and tags for every active language
• Stores the primary language version separately from the translations
• Checks that the language columns exist before saving to avoid database errors
• Tag fallback: if the AI could not translate the tags, the module builds them from the already translated content
• Supports different language scenarios: the site’s primary language can differ from the language of the source site

Filters, replacements and per-source publishing

• Required words and stop words, minimum text length, maximum article age — checked before AI so no tokens are wasted
• Rule-based text replacements, including regular expressions
• For each source: publish immediately or send to moderation, news author, show on the homepage, allow comments and a daily news limit
• Export and import sources to a file — move settings between sites in a minute

Testing and preview before publishing

Real dry-run pipeline

• Source testing uses the same pipeline as real parsing
• Dry-run mode does not save the item to the database but shows the final processing result
• The preview includes the final title, short description, full text, categories, images, xfields, tags and duplicate verdict
• The test shows the real result after cleanup, AI rewriting, translation, image downloading and additional field processing
• The interface warns that a full simulation may take several minutes and use AI/API requests
• The test result matches as closely as possible what will be published on a real CRON run

CSS selectors and data extraction

Advanced selector engine

• Basic CSS selectors: tags, classes, IDs, attributes
• Comma-separated groups: h1, .title, [itemprop="headline"]
• Descendants, direct children and adjacent siblings: .article .text, .post > p, h1 + .lead
• Multiple classes on one element: .post.featured
• Attributes: [href], [data-src], [itemprop="articleBody"], [class*="content"]
• A range of pseudo-classes: :first-child, :last-child, :nth-child(), :contains(), :not(), :is(), :where(), :has() within supported limits
• Test-mode warnings for selectors that are too complex or unsupported
• Detailed help in the admin panel with examples and an explanation of the supported syntax

DLE additional fields

Downloading files, media and galleries

• Any number of additional fields per source
• Each field has its own CSS selector, extraction attribute and action type
• Supported attributes: href, src, data-src, data-href, content, text, html
• Supported actions: save URL/text, download file, download image, download video, save external video, build gallery
• Galleries automatically go through every element found and save the result in DLE format
• Files and videos are saved to uploads/public_files/
• Images are saved to uploads/posts/ with size metadata
• HTML blocks can be saved to additional fields with the content cleanup policy applied

Images and media

Professional image processing

• Automatic downloading of images from the full article text
• Supports HTTPS, redirects and relative URLs
• Images are stored in date-based folders
• External URLs in the content are replaced with local copies
• Supports JPEG, PNG, GIF, WebP and AVIF
• Optional conversion to WebP or AVIF
• Minimum image width — icons and counters stay out of the news post
• Watermark from your DLE template
• Proportional downscaling to a maximum width and height
• Keeps the original format when conversion is disabled
• Aspect-ratio-preserving resize via GD or Imagick
• Extracts images from figure, picture, img, srcset, lazy-load attributes and meta tags

Categories and tags

Smart category management

• Extracts categories from RSS, HTML and Sitemap pages
• Visual mapping of source-site categories to DLE categories
• Default category for unmapped items
• Multiple categories per item
• Hybrid mode offers a merge policy for RSS and HTML categories

Tag generation and translation

• Automatic tag generation with AI
• Fallback tag generation from categories, title and text if the AI did not return a valid result
• Tag translation for DLE Multi-Language
• Fallback for tags_{iso} from translated content if AI tag translation is unavailable
• Tag list normalization: removes junk, duplicates and overly long phrases

Duplicate protection and reliable progress

Duplicate prevention system

• URL normalization before duplicate checks
• Checks by source_url, GUID, title and content fingerprint
• Handles duplicate skips correctly, without a false save error
• Clears dedupe progress on a full source reset
• Prevents republishing after changes to URL parameters, trailing slashes or tracking parameters

Progress and cursor model

• HTML sources use a URL cursor instead of a position counter on the page
• RSS and Sitemap use separate strategies for new_to_old and old_to_new
• Hybrid mode advances the cursor even on controlled errors, so a source never gets stuck on one item
• Resetting progress clears the source’s related dedupe records

Structured logs, statistics and monitoring

Structured logs

• Every parsing stage is recorded as a structured event
• Each log entry contains source_id, source_type, source_name, item_url, stage, status, message, duration_ms and additional context
• The fetch, parse, download_images, ai_rewrite, translate, generate_tags, translate_tags, db_save and duplicate check stages are logged separately
• Logs make it easy to see exactly where time is spent or where an error occurs

Source health monitoring

• Tracks the last successful run
• Fail streak shows the number of consecutive errors
• Duplicate rate to assess source quality
• Average fetch time and average run time
• More accurate statistics of processed items for each source

Cloudflare, proxy and network layer

Two-level protection bypass system

• Enhanced cURL with browser header emulation
• Cookie persistence between requests
• FlareSolverr integration to get past Cloudflare and JavaScript challenges
• Optional HTTP/SOCKS5 proxy support
• Automatic switch to FlareSolverr when a protection page is detected
• Graceful fallback to plain cURL when no bypass is needed

Admin panel

Managing sources and settings

• Add and edit HTML, RSS/Atom, Hybrid and Sitemap sources
• Enable and disable sources
• Reset progress and re-process
• Interface in Russian, English, Ukrainian and Spanish, styled like the DLE admin panel, with no horizontal scrolling on any screen
• Clear hints when something goes wrong: if a site builds its pages with scripts or blocks robots, the module tells you straight away to connect its RSS or sitemap instead
• Detailed documentation at doc.dlemod.com
• Server limit warnings when running a full parsing simulation
• Built-in preview of the final item before publishing
• Supports both old and new DLE AJAX endpoints

Security

• Cleans source content of malicious code: scripts, dangerous attributes and iframes from third-party sites are removed
• Protection against requests to internal server addresses when loading pages, images and files
• All admin actions are protected against request forgery; AI keys are hidden in the settings

Round-Robin scheduler

• Even load distribution across active sources
• Automatic source rotation
• Configurable number of items per CRON run
• CRON endpoint protected by a secret key
• Structured logging of every run
• Lock protection against parallel runs of the same cron process

Benefits

• Automation: the full cycle from fetching content to saving a ready-to-publish post in DLE
• Multilingual: automatic filling of DLE Multi-Language language fields
• Flexibility: four operating modes for different types of sources
• Content quality: AI rewriting, translation, tag generation and preserved HTML structure
• Reliability: duplicate protection, a robust cursor, controlled failure handling
• Transparency: structured logs, health metrics and a clear preview before publishing
• Media: local storage of images, files, videos and galleries
• Scalability: unlimited sources with Round-Robin processing
• Compatibility: supports current DLE versions and PHP 7.4-8.4

Use cases

• News aggregators: regular import from multiple RSS, HTML and Sitemap sources
• Niche blogs: adapting and translating foreign content into the site’s primary language
• Multilingual sites: automatic filling of language versions of articles
• Review portals: importing reviews, how-tos, guides and technical articles
• Regional media: collecting publications from local sources with further processing
• Educational projects: importing learning materials, manuals and documentation
• Content portals: bulk-filling sections with control over duplicates, categories and tags

Technical requirements and compatibility

• DLE: 15.1 – 21.0
• PHP: 7.4 - 8.4
• PHP extensions: CURL, DOM, XPath, libxml, GD or Imagick, JSON, mbstring
• MySQL/MariaDB: a compatible version supported by your DLE version
• Permissions: write access to /uploads/posts/, /uploads/public_files/, /engine/data/, /engine/cache/
• AI API: a key from any supported provider (or a local Ollama / LM Studio server) if you use rewriting, translation or tag generation
• CRON: access to configure the task scheduler
• FlareSolverr: optional, only needed for sites with Cloudflare/JavaScript protection