DLE Parser PRO

DLE Parser PRO is a complete DataLife Engine solution for automatically importing, processing, translating and publishing content from external sources. It brings together HTML parsing, RSS/Atom feeds, a hybrid RSS + HTML mode, Sitemap import, image processing, DLE additional fields, duplicate protection, AI rewriting and integration with DLE Multi-Language.

It is built for news sites, niche portals, blogs, aggregators and content projects that need more than a copied article — they need a ready-to-publish post with clean HTML, local images, categories, tags, additional fields, SEO-friendly URLs, translations and diagnostics for every processing step.

Architecture: four parsing modes

HTML Parser — classic web scraping

• Extracts items straight from the HTML structure of the source site
• Handles item lists correctly: the parser walks through every container it finds and looks for the link inside each block
• URL-based progress: tracks the last processed item by its URL rather than by its position on the page
• Processing direction of your choice: newest to oldest or oldest to newest
• Configurable pagination: /page/{page}/, ?page={page}, /p/{page}, /offset/{page} and custom patterns
• Title, full text, categories, main image and additional fields are extracted with CSS selectors
• Content is cleaned locally inside the selected block, without removing elements from the whole document
• Images from the article text are downloaded automatically and external links are replaced with local ones
• Supports DLE additional fields: files, images, videos, galleries, HTML blocks and text values

RSS/Atom Parser — working with news feeds

• Supports RSS 2.0, RSS 1.0 (RDF) and Atom 1.0
• Reads content from content:encoded, description and Atom content
• Handles the media, content, dc and atom namespaces
• Picks up images from enclosure, media:content, media:thumbnail, media:group and the HTML content
• Looks for the main image on the article page as well, through Open Graph and Twitter meta tags
• Extracts categories from the feed and maps them to DLE categories
• Separate cursor strategies for new_to_old and old_to_new, so new items at the top of the feed are never skipped
• Removes ads and service blocks from RSS content with exclusion selectors

Hybrid Parser — RSS and HTML combined

• RSS is used to get the list of fresh items quickly
• The HTML page is used to extract the full text, images, categories and additional fields
• HTML category selector with a merge policy: RSS first, HTML fallback or HTML priority
• Never stuck on a problematic item: item-level failure tracking, fail cursor advancement and a retry limit
• Merges data from RSS and HTML, preferring the more complete source
• Downloads media and additional fields from the HTML version of the article
• The best mode for sites whose RSS contains only a teaser while the full article lives on the page

Sitemap Parser — import from sitemap.xml

• Supports regular sitemap.xml files and sitemap indexes with nested sitemaps
• Gets the list of item URLs from the Sitemap automatically
• Caches the sitemap inventory to speed up repeat runs and reduce the load on the source site
• Processing direction of your choice: newest to oldest or oldest to newest
• Title, full text, categories, images and additional fields are extracted with CSS selectors
• Tracks the total number of URLs and the processing progress
• Protects against publishing already imported items again

AI rewriting, translation and DLE Multi-Language

AI processing with DeepSeek and OpenAI

• DeepSeek or OpenAI as the AI provider
• Rewrites the title, the short description and the full text
• Chunk-based processing of long articles, split into safe fragments
• Keeps the HTML structure intact while rewriting and translating: paragraphs, lists, headings, blockquote, code/pre
• Protects media before AI processing: images, figure, picture, iframe, video and pre/code blocks are temporarily replaced with markers
• Restores media elements automatically after AI processing
• Configurable prompts for the title, short description, full text and tags
• Re-processes suspicious fragments if the AI left part of the text in the source language
• Normalises the HTML after the AI response to prevent broken lists and unclosed tags

Integration with DLE Multi-Language

• Detects active languages from the DLE Multi-Language configuration automatically
• Fills the language columns: title_{iso}, short_story_{iso}, full_story_{iso}, tags_{iso}
• Translates the full text, short description, title and tags into every active language
• Keeps the main language version separate from the translations
• Generates an alt_name for each language version from the translated title
• Checks that the language columns exist before saving, to avoid database errors
• Tag fallback: if the AI could not translate the tags, the module builds them from the already translated content
• Supports different language scenarios: the site’s main language may differ from the language of the source

Testing and preview before publishing

A real dry-run pipeline

• Testing a source uses exactly the same pipeline as real parsing
• Dry-run mode saves nothing to the database but shows the final result of processing
• The preview includes the final title, short description, full text, categories, images, xfields, tags and the duplicate verdict
• You see the real result after cleaning, AI rewriting, translation, image downloads and additional-field processing
• The interface warns that a full simulation may take several minutes and use AI/API requests
• The test result matches what will actually be published on a real CRON run as closely as possible

CSS selectors and data extraction

An extended selector engine

• Basic CSS selectors: tags, classes, IDs, attributes
• Comma-separated groups: h1, .title, [itemprop="headline"]
• Nesting, direct children and adjacent elements: .article .text, .post > p, h1 + .lead
• Several classes on one element: .post.featured
• Attributes: [href], [data-src], [itemprop="articleBody"], [class*="content"]
• A range of pseudo-classes: :first-child, :last-child, :nth-child(), :contains(), :not(), :is(), :where(), :has() within supported limits
• Warnings in test mode for selectors that are too complex or not supported
• Detailed help in the admin panel with examples and an explanation of the supported syntax

DLE additional fields

Downloading files, media and galleries

• Any number of additional fields for each source
• Each field has its own CSS selector, extraction attribute and action type
• Supported attributes: href, src, data-src, data-href, content, text, html
• Supported actions: save the URL/text, download a file, download an image, download a video, save an external video, build a gallery
• Galleries go through every element found and save the result in the DLE format
• Files and videos are saved to uploads/public_files/
• Images are saved to uploads/posts/ together with size metadata
• HTML blocks can be saved to additional fields with the content cleaning policy applied

Images and media

Professional image processing

• Images in the full text are downloaded automatically
• HTTPS, redirects and relative URLs are supported
• Images are stored in date-based folders
• External URLs in the content are replaced with local copies
• JPEG, PNG, GIF and WebP are supported
• Optional conversion to WebP
• The original format is kept correctly when conversion is off
• Resizing with preserved proportions through GD or Imagick
• Images are extracted from figure, picture, img, srcset, lazy-load attributes and meta tags

Categories and tags

Smart category management

• Categories are extracted from RSS, HTML and Sitemap pages
• Visual mapping of source categories to DLE categories
• A default category for items that have no mapping
• Several categories for a single item
• In Hybrid mode, a merge policy for categories coming from RSS and HTML

Tag generation and translation

• Tags are generated automatically by the AI
• Fallback generation from categories, the title and the text if the AI did not return a valid result
• Tag translation for DLE Multi-Language
• Fallback for tags_{iso} from the translated content when AI tag translation is unavailable
• Tag list normalisation: junk, duplicates and overly long phrases are removed

Duplicate protection and reliable progress

Duplicate prevention

• URLs are normalised before the duplicate check
• Checks by source_url, GUID, title and content fingerprint
• A duplicate skip is handled cleanly, without a false save error
• Dedupe progress is cleared when a source is fully reset
• No repeat publishing after changed URL parameters, a trailing slash or tracking parameters

Progress and cursor model

• HTML sources use a URL cursor instead of a position counter on the page
• RSS and Sitemap use separate strategies for new_to_old and old_to_new
• Hybrid mode moves the cursor forward even on controlled errors, so a source never gets stuck on a single item
• Resetting progress also clears the source’s dedupe records

Structured logs, statistics and monitoring

Structured logs

• Every parsing stage is recorded as a structured event
• Each log entry contains source_id, source_type, source_name, item_url, stage, status, message, duration_ms and extra context
• Fetch, parse, download_images, ai_rewrite, translate, generate_tags, translate_tags, db_save and the duplicate check are logged separately
• The logs show at a glance where time is spent or where an error occurs

Source health monitoring

• The last successful run is tracked
• Fail streak — how many errors in a row
• Duplicate rate — to judge the quality of a source
• Average fetch time and average run time
• More accurate statistics of processed items for every source

Cloudflare, proxies and the network layer

Two-level protection bypass

• Enhanced cURL that imitates browser headers
• Cookie persistence between requests
• FlareSolverr integration to get past Cloudflare and JavaScript challenges
• Optional HTTP/SOCKS5 proxy support
• Automatic switch to FlareSolverr when a protection page is detected
• Graceful fallback to plain cURL when no bypass is needed

Admin panel

Sources and settings

• Add and edit HTML, RSS/Atom, Hybrid and Sitemap sources
• Enable and disable sources
• Reset progress and reprocess
• Detailed help on modes, selectors, additional fields and settings
• Warnings about server limits when a full parsing simulation is started
• A built-in preview of the final post before publishing
• Supports both the old and the new DLE AJAX endpoints

Round-robin scheduler

• Load is spread evenly across active sources
• Sources rotate automatically
• Set the number of items per CRON run
• The CRON endpoint is protected with a secret key
• Every run is logged in a structured way
• A lock prevents the same cron process from running twice in parallel

Why use it

• Automation: the whole cycle, from fetching an item to saving a finished post in DLE
• Multilingual: DLE Multi-Language language fields are filled automatically
• Flexibility: four modes for different kinds of sources
• Content quality: AI rewriting, translation, tag generation and preserved HTML structure
• Reliability: duplicate protection, a robust cursor and controlled failure handling
• Transparency: structured logs, health metrics and a clear preview before publishing
• Media: images, files, videos and galleries are stored locally
• Scalability: unlimited sources processed round-robin
• Compatibility: current DLE versions and PHP 7.4–8.4

Use cases

• News aggregators: regular import from several RSS, HTML and Sitemap sources
• Niche blogs: adapting and translating foreign content into the site’s main language
• Multilingual sites: language versions of every item filled automatically
• Review portals: importing reviews, how-tos, guides and technical articles
• Regional media: collecting publications from local sources and processing them
• Educational projects: importing learning materials, manuals and documentation
• Content portals: filling whole sections at scale with control over duplicates, categories and tags

Requirements and compatibility

• DLE: 14.x – 20.0
• PHP: 7.4 – 8.4
• PHP extensions: CURL, DOM, XPath, libxml, GD or Imagick, JSON, mbstring
• MySQL/MariaDB: any version supported by your DLE
• Permissions: write access to /uploads/posts/, /uploads/public_files/, /engine/data/, /engine/cache/
• AI API: DeepSeek or OpenAI, if you use rewriting, translation or tag generation
• CRON: access to the task scheduler
• FlareSolverr: optional, only needed for sites behind Cloudflare/JavaScript protection