Ecommercen Docs

Appearance

Sitemap Generation ​

Flow ID: SY-05 | Module(s): job, eshop, blog, category | Complexity: Medium Last Updated: 2026-07-21 — 4.119.0 release resync: corrected the #459 fix version to 4.119.0; added product_codes to the Data Model

Business Overview ​

The AdvGenerateSitemaps job generates XML sitemaps for search engine crawlers. It streams URLs for all public-facing content -- products, categories, vendors, vendor product lines, vendor-category intersections, blog articles, and CMS pages -- through a bounded in-memory buffer, flushing chunked sitemap files (up to 25,000 URLs per file) plus a sitemap index to the dedicated storage('sitemap') disk under the sitemap/ prefix as soon as the buffer fills. This keeps peak memory flat regardless of catalogue size -- large tenants previously exceeded the CLI's memory/time budget building one big in-memory URL array before writing anything.

The sitemap index is a static file written by the cron alongside the chunks, not a controller endpoint. For local-disk tenants nginx serves all files (chunks + index) directly from public/sitemap/; for K8s tenants the same files live on S3 and are served via a CDN edge mounted at /sitemap/. Crawlers always hit the same registered URL /sitemap/sitemap_index.xml.

The job runs daily at 02:30 by default in its own dedicated sitemap queue (moved off the locked core queue so it can use a larger grace time for the streaming run).

Architecture ​

Nightly cron (no per-request work — no controller in the path):
  AdvGenerateSitemaps::executeCommand()
    |
    +--> @set_time_limit(0) / @ini_set('memory_limit', '1G')   raise CLI limits for the streaming run
    +--> storage('sitemap')->listContents / ->delete(each)     wipe the previous run's chunks + index
    |
    +--> addUrls([homepage, vendors page])    static entry points
    +--> addUrls(getCategories())              product category tree (4 levels deep); also populates $publishCategories
    +--> addUrls(getVendors())                 vendors + exclusive pages + product lines + vendor-category pages
    +--> addUrls(getBlog())                    published blog articles
    +--> addUrls(getPages())                   CMS static pages
    |
    +--> getTopSaleProductIdsForSiteMap(500)   one upfront query resolving top-seller priority (1.0)
    +--> loop: getProductsForSiteMapBatch(afterId, 25000)   keyset (id > afterId) batches until a short batch ends the loop
    |      +--> addUrls(batchUrls)             buffer each batch's URLs
    |
    +--> addUrls() buffers URLs and, once the buffer reaches CHUNK_SIZE (25000), calls flushChunk() immediately
    |      +--> siteMap(chunk)                 generate XML per chunk
    |      +--> ->put(sitemap/sitemap-N.xml)   write the chunk
    |      +--> base_url(path) → $chunkUrls[]  canonical-host URL for the index
    |
    +--> flushChunk(residual buffer)          flush whatever's left after all getters/batches run
    +--> siteMapIndex($chunkUrls)              generate the index XML
    +--> ->put(sitemap/sitemap_index.xml)      write the index alongside the chunks

Request path (crawler hits the registered URL):
  GET /sitemap/sitemap_index.xml
    +--> Local-disk tenants  → nginx serves public/sitemap/sitemap_index.xml directly
    +--> K8s tenants         → CDN edge serves it from S3 at the same path

Key Files ​

FileRole
ecommercen/job/libraries/AdvGenerateSitemaps.phpJob implementation
application/modules/job/libraries/GenerateSitemaps.phpClient-overridable subclass
ecommercen/helpers/xml_helper.phpsiteMap() and siteMapIndex() XML generators
ecommercen/eshop/models/Adv_product_category_model.phpCategory tree and vendor-category queries
ecommercen/eshop/models/Adv_vendors_model.phpVendor sitemap queries
ecommercen/eshop/models/Adv_lines_model.phpVendor product lines
ecommercen/eshop/models/Adv_product_model.phpgetTopSaleProductIdsForSiteMap() (upfront top-seller IDs) and getProductsForSiteMapBatch() (keyset-paginated product rows); the older getProductsForSiteMap() still exists but is now entirely unused
ecommercen/blog/models/Adv_blog_model.phpBlog article list
ecommercen/category/models/Adv_categories_model.phpCMS pages list
src/Storage/Storage.phpLeague Flysystem wrapper used to read/write chunks on local or S3
application/config/storage.phpsitemap storage type — local (root = FCPATH, visibility public) or s3 (with public CDN URL)

Code Flow ​

1. Initialization ​

  • Resolves the current language abbreviation for URL generation (constructor).
  • executeCommand() raises CLI limits for the full-catalogue streaming run: @set_time_limit(0) and @ini_set('memory_limit', '1G').
  • Lists every file under storage('sitemap')->listContents('sitemap') and deletes each one. This wipes the previous run's chunks and index before the new set is written. There is a brief window during regeneration where crawlers may see a 404 for a chunk URL they cached earlier — accepted given the daily cadence; matches the behaviour of the pre-storage-refactor implementation. A first run on a fresh disk (or an adapter that throws on a missing directory) is caught and treated as "nothing to clean".

2. URL Collection ​

Each content type is collected independently and passed to addUrls(), which appends to an in-memory buffer and flushes a chunk file to disk (flushChunk()) as soon as the buffer reaches CHUNK_SIZE (25,000 URLs) — content types are no longer merged into one big array before anything is written. Each URL entry is a map with loc, changefreq, and priority keys.

Static Pages (always included) ​

URLFrequencyPriority
Homepage (base_url())daily1.0
Vendors listing (/vendors)weekly1.0

Categories (gated by XML_FEEDS.IS_ENABLED_SITEMAP_CATEGORIES) ​

Traverses the published category tree up to 4 levels deep via product_category_model->getRecordsTreeMemory(). All published categories are added with:

  • Change frequency: daily
  • Priority: 0.9

Published category IDs are collected for use in the vendor-category intersection later.

Products (gated by XML_FEEDS.IS_ENABLED_SITEMAP_PRODUCTS) ​

Products are streamed rather than loaded in one query:

  1. product_model->getTopSaleProductIdsForSiteMap(500) runs once upfront and returns up to 500 non-deleted, non-zero-price product IDs ordered by tmp_shop_order_basket order count (DESC) -- these keep priority 1.0 regardless of which batch they land in.
  2. product_model->getProductsForSiteMapBatch($afterId, 25000) is called in a loop, each call returning non-deleted, non-zero-price products with id > $afterId ordered by id ASC (keyset pagination), joined to product_codes for COALESCE(SUM(stock), 0) instead of the old per-row correlated subquery. The loop advances $afterId to the last row's id and stops once a batch returns fewer than 25,000 rows.

Priority assignment (per row, checked against the upfront top-seller ID set):

ConditionPriority
In the top-500 seller ID set1.0
Inactive products0.1
Negative stock with negative-stock disabled0.5
All other products0.7

URL resolution: uses the product's url field if set; otherwise falls back to {vendors_base_url}/{vendor_slug}/{product_slug}.

The older product_model->getProductsForSiteMap() (single-query, order-count/active/stock sort) still exists in the codebase but is now entirely unused.

Vendors (gated by XML_FEEDS.IS_ENABLED_SITEMAP_VENDORS) ​

Three sub-collections:

Vendor pages via vendors_model->getVendorsForSiteMap():

  • Top 15% of vendors: priority 1.0 (tracked as "top vendors")
  • Remaining vendors: priority 0.8
  • Exclusive vendors also get a /details page at priority 0.5
  • Change frequency: weekly

Vendor product lines via lines_model->getLinesForSiteMap():

  • Lines belonging to top vendors: priority 0.8
  • Other lines: priority 0.6
  • URL: /vendors/{vendor_slug}/{line_slug}

Vendor-category pages via product_category_model->getVendorsCategoriesForSiteMap():

  • Only for categories that were published (collected during category traversal)
  • Top vendor categories: priority 0.8
  • Other categories: priority 0.6
  • URL: /vendors/{vendor_slug}/{category_slug}

Blog Articles (gated by XML_FEEDS.IS_ENABLED_SITEMAP_BLOG) ​

Published articles with blog_date <= today via blog_model->getBlogsList():

  • Change frequency: weekly
  • Priority: 0.7
  • URL: /blog/{slug}

CMS Pages (gated by XML_FEEDS.IS_ENABLED_SITEMAP_STATIC_PAGES) ​

All CMS pages via categories_model->get_list():

  • Change frequency: yearly
  • Priority: 0.3
  • URL: /category/{slug}

3. Chunk and Index Writing ​

  1. addUrls() appends every collected URL to an in-memory buffer; whenever the buffer reaches CHUNK_SIZE (25,000 URLs) it immediately calls flushChunk() on that slice, so a chunk can be written to disk mid-collection rather than only at the end.
  2. flushChunk() passes the chunk to siteMap() (ecommercen/helpers/xml_helper.php), which generates a compliant <urlset> XML document using DOMDocument.
  3. The chunk is written to sitemap/sitemap-{N}.xml via storage('sitemap')->put(...). The job also captures base_url($chunkPath) into a $chunkUrls array — so the index always references the site's canonical host (/sitemap/sitemap-N.xml) regardless of the storage-disk backend, rather than the storage/CDN host that storage('sitemap')->url() would have returned on S3 tenants.
  4. After all content types and product batches have been collected, any residual buffered URLs (fewer than CHUNK_SIZE) are flushed as one final chunk.
  5. siteMapIndex($chunkUrls) builds the <sitemapindex> XML referencing every chunk written, and the job writes it to sitemap/sitemap_index.xml. If no URLs were ever produced ($chunkUrls empty), the index write is skipped entirely.

There is no controller in the request path — both chunks and index are static files served directly by nginx (local-disk tenants) or by the CDN edge (K8s tenants).

XML Structure ​

Sitemap chunk (sitemap-0.xml):

xml
<?xml version="1.0" encoding="utf-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <changefreq>daily</changefreq>
    <priority>1</priority>
  </url>
  ...
</urlset>

Sitemap index (sitemap_index.xml):

xml
<?xml version="1.0" encoding="utf-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap/sitemap-0.xml</loc>
  </sitemap>
  ...
</sitemapindex>

Data Model ​

No dedicated tables. The job reads from:

TableContent Type
shop_product + shop_product_mui + shop_vendor_muiProducts
shop_product_category + shop_product_category_muiCategories
shop_vendor + shop_vendor_muiVendors
shop_product_lines + shop_product_lines_muiProduct lines
shop_product_category_lpVendor-category intersections
product_codesProduct stock (COALESCE(SUM(stock),0) aggregated per product; feeds the product URL priority 0.5/0.7)
blog + blog_muiBlog articles
categories + categories_muiCMS pages
tmp_shop_order_basketProduct popularity (upfront top-500 seller ID lookup for priority 1.0)

Configuration ​

Job Scheduling (application/config/jobs.php) ​

php
['command' => 'GenerateSitemaps', 'schedule' => '30 2 * * *', 'graceTime' => 1800, 'retryTimes' => 1]

Runs daily at 02:30 in its own dedicated sitemap queue (moved off the shared, locked core queue). The larger graceTime (1800s vs. the 300s the job used as a core command) accommodates the streaming run against large catalogues; retryTimes is reduced to 1 since retrying a full regeneration repeatedly is expensive.

Registry Settings (feature toggles) ​

GroupKeyPurpose
XML_FEEDSIS_ENABLED_SITEMAP_CATEGORIESInclude categories in sitemap
XML_FEEDSIS_ENABLED_SITEMAP_PRODUCTSInclude products in sitemap
XML_FEEDSIS_ENABLED_SITEMAP_VENDORSInclude vendors, lines, and vendor-categories
XML_FEEDSIS_ENABLED_SITEMAP_BLOGInclude blog articles
XML_FEEDSIS_ENABLED_SITEMAP_STATIC_PAGESInclude CMS pages

Output Location ​

Files are written to the dedicated sitemap storage type under the sitemap/ prefix. The actual location depends on the driver configured in application/config/storage.php:

  • Local driver (single-server deployments): chunks and the index land at FCPATH/sitemap/sitemap-N.xml and FCPATH/sitemap/sitemap_index.xml. Nginx serves them as static files at /sitemap/....
  • S3 driver (Kubernetes deployments): same paths but on s3://{SITEMAP_S3_BUCKET}/{FILES_S3_PREFIX}/sitemap/. The configured SITEMAP_S3_URL CDN endpoint serves them. Infra is expected to mount the CDN at https://{app-host}/sitemap/ (typically via CloudFront path-based routing) so the registered URL /sitemap/sitemap_index.xml resolves to the S3 object without changing the public URL.

Driver selection is per-tenant via the SITEMAP_STORAGE_DISK env var (local or s3).

Client Extension Points ​

  1. Override the job class: Extend AdvGenerateSitemaps in application/modules/job/libraries/GenerateSitemaps.php to:

    • Add custom content types (e.g., event pages, landing pages)
    • Exclude specific categories or vendors
    • Change priority assignment logic
    • Modify URL patterns

  2. Override executeCommand(): There is no longer a single getUrls() aggregator to override — URL collection (the addUrls() calls for static pages, categories, vendors, blog, pages) and the product streaming loop both live directly in executeCommand(). Override the whole method to add, remove, or reorder content types, keeping the addUrls() / flushChunk() buffering intact so memory stays bounded.

  3. Override individual getters: Override getCategories(), getVendors(), getBlog(), or getPages() to customize filtering, ordering, or priority logic for those content types. Product URLs are no longer built by a single getProducts() getter — they come from the inline batch loop in executeCommand() calling product_model->getProductsForSiteMapBatch(); override that model method (or the loop itself) to customize product filtering/ordering.

  4. Override XML helpers: Replace siteMap() and siteMapIndex() in application/helpers/xml_helper.php to change XML formatting (e.g., add <lastmod> timestamps or <image:image> extensions).

  5. Custom content types: Add new getter methods in a subclass and feed their output through addUrls() inside an overridden executeCommand().

Business Rules ​

RuleDescription
Full regenerationThe job lists everything under sitemap/ and deletes it before writing the new chunk set + index; no incremental updates
Static indexThe sitemap index is written by the cron as a static file (sitemap/sitemap_index.xml) — no PHP request path
25,000 URLs per fileSitemaps are chunked at CHUNK_SIZE = 25,000 URLs per file, flushed to disk as soon as the in-memory buffer fills (streamed, not a single array_chunk at the end)
Feature-gated contentEach content type can be independently enabled/disabled via registry
Priority-based rankingThe top 500 sellers (from a separate upfront tmp_shop_order_basket query) get maximum priority; the main product stream itself is ordered by id ASC for keyset pagination, not by popularity
Exclusive vendor pagesVendors with is_exclusive = 1 get an additional /details page
Top vendor boostTop 15% of vendors (by position) receive priority 1.0
Published content onlyCategories must be published = 1; blog articles must have blog_date <= today
Non-zero priceProducts with zero price are excluded
Non-deleted productsSoft-deleted products are excluded
Language-aware URLsProduct and category URLs use the configured language abbreviation

Known Issues & Security Gaps ​

None currently known. (Previously: the exclusive-vendor /details page was never emitted because AdvGenerateSitemaps.php:253 compared is_exclusive — returned as the string "1" from the raw CI3 query in getVendorsForSiteMap(), Adv_vendors_model.php:507 — with strict === 1. Fixed in 4.119.0 by casting: (int) $vendor->is_exclusive === 1 — Advisable-com/ecommercen#459.)