Ecommercen Docs

Appearance

Job Manager & Queue Processing ​

Flow ID: SY-22 | Module(s): job | Complexity: High Last Updated: 2026-04-04

Business Context ​

The Job Manager is the core orchestration system for all background task processing in Ecommercen. It implements a multi-queue job scheduling system where each queue (core, personalization, tracking, import, giftCards, erp, etc.) processes its registered commands according to cron schedules.

The system operates in two tiers:

  1. Job Manager (AdvJobManager): Entry point called by the system cron. Spawns one subprocess per enabled queue in parallel.
  2. Job Queue Manager (AdvJobQueueManager): Per-queue processor. Initializes the schedule, picks the next job, executes it as a subprocess, handles retries and timeouts.

Each job is tracked in the job_schedule database table with status transitions: NOT_STARTED -> RUNNING -> FINISHED or FAILED.

API Reference ​

Health Check Endpoints ​

The application exposes Kubernetes-style health check endpoints (not part of the Job Manager itself, but relevant to system health monitoring):

MethodPathDescription
GET/_healthz/liveLiveness probe -- returns {"error":false,"version":"04.99.005.000"}. Confirms the application process is running.
GET/_healthz/readyReadiness probe -- checks DB connectivity, cache L1/L2 availability, and Solr status. Returns HTTP 500 if any dependency is unhealthy.

Version format: MM.VV.PPP.CCC (e.g., 04.99.005.000) -- matches the format defined in application/config/version.php.

REST Endpoints ​

No REST API. Job management is CLI-only.

CLI Routes ​

RouteControllerMethodDescription
job_managerAdvJobManagerindexProcess all enabled queues in parallel
job_manager/{queue}AdvJobManagerindexProcess a specific queue only
job_queue_manager/{queue}AdvJobQueueManagerindexProcess next job in a specific queue

CLI invocation: php cli.php job_manager (all queues) or php cli.php job_manager/core (specific queue).

Cron entry (typical): * * * * * php /path/to/cli.php job_manager -- runs every minute, each invocation processes one job per queue.

Code Flow ​

Job Manager: Queue Orchestration ​

  1. System cron triggers php cli.php job_manager.
  2. AdvJobManager::index() receives optional $requestedQueue parameter.
  3. getQueues() builds the list of queues to process:
    • If a specific queue is requested, validates it exists in config and is enabled.
    • Otherwise, collects all queues where enabled = true from application/config/jobs.php.
  4. For each queue, creates a Symfony\Component\Process\Process instance:
    php cli.php job_queue_manager/{queue}
  5. Starts all processes in parallel via $process->start().
  6. Blocks until all complete via $process->wait().

Job Queue Manager: Per-Queue Processing ​

  1. AdvJobQueueManager::index($queue) is invoked for a single queue.
  2. Initialize schedule: initializeJobs($queue):
    • Checks if any JOB_NOT_STARTED jobs exist for this queue.
    • If none exist and hasScheduler is true, runs CreateJobSchedule to generate schedule entries from commands config.
  3. Lock check (if lockJobs is true):
    • Calls getRunningJob($queue) to check for JOB_RUNNING jobs.
    • If a running job exists, keepRunningJob() checks if it has exceeded its maximum allowed runtime (graceTime * (retryTimes + 1)).
    • If exceeded, cancels it (sets JOB_FAILED). Otherwise, returns without processing (queue is locked).
  4. Get next job: getNextJob($queue) queries job_schedule for the oldest JOB_NOT_STARTED record where run_at <= now().
  5. Execute job: executeJob($nextJob, $queue, $jobCount): a. Checks if retry count exceeds configured max -- if so, marks as JOB_FAILED. b. Creates a Symfony\Component\Process\Process:
    php cli.php job/job/{jobId}
    c. Sets timeout from grace_time (per-job or per-queue or global default). d. Updates job status to JOB_RUNNING with start_date. e. Runs the process synchronously with $process->run(). f. Retry loop: While not successful and retries remaining, waits 3 seconds and restarts. g. Timeout handling: Catches ProcessTimedOutException, increments retry count, recursively calls executeJob(). h. On success: marks JOB_FINISHED with end_date. i. On failure after all retries: marks JOB_FAILED with end_date.

Job Lifecycle ​

CreateJobSchedule generates entries
          |
          v
    JOB_NOT_STARTED (status=0)
          |
     Queue Manager picks it
          |
          v
    JOB_RUNNING (status=1)
         / \
        /   \
       v     v
JOB_FINISHED  JOB_FAILED
  (status=3)   (status=2)
                   |
              retry? -> JOB_RUNNING again

Domain Layer ​

No modern domain layer. Job management is fully legacy.

Architecture ​

ComponentPathPurpose
AdvJobManagerecommercen/job/controllers/AdvJobManager.phpQueue orchestrator -- spawns parallel queue processors (56 lines)
AdvJobQueueManagerecommercen/job/controllers/AdvJobQueueManager.phpPer-queue job processor -- picks, executes, retries (151 lines)
AdvJobsModelecommercen/job/models/AdvJobsModel.phpModel for job_schedule table (35 lines)
Jobs_modelapplication/modules/job/models/Jobs_model.phpApp-level wrapper
JobStatusecommercen/job/libraries/JobStatus.phpStatus constants (0-3)
CreateJobScheduleecommercen/job/libraries/AdvCreateJobSchedule.phpGenerates job_schedule entries from cron config
ClearJobScheduleecommercen/job/libraries/AdvClearJobSchedule.phpCleans up old completed/failed jobs
Jobs configapplication/config/jobs.phpQueue definitions, commands, schedules
Routesapplication/config/routes.php:630-640CLI route definitions

Dependencies ​

  • Symfony Process: Symfony\Component\Process\Process for subprocess management.
  • Symfony Process Exception: Symfony\Component\Process\Exception\ProcessTimedOutException for timeout handling.

Data Model ​

job_schedule ​

ColumnTypeDescription
idint (PK)Auto-increment primary key
jobvarcharJob command class name
queuevarcharQueue name (core, personalization, tracking, etc.)
job_statusintStatus: 0=not_started, 1=running, 2=failed, 3=finished
run_atdatetimeScheduled execution time
start_datedatetime (nullable)When execution began
end_datedatetime (nullable)When execution completed
grace_timeint (nullable)Per-job timeout override (seconds)
max_retriesint (nullable)Per-job maximum retry limit override
retry_countintCurrent retry attempt count
optionstext (nullable)JSON-encoded job-specific options

Configuration ​

Queue Configuration (application/config/jobs.php) ​

Global defaults:

  • graceTime: 60 seconds (default timeout per job)
  • retryTimes: 0 (default retry count)

Per-queue settings:

QueueEnabledLockSchedulerGraceRetriesPurpose
coretruetruetrue--Primary business jobs (orders, cache, sitemap, backup)
personalizationfalsetruetrue3000Customer tagging and audience population
trackingtruefalsetrue--Price tracking (runs every 5 min)
importfalsetruefalse3000Bulk data imports (no scheduler, on-demand)
giftCardstruefalsetrue--Gift card processing
erpfalsetruetrue--ERP integration jobs
public_marketplacefalsetruetrue--Public marketplace order sync
shopflix_marketplacefalsetruetrue--Shopflix marketplace order sync
reportingtruefalsetrue18003Analytics reporting to external services
fileManagertruefalsefalse3000File processing (no scheduler, on-demand)

Lock behavior:

  • lockJobs: true: Only one job at a time per queue. If a job is running, no new jobs start until it finishes or times out.
  • lockJobs: false: Multiple jobs can run concurrently (each cron invocation processes the next available job).

Scheduler behavior:

  • hasScheduler: true: CreateJobSchedule generates job_schedule entries based on cron expressions in commands.
  • hasScheduler: false: Jobs are inserted into job_schedule by external code (e.g., admin triggers, webhook handlers).

Job Command Options ​

The commandOptions section maps each job class to its getOptions() return value, providing per-command configuration metadata. This is used by CreateJobSchedule to pass options to generated schedule entries.

Client Extension Points ​

  • Custom commands: Add job command entries to queue commands arrays in application/config/jobs.php.
  • Custom queues: Add entirely new queue configurations for client-specific processing needs.
  • Override grace times: Per-job graceTime and max_retries in the config override global defaults.
  • Job options: Pass options array to commands for client-specific parameters (e.g., ['days' => 5] for ClearOldEmails).
  • Custom schedulers: Override CreateJobSchedule to implement custom scheduling logic.
  • On-demand jobs: For queues with hasScheduler: false (like import), client code directly inserts rows into job_schedule to trigger processing.

Business Rules ​

  1. Parallel queues: All enabled queues run in parallel via separate subprocesses, but jobs within a locked queue are sequential. (ecommercen/job/controllers/AdvJobManager.php, index())
  2. Locking: Locked queues (lockJobs: true) prevent concurrent job execution. A running job blocks the queue until it finishes or its maximum runtime is exceeded. (ecommercen/job/controllers/AdvJobQueueManager.php, keepRunningJob())
  3. Maximum runtime: Calculated as graceTime * (retryTimes + 1). If a running job exceeds this, it is forcibly marked as FAILED and the queue resumes. (ecommercen/job/controllers/AdvJobQueueManager.php, keepRunningJob())
  4. Retry with backoff: Between retries, the system waits 3 seconds (sleep(3)) before restarting the process. (ecommercen/job/controllers/AdvJobQueueManager.php, executeJob())
  5. Timeout recovery: On ProcessTimedOutException, the retry count is incremented and the job is re-executed recursively. (ecommercen/job/controllers/AdvJobQueueManager.php, executeJob())
  6. Schedule initialization: On each invocation, if no NOT_STARTED jobs exist for a queue, the scheduler regenerates the schedule from config. (ecommercen/job/controllers/AdvJobQueueManager.php, initializeJobs())
  7. Run-at ordering: Jobs are picked in run_at ASC order -- earliest scheduled job runs first. (ecommercen/job/models/AdvJobsModel.php, getNextJob())
  8. DB reconnect: The AdvJobsModel::update() method calls $this->db->close(); $this->db->initialize() before updates to handle potential MySQL "gone away" errors during long-running jobs. (ecommercen/job/models/AdvJobsModel.php, update())
  9. Status constants: JOB_NOT_STARTED=0, JOB_RUNNING=1, JOB_FAILED=2, JOB_FINISHED=3. (ecommercen/job/libraries/JobStatus.php)
  10. On-demand queues: Queues with hasScheduler: false (like import, fileManager) do not auto-generate schedules. External code inserts rows into job_schedule to trigger processing. (application/config/jobs.php)
  11. Clear job schedule: AdvClearJobSchedule purges old completed/failed records per the queue's keepLogPeriod (DateInterval). (ecommercen/job/libraries/AdvClearJobSchedule.php)

Core Queue Jobs ​

Other Queue Jobs ​

System Infrastructure ​

Integration Flows ​

Wiki Guide: Job Manager Guide -- operational guide for the job scheduling system