Storage Migration to S3

Flow ID: SY-34 | Module(s): ecommercen/job/libraries/, src/Storage/ | Complexity: Medium Last Updated: 2026-07-13

Business Context

AdvMigrateStorageToS3 is a one-time operator job that copies every object from a storage type's local disk to its s3 disk, file-by-file, so a deployment can populate an S3 bucket before flipping the disk-selector env var (e.g. PRIVATE_STORAGE_DISK=s3). Without it, the cutover would read against an empty bucket and start 404-ing on existing invoices, vouchers, and contact-form attachments.

It is the operator-facing companion to the "private storage type" foundation (#235) — see SY-28 Storage Abstraction for the disk/driver layer this job builds on. The job is idempotent (skip-if-exists on the destination), resilient (per-file and per-target failures are logged and skipped, never aborting the run), supports dry-run and deleteSource (with post-write re-verification), and streams every file (never fully buffers). It touches no database tables and rewrites no path/disk columns — disk selection is purely env-driven, so "migration" here means copying blobs, not rewriting DB rows.

API Reference

REST Endpoints (Modern Layer)

None. This is a CLI/queue-only background job with no HTTP surface and no modern src/Rest controller.

Legacy Admin Routes

CLI-only — no admin UI. Reached through the job HMVC controller:

Route	Maps to	file:line
job/(.+)	job/job/index/$1 (direct CLI)	application/config/routes.php:634 (locale twin :642)
job/job/(.+)	job/job/job/$1 (DB-queued)	application/config/routes.php:633 (locale twin :641)

AdvJob's constructor hard-rejects non-CLI requests with error_401() (ecommercen/job/controllers/AdvJob.php:9-12).

Code Flow

operator / cron
  └─ php cli.php job/AdvMigrateStorageToS3 --storages=private --dryRun=true
       └─ AdvJob::index(jobName, param)            ecommercen/job/controllers/AdvJob.php:22-31
            └─ new AdvMigrateStorageToS3()->execute(param)   (CLI arg parser :264-278)
                 └─ executeCommand(options)         AdvMigrateStorageToS3.php:38-55
                      ├─ normaliseStorages()        :196-226  (CSV/array -> [type, subpath] pairs)
                      └─ for each target:
                           migrateStorageType()      :62-84  (resolve storage($type,'local') + ('s3'); skip target on disk-config error)
                             └─ migrate()            :97-173
                                  ├─ local->listContents(rootPath, deep=true)   :117
                                  ├─ for each file:
                                  │    ├─ if s3->exists(path) -> skipped++       :133-134  (idempotency: natural guard)
                                  │    ├─ else stream local->getStream -> s3->putStream -> fclose; copied++   :135-147
                                  │    └─ if deleteSource && !dryRun: re-verify s3->exists then local->delete; else failed++   :149-160
                                  └─ summary log {copied,skipped,failed,deleted}  :171

The DB-queued path is equivalent: AdvJob::job(jobId) (AdvJob.php:67-100) loads the job_schedule row, decodes job_arguments JSON, and calls executeCommand($arguments). Queue mechanics are owned by SY-22 Job Manager.

Data Model

This flow reads and writes no database tables of its own (verified: no db->/model/query/registry usage in AdvMigrateStorageToS3.php). It operates exclusively on the Flysystem storage layer.

The only relevant table is the job queue (used by the DB-queued invocation path, not by the job's logic). job_schedule is the canonical property of SY-22 Job Manager — see database/initial/initial.sql:661-677. The job's storages/dryRun/deleteSource/progressEvery options are stored as JSON in job_schedule.job_arguments.

No Phinx migration is associated with this flow.

Domain Layer

Modern Domain (src/)

None. No src/Domains/.../ entity, repository, service, or REST controller exists for this flow (verified: no MigrateStorage/StorageMigration match under src/). The only modern code it consumes:

File	Responsibility
src/Storage/Storage.php	Flysystem wrapper — listContents (:145-148), exists (:119-122), getStream (:114-117), putStream (:104-107), delete (:124-127); driver dispatch local/s3/sftp (:44-97). Canonical home: SY-28.
src/Logger/NamedLoggerInterface.php	Channel-tagging logger contract (:32-44); DI alias at application/config/container/logger.php:49-50.

Legacy Layer (ecommercen/)

File	Responsibility
ecommercen/job/libraries/AdvMigrateStorageToS3.php	The job (class :22). executeCommand :38-55, migrateStorageType :62-84, migrate :97-173, normaliseStorages :196-226, normaliseBool :232-235, getOptions :237-257, execute CLI parser :264-278, logger :32-36 (channel migrate-storage-to-s3).
ecommercen/core/JobCommand.php:3-7	Interface (executeCommand, getOptions).
application/helpers/storage_helper.php:3-14	storage($type, $disk) — process-static-cached Storage instance.

Configuration

The job operates between the local and s3 disks of whichever storage type(s) are named in the storages option. Storage types are defined in application/config/storage.php (canonical home: SY-28):

Type	Disk-selector env	Notes
files (storage.php:5-36)	FILES_STORAGE_DISK	public bucket, has url
sitemap (:37-64)	SITEMAP_STORAGE_DISK	public
private (:65-100)	PRIVATE_STORAGE_DISK	url=null on both disks (private)
iqvia (:101-116)	IQVIA_STORAGE_DISK	sftp only — no s3 disk (migrating it fails-and-skips)

S3 credential env vars per type are listed in .env.example (FILES_S3_* :138-145, SITEMAP_S3_* :153-159, PRIVATE_S3_* :169-174) — see SY-28 for the full disk-config reference.

The private-disk subtrees a tenant typically migrates correspond to the #235-relocated *_PATH constants in application/config/constants.php: INVOICE_PDF_BASE_PATH='invoices/' (:201), DHL_VOUCHER_SAVE_PATH='vouchers/dhl/' (:204), TEMPORARY_STORAGE_PATH='tmp/' (:209), FORMS_ATTACHMENTS_PATH='forms/' (:212), IMPORT_FILES_PATH='import/' (:216), GENERATED_VIEWS_PATH='views/' (:220).

Options (getOptions() AdvMigrateStorageToS3.php:237-257):

Option	Required	Default	Meaning
storages	yes	—	CSV or array; each token type or type/subpath (first / splits)
dryRun	no	false	List + count but never write or delete
deleteSource	no	false	After verified copy, delete the local source
progressEvery	no	100 (DEFAULT_PROGRESS_EVERY :28)	Progress-log cadence (seen files)

Scheduling: the job is intentionally not registered in application/config/jobs.php (it is a manual one-time job, not scheduled). No registry key or feature flag gates it.

Client Extension Points

None specific. The job is upstream-only. Client forks pulling #235 must run this job before flipping their *_STORAGE_DISK env vars (see Known Issues #5 cross-system caution).

Business Rules

1. Idempotency via natural guard — a destination object that already exists is never overwritten; the file is counted as skipped (AdvMigrateStorageToS3.php:133-134). Re-runs are safe.
2. Streaming, never buffering — each file is copied via getStream → putStream, and the source stream is always fclose-d in a finally (:135-147). Peak memory is independent of file size.
3. deleteSource re-verifies before deleting — with deleteSource, the destination is re-checked with exists() after the write; only then is the local source deleted. A failed re-verification keeps the source and increments failed (:149-160).
4. Per-file resilience — any Throwable during a file's copy/delete increments failed, logs an error, and the loop continues (:161-164).
5. Per-target resilience — a disk-config Throwable (e.g. naming a type with no s3 disk) is logged and that target is skipped without aborting the remaining targets (:62-84).
6. dryRun suppresses all mutations — listing and counting still happen, but no putStream/delete runs (:136, :149).
7. No DB or cache side effects — the job writes no tables, fires no events, invalidates no cache. The env-var disk flip is a separate, manual operator step.

Known Issues & Security Gaps

1. Documented short invocation name MigrateStorageToS3 does not resolve to any class. RESOLVED in 4.113.0 (Advisable-com/ecommercen#391). Previously this job had no thin non-Adv wrapper in application/modules/job/libraries/ (every other job does, e.g. BackUpDataBase extends AdvBackUpDataBase), so the dispatcher's new $jobName() on the literal route/DB string (ecommercen/job/controllers/AdvJob.php:26,78) raised a class-not-found for php cli.php job/MigrateStorageToS3 … and for a queued 'job' => 'MigrateStorageToS3' — only job/AdvMigrateStorageToS3 worked (the CLI path actually surfaced an uncaught \Error, since AdvJob::index catches only \Exception; the queue path caught it and exit(1)-ed). Fixed: application/modules/job/libraries/MigrateStorageToS3.php (class MigrateStorageToS3 extends AdvMigrateStorageToS3 {}) was added, so the documented short name now resolves through the composer classmap; both job/MigrateStorageToS3 and job/AdvMigrateStorageToS3 (and the matching DB-queued 'job' values) work.
2. Not registered in application/config/jobs.php commandOptions (:168-226) — a deliberate choice for this one-time ops migration with a destructive deleteSource option (kept off the scheduler), but the job-manager scheduling UI (SY-22) consequently cannot offer it or validate its options via getOptions(). The DB-queued path itself works now that the short-name wrapper exists (#1).
3. INFO progress/summary lines may be invisible at the default log threshold. Channel migrate-storage-to-s3 is unlisted in application/config/monolog.php $channelThresholds (:8-13), so it inherits APP_LOG_THRESHOLD (default NOTICE). INFO < NOTICE, so the start line (:114), progress lines (:167), and summary (:171) are filtered unless the operator lowers the threshold or sets APP_LOG_CHANNEL_OVERRIDES='{"migrate-storage-to-s3":"INFO"}'. ERROR lines always pass. A fully-successful dry-run may emit no visible output. See SY-32 Logging.
4. iqvia storage type has no s3 disk (storage.php:101-116). Naming it makes storage('iqvia','s3') throw, which is caught and the target skipped (:76-81) — graceful, but a footgun.
5. Cross-system caution (#235): running this job populates the bucket but does not fix external readers of storage/* (e.g. a Laravel invoice-sending API reading a shared mount). After PRIVATE_STORAGE_DISK=s3, those readers must switch to HTTP or mirror the PRIVATE_S3_* env vars — a separate manual step (docs/changelog/Changelog.4.102.md:142-144).
6. USER_LANGS_JSON_PATH was not relocated to the relative-key form — application/config/constants.php:213 still holds '../storage/user_langs/' (old FCPATH . '../storage/…' semantics), unlike the six #235-migrated constants. If user_langs/ data lives under storage/, confirm whether it needs migrating on an S3 cutover.
7. No content/size integrity check on copy — deleteSource re-verification uses exists() only (:153), not a size/checksum compare. A truncated-but-present write would pass and the source would be deleted. High-assurance cutovers should run without deleteSource and verify out-of-band.

Tests

tests/Legacy/Job/AdvMigrateStorageToS3Test.php (522 lines) — 22 tests via reflection + a FakeStorage double, covering: empty/null storages guard, CSV/array normalisation, type/subpath parsing, bool normalisation, happy-path copy + summary, skip-if-exists idempotency, dry-run suppression, deleteSource happy path, post-write-verification-failure keeps source, per-file failure resilience, directory-entry skipping, progress cadence, rootPath filtering, and local-listing-failure handling.

Coverage gaps: the real storage() disk-resolution / storage($type,'s3') failure-skip path (:76-81) and the CLI execute() arg parser (:264-278) are untested. (The MigrateStorageToS3 wrapper added for #1 is a no-logic subclass — nothing to test.)

Related Flows

• SY-28 Storage Abstraction — canonical home for Storage, the storage() helper, disk/driver config, and S3 env vars. This job builds on it.
• SY-22 Job Manager — canonical home for job_schedule, the AdvJob dispatcher, and the CLI/queue mechanics; also the wrapper pattern this job's class now follows (the MigrateStorageToS3 wrapper added in 4.113.0).
• SY-25 File Upload & Storage — the upload side that writes into these same disks.
• SY-08 Database Backup — sibling ops job; a good example of the Adv* + non-Adv wrapper pattern.
• SY-09 Customer Data Jobs — sibling operational jobs.
• SY-27 Deferred Task Runner — explicitly not used by this flow (the job runs synchronously in-process).
• SY-32 Logging — Monolog channel/threshold behavior relevant to Known Issues #3.