SY-33: Bot Detection and Session Exclusion

Flow ID: SY-33 | Module(s): application/libraries/Session, application/config | Complexity: Low Last Updated: 2026-06-12

Status: Active

Business Context

Every HTTP request that reaches the PHP process causes MY_Session to run before the controller dispatches. Prior to commit 03226835d, bot detection relied on CodeIgniter's built-in User_agent::is_robot(), which maintains a hand-curated list of approximately 88 user-agent patterns. That list predates the current generation of AI web crawlers. Agents such as GPTBot, ClaudeBot, PerplexityBot, CCBot, Amazonbot, Applebot, and Bytespider were not present in the list, so they each received a full CI3 session: a database row write on first visit, a ci_session cookie set in the response, and potential session-ID regeneration on subsequent requests. At meaningful crawler volumes this creates write pressure on the session store, bloats session tables with rows that are never read again, and sets cookies on clients that ignore them.

Commit 03226835d replaced the CI3 list with matomo/device-detector, which is actively maintained and covers modern AI crawler signatures. Commit cbefa221d removed a broken attempt to wire a PSR-6 cache — the di() helper takes no arguments, so passing a class name returned the container itself and caused a TypeError on every request; pod-local caching was dropped until correct wiring could be implemented.

Commit c6211d80cd (#285) resolved both outstanding problems. It switched from the full DeviceDetector class to the standalone \DeviceDetector\Parser\Bot, which loads only bots.yml and stops at the combined pre-match — it never walks the much larger OS/client/device regex tree for non-bot UAs. It also added a working cross-request PSR-6 cache via \DeviceDetector\Cache\PSR6Bridge over di()->get(\Advisable\Cache\CachePool::class). The measured effect: ~3325 ms down to ~24 ms warm for a browser UA; the first request per pod/TTL still costs ~1.4 s to populate the bots-regex cache entry; the uncached fallback (cache pool unavailable) also costs ~1.4 s per request and is logged as a WARNING.

Route exclusion — skipping session creation for specific controller/method pairs regardless of bot status — was already present before this change and is unchanged.

Architecture Overview

Incoming request
  └─ MY_Session::__construct()
       ├─ load config/session_excludes.php
       ├─ isBotSession()  ──────────────────────────────────────────────┐
       │    ├─ empty User-Agent?  → true (skip session)                 │
       │    ├─ botDetectorCache()                                        │
       │    │    ├─ PSR6Bridge(di()->get(CachePool::class)) → cache     │
       │    │    └─ Throwable? → log WARNING, return null (uncached)    │
       │    ├─ Bot::setUserAgent() + discardDetails() + setCache()      │
       │    ├─ Bot::parse() !== null? → true (is bot)                   │
       │    └─ Throwable? → log error, return false (fail open)         │
       ├─ isRouteExcluded()  ──────────────────────────────────────────┐│
       │    └─ router class+method in session_excludes list? → true    ││
       │                                                                ││
       ├─ isBotSession() || isRouteExcluded() == true ─────────────────┘┘
       │    → return early: NO session created, NO cookie set
       │
       └─ neither condition → parent::__construct($params)
              → standard CI3 session lifecycle (DB row, cookie, regeneration)

Key Files

File	Role
`application/libraries/Session/MY_Session.php`	Session subclass — `isBotSession()` and `isRouteExcluded()` intercept `__construct()` before `parent::__construct()` runs
`application/config/session_excludes.php`	Array of `[ControllerClass, 'method']` pairs whose routes skip session creation entirely

Code Flow

Session entry point

MY_Session::__construct() (lines 8–19) loads the exclusion config, evaluates both guards, and short-circuits before calling parent::__construct(). (MY_Session.php:8-19)

php

// application/libraries/Session/MY_Session.php:8-19
public function __construct($params = array())
{
    get_instance()->load->config('session_excludes');
    $this->excludes = get_instance()->config->item('session_excludes');

    //parent construct calls sess_<methods> so it needs to have the config already loaded
    if ($this->isBotSession() || $this->isRouteExcluded()) {
        return;
    }

    parent::__construct($params);
}

The comment on MY_Session.php:13 is significant: parent::__construct() triggers CI3's internal sess_* methods (database write, cookie dispatch). Returning early before that call is the mechanism that prevents all session side-effects.

Bot Detection (`isBotSession()`)

isBotSession() (lines 21–47) reads the raw HTTP_USER_AGENT server variable and passes it to the standalone \DeviceDetector\Parser\Bot:

php

// application/libraries/Session/MY_Session.php:21-47
protected function isBotSession(): bool
{
    $userAgent = $_SERVER['HTTP_USER_AGENT'] ?? '';

    if ($userAgent === '') {
        return true;
    }

    try {
        // Standalone bot parser: only loads regexes/bots.yml and stops at the
        // combined pre-match — a full DeviceDetector->parse() would also walk
        // the (much larger) OS/client/device regex tree for every non-bot UA.
        $botParser = new \DeviceDetector\Parser\Bot();
        $botParser->setUserAgent($userAgent);
        $botParser->discardDetails();

        $cache = $this->botDetectorCache();
        if ($cache !== null) {
            $botParser->setCache($cache);
        }

        return null !== $botParser->parse();
    } catch (\Throwable $detectorError) {
        log_message('error', 'MY_Session bot detection failed: ' . $detectorError->getMessage());
        return false;
    }
}

Key behaviors from this implementation:

\DeviceDetector\Parser\Bot is used rather than the full DeviceDetector class. The bot parser loads only regexes/bots.yml and exits at the combined pre-match; it never walks the OS/client/device regex tree. This is the primary source of the ~3325 ms → 24 ms warm improvement for browser UAs. (MY_Session.php:33)
discardDetails() is called before parse(). The bot parser has two modes: a full mode that retains bot name, category, and URL metadata, and a lightweight mode (after discardDetails()) that only records whether a match was found. The lightweight mode is faster; the metadata is not needed here. (MY_Session.php:35)
Bot detection result is null !== $botParser->parse(): parse() returns a match array for bots and null for non-bots. (MY_Session.php:42)
botDetectorCache() is called to obtain a cross-request PSR-6 cache. If the cache is unavailable the method returns null and parsing proceeds uncached. (MY_Session.php:37-40)
A missing or empty User-Agent short-circuits to true (bot) before any parser is constructed. Empty UAs are characteristic of headless/automation clients and raw HTTP library calls; treating them as bots is a safe default. (MY_Session.php:25-27)
Any \Throwable from the parser logs an error via log_message() and returns false (fail open). The session will be created for that request. This prevents a faulty or incompatible matomo/device-detector version from silently breaking session creation for all real users. (MY_Session.php:43-46)

Cache wiring (`botDetectorCache()`)

botDetectorCache() (lines 56–69) returns a \DeviceDetector\Cache\PSR6Bridge wrapping di()->get(\Advisable\Cache\CachePool::class), or null if the cache pool is unavailable:

php

// application/libraries/Session/MY_Session.php:56-69
protected function botDetectorCache(): ?\DeviceDetector\Cache\CacheInterface
{
    try {
        return new \DeviceDetector\Cache\PSR6Bridge(
            di()->get(\Advisable\Cache\CachePool::class)
        );
    } catch (\Throwable $cacheError) {
        log_message(
            'warning',
            'MY_Session bot detector cache unavailable — parsing uncached: ' . $cacheError->getMessage()
        );
        return null;
    }
}

The PSR6Bridge wraps the same \Advisable\Cache\CachePool instance that Adv_front_controller::getUserDeviceInformation() uses at ecommercen/core/Adv_front_controller.php:1524. Both use the same cache key (DeviceDetector-{VERSION}regexes-bot), so whichever fires first in a request cycle — session boot or the front controller's device parse — pre-warms the entry for the other. On cache failure, a WARNING is logged via log_message() and null is returned; isBotSession() then runs without a cache, costing ~1.4 s per request (the YAML is compiled at runtime; this cost is by design for the degraded path).

Route Exclusion (`isRouteExcluded()`)

isRouteExcluded() (lines 71–87) compares the current CI3 router's resolved class and method against the exclusion list:

php

// application/libraries/Session/MY_Session.php:71-87
protected function isRouteExcluded(): bool
{
    [$targetClass, $targetMethod] = $this->normalizeRouteElements(
        [get_instance()->router->class, get_instance()->router->method]
    );

    return array_reduce(
        array_map(function ($exclude) use ($targetClass, $targetMethod) {
            [$excludeClass, $excludeMethod] = $this->normalizeRouteElements($exclude);
            return ($excludeClass === $targetClass && $excludeMethod === $targetMethod);
        }, $this->excludes),
        function ($isExcluded, $item) {
            return $isExcluded || $item;
        },
        false
    );
}

normalizeRouteElements() (lines 89–94) lowercases and trims slashes and backslashes from both sides of each element before comparison. This means matching is case-insensitive and tolerant of leading/trailing path separators. (MY_Session.php:89-94)

Route exclusion config

application/config/session_excludes.php defines the excluded pairs at the time of writing:

php

// application/config/session_excludes.php:3-15
$config['session_excludes'] = [
    [Api_view_metrics_recorder::class, 'index'],
    [Healthz::class, 'live'],
    [Healthz::class, 'ready']
];

Three routes are excluded: the view-metrics recorder endpoint and both health-check probes (/healthz/live, /healthz/ready). Health-check probes are called by Kubernetes liveness/readiness checks at high frequency; creating sessions for them would flood the session store.

Data Model

No data is written for bot or excluded-route requests — that is the point of this feature. For non-bot, non-excluded requests the session lifecycle follows the standard CI3 session driver (typically the database driver writing to ci_sessions).

Domain Layer

This feature lives entirely in the legacy CI3 layer. There are no src/Domains/ or src/Rest/ components involved.

Configuration

Config key	File	Description
`session_excludes`	`application/config/session_excludes.php`	Array of `[ClassName, 'method']` pairs. Each pair bypasses session creation for matching routes.

No environment variables govern bot detection. The bots.yml regex set is compiled at runtime on the first request per pod, then stored cross-request in the shared CachePool (the same pool used by Adv_front_controller::getUserDeviceInformation()). Subsequent requests read the compiled regex set from the cache, avoiding the ~1.4 s YAML compilation cost. If the cache pool is unavailable, botDetectorCache() returns null, the Bot parser uses only its own per-process static cache (StaticCache), and each cold process restart costs ~1.4 s on the first bot-detection call. (MY_Session.php:56-69)

Client Extension Points

Client repos that need additional sessionless routes (e.g. a custom API endpoint polled by a monitoring system) should add entries to application/config/session_excludes.php in the client's application/ overlay. The format is the same array of [ClassName, 'method'] pairs.

There is no hook or override mechanism for isBotSession() itself. To swap the detection library a client would need to override application/libraries/Session/MY_Session.php.

Business Rules

Empty User-Agent equals bot. A request with no HTTP_USER_AGENT header (or an empty string) is unconditionally treated as a bot and receives no session. (MY_Session.php:25-27)
Standalone bot parser only. \DeviceDetector\Parser\Bot loads only bots.yml and stops at the combined pre-match. The full DeviceDetector class (which also walks the OS/client/device tree) is not used. This is the primary source of the warm-path performance improvement. (MY_Session.php:33)
discardDetails() is called before parse(). The bot parser only records whether a match was found, not the bot's name, category, or URL. This reduces per-request memory usage and parse time. (MY_Session.php:35)
Cache wiring is best-effort. botDetectorCache() catches any \Throwable from the DI container or cache construction, logs a WARNING, and returns null. isBotSession() then proceeds without a cache. The session subsystem will not fail due to an unavailable cache pool. (MY_Session.php:56-69)
Fail open on detection error. If the bot parser throws any \Throwable, the error is logged and the request is treated as a non-bot — a session is created. This prioritises user experience over infrastructure protection. (MY_Session.php:43-46)
Route exclusion is evaluated after bot detection but is independent of it. A bot hitting an excluded route still returns early, but so does a real user hitting an excluded route. Both guards are ||-combined before parent::__construct() is called. (MY_Session.php:14)
Route matching is case-insensitive. normalizeRouteElements() lowercases and trims separators before comparing class and method names. (MY_Session.php:89-94)
Exclusion is exact (class + method pair). A controller class with multiple methods must have each sessionless method listed separately. There is no wildcard or prefix matching.

Known Issues & Security Gaps

[OPEN — follow-up, highest priority] Adv_front_controller::shouldLoadAdvisableAI() still calls $this->agent->is_robot() (CI3's legacy ~88-entry list) at ecommercen/core/Adv_front_controller.php:321. Modern AI crawlers (GPTBot, ClaudeBot, PerplexityBot, etc.) are not in that list, so the AdviseableAI feature loads and fires HTTP calls on every such crawler page view. This is the highest financial-impact follow-up cited in the body of commit 03226835d.
[OPEN — follow-up] AdvApiCartController::destroy() checks $this->agent->is_robot() at ecommercen/api/controllers/AdvApiCartController.php:275 before denying cart-destroy calls. Modern AI crawlers that bypass the legacy list can trigger the cart-destroy path. Cited as a follow-up candidate in commit 03226835d.
[FIXED — #285, commit c6211d80cd] Cross-request bot-detector caching was absent after commit cbefa221d removed a broken PSR-6 wiring attempt. c6211d80cd re-added a working botDetectorCache() method using PSR6Bridge(di()->get(CachePool::class)). The bots-regex set is now cached cross-request, reducing warm-path parse time from ~3325 ms to ~24 ms for browser UAs. Residual: if the cache pool is unavailable the uncached fallback costs ~1.4 s per request (by design — fail-open); this path is logged as a WARNING. (MY_Session.php:56-69)

Tests

tests/Legacy/Session/MySessionBotDetectionTest.php covers isBotSession() via reflection (ReflectionClass::newInstanceWithoutConstructor() + ReflectionMethod::invoke()). MY_Session is reflected without constructing it to avoid booting real session machinery. The test suite covers eight scenarios:

Test method	Scenario
`test_empty_user_agent_string_returns_true`	`HTTP_USER_AGENT = ''` → `true` (short-circuit before parser)
`test_absent_user_agent_header_returns_true`	`HTTP_USER_AGENT` key absent from `$_SERVER` → `true`
`test_googlebot_user_agent_returns_true`	Googlebot UA → `true` (uncached path — `CachePool` not wired in test container)
`test_claudebot_user_agent_returns_true`	ClaudeBot UA → `true` (uncached path)
`test_chrome_browser_user_agent_returns_false`	Chrome browser UA → `false` (also exercises uncached degradation path implicitly)
`test_cache_unavailable_fallback_does_not_throw_and_returns_false`	`di()->get(CachePool::class)` throws `ServiceNotFoundException`; `botDetectorCache()` must swallow it, log a WARNING, return `null`; result still `false` for browser UA
`test_cached_path_returns_correct_results_and_populates_pool`	Anonymous subclass overrides `botDetectorCache()` with in-memory `PSR6Bridge`; browser → `false`, Googlebot → `true`; asserts the `DeviceDetector-{VERSION}regexes-bot` key is written to the pool
`test_warm_cache_call_completes_faster_than_cold_parse`	Two anonymous-subclass instances sharing a pool; asserts warm call is strictly faster than cold and completes within 500 ms

The test container does not register CachePool, so tests 1–5 implicitly exercise the uncached degradation path (botDetectorCache() catches ServiceNotFoundException, returns null). Tests 6–8 (cached path) use anonymous subclasses that override botDetectorCache() directly with an InMemoryAdapter-backed pool.

SY-32 Logging — log_message('error', ...) in the catch block routes through the PSR-3 logging architecture described there
CF-10 Customer Auth — session state is a prerequisite for customer authentication; bots correctly receive no session and therefore cannot acquire an auth session

SY-33: Bot Detection and Session Exclusion ​

Business Context ​

Architecture Overview ​

Key Files ​

Code Flow ​

Session entry point ​

Bot Detection (isBotSession()) ​

Cache wiring (botDetectorCache()) ​

Route Exclusion (isRouteExcluded()) ​

Route exclusion config ​

Data Model ​

Domain Layer ​

Configuration ​

Client Extension Points ​

Business Rules ​

Known Issues & Security Gaps ​

Tests ​

Related Flows ​