apiVersion: jenkins.io/v1 kind: PipelineActivity metadata: annotations: lighthouse.jenkins-x.io/cloneURI: https://github.com/greencapitaltrade/deadpool.git lighthouse.jenkins-x.io/job: release pipeline.jenkins-x.io/traceID: 740d258d4c868e3a561514685eab2507 creationTimestamp: "2026-07-27T16:45:24Z" generation: 8 labels: branch: main build: "89" context: release created-by-lighthouse: "true" event-GUID: 8e1aacb2-89da-11f1-802c-69898093d08f lighthouse.jenkins-x.io/baseSHA: 18d80ae28d6bddcd79a05bd5eb7d25ba485b5ec9 lighthouse.jenkins-x.io/branch: main lighthouse.jenkins-x.io/buildNum: "1785170724324" lighthouse.jenkins-x.io/context: release lighthouse.jenkins-x.io/id: pitaltrade-deadpool-main-release-wrb2n lighthouse.jenkins-x.io/job: release lighthouse.jenkins-x.io/lastCommitSHA: 18d80ae28d6bddcd79a05bd5eb7d25ba485b5ec9 lighthouse.jenkins-x.io/refs.org: greencapitaltrade lighthouse.jenkins-x.io/refs.repo: deadpool lighthouse.jenkins-x.io/type: postsubmit owner: greencapitaltrade podName: pitaltrade-deadpool-main-release-qm6kt-from-build-pack-pod provider: github repository: deadpool tekton.dev/pipeline: pitaltrade-deadpool-main-release managedFields: - apiVersion: jenkins.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:labels: f:branch: {} f:context: {} f:owner: {} f:provider: {} f:repository: {} f:spec: f:lastCommitMessage: {} f:lastCommitSHA: {} f:releaseNotesURL: {} f:version: {} manager: jx-changelog-0.10.22 operation: Update time: "2026-07-27T16:53:48Z" - apiVersion: jenkins.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:lighthouse.jenkins-x.io/cloneURI: {} f:lighthouse.jenkins-x.io/job: {} f:pipeline.jenkins-x.io/traceID: {} f:labels: .: {} f:build: {} f:created-by-lighthouse: {} f:event-GUID: {} f:lighthouse.jenkins-x.io/baseSHA: {} f:lighthouse.jenkins-x.io/branch: {} f:lighthouse.jenkins-x.io/buildNum: {} f:lighthouse.jenkins-x.io/context: {} f:lighthouse.jenkins-x.io/id: {} f:lighthouse.jenkins-x.io/job: {} f:lighthouse.jenkins-x.io/lastCommitSHA: {} f:lighthouse.jenkins-x.io/refs.org: {} f:lighthouse.jenkins-x.io/refs.repo: {} f:lighthouse.jenkins-x.io/type: {} f:podName: {} f:tekton.dev/pipeline: {} f:spec: .: {} f:baseSHA: {} f:batchPipelineActivity: {} f:build: {} f:buildLogsUrl: {} f:completedTimestamp: {} f:context: {} f:gitBranch: {} f:gitOwner: {} f:gitRepository: {} f:gitUrl: {} f:message: {} f:pipeline: {} f:startedTimestamp: {} f:status: {} f:steps: {} manager: jx-build-controller operation: Update time: "2026-07-27T16:55:00Z" name: greencapitaltrade-deadpool-main-89 namespace: jx resourceVersion: "221256606" uid: 4f254c2d-452f-4d5a-8bf0-a0c8addcd441 spec: baseSHA: 18d80ae28d6bddcd79a05bd5eb7d25ba485b5ec9 batchPipelineActivity: {} build: "89" buildLogsUrl: s3://logs-gct-prod-20260425045301534100000007/jenkins-x/logs/greencapitaltrade/deadpool/main/89.log completedTimestamp: "2026-07-27T16:54:51Z" context: release gitBranch: main gitOwner: greencapitaltrade gitRepository: deadpool gitUrl: https://github.com/greencapitaltrade/deadpool.git lastCommitMessage: | fix(tcp-server): run blocking BRPOP on a dedicated Redis connection Command routing is dead in production: the command-queue reader loops on "Redis error reading command queue: timed out" (843 timeouts in 5 min on one gt06 pod), so no command reaches any device. Verified end-to-end — an engine_restore sent to a live device sat in Kafka, and when finally consumed the TCP server could not dispatch it. Cause: `process_commands` issued a *blocking* `BRPOP 5` on the SHARED `ConnectionManager` returned by `DeviceRegistry::get_connection` (a clone of the same manager the registry hot path uses). redis-rs multiplexes every command over one socket, and Redis serves a connection in order, so a parked BRPOP head-of-line-blocks everything behind it. The reader re-issues BRPOP immediately after each 5s expiry, so that socket is blocked ~100% of the time and `is_verified` / `refresh_device` / `register_device` / `unregister_device` all queue behind it. This also explains the long-standing "app-side" latency: Tempo showed redis_exists taking 2-3s while redis-cli from a debug pod answered in ~0ms — a blocking command stalls only its own client connection, not the server, so an independent client always looked healthy. The redis 1.x upgrade added a response timeout, turning that stall into hard failures. Fix: `dedicated_connection()` opens a NEW connection for the blocking reader; the shared manager is never used for blocking commands. BRPOP stays — it is the right primitive for the per-instance command mailbox (no polling); only the socket it ran on was wrong. Regression test asserts a registry op still returns in <1s while a BRPOP is parked. Co-Authored-By: Claude Opus 4.8 (1M context) lastCommitSHA: f761ea000bca3ef0f5b28d9ca2d39bb77d968191 message: 'Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0' pipeline: greencapitaltrade/deadpool/main releaseNotesURL: https://github.com/greencapitaltrade/deadpool/releases/tag/v0.60.3 startedTimestamp: "2026-07-27T16:45:24Z" status: Succeeded steps: - kind: Stage stage: completedTimestamp: "2026-07-27T16:54:51Z" name: from build pack startedTimestamp: "2026-07-27T16:45:48Z" status: Succeeded steps: - completedTimestamp: "2026-07-27T16:45:49Z" name: Git Clone startedTimestamp: "2026-07-27T16:45:48Z" status: Succeeded - completedTimestamp: "2026-07-27T16:45:52Z" name: Next Version startedTimestamp: "2026-07-27T16:45:50Z" status: Succeeded - completedTimestamp: "2026-07-27T16:46:02Z" name: Jx Variables startedTimestamp: "2026-07-27T16:45:59Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:27Z" name: Build Cargo Install startedTimestamp: "2026-07-27T16:46:18Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:30Z" name: Check Registry startedTimestamp: "2026-07-27T16:53:27Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:32Z" name: Build Server Image startedTimestamp: "2026-07-27T16:53:30Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:34Z" name: Build Consumer Image startedTimestamp: "2026-07-27T16:53:32Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:36Z" name: Build Mqtt Image startedTimestamp: "2026-07-27T16:53:34Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:48Z" name: Promote Changelog startedTimestamp: "2026-07-27T16:53:36Z" status: Succeeded - completedTimestamp: "2026-07-27T16:53:53Z" name: Promote Helm Release startedTimestamp: "2026-07-27T16:53:49Z" status: Succeeded - completedTimestamp: "2026-07-27T16:54:35Z" name: Promote Jx Promote startedTimestamp: "2026-07-27T16:53:53Z" status: Succeeded - completedTimestamp: "2026-07-27T16:54:51Z" name: Publish Org Kb startedTimestamp: "2026-07-27T16:54:35Z" status: Succeeded - kind: Promote promote: environment: staging pullRequest: pullRequestURL: https://github.com/greencapitaltrade/mcu/pull/4922 startedTimestamp: "2026-07-27T16:54:15Z" status: Succeeded startedTimestamp: "2026-07-27T16:54:15Z" status: Succeeded - kind: Promote promote: environment: production pullRequest: pullRequestURL: https://github.com/greencapitaltrade/mcu/pull/4923 startedTimestamp: "2026-07-27T16:54:32Z" status: Succeeded startedTimestamp: "2026-07-27T16:54:32Z" status: Succeeded version: 0.60.3 status: {}