Background: Where Temenos Quantum Shines—and Where It Bites
Platform Overview
Temenos Quantum (formerly Kony) couples a cross-platform UI runtime with a powerful middleware layer (Temenos Fabric). Teams design forms and controllers in the Visualizer, package for iOS/Android with native tooling, and orchestrate data, identity, analytics, and offline through Fabric. This separation is productive, yet the boundaries between client, middleware, and back-end adapters are where misconfigurations multiply.
Why Troubleshooting Is Hard at Enterprise Scale
- Multiple environments (dev, SIT, UAT, prod) with different Fabric configuration JSON, identity providers, and app keys.
- Hybrid codebase: platform widgets plus custom native modules across iOS/Android, each with their own dependency graphs and signing pipelines.
- Offline-first: sync rules, conflict handlers, and schema migrations must converge across teams and release trains.
- Security overlays: device attestation, jailbreak/root detection, and keychain/Keystore use intersect with offline storage and SSO.
Architecture: Mental Model for Systematic Diagnosis
Four Planes of Responsibility
- Client UI Plane: Visualizer forms, widgets, skins, navigation, and accessibility.
- Client Logic Plane: JavaScript controllers, action sequences, custom modules, and native bridges.
- Sync & Data Plane: Offline store, object services, integration services, identity, conflict resolution, transformers.
- Delivery Plane: Packager, signing, native SDK versions, CI/CD, app versioning, feature flags.
Map every incident to one plane first, then follow cross-plane edges: e.g., a UI freeze can be a logic-plane promise deadlock triggered by a sync-plane pagination bug after a delivery-plane downgrade of Android Gradle Plugin.
Configuration Sources of Truth
- appConfig.json and environment-specific Fabric endpoints.
- Object Service descriptors and sync rules.
- Identity provider configuration (OIDC/SAML) and token lifetimes.
- Native packager settings (iOS bundle id, provisioning profile; Android package, signing config).
Common Failure Patterns (Symptoms → Likely Planes)
1) "Works on Wi-Fi, fails on 4G" API calls
Likely: Sync/Data plane & network transport. Misconfigured TLS ciphers, certificate pinning drift, or MTU issues when large payloads meet mobile carrier proxies. Often appears after certificate rotation or Fabric gateway WAF changes.
2) Offline store diverges after upgrade
Likely: Sync/Data plane. Schema migration ordering, missing default values, or conflict handler skipping a specific status code path. Usually surfaces during blue/green or phased mobile rollout.
3) iOS archive fails after Xcode upgrade
Likely: Delivery plane. Packager templates lag behind Xcode/Swift changes, or entitlements mismatch (Keychain groups, associated domains). Reproducible only in CI due to different provisioning profile capabilities.
4) Android ANRs on older devices
Likely: Client Logic plane. Synchronous JSON transforms inside UI thread, large image decoding without sampling, or WebView bridge marshalling floods.
5) Intermittent 401/403 after idle
Likely: Identity configuration. Access token expiry shorter than offline sync window, refresh token audience mismatch, or clock skew between device and identity provider.
Diagnostics: A Repeatable Playbook
Step 1: Capture the Minimal Failure Path
On the device, turn on verbose logging for Quantum runtime, network layer, and Fabric adapters. Ensure each log line includes a correlation id propagated from the client to Fabric and back-end.
// Pseudocode: set log levels centrally
function configureLogging() {
kony.print("[BOOT] enabling verbose logs");
kony.logger.activatePersistor();
kony.logger.setLevel("TRACE");
}
function withCorrId(headers) {
var id = generateCorrId();
headers["X-Correlation-Id"] = id;
return { id: id, headers: headers };
}
Step 2: Environment Parity Matrix
Create a machine-readable manifest describing each environment’s key versions, endpoints, and security toggles. Diff manifests on every deployment to catch silent drifts.
{
"env": "UAT",
"fabricUrl": "https://uat-fabric.company.com",
"objectServicesVersion": "7.4.2",
"identity": { "provider": "OIDC", "tokenTTL": 3600 },
"mobile": {
"ios": { "xcode": "15.4", "swift": "5.10", "minOS": "13.0" },
"android": { "agp": "8.4.1", "kotlin": "1.9.24", "minSdk": 24 }
},
"security": { "pinning": true, "jailbreakDetection": true }
}
Step 3: Binary Repro in CI
Archive the exact built artifacts (IPA/AAB), Fabric adapter configs, and the Visualizer project export for any failing build. Keep a 30-day rolling cache to reproduce issues that only surface after dependent service changes.
Step 4: Thundering Herd Tests
Use synthetic users to hammer problematic flows with poor network, battery saver mode, and background/foreground transitions. Record heap snapshots and network traces. Look for steady-state drifts after 20–30 minutes, not just initial runs.
Root Causes & How to Recognize Them Quickly
TLS & Certificate Pinning Drift
Symptom: Random SSL failures on mobile data only, or after a backend maintenance window. Signal: Failure aligns with certificate rotation or intermediate CA change; Wi-Fi corporate proxy hides it. Fix vector: Use pinning by public key hash, track upcoming rotations, and maintain a dual-pinned grace period.
Offline Schema Mismatch
Symptom: Offline updates succeed locally but vanish after sync; conflicts spike post-upgrade. Signal: Migration scripts skipped default value backfills; object service metadata changed without version bump. Fix vector: Version your object schemas and enforce client-side preflight migrations with linearized order.
Token Refresh Race
Symptom: 401 after idle, immediately followed by 200 upon retry; analytics show bursts aligned with device wake cycles. Signal: Multiple in-flight requests try to refresh simultaneously, invalidating each other. Fix vector: Single-flight token refresh gate and retry with backoff.
Packaging Entitlement Mismatch
Symptom: iOS builds run locally, fail in CI with "Provisioning profile doesn’t include keychain group". Signal: CI uses a different provisioning profile or missing capability. Fix vector: Codify signing assets, validate entitlements before archive.
Large Payload Backpressure
Symptom: Android freezes during "export to PDF and upload". Signal: UI thread is blocked by Base64 encoding and JSON serialization. Fix vector: Stream to disk, chunk uploads, move transforms to worker threads.
Step-by-Step Fixes (Battle-Tested)
1) Stabilize Identity & Token Management
Implement a single-flight refresh gate, clock skew tolerance, and durable token storage. Ensure Fabric and the identity provider agree on audiences and scopes.
// Token manager with single-flight refresh
var TokenManager = (function () {
var access = null, refresh = null, expiresAt = 0;
var inflight = null; // Promise gate
function now() { return Math.floor(Date.now() / 1000); }
function isExpiringSoon() { return (expiresAt - now()) < 60; }
function refreshOnce() {
if (inflight) return inflight;
inflight = new Promise(function (resolve, reject) {
kony.sdk.getCurrentInstance().getIdentityService("OIDC").refreshToken(function (res) {
access = res.access_token;
refresh = res.refresh_token;
expiresAt = now() + res.expires_in;
inflight = null;
resolve(access);
}, function (err) { inflight = null; reject(err); });
});
return inflight;
}
async function withToken() {
if (!access || isExpiringSoon()) { await refreshOnce(); }
return access;
}
return { withToken: withToken };
})();
async function callApi(path, body) {
var t = await TokenManager.withToken();
var headers = { "Authorization": "Bearer " + t, "Content-Type": "application/json" };
return kony.net.invokeServiceAsync(path, body, headers);
}
2) Make Offline Schema Migrations Boring
Enforce versioned migrations and linearize them with explicit preconditions. Never rely on implicit widget defaults to populate required fields.
// Example migration registry
var Migrations = [
{ id: 1, run: function (db) { db.execSQL("ALTER TABLE Orders ADD COLUMN status TEXT DEFAULT 'PENDING'"); } },
{ id: 2, run: function (db) { db.execSQL("CREATE INDEX IF NOT EXISTS idx_orders_ts ON Orders(updatedAt)"); } }
];
function migrate(db, currentVersion) {
for (var i = 0; i < Migrations.length; i++) {
var m = Migrations[i];
if (m.id > currentVersion) {
kony.print("[MIGRATE] " + m.id);
m.run(db);
currentVersion = m.id;
}
}
return currentVersion;
}
3) Certificate Pinning Without Pager Fatigue
Pin to SPKI hashes and maintain a dual-window during rotations. Automate a runtime pin set fetch signed by a long-lived root to avoid forced app updates for emergency cert changes.
// Pseudocode for dual-pin verification
var pinSet = { primary: "sha256/Abc...", backup: "sha256/Xyz..." };
function verifyPin(chain) {
var spki = extractSpki(chain[0]);
return spki === pinSet.primary || spki === pinSet.backup;
}
4) Fix Android Freezes on Heavy Transforms
Offload encoding/decoding to a background thread and stream uploads. Avoid base64-in-JSON for large files; use multipart and chunking.
// Pseudocode: chunked upload
async function uploadFile(path) {
var stream = openFileStream(path);
var chunk;
while ((chunk = stream.read(256 * 1024)) != null) {
await kony.net.invokeServiceAsyncMultipart("/upload", { part: chunk });
}
}
5) Make iOS Signing Deterministic
Codify bundle id, provisioning profile UUID, and entitlements in CI variables; validate before archive. Fail fast if the provisioning profile does not include required capabilities.
# Fastlane snippet match(type: "appstore", app_identifier: "com.company.app") update_project_provisioning(xcodeproj: "App.xcodeproj", target_filter: "App", profile: ENV["PROVISIONING_PROFILE_SPECIFIER"]) # Validate entitlements /usr/libexec/PlistBuddy -c "Print :com.apple.developer.associated-domains" "App/App.entitlements"
Pitfalls That Bite Senior Teams
Hidden Environment Drift
Fabric adapter JSON or object service versions differ by one environment; defects only appear at scale. Solution: declarative configs, drift detection, and promotion pipelines that move immutable artifacts, not rebuilt ones.
Overloaded Action Sequences
Visual flows hide complexity; a long chain that mixes network calls, transforms, and UI updates becomes impossible to test. Break into controller functions, centralize error handling, and reserve action sequences for orchestration only.
Global State in Controllers
Singleton-like modules keep transient state (filters, pagination cursors) and get corrupted across navigation or backgrounding. Push state into view models or pass explicitly per screen.
Inconsistent Error Taxonomy
Mixing Fabric errors (integration, identity) with controller exceptions produces unactionable logs. Establish a canonical error envelope with codes, categories, and correlation ids.
Operational Excellence: Observability & Guardrails
Golden Signals for Mobile + Fabric
- Client: Crash-free sessions, ANR rate, p95 input latency, cold start time, offline backlog size, foreground/background sync success.
- Fabric: p95/p99 per adapter, error ratio, auth failures, queue depth, downstream timeouts, cache hit ratio.
- Identity: token refresh success rate, refresh latency, SSO reauth prompts per user per week.
Unified Error Envelope
{
"corrId": "6c2c-...",
"when": 1723492331,
"plane": "SYNC",
"code": "FABRIC_TIMEOUT",
"http": 504,
"detail": "Adapter OrderService timed out after 25s"
}
Client-Side Circuit Breakers
Avoid cascading failures by short-circuiting known-bad backends and queueing offline work. Expose breaker state to a diagnostics screen for support staff.
// Simple circuit breaker
function CircuitBreaker(name) {
var openUntil = 0; var failures = 0;
this.exec = async function (fn) {
var now = Date.now();
if (now < openUntil) throw new Error("OPEN");
try {
var res = await fn(); failures = 0; return res;
} catch (e) {
failures++; if (failures >= 3) openUntil = now + 30000; throw e;
}
};
}
var orderSvcBreaker = new CircuitBreaker("OrderService");
Performance Engineering: Patterns That Scale
Defer, Stream, Cache
- Lazy-load heavy screens; prefetch critical lightweight data during splash.
- Stream binary uploads; avoid base64 inflation.
- Cache immutable reference data in the offline store with TTLs.
Batch & Compress
Group small writes into a single sync transaction. Enable gzip/deflate where intermediaries allow it; verify WAFs and Fabric honor content encoding and do not double-compress.
Telemetry-Driven Budgets
Define budgets: cold start < 2s (p90), first interactive < 3.5s, p95 API < 600ms, average daily data usage < 25MB. Tie budgets to CI gates using synthetic runs on representative devices.
Security & Compliance Considerations
Key Storage & Device State
Use OS keystores; avoid storing secrets in Quantum preferences. If jailbreak/root detection is enabled, align behaviors with offline-first: permit read-only offline access with masked PII until re-authenticated.
PII & Analytics
Never log raw tokens or PII. Implement a privacy scrubber that redacts fields before logs are persisted or uploaded. Test redaction with fuzzed payloads.
Architecting for Upgrades Without Fear
Establish a Compatibility Matrix
Pin and test combinations of Visualizer, Fabric server, Android/iOS SDKs, Xcode/AGP, and third-party native modules. Publish a "known good" set per release train.
Shadow Deployments
Before rolling a new Fabric adapter, mirror a small percentage of traffic and compare responses, latency, and error taxonomy. Promote only if regressions stay below threshold for a full business cycle.
End-to-End Example: From Incident to Permanent Fix
Incident
After a Fabric rollout, field agents report intermittent "Unable to sync orders" on 4G. Crash-free rate is unaffected, but support sees spikes in 401 and 0-byte uploads.
Diagnosis
- Correlate by id: 92% of failures share a cert chain with a new intermediate CA.
- Network traces show TLS handshake failures on carrier networks only.
- Client logs reveal pinning to leaf certificate fingerprint.
- Parallel: token refresh storms coincide with connectivity resumption.
Fix
- Switch to SPKI pinning with dual pins and a signed runtime pin set.
- Add a single-flight token refresh gate with jittered retries.
- Ship a canary to 10% of devices, monitor p95 handshake success and 401 ratio.
- Postmortem: add rotation runbook and alert 30 days before cert expiry.
Testing Strategies That Actually Catch Regressions
Contract Tests for Object Services
Define JSON schemas for each object and write contract tests that run against Fabric UAT on every PR. Fail on additional required fields or enum changes without version bumps.
Soak Tests on Real Devices
Run 60-minute scripted sessions on two low-end Androids and one mid-range iPhone, cycling through foreground/background, toggling connectivity, and performing offline/online transitions. Trigger a failure if memory exceeds a budget curve.
CI Artifacts as Evidence
- Store offline DB snapshots before and after migration.
- Attach Fabric adapter JSON and identity config digests.
- Record app "About" screen metadata (build number, SDKs, environment).
Playbooks (Copy/Paste Ready)
Temenos Fabric Timeout Playbook
IF p99 latency > SLO AND errors spike CHECK downstream health (DB, core) INCREASE client timeout only if downstream OK ENABLE circuit breaker for failing adapter ADD cache for idempotent GETs SCALE adapter pods and set connection pools REVIEW WAF logs for blockage
Offline Conflict Playbook
IF conflicts > 3% on release week CHECK schema versions across cohorts ENSURE migration ran (look for new columns defaulted) ENABLE server-last-write for non-critical fields SEND targeted re-sync for corrupted partitions ADD reconciliation job to flag orphans
Identity Drift Playbook
IF 401 after idle increases VERIFY token TTL and refresh window ADD single-flight refresh ADD 2-min skew tolerance AUDIT scopes/audiences SOAK test sleep/wake cycles
Governance: Make Good Habits Easy
Checklists Embedded in CI
- Fail build if environment manifests differ (hash mismatch).
- Static scan for synchronous heavy work on UI thread patterns.
- Entitlement validation step before iOS archive.
- Schema migration presence for any object change.
Runbooks and SLOs
Define SLOs (e.g., sync success ≥ 99.5%, crash-free ≥ 99.8%, 401 rate < 0.5%). Link every SLO breach to a runbook with "Verify", "Mitigate", "Fix" steps and owners.
Conclusion
Temenos Quantum accelerates delivery, but its real power emerges when teams treat it as a layered system with clear contracts across UI, logic, sync, and delivery planes. Enterprise outages usually trace to configuration drift, identity races, brittle offline migrations, or package signing entropy—not "mystery bugs". Standardize manifests, codify migrations, pin and rotate certificates sanely, and gate token refreshes. Invest in synthetic soak tests, unified error envelopes, and deterministic packaging. With these guardrails, you can compress MTTR, stabilize release trains, and scale confidently across lines of business.
FAQs
1. How do we prevent offline store corruption during phased rollouts?
Version schemas and include forward-compatible defaults; linearize migrations with preconditions. Gate rollouts with a canary cohort and compare conflict rates before promoting.
2. What’s the safest approach to TLS pinning in Quantum apps?
Pin SPKI hashes with a dual-pin window and a signed runtime pin set so you can rotate without forcing app updates. Track expiries and test on cellular networks where proxies can alter behavior.
3. Why do we see 401s after devices wake from sleep?
Concurrent refresh attempts race and invalidate tokens, especially with short TTLs. Implement a single-flight gate, add clock-skew tolerance, and retry idempotently with jitter.
4. How can we make iOS/Android packaging deterministic in CI?
Codify signing assets, validate entitlements/capabilities, and fix SDK versions in a compatibility matrix. Archive the exact Visualizer export and Fabric configs used to build each artifact.
5. What metrics predict user-visible degradation before incidents?
Watch p95 sync latency, 401 ratio, offline backlog size, ANR rate, and crash-free sessions. Alert on trends over 30–60 minutes, not just spikes, and correlate with environment manifest diffs.