Client-Side and Mobile Data Persistence Questions

High-level goal
Minimize device wakeups, avoid large transfers on cellular/low battery, and adapt to real conditions.

Scheduling & APIs

• Use platform schedulers: Android WorkManager/JobScheduler and iOS BackgroundTasks/NSURLSession backgroundUpload. Let OS coalesce jobs and respect Doze/Low Power Mode.
• Prefer scheduled windows (flexible windows) vs strict periodic cron-like tasks.

Batching & Large Uploads

• Aggregate small changes into single payloads; compress and delta-sync.
• For large media, upload when on Wi‑Fi and charging; use background transfer APIs that survive app kill.

Opportunistic sync

• Detect network type and battery state; defer noncritical work until Wi‑Fi or charging.
• Use connectivity callbacks and adapt priority (low-priority when on cellular).

Backoff & retry

• Exponential backoff with jitter to avoid thundering herd:
  • backoff = base * 2^attempt +/- random_jitter
• Cap retries and fail gracefully, queue for next opportunistic window.

Push vs Polling

• Use push (silent push notifications) to trigger sync when server has changes; falls back to infrequent polling for liveness.
• Polling only when necessary and with increasing intervals if idle.

Observe & tune

• Measure: Android Battery Historian, adb shell dumpsys batterystats, Android Network Profiler, iOS Energy Gauge and Instruments (Energy, Network).
• Track metrics: wakelocks, number of network sessions, bytes over cellular, sync latency, failure rates.
• Iterate: correlate spikes with code paths, reduce frequency or increase batching, move work to OS-scheduled windows.

Example: switch from 5‑min polling to push + daily batch led to 70% fewer wakes and 60% less cellular data usage.

Practice conservative defaults, expose user controls (sync over Wi‑Fi only), and log metrics to validate changes.

Compare CRDTs vs OT (brief, mobile-focused)

• CRDTs

  • Convergence via mathematically commutative/associative/ idempotent ops or state merges; works well with unreliable mobile networks and offline-first apps because merges are safe after partitions.
  • Simpler client logic: local updates always allowed; sync uses merge/replicate; good for per-device state (counters, sets, maps).
  • Trade-offs: some semantics harder (list/position-aware editing), metadata overhead (per-replica state).

• Operational Transformation (OT)

  • Transforms concurrent operations against each other to preserve intention (used in collaborative text editors).
  • Requires operation history, central ordering or agreement protocol; more complex to implement on mobile (synchronization, transformation functions, undo/redo).
  • Trade-offs: better at preserving user intent for linear structures (text), but harder under mobile churn and partitions.

For mobile apps I prefer CRDTs for offline-first sync and simpler client code; use OT when exact edit intentions for shared rich text matter and you can maintain reliable coordination.

G-Counter (state-based) — Kotlin implementation

// G-Counter: grow-only counter per-replica; merge = pointwise max
data class GCounter(private val counts: MutableMap<String, Long> = mutableMapOf()) {

    // increment local replica by 1 (or delta)
    fun increment(replicaId: String, delta: Long = 1L) {
        require(delta >= 0)
        counts[replicaId] = (counts[replicaId] ?: 0L) + delta
    }

    // merge another replica's state into this one (in-place)
    fun merge(other: GCounter) {
        for ((id, value) in other.counts) {
            val local = counts[id] ?: 0L
            counts[id] = maxOf(local, value)
        }
    }

    // total value across replicas
    fun value(): Long = counts.values.sum()
    
    // snapshot/copy for sending over network
    fun snapshot(): Map<String, Long> = counts.toMap()
}

Explanation, complexity, edge cases

• Why merge works: each replica keeps monotonically increasing per-replica counters; taking max ensures no double-counting and commutativity/associativity/idempotence.
• Time: increment O(1); merge O(n) where n = number of replicas present in states; space O(r).
• Edge cases: replica id collisions (use UUIDs), clock resets (avoid by only incrementing local counter), deletions not supported (G-Counter is grow-only). For mobile, compress maps, garbage-collect stale replica entries, and sign/verify replica ids for security.

Plan overview (goal): rename Note.title → Note.name and add relationship owner:User with minimal user-visible risk. Prefer lightweight migration; fall back to a custom mapping if needed.

Lightweight migration steps

• Create a new Core Data model version (Editor → Add Model Version) and set it as current.
• In the new model: rename attribute title → name. Set the attribute's Renaming ID to the old name (title) (in Xcode inspector, set "Renaming Identifier").
• Add relationship owner on Note pointing to User. If owner can be nil for existing records, mark relationship optional.
• Enable automatic lightweight migration when adding the store:
  swift

  let options = [
    NSMigratePersistentStoresAutomaticallyOption: true,
    NSInferMappingModelAutomaticallyOption: true
  ]
  container.persistentStoreDescriptions.first?.setOption(true as NSNumber, forKey: NSPersistentHistoryTrackingKey)
  container.loadPersistentStores(completionHandler: { _, error in ... })
  

• Reasoning: Renaming with Renaming ID and adding an optional relationship are supported by lightweight inference. This preserves data and requires no custom code.

When lightweight is insufficient

• If owner must be populated from existing info (e.g., Note has ownerID stored externally), or relationship cardinality/transformations are complex, create a custom mapping:
  • Create an explicit Mapping Model (File → New → Mapping Model) between old and new versions.
  • Implement an NSEntityMigrationPolicy subclass for Note to set the owner relationship during migration:
    swift

    class NoteMigrationPolicy: NSEntityMigrationPolicy {
      override func createDestinationInstances(forSource sInstance: NSManagedObject, in m: NSMigrationManager) throws {
        try super.createDestinationInstances(forSource: sInstance, in: m)
        let dest = m.destinationInstances(forEntityMappingName: "NoteToNote", sourceInstances: [sInstance]).first as? NSManagedObject
        if let ownerID = sInstance.value(forKey: "ownerID") as? String {
          // lookup or create User in destination context then set relationship
        }
      }
    }
    

  • Attach the policy to the Note entity mapping in the mapping model.
  • Use NSMigrationManager to run the custom mapping when replacing the store.

Testing strategy (across live data / versions)

• Automated migration tests:
  • Keep an archive of old model .momd versions. Programmatically load an old store seeded with representative records and run migration code to the new model; assert renamed attribute values and owner relationships.
• Manual / integration testing on devices:
  • Build an app binary with the old model, seed devices via TestFlight or local install with realistic data (create notes, set owner scenarios).
  • Install new version over the old on device and run acceptance tests covering UI, CRUD, and background sync. Verify no data loss, name populated, and owner set/optional as expected.
• Edge checks:
  • Test corrupted/partially-migrated stores — ensure graceful error and clear recovery (backup & restore prompt).
  • Test on multiple iOS versions and large stores for performance.
• Rollback / telemetry:
  • Log migration success/failure with non-sensitive metrics and provide a safe rollback path (keep pre-migration backup copy of the sqlite + WAL files and surface an in-app restore option for support).

This approach minimizes user disruption, uses lightweight migration where possible, and provides a clear custom path and test plan when transformations require code.

Clarify goals & constraints

• Support upgrades from many historical app versions on Android (Room/SQLite) and iOS (Core Data/Realm).
• Minimize crash risk, preserve user data, optimize startup time.

High-level strategy

• Use strictly incrementing schema version numbers per platform.
• Ship only small, incremental migrations; never big monolithic jumps.
• Keep migrations deterministic, idempotent, and reversible where possible (backup before apply).

Migration ordering

• Always apply migrations sequentially from current DB version -> next -> ... -> target version.
• Embed migration entrypoint at open-time; fail-safe to rollback or run lightweight compatibility path.

Backward / forward compatibility

• Forward-compatible reads: add nullable columns, new tables; avoid removing columns without shadowing.
• Backward-compatible writes: feature-gate writes to new fields until migration completes and server validates.

Migration scripts

• Maintain per-version migration scripts in code (Kotlin/Swift) and as SQL for SQLite.
• Example: Room provides Migration(oldVersion, newVersion) with execSQL; Core Data uses mapping models/NSMigrationManager or lightweight migrations; Realm uses migration blocks.
• Include checksums and migration metadata table (applied_at, from_version, to_version, checksum).

Incremental migration

• Write small scripts that transform minimal state (e.g., add column, create table, backfill in batches).
• For large backfills, do lazy migration: add column, mark rows as unfilled, backfill asynchronously with background jobs.

Testing at scale

• Maintain canonical DB snapshots for representative historical versions (automated via CI).
• Create matrix: each historical snapshot -> run migrations sequentially -> assert schema, data invariants, and app flows.
• Use fuzzing, randomized data, and property-based tests for edge cases.
• Run migrations on device farm/emulators across OS versions; measure migration time and memory.
• Include automated rollback tests and corruption simulations.

Operational practices

• Ship telemetry around migration success/time/failure; auto-collect safe logs.
• Rollout migrations behind feature flags / phased rollout.
• Document each migration: intent, invariants, testing steps, rollback plan.

This approach balances safety, user-data integrity, and maintainability for mobile local stores.

Approach summary
Design uses envelope encryption: strong symmetric data-keys (AES-256-GCM) to encrypt PII, with those data-keys wrapped by hardware-backed keys in Keychain (iOS) / Keystore or StrongBox (Android). Biometric unlock gates the unwrapping operation rather than the raw key material. Key rotation, encrypted backups, and optional server-side recovery use key-wrapping and split/escrow techniques so plaintext PII never leaves device.

Encryption-at-rest

• Use AES-256-GCM for authenticated encryption (integrity + confidentiality).
• Per-record or per-user data keys (DEKs) to limit blast radius.
• Store ciphertext + associated data (version, key ID, IV, auth tag).

Key management

• Envelope pattern: DEK (symmetric) encrypted by a KEK (Key Encryption Key).
• iOS:
  • Create KEK in Keychain with kSecAttrKeyTypeEC or kSecAttrKeyTypeRSA stored in Secure Enclave.
  • Access control: kSecAttrAccessibleWhenPasscodeSetThisDeviceOnly and use SecAccessControlCreateWithFlags with .biometryAny or .userPresence.
  • Use Secure Enclave for signing/unwrap via private key; derive ephemeral symmetric via ECDH if needed.
• Android:
  • Generate KEK with KeyGenParameterSpec setUserAuthenticationRequired(true) and requireSecureHardware(true); prefer StrongBox if available.
  • setInvalidatedByBiometricEnrollment(true) to avoid stale-biometrics.
  • Use Android Keystore to wrap/unwrap DEKs via RSA/EC or use AES key with keymaster-based wrap.

Biometric unlocking

• Do not store DEK plaintext; require user auth to authorize key use.
• On iOS, set SecAccessControl with biometrics so SecKeyCreateDecryptedData requires biometric unlock.
• On Android, require user authentication on key usage (setUserAuthenticationValiditySeconds = 0 for per-use auth) and integrate BiometricPrompt to gate key operations.
• Mark keys invalidated on credential change to prevent unauthorized reuse.

Key rotation

• Rotate KEK periodically or when device compromise suspected.
• Process:
  • Generate new KEK (hardware-backed).
  • For each DEK: unwrap with old KEK (authorized operation), rewrap with new KEK, update metadata atomically.
• Prefer lazy rotation: rewrap DEKs on next access to avoid heavy offline migration.
• Keep key versioning metadata; support rollbacks for failed rotations.

Encrypted backups

• Backups must not expose KEKs or DEKs in plaintext.
• Options:
  • Device-only backups: use platform backup APIs that store Keychain/Keystore items as device-only (iOS: thisDeviceOnly prevents iCloud backup).
  • Encrypted, user-authorized cloud backup: wrap DEKs with a recovery-wrapping-key derived from user passphrase (SRP/Argon2 + salt) or an asymmetric server public key, then upload ciphertext. Keep privacy: do not upload plaintext PII.
  • Use client-side encryption before sending backups; server stores only ciphertext + metadata.

Selective server-side recovery while preserving privacy

• Use split-key/escrow:
  • Generate Recovery Key (RK) and split into parts (Shamir) or use threshold crypto where server holds a share and user holds a share (or a passphrase-derived share).
  • To recover: client authenticates (multi-factor) to server to retrieve server share; combine with user share/passphrase to reconstruct RK, then unwrap DEKs.
• Alternative: Wrap DEKs with server's public key but encrypt that wrapped blob with user-controlled passphrase so server cannot decrypt without user consent.
• Implement policy checks: require strong server-side authentication (OAuth + device attestation) and user consent; minimize metadata saved server-side.
• Use attestation (Apple DeviceCheck/Android SafetyNet/Play Integrity) to ensure recovery requests come from legit device.

Operational and security details

• Store metadata: key IDs, versions, wrapping algorithm, invalidation flags.
• Monitor and log key operations (without logging secrets).
• Implement secure wipe: delete wrapped DEKs + remove keys from Keychain/Keystore.
• Test: biometric enrollment changes, OS upgrades, backup/restore flows, rotation edge-cases.
• Threat model: detect malware/rooted device and degrade functionality (deny access, require re-authentication via server).

This design keeps plaintext PII off persistent storage and out of backups, leverages hardware-backed keys for strong protection, uses biometrics only to gate key usage, supports rotation and recovery while preserving user privacy via split-key or passphrase-protected wraps.