Mobile Build and Release Engineering Questions

Difference: Android App Bundle vs APK; IPA vs App Store distribution

• Android App Bundle (AAB): publishing artifact containing all compiled code/resources for every device/configuration. Google Play generates optimized APKs (split APKs / signed app bundles → Dynamic Delivery) per device. APK: a single installable package that may be universal or targeted; developer-built APKs can be larger and require more manual splits.
• iOS IPA vs App Store distribution: IPA is the packaged app binary (signed). For App Store distribution you upload an archive (IPA inside .xcarchive) to App Store Connect; Apple performs on-device thinning (slicing) and manages re-signing and distribution.

Benefits of bundles / app thinning

• Smaller download/install size (reduced network usage, faster installs)
• Device targeting: delivers only needed architectures, locales, screen assets, and modules
• Easier management of dynamic features (Android Dynamic Delivery) and on-demand resources (iOS)

How they affect size & targeting

• Reduce user-facing download by excluding unused ABIs, languages, and large assets
• Enable conditional delivery (Android) or on-demand resources (iOS) for feature modules

Basic steps to prepare & upload

Google Play:

1. Build AAB (./gradlew bundleRelease)
2. Ensure Play App Signing enabled, set versionCode/versionName, sign upload key
3. Test with internal test track; generate release notes
4. Upload to Play Console, configure device targeting, rollout percentage, submit

App Store Connect:

1. Archive in Xcode (Product → Archive), export for App Store distribution or use Transporter/Xcode upload
2. Increment CFBundleVersion/CFBundleShortVersionString, enable App Thinning/on-demand if used
3. Create app record, add screenshots/metadata, choose build in TestFlight for testing
4. Submit for review and release (manual or automatic)

Pipeline overview (stages)

• PR -> CI: lint, unit tests, build checks
• Merge to main -> CD: full builds, UI tests on device cloud, artifact storage, staged release
• Release gates: manual approval for production

Build matrix & parallelism

• Use GitHub Actions matrix strategy for OS/ABI/Flavors (android: API 29/31, iOS: 14/16) and variants (debug/release).
• Split heavy jobs (android build, ios build, android UI, ios UI) into parallel workflows; use job concurrency to limit device cloud slots.

Secrets & signing

• Store keystore, KEYSTORE_PASSWORD, FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD, Apple API key JSON, service account JSON in GitHub Secrets.
• Decrypt signing artifacts at runtime using actions/checkout + actions/upload-artifact for transient use; restrict secrets to protected branches and require approvals.
• For iOS, use Fastlane match (private git-backed certs) or Apple App Store Connect API keys; for Android, use Google Play service account JSON.

Testing on device clouds

• Integrate Firebase Test Lab (Android) and BrowserStack or Bitrise device cloud (iOS) via CLI; upload APK/IPA and run matrix of device+OS tests. Collect logs, screenshots, and video as artifacts.

Caching & dependency management

• Cache Gradle, .gradle/caches, ~/.m2, CocoaPods/Carthage/SwiftPM artifacts using actions/cache keyed by lockfile + OS + toolchain. Restore to speed builds.

Artifacts & storage

• Upload build outputs and test reports to GitHub artifacts or an S3 bucket. Tag artifacts with build number, git commit, branch.

Publishing

• Android: use Fastlane supply with service account to upload to internal, alpha, beta tracks; automate track promotion after approval.
• iOS: use Fastlane pilot / deliver with App Store Connect API key to upload dSYMs and IPA, submit to TestFlight, require manual review step for production release.

Observability & rollbacks

• Publish build status to Slack, include test summaries. Keep signed artifact snapshots for fast rollback and use staged rollouts (Play Store) and phased releases (App Store).

High-level tradeoffs

• Monorepo: single source for iOS/Android + core library — easier refactors, single dependency graph, atomic changes. Downsides: larger CI surface, tooling complexity.
• Multi-repo: clear ownership, smaller checkouts, simpler per-repo pipelines. Downsides: cross-repo coordination, harder refactors, version drift.

CI design goals

• Fast feedback per team
• Run only affected builds/tests
• Safe promotion of shared artifacts
• Manage breaking API changes without blocking clients

Detect affected projects

• Use path-based change detection plus dependency graph.
• Example (PR pipeline):

git fetch origin main
git diff --name-only origin/main...HEAD > changed.txt

• Map paths to modules (core/, android/app/, ios/) and traverse module deps (Gradle / CocoaPods / SPM manifests or a precomputed graph).

Minimal builds/tests

• For each affected module run:
  • Static checks + unit tests
  • For mobile: run JVM unit tests and iOS Swift unit tests
  • Only run instrumented/emulator/real-device tests for apps or when platform-specific code changed
• Use build caching (Gradle remote cache, Bazel) and incremental compilation.

Promote shared artifacts safely

• Publish core library artifacts to internal registries (Maven for Android, binary SPM/CocoaPods for iOS) with semantic versions and metadata about source commit.
• CI flow:
  • On core-only PR: run build + full test matrix, then publish to a "canary" channel (e.g., 0.x-canary) and create a release candidate.
  • App pipelines can opt-in to canary or pin stable semver.
• Automate artifact signing and provenance.

Manage breaking API changes

• Enforce semver and require major-version bump for breaking changes.
• Provide CI compatibility job that checks downstream repos/apps by running consumer build/test against the PR branch (lightweight: unit-tests + smoke app build). Use feature flags or dual-api shims so core can keep backward compatibility for one major.
• Add linter/generator that flags incompatible changes (missing symbols, changed method signatures).
• Communicate via automated migration notes and codemods in PR.

Tooling recommendations

• Monorepo: Bazel or Gradle with composite builds + remote cache
• Multi-repo: small repo + automated cross-repo test runner (CI job that checks downstream)
• Use artifact registry, semantic release, feature flags, and automated migration tooling.

This approach keeps mobile teams independent while enabling safe, auditable shared-library evolution.

Clarify requirements

• Multiple cross-platform apps in one monorepo, each with flavors (e.g., free/paid, staging/prod), environment-specific native libs, and distinct signing keys. CI must store credentials securely, allow key rotation, and run per-flavor pipelines with traceability and minimal duplication.

High-level approach

• Use flavor-aware build config (Android Gradle productFlavors, iOS xcconfigs + build schemes/targets).
• Store environment-native libs under flavor-specific dirs and expose via sourceSets (Android) / Build Phases or pods (iOS).
• Centralize signing metadata (keystore paths, provisioning profiles, key IDs) in a single build-metadata layer referenced by flavors.

Repo layout (example)

• /apps/{app}/android/{app}/src/{flavor}/...
• /apps/{app}/ios/{flavor}/...
• /ci/fastlane/{app}/Fastfile (shared lanes)
• /keys/ (not checked in; placeholders only)

Signing & native libs

• Android: use signingConfigs in Gradle that read key alias/password from environment variables; flavor -> signingConfig mapping.
• iOS: use xcconfig files per flavor with CODE_SIGN_IDENTITY and PROVISIONING_PROFILE_SPECIFIER; Fastlane match or Sigh to manage profiles/certs.
• Load native libs at build time from flavor dirs; guard with compilation flags.

Secure credential storage in CI

• Use cloud KMS + secret storage (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or GitHub Actions Secrets for smaller teams).
• Store encrypted artifacts (keystores, p12, provisioning profiles) in an artifact store accessible by CI after decryption with a short-lived key.
• CI jobs fetch secrets at runtime using IAM roles / short-lived tokens; never store plaintext in repo or logs.

Key rotation strategy

• Maintain versioned keys/certs in Vault or S3 encrypted; tag versions with metadata (created_by, expires_at).
• Automate rotation pipeline: generate new keypair -> upload encrypted -> update matching configs in staging -> run smoke-deploy -> promote and update production references.
• Use Fastlane match with git-backed certs or a secure backend and rotate by creating new certs and updating provisioning profiles; keep old keys for a rollback window.

Automating per-flavor pipelines

• Define a single templated pipeline with a flavor matrix (CI matrix builds flavors/platforms). Pipeline reads flavor metadata file (YAML) that maps:
  • flavor id -> signing secret name, native lib path, bundle id, app name
• Use reusable pipeline templates (GitHub Actions reusable workflows, GitLab includes) to avoid duplication.
• Fastlane: one Fastfile with parameterized lanes: lane(:build) { |options| build_flavor = options[:flavor]; ... }
• Cache common steps (dependency install, toolchain setup) and run flavor-specific steps in parallel.

Traceability & audit

• Produce signed build artifacts with metadata: flavor, git commit, build number, signing key version. Embed provenance in app metadata (CFBundle/Android manifest or a BUILD_INFO resource).
• Store build artifacts and logs in an immutable artifact repository (S3, Nexus) and log secret access via Vault audit logs and cloud IAM audit trails.

Example: CI flow (Android, flavor "staging")

1. CI matrix picks flavor=staging.
2. CI authenticates to Vault using workload identity.
3. Fetch encrypted keystore and decrypt to ephemeral path.
4. Run ./gradlew assembleStagingRelease using signingConfigs that read keystore path and env vars.
5. Run Fastlane supply or pilot deployment using short-lived API tokens.
6. Publish artifact with metadata and rotate/delete ephemeral secrets.

Trade-offs / best practices

• Favor centralized secret management and short-lived credentials.
• Keep pipelines declarative and parameterized to reduce duplication.
• Test rotation on non-prod first; keep rollback key material for a defined window.
• Log and monitor secret access; restrict who can update flavor metadata.

This approach balances security, auditability, and maintainability across many apps/flavors in a monorepo while enabling automated, traceable CI/CD.

Brief stance
OTA JS/bytecode updates (e.g., CodePush for React Native) speed bug fixes and UX tweaks but carry policy, security, compatibility and legal risks. Use them selectively and with strict controls.

Pros

• Fast bug fixes and A/B experiments without store resubmits
• Reduced user friction; faster mean time to fix
• Smaller payloads than full binary updates

Cons / Risks

• App store constraints: Apple allows JS/content updates but forbids changing app’s primary purpose or delivering new native code/feature sets that bypass review; Google’s policies similar — risk of rejection or removal if abused.
• Security/integrity: remote code delivery opens supply-chain attack surface.
• Compatibility: native API or ABI changes require binary updates; JS patches can break if native bridge signatures change.
• Debuggability & support complexity; increased testing matrix.

Safe implementation plan

• Policy & legal:
  • Document OTA usage in TOS and privacy policy.
  • Limit OTA to non-native-feature changes; keep features requiring native modules to store updates.
  • Maintain audit logs of released bundles and rollbacks for compliance.
• Verification & delivery:
  • Serve bundles over TLS with HSTS.
  • Sign bundles server-side with asymmetric keys; verify signature before applying.
  • Validate bundle schema and runtime compatibility metadata (min native SDK version, ABI hash).
• Deployment strategy:
  • Canary/staged rollouts (e.g., 1% → 10% → 100%) with metrics gates.
  • Use feature flags to toggle risky changes server-side.
• Rollback & safety nets:
  • Automatic health checks and crash-rate monitoring; revert on threshold breach.
  • Keep previous known-good bundle on-device for instant rollback.
  • Remote “kill switch” to disable OTA execution if compromise suspected.
• Testing & QA:
  • Preflight E2E tests on device farm and on every supported native ABI/version.
  • CI to verify bundle signs, integrity, and compatibility metadata.

Example checks

• Before apply: verify signature, check ABI hash matches native app, validate min/native SDK constraints.
• On failure: load local fallback bundle and report telemetry with logs.

Using CodePush-like tools can be powerful if constrained to JS/UI fixes, cryptographically verified, staged, observability-driven, and aligned with app store policies and legal disclosures.