Laptop-green, store-red

A release candidate built on a laptop “worked fine.” Sideload install, smoke the critical path, ship the AAB from a different machine — or from a CI job whose image nobody audited last quarter. The store build failed: R8 behaved differently under another JDK patch, a dependency resolved from a developer’s warm cache instead of a clean graph, and signing picked the wrong signingConfigs block because local.properties on the laptop was not what the release job used. Nobody changed application logic that week; the assembly path did.

That gap is a pipeline honesty problem. CI/CD exists to make assemble, test, and promotion repeatable with an audit trail — not as Agile theater or a DevOps rebrand. Green on an unverifiable laptop is not evidence.

I have shipped Android through classic Jenkins jobs and through org-scale platforms including Screwdriver (Verizon Media / Yahoo open-source lineage). The product names differ; the honesty question does not: will the artifact users get match the environment the team actually verified?

What the pipeline must pin

When laptop-green becomes store-red without a code change, one of these usually drifted:

1. JDK — patch-level mismatch changes compiler behavior, desugaring, or which warnings become errors.
2. Android SDK / build-tools / NDK — “I have a recent SDK” on a laptop is not a pin; CI must name versions in an image or locked config.
3. Signing — release keystore, upload key, and Play App Signing alignment. A debug-signed local install proves nothing about the store binary.
4. Gradle + dependency graph — wrapper version, remote cache contents, and whether local.properties secrets came from a vault or a sticky laptop file.

The failure mode is social as much as technical: someone “just rebuilt locally” to unblock a release, then promoted that binary. CI is the gate that refuses to promote unless those pins match. CD is the path that pushes only artifacts produced under those pins — signed AAB/APK, internal track, then wider rollout. Review LGTM on a laptop assemble is not a substitute for either.

Typical Android stages that keep the story honest:

1. Checkout + cache Gradle dependencies
2. Static analysis (ktlint/detekt)
3. Unit tests (:test)
4. Assemble the same variants the store will see
5. Optional instrumentation / smoke (slower; often nightly)
6. Archive signed AAB/APK + upload to a controlled track

flowchart LR
  commit[Git push] --> ci[CI controller]
  ci --> agent[Pinned agent / image]
  agent --> build[Gradle assemble]
  build --> tests[Tests + lint]
  tests --> artifact[Signed artifact]
  artifact --> track[Internal track / store]

Figure 1. Honesty path: commit → pinned environment → signed artifact → track — not “whatever was on my laptop.”

Rule of thumb — pick the platform that keeps JDK and SDK versions honest; Android failures from environment skew look like flaky tests and waste weeks.

Who owns that honesty — Jenkins vs org platform

Jenkins is the mental model many teams learn first: a controller schedules jobs, agents run stages, a Jenkinsfile encodes the graph. It can pin toolchains well — if someone treats agent images and credentials as product, not tribal knowledge. Flexibility is the failure mode: plugin glue, Groovy drift, and snowflake agents recreate “works on that one executor.”

Org-scale platforms like Screwdriver exist when that maintenance becomes its own product. Pipeline definitions in repo, containerized builds, event-driven triggers (PR, tag, cron), centralized secrets and audit — the goal is one honest environment many teams share, not a prettier dashboard. Buildkite, org-tier GitHub Actions, and similar tools make the same bet. Screwdriver’s cdCon 2020 talk slides (Verizon Media) describe the operations angle better than a README skim.

Inside the container, Android specifics do not change: Gradle wrapper, JDK pin, secrets from vault, emulator farms optional and expensive. On large Android monorepos I have worked in, PR gates usually run unit tests and static analysis; heavier instrumentation moves to nightly or release tracks so honesty stays fast enough that people trust the red.

Jenkins vs Screwdriver is not a theology fight. It is who is accountable for the pins — a team babysitting VMs, or a platform that makes ephemeral, versioned builders the default. Migrate when copy-paste Jenkinsfiles and drifting agents are lying faster than you can patch them.

Lessons that survive any migration

What survives Jenkins → Screwdriver, or any other move:

1. Pin toolchains — JDK, Android SDK, NDK in image or config; document bumps as intentional changes.
2. Gradle remote cache — pays off on modular Android monorepos when inputs are truly reproducible.
3. Separate PR vs release pipelines — fast feedback (lint + unit) under a human patience budget; full suite on nightly/release.
4. Signed releases only from CI — no manual keystore on a laptop that “also worked.”
5. Treat flaky green as debt — a pipeline that lies erodes trust faster than an honest red.

Android CI at that scale looks boring on purpose. Exciting pipelines usually mean someone is firefighting environment skew. If your release broke without a code change, fix the honesty of assemble/test/promotion before renaming the tool.

References

• Jenkins
• Screwdriver CD — open-source CI/CD platform
• Jenkins CI/CD overview (YouTube)
• Screwdriver at Verizon Media — cdCon 2020 slides (PDF)