Automating Nightly Builds for Continuous Mobile App Health Monitoring

Nov 11, 2024

At Jar, we prioritize delivering a seamless and efficient mobile experience to our users. A critical part of achieving that goal is our nightly build system, which runs automated health checks across our mobile repositories every night at around 12 AM. This system ensures that both our Android and iOS apps are always in their best shape and ready for production, while also boosting developer productivity as they wouldn't have to deal with build issues.

Why Are We Doing This?

Mobile app development is inherently complex, especially when it involves multiple repositories and features being worked on simultaneously. To maintain the integrity and performance of the app across various environments, we have automated several key processes:

  • Gradle Health Checks: These ensure that our Android builds for different flavours (different UAT environment) across different build flavours.
  • iOS Binary Builds: Our build process also includes nightly iOS builds, ensuring the app compiles with the latest shared code as well as older branches for backward compatibility.
  • App Performance Metrics: We continuously measure performance metrics like app startup times and home screen loading times using macro-benchmark tests run on AWS Device Farm.
  • Firebase Test Lab: Automated tests are uploaded to Firebase Test Lab, where we run a suite of tests on real devices to catch potential crashes or performance issues.
  • Diffuse daily size checker: This is a size checker step executed to post analysis of size comparison from yesterday's build.

While when pull request's are merged into our main branch in our mobile repositories we cannot run all the extensive testing for all the configuration/UAT environment's, therefore we utilise the idle time our CI systems would be in the night for checking these metrics.

Impact on Developer Productivity

Automating these builds and tests has a significant impact on developer productivity:

  1. Ensuring Build Stability: Developers can rely on nightly builds to identify issues early, preventing broken code from making it to production. This minimizes firefighting and allows developers to focus on delivering new features.
  2. Faster Turnaround: By the time developers start their day, they already have the results of the nightly builds and can quickly address any issues or performance regressions identified overnight.
  3. Cross-platform Consistency: Regular testing across both Android, shared (Multiplatform), iOS platforms ensures that the app’s build performance is consistent.
  4. Runtime crash identification: This also runs few monkey testing using all the critical Deeplinks using firebase and Maestro testing to identify crashes AKA smoke testing.

Steps in Nightly CI - 

We use Jenkins as our runner and it executes a serious of jobs in the pipeline, as of now we have these steps given below -

Build health checking
Our nightly build process focuses heavily on optimizing build times, ensuring we can test the app efficiently across environments. Below is an overview of the tasks we run during nightly builds, along with their build times and explanations:

  • AssembleProdDebug: Builds the Android app in debug mode for production testing. This mode does not include release-level optimizations.
  • AssembleProdDebugNoBuildCacheAndPro: Builds the Android app in debug mode without using Gradle’s build cache. This helps ensure the build is consistent and not reliant on cached data.
  • AssembleProdRelease: Builds the app in release mode, optimized for production. This build includes code shrinking, obfuscation, and other optimizations.
  • JarPropConfig1 [include_dfm=false]: Excludes Dynamic Feature Modules (DFM) from the build process. This reduces build times by skipping non-critical features.
  • fnJarPropConfig2 [include_dfm=true, include_firebase_pref=false, jar.keep_en_res_only=true]: Includes DFMs but excludes Firebase Pref plugin tasks and generates only English-language resources. This speeds up the build by reducing unnecessary tasks.
  • fnJarPropConfig3 [include_dfm=false, include_firebase_pref=false, jar.keep_en_res_only=false]: Excludes DFMs and Firebase Pref plugin tasks, but generates all language resources. This is useful for testing the app’s full experience across different languages.
  • JarPropConfig4 [include_dfm=true, include_firebase_pref=true, jar.keep_en_res_only=true]: This build includes both DFMs and Firebase Pref plugin, but limits resource generation to English-only. It’s useful for Firebase testing and modularization with reduced build overhead.
  • AssembleProdReplicaDebugUnitTest: Builds the ProdReplica debug build and runs unit tests, ensuring that alternative production environments are stable.
  • :app:bundleProdRelease: Packages the app into a production-ready bundle (AAB), which is uploaded to the Play Store. This task includes app compression and asset packaging.
  • linkPodDebugFrameworkIosX64: Links the iOS debug framework, ensuring that the app is compatible with iOS builds and works across shared codebases.

Firebase Test Lab monkey testing
FTL plays a crucial role in identifying platform-specific issues. By using our set presets in firebase of running monkey testing across flows helps us to identify any crashes,

For example, a recent run detected a fatal exception related to font loading in Android, which was not caught in QA-regression nor while in development.

Using Firebase Test Lab, we could pinpoint this issue and resolve it before it affected our users in production. Test results are automatically uploaded to a shared Google Cloud Storage bucket, where developers can review logs, screenshots, and stack traces of any failed tests.

Maestro deeplink smoke testing
In this step we use Maestro framework, to execute all the deep links we have in the app for smoke testing and some tests for core flows.

This gives us an idea of what is breaking.

Diffuse daily size checker

Using diffuse, we keep a track of release apk with minification enabled on to track and ensure we aren't increasing the size at a daily basis.

Conclusion

Our nightly build system is an integral part of maintaining the stability and performance of our mobile applications. By automating these builds, tests, and performance checks, we catch potential issues early, optimize developer workflows, and ensure that our apps perform smoothly across platforms. This continuous testing process allows us to deliver a better product to our users every single day while ensuring the development cycle is both fast and efficient.

This system isn’t just a tool for catching errors—it’s a vital part of our strategy to continuously improve the quality and performance of our mobile apps.

Kunal Gupta