Category: Google-Cloud-Platform

One Button, Two Stores: Shipping Your Flutter App to Play Store and App Store with GitHub Actions and Fastlane How we went from “whose laptop has the keystore?” to a single Run workflow button that tests, builds, signs and uploads our Android and iOS apps, without anyone on the team owning a Mac. The problem We build a Flutter app for Android and iOS. Like most teams, we started out releasing it by hand, and every release was slow and a little risky. The biggest blocker was iOS. To build an iOS app and upload it to the App Store or TestFlight, you need Xcode, and Xcode only runs on a Mac. Every iOS release meant finding a Mac, installing a distribution certificate and a provisioning profile, and hoping none of them had expired since the last time. Android was easier to build but had its own problems: The upload keystore lived on one or two laptops. If that person was on leave, nothing shipped. Someone had to bump the build number in pubspec.yaml by hand. Sometimes two people uploaded the same number and Play rejected the second one. Every so often a “quick test build” went out pointing at the wrong API. We wanted releases to be boring: repeatable, one click, the same steps every time, with no secrets on anybody’s laptop and no Mac on anybody’s desk. What we ended up with A single manually-triggered GitHub Actions workflow with three inputs: Input Choices Meaning platform android, ios, both Which app(s) to release api dev, prod Which backend the build talks to version Text such as 0.1.5 (optional) The version name shown in the stores and the app; empty uses pubspec.yaml Anyone on the team presses Actions → Mobile release → Run workflow, from any computer. About 15 to 25 minutes later: Android: a signed App Bundle is released to Google Play’s Internal testing track, and testers can install it straight away. iOS: a signed build is processing in TestFlight, and internal testers get it automatically. Why GitHub Actions and Fastlane? We split the work between two tools, each doing what it’s best at. Fastlane handles the how: talking to Google Play and App Store Connect, working out version numbers, running the Flutter build, signing and uploading. It runs the same way on a laptop as it does on CI, so we can still debug a release locally. GitHub Actions handles the where and when: which machine runs the build (Linux for Android, macOS for iOS), which secrets are injected, who has to approve a production build, and keeping the built files for later. That split keeps the workflow file short and easy to read. iOS without a Mac and without Xcode This was the part we were most worried about, and it ended up being the cleanest. GitHub provides hosted macOS machines that already have Xcode installed. Our iOS job simply runs on one of them, so the Mac is GitHub’s problem, not ours. Signing was the other half. The traditional approach is to export a certificate (.p12) and a provisioning profile, store them as secrets, and install them on every run. That’s a lot of moving parts, and certificates expire every year. Instead we use automatic signing with an App Store Connect API key. The only iOS secrets are the API key itself and our Apple Team ID. During the build, Xcode uses that key to create or reuse the distribution certificate and App Store profile on the fly. There are no certificate files to store, renew or lose. Two small settings make iOS releases even smoother: The app’s Info.plist declares that it only uses standard encryption (HTTPS), so TestFlight doesn’t stop each upload to ask about export compliance. The workflow doesn’t wait for Apple to finish processing the build, which takes 10 to 30 minutes. There’s no reason to pay for a macOS machine to sit and watch that. Every release follows the same four stages Android and iOS run the same shape of pipeline, so once you understand one you understand both: Stage What happens Test flutter analyze and flutter test. Any failure stops everything; nothing is built or uploaded. Version Ask the store for the latest build number and add 1. The version name comes from the version input, or pubspec.yaml when it’s empty. Build A clean release build with the right version and API URL baked in. Upload Android goes to Play Internal testing. iOS goes to TestFlight. Key decisions The store decides the build number Keeping the build number in a file means someone has to remember to bump it, and two branches can bump it to the same value. So we stopped trusting the file. Before each build, Fastlane asks Google Play (or TestFlight) for the highest build number it has ever seen and adds one. Nothing is written back to the repo, so the workflow only ever needs read access. Type the version once, see it everywhere The optional version input sets the version name in one go. Type 0.1.5 and it shows up in Play Console, App Store Connect, TestFlight and on the app’s own profile screen. Only a strict format like 0.1.5 is accepted, so a typo fails in seconds instead of after a 20-minute build. Leave it empty and the version from pubspec.yaml is used. Release to testers directly, keep production human Android builds go straight to the Internal testing track instead of waiting as drafts, and iOS builds go straight to TestFlight. Only the app itself is uploaded; the store listing, screenshots and descriptions are never touched. Going live stays a human decision. Builds that use the production API pause until an approved reviewer clicks Approve in GitHub, and promoting a build to Production or submitting it for App Store review is still done by hand, after someone has tested it. Secrets you need Everything sensitive lives in Settings → Secrets and variables → Actions. Nothing is committed to the repo and nothing is needed on a developer’s machine. Secret Platform What it is ANDROID_KEYSTORE_BASE64 Android The upload keystore, base64-encoded ANDROID_KEYSTORE_PASSWORD Android Keystore password ANDROID_KEY_ALIAS Android Key alias ANDROID_KEY_PASSWORD Android Key password PLAY_STORE_JSON_KEY Android Google Cloud service-account JSON key APPLE_TEAM_ID iOS 10-character Apple Developer team ID ASC_KEY_ID iOS App Store Connect API Key ID ASC_ISSUER_ID iOS App Store Connect API Issuer ID ASC_KEY_P8_BASE64 iOS The AuthKey_XXXX.p8 file, base64-encoded The dev and production API URLs are stored as plain variables, since they aren’t secret. One-time store setup The workflow is the easy part. Most of the time goes into store-side setup, and you only do it once. Google Play Upload the first build by hand. The Play API can only upload to an app that already has at least one release. Enable the Google Play Android Developer API in any Google Cloud project. Create a service account and download a JSON key. Invite the service account’s email in Play Console under Users and permissions, and allow it to release to testing tracks. Wait about five minutes for the permissions to take effect. Don’t generate a new keystore: Play only accepts builds signed with the upload key it already knows. Apple Join the Apple Developer Program for your organisation. Create the app in App Store Connect with the right bundle ID. Create an App Store Connect API key with the Admin role, so it can create certificates and profiles. Download the .p8 file straight away, because Apple only lets you download it once. Create an internal TestFlight group and add your testers. The workflow file Save this as .github/workflows/mobile-release.yml: name: Mobile release on: workflow_dispatch: inputs: platform: description: App to release required: true type: choice options: [android, ios, both] default: android api: description: API the build talks to (prod waits for approval) required: true type: choice options: [dev, prod] default: dev version: description: Version shown in the stores, e.g. 0.1.5 (empty = pubspec.yaml) required: false type: string default: '' permissions: contents: read env: FLUTTER_VERSION: 3.47.7 RUBY_VERSION: '3.3' APP_VERSION_NAME: ${{ inputs.version }} jobs: test: name: Analyze and test runs-on: ubuntu-latest defaults: run: working-directory: apps/mobile steps: - uses: actions/checkout@v4 - name: Check the version input if: inputs.version != '' run: | if ! [[ "$APP_VERSION_NAME" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then echo "::error::version must look like 0.1.5, got '$APP_VERSION_NAME'" exit 1 fi - uses: subosito/flutter-action@v2 with: channel: stable flutter-version: ${{ env.FLUTTER_VERSION }} cache: true - run: flutter pub get - run: flutter analyze - run: flutter test android: name: Android ? Play Internal testing needs: test if: inputs.platform != 'ios' runs-on: ubuntu-latest concurrency: group: mobile-release-android cancel-in-progress: false environment: ${{ inputs.api == 'prod' && 'production' || 'development' }} env: SKIP_FLUTTER_TESTS: 'true' APP_PLAY_RELEASE_STATUS: completed APP_DEV_API_URL: ${{ vars.APP_DEV_API_URL }} APP_PROD_API_URL: ${{ vars.APP_PROD_API_URL }} APP_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }} APP_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }} APP_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }} steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: '17' - uses: subosito/flutter-action@v2 with: channel: stable flutter-version: ${{ env.FLUTTER_VERSION }} cache: true - uses: ruby/setup-ruby@v1 with: ruby-version: ${{ env.RUBY_VERSION }} bundler-cache: true working-directory: apps/mobile - name: Restore the upload keystore and the Play key env: KEYSTORE_BASE64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }} PLAY_JSON: ${{ secrets.PLAY_STORE_JSON_KEY }} run: | echo "$KEYSTORE_BASE64" | base64 --decode > "$RUNNER_TEMP/upload-keystore.jks" printf '%s' "$PLAY_JSON" > "$RUNNER_TEMP/play-store.json" echo "APP_KEYSTORE_PATH=$RUNNER_TEMP/upload-keystore.jks" >> "$GITHUB_ENV" echo "APP_PLAY_STORE_JSON_KEY=$RUNNER_TEMP/play-store.json" >> "$GITHUB_ENV" - name: Build and release to Internal testing working-directory: apps/mobile/android run: bundle exec fastlane ${{ inputs.api == 'prod' && 'internal' || 'dev' }} - uses: actions/upload-artifact@v4 with: name: android-${{ inputs.api }}-${{ inputs.version && format('{0}-', inputs.version) || '' }}${{ github.run_number }} path: apps/mobile/build/app/outputs/bundle/release/app-release.aab retention-days: 14 - name: Remove the keystore and key if: always() run: rm -f "$RUNNER_TEMP/upload-keystore.jks" "$RUNNER_TEMP/play-store.json" ios: name: iOS ? TestFlight needs: test if: inputs.platform != 'android' runs-on: macos-latest concurrency: group: mobile-release-ios cancel-in-progress: false environment: ${{ inputs.api == 'prod' && 'production' || 'development' }} env: SKIP_FLUTTER_TESTS: 'true' APP_DEV_API_URL: ${{ vars.APP_DEV_API_URL }} APP_PROD_API_URL: ${{ vars.APP_PROD_API_URL }} APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }} ASC_KEY_ID: ${{ secrets.ASC_KEY_ID }} ASC_ISSUER_ID: ${{ secrets.ASC_ISSUER_ID }} ASC_KEY_P8_BASE64: ${{ secrets.ASC_KEY_P8_BASE64 }} steps: - uses: actions/checkout@v4 - uses: maxim-lobanov/setup-xcode@v1 with: xcode-version: latest-stable - uses: subosito/flutter-action@v2 with: channel: stable flutter-version: ${{ env.FLUTTER_VERSION }} cache: true - uses: ruby/setup-ruby@v1 with: ruby-version: ${{ env.RUBY_VERSION }} bundler-cache: true working-directory: apps/mobile - name: Build, sign and upload to TestFlight working-directory: apps/mobile/ios run: bundle exec fastlane ${{ inputs.api }} - uses: actions/upload-artifact@v4 with: name: ios-${{ inputs.api }}-${{ inputs.version && format('{0}-', inputs.version) || '' }}${{ github.run_number }} path: apps/mobile/build/ios/ipa/App.ipa retention-days: 14 How the workflow works, in short Inputs: when you click Run workflow, you choose the platform (Android, iOS or both), the API (dev or prod) and, optionally, the version number. permissions: the workflow can only read the repo. It never commits anything back. test job: checks the version format, then runs flutter analyze and flutter test once on Linux. If anything fails, the build jobs never start. The build jobs set SKIP_FLUTTER_TESTS so the tests don’t run again on the more expensive machines. android job: runs on Linux. It decodes the keystore and Play key from secrets into a temporary folder, then Fastlane builds, signs and releases the app to Play Internal testing. The last step deletes the keystore and key, even if the build failed. ios job: runs on GitHub’s macOS machine with the latest stable Xcode. Fastlane builds the app, signs it using the App Store Connect API key and uploads it to TestFlight. This is the job that replaces your own Mac. environment: builds using the prod API run in the production environment, which waits for an approved reviewer. Dev builds run straight away. concurrency: only one upload per store runs at a time. A second run waits its turn, so two releases can never pick the same build number. upload-artifact: keeps a copy of the .aab and .ipa for 14 days, named with the API, version and run number. Troubleshooting These are the errors we actually hit, and what fixed them: Message Fix The caller does not have permission Invite the service account in Play Console and grant app permissions, then wait about 5 minutes. Could not read any versionCode from Play The service account can’t see the app, or no build was ever uploaded by hand. Version code X has already been used Two uploads raced. The concurrency group prevents this on CI; just rerun. The Android App Bundle was signed with the wrong key The keystore secret isn’t the upload key Play expects. iOS export fails after a successful archive Put the signing settings in the Xcode project instead of passing them as build arguments. What we got out of it Anyone on the team can ship, from Windows, Mac or Linux, with nothing installed locally. No Mac and no Xcode needed for iOS releases, and no certificate files to renew. No secrets on laptops. Keys exist only as encrypted GitHub secrets and on a build machine for the length of a job. No more build number clashes, because the store decides and uploads run one at a time. Tests always gate a release. A red test means nothing is built. People still make the go-live decision for production. If you’re running a Flutter app with a small team and still releasing from someone’s laptop, this setup takes about a day, and most of that day is store-console setup you only do once. After that, every release is one button. Stack: Flutter 3.47, Fastlane 2.228, Ruby 3.3, GitHub Actions (ubuntu-latest and macos-latest).

Apache Kafka is the numerous common buffer solution deployed together with the ELK Stack. Kafka is deployed within the logs delivery and the indexing units, acting as a segregation unit for the data being collected: In this blog, we’ll see how to deploy all the components required to set up a resilient logs pipeline with Apache Kafka and ELK Stack: Filebeat – collects logs and forwards them to a Kafka topic. Kafka – brokers the data flow and queues it. Logstash – aggregates the data from the Kafka topic, processes it and ships to Elasticsearch. Elasticsearch – indexes the data. Kibana – for analyzing the data.   My environment: To perform the steps below, I set up a single Ubuntu 18.04 VM machine on AWS EC2 using local storage. In real-life scenarios, you will probably have all these components running on separate machines. I started the instance in the public subnet of a VPC and then set up a security group to enable access from anywhere using SSH and TCP 5601 (for Kibana). Using Apache Access Logs for the pipeline, you can use VPC Flow Logs, ALB Access logs etc. We will start by installing the main component in the stack — Elasticsearch. Login to your Ubuntu system using sudo privileges. For the remote Ubuntu server using ssh to access it. Windows users can use putty or Powershell to log in to Ubuntu system. Elasticsearch requires Java to run on any system. Make sure your system has Java installed by running the following command. This command will show you the current Java version. sudo apt install openjdk-11-jdk-headless Check the installation is successful or not by the below command ~$ java — versionopenjdk 11.0.3 2019–04–16OpenJDK Runtime Environment (build 11.0.3+7-Ubuntu-1ubuntu218.04.1)OpenJDK 64-Bit Server VM (build 11.0.3+7-Ubuntu-1ubuntu218.04.1, mixed mode, sharing) Finally, I added a new elastic IP address and associated it with the running instance. The example logs used for the tutorial are Apache access logs.   Step 1: Installing Elasticsearch We will start by installing the main component in the stack — Elasticsearch. Since version 7.x, Elasticsearch is bundled with Java so we can jump right ahead with adding Elastic’s signing key: Download and install the public signing key: wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - Now you may need to install the apt-transport-https package on Debian before proceeding: sudo apt-get install apt-transport-https echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list Our next step is to add the repository definition to our system: echo “deb https://artifacts.elastic.co/packages/7.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list You can install the Elasticsearch Debian package with: sudo apt-get update && sudo apt-get install elasticsearch Before we bootstrap Elasticsearch, we need to apply some basic configurations using the Elasticsearch configuration file at: /etc/elasticsearch/elasticsearch.yml: sudo su nano /etc/elasticsearch/elasticsearch.yml Since we are installing Elasticsearch on AWS, we will bind Elasticsearch to the localhost. Also, we need to define the private IP of our EC2 instance as a master-eligible node: network.host: "localhost" http.port:9200 cluster.initial_master_nodes: ["<InstancePrivateIP"] Save the file and run Elasticsearch with: sudo service elasticsearch start To confirm that everything is working as expected, point curl to: http://localhost:9200, and you should see something like the following output (give Elasticsearch a minute or two before you start to worry about not seeing any response): {   "name" : "elasticsearch",   "cluster_name" : "elasticsearch",   "cluster_uuid" : "W_Ky1DL3QL2vgu3sdafyag",   "version" : {     "number" : "7.2.0",     "build_flavor" : "default",     "build_type" : "deb",     "build_hash" : "508c38a",     "build_date" : "2019-06-20T15:54:18.811730Z",     "build_snapshot" : false,     "lucene_version" : "8.0.0",     "minimum_wire_compatibility_version" : "6.8.0",     "minimum_index_compatibility_version" : "6.0.0-beta1"   },   "tagline" : "You Know, for Search" }   Step 2: Installing Logstash Next up, the “L” in ELK — Logstash. Logstash and installing it is easy. Just type the following command. sudo apt-get install logstash -y Next, we will configure a Logstash pipeline that pulls our logs from a Kafka topic, processes these logs and ships them on to Elasticsearch for indexing. Verify Java is installed: java -version openjdk version "1.8.0_191" OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12) OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode) Let’s create a new config file: Since we already defined the repository in the system, all we have to do to install Logstash is run: sudo nano /etc/logstash/conf.d/apache.conf Next, we will configure a Logstash pipeline that pulls our logs from a Kafka topic, processes these logs, and ships them on to Elasticsearch for indexing. Let’s create a new config file: input {   kafka {     bootstrap_servers => "localhost:9092"     topics => "apache"     } } filter {     grok {       match => { "message" => "%{COMBINEDAPACHELOG}" }     }     date {     match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]     }   geoip {       source => "clientip"     } } output {   elasticsearch {     hosts => ["localhost:9200"]   } } As you can see — we’re using the Logstash Kafka input plugin to define the Kafka host and the topic we want Logstash to pull from. We’re applying some filtering to the logs and we’re shipping the data to our local Elasticsearch instance.   Step 3: Installing Kibana Let’s move on to the next component in the ELK Stack — Kibana. As before, we will use a simple apt command to install Kibana: sudo apt-get install kibana We will then open up the Kibana configuration file at: /etc/kibana/kibana.yml, and make sure we have the correct configurations defined: server.port: 5601 server.host: "<INSTANCE_PRIVATE_IP>" elasticsearch.hosts: ["http://<INSTANCE_PRIVATE_IP>:9200"] Then enable and start the Kibana service: sudo systemctl enable kibana sudo systemctl start kibana We would need to install Firebeat. Use: sudo apt install filebeat   Open up Kibana in your browser with http://<PUBLIC_IP>:5601. You will be presented with the Kibana home page.

You may have heard infrastructure as code(IaC), But do you know what infrastructure is? Why do we need infrastructure as code? What are the benefits of infrastructure as code? Is it safe and secure?    What is Infrastructure as Code(IoC)? Infrastructure as code (IaC) means to manage and upgrade your environments as infrastructure using configuration files. Terraform provides infrastructure as code for provisioning, compliance, and management across any public cloud, private data center, and third-party service. Enables teams to write, share, manage, and automate any infrastructure using version control With automated policy enforcement for security, compliance, and operational best practices and Enable developers to provision their desired infrastructure from within their workflows. IOC has a high impact on the Business perspective by providing Increased Productivity, Reduced Risk, Reduced Cost   Why do we use Infrastructure as Code(IoC)? Terraform is a simple human-readable configuration language, to define the desired topology of infrastructure resources VCS Integration Write, version, review, and collaborate on Terraform code using your preferred version control system Workspaces Workspaces decompose monolithic infrastructure into smaller components, or "micro-infrastructures". These workspaces can be aligned to teams for role-based access control. Variables Granular variables allow easy reuse of code and enable dynamic changes to scale resources and deploy new versions. Runs Terraform uses two-phased provisioning a plan (dry run) & apply (execution). Plans can be inspected before execution to ensure expected behavior and safety. Infrastructure State The state file is a record of currently provisioned resources. State files enable a versioned history of the infrastructure and are encrypted at rest. Versions can be inspected to see incremental changes. Policy as Code Sentinel is a policy as a code framework to automate multi-cloud governance.   What are the benefits of Infrastructure as Code(IoC)? Infrastructure as Code enables Infrastructure teams to test the applications in staging environments or development environment early - likely in the development cycle Infrastructure as Code Saves You Time and Money We can have a version history like when the infrastructure is upgraded and who has done it from the code itself. Else we have to ask to check the Infrastructure admin to look into logs and which is very time-consuming. We can check it into version control and I get versioning. Now we can see an incremental history of who changed what Use Infrastructure as Code to build update and manage any cloud, infrastructure, or services Terraform makes it easy to re-use configurations for the environment for similar infrastructure, helping you avoid mistakes and save time. We can use the same configuration code for the different staging Production and development environments. Terraform supports many Providers to be built from just a simple and less line of code. Major providers are as follows AWS Azure GitHub GitLab Google Cloud Platform VMWare Docker  and  200+ more. A Simple example to create an Ec2 Instance with just a few lines of code. resource "aws_instance" "ec2_instance" {   ami = "ami-*******"   instance_type = "t2.micro"   vpc_security_group_ids = ["${aws_security_group.*****.id}"]   key_name = "${aws_key_pair.****.id}"   tags {     Name = "New-EC2-Instance"   } } But First, we have to write code for which provider we are writing our code. To do so  here is the simple basic code to assign a provider provider "aws" {   region = "us-west-2"   ## PROVIDE CREDENTIALS } Now to Create your Ec2 Instance in AWS. We have to run the commands. So terraform has Four commands to check and apply the infrastructure changes, Init Plan Apply Destroy.   1. Init $ terraform init We can understand from the name of the command that is used to initialize something. So here terraform will be initialized in our code which will create some basic backend and tfstate files in folders for internal use. 2. Plan $ terraform plan As we do compile in some code languages, it will check for the compilation errors and plan what is going to happen when we run the script to generate infrastructure code. It will show you what resources are going to be created and what will be the configuration. 3. Apply $ terraform apply It is time to run the script and check what is being generated from the scripts. So the command will execute the script and apply the changes in our infrastructure, which will generate some resources for what we have written in the code.  4. Destroy $ terraform destroy This command is used when we want to remove or destroy the resource. After some time we don't need that resource then we just run the command which will destroy the resource. And your money is saved.