Skip to main content

Developer Workflow

Overview​

This document outlines our developer workflow, we are using the Trunk Based Development process which essentially means we aim to be merging to main branch frequently and not letting feature branches diverge from main significantly.

We do not maintain a branch per environment, but instead deploy to all environments from main. The key to this is that any code that is merged into main should be considered production ready.

Currently we deploy to dev & uat when the CI workflow triggered by a push to main completes successfully and manually deploy to sandbox & prod environments.

You can read more about TBD here

https://trunkbaseddevelopment.com/

https://www.freecodecamp.org/news/what-is-trunk-based-development/

Hecter​

We have developed a Slack app called Hecter which is responsible for orchestrating our PR and CI/CD workflows based on GitHub triggers (branch pushed, pull request updates etc) and manual workflow triggers from within Slack. Any automated messaging in Slack will come from the Hecter Slack app

Codebase is here https://github.com/Hectare-Agritech/hecter

Documentation for Hecter is here

Feature Development​

In general we are using short lived feature branches for development and pull requests for code reviews with the expectation that you are committing to main frequently and not maintaining feature branches for more that a day or 2.

We are using bit to manage component versioning which forms part of our developer workflow, see here for more information on Bit.

The standard development workflow is as follows...

  1. Create a short-lived feature branch from main with a prefix feature/ for example feature/my-new-feature

  2. Code and commit in your feature branch. After your first commit, push your branch to the remote, on push to any branch prefixed with feature/ a new pull request will be created in GitHub in draft state.

  3. Continue coding and committing as frequently as you like until your feature is ready to be reviewed and merged, once its ready there are a few steps needed to prepare your feature branch for review and merging to main

  4. Firstly you should rebase main onto your feature branch to get any changes from main before you request a code review and run the tests.

  5. After you've rebased main you need to verify your changes have not broken any tests and everything compiles successfully, to check this run the pnpm run build command. This will perform a number of steps including

    • Import the latest bit components
    • Compile all code
    • Regenerate Open API schemas & clients
    • Re-seed the test database
    • Run unit and integration tests for any modified components (Bit will detect modified components)

  6. Assuming step 5 passes you now need to create a final commit message for your branch which will be used to trigger the code review process. Your final commit message must be prefixed with the Jira ticket number, i.e. HE-1234 commit message there is a git pre-push script which validates this via a regex. If you push to main directly or push to a feature branch with the [review] flag this validation will occur. The commit message can be formatted over multiple lines, the first line must provide a title which succinctly describes the feature and should end with the tag [review]. If you want to add a description to your pull request add a carriage return after the [review] tag and enter a description. For example

HE-2379 Get business units with storage counts [review]
This PR adds counts for stores and fields for each BU. To do this we need to call a new endpoint in Inventory (GET /storage/businessunits) which does a facet query on storage index to get the counts for each BU by field and store. We then combine these counts with the business unit information from the GET /business-units/:id endpoint in Customers

  1. Once you have created your final commit you need to soft tag your feature. This is a bit process which looks through all our components, detects which components have changed and marks each modified component as needing a version bump when the CI workflow runs. This tagging process will update the .bitmap file in the root of the repo. To run the tag process run the following command pnpm run tag. This will update the .bitmap file and amend your last commit with the change.

  2. Finally push your final commit(s) to your feature branch. The GitHub action ci-manage-pr.yml will detect the [review] tag and trigger a Slack workflow which will move your draft PR into a ready to review state, request a review from all back end developers and send a message to the #devops-be-pr Slack channel prompting the back end team to pick up the review.

Commit Notes​

There are a few options you can apply to commit messages to control the CI and deployment process.

  1. Skip tagging: If you are pushing changes to services without any code changes to service dependencies (modules), for example migration scripts or changes to the api or lambda cloudformation templates you do not need the bit tagging process. Hecter will detect these service changes and trigger the appropriate build / deployment. In order to skip tagging add the [skip-tag] token to your commit message

  2. Skip tests: if you skip tagging the CI will automatically skip running the tests, but if for some reason you want the CI to skip just the tests you can include the [skip-tests] token in your commit message.

  3. Environments: by default we will deploy any push to main to dev & uat, there may be occasions when you want to control which environments we deploy to once we've completed the build for a given commit. Another use case is an urgent hotfix to production where we may (with obvious caution) want to deploy straight to production. You can control which environments we initialiiy deploy a build to using the following token on your commit message [env:dev,uat,sandbox,prod] or any combination of, to deploy straight to production use the [env:prod] token.

Code Reviews​

As mentioned above we are using GitHub pull requests for code reviews. The process of creating code reviews is automated and any comments / reviews made in GitHub will trigger notifications to the #devops-be-pr Slack channel. The code review process works as follows.

  1. After pushing your final feature commit, the Slack PR workflow will send a message to #devops-be-pr asking for someone from the backend dev team to review the feature.

  2. If you intend to pick up a review you should respond to the message with the :white-check-mark: emoji so the other developers know someone is reviewing the feature.

  3. The Slack message has a link to the PR in GitHub where the code review can be performed. Each review added to the PR will trigger a message to a dedicated thread in Slack which is linked to the original PR message.

  4. If any code changes are required you should make the changes and amend your final commit on your feature branch, this ensures the branch has a single commit with the Jira ticket number in it. A new slack notification will be triggered mentioning the reviewer so they can continue their review.

  5. When the PR is ready to approve, the reviewer should aprove the PR in GitHub but NOT merge the PR in GitHub, this is done automatically.

  6. On approval a GitHub workflow will be triggered (ci-merge-pr.yml) which will rebase main again onto the feature branch and push the changes. If there are merge conflicts and the feature branch cannot be merged to main a message will be triggered to Slack mentioning the developer who created the PR asking them to manually merge the feature branch into main.

  7. After the merge has completed the Slack workflow will trigger the GitHub API to merge the feature branch into main. We use the squash and fast-forward merge process by default, so all your commits on the feature branch will be squashed into a single commit, we use the PR title as the final commit message and include a link to the PR in the commit description

  8. If do not want your commits squashed, but want to retain all the commits you can tell the Slack PR workflow to use the rebase and merge process in GitHub, this will then fast forward merge all your commits without squashing. To do this you need to add a tag to your final commit message before you push it for review, the tag should be [rebase]. If you forget to do this you can go to the PR in GitHub and add the same tag to the top of the PR description.

  9. Once the PR has merged successfully it will automatically trigger the CI workflow to deploy the feature to dev & uat.

Pull Request Notes​

  1. If your feature branch cannot be merged automatically a message will be sent to the Slack PR channel mentioning you indicating you need to merge the PR manunally. You will then need to rebase main, resolve any conflicts and fast-forward merge your feature branch into main.

  2. You can tell Hecter not to merge your feature branch, if you want it reviewed but are not ready to have it merged into main you can add the [skip-merge] flag into the review body in GitHub. This will prevent the ci-merge-pr.yml workflow from being triggered on PR approval.

  3. The CI cannot run concurrently because of the tagging process (see below for more info) if the CI is running when your PR is approved the CI will not be triggered. There is nothing you need to do if this happens because the build workflow triggered at the end of the CI will check to see if there are any pending PRs to merge and automatically trigger the CI if there are. We use PR label called merge to indicate a PR is waiting to be merged after the CI is complete.

CI Workflow​

After your PR has been approved and merged into main it will trigger the CI workflow in GitHub the CI workflow performs the following tasks...

  1. Checkout the repo, install dependencies including bit and run the unit & integration tests for all components that have changed in the commits that triggered the CI. Bit detects this for us via the bit test command

  2. Assuming all the test pass we need to hard tag the modified bit components. Developers are responsible for soft tagging as described in Developer Workflow step 7 above, hard tagging is the process of bundling each component and publishing a new version to the bit cloud. The bit cloud is analygous to an npm registry and is where all our component dependencies are pulled from when building resources for deployment.

  3. We use this tagging mechanism to determine which resources need to be deployed. Each service in the repo contains both an api and lambda resource folder with the configuration files needed to deploy the resource. These files include the Cloudformation templates and a package.json to indicate the dependencies the service has on internal modules (and a few external dependencies). The CI uses the pnpm up command recursively in each resource folder to update the version numbers of our internal components that changed from the tagging process, this will modify the package.json for each resource.

  4. We then use git add to stage any files that the tagging (.bitmap) and version update (package.json) modified and use a bash script to build these modified files into a variable we pass to the Slack build workflow.

  5. We then commit the modified files using the same commit message that triggered the CI with [tag] appended and push this to the repo.

  6. Finally we trigger the Slack build workflow passing in the commit sha, modified files and some other contextual information

Build Workflow​

Once the CI completes successfully the Slack build workflow is triggered. This workflow performs the following tasks...

  1. Detects which resources have changed via both the commit files (to detect service configuration changes) and the package.json changes for dependency version updates.

  2. For each modified resource we create a new build (stored in Slack database) which contains a reference to the commit sha, timestamp, user and creates a new semantic version for the resource. Versions for each resource are maintained in a separate Slack database so we can track the current version of each resource.

  3. After creating the build and incrementing the version we invoke the ci-build.yml Github action which is responsible for building the artifacts for each modified resource and uploading them to S3 from where we deploy. Each built artifact is tagged with the new version number for each resource.

  4. After the build workflow completes (only modified resources are built) we trigger the deploy Github actions for any resources that have been modified (there are separate actions for each resource). Afer building the artifacts successfully the build workflow will trigger deployments to dev and uat unless the environments have been overridden via the commit message as described above.

Release Alerts & Jira Integration​

After a Deploy workflow for a given resource (API, Events etc) has completed successfully it will trigger a Release Notes workflow in Hecter. This workflow searches the builds database to locate the last deployed build for the resource. We use the commitHash field on the last build to find all commits since the last deployment and have logic to determine which of those commits are related to the resource in question.

Once we've identified all the commits related to this release we publish a message to the devops-be-releases channel and using the Jira issue number in the commit messages transition the related issues to Done and set a Deployed date on the issue.

There is an automation set up in Jira which will automatically archive issues once they have been Deployed for > 5 days.

Build Notes​

Manual Deployments​

Once the initial deployment has been completed you can deploy the latest version of any resource to any environment you choose. This is done via a Slack workflow integrated into the devops-be-ci-cd Slack channel. To deploy follow these steps

  1. Click the Workflows link in the devops-be-ci-cd channel and choose CI/CD: Deploy

  2. Choose the environment you want to deploy to from the dropdown

  3. Specify the resources you want to deploy in the textbox. Enter all to deploy the latest version of all resources or use the following comma separated format to deploy specific resources inv:a,inv:e,sys:e the service prefixes are

    • inv Inventory
    • cus Customers
    • tra Trading
    • sys System
    • log Logistics

Use :a to deploy the API and :e to deploy all the Lambda functions, for example inv:a,inv:e,cus:a will deploy the Inventory API and Events and the Customers API

  1. Click deploy, you will see notifications in the devops-be-ci-cd channel as the deployment proceeds.

Production Status​

You can check the versions deployed to production at any time by triggering the CI/CD Production Status workflow in the devops-be-ci-cd channel. Simply click the Workflows link in the devops-be-ci-cd and choose CI/CD: Production Status. This workflow will look through the latest builds in the Slack database and find the latest versions on production and write a message to the Slack channel showing information about the latest build on prod with links to the commit and workflow that triggered the release. This allows you to easily find the latest commit on prod at any time and the latest version.

Latest Build Status​

You can check at any time which environments the latest build is deployed to. We have another workflow in the devops-be-ci-cd for this. Click the Workflows link in the devops-be-ci-cd and choose CI/CD: Latest Build Status. This workflow will post a message to the devops-be-ci-cd with a row for each resource, each row shows the current version, a link to the build workflow and commit and a list of environments the latest build is deployed to.

Resource Versioning​

Each resource has a semantic version associated with it which is incremented each time we build that resource. These version numbers are output in a response header for all APIs (x-api-version) and are included in all logs for both APIs and Lambdas.