Moving a local sfdx project under GitLab CD/CI for automatic push to production

There are two main ways to carry our CD/CI process with GitLab:

• Using the sfdx-project-template and the instructions at https://gitlab.com/sfdx/sfdx-project-template
• Import the yml file locally and run locally

Unfortunately the sfdx project is not being kept up to date and suffers from 3 main problems (as at Winter ’20):

• Code coverage is not included out of the box
• The initial build of the package is not supported
• Package dependencies is not supported

The lack of codecoverage will prevent any deployments to live so really the first option is not viable. So this post will explore how to run the yml file locally within your project.

So let’s say that you have a local project on your machine and you want to move this across to GitLab under the CD/CI process for package deployment. The steps are:

1. Setup a new project in GitLab
2. Set up your code to be linked to the GitLab repository
3. Set up the variables for the CD/CI processing
4. Add in the yml file for processing

Step 1 – New Project in GitLab

This is easy and simply follow these instructions https://gitlab.com/projects/new to create a new blank project.

Step 2 – Locally link to the remote GitLab repository

At this point you have a remote repository but no links between the local and no concept of where the master repository is. Gitlab will provide the code snippet to run locally in the directory of your code. You will insert your username and project name in the code below.

git init
git remote add origin https://gitlab.com/USERNAME/PROJECTNAME.git
git add .
git commit -m "Initial commit"
git push -u origin master

Now if you modify any code locally you can push this to the remote repository. This is well documented for Visual Code for instance.

Step 3 – Setup the CD/CI variables

From the main project menu in GitLab choose settings and then CD/CI

Scroll down to the Variables section

There need to be three variables set up – DEVHUB_AUTH_URL, SANDBOX_AUTH_URL and PACKAGE_NAME

Use the code below to find the ‘Sfdx Auth Url’ for your production/dev hub and your sandbox (UAT sandbox for testing before production deployment).

sfdx force:org:display --targetusername ORGNAME --verbose

The PACKAGE_NAME is simply the name of your package in the sfdx-project.json file (not strictly needed as the yml file will read the package name).

Step 4 – Adding the yml files

I have found it is better to have a folder to hold the main yml file and then refer to this from the root directory. So create a cdci directory and copy the Salesforce.gitlab-ci.yml from https://gitlab.com/sfdx/sfdx-cicd-template/-/blob/master/Salesforce.gitlab-ci.yml and copy to the cdci folder.

For CD/CI GitLab is expecting a yml file in the root so create a .gitlab-ci.yml file with the following code:

include:
  - local: 'cdci/Salesforce.gitlab-ci.yml'
  - template: SAST.gitlab-ci.yml

sast:
  # Run Security scanning job 'SAST' within the 'preliminary-testing' stage
  stage: preliminary-testing
  # Override executing any pre-scripts and SFDX helper functions
  before_script:
    - echo "Skipping sfdx helper scripts"
  # Run only APEX and Secret scanning and run each analyzer as an isolated job instead of single job in DinD
  variables:
    SAST_DEFAULT_ANALYZERS: "pmd-apex,secrets"
    SAST_DISABLE_DIND: "true"

This only links to the other yml file which has our main processing. For more information on yml files wee https://docs.gitlab.com/ee/ci/yaml/

I have made some updates to the yml file. First add –codecoverage to the apex tests, find this code and update :

    # If no "test:scratch" script property, then add one
    if [[ -z "$scriptValue" || $scriptValue == null ]]; then
      local tmp=$(mktemp)
      jq '.scripts["test:scratch"]="sfdx force:apex:test:run --codecoverage --resultformat junit --wait 10 --outputdir ./tests/apex"' package.json > $tmp
      mv $tmp package.json
      echo "added test:scratch script property to package.json" >&2
      cat package.json >&2
    fi

.
.
.

 # Create a new package version
    cmd="sfdx force:package:version:create --targetdevhubusername $devhub_username --package $package_id --versionnumber $version_number --installationkeybypass --wait 10 --json  --codecoverage" && (echo $cmd >&2)
    output=$($cmd) && (echo $output | jq '.' >&2)
    local subscriber_package_version_id=$(jq -r '.result.SubscriberPackageVersionId' <<< $output)

Then add in the setting of base version

    # Calculate next version number.
    # If the latest package version is released then
    # we need to increment the major or minor version numbers.
    local cmd="sfdx force:package:version:list --targetdevhubusername $devhub_username --packages $package_id --concise --released --json" && (echo $cmd >&2)
    local output=$($cmd) && (echo $output | jq '.' >&2)
    local last_package_version=$(jq '.result | sort_by(-.MajorVersion, -.MinorVersion, -.PatchVersion, -.BuildNumber) | .[0]' <<< $output)
    local is_released=$(jq -r '.IsReleased' <<< $last_package_version)
    local major_version=$(jq -r '.MajorVersion' <<< $last_package_version)
    local minor_version=$(jq -r '.MinorVersion' <<< $last_package_version)
    local patch_version=$(jq -r '.PatchVersion' <<< $last_package_version)
    local build_version="NEXT"
    if [ $major_version == 'null' ]; then major_version=1; fi;
    if [ $minor_version == 'null' ]; then minor_version=0; fi;
    if [ $patch_version == 'null' ]; then patch_version=0; fi;
    if [ $is_released == true ]; then minor_version=$(($minor_version+1)); fi;
    local version_number=$major_version.$minor_version.$patch_version.$build_version
    echo "version_number=$version_number" >&2

and then the code for installing the package dependencies:

  function deploy_scratch_org() {
    local devhub=$1
    local orgname=$2
    assert_within_limits $devhub DailyScratchOrgs
    local scratch_org_username=$(create_scratch_org $devhub $orgname)
    echo $scratch_org_username > SCRATCH_ORG_USERNAME.txt
    get_org_auth_url $scratch_org_username > SCRATCH_ORG_AUTH_URL.txt
    install_dependencies $scratch_org_username
    push_to_scratch_org $scratch_org_username
    populate_scratch_org_redirect_html $scratch_org_username
    echo "Deployed to scratch org $username for $orgname"
  }
.
.
.

  # install_dependencies 
  # Arguments
  #     $1 = scratch org username


  function install_dependencies() {
    local scratch_org_username=$1
    echo "Installing package dependencies"

    for package_version_id in $(jq -r '.packageDirectories[].dependencies[].subscriberPackageVersionId' < sfdx-project.json)
    do
      echo $package_version_id
      if [ $package_version_id ]; then
         # install the package
         local cmd="sfdx force:package:install --targetusername $scratch_org_username --package $package_version_id --wait 10 --publishwait 10 --noprompt --json" && (echo $cmd >&2)
        local output=$($cmd) && (echo $output | jq '.' >&2)
      fi
    done
  }

When a new update is push to the master then this will create the scratch org, with the dependent packages and then install into UTA and then production. Any dependent packages already need to be installed in the UAT and production – which is what is most likely anyway.