CircleCI is a simple, yet powerful Continuous Integration platform, but it doesn't provide any guarantees or even tight ranges on outbound traffic IP addresses.

If you mock your database or spin up a temporary CI database in CircleCI, this may not be an issue.  But in some cases, that simply isn't practical and you are faced with a dilemma – do you skip the data driven tests altogether (until you can reduce or remove the dependency) or do you grant database access to the CI server?

Fortunately, cloud providers like AWS and Azure make it simple to grant CircleCI (or the CI tool of your choice) access to your cloud based CI database.  

This assumes that you have a firewall restricting access to specific IP addresses.  This also assumes that your CircleCI server accesses the internet via IPv4 (which has been our experience).  For IPv6, you would need to change the CIDR in the allowIP script to /64 instead of /32.

Azure has a variety of authentication mechanisms.  Less ideal: create a user account and specify that user name and password as environment variables to use in your CircleCI script.  More ideal: create an Application Registration for CircleCI.

Creating an App Registration in the Azure Portal

  1. In Azure Active Directory, on the Overview tab, find Tenant ID.  Record this value, as it will be required for CircleCI.
  2. In Azure Active Directory | Manage, click App registrations.  Click: New Registration.  Enter the name, 'CircleCI' then click 'Register'

Once the App Registration has been created, select 'Certificates & secrets' and click on 'New client secret'.

This will generate and display an application secret, essentially, a password or token, that can be used by the App to log in.  Be sure to record it, as it won't be displayed again.

Important: once you have created the App Registration, you need to authorize it to administer your firewall.  In the Azure Portal, find the Network Security Group controlling access to your database.  Select 'Access Control (IAM)', then 'Add a role assignment'.  For Role, select 'Contributor'.  The in 'Select', choose the App Registration that you just created.  You may need to type the first few letters of the name to find it.  Once complete, click 'Save'.

In CircleCI:

Select Settings | Build Settings | Environment Variables.

Add:

  • AZURE_APP_ID
  • AZURE_APP_SECRET
  • AZURE_TENANT_ID

Setting their values appropriately.

In Your Code Repository:

Add the following bash script to your CI code.  Name it: allowIP.sh

public_ip=$(wget -qO- http://checkip.amazonaws.com)

echo "Public IP $public_ip"

az network nsg rule create /
  --name CircleCI /
  --nsg-name {NSG (Firewall) Name} /
  --resource-group {YOUR RESOURCE GROUP NAME} /
  --subscription {YOUR SUBSCRIPTION ID} /
  --priority 200 /
  --source-address-prefixes $public_ip/32 /
  --destination-address-prefixes '*' /
  --destination-port-ranges 5781 /
  --direction Inbound /
  --access Allow /
  --protocol Tcp
allowIP.sh

Create an additional file, removeIP.sh

az network nsg rule delete /
  --name CircleCI /
  --nsg-name {NSG (Firewall) Name} /
  --resource-group {YOUR RESOURCE GROUP NAME} /
  --subscription {YOUR SUBSCRIPTION ID}
removeIP.sh

Update your CircleCI YAML script with the following steps:

    - run: 
      name: Install Azure CLI
      command: curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

    - run:
      name: Login to Azure CLI
      command: az login /
        --service-principal /
        -u $AZURE_APP_ID /
        -p $AZURE_APP_SECRET /
        --tenant $AZURE_TENANT_ID

    - run:
      name: Whitelist IP
      command: ./allowIP.sh
      
#run CI processes
     
#Finally, cleanup.  Each cleanup step is marked as 'when: always' to ensure that they run, even if CI fails

    - run:
      name: Remove Whitelisted IP
      command: ./removeIP.sh
      when: always
    
    - run:
      name: Logout from Azure CLI
      command: az logout
      when: always

You should be all set.  Again, it isn't ideal, but when required, it isn't too difficult to access IP restricted databases from your favorite CI tool.

What are your thoughts?  Any suggestions or comments?  I'd love to hear them.  Thank you!