Moving Jenkins Pipelines to CircleCI: A Quick and Painless Transition

Last September we set up a Jenkins instance for internal deployments, with high hopes that it would solve all of our CI/CD problems. We used CDK and Jenkins Configuration as Code to ensure that our pipelines and configuration would be persisted if the instance needed to reboot. We read through endless Jenkins documentation and articles. We got it working!

For a while, Jenkins was solving our problems. But what I’ve come to notice is that those of us familiar with the original setup are able to create and update the pipelines - whilst the learning curve for those new to Jenkins, CDK or CasC is often too high to justify. Every pipeline requires an as-code definition, and Jenkins is flakey - so sometimes (more often than seems reasonable) the instance will go into a crash loop without any helpful error messages. Understanding how to structure a Jenkinsfile isn’t easy, either, especially when you’re dealing with multiple AWS accounts and access roles for a deployment.

We want our CI to be quick to configure, use well-maintained tech and be easy enough for everyone to get started. With that in mind, we looked for some alternatives to Jenkins, and CircleCI became a clear contender because of its enormous free tier, comprehensive platform support, and excellent documentation.

So all that was left was to transfer our pipelines over from Jenkins to CircleCI. Swapping platforms can seem like a daunting task (AngularJS to literally anything, anyone?), but I gave this a shot and found it to be much easier than I’d expected.

In this article, I’ll introduce some basic concepts of CircleCI whilst using a real example of a pipeline we’ve transferred over from Jenkins to show you how easy it can be!

Note: This tutorial assumes that you’ve already got a CircleCI account set up and connected to the GitHub account where your projects live.

Making the Switch

1. Find that Jenkinsfile

Track down the Jenkinsfile in your project repo, and read through it so you’re familiar with what it’s trying to do. Unlike pretty much everything else Jenkins-related, the Jenkinsfile is actually pretty simple!

Below is a rough copy of the Jenkinsfile I’m working with:

As you can see, this is a Serverless project using Yarn and CDK. In the tools section, Jenkins is installing nodejs, and we’ve got the slackSend plugin configured to send Slack messages to our channels.

2. Config.yml is the new Jenkinsfile

Create a new branch to faff around with CircleCI, and make yourself a top-level .circleci folder. Whack an empty config.yml file in there. Boom 💥

Now you need some config to justify that file name - so cue that piracy ad in your head and steal this template. Go on. You know you want to.

This is actually pretty simple. At the top we define the CircleCI version, and then we install orbs for the AWS CLI and Slack. Orbs are just config packages that make our lives a lot, lot easier.

Then we get to defining our jobs. Unlike in Jenkins, these don’t just run automatically - we cue them using workflows, which are lower down in the file. Here we have one job, called deploy_api, and because of the clever workflow stuff, we can use the same one with different config to deploy both staging and production. 💃

To save ourselves some time and energy, we’re using the aws-cli/default executor, which comes with a bunch of AWS functionality pre-installed. If you’re not using AWS, you can use a nodejs version, an OS or shell - have a look at the basic options here.

Inside a job, we can define some environment variables that aren’t sensitive - be careful here, especially if your repo isn’t private. We often use this block to point to the ARN of our AWS Secrets Manager resource, where we store the proper secrets!

After the environment section, there’s the actual steps to run in your pipeline. This one checks out the branch from version control, configures the AWS profile for this project, installs all the dependencies and deploys - and then messages a slack channel to inform them of the success/failure.

3. Make it work for your project

Just borrowing my template and expecting to magically deploy won’t work for every project - you might be using a different language, framework or cloud provider.

This is basically a case of looking through your Jenkinsfile and converting each step from one syntax to the other. Here’s an example:

I’ve got this in my Jenkinsfile:

That won’t do a whole lot of good in my config.yml, but it’s easy to convert! This becomes:

in the steps section of my config.yml file. Simple, right? Go ahead and get that whole pipeline copied over.

4. Start using CircleCI

Once you’ve committed your config.yml to the repo, you’ll need to go into your CircleCI account and set up the project so you can start deploying. This is easy - assuming that CircleCI is set up for your GitHub organisation, you just need to go to Projects and pick the one you’re configuring. Then when it asks for an existing config.yml, type in the branch name where you’ve committed it and ta-da! CircleCI does the magic and you have a pipeline.

Obviously, this will immediately fail because of a significant lack of environment variables - we’ll fix that now.

5. Save the environment

An awesome thing about CircleCI is how it handles environment variables - they’re super easy to add, and automatically become available in your pipelines when they’re run.

Now we’ve got a pipeline set up, we’ll add those environment variables so it actually works!

There are two ways to do this, depending on whether the environment variables are specific to your project or shared between projects.

For config that’s shared between projects, use Contexts - find these in Organisation Settings. Each one can be referenced by name in your config.yml, and saves you writing out the same environment variables for every project. It’s great for things like AWS and Slack credentials!

For project-specific config, use the basic environment variables for that project. To do this, click into your project in CircleCI, and find Environment Variables in the Project Settings menu.

Add those variables, and try running your pipeline again!

6. Goodbye Jenkinsfile

Don’t forget to delete the Jenkinsfile from your repo - you don’t want to deploy in two places at once! 

You can also remove the job from your Jenkins instance…doesn’t that feel good?

That’s it - you should now have your project swapped over from Jenkins to CircleCI, with pipelines succeeding. 

Obviously, there’s a lot more that can be done with CircleCI than what’s been covered here, so make sure to take a look at their docs for anything else you might want to take advantage of.