CI/CD tool evaluation

Owned by Bert Viikmäe (Deactivated)
Last updated: Jan 05, 2023 by Meelis Zujev (Deactivated)
2 min read

 

Tool

Hosting

Execution environments

Integrations and plugins

Pros and cons

Pricing

Tool

Hosting

Execution environments

Integrations and plugins

Pros and cons

Pricing

Jenkins

On-premise

  • Windows

  • MacOS

  • Linux

  • Docker

  • Kubernetes

Limitless due to 1800+ plugins

Open Source

Free of charge

Platform-agnostic

Easy to set up

Poor UI

Free (+ hosting fees)

TeamCity

SaaS

  • Windows

  • MacOS

  • Linux

  • Docker

  • AWS

  • .NET

  • JIRA, Bugzilla

  • Maven

  • Visual Studio

  • 400+ plugins

Feature rich

Easy to set up

Great documentation

Steep learning curve

Manual upgrading

 

From 45$ a month

Bamboo Data Center

On-premise

  • Windows

  • MacOS

  • Linux

  • Docker

  • Kubernetes

  • JIRA

  • Bitbucket

  • Confluence

  • AWS Codedeploy

  • 185 plugins

Native integrations with Atlassian tools

Ease of deployment

Great documentation

No cloud version

High price

From 100$ a month

TravisCI

SaaS

  • Windows

  • MacOS

  • Ubuntu

  • Docker (except macOS)

  • Github

  • Gitlab

  • Assembla

  • Bitbucket

Support for 30+ programming languages

Numerous deployment options

Limited plugins

Lack of scalability

From 30$ a month

CircleCI

 

SaaS

  • WIndows

  • Linux

  • MacOS (for cloud only)

  • Docker

  • Kubernetes

  • ARM machines

  • Github

  • Bitbucket

  • JIRA

  • Slack

  • AWS

  • Google

  • Microsoft Azure

  • 150 integrations in total

Pre-built configuration packages

Great tutorials, custom services and community support

Hard-to-predict price

No visual designer tool

From 15$ a month

CodeShip

SaaS

  • AWS

  • Ubuntu

  • Docker (only for Pro version)

  • Github

  • Bitbucket

  • About 30 integrations

Role permissions

Parallel tests

Intuitive UI

One-click cross-platform deployments

Lack of documentation

Lack of integrations

From 49$ a month

Github Actions

SaaS/on-premise

Github

  • Github

Integration with GitHub

Easy to duplicate a workflow

Active development roadmap

Fast

  • BASIC FREE 2000 minutes/month

  • TEAM 4$ user/month 10000 minutes/month

  • ENTERPRISE 21$ user/month50000 minutes/month

Github minutes charged per 60s:

  • Ubuntu 1

  • Windows 2

  • MacOS 10

 

CONCLUSIONS

 

  1. In case of on-premise CI/CD solution, there is the need for self-hosting and self-management, which in case of Sandbox is unwanted, hence Jenkins and Bamboo will drop from the list of favorable tools.

  2. TeamCity can be dropped due to the steep learning curve and manual upgrading.

  3. TravisCI can be dropped due to limited plugins, lack of scalability and not so many integrations

  4. CodeShip can be dropped because of lack of documentation and lack of integrations

This leaves us with two choices for the tool: CircleCI and GitHub Actions.

When comparing these two, the CircleCI major con is hard-to-predict price, but the pros when comparing against GitHub Actions is:

  • The widest choice to customize operating systems, CPUs, GPUs, memory, and images for each job

  • CircleCI lets you customize the environment and conditions for your builds, tests, deployments, and releases

  • Securely access any job on CircleCI to debug builds and tests in real-time

  • Shorten the feedback loop by automatically splitting your tests across parallel instances of the same job

  • Supports also other VCSs: Bitbucket, GitLab

  • Faster than GitHub Actions: 2min vs 3min per build