CI/CD pipeline for application and micro-service deployment on Amazon Elastic Container Service using Docker, Ansible & Gitlab-CI

FrieslandCampina China have an external application development partner who work closely with the business units on fast moving projects with very short deadlines. Traditionally they operated on the basis of being given administrative access to servers which they configured as required and on which they deployed and tested their applications. This level of autonomy allowed them to react quickly and work without dependencies on other teams.

They were now moving to an AWS environment that is managed by a different external partner. It was necessary to provide them with the ability to work quickly with autonomy but while also ensuring adherence to design and security standards necessary for stable operations and compliance. The application development done in local development environments needs to deploy and work consistently in the AWS test, acceptance and production environments. The solution needs to be reusable across multiple projects and development team with minimal adjustments.

Solution

There was a need for more sophisticated practices in their application development life-cycle to enable them in the AWS environment. We needed to provide them with the ability to configure their application environment and deploy their applications with minimal or no input from the DevOPs team.

It was decided to use containerization for consistent deployment and functionality of their application across different environments. We enabled them to do automated build of Docker images, Java source code compiling and application deployment, initiated by development team commits to code repositories.

AWS CloudFormation is used to deploy and perform ongoing configuration management of the AWS VPC network, AWS Identity & Access Management and separate Amazon Elastic Container Service clusters for the Production and for the Test & Acceptance environments.

As it was not allowed to use external Git repository services and because we needed certain features of Gitlab CI to facilitate code compiling and automated deployments we deployed Gitlab & Gitlab-CI on EC2. Additonally we deployed an AWX (Ansible opensource) server with Maven for compiling Java applications, building docker images and doing docker deployments.

The development team were provided with suitable docker files templates, which they can edit to configure their application environment. We defined and communicated a standard process for committing Docker files and or application code updates, so as to initiate a deployment in test, acceptance or production.

• There is an approval step to launch the deployment, as requested by the development team.
• Approval of merge requests into the ‘master’ branch for Production is access controlled in Git for a limited number of authorized persons.

Outcomes

• Development team can now work independently of DevOPs team and other teams on application development, testing and deployment, but a consistent standard is being maintained for operational and compliance purpose.
• New application stack deployment times have been reduced from weeks of cooperative engagement to hours of self-service activity.
• Billable deployment requests from the development team to the DevOPs team have been reduced to zero for projects that fit in this model.
• Business teams are now enabled for final approval on deployments to production.
• We have overcome language and culture barriers to servicing a remote and Mandarin speaking development team, by enabling them for self service with DevOPs technologies.
• The pipeline and process is reusable for other development teams.

Lessons learned

• Gitlab-CI has features and functionality that can be successfully be used to displace the need for a Jenkins server.