AWS EC2 Plugin for Jenkins for jobs requiring higher capacity machine
In this article, I will go through the step by step process on how we can leverage Jenkins for building jobs that require higher capacity AWS EC2 instance or when the master node in Jenkins is loaded or to speed up building jobs.
Problem Description: Consider we have to create an build or an artifact through Jenkins's job that requires higher capacity EC2 instance. If we run the same build job in a lower volume or capacity machine, it takes more than 2 hrs to build. The problem of running a higher capacity instance for Jenkins jobs is the cost, so we need to make sure this EC2 instance starts and stops only when the build job is triggered.
Solution: In order to set up the trigger of starting the EC2 instance and stopping it, we can use a plugin in Jenkins called AWS EC2 plugin.
Steps to setup this plugin:
- In Jenkins, go to Manage Jenkins -> Manage Plugins -> Select the ‘Available’ tab
2. Search for AWS EC2 Plugin and install it
3. Once after installation and restart of Jenkins. Go to Manage Jenkins -> Configure System and move to the end of the page
Note: We can also go directly via Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds
4. Now lets set up the EC2 instance. In order to set this up, we need the following,
(All the below steps are to be done by logging into the AWS management console)
AWS Credentials (AWS IAM Access Key used to connect to EC2) : AWS IAM -> Users -> Create User -> Go to Security Credentials Tab -> Create Access Keys -> Note down the Access Key Id and .pem file downloaded
EC2 Key Pair for the user ec2-user to access the EC2 box: AWS EC2 -> Key Pairs (under Network and Security section) -> Create key Pair
AMI ID, Some of the builds need certain configuration for the job to run so pick the right AMI ID or build your AMI: AWS EC2 -> AMIs (under Image section), pick the AMI ID compatible for the build job. Note down the AMI ID
Security Group, Define a security group with the required inbound and outbound rule: AWS EC2 -> Security Groups (under Network and Security section) -> Create a security group with the inbound and outbound rule. Note down the security group name.
Subnet ID of the VPC get the subnet id of the VPC running: AWS VPC -> Subnet ID -> Note the subnet ID
IAM Instance Role: AWS IAM -> Roles -> Create a role with Administrator Access policy-> tag it to the instance role -> Note down the ARN for Instance Profile ARNs
5. Once the above step is completed, let's configure Jenkins to set this up.
Go to Manage Jenkins -> Manage Nodes and Clouds -> Configure Clouds -> The page contains a drop-down, select AWS EC2 from the list
Provide the access key and the private key created in step 4 as mentioned below,
The Labels and Usage is required for the job tagged to this instance. This means, when you are running your job, the job to pick the right instance.
Once the above data is entered and saves, the instance is ready to be launched.
In order to see if it is working, Go to Manage Jenkins -> Manage Nodes and Clouds -> Provision the EC2 created. The instance will start automatically.
How do we set a build job in Jenkins to pick this instance that we created now?
Go to Jenkins -> select the job -> click on configure, give the label name as we had given above
Now run the job, if the instance is up, build job executes immediately.
After 5 mins of the execution, if the instance is idle, the EC2 instance terminates automatically. When you try building the job again, automatically the instance gets starts after a couple of mins. (so need of any manual intervention to start the instance every time)
Happy Learning !!!
