Spot Fleet Termination and Remove Stale Route53 Entries

When it comes to Cluster management, we may need many slaves (machines) which will run our tasks/applications. Similarly, in our project, we do have Apache Mesos Cluster which nearly runs around 200 EC2 instances in production itself and for real-time data processing, we use Apache Storm cluster which has supervisors machine count around 150. All these machines rather than running on on-demand we run on Spotfleet in AWS.

Now the question comes what is spotfleet? To understand SpotFleet first look what is Spot Instances.

Spot Instances

AWS must maintain a huge infrastructure with a lot of unused capacity. This unused capacity is basically the available spot instance pool – AWS lets users bid for these unused resources (usually on a significantly lower price than the on-demand price). So we can get AWS ec2 boxes at a much lower price as compared to their on-demand price.

SpotFleet

A Spot Fleet is a collection, or fleet, of Spot Instances, and optionally On-Demand Instances. The Spot Fleet attempts to launch the number of Spot Instances and On-Demand Instances to meet the target capacity that you specified in the Spot Fleet request.

Above is the screenshot of AWS SpotFleet, in which we are launching 20 instances.

Spot instance lifecycle:

  • User submits a bid to run the desired number of EC2 instances of a particular type. The bid includes the price that the user is willing to pay to use the instance.
  • If the bid price exceeds the current spot price (that is determined by AWS based on current supply and demand) the instances are started.
  • If the current spot price rises above the bid price or there are no available capacity, the spot instance is interrupted and reclaimed by AWS. 2 minutes before the interruption the internal metadata endpoint on the instance is updated with the termination info.

Spot instance termination notice

The Termination Notice is accessible to code running on the instance via the instance’s metadata at http://169.254.169.254/latest/meta-data/spot/termination-time. This field becomes available when the instance has been marked for termination and will contain the time when a shutdown signal will be sent to the instance’s operating system.

The most common way discussed to detect that the Two Minute Warning has been issued is by polling the instance metadata every few seconds. This is available on the instance at:

http://169.254.169.254/latest/meta-data/spot/termination-time

This field will normally return a 404 HTTP status code but once the two-minute warning has been issued, it will return the time that shutdown will actually occur.

This can only be accessed from the instance itself, so you have to put this code on every spot instance that you are running. A simple curl to that address will return the value. You might be thinking to set up a cron job, but do not go down that path. The smallest interval you can run something with cron is once a minute. If you miss it by a second or two you are not going to detect it until the next minute and you lose half of the time available to you.

 Sample Snippet:

Below is alert that we receive in SLACK.

Delete Route53 Entries

We do create DNS entries for all our mesos and storm boxes, but whenever those instances get deleted their DNS entries still remain there, which cause lots of entries under Route53 which are of no use. So, we come up with an idea why not have a lambda function that will be triggered whenever a spotfleet instance got terminated.

Created a cloudwatch rule

So, whenever an instance got terminated this rule run and it triggers a Lambda function, which deletes the route53 of the terminated instance.

Below is the code snippet:

For each mesos and storm Instance, we do have Tagging and that tagging we use to destroy entries

Conclusion:

From Spot instances Termination we get a two-minute warning that they are about to be reclaimed by AWS. These are precious moments where we can have the instance deregister from accepting any new work, finish up any work in progress. Apart from these,  using lambda function we can remove stale route53 entries.

ELK Setup And Email Alerting

What’s ELK Stack?

“ELK” is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana.

Introduction: ELK services:

Elasticsearch: It’s a search and analytics engine.

Logstash: It’s a server‑side data processing pipeline that is for centralized logging, log enrichment and parsing

Kibana: It lets users visualize data with charts and graphs in Elasticsearch.

Note: I will be using Amazon EC2 instances with Ubuntu 16.04 LTS operating system for whole ELK and alerting setup.

We need Java8 for this setup first!!!

  • Setup Java8 on Ubuntu EC2 server by following commands:

$ sudo add-apt-repository -y ppa:webupd8team/java

$ sudo apt-get update -y

$ sudo apt-get -y install oracle-java8-installer

Lets Start with ELK services setup !!!

1) Setup Elasticsearch in our Ubuntu EC2 servers:

  • Download and install the public signing key:

$ wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

  • Installing from the APT repository

$ sudo apt-get install apt-transport-https

  • Save the repository definition to /etc/apt/sources.list.d/elastic-5.x.list:

$ echo “deb https://artifacts.elastic.co/packages/5.x/apt stable main” | sudo tee -a  /etc/apt/sources.list.d/elastic-5.x.list

  • Finally install Elasticsearch

$ sudo apt-get update && sudo apt-get install elasticsearch

  • After installing Elasticsearch , do following changes in configuration file “elasticsearch.yml”:
  • Changes in configuration file:

$ sudo vim /etc/elasticsearch/elasticsearch.yml

“network.host” to localhost

  • After saving and exiting file:

$ sudo systemctl restart elasticsearch

$ sudo systemctl daemon-reload

$ sudo systemctl enable elasticsearch

 

2) Setup Kibana in our Ubuntu EC2 servers:

  • Download and install the public signing key:

$ wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

  • Installing from the APT repository

$ sudo apt-get install apt-transport-https

  • Save the repository definition to /etc/apt/sources.list.d/elastic-5.x.list:

$ echo “deb https://artifacts.elastic.co/packages/5.x/apt stable main” | sudo tee -a  /etc/apt/sources.list.d/elastic-5.x.list

  • Finally install Kibana

$ sudo apt-get update && sudo apt-get install kibana

  • After installing Kibana, do following changes in configuration file “kibana.yml”:
  • For doing changes in cnfiguration file:

$ sudo vim /etc/kibana/kibana.yml

“server.host” to localhost.

  • After saving and exiting file:-

$ sudo systemctl restart kibana

$ sudo systemctl daemon-reload

$ sudo systemctl enable kibana

 

3) Setup Nginx for Kibana UI:

  • Following commands to install nginx:

$ sudo apt-get -y install nginx

$ echo “kibanaadmin:`openssl passwd -apr1`” | sudo tee -a /etc/nginx/htpasswd.users

  • Change nginx configuration file to listen at port 80 for Kibana:

$ sudo nano /etc/nginx/sites-available/default

  • Add the following in the file and save and exit file:
server {

listen 80;

server_name example.com;

auth_basic "Restricted Access";

auth_basic_user_file /etc/nginx/htpasswd.users;

location / {

proxy_pass http://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

}

 

  • For checking Nginx configuration:-

$ sudo nginx -t

$ sudo systemctl restart nginx

$ sudo ufw allow ‘Nginx Full’

 

4) Setup Filebeat on a different EC2 server with Amazon Linux image, from where logs will come to ELK:

  • Following commands to install filebeat:

$ sudo yum install filebeat

$ sudo chkconfig –add filebeat

  • Changes in Filebeat config file, here we can add different types of logs [ tomcat logs, application logs, etc] with their paths:-

filebeat:

prospectors:

paths:

– /var/log/tomcat/

input_type: log

document_type: tomlog

registry_file: /var/lib/filebeat/registry

output:

logstash:

hosts: [“elk_server_ip:5044”]

bulk_max_size: 1024

  • After changes done, save and exit the file and restart filebeat service with:-

$ service filebeat restart

$ service filebeat status [It should be running].

5) Setup Logstash in our ELK Ubuntu EC2 servers:

  • Following commands via command line terminal:

$ sudo apt-get update && sudo apt-get install logstash

$ sudo systemctl start logstash.service

  • Logstash Parsing of variables to show in Kibana:-

Make file in /etc/logstash/conf.d as “tomlog.conf” and add the following:

input {

beats {

port => 5044

ssl => false

}

}

filter {

if [type] == "tomlogs" {

grok {

patterns_dir => ["./patterns"]

match => { "message" => "%{IPORHOST:IP} - - %{SYSLOG5424SD:timestamp}%{SPACE}%                                                              {NOTSPACE:deviceId}%{SPACE}%{NOTSPACE:sessionId}%{SPACE}%                                                                                  {NOTSPACE:latitude}%{SPACE}%{NOTSPACE:longitude}%{SPACE}%                                                                                  {QUOTEDSTRING:query_string}%{SPACE}%{NOTSPACE:response}%{SPACE}%                                                                {NOTSPACE:byte_sent}%{SPACE}%{NOTSPACE:response_in_milli}" }

}

}

}

output {

elasticsearch {

hosts => ["localhost:9200"]

sniffing => true

manage_template => false

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

document_type => "%{[@metadata][type]}"

}

}
  • For testing Logstash config:

$ sudo /usr/share/logstash/bin/logstash –configtest -f /etc/logstash/conf.d/

  • After that restart all ELK services:

$ service logstash restart

$ service elacticsearch restart

$ service kibana restart

 

6) Browser: Kibana UI setup:

  • To load Kibana UI:

Search: http://elk_server_public_ip

  • Then add username and password for Kibana: admin/change_password.
  1. Go to Management: Add index: filebeat-* .
  2. Discover page should now show your system logs parsed under filebeat-* index.

7) Setup Elastalert for Email Alerting system:

  • SSH again in ELK EC2 Ubuntu server and do following:
  • Add cron in /etc/crontab:-
00 7    * * *   ubuntu    cd /home/ubuntu/elastalert && ./run_elastalert_with_cron.sh
  • Install elastalert on Ubuntu server:

$ git clone https://github.com/Yelp/elastalert.git

$ pip install “setuptools>=11.3”

$ python setup.py install

$ elastalert-create-index

  • Create rule:

$ cd  ~/elastalert/rules && touch rule.yml

$ cd ~/elastalert

$ vim config.yml

  • Add following for main config.yml file:

es_host: “localhost”

es_port: 9200

name: “ELK_rule_twiin_production”

type: “frequency”

index: “filebeat-*”

num_events: 1

timeframe:

hours: 24

filter:

– query:

query_string:

query: ‘type:syslogs AND SYSLOGHOST:prod_elk_000’

alert:

– “email”

realert:

minutes: 0

email: “example@email.com

  • Changes in rules/rule.yml:

rules_folder: rules

run_every:

minutes: 30

buffer_time:

hours: 24

es_host: ‘localhost’

es_port: 9200

writeback_index: elastalert_status

smtp_host: ‘localhost’

smtp_port: 25

alert_time_limit:

days: 0

from_addr: “your_addr@domain.com

 

  • Cron script for running Elkastalert to send Email once a day:-

 

#!/bin/bash

curl -X GET “http://localhost:9200/elastalert_status

curl -X DELETE “http://localhost:9200/elastalert_status

pkill -f elastalert.elastalert

cd /home/ubuntu/elastalert

cd /home/ubuntu/elastalert && python -m elastalert.elastalert –config config.yaml –rule rules/frequency.yaml –verbose

Save and exit Shell script.

Now, it will run daily at 7:00 UTC according to my cron setup in the first step!!!

Let’s have a demo !!!

1) SSH to EC2 server [my case its Amazon Linux machine] where logs are getting generated:-

  • Go to the directory where we installed Filebeat service to create indices of our logs:

$ cd /etc/filebeat

$ vi filebeat.yml

  • Put your ELK server’s IP address for getting output in that server:-

  • Logs for Filebeat service will be stored at location: /var/log/filebeat:

$ cd /var/log/filebeat

$ ls –ltrh

  • The logs we need for indexing via Filebeat will be our tomcat logs, we can have them anywhere:-
  • In my case , they are at location: /var/log/containers/:

  • Now, let’s jump to ELK server to check how indices are coming there:-
  • Check the status of all ELK services:-

  • Go to Logstash conf directory to see the parsing of logs:

$ cd /etc/logstash/conf.d

$ vi tomlog.conf

  • Logs for Logstash service , will be at location: /var/log/logstash:

  • Now let’s check our Elasticsearch conf, at location /etc/elasticsearch:
  • You can define indices storage location and host in elasticsearch.yml:-

  • We can check elasticsearch services’s logs at /var/log/elasticsearch:

  • Lets checkout Kibana conf at location /etc/kibana:
  • We just need to define the host for Kibana UI:

  • Now, let’s move to Kibana UI

In browser: http://public_ip_elk_server

  • Enter username and password for Kibana, then it all starts:

  • Once you are logged in, you will go to Management:

  • Create the index for logs, for me, it’s filebeat-*:

  • As soon we create index pattern: All parsed fields will come there with details as

  • Then we can search our query normally and add fields we want to check values of as:

  • Let’s check, how to create a visualization in Kibana:

  • We can choose our desired graph pattern:

  • I am choosing Vertical bar:

  • Now for graphs, we need parameters to be framed in X-Y axis:
  • Let’s make a new search and save it first for Visualizations:

My search is:- “type:tomlogs AND response:200”:

  • Now save the search in Kibana for further making visualizations:

  • Now go to the Visualization board again:

  • Now we can select our saved search to make visualizations:
  • We selected our Y-axis to be a field: Response with Aggregation: Average:

  • Now save your visualization:

  • Now I made more responses to make a dashboard further:

  • Now let’s jump to Kibana Dashboards:

  • Create a dashboard:
  • As we start, Kibana will ask to add visualizations:

  • I chose both of my visualizations:

  • Save your dashboard:

  • Now you can browse more out of your dashboard according to Time Range at right top corner:

  • Just for little insight of Dev Tools:
  • We can query using APIs for our mappings or templates or Indices:
  • Let’s have all indices for the query as:

Hence, we have covered almost all features of Kibana !!!

Conclusion:

The ELK Stack is a fantastic piece of software. ELK stack is most commonly used in log analysis in IT environments (though there are many more use cases for the ELK Stack starting including business intelligence, security and compliance, and web analytics).

Logstash collects and parses logs, and then Elasticsearch indexes and stores the information.

Kibana then presents the data in visualizations that provide actionable insights into one’s environment.

Custom Debian package creation method

This blog explains how to create a custom package in Debian OS. We will see how to configure a customize Debian package and create customize command.

Debian package creation in Ubuntu/Debian:

We will create the helloworld package without any command and with command like e.g.- ipaddr (to get external ip)

Create a debian package:

Let’s start creating empty directory helloworld. This folder will contain both source code and the Debian build instructions.

1.1 Create the debian/ Directory

Here we will use dh_make command to create the debian/ Directory:

$ mkdir -p Debian/helloworld-1.0
$ cd Debian/helloworld-1.01.    Create debian/ folder with example files (.ex)
$ dh_make \
–native \
–single \
–packagename helloworld_1.0.0 \
–email mukesh.waghadhare@talentica.com

This created the debian/ folder. Explore them! Especially the example files (*.ex) as well as most importantly the files:

  • debian/control
  • debian/changelog
  • debian/rules

1.2 Build the first (empty) Package

The dpkg-buildpackage command can be used to build  first package:

$ dpkg-buildpackage -uc -us

(-us, –unsigned-source Do not sign the source package (long option since dpkg 1.18.8).

-uc, –unsigned-changes Do not sign the .changes file (long option since dpkg 1.18.8).)

That’s it! This command build four files:

root@ip-172-31-19-221:/final_dpkg# ls -lrth

-rw-r–r– 1 root root 7.2K Jun 20 10:47 helloworld_1.0.0.tar.xz

-rw-r–r– 1 root root 573 Jun 20 10:47 helloworld_1.0.0.dsc

-rw-r–r– 1 root root 2.0K Jun 20 10:47 helloworld_1.0.0_amd64.deb

-rw-r–r– 1 root root 4.7K Jun 20 10:47 helloworld_1.0.0_amd64.buildinfo

-rw-r–r– 1 root root 1.5K Jun 20 10:47 helloworld_1.0.0_amd64.changes

  • .tar.gz: Source package, contains the contents of the helloworld/ folder
  • .deb: Debian package, contains the installable package
  • .dsc/.changes: Signature files, cryptographic signatures of all files

Let’s examine the contents with the dpkg -c command:

root@ip-172-31-19-221:/final_dpkg# dpkg -c helloworld_1.0.0_amd64.deb
drwxr-xr-x root/root 0 2018-06-20 09:03 ./
drwxr-xr-x root/root 0 2018-06-20 09:03 ./usr/
drwxr-xr-x root/root 0 2018-06-20 09:03 ./usr/share/
drwxr-xr-x root/root 0 2018-06-20 09:03 ./usr/share/doc/
drwxr-xr-x root/root 0 2018-06-20 09:03 ./usr/share/doc/helloworld/
-rw-r–r– root/root 187 2018-06-20 09:03 ./usr/share/doc/helloworld/README.Debian
-rw-r–r– root/root 144 2018-06-20 09:03 ./usr/share/doc/helloworld/changelog.gz
-rw-r–r– root/root 1413 2018-06-20 09:03 ./usr/share/doc/helloworld/copyright
root@ip-172-31-19-221:/final_dpkg#

Nothing really in the package except the default changelog, copyright and README file.

1.3 Install the (empty) package

Let’s install it with dpkg -i command:

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg -i ../helloworld_1.0.0_amd64.deb
(Reading database … 49055 files and directories currently installed.)
Preparing to unpack ../helloworld_1.0.0_amd64.deb …
Unpacking helloworld (1.0.0) over (1.0.0) …
Setting up helloworld (1.0.0) …
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0#

Done. That installed the package. Check out that it was actually installed by listing the installed packages with dpkg -l

# Proof that it’s actually installed

$ dpkg -l | grep helloworld

ii  helloworld       1.0.0        amd64  <insert up to 60 chars description>

# ^^     ^                      ^            ^           ^

# ||       |                       |            |            |

# ||       |                       |            |             – Description

# ||       |                       |             – Architecture

# ||       |                        – Version

# ||       – Package name

# | – Actual/current package state (n = not installed, i = installed, …)

#  – Desired package state (i = install, r = remove, p = purge, …)

The columns in the list are desired package stateactual package statepackage namepackage versionpackage architecture and a short package description. If all is well, the first column will contain ii, which means that the package is properly installed.

Now that the package is installed, you can list its contents with the dpkg -L

root@ip-172-31-19-221:/final_dpkg# dpkg -L helloworld
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/helloworld
/usr/share/doc/helloworld/README.Debian
/usr/share/doc/helloworld/changelog.gz
/usr/share/doc/helloworld/copyright
root@ip-172-31-19-221:/final_dpkg#

2.0 Adding files and updating the changelog

Let’s now add actual files in empty package. To do that, let’s create a folder file/usr/bin and add scripts to check IP address under ipaddr file.

$ mkdir -p files/usr/bin

$ touch files/usr/bin/ipaddr

$ chmod +x files/usr/bin/ipaddr

$ vi files/usr/bin/ipaddr

# Script contents see below

For this demo, we’ll use a short script to grab the public IP address from a service called ipify. It provides an API to return the IP address in various formats. We’ll grab it with curl in JSON and then use jq to parse out the ‘ip’ field:

#!/bin/bash

curl –silent ‘https://api.ipify.org?format=json’ | jq .ip –raw-output

If we were to rebuild the package now, the dpkg-buildpackage command, we wouldn’t know which files to include in the package. So we’ll create the Debian/install file to list directories to include (e.g. vi Debian/install):

files/usr/* usr

This basically means that everything in the files/usr/ folder will be installed at /usr/ on the target file system when the package is installed.

Once this is done, we can just rebuild and reinstall the package, but let’s go one step further and update the version of the package to 1.1.0. Versions are handled by the Debian/changelog file. You can update it manually, or use the dchscript (short for “Debian changelog”) to do so:

# We changed the package, let’s update the version and changelog for it to 1.0.3

$ dch -im

# Opens the editor …

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# cat debian/changelog

helloworld (1.0.1) unstable; urgency=medium

* Added ‘ipaddr’ script
*

— root <mukesh.waghadhare@talentica.com> Thu, 21 Jun 2018 07:19:02 +0000

helloworld (1.0.0) unstable; urgency=medium

* Initial Release.

— root <mukesh.waghadhare@talentica.com> Wed, 20 Jun 2018 09:03:21 +0000
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0#

Let’s rebuild and reinstall the package:

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg-buildpackage -uc -us

(-us, –unsigned-source Do not sign the source package (long option since dpkg 1.18.8).

-uc, –unsigned-changes Do not sign the .changes file (long option since dpkg 1.18.8).)

Looks like it installed correctly. Let’s check if the ipaddr script is where it’s supposed to be. And then let’s try to run it:

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg -l | grep helloworld
ii helloworld 1.0.1 amd64 <insert up to 60 chars description>
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# which ipaddr
/usr/bin/ipaddr
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# ipaddr
/usr/bin/ipaddr: line 2: jq: command not found
(23) Failed writing body

2.1 Updating description and adding dependencies

Each Debian package can depend on other packages. In the case of our ipaddr script, we use the curl and jqcommands, so the ‘helloworld’ package depends on these commands.

To add the dependencies to the ‘helloworld’ package, edit the Depends: section in the debian/control file (e.g. via vi debian/control):

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# vim debian/control
Source: helloworld
Section: utils
Priority: optional
Maintainer: mukeshw <mukesh.waghadhare@talentica.com>
Build-Depends: debhelper (>= 9)
Standards-Version: 3.9.8
Homepage: <insert the upstream URL, if relevant>Package: helloworld
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}, curl, jq
Description: Network management tools
Includes various tools for network management.
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0#

Again update the version and rebuild the package

Change the version:

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dch -im

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# cat debian/changelog
helloworld (1.0.3) stable; urgency=medium

* Updated description and depndancy

— root <mukesh.waghadhare@talentica.com> Thu, 21 Jun 2018 07:19:48 +0000

helloworld (1.0.2) unstable; urgency=medium

* Added ‘ipaddr’ script

— root <mukesh.waghadhare@talentica.com> Thu, 21 Jun 2018 07:19:02 +0000

helloworld (1.0.0) unstable; urgency=medium

* Initial Release.

— root <mukesh.waghadhare@talentica.com> Wed, 20 Jun 2018 09:03:21 +0000
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0#

Rebuild:

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg-buildpackage -uc -us

Reinstall:

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg -i ../helloworld_1.0.3_amd64.deb
(Reading database … 49056 files and directories currently installed.)
Preparing to unpack ../helloworld_1.0.3_amd64.deb …
Unpacking helloworld (1.0.3) over (1.0.1) …
dpkg: dependency problems prevent configuration of helloworld:
helloworld depends on jq; however:
Package jq is not installed.

dpkg: error processing package helloworld (–install):
dependency problems – leaving unconfigured
Errors were encountered while processing:
helloworld
root@ip-172-31-19-221:/final_dpkg/helloworld-1.0#

unlike apt-get install, the dpkg -i command does not automatically resolve and install missing dependencies. It just highlights them. That is perfectly normal and expected. In fact, it gives us the perfect opportunity to check the package state (like we did above):

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg -l | grep helloworld
iU  helloworld 1.0.3 amd64 Network management tools# ^^                                           ^# ||                                             |# ||                                             – Yeyy, the new description# | – Actual package state is ‘U’ / ‘Unpacked’#  – Desired package state is ‘i’ / ‘install’

As you can see by the output, the desired state for the package is ‘i’ (= installed), but the actual state is ‘U’ (= Unpacked). That’s not good. Luckily though, dependencies can be automatically resolved by apt-get install -f

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# apt-get install -f
Reading package lists… Done
Building dependency tree
Reading state information… Done
Correcting dependencies… Done
The following additional packages will be installed:
jq libjq1 libonig4
The following NEW packages will be installed:
jq libjq1 libonig4
0 upgraded, 3 newly installed, 0 to remove and 28 not upgraded.
1 not fully installed or removed.
Need to get 327 kB of archives.
After this operation, 1,157 kB of additional disk space will be used.
Do you want to continue? [Y/n] y

Processing triggers for man-db (2.7.6.1-2) …

Setting up jq (1.5+dfsg-1.3) …

Setting up helloworld (1.0.3) …

Finally, its installed correctly. Now, let’s test it…

root@ip-172-31-19-221:/final_dpkg/helloworld-1.0# dpkg -l | grep helloworld
ii helloworld 1.0.3 amd64 Network management toolsroot@ip-172-31-19-221:/final_dpkg/helloworld-1.0# ipaddr
18.184.205.204

Conclusion:

This method makes it easy to install dependencies and multiple packages which we might require at bootup. It also installs a cluster of dependencies and customized packaged with the help of just one customized package.

AWS Batch Jobs

What is batch computing?

Batch computing means running jobs asynchronously and automatically, across one or more computers.

What is AWS Batch Job?

AWS Batch enables developers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (for example, CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as Amazon EC2 and Spot Instances.

Why use AWS Batch Job ?

  • Fully managed infrastructure – No software to install or servers to manage. AWS Batch provisions, manages, and scales your infrastructure.
  • Integrated with AWS – Natively integrated with the AWS Platform, AWS Batch jobs can easily and securely interact with services such as Amazon S3, DynamoDB, and Recognition.
  • Cost-optimized Resource Provisioning – AWS Batch automatically provisions compute resources tailored to the needs of your jobs using Amazon EC2 and EC2 Spot.

AWS Batch Concepts

  • Jobs
  • Job Definitions
  • Job Queue
  • Compute Environments

Jobs

Jobs are the unit of work executed by AWS Batch as containerized applications running on Amazon EC2. Containerized jobs can reference a container image, command, and parameters or users can simply provide a .zip containing their application and AWS will run it on a default Amazon Linux container.

$ aws batch submit-job –job-name poller –job-definition poller-def –job-queue poller-queue

Job Dependencies

Jobs can express a dependency on the successful completion of other jobs or specific elements of an array job.

Use your preferred workflow engine and language to submit jobs. Flow-based systems simply submit jobs serially, while DAG-based systems submit many jobs at once, identifying inter-job dependencies.

Jobs run in approximately the same order in which they are submitted as long as all dependencies on other jobs have been met.

$ aws batch submit-job –depends-on 606b3ad1-aa31-48d8-92ec-f154bfc8215f …

Job Definitions

Similar to ECS Task Definitions, AWS Batch Job Definitions specify how jobs are to be run. While each job must reference a job definition, many parameters can be overridden.

Some of the attributes specified in a job definition are:

  • IAM role associated with the job
  • vCPU and memory requirements
  • Mount points
  • Container properties
  • Environment variables
$ aws batch register-job-definition –job-definition-name gatk –container-properties …

Job Queues

Jobs are submitted to a Job Queue, where they reside until they are able to be scheduled to a compute resource. Information related to completed jobs persists in the queue for 24 hours.

$ aws batch create-job-queue –job-queue-name genomics –priority 500 –compute-environment-order …

 

Compute Environments

Job queues are mapped to one or more Compute Environments containing the EC2 instances that are used to run containerized batch jobs.

Managed (Recommended) compute environments enable you to describe your business requirements (instance types, min/max/desired vCPUs, and EC2 Spot bid as x % of On-Demand) and AWS launches and scale resources on your behalf.

We can choose specific instance types (e.g. c4.8xlarge), instance families (e.g. C4, M4, R3), or simply choose “optimal” and AWS Batch will launch appropriately sized instances from AWS more-modern instance families.

Alternatively, we can launch and manage our own resources within an Unmanaged compute environment. Your instances need to include the ECS agent and run supported versions of Linux and Docker.

$ aws batch create-compute-environment –compute- environment-name unmanagedce –type UNMANAGED …

AWS Batch will then create an Amazon ECS cluster which can accept the instances we launch. Jobs can be scheduled to your Compute Environment as soon as the instances are healthy and register with the ECS Agent.

Job States

Jobs submitted to a queue can have the following states:

  • SUBMITTED: Accepted into the queue, but not yet evaluated for execution
  • PENDING: The job has dependencies on other jobs which have not yet completed
  • RUNNABLE: The job has been evaluated by the scheduler and is ready to run
  • STARTING: The job is in the process of being scheduled to a compute resource
  • RUNNING: The job is currently running
  • SUCCEEDED: The job has finished with exit code 0
  • FAILED: The job finished with a non-zero exit code or was cancelled or terminated.

AWS Batch Actions

  • Jobs: SubmitJob, ListJobs, DescribeJobs, CancelJob, TerminateJob
  • Job Definitions: RegisterJobDefinition, DescribeJobDefinitions, DeregisterJobDefinition
  • Job Queues: CreateJobQueue, DescribeJobQueues, UpdateJobQueue, DeleteJobQueue
  • Compute Environments: CreateComputeEnvironment, DescribeComputeEnvironments, UpdateComputeEnvironment, DeleteComputeEnvironment

AWS Batch Pricing

There is no charge for AWS Batch. We only pay for the underlying resources we have consumed.

Use Case

Poller and Processor Service

Purpose

Poller service needs to be run every hour like a cron job which submits one or more requests to a processor service which has to launch the required number of EC2 resource, process files in parallel and terminate them when done.

Solution

We plan to go with Serverless Architecture approach instead of using the traditional beanstalk/EC2 instance, as we don’t want to maintain and keep running EC2 server instance 24/7.

This approach will reduce our AWS billing cost as the EC2 instance launches when the job is submitted to Batch Job and terminates when the job execution is completed.

Poller Service Architecture Diagram

Processor Service Architecture Diagram

First time release

For Poller and Processor Service:

  • Create Compute environment
  • Create Job queue
  • Create Job definition

To automate above resource creation process, we use batchbeagle (for Installaion and configuration, please refer batch-deploymnent repository)

Command to Create/Update Batch Job Resources of a Stack (Creates all Job Descriptions, Job Queues and Compute Environments)

beagle -f stack/stackname/servicename.yml assemble

To start Poller service:

  • Enable a Scheduler using AWS CloudWatch rule to trigger poller service batch job.

Incremental release

We must create a new revision of existing Job definition environment which will point to the new release version tagged ECR image to be deployed.

Command to deploy new release version of Docker image to Batch Job (Creates a new revision of an existing Job Definition)

 

beagle -f stack/stackname/servicename.yml job update job-definition-name

Monitoring

Cloudwatch Events

We will use AWS Batch event stream for CloudWatch Events to receive near real-time notifications regarding the current state of jobs that have been submitted to your job queues.

AWS Batch sends job status change events to CloudWatch Events. AWS Batch tracks the state of your jobs. If a previously submitted job’s status changes, an event is triggered. For example, if a job in the RUNNING status moves to the FAILED status.

We will configure an Amazon SNS topic to serve as an event target which sends notification to lambda function which will then filter out relevant content from the SNS message (json) content and beautify it and send to the respective Environment slack channel .

CloudWatch Event Rule → SNS Topic → Lambda Function → Slack Channel

Batch Job Status Notification in Slack

Slack notification provides the following details:

  • Job name
  • Job Status
  • Job ID
  • Job Queue Name
  • Log Stream Name

Nuances of Riemann

Introduction

Riemann is a next generation Open-Source Push-Based monitoring system, which is designed to scale vertically and horizontally with the infrastructure, making it a good choice for monitoring elastic infrastructure. Riemann provides transient shared state for system with many moving parts. Riemann aggregates events from the servers and applications with a powerful stream processing language.   Continue reading Nuances of Riemann

Boot2Docker for Windows

Development environment – It can take too long to set it up

Have you ever thought that how long does it take for an entry level developer to setup a new project development environment? The answer depends on many factors  like the project age, the number of developers worked on it etc. but don’t get surprised if you get answer half a day or more.

Hey! It should be much faster than that: checkout a script and execute it. That’s all. Two steps should be sufficient to setup your environment and get ready for development.

 

Continue reading Boot2Docker for Windows