DNS routing traffic for subdomains using AWS Route 53

Long explanation in AWS Routing traffic for subdomains – Amazon Route 53

GROUND RULE or RULE OF THUMB of DNS, NEVER change the NS record of your domain in your Domain registrar before completing setup of your DNS server. The result may cause 72 hours of outtage.

Using my domain karmeng.my as an example, the objective is to have chow.karmeng.my to be a valid domain for email hosting which offers customed domain.

Continue reading

Working with AWS EC2 burstable Instances

Technically incline folks may choose to use the “free” tier for the first 2 months from the AWS lightsail. Free comes at a cost of burstable CPU consumption. User of lightsail needs to get familiar with the overview of the Average CPU Utilization and Remaining CPU burst capacity as shown in the screenshot below.

Once the AWS EC2 burstable instance started, AWS will add a small amount of burstable credit as long as the EC2 instance CPU usage are not exceeding 10%. If deployment of software into the instances uses CPU exceeds 10%, the CPU burst capacity will be deducted. Remaining CPU burst at 0% capacity will cause capping of the performance for the EC2 instance.

In order for the CPU burst capacity the build to 100%, the EC2 must CPU utilization must not exceed 10%. This will the CPU burst capacity to accumulate to 100% capacity, over the period of 27 hours.

Therefore, before developers or system architects needs to understand this feature of EC2 burstable CPU, allow ample time for the CPU burst capacity to build up. In real world, when deploying applications, the software provisioning may utilized all the CPU burst capacity and causes bad user experience of software performance in the AWS cloud. If I were to be an enterprise solutions architect, I would not use EC2 that has burstable CPU capacity.

How to custom install AWSCli into linux based machine

Quote

Good new for AWS users, Amazon has release the new unified AWSCli was released in September 2013. Amazon did provide multiple ways to have the new AWSCli installed.

I have to admit the task of installation is more straight forward and simplified compared to the old AWS Cli.

At the time of this post, the AWSCli released version 1.2.6, and it runs on Python 2.6. Hence, this post will provide custom install of AWSCli into linux based machine.

For users who are planning to use the AWSCli bundle provided by Amazon here is the recommended steps in sequence. Disclaimer and note : I have not added any form of error catching or linux distro detection and I am assuming the linux distro used is redhat.

Installing AWSCli using the Amazon awscli-bundle

mkdir -p /opt/apps/tmp
cd /opt/apps/tmp
wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
unzip awscli-bundle.zip
mkdir -p /opt/apps/$(ls awscli-bundle/packages/ | egrep -o 'awscli-[0-9]\.[0-9]\.[0-9]')
./awscli-bundle/install -i /opt/apps/$(ls awscli-bundle/packages/ | egrep -o 'awscli-[0-9]\.[0-9]\.[0-9]')
/opt/apps/awscli/bin/aws --version
ln -s /opt/apps/$(ls awscli-bundle/packages/ | egrep -o 'awscli-[0-9]\.[0-9]\.[0-9]') /opt/apps/awscli
ln -s /opt/apps/awscli/bin/aws /usr/bin/aws
ln -s /opt/apps/awscli/bin/aws.cmd /usr/bin/aws.cmd
cd ~
rm -Rf /opt/apps/tmp

Installing AWS via pip

python --version
apt-get install python-pip
yum install python-pip
cd /opt/apps/
mkdir tmp
cd tmp
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python ez_setup.py
python get-pip.py
pip install awscli==1.2.6
aws help
cd ~
rm -Rf /opt/apps/tmp

The advantage of AWSCli bundle over the pip method is, ease of install without need to get ez_setup and pip installed. Since the AWSCli bundle zip is hosted within Amazon Web Service network, it took me less than 1 seconds to download the 5MB AWSCli-bundle.zip file.

[root@ip-10-255-255-1 ~]# time wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
--2013-11-28 05:25:16-- https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
Resolving s3.amazonaws.com... 176.32.99.46
Connecting to s3.amazonaws.com|176.32.99.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5139105 (4.9M) [application/zip]
Saving to: `awscli-bundle.zip'

100%[================================================================================================================
===================================================================================>] 5,139,105 16.0M/s in 0.3s

2013-11-28 05:25:17 (16.0 MB/s) - `awscli-bundle.zip' saved [5139105/5139105]
real 0m0.611s
user 0m0.096s
sys 0m0.036s
[root@ip-10-255-255-1 ~]# time wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
--2013-11-28 05:25:49-- https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
Resolving s3.amazonaws.com... 176.32.99.46
Connecting to s3.amazonaws.com|176.32.99.46|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5139105 (4.9M) [application/zip]
Saving to: `awscli-bundle.zip'

100%[================================================================================================================
===================================================================================>] 5,139,105 19.6M/s in 0.2s

2013-11-28 05:25:49 (19.6 MB/s) - `awscli-bundle.zip' saved [5139105/5139105]
real 0m0.338s
user 0m0.076s
sys 0m0.036s

The downside of using the awscli bundle installation is users need to upload the awscli-bundle.zip into personal version control servers/services (such as github) in order to have version control of awscli. Therefore, there will be overhead of maintaining the version of the awscli and it will be labor intensive or makes processes complicated.

The only disadvantage of pip AWSCli is the pre-requisite of installing pip. And maybe in the future, would be a permanent removal of older awscli version from the pip public repository.

Administrator using pip will be able to make awscli to be installed into a customed directory such as /opt/apps by using the following pip command

pip install --install-option="--prefix=/opt/apps" awscli==1.2.6

Unfortunately, in doing so pip will no longer able to manage the awscli package. Administrators will need to have a small effort to remove the installed version manually before upgrading the AWSCli using the similar command.

As a closing, to my personal opinion pip is a better way to install and maintaining the version of AWSCli.

Adding Amazon Web Service EC2 instances IP and names into PuTTY session automagically

Introduction :

The motivation of creation of this script were due to non-persistent and ever changing state of Amazon Web Service(AWS) that causes my infrastructure changes more frequently. It will be labor intensive to create, update and remove session manually in PuTTY to reflect the changes in the AWS.

The pre-requisite :

The idea :

Using a batch scripting to warp and call the powershell. Then the powershell script will call the installed EC2 API tools provided by Amazon.

 

Then, powershell is too used to do the parsing of text returned by the EC2 API tools. In all the complexity of powershell will generate a new registry file for windows.

 

Finally, the batch script will call registry editor that would import the exported EC2 instance values into the PuTTY session repository.

 

In implementing the script, I have created 4 files, the batch script generate_putty_session.bat, the poweshell script generate_putty_session.ps1 , the registry file header reg_header.txt and lastly, the reg_putty.txt contains the text of default PuTTY configuration in a form of windows registry format.

 

Codes of generate_putty_session.bat :


@echo off
powershell -version 2.0 -ExecutionPolicy unrestricted %~dp0generate_putty_sessions.ps1
regedit.exe /s %userprofile%\putty_list.reg

 

Codes of generate_putty_session.ps1 :

 


#Preloading scripts
#Removing old reg file
if ( Test-Path $env:userprofile\putty_list.reg){
  del $env:userprofile\putty_list.reg
}

#Check environment for Windows x86 or x86_64
if ([IntPtr]::Size -eq 4){
  if ( Test-Path "C:\Program Files\AWS Tools\PowerShell\AWSPowerShell"){
    import-module "C:\Program Files\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
  }
  else{
    write-host "AWS Tools for PowerShell was not install, exiting. Download at http://aws.amazon.com/powershell/"
    exit
  }
}
else{
  if ( Test-Path "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell"){
    import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
  }
  else{
    write-host "AWS Tools for PowerShell was not install, exiting. Download at http://aws.amazon.com/powershell/"
    exit
  }
}

#Check env variable for required EC2 configuration
if (-not (Test-Path Env:\EC2_HOME)){
  write-host "Environment variable EC2_HOME was not found, please ensure your EC2 API Tools were properly installed or configured or setup."
  exit
}

if (-not (Test-Path Env:\EC2_CERT)){
  write-host "Environment variable EC2_CERT was not found, please ensure your EC2 API Tools were properly installed or configured or setup."
  exit
}

if (-not (Test-Path Env:\EC2_PRIVATE_KEY)){
  write-host "Environment variable EC2_PRIVATE_KEY was not found, please ensure your EC2 API Tools were properly installed or configured or setup."
  exit
}

#Get my script path
$myPath = split-path -parent $MyInvocation.MyCommand.Definition

if(-not (Test-Path -path $myPath\reg_header.txt)){
  write-host "Please make sure reg_header.txt is in " $myPath
  exit
}

if(-not (Test-Path -path $myPath\reg_putty.txt)){
  write-host "Please make sure reg_header.txt is in " $myPath
  exit
}

Copy-Item $myPath\reg_header.txt $env:userprofile
Copy-Item $myPath\reg_putty.txt $env:userprofile

#Main body and function of the script.
#Creating file to link instance ID with Public DNS
ec2-describe-instances --filter `"virtualization-type=paravirtual`" --filter `"instance-state-name=running`" --filter `"tag:Name=/*/*`" | Select-String -pattern INSTANCE -caseSensitive | foreach { "$($_.ToString().split()[1,3])" >> $env:userprofile\awsinstanceIP.tmp}

#Creating a file to link instance ID with Name tag
ec2-describe-instances --filter `"virtualization-type=paravirtual`" --filter `"instance-state-name=running`" --filter `"tag:Name=/*/*`" | Select-String -pattern Name -caseSensitive | foreach { "$($_.ToString().split()[2,4])" >> $env:userprofile\awsinstanceName.tmp}

# Clean up results, removing RenderWorkerGroup
Get-Content $env:userprofile\awsinstanceName.tmp | Select-String -pattern RenderWorkerGroup -NotMatch | foreach { "$($_.ToString().split()[0,1])" >> $env:userprofile\awsinstanceNameClean.tmp}

#$awsInstanceIDIP = Get-Content $env:userprofile\awsinstanceIP.tmp
$awsInstanceCleanName =  Get-Content $env:userprofile\awsinstanceNameClean.tmp
$count = 0

# Create HashTable from File.
ForEach ($line in $awsInstanceCleanName) {
  if ($count -le 0 ) {
    $myHash = @{ $line.ToString().Split()[0] = $line.ToString().Split()[1]}
  }
  else{
    $myHash.Set_Item($line.ToString().Split()[0], $line.ToString().Split()[1])
  }
  $count = $count + 1
}
$count = 0

Get-Content $env:userprofile\awsinstanceIP.tmp | ForEach-Object {

  $line = $_
  $myHash.GetEnumerator() | ForEach-Object {
    if ($line -match $_.Key)
    {
      if ($_.value.ToString().Contains("render-worker")){
        $replacement = $_.Key.ToString() + " " + $_.Value.ToString() + "/" + $_.Key.ToString()
      }
      else{
        $replacement = $_.Key.ToString() + " " + $_.Value.ToString()
      }
      $line = $line -replace $_.Key, $replacement
    }
  }
  $line
} | Set-Content -Path $env:userprofile\awsinstanceResult.tmp

del $env:userprofile\awsinstanceIP.tmp
del $env:userprofile\awsinstanceName.tmp
del $env:userprofile\awsinstanceNameClean.tmp

$awsinstanceResult = Get-Content $env:userprofile\awsinstanceResult.tmp

#Adding header into file content.
Add-Content $env:userprofile\awsinstanceReg_List.tmp $(Get-Content $env:userprofile\reg_header.txt)
Add-Content $env:userprofile\awsinstanceReg_List.tmp "`r"

# Populating body of the file before converting into registry file.
foreach ($line in $awsinstanceResult){
  $reg_line = "`[HKEY_CURRENT_USER\Software\Simontatham\PuTTY\Sessions\" + $line.ToString().Split()[1] + "]"
  Add-Content $env:userprofile\awsinstanceReg_List.tmp $reg_line
  $reg_line = "`"HostName`"=`"" + $line.ToString().Split()[2] + "`""
  Add-Content $env:userprofile\awsinstanceReg_List.tmp $reg_line
  # Add fillers to the sessions
  Add-Content $env:userprofile\awsinstanceReg_List.tmp $(Get-Content $env:userprofile\reg_putty.txt)
  Add-Content $env:userprofile\awsinstanceReg_List.tmp "`r"
}

Get-Content $env:userprofile\awsinstanceReg_List.tmp | Add-Content $env:userprofile\putty_list.reg

#Removing all temporary files
del $env:userprofile\awsinstanceReg_List.tmp
del $env:userprofile\awsinstanceResult.tmp
del $env:userprofile\reg_header.txt 
del $env:userprofile\reg_putty.txt
Before running the script, pay special attention to the powershell code of generate_putty_sessions.ps1 at line 61, 64, 67 and 91. Make needed change to the format of your AWS Tag “Name”.

 

The filter in line 61, 64 would create 2 different files using the same ec2 api tools command. It is working under assumption that you have named your AWS instances using format as such /[product_name]/[environment]/[sub-system]/[server-number] .

 

Line 67 would use the similar pattern that I have used in my environment to remove unwanted servers from being added into the PuTTY sessions. In my code I am removing output that contains RenderWorkerGroup.

 

Line 91, just enforcing the format name for instances which is generated by AutoScaling /[product_name]/[environment]/[sub-system]/[aws-instance-id]

 
 

Using the script :

Place all the files into a single folder in your Windows machine.

 

Run the generate_putty_sessions.bat in the Administrator command prompt. In less than 2 minutes

 

Your PuTTY should contains all the session imported from Amazon Web Service EC2.

 
 

Code Download :

putty_session_generator