Have added changes to incorporate both generating AWStats logs and starting up AWStats service in a single docker compose file at KarMeng / docker_awstats — Bitbucket
This is an easy and simple example that beginners can use to generate web statistics using AWStats.
Required softwares:
Hashicorp Vagrant 2.4.1
Oracle VirtualBox 7.0.14
The first error that will be face on get go is the error “kibana Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap” and exiting error “dependency failed to start: container docker-es03-1 exited (137)”.
Searching mentioned error on google or the internet will yield result that advice swap memory and memory limit hit.
Continue readingPre-requisite:
– VirtualBox 7.0.14
– Vagrant 2.4.1
– Windows 10 or better OS
– 16GB RAM (10GB RAM are required for creating ElasticSearch with kibana; 1GB and 2 x ElasticSearch node; 4GB each, rest of the RAM for VM host OS)
Overview:
There are 2 layers of virtualization, first the Virtual Box, then the docker engine running in the Virtual Box VM running on Ubuntu 20.04 focal.
Orchestration used in the host OS level; Windows 10 are the hashicorp vagrant. The vagrant is used to configure the VM Ubuntu OS to be configured to run properly configured docker and Ubuntu 20.04.
Then docker compose v2 are used to create the ElasticSearch 8.12 cluster or stack.
The downside of this example, vagrant up needs to be run initially to configure the VM Ubuntu 20.04 OS. I have yet to discover if Vagrant has the ability to bootstrap grub and configuring the sysctl to allow the docker engine to run properly with the ElasticSearch 8.12 stack.
Continue reading
copy con is a legacy command that allows a plain MSDOS to create text file such as autoexec.bat and config.sys
From my experience, the command is supported from MSDOS 3.30 till today. Have yet to test this command in Windows 11 OS.
To commit and save the file into disk, use CTRL + Z
This command was learned/discovered during the days before existence of Internet, by attentively looking over the shoulder of seniors, who are not willing to teach during computer club session.
@echo off
netsh wlan show profiles |findstr All > tmp.txt
for /f "tokens=2 delims=:" %%a in (tmp.txt) do (
echo %%a
netsh wlan show profiles name=%%a key=clear
)
del tmp.txtThis batch script will list out list of saved wifi profiles, save the profile into text file tmp.txt
From the file tmp.txt, print out the 2nd token from the text file which will returns the WiFi profile name, then using the same command to expose WiFi key in clear text.
First assuming that compatible hardware is available, get compatible SODIMM RAM. and identify type of hard disk your laptop can use.
General idea, check the laptop user manual to identify what are the maximum RAM that are supported total, what are the maximum RAM supported in a slot. And importantly does the motherboard chipset of your laptop supports SODIMM that are different capacity. Clock speed as long as its higher than the laptop spec, the motherboard chipset will automatically tune the RAM Clock speed to match the maximum speed supported by the motherboard.
For context, the old laptop for upgrade is an Acer Aspire E5-475 which comes with 4GB RAM, 1TB HDD that has an Intel i5-7200U CPU at 2.5GHz.
A little bit of effort to search the net and encountering the solution, powershell will is a simple way to measure time spent executing a command. In Linux environment, time command can be used, but in this post, it is limited to Windows environment.
Based on the conventional wisdom from the documentation of Measure-Command of powershell. The command would be as follows:
Measure-Command {vagrant up}However, by running the above powershell command, the output to the screen will be not available.
Hence, more searching in the internet (Source: time – Timing a command’s execution in PowerShell – Stack Overflow) leads to the complete command below:
Measure-Command {vagrant up|Out-Default}
Based on the previous post of the vagrant orchestration of GitLab, the orchestration takes about 50 minutes in my computer with timedotcom broadband connection.
Conclusion, measure-command is easy way but lack of detail such as time spent on CPU, IDLE time and network operations time. It will not be suitable for use if more details are needed.
The code of the project is available at chowkarmeng/vagrant_gitlab (github.com), the docker compose is based on the sample provided in GitLab Docker images | GitLab
The improvement done was to change the folder sync for virtual box into docker volumes.
First, git clone the repository https://github.com/chowkarmeng/vagrant_gitlab.git
The process will take hours depending on the speed of your computer and speed of your internet connection.
Continue readingA month ago, I was working on migrating a system from Azure to another cloud platform. Azure web management portal at the time of this post has no form of UI to identify the amount of space used by your Blob Storage. One can only know how much he owe Azure from the Billing portal.
Fortunately, Microsoft Azure team has came out with a simple powershell script that will provide better insight of how much blob object and how much space used by your Azure Blob. The solution were provided at Get Billable size for Azure Blob at MSDN.
The script provided works fine, until the script stopped due to System.OutOfMemoryException :
PS C:\dev\powershell> .\CalculateBlobCost.ps1 -storageaccountname "myProdatAmerNorth"
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Services\Microsoft.WindowsAzure.Commands.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Automation\Microsoft.Azure.Commands.Automation.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\TrafficManager\Microsoft.WindowsAzure.Commands.TrafficManager.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Services\Microsoft.WindowsAzure.Commands.Profile.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Compute\Microsoft.WindowsAzure.Commands.ServiceManagement.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Sql\Microsoft.WindowsAzure.Commands.SqlDatabase.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Storage\Microsoft.WindowsAzure.Commands.Storage.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\ManagedCache\Microsoft.Azure.Commands.ManagedCache.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\HDInsight\Microsoft.WindowsAzure.Commands.HDInsight.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\Networking\Microsoft.WindowsAzure.Commands.ServiceManagement.Network.dl
l'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\StorSimple\Microsoft.WindowsAzure.Commands.StorSimple.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\RemoteApp\Microsoft.WindowsAzure.Commands.RemoteApp.dll'.
VERBOSE: Loading module from path 'C:\Program Files (x86)\Microsoft
SDKs\Azure\PowerShell\ServiceManagement\Azure\.\RecoveryServices\Microsoft.Azure.Commands.RecoveryServices.dll'.
VERBOSE: 11:35:12 AM - Begin Operation: Get-AzureStorageAccount
VERBOSE: 11:35:14 AM - Completed Operation: Get-AzureStorageAccount
WARNING: GeoReplicationEnabled property will be deprecated in a future release of Azure PowerShell. The value will be
merged into the AccountType property.
VERBOSE: 11:35:14 AM - Begin Operation: Get-AzureStorageKey
VERBOSE: 11:35:16 AM - Completed Operation: Get-AzureStorageKey
VERBOSE: Container 'mySettings' with 5 blobs has a size of 0.00MB.
Exception of type 'System.OutOfMemoryException' was thrown.
At C:\dev\powershell\CalculateBlobCost.ps1:77 char:9
+ $Blob.ICloudBlob.DownloadBlockList() |
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException
Exception calling "DownloadBlockList" with "0" argument(s): "Exception of type 'System.OutOfMemoryException' was
thrown."
At C:\dev\powershell\CalculateBlobCost.ps1:77 char:9
+ $Blob.ICloudBlob.DownloadBlockList() |
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : StorageException
Exception calling "DownloadBlockList" with "0" argument(s): "Exception of type 'System.OutOfMemoryException' was
thrown."
At C:\dev\powershell\CalculateBlobCost.ps1:77 char:9
+ $Blob.ICloudBlob.DownloadBlockList() |
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : StorageException
Exception calling "DownloadBlockList" with "0" argument(s): "Exception of type 'System.OutOfMemoryException' was
thrown."
At C:\dev\powershell\CalculateBlobCost.ps1:77 char:9
+ $Blob.ICloudBlob.DownloadBlockList() |
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : StorageException
Exception of type 'System.OutOfMemoryException' was thrown.
At C:\dev\powershell\CalculateBlobCost.ps1:124 char:13
+ $containerSizeInBytes += Get-BlobBytes $_
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException
ForEach-Object : Exception of type 'System.OutOfMemoryException' was thrown.
At C:\dev\powershell\CalculateBlobCost.ps1:123 char:9
+ ForEach-Object {
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [ForEach-Object], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException,Microsoft.PowerShell.Commands.ForEachObjectCommand
VERBOSE: Container 'whoamI-version-2012' with 79895 blobs has a size of 33.56MB.
VERBOSE: Container 'whoamI-version-2' with 0 blobs has a size of 0.00MB.
VERBOSE: Container 'whoamI-version-3' with 0 blobs has a size of 0.00MB.
Exception of type 'System.OutOfMemoryException' was thrown.
At C:\dev\powershell\CalculateBlobCost.ps1:77 char:9
+ $Blob.ICloudBlob.DownloadBlockList() |
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [], OutOfMemoryException
+ FullyQualifiedErrorId : System.OutOfMemoryException
VERBOSE: Container 'whoamI-version-2' with 10070 blobs has a size of 7.29MB.
VERBOSE: Container 'whoamI-version-3' with 423186 blobs has a size of 1337.75MB.
VERBOSE: Container 'whoamI-version-4' with 95 blobs has a size of 95.96MB.
After few hours later, identify that one of my Azure Blob container “whoamI-version-3” has more than 1 Million object, with size exceeding 1GB was the culprit that caused the script to fail. The cmdlet that failed was Get-AzureStorageBlob.
From, MSDN Get-AzureStorageBlob has the parameter -MaxCount that would be able to calculate the object in the AzureBlob in batches; to SQL paging.
By adding the use of MaxCount parameter and adding the logic to have a continuation token, the improved script is now able to run without any fear of OutOfMemoryException.
To immediately use the improved powershell script, kindly download the file attached in this post.CalculateBlobCost powershell script
@echo off
powershell -version 2.0 -ExecutionPolicy unrestricted %~dp0generate_putty_sessions.ps1
regedit.exe /s %userprofile%\putty_list.reg
#Preloading scripts
#Removing old reg file
if ( Test-Path $env:userprofile\putty_list.reg){
del $env:userprofile\putty_list.reg
}
#Check environment for Windows x86 or x86_64
if ([IntPtr]::Size -eq 4){
if ( Test-Path "C:\Program Files\AWS Tools\PowerShell\AWSPowerShell"){
import-module "C:\Program Files\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
}
else{
write-host "AWS Tools for PowerShell was not install, exiting. Download at http://aws.amazon.com/powershell/"
exit
}
}
else{
if ( Test-Path "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell"){
import-module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"
}
else{
write-host "AWS Tools for PowerShell was not install, exiting. Download at http://aws.amazon.com/powershell/"
exit
}
}
#Check env variable for required EC2 configuration
if (-not (Test-Path Env:\EC2_HOME)){
write-host "Environment variable EC2_HOME was not found, please ensure your EC2 API Tools were properly installed or configured or setup."
exit
}
if (-not (Test-Path Env:\EC2_CERT)){
write-host "Environment variable EC2_CERT was not found, please ensure your EC2 API Tools were properly installed or configured or setup."
exit
}
if (-not (Test-Path Env:\EC2_PRIVATE_KEY)){
write-host "Environment variable EC2_PRIVATE_KEY was not found, please ensure your EC2 API Tools were properly installed or configured or setup."
exit
}
#Get my script path
$myPath = split-path -parent $MyInvocation.MyCommand.Definition
if(-not (Test-Path -path $myPath\reg_header.txt)){
write-host "Please make sure reg_header.txt is in " $myPath
exit
}
if(-not (Test-Path -path $myPath\reg_putty.txt)){
write-host "Please make sure reg_header.txt is in " $myPath
exit
}
Copy-Item $myPath\reg_header.txt $env:userprofile
Copy-Item $myPath\reg_putty.txt $env:userprofile
#Main body and function of the script.
#Creating file to link instance ID with Public DNS
ec2-describe-instances --filter `"virtualization-type=paravirtual`" --filter `"instance-state-name=running`" --filter `"tag:Name=/*/*`" | Select-String -pattern INSTANCE -caseSensitive | foreach { "$($_.ToString().split()[1,3])" >> $env:userprofile\awsinstanceIP.tmp}
#Creating a file to link instance ID with Name tag
ec2-describe-instances --filter `"virtualization-type=paravirtual`" --filter `"instance-state-name=running`" --filter `"tag:Name=/*/*`" | Select-String -pattern Name -caseSensitive | foreach { "$($_.ToString().split()[2,4])" >> $env:userprofile\awsinstanceName.tmp}
# Clean up results, removing RenderWorkerGroup
Get-Content $env:userprofile\awsinstanceName.tmp | Select-String -pattern RenderWorkerGroup -NotMatch | foreach { "$($_.ToString().split()[0,1])" >> $env:userprofile\awsinstanceNameClean.tmp}
#$awsInstanceIDIP = Get-Content $env:userprofile\awsinstanceIP.tmp
$awsInstanceCleanName = Get-Content $env:userprofile\awsinstanceNameClean.tmp
$count = 0
# Create HashTable from File.
ForEach ($line in $awsInstanceCleanName) {
if ($count -le 0 ) {
$myHash = @{ $line.ToString().Split()[0] = $line.ToString().Split()[1]}
}
else{
$myHash.Set_Item($line.ToString().Split()[0], $line.ToString().Split()[1])
}
$count = $count + 1
}
$count = 0
Get-Content $env:userprofile\awsinstanceIP.tmp | ForEach-Object {
$line = $_
$myHash.GetEnumerator() | ForEach-Object {
if ($line -match $_.Key)
{
if ($_.value.ToString().Contains("render-worker")){
$replacement = $_.Key.ToString() + " " + $_.Value.ToString() + "/" + $_.Key.ToString()
}
else{
$replacement = $_.Key.ToString() + " " + $_.Value.ToString()
}
$line = $line -replace $_.Key, $replacement
}
}
$line
} | Set-Content -Path $env:userprofile\awsinstanceResult.tmp
del $env:userprofile\awsinstanceIP.tmp
del $env:userprofile\awsinstanceName.tmp
del $env:userprofile\awsinstanceNameClean.tmp
$awsinstanceResult = Get-Content $env:userprofile\awsinstanceResult.tmp
#Adding header into file content.
Add-Content $env:userprofile\awsinstanceReg_List.tmp $(Get-Content $env:userprofile\reg_header.txt)
Add-Content $env:userprofile\awsinstanceReg_List.tmp "`r"
# Populating body of the file before converting into registry file.
foreach ($line in $awsinstanceResult){
$reg_line = "`[HKEY_CURRENT_USER\Software\Simontatham\PuTTY\Sessions\" + $line.ToString().Split()[1] + "]"
Add-Content $env:userprofile\awsinstanceReg_List.tmp $reg_line
$reg_line = "`"HostName`"=`"" + $line.ToString().Split()[2] + "`""
Add-Content $env:userprofile\awsinstanceReg_List.tmp $reg_line
# Add fillers to the sessions
Add-Content $env:userprofile\awsinstanceReg_List.tmp $(Get-Content $env:userprofile\reg_putty.txt)
Add-Content $env:userprofile\awsinstanceReg_List.tmp "`r"
}
Get-Content $env:userprofile\awsinstanceReg_List.tmp | Add-Content $env:userprofile\putty_list.reg
#Removing all temporary files
del $env:userprofile\awsinstanceReg_List.tmp
del $env:userprofile\awsinstanceResult.tmp
del $env:userprofile\reg_header.txt
del $env:userprofile\reg_putty.txt
Pre-requisite :
Steps : Continue reading