The first error that will be face on get go is the error “kibana Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap” and exiting error “dependency failed to start: container docker-es03-1 exited (137)”.
Searching mentioned error on google or the internet will yield result that advice swap memory and memory limit hit.
Continue readingPre-requisite:
– VirtualBox 7.0.14
– Vagrant 2.4.1
– Windows 10 or better OS
– 16GB RAM (10GB RAM are required for creating ElasticSearch with kibana; 1GB and 2 x ElasticSearch node; 4GB each, rest of the RAM for VM host OS)
Overview:
There are 2 layers of virtualization, first the Virtual Box, then the docker engine running in the Virtual Box VM running on Ubuntu 20.04 focal.
Orchestration used in the host OS level; Windows 10 are the hashicorp vagrant. The vagrant is used to configure the VM Ubuntu OS to be configured to run properly configured docker and Ubuntu 20.04.
Then docker compose v2 are used to create the ElasticSearch 8.12 cluster or stack.
The downside of this example, vagrant up needs to be run initially to configure the VM Ubuntu 20.04 OS. I have yet to discover if Vagrant has the ability to bootstrap grub and configuring the sysctl to allow the docker engine to run properly with the ElasticSearch 8.12 stack.
Continue readingThe code of the project is available at chowkarmeng/vagrant_gitlab (github.com), the docker compose is based on the sample provided in GitLab Docker images | GitLab
The improvement done was to change the folder sync for virtual box into docker volumes.
First, git clone the repository https://github.com/chowkarmeng/vagrant_gitlab.git
The process will take hours depending on the speed of your computer and speed of your internet connection.
Continue readingAt the time of this post, the compatibility matrix of vagrant and VirtualBox is as follows:
| Vagrant version | VirtualBox version | |
| 2.3.7 | 7.0.10 | |
| 7.0.12 |
Unfortunately, Vagrant 2.4.0 does not work well with VirtualBox 7.0
This post was created using Vagrant 2.3.7 and VirtualBox 7.0.10
To make Vagrant possible, after installing the Vagrant from hashicorp webpage, a Vagrant file needs to be created. The most basic file that needs to exist in your Vagrant to work is a folder with Vagrantfile
Additional post start up scripts to complete the installations that are used in this example are setup.sh, install_ansible.sh, install_boto3.sh and install_python3.sh
Continue readingIn the linux ubuntu or redhat/centos (yum) just use the default package manager. For my post I am using ubuntu as an example to illustrate my point.
sudo apt-get install mysql-proxy
You should be getting the following confirmation
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
ttf-dejavu-extra libevent-extra-2.0-5 libmysql++3 libdbi1 libapr1 librrd4 libcairo2 libmysqlclient-dev libaprutil1-ldap libthai-data libreadline6-dev libpcrecpp0 libsvn1 libdatrie1
fontconfig libtinfo-dev libpixman-1-0 libevent-openssl-2.0-5 libaprutil1-dbd-sqlite3 libonig2 libthai0 libneon27-gnutls zlib1g-dev libevent-pthreads-2.0-5 libzzip-0-13 ttf-dejavu
libdb4.8 libpq5 libpcre3-dev libpango1.0-0 libxcb-render0 libxcb-shm0 libaprutil1 libevent-core-2.0-5 libfcgi0ldbl libreadline-dev
However, to run the mysql-proxy there is a few tweak needed, assuming that you are not using mysql-proxy beyond just a proxy.
To start up mysql-proxy, you may issue the command such as this :
sudo mysql-proxy --defaults-file=/etc/mysql/mysql_proxy.cnf &
The configuration of mysql_proxy.cnf :
log-file = /opt/apps/logs/mysql-proxy/mysql-proxy.log
log-level = debug
proxy-backend-addresses = 10.161.89.64:3306
admin-username = root
admin-password = for_my_eyes_only
If the mysql-proxy fails to start as a daemon, it is best to check the logs at /opt/apps/logs/mysql-proxy/mysql-proxy.log :
root@ckm-myprox:/opt/apps/logs/mysql-proxy$ sudo tail -f mysql-proxy.log
2014-06-11 20:58:27: (message) mysql-proxy 0.8.1 started
2014-06-11 20:58:27: (debug) max open file-descriptors = 1024
2014-06-11 20:58:27: (critical) admin-plugin.c:579: --admin-lua-script needs to be set, /lib/mysql-proxy/lua/admin.lua may be a good value
2014-06-11 20:58:27: (critical) mainloop.c:267: applying config of plugin admin failed
2014-06-11 20:58:27: (critical) mysql-proxy-cli.c:596: Failure from chassis_mainloop. Shutting down.
2014-06-11 20:58:27: (message) Initiating shutdown, requested from mysql-proxy-cli.c:597
2014-06-11 20:58:27: (message) shutting down normally, exit code is: 1
The last line of the log, just confirmed the mysql-proxy were failed to start due admin lua were not set. To skip the admin lua function(assuming that it will not be used). Start the mysql with :
sudo mysql-proxy --defaults-file=/etc/mysql/mysql_proxy.cnf --plugins=proxy &
Take note that my post on installing mysql-proxy, the version is 0.8.4 and the stock from ubuntu repository is 0.8.1 .
sudo wget http://dev.mysql.com/get/Downloads/MySQL-Proxy/mysql-proxy-0.8.4.tar.gzsudo tar -xzvf mysql-proxy-0.8.4.tar.gzsudo apt-get install libmysql++-devsudo ./configureconfigure: error: The pkg-config script could not be found or is too old. Make sure it is in your PATH or set the PKG_CONFIG environment variable to the full path to pkg-config.sudo apt-get install pkg-configchecking pkg-config is at least version 0.9.0... yes checking for LUA... no ... checked for Lua via pkg-config: No package 'lua' found. retrying with lua5.1 checking for LUA... no configure: error: checked for Lua via pkg-config: No package 'lua5.1' found. Make sure lua and its devel-package, which includes the lua5.1.pc (debian and friends) or lua.pc (all others) file, is installedsudo apt-get install lua5.1
sudo apt-get install liblua5.1 sudo apt-get install liblua5.1-sql-mysql2checking pkg-config is at least version 0.9.0... yes checking for LUA... no ... checked for Lua via pkg-config: No package 'lua' found. retrying with lua5.1 checking for LUA... yes checking for GLIB... configure: error: Package requirements (glib-2.0 >= 2.16.0) were not met:sudo apt-get install glib2.0 sudo apt-get install libglib2.0-0configure: error: libevent is requiredsudo apt-get install libevent-2.0-5 sudo apt-get install libevent-devsudo make
sudo make installsudo mysql-proxymysql-proxy: error while loading shared libraries: libmysql-chassis.so.0: cannot open shared object file: No such file or directorysudo ldconfigUsage:
mysql-proxy [OPTION...] - MySQL Proxy
Help Options:
-h, --help Show help options
--help-all Show all help options
--help-proxy Show options for the proxy-module
Application Options:
-V, --version Show version
--defaults-file= configuration file
--verbose-shutdown Always log the exit code when shutting down
--daemon Start in daemon-mode
--user= Run mysql-proxy as user
--basedir= Base directory to prepend to relative paths in the config
--pid-file= PID file in case we are started as daemon
--plugin-dir=
path to the plugins
--plugins= plugins to load
--log-level=(error|warning|info|message|debug) log all messages of level ... or higher
--log-file= log all messages in a file
--log-use-syslog log all messages to syslog
--log-backtrace-on-crash try to invoke debugger on crash
--keepalive try to restart the proxy if it crashed
--max-open-files maximum number of open files (ulimit -n)
--event-threads number of event-handling threads (default: 1)
--lua-path= set the LUA_PATH
--lua-cpath= set the LUA_CPATH
Good new for AWS users, Amazon has release the new unified AWSCli was released in September 2013. Amazon did provide multiple ways to have the new AWSCli installed.
I have to admit the task of installation is more straight forward and simplified compared to the old AWS Cli.
At the time of this post, the AWSCli released version 1.2.6, and it runs on Python 2.6. Hence, this post will provide custom install of AWSCli into linux based machine.
For users who are planning to use the AWSCli bundle provided by Amazon here is the recommended steps in sequence. Disclaimer and note : I have not added any form of error catching or linux distro detection and I am assuming the linux distro used is redhat.
Installing AWSCli using the Amazon awscli-bundle
mkdir -p /opt/apps/tmp cd /opt/apps/tmp wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip unzip awscli-bundle.zip mkdir -p /opt/apps/$(ls awscli-bundle/packages/ | egrep -o 'awscli-[0-9]\.[0-9]\.[0-9]') ./awscli-bundle/install -i /opt/apps/$(ls awscli-bundle/packages/ | egrep -o 'awscli-[0-9]\.[0-9]\.[0-9]') /opt/apps/awscli/bin/aws --version ln -s /opt/apps/$(ls awscli-bundle/packages/ | egrep -o 'awscli-[0-9]\.[0-9]\.[0-9]') /opt/apps/awscli ln -s /opt/apps/awscli/bin/aws /usr/bin/aws ln -s /opt/apps/awscli/bin/aws.cmd /usr/bin/aws.cmd cd ~ rm -Rf /opt/apps/tmp
Installing AWS via pip
python --version apt-get install python-pip yum install python-pip cd /opt/apps/ mkdir tmp cd tmp wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py python ez_setup.py python get-pip.py pip install awscli==1.2.6 aws help cd ~ rm -Rf /opt/apps/tmp
The advantage of AWSCli bundle over the pip method is, ease of install without need to get ez_setup and pip installed. Since the AWSCli bundle zip is hosted within Amazon Web Service network, it took me less than 1 seconds to download the 5MB AWSCli-bundle.zip file.
[root@ip-10-255-255-1 ~]# time wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip --2013-11-28 05:25:16-- https://s3.amazonaws.com/aws-cli/awscli-bundle.zip Resolving s3.amazonaws.com... 176.32.99.46 Connecting to s3.amazonaws.com|176.32.99.46|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 5139105 (4.9M) [application/zip] Saving to: `awscli-bundle.zip' 100%[================================================================================================================ ===================================================================================>] 5,139,105 16.0M/s in 0.3s 2013-11-28 05:25:17 (16.0 MB/s) - `awscli-bundle.zip' saved [5139105/5139105] real 0m0.611s user 0m0.096s sys 0m0.036s [root@ip-10-255-255-1 ~]# time wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip --2013-11-28 05:25:49-- https://s3.amazonaws.com/aws-cli/awscli-bundle.zip Resolving s3.amazonaws.com... 176.32.99.46 Connecting to s3.amazonaws.com|176.32.99.46|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 5139105 (4.9M) [application/zip] Saving to: `awscli-bundle.zip' 100%[================================================================================================================ ===================================================================================>] 5,139,105 19.6M/s in 0.2s 2013-11-28 05:25:49 (19.6 MB/s) - `awscli-bundle.zip' saved [5139105/5139105] real 0m0.338s user 0m0.076s sys 0m0.036s
The downside of using the awscli bundle installation is users need to upload the awscli-bundle.zip into personal version control servers/services (such as github) in order to have version control of awscli. Therefore, there will be overhead of maintaining the version of the awscli and it will be labor intensive or makes processes complicated.
The only disadvantage of pip AWSCli is the pre-requisite of installing pip. And maybe in the future, would be a permanent removal of older awscli version from the pip public repository.
Administrator using pip will be able to make awscli to be installed into a customed directory such as /opt/apps by using the following pip command
pip install --install-option="--prefix=/opt/apps" awscli==1.2.6
Unfortunately, in doing so pip will no longer able to manage the awscli package. Administrators will need to have a small effort to remove the installed version manually before upgrading the AWSCli using the similar command.
As a closing, to my personal opinion pip is a better way to install and maintaining the version of AWSCli.
For linux or ubuntu user who is lazy to login to remote by typing password or plainly to improve security measure. You may generate a RSA key type to authenticate the login process. Generating the RSA key is the very first step.
Rsync is a common command that users and administrator alike uses it to synchronize files. The closest function that Microsoft uses is xcopy .
Scope and conditions :