# posts

uuid: 1f18861f-7597-481c-b20e-e1346fff85f2
title: About me
slug: about-me
status: published
language: en_US
markdown:
  Welcome to my site.

  My name is Nemanja Djuric. I have more than ten years of Information Technology and Web Hosting experience. This includes Technical Support, Customer Service, System Administrator and Project Management Roles, using SAS, Cloud, VMware ESX / Server, XEN, OpenVZ and Parallels Virtuozzo Virtualization Platforms running Parallels Plesk Panel, cPanel/WHM, Direct Admin, H-Sphere and etc.

  General
  ------------------------------------------
  SSH,Shell/Bash Scripting
  SSL, FTP, DNS
  HTML, CSS, Javascript 
  PHP, JSP, AJAX, MySQL, MSSQL, PostgreSQL
  POP3/SMTP, Sendmail, Qmail, Exim
  Webalizer, AWStats
  Kayako, WHMCS, Zimbra, Lotus Notes, IIS 6, 7, 7.5, Adobe CS5/CS6
  WordPress / Drupal / Joomla, SEO

  Operating Systems
  ------------------------------------------
  RHE
  CentOS 4/5/6
  Ubuntu
  Windows Server 2003, Windows Server 2008 R2 
  Lotus Domino

  Networking
  ------------------------------------------
  General Networking, Bind, IPTables, VPN

  Virtualization 
  ------------------------------------------
  VMware ESX / Server
  XEN
  OpenVZ
  Parallels Virtuozzo

  Control Panels
  ------------------------------------------
  Parallels Plesk Panel
  cPanel/WHM
  Direct Admin
  H-Sphere
  Lotus Domino

html:
  <p>Welcome to my site.</p>

  <p>My name is Nemanja Djuric. I have more than ten years of Information Technology and Web Hosting experience. This includes Technical Support, Customer Service, System Administrator and Project Management Roles, using SAS, Cloud, VMware ESX / Server, XEN, OpenVZ and Parallels Virtuozzo Virtualization Platforms running Parallels Plesk Panel, cPanel/WHM, Direct Admin, H-Sphere and etc.</p>

  <h2 id="general">General  </h2>

  <p>SSH,Shell/Bash Scripting <br />
  SSL, FTP, DNS <br />
  HTML, CSS, Javascript <br />
  PHP, JSP, AJAX, MySQL, MSSQL, PostgreSQL <br />
  POP3/SMTP, Sendmail, Qmail, Exim <br />
  Webalizer, AWStats <br />
  Kayako, WHMCS, Zimbra, Lotus Notes, IIS 6, 7, 7.5, Adobe CS5/CS6 <br />
  WordPress / Drupal / Joomla, SEO</p>

  <h2 id="operatingsystems">Operating Systems  </h2>

  <p>RHE <br />
  CentOS 4/5/6 <br />
  Ubuntu <br />
  Windows Server 2003, Windows Server 2008 R2 <br />
  Lotus Domino</p>

  <h2 id="networking">Networking  </h2>

  <p>General Networking, Bind, IPTables, VPN</p>

  <h2 id="virtualization">Virtualization  </h2>

  <p>VMware ESX / Server <br />
  XEN <br />
  OpenVZ <br />
  Parallels Virtuozzo</p>

  <h2 id="controlpanels">Control Panels  </h2>

  <p>Parallels Plesk Panel <br />
  cPanel/WHM <br />
  Direct Admin <br />
  H-Sphere <br />
  Lotus Domino</p>


uuid: e92429c5-55a1-4e9c-b728-c6e9221c9600
title: How to Install CPanel on CentOS 7
slug: how-to-install-cpanel-on-centos-7
status: published
language: en_US
meta_title: How to Install CPanel on CentOS 7
markdown:
  In September 2015, the product was launched. And it is available in the STABLE release tier now. The new brand cPanel 54 version is launched and it is available in the current release tier for this installation. Cpanel & WHM version 11.54+ supports the CloudLinux 7/RHEL 7/CentOS 7 operating systems for fresh installations.

  In this article, I'm going to discuss about the features/Pre-requisites/installation of latest LTS release of cPanel & WHM 11.52 on CentOS 7 OS version. Before going to the installation, we can take a look on some of the new features of this release comparing the rest of the releases.

  New features:

  CloudLinux/RHEL/CentOS 7 support
  Quota support & XFS file system
  Amazon Linux AMI support
  Webmail Branding
  MultiPHP
  Easy Apache 3 to 4 updates
  Linux Container support
  Common Mail providers tabs to WHM's Greylisting.

  Network Requirements

  Hostname should be a fully qualified domain name.
  Should have a valid ipaddress, subnet address and default gateway
  Ethernet device with a static IP & hostname
  Disable OS Firewall
  Hardware Requirements

  Processor : 226 Mhz
  RAM : Minimum 1GB ( 2GB recommended for smooth functioning)
  Disk Space : Minimun 20 GB (40GB recommended for smooth functioning )
  Disable Selinux

  Need to edit the selinux configuration file located at : /etc/selinux/config

  Modify the SELINUX parameter to disabled and reboot the server.

  Installation steps followed for installing cPanel 11.52 on CentOS 7

  Before installation make sure the server meets all the pre-requisites.

  1. Set hostname to a fully qualified domain

  This is how we set the hostname for the server to a fully qualified domain name "server1.centos7-test.com" and confirm the hostname status once it is done.

  root@server1 [~]# `hostnamectl set-hostname server1.centos7-test.com –static`
  root@server1 [~]# `hostnamectl status`
  Static hostname: server1.centos7-test.com
  Icon name: computer-vm
  Chassis: vm
  Machine ID: 72863e389b584a4dab36fae7f3bffda2
  Boot ID: 8bd4f714d7ba4ebf9d53f059d0a1fe8b
  Virtualization: xen
  Operating System: CentOS Linux 7 (Core)
  CPE OS Name: cpe:/o:centos:centos:7
  Kernel: Linux 4.1.5-x86_64-linode61
  Architecture: x86-64
  root@server1 [~]#

  2. Check for the IP addr, subnet addr and gateway

  Make sure we have a static IP with a proper subnet mask and default gateway. In this case our IP address is 45.79.183.73 with a proper subnet mask and gateway.

  root@server1 [~]# `ifconfig`
  eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
  inet 45.79.183.73 netmask 255.255.255.0 broadcast 45.79.183.255
  inet6 fe80::f03c:91ff:fef1:78d9 prefixlen 64 scopeid 0x20<link>
  inet6 2600:3c03::f03c:91ff:fef1:78d9 prefixlen 64 scopeid 0x0<global>
  ether f2:3c:91:f1:78:d9 txqueuelen 1000 (Ethernet)
  RX packets 38095 bytes 13241381 (12.6 MiB)
  RX errors 0 dropped 0 overruns 0 frame 0
  TX packets 41505 bytes 45527241 (43.4 MiB)
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

  3. Check the OS, disk usage and RAM usage for the server

  Our server is installed with CentOS 7.2, having a disk space of 20GB and around 1GB RAM. Please see the results of the hardware checks below:

  OS version check:

   # `cat/etc/redhat-release`

  root@server1 [~]# `cat /etc/redhat-release`
  CentOS Linux release 7.2.1511 (Core)
  root@server1 [~]#

  Disk usage check:

   # `df-h`

  root@server1 [~]# `df -h`
  Filesystem Size Used Avail Use% Mounted on
  /dev/root 20G 5.9G 14G 31% /
  devtmpfs 492M 0 492M 0% /dev
  tmpfs 494M 0 494M 0% /dev/shm
  tmpfs 494M 14M 480M 3% /run
  tmpfs 494M 0 494M 0% /sys/fs/cgroup
  /dev/loop0 662M 736K 627M 1% /tmp
  tmpfs 99M 0 99M 0% /run/user/0

  RAM usage check:

  `free -m or cat /proc/meminfo`

  root@server1 [~]# `free -m`
  total used free shared buff/cache available
  Mem: 987 336 275 12 375 607
  Swap: 511 1 510

  4. Disable Firewall

  It is recommended to disable the OS firewalls before the cPanel & WHM installations. First of all we need to save the firewall rules and then stop the firewalld service. And make sure the service is disabled.

  `iptables-save > ~/firewall.rules`
  `systemctl stop firewalld.service`
  `systemctl disable firewalld.service`

  root@server1 [~]# systemctl status firewalld.service
  ● firewalld.service - firewalld - dynamic firewall daemon
  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
  Active: inactive (dead)

  Jan 09 06:50:28 server1.centos7-test.com systemd[1]: Stopped firewalld - dynamic firewall daemon.

  5. Disable Selinux

  Modify the SELINUX parameter to disabled in /etc/selinux/config and reboot the server.
  root@server1 [~]# `cat /etc/selinux/config`

   # This file controls the state of SELinux on the system.
   # SELINUX= can take one of these three values:
   # enforcing - SELinux security policy is enforced.
   # permissive - SELinux prints warnings instead of enforcing.
   # disabled - No SELinux policy is loaded.
  SELINUX=disabled
   # SELINUXTYPE= can take one of these two values:
   # targeted - Targeted processes are protected,
   # minimum - Modification of targeted policy. Only selected processes are protected.
   # mls - Multi Level Security protection.
  SELINUXTYPE=targeted

  5. Installation of Perl

  cPanel is programmed in Perl language. Hence, we need to install Perl prior to the cPanel installation if it isn't installed in the server before.

  `yum install -y perl`

  6. Download/Install/Run cPanel/WHM script.

  We need to download the cPanel installation script into the /home directory and run the script from there.

   # `cd /home`

   # `curl -o latest -L https://securedownloads.cpanel.net/latest`
   # `sh latest`

  This is how we download and install the cPanel installation script from our server's home directory.

  The script can take half an hour or more to complete the installation procedures.

  Installation log

  2016-01-07 07:15:51 227 ( INFO): cPanel install finished in 32 minutes and 14 seconds!
  2016-01-07 07:15:51 964 ( INFO): Congratulations! Your installation of cPanel & WHM 11.52 is now complete. The next step is to configure your server.
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): Before you configure your server, ensure that your firewall allows access on port 2087.
  2016-01-07 07:15:51 964 ( INFO): On RHEL, CentOS, and CloudLinux systems, execute `/scripts/configure_firewall_for_cpanel` to accomplish this.
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): After ensuring that your firewall allows access on port 2087, you can configure your server.
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): 1. Open your preferred browser
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): 2. Type https://45.79.183.73:2087 in the address bar
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): 3. Enter the word root in the Username text box
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): 4. Enter your root password in the Password text box
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): 5. Click the Login button
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): Visit http://go.cpanel.net/whminit for more information about first-time configuration of your server.
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): Visit http://support.cpanel.net or http://go.cpanel.net/whmfaq for additional support
  2016-01-07 07:15:51 964 ( INFO):
  2016-01-07 07:15:51 964 ( INFO): Thank you for installing cPanel & WHM 11.52!
  2016-01-07 07:15:51 835 ( WARN): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  7. Adding ports 2087/2086 to the firewalld service

  After completing the installation procedures, we need to ensure that our server allows the required service ports in the firewall.

  [root@server1 zones]# `firewall-cmd --zone=public --add-port=2087/udp`
  success
  [root@server1 zones]# `firewall-cmd --zone=public --add-port=2086/udp`
  success
  [root@server1 zones]# `firewall-cmd --zone=public --add-port=2086/tcp`
  success
  [root@server1 zones]# `firewall-cmd --zone=public –add-port=2087/tcp`

  [root@server1 zones]# `firewall-cmd --reload`

  If your server has enough resources, then it is always advised to disable the OS firewall and configure CSF/APF to ensure better security and allow all required cPanel service ports by default. I would recommend to go with CSF for a cPanel server.

  CSF Installation steps:

  `wget http://configserver.com/free/csf.tgz`

  `tar -xzf csf.tgz`

  `cd csf & ./install.cpanel.sh`

  Access the WHM via the URL

  You can access WHM using the URL https://server-hostname:2087 or http://server-hostname:2086

  After accessing the URL mentioned above, you can put your username as root and its password to login.

  Read through the terms and conditions & accept the license agreement.

  Now you can do your initial configuration steps one by one as below:

  1. Setup Networking
  2. Setup IP address
  3. Nameservers
  4. Services
  5. Setup quotas

  Visit http://go.cpanel.net/whminit for more information about first-time configuration of your server.
   
  I believe this article will be a great help for you guys! Thank you for referring to this. I would appreciate your valuable comments for further improvements.

html:
  <p>In September 2015, the product was launched. And it is available in the STABLE release tier now. The new brand cPanel 54 version is launched and it is available in the current release tier for this installation. Cpanel &amp; WHM version 11.54+ supports the CloudLinux 7/RHEL 7/CentOS 7 operating systems for fresh installations.</p>

  <p>In this article, I'm going to discuss about the features/Pre-requisites/installation of latest LTS release of cPanel &amp; WHM 11.52 on CentOS 7 OS version. Before going to the installation, we can take a look on some of the new features of this release comparing the rest of the releases.</p>

  <p>New features:</p>

  <p>CloudLinux/RHEL/CentOS 7 support <br />
  Quota support &amp; XFS file system <br />
  Amazon Linux AMI support <br />
  Webmail Branding <br />
  MultiPHP <br />
  Easy Apache 3 to 4 updates <br />
  Linux Container support <br />
  Common Mail providers tabs to WHM's Greylisting.</p>

  <p>Network Requirements</p>

  <p>Hostname should be a fully qualified domain name. <br />
  Should have a valid ipaddress, subnet address and default gateway <br />
  Ethernet device with a static IP &amp; hostname <br />
  Disable OS Firewall <br />
  Hardware Requirements</p>

  <p>Processor : 226 Mhz <br />
  RAM : Minimum 1GB ( 2GB recommended for smooth functioning) <br />
  Disk Space : Minimun 20 GB (40GB recommended for smooth functioning ) <br />
  Disable Selinux</p>

  <p>Need to edit the selinux configuration file located at : /etc/selinux/config</p>

  <p>Modify the SELINUX parameter to disabled and reboot the server.</p>

  <p>Installation steps followed for installing cPanel 11.52 on CentOS 7</p>

  <p>Before installation make sure the server meets all the pre-requisites.</p>

  <ol>
  <li>Set hostname to a fully qualified domain</li>
  </ol>

  <p>This is how we set the hostname for the server to a fully qualified domain name "server1.centos7-test.com" and confirm the hostname status once it is done.</p>

  <p>root@server1 [~]# <code>hostnamectl set-hostname server1.centos7-test.com –static</code> <br />
  root@server1 [~]# <code>hostnamectl status</code> <br />
  Static hostname: server1.centos7-test.com <br />
  Icon name: computer-vm <br />
  Chassis: vm <br />
  Machine ID: 72863e389b584a4dab36fae7f3bffda2 <br />
  Boot ID: 8bd4f714d7ba4ebf9d53f059d0a1fe8b <br />
  Virtualization: xen <br />
  Operating System: CentOS Linux 7 (Core) <br />
  CPE OS Name: cpe:/o:centos:centos:7 <br />
  Kernel: Linux 4.1.5-x86_64-linode61 <br />
  Architecture: x86-64 <br />
  root@server1 [~]#</p>

  <ol>
  <li>Check for the IP addr, subnet addr and gateway</li>
  </ol>

  <p>Make sure we have a static IP with a proper subnet mask and default gateway. In this case our IP address is 45.79.183.73 with a proper subnet mask and gateway.</p>

  <p>root@server1 [~]# <code>ifconfig</code> <br />
  eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 <br />
  inet 45.79.183.73 netmask 255.255.255.0 broadcast 45.79.183.255 <br />
  inet6 fe80::f03c:91ff:fef1:78d9 prefixlen 64 scopeid 0x20<link> <br />
  inet6 2600:3c03::f03c:91ff:fef1:78d9 prefixlen 64 scopeid 0x0<global> <br />
  ether f2:3c:91:f1:78:d9 txqueuelen 1000 (Ethernet) <br />
  RX packets 38095 bytes 13241381 (12.6 MiB) <br />
  RX errors 0 dropped 0 overruns 0 frame 0 <br />
  TX packets 41505 bytes 45527241 (43.4 MiB) <br />
  TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0</p>

  <ol>
  <li>Check the OS, disk usage and RAM usage for the server</li>
  </ol>

  <p>Our server is installed with CentOS 7.2, having a disk space of 20GB and around 1GB RAM. Please see the results of the hardware checks below:</p>

  <p>OS version check:</p>

  <p># <code>cat/etc/redhat-release</code></p>

  <p>root@server1 [~]# <code>cat /etc/redhat-release</code> <br />
  CentOS Linux release 7.2.1511 (Core) <br />
  root@server1 [~]#</p>

  <p>Disk usage check:</p>

  <p># <code>df-h</code></p>

  <p>root@server1 [~]# <code>df -h</code> <br />
  Filesystem Size Used Avail Use% Mounted on <br />
  /dev/root 20G 5.9G 14G 31% /
  devtmpfs 492M 0 492M 0% /dev <br />
  tmpfs 494M 0 494M 0% /dev/shm <br />
  tmpfs 494M 14M 480M 3% /run <br />
  tmpfs 494M 0 494M 0% /sys/fs/cgroup <br />
  /dev/loop0 662M 736K 627M 1% /tmp
  tmpfs 99M 0 99M 0% /run/user/0</p>

  <p>RAM usage check:</p>

  <p><code>free -m or cat /proc/meminfo</code></p>

  <p>root@server1 [~]# <code>free -m</code> <br />
  total used free shared buff/cache available <br />
  Mem: 987 336 275 12 375 607 <br />
  Swap: 511 1 510</p>

  <ol>
  <li>Disable Firewall</li>
  </ol>

  <p>It is recommended to disable the OS firewalls before the cPanel &amp; WHM installations. First of all we need to save the firewall rules and then stop the firewalld service. And make sure the service is disabled.</p>

  <p><code>iptables-save &gt; ~/firewall.rules</code>
  <code>systemctl stop firewalld.service</code>
  <code>systemctl disable firewalld.service</code></p>

  <p>root@server1 [~]# systemctl status firewalld.service <br />
  ● firewalld.service - firewalld - dynamic firewall daemon
  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) <br />
  Active: inactive (dead)</p>

  <p>Jan 09 06:50:28 server1.centos7-test.com systemd[1]: Stopped firewalld - dynamic firewall daemon.</p>

  <ol>
  <li>Disable Selinux</li>
  </ol>

  <p>Modify the SELINUX parameter to disabled in /etc/selinux/config and reboot the server. <br />
  root@server1 [~]# <code>cat /etc/selinux/config</code></p>

  <p># This file controls the state of SELinux on the system.
   # SELINUX= can take one of these three values:
   # enforcing - SELinux security policy is enforced.
   # permissive - SELinux prints warnings instead of enforcing.
   # disabled - No SELinux policy is loaded.
  SELINUX=disabled <br />
   # SELINUXTYPE= can take one of these two values:
   # targeted - Targeted processes are protected,
   # minimum - Modification of targeted policy. Only selected processes are protected.
   # mls - Multi Level Security protection.
  SELINUXTYPE=targeted</p>

  <ol>
  <li>Installation of Perl</li>
  </ol>

  <p>cPanel is programmed in Perl language. Hence, we need to install Perl prior to the cPanel installation if it isn't installed in the server before.</p>

  <p><code>yum install -y perl</code></p>

  <ol>
  <li>Download/Install/Run cPanel/WHM script.</li>
  </ol>

  <p>We need to download the cPanel installation script into the /home directory and run the script from there.</p>

  <p># <code>cd /home</code></p>

  <p># <code>curl -o latest -L https://securedownloads.cpanel.net/latest</code>
   # <code>sh latest</code></p>

  <p>This is how we download and install the cPanel installation script from our server's home directory.</p>

  <p>The script can take half an hour or more to complete the installation procedures.</p>

  <p>Installation log</p>

  <p>2016-01-07 07:15:51 227 ( INFO): cPanel install finished in 32 minutes and 14 seconds! <br />
  2016-01-07 07:15:51 964 ( INFO): Congratulations! Your installation of cPanel &amp; WHM 11.52 is now complete. The next step is to configure your server. <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): Before you configure your server, ensure that your firewall allows access on port 2087. <br />
  2016-01-07 07:15:51 964 ( INFO): On RHEL, CentOS, and CloudLinux systems, execute <code>/scripts/configure_firewall_for_cpanel</code> to accomplish this. <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): After ensuring that your firewall allows access on port 2087, you can configure your server. <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): 1. Open your preferred browser <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): 2. Type <a href="https://45.79.183.73:2087">https://45.79.183.73:2087</a> in the address bar <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): 3. Enter the word root in the Username text box <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): 4. Enter your root password in the Password text box <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): 5. Click the Login button <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): Visit <a href="http://go.cpanel.net/whminit">http://go.cpanel.net/whminit</a> for more information about first-time configuration of your server. <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): Visit <a href="http://support.cpanel.net">http://support.cpanel.net</a> or <a href="http://go.cpanel.net/whmfaq">http://go.cpanel.net/whmfaq</a> for additional support <br />
  2016-01-07 07:15:51 964 ( INFO): <br />
  2016-01-07 07:15:51 964 ( INFO): Thank you for installing cPanel &amp; WHM 11.52! <br />
  2016-01-07 07:15:51 835 ( WARN): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!</p>

  <ol>
  <li>Adding ports 2087/2086 to the firewalld service</li>
  </ol>

  <p>After completing the installation procedures, we need to ensure that our server allows the required service ports in the firewall.</p>

  <p>[root@server1 zones]# <code>firewall-cmd --zone=public --add-port=2087/udp</code>
  success <br />
  [root@server1 zones]# <code>firewall-cmd --zone=public --add-port=2086/udp</code>
  success <br />
  [root@server1 zones]# <code>firewall-cmd --zone=public --add-port=2086/tcp</code>
  success <br />
  [root@server1 zones]# <code>firewall-cmd --zone=public –add-port=2087/tcp</code></p>

  <p>[root@server1 zones]# <code>firewall-cmd --reload</code></p>

  <p>If your server has enough resources, then it is always advised to disable the OS firewall and configure CSF/APF to ensure better security and allow all required cPanel service ports by default. I would recommend to go with CSF for a cPanel server.</p>

  <p>CSF Installation steps:</p>

  <p><code>wget http://configserver.com/free/csf.tgz</code></p>

  <p><code>tar -xzf csf.tgz</code></p>

  <p><code>cd csf &amp; ./install.cpanel.sh</code></p>

  <p>Access the WHM via the URL</p>

  <p>You can access WHM using the URL <a href="https://server-hostname:2087">https://server-hostname:2087</a> or <a href="http://server-hostname:2086">http://server-hostname:2086</a></p>

  <p>After accessing the URL mentioned above, you can put your username as root and its password to login.</p>

  <p>Read through the terms and conditions &amp; accept the license agreement.</p>

  <p>Now you can do your initial configuration steps one by one as below:</p>

  <ol>
  <li>Setup Networking  </li>
  <li>Setup IP address  </li>
  <li>Nameservers  </li>
  <li>Services  </li>
  <li>Setup quotas</li>
  </ol>

  <p>Visit <a href="http://go.cpanel.net/whminit">http://go.cpanel.net/whminit</a> for more information about first-time configuration of your server.</p>

  <p>I believe this article will be a great help for you guys! Thank you for referring to this. I would appreciate your valuable comments for further improvements.</p>


uuid: aff22271-caee-453c-8d79-ab3799faef66
title: Reliable Mail relay solution
slug: reliable-mail-relay-solution
image: /content/images/2016/08/email-spam-600.jpg
status: published
language: en_US
markdown:
  Recently I discovered very good way how to create reliable SMTP relay solution and outbound SPAM protect for any cPanel server which I want to share with you. Many known Hosting providers sell you service with already blacklisted ip address. That can be very frustrating, but fortunately we have fix for that. Basically here is the thing you need to do.

  1) As you and I know every cPanel come with Spam Assassin. By default it's disabled and that is okay, but hey why don't we enable Spam Assassin to filter outgoing emails as much as it is awesome for incoming email. How to do that? Once you are in the WHM you can go to the search bar in the top left and type Exim and then choose the Exim Configuration Manager option. Once you are on the Exim Configuration Manager page you will stay on the Basic Editor, then you will select the SpamAssassin Options. If you do not see it then there will be a little blue arrow pointing to the right to let you scroll to see the other options. The last one being SpamAssassin Options. 

  More details you can read here:
  http://carvertown.com/enable-outgoing-spam-filtering-using-spamassassin/

  2) Okay now that this is enabled sign up here https://sendgrid.com/ . What is Sendgrid? SendGrid is a Boulder, Colorado-based transactional email delivery and management service. The company was founded by Isaac Saldana, Jose Lopez, and Tim Jenkins in 2009, and incubated through the TechStars accelerator program. So yes we can say its very similar business as Mail channels do, BUT big difference here is that they give 12,000 emails FREE OF CHARGE for first 45 days (enough to sort all things on server and stabilize things). Prices are very competitive after with only $9 for 40,000 emails / monthly.

  Check price list here:
  https://sendgrid.com/pricing

  3) Now that you got access to Sendgrid Dashboard please go to https://app.sendgrid.com/settings/mail_settings , scroll down to the bottom of page and enable Spam Checker (this will be secondary checker so yes filtered here even :)

  4) When this is finished we need to integrate Sendgrid with cPanel server. Detailed instructions how to integrate can be found here:
  https://sendgrid.com/docs/Integrate/Mail_Servers/exim.html
  (Section cPanel)

  One note:
  Restart all email exchange services and not just exim. That's pretty much all. Now you can follow up in real time how outgoing emails are going from app.sendgrid dashboard using awesome functions there like overview, suppression and etc.. Even you can get there review about bounces and spam messages so you can track source on server and easily disable.

  I hope this article can be useful to anyone who experience email delivery or spam issues.

html:
  <p>Recently I discovered very good way how to create reliable SMTP relay solution and outbound SPAM protect for any cPanel server which I want to share with you. Many known Hosting providers sell you service with already blacklisted ip address. That can be very frustrating, but fortunately we have fix for that. Basically here is the thing you need to do.</p>

  <p>1) As you and I know every cPanel come with Spam Assassin. By default it's disabled and that is okay, but hey why don't we enable Spam Assassin to filter outgoing emails as much as it is awesome for incoming email. How to do that? Once you are in the WHM you can go to the search bar in the top left and type Exim and then choose the Exim Configuration Manager option. Once you are on the Exim Configuration Manager page you will stay on the Basic Editor, then you will select the SpamAssassin Options. If you do not see it then there will be a little blue arrow pointing to the right to let you scroll to see the other options. The last one being SpamAssassin Options. </p>

  <p>More details you can read here: <br />
  <a href="http://carvertown.com/enable-outgoing-spam-filtering-using-spamassassin/">http://carvertown.com/enable-outgoing-spam-filtering-using-spamassassin/</a></p>

  <p>2) Okay now that this is enabled sign up here <a href="https://sendgrid.com/">https://sendgrid.com/</a> . What is Sendgrid? SendGrid is a Boulder, Colorado-based transactional email delivery and management service. The company was founded by Isaac Saldana, Jose Lopez, and Tim Jenkins in 2009, and incubated through the TechStars accelerator program. So yes we can say its very similar business as Mail channels do, BUT big difference here is that they give 12,000 emails FREE OF CHARGE for first 45 days (enough to sort all things on server and stabilize things). Prices are very competitive after with only $9 for 40,000 emails / monthly.</p>

  <p>Check price list here: <br />
  <a href="https://sendgrid.com/pricing">https://sendgrid.com/pricing</a></p>

  <p>3) Now that you got access to Sendgrid Dashboard please go to <a href="https://app.sendgrid.com/settings/mail_settings">https://app.sendgrid.com/settings/mail_settings</a> , scroll down to the bottom of page and enable Spam Checker (this will be secondary checker so yes filtered here even :)</p>

  <p>4) When this is finished we need to integrate Sendgrid with cPanel server. Detailed instructions how to integrate can be found here: <br />
  <a href="https://sendgrid.com/docs/Integrate/Mail_Servers/exim.html">https://sendgrid.com/docs/Integrate/Mail_Servers/exim.html</a> <br />
  (Section cPanel)</p>

  <p>One note: <br />
  Restart all email exchange services and not just exim. That's pretty much all. Now you can follow up in real time how outgoing emails are going from app.sendgrid dashboard using awesome functions there like overview, suppression and etc.. Even you can get there review about bounces and spam messages so you can track source on server and easily disable.</p>

  <p>I hope this article can be useful to anyone who experience email delivery or spam issues.</p>


uuid: 32645465-2b2a-4fb9-a5fd-c78c5bdfe9f3
title: How to Set Nginx as Reverse Proxy on Centos7 cPanel server
slug: how-to-set-nginx-as-reverse-proxy-on-centos7-cpanel
image: /content/images/2016/08/apache-nginx-varnish-530x295.jpg
status: published
language: en_US
markdown:
  Nginx is one of the fastest and most powerful web-server. It is known for its high performance and low resource utilization. It can be installed as both a standalone and a Reverse Proxy Web-server. In this article, I'm discussing about the installation of Nginx as a reverse proxy along with Apache on a cPanel server with latest CentOS 7 installed.

  Nginx as a reverse proxy will work as a frontend webserver serving static contents along with Apache serving the dynamic files in backend. This setup will boost up the overall server performance.

  Let's walk through the installation steps for Nginx as reverse proxy in CentOS7 x86_64 bit server with cPanel 11.54 installed.

  First of all, we need to install the EPEL repo to start-up with the process.

  Step 1 : Install the EPEL repo.

  root@server1 [/usr]# `yum -y install epel-release`
  Loaded plugins: fastestmirror, tsflags, universal-hooks
  Loading mirror speeds from cached hostfile
  * EA4: 66.23.237.210
  * base: mirrors.linode.com
  * extras: mirrors.linode.com
  * updates: mirrors.linode.com
  Resolving Dependencies
  --> Running transaction check
  ---> Package epel-release.noarch 0:7-5 will be installed
  --> Finished Dependency Resolution

  Dependencies Resolved
  Package Arch Version Repository Size
  Installing:
  epel-release noarch 7-5 extras 14 k

  Step 2: After installing the repo, we can start with the installation of the nDeploy RPM repo for CentOS to install our required nDeploy Webstack and Nginx plugin.

  root@server1 [/usr]# `yum -y install http://rpm.piserve.com/nDeploy-release-centos-1.0-1.noarch.rpm`
  Loaded plugins: fastestmirror, tsflags, universal-hooks
  nDeploy-release-centos-1.0-1.noarch.rpm | 1.7 kB 00:00:00
  Examining /var/tmp/yum-root-ei5tWJ/nDeploy-release-centos-1.0-1.noarch.rpm: nDeploy-release-centos-1.0-1.noarch
  Marking /var/tmp/yum-root-ei5tWJ/nDeploy-release-centos-1.0-1.noarch.rpm to be installed
  Resolving Dependencies
  --> Running transaction check
  ---> Package nDeploy-release-centos.noarch 0:1.0-1 will be installed
  --> Finished Dependency Resolution

  Dependencies Resolved
  Package Arch Version Repository Size

  Installing:
  nDeploy-release-centos noarch 1.0-1 /nDeploy-release-centos-1.0-1.noarch 110

  Step 3: Install the nDeploy and Nginx nDeploy plugins.

  root@server1 [/usr]# `yum --enablerepo=ndeploy install nginx-nDeploy nDeploy`
  Loaded plugins: fastestmirror, tsflags, universal-hooks
  epel/x86_64/metalink | 9.9 kB 00:00:00
  epel | 4.3 kB 00:00:00
  ndeploy | 2.9 kB 00:00:00
  (1/4): ndeploy/7/x86_64/primary_db | 14 kB 00:00:00
  (2/4): epel/x86_64/group_gz | 169 kB 00:00:00
  (3/4): epel/x86_64/primary_db | 3.7 MB 00:00:02

  Dependencies Resolved
  Package Arch Version Repository Size

  Installing:
  nDeploy noarch 2.0-11.el7 ndeploy 80 k
  nginx-nDeploy x86_64 1.8.0-34.el7 ndeploy 36 M
  Installing for dependencies:
  PyYAML x86_64 3.10-11.el7 base 153 k
  libevent x86_64 2.0.21-4.el7 base 214 k
  memcached x86_64 1.4.15-9.el7 base 84 k
  python-inotify noarch 0.9.4-4.el7 base 49 k
  python-lxml x86_64 3.2.1-4.el7 base 758 k

  Transaction Summary
  Install 2 Packages (+5 Dependent packages)

  With these steps, we've completed with the installation of Nginx plugin in our server. Now we need to configure Nginx as reverse proxy and create the virtualhost for the existing cPanel user accounts. For that we can run the following script.

  Step 4: To enable Nginx as a front end Web Server and create the default configuration files.

  root@server1 [/usr]# `/opt/nDeploy/scripts/cpanel-nDeploy-setup.sh enable`
  Modifying apache http and https port in cpanel

  httpd restarted successfully.
  Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service.
  Created symlink from /etc/systemd/system/multi-user.target.wants/ndeploy_watcher.service to /usr/lib/systemd/system/ndeploy_watcher.service.
  Created symlink from /etc/systemd/system/multi-user.target.wants/ndeploy_backends.service to /usr/lib/systemd/system/ndeploy_backends.service.
  ConfGen:: saheetha
  ConfGen:: satest

  As you can see these script will modify the Apache port from 80 to another port to make Nginx run as a front end web server and create the virtual host configuration files for the existing cPanel accounts. Once it is done, confirm the status of both Apache and Nginx.

  Apache Status:

  root@server1 [/var/run/httpd]# `systemctl status httpd`
  ● httpd.service - Apache Web Server
  Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Active: active (running) since Mon 2016-01-18 06:34:23 UTC; 12s ago
  Process: 25606 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
  Main PID: 24760 (httpd)
  CGroup: /system.slice/httpd.service
  ‣ 24760 /usr/local/apache/bin/httpd -k start

  Jan 18 06:34:23 server1.centos7-test.com systemd[1]: Starting Apache Web Server...
  Jan 18 06:34:23 server1.centos7-test.com apachectl[25606]: httpd (pid 24760) already running
  Jan 18 06:34:23 server1.centos7-test.com systemd[1]: Started Apache Web Server.

  Nginx Status:

  root@server1 [~]# `systemctl status nginx`
  ● nginx.service - nginx-nDeploy - high performance web server
  Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
  Active: active (running) since Sun 2016-01-17 17:18:29 UTC; 13h ago
  Docs: http://nginx.org/en/docs/
  Main PID: 3833 (nginx)
  CGroup: /system.slice/nginx.service
  ├─ 3833 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
  ├─25473 nginx: worker process
  ├─25474 nginx: worker process
  └─25475 nginx: cache manager process

  Jan 17 17:18:29 server1.centos7-test.com systemd[1]: Starting nginx-nDeploy - high performance web server...
  Jan 17 17:18:29 server1.centos7-test.com nginx[3804]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
  Jan 17 17:18:29 server1.centos7-test.com nginx[3804]: nginx: configuration file /etc/nginx/nginx.conf test is successful
  Jan 17 17:18:29 server1.centos7-test.com systemd[1]: Started nginx-nDeploy - high performance web server.

  Nginx act as a frontend webserver running on port 80 and Apache configuration is modified to listen on http port 9999 and https port 4430. Please see their status below:

  root@server1 [/usr/local/src]# `netstat -plan | grep httpd`
  tcp 0 0 0.0.0.0:4430 0.0.0.0:* LISTEN 17270/httpd
  tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 17270/httpd
  tcp6 0 0 :::4430 :::* LISTEN 17270/httpd
  tcp6 0 0 :::9999 :::* LISTEN 17270/httpd

  apacheport
  root@server1 [/usr/local/src]# `netstat -plan | grep nginx`
  tcp 0 0 127.0.0.1:80 0.0.0.0:* LISTEN 17802/nginx: master
  tcp 0 0 45.79.183.73:80 0.0.0.0:* LISTEN 17802/nginx: master

  Or simply you can rung lsof -i :80 and lsof -i :9999 to see same.  

  The virtualhost entries created for the existing users as located in the folder "/etc/nginx/sites-enabled". This file path is included in the Nginx main configuration file.

  root@server1 [/etc/nginx/sites-enabled]# `ll | grep .conf`
  -rw-r--r-- 1 root root 311 Jan 17 09:02 saheetha.com.conf
  -rw-r--r-- 1 root root 336 Jan 17 09:02 saheethastest.com.conf

  Sample Vhost for a domain:

  server {

  listen 45.79.183.73:80;
   #CPIPVSIX:80;

   # ServerNames
  server_name domain.com www.domain.com;
  access_log /usr/local/apache/domlogs/domain.com main;
  access_log /usr/local/apache/domlogs/domain.com-bytes_log bytes_log;

  include /etc/nginx/sites-enabled/domain.com.include;

  }

  We can confirm the working of the web server status by calling a website in the browser. Please see the web server information on my server after the installation.

  root@server1 [/home]# `ip a | grep -i eth0`
  3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
  inet 45.79.183.73/24 brd 45.79.183.255 scope global dynamic eth0
  root@server1 [/home]# nginx -v
  nginx version: nginx/1.8.0

  webserver-status

  Nginx will create the virtual host automatically for any newly created accounts in cPanel. With these simple steps we can configure Nginx as reverse proxy on a CentOS 7/CPanel server.

  Advantages of Nginx as Reverse Proxy:

  Easy to install and configure
  Performance and efficiency
  Prevent DDOS attacks
  Allows .htaccess PHP rewrite rules

  I hope this article is useful for you guys. Thank you for referring to this. I would appreciate your valuable comments and suggestions on this for further improvements.

html:
  <p>Nginx is one of the fastest and most powerful web-server. It is known for its high performance and low resource utilization. It can be installed as both a standalone and a Reverse Proxy Web-server. In this article, I'm discussing about the installation of Nginx as a reverse proxy along with Apache on a cPanel server with latest CentOS 7 installed.</p>

  <p>Nginx as a reverse proxy will work as a frontend webserver serving static contents along with Apache serving the dynamic files in backend. This setup will boost up the overall server performance.</p>

  <p>Let's walk through the installation steps for Nginx as reverse proxy in CentOS7 x86_64 bit server with cPanel 11.54 installed.</p>

  <p>First of all, we need to install the EPEL repo to start-up with the process.</p>

  <p>Step 1 : Install the EPEL repo.</p>

  <p>root@server1 [/usr]# <code>yum -y install epel-release</code> <br />
  Loaded plugins: fastestmirror, tsflags, universal-hooks <br />
  Loading mirror speeds from cached hostfile <br />
  * EA4: 66.23.237.210
  * base: mirrors.linode.com
  * extras: mirrors.linode.com
  * updates: mirrors.linode.com
  Resolving Dependencies <br />
  --> Running transaction check
  ---> Package epel-release.noarch 0:7-5 will be installed
  --> Finished Dependency Resolution</p>

  <p>Dependencies Resolved <br />
  Package Arch Version Repository Size <br />
  Installing: <br />
  epel-release noarch 7-5 extras 14 k</p>

  <p>Step 2: After installing the repo, we can start with the installation of the nDeploy RPM repo for CentOS to install our required nDeploy Webstack and Nginx plugin.</p>

  <p>root@server1 [/usr]# <code>yum -y install http://rpm.piserve.com/nDeploy-release-centos-1.0-1.noarch.rpm</code> <br />
  Loaded plugins: fastestmirror, tsflags, universal-hooks <br />
  nDeploy-release-centos-1.0-1.noarch.rpm | 1.7 kB 00:00:00 <br />
  Examining /var/tmp/yum-root-ei5tWJ/nDeploy-release-centos-1.0-1.noarch.rpm: nDeploy-release-centos-1.0-1.noarch <br />
  Marking /var/tmp/yum-root-ei5tWJ/nDeploy-release-centos-1.0-1.noarch.rpm to be installed <br />
  Resolving Dependencies <br />
  --> Running transaction check
  ---> Package nDeploy-release-centos.noarch 0:1.0-1 will be installed
  --> Finished Dependency Resolution</p>

  <p>Dependencies Resolved <br />
  Package Arch Version Repository Size</p>

  <p>Installing: <br />
  nDeploy-release-centos noarch 1.0-1 /nDeploy-release-centos-1.0-1.noarch 110</p>

  <p>Step 3: Install the nDeploy and Nginx nDeploy plugins.</p>

  <p>root@server1 [/usr]# <code>yum --enablerepo=ndeploy install nginx-nDeploy nDeploy</code> <br />
  Loaded plugins: fastestmirror, tsflags, universal-hooks <br />
  epel/x86<em>64/metalink | 9.9 kB 00:00:00 <br />
  epel | 4.3 kB 00:00:00 <br />
  ndeploy | 2.9 kB 00:00:00 <br />
  (1/4): ndeploy/7/x86</em>64/primary<em>db | 14 kB 00:00:00
  (2/4): epel/x86</em>64/group<em>gz | 169 kB 00:00:00
  (3/4): epel/x86</em>64/primary_db | 3.7 MB 00:00:02</p>

  <p>Dependencies Resolved <br />
  Package Arch Version Repository Size</p>

  <p>Installing: <br />
  nDeploy noarch 2.0-11.el7 ndeploy 80 k <br />
  nginx-nDeploy x86<em>64 1.8.0-34.el7 ndeploy 36 M <br />
  Installing for dependencies: <br />
  PyYAML x86</em>64 3.10-11.el7 base 153 k <br />
  libevent x86<em>64 2.0.21-4.el7 base 214 k <br />
  memcached x86</em>64 1.4.15-9.el7 base 84 k <br />
  python-inotify noarch 0.9.4-4.el7 base 49 k <br />
  python-lxml x86_64 3.2.1-4.el7 base 758 k</p>

  <p>Transaction Summary <br />
  Install 2 Packages (+5 Dependent packages)</p>

  <p>With these steps, we've completed with the installation of Nginx plugin in our server. Now we need to configure Nginx as reverse proxy and create the virtualhost for the existing cPanel user accounts. For that we can run the following script.</p>

  <p>Step 4: To enable Nginx as a front end Web Server and create the default configuration files.</p>

  <p>root@server1 [/usr]# <code>/opt/nDeploy/scripts/cpanel-nDeploy-setup.sh enable</code> <br />
  Modifying apache http and https port in cpanel</p>

  <p>httpd restarted successfully. <br />
  Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service. <br />
  Created symlink from /etc/systemd/system/multi-user.target.wants/ndeploy<em>watcher.service to /usr/lib/systemd/system/ndeploy</em>watcher.service. <br />
  Created symlink from /etc/systemd/system/multi-user.target.wants/ndeploy<em>backends.service to /usr/lib/systemd/system/ndeploy</em>backends.service. <br />
  ConfGen:: saheetha <br />
  ConfGen:: satest</p>

  <p>As you can see these script will modify the Apache port from 80 to another port to make Nginx run as a front end web server and create the virtual host configuration files for the existing cPanel accounts. Once it is done, confirm the status of both Apache and Nginx.</p>

  <p>Apache Status:</p>

  <p>root@server1 [/var/run/httpd]# <code>systemctl status httpd</code> <br />
  ● httpd.service - Apache Web Server
  Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) <br />
  Active: active (running) since Mon 2016-01-18 06:34:23 UTC; 12s ago <br />
  Process: 25606 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS) <br />
  Main PID: 24760 (httpd) <br />
  CGroup: /system.slice/httpd.service <br />
  ‣ 24760 /usr/local/apache/bin/httpd -k start</p>

  <p>Jan 18 06:34:23 server1.centos7-test.com systemd[1]: Starting Apache Web Server... <br />
  Jan 18 06:34:23 server1.centos7-test.com apachectl[25606]: httpd (pid 24760) already running <br />
  Jan 18 06:34:23 server1.centos7-test.com systemd[1]: Started Apache Web Server.</p>

  <p>Nginx Status:</p>

  <p>root@server1 [~]# <code>systemctl status nginx</code> <br />
  ● nginx.service - nginx-nDeploy - high performance web server
  Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled) <br />
  Active: active (running) since Sun 2016-01-17 17:18:29 UTC; 13h ago <br />
  Docs: <a href="http://nginx.org/en/docs/">http://nginx.org/en/docs/</a> <br />
  Main PID: 3833 (nginx) <br />
  CGroup: /system.slice/nginx.service <br />
  ├─ 3833 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
  ├─25473 nginx: worker process
  ├─25474 nginx: worker process
  └─25475 nginx: cache manager process</p>

  <p>Jan 17 17:18:29 server1.centos7-test.com systemd[1]: Starting nginx-nDeploy - high performance web server... <br />
  Jan 17 17:18:29 server1.centos7-test.com nginx[3804]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok <br />
  Jan 17 17:18:29 server1.centos7-test.com nginx[3804]: nginx: configuration file /etc/nginx/nginx.conf test is successful <br />
  Jan 17 17:18:29 server1.centos7-test.com systemd[1]: Started nginx-nDeploy - high performance web server.</p>

  <p>Nginx act as a frontend webserver running on port 80 and Apache configuration is modified to listen on http port 9999 and https port 4430. Please see their status below:</p>

  <p>root@server1 [/usr/local/src]# <code>netstat -plan | grep httpd</code> <br />
  tcp 0 0 0.0.0.0:4430 0.0.0.0:* LISTEN 17270/httpd <br />
  tcp 0 0 0.0.0.0:9999 0.0.0.0:* LISTEN 17270/httpd <br />
  tcp6 0 0 :::4430 :::* LISTEN 17270/httpd <br />
  tcp6 0 0 :::9999 :::* LISTEN 17270/httpd</p>

  <p>apacheport <br />
  root@server1 [/usr/local/src]# <code>netstat -plan | grep nginx</code> <br />
  tcp 0 0 127.0.0.1:80 0.0.0.0:* LISTEN 17802/nginx: master <br />
  tcp 0 0 45.79.183.73:80 0.0.0.0:* LISTEN 17802/nginx: master</p>

  <p>Or simply you can rung lsof -i :80 and lsof -i :9999 to see same.  </p>

  <p>The virtualhost entries created for the existing users as located in the folder "/etc/nginx/sites-enabled". This file path is included in the Nginx main configuration file.</p>

  <p>root@server1 [/etc/nginx/sites-enabled]# <code>ll | grep .conf</code> <br />
  -rw-r--r-- 1 root root 311 Jan 17 09:02 saheetha.com.conf
  -rw-r--r-- 1 root root 336 Jan 17 09:02 saheethastest.com.conf</p>

  <p>Sample Vhost for a domain:</p>

  <p>server {</p>

  <p>listen 45.79.183.73:80; <br />
   #CPIPVSIX:80;</p>

  <p># ServerNames
  server<em>name domain.com www.domain.com; <br />
  access</em>log /usr/local/apache/domlogs/domain.com main; <br />
  access<em>log /usr/local/apache/domlogs/domain.com-bytes</em>log bytes_log;</p>

  <p>include /etc/nginx/sites-enabled/domain.com.include;</p>

  <p>}</p>

  <p>We can confirm the working of the web server status by calling a website in the browser. Please see the web server information on my server after the installation.</p>

  <p>root@server1 [/home]# <code>ip a | grep -i eth0</code> <br />
  3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 <br />
  inet 45.79.183.73/24 brd 45.79.183.255 scope global dynamic eth0 <br />
  root@server1 [/home]# nginx -v <br />
  nginx version: nginx/1.8.0</p>

  <p>webserver-status</p>

  <p>Nginx will create the virtual host automatically for any newly created accounts in cPanel. With these simple steps we can configure Nginx as reverse proxy on a CentOS 7/CPanel server.</p>

  <p>Advantages of Nginx as Reverse Proxy:</p>

  <p>Easy to install and configure <br />
  Performance and efficiency <br />
  Prevent DDOS attacks <br />
  Allows .htaccess PHP rewrite rules</p>

  <p>I hope this article is useful for you guys. Thank you for referring to this. I would appreciate your valuable comments and suggestions on this for further improvements.</p>


uuid: b4f07755-d1f8-41c8-bb54-d0cbe9fb9c54
title: My Resume and LinkedIn
slug: resume
image: /content/images/2016/08/sysadmin.jpg
status: published
language: en_US
markdown:
  ![alt](http://repo.nemanja.io/nemke-card.jpg)
  If you need to reach me directly you can always email me ne(at)nemanja.io

  wget -c http://repo.nemanja.io/nemke-card.jpg

  My LinkedIn page is:
  https://www.linkedin.com/in/nemke82

  My GitHub page is:
  https://github.com/nemke82

  Please view my resume from here:
  http://repo.nemanja.io/nemke-resume.pdf

  If you need my services or to reach me, you can schedule an appointment here:
  https://calendly.com/nemanjadjuric/60min

html:
  <p><img src="http://repo.nemanja.io/nemke-card.jpg" alt="alt" />
  If you need to reach me directly you can always email me ne(at)nemanja.io</p>

  <p>wget -c <a href="http://repo.nemanja.io/nemke-card.jpg">http://repo.nemanja.io/nemke-card.jpg</a></p>

  <p>My LinkedIn page is: <br />
  <a href="https://www.linkedin.com/in/nemke82">https://www.linkedin.com/in/nemke82</a></p>

  <p>My GitHub page is: <br />
  <a href="https://github.com/nemke82">https://github.com/nemke82</a></p>

  <p>Please view my resume from here: <br />
  <a href="http://repo.nemanja.io/nemke-resume.pdf">http://repo.nemanja.io/nemke-resume.pdf</a></p>

  <p>If you need my services or to reach me, you can schedule an appointment here: <br />
  <a href="https://calendly.com/nemanjadjuric/60min">https://calendly.com/nemanjadjuric/60min</a></p>


uuid: 1f31d8f6-2664-45e7-a2f1-d4e6536c2f60
title: How to install Varnish on cPanel server
slug: how-to-install-varnish-on-cpanel-server
image: /content/images/2016/08/varnish-cache-step-by-step-1-638.jpg
status: published
language: en_US
markdown:
  In my previous post, I explained how to setup Nginx as a reverse proxy on a CentOS 7 server with cPanel installed. Now, I will explain how to install Varnish which will provide additional performance to the server and boost the site's page load time.

  Varnish is an HTTP accelerator designed for content-heavy dynamic web sites as well as heavily consumed APIs. In contrast to other web accelerators, such as Squid, which began life as a client-side cache, or Apache and Nginx, which are primarily origin servers, Varnish was designed as an HTTP accelerator. The main developers being Poul-Henning Kamp (https://en.wikipedia.org/wiki/Poul-Henning_Kamp) and Redpill-Linpro who are the founders of the Varnish Software licensed under a two-clause BSD license.

  This tutorial will show you how to use the Varnish caching proxy on your server.

  When you are done with this tutorial, the request handling on your server will look like this:

  **Browser --> Nginx --> Varnish --> Apache --> PHP**

  Nginx will still be the public-facing server because Nginx supports SSL and HTTP/2 and uses a more scalable asynchronous event-driven model. That is, Varnish will only be used for what it's best at: caching.

  1) First, SSH into your server as the root user so we can install and configure the system. You can work from /root folder or any other you prefer.

  2) Visit http://repo.varnish-cache.org/ and choose the proper version (for cPanel installed on CentOS 6 — Varnish 3.0.7 is optimal) if CentOS 7 then Varnish 4 can be installed (for example: http://repo.varnish-cache.org/redhat/varnish-4.0/el7/noarch/varnish-release/varnish-release-4.0-4.el7.noarch.rpm)

  3) Make sure that EPEL repo is installed: 
  `yum install epel-release –y`
  (if you have followed my previous post then it is).

  4) `rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-4.0/el7/noarch/varnish-release/varnish-release-4.0-4.el7.noarch.rpm` 
  (or whatever you choose from the Varnish repo site)

  5) `yum install varnish` 
  (so we can get varnish installed)

  6) After Varnish is successfully installed you can edit the `/etc/sysconfig/varnish` file if you want Varnish to run on a port other than 6081 (6082 for admin)

  7) edit /etc/varnish/default.vcl file and adjust section: 

  `backend default {
  .host = "x.x.x.x";
  .port = "8080";
  }`

  Note: adjust x.x.x.x with your server ip address to where Apache is listening.

  At the end let's say what will be cached, in this case uncomment or add: sub vcl_recv { if (req.url ~ “.(png|gif|jpg|swf|css|js)$”) { return(lookup); } }

  set req.http.X-Forwarded-For = client.ip;

  }

  sub vcl_fetch { if (req.url ~ “.(png|gif|jpg|swf|css|js)$”) { unset beresp.http.set-cookie; } }

  Note: You can un-comment lines vclrecv { , sub vclpipe { , sub vclpass { , vclhash, vclhit and vclmiss if you wish additional things to be parsed.

  At the end push: `chkconfig varnish on && service varnish start`

  NOTE: Make sure that 6081 / 6082 is enabled within APF/CSF or whichever you choose. You can use `varnishstat` or `varnishtop` to check for next.

  Also, you can use commands like : 

  `lsof -i :6081`

  `lsof -i :6082`

  To check if Varnish is listening and online.

  8) Now you can adjust (if you are using nDeploy Nginx version) /etc/nginx/conf.d/defaultserver.conf file and add proxypass / upstream to listen to 6081 port. If you want Varnish to be stand alone move Apache to port 8080 and then put Varnish on 80 port. If you want to enable Varnish selectively for some sites, please adjust the configuration file for the website at /etc/nginx/sites-enabled folder.

  That's all! Test your site using curl -I domain.com to see if Varnish will be displayed.

  Additional issues which could happen: If you have an issue with the IP address not being logged well, please install mod_rpaf and make sure the following line is added : <IfModule mod_rpaf-2.0.c>

  RPAFenable On
  RPAFsethostname On
  RPAFproxy_ips 127.0.0.1 127.0.0.1 <my-public-ip>
  RPAFheader X-Real-IP   <--- From X-Forwarded-For  to X-Real-IP
  </IfModule>

  Restart Apache and that's all!

  Sometimes you need to purge the Varnish cache. If you need to, please push the following on your server: 
  `varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret “ban req.url ~ /”`

  I hope this article is useful for you guys. Thank you for referring to this. I would appreciate your valuable comments and suggestions on this for further improvements. In next article, I will write about Memcached which is used for MySQL query caching.

html:
  <p>In my previous post, I explained how to setup Nginx as a reverse proxy on a CentOS 7 server with cPanel installed. Now, I will explain how to install Varnish which will provide additional performance to the server and boost the site's page load time.</p>

  <p>Varnish is an HTTP accelerator designed for content-heavy dynamic web sites as well as heavily consumed APIs. In contrast to other web accelerators, such as Squid, which began life as a client-side cache, or Apache and Nginx, which are primarily origin servers, Varnish was designed as an HTTP accelerator. The main developers being Poul-Henning Kamp (<a href="https://en.wikipedia.org/wiki/Poul-Henning_Kamp">https://en.wikipedia.org/wiki/Poul-Henning_Kamp</a>) and Redpill-Linpro who are the founders of the Varnish Software licensed under a two-clause BSD license.</p>

  <p>This tutorial will show you how to use the Varnish caching proxy on your server.</p>

  <p>When you are done with this tutorial, the request handling on your server will look like this:</p>

  <p><strong>Browser --> Nginx --> Varnish --> Apache --> PHP</strong></p>

  <p>Nginx will still be the public-facing server because Nginx supports SSL and HTTP/2 and uses a more scalable asynchronous event-driven model. That is, Varnish will only be used for what it's best at: caching.</p>

  <p>1) First, SSH into your server as the root user so we can install and configure the system. You can work from /root folder or any other you prefer.</p>

  <p>2) Visit <a href="http://repo.varnish-cache.org/">http://repo.varnish-cache.org/</a> and choose the proper version (for cPanel installed on CentOS 6 — Varnish 3.0.7 is optimal) if CentOS 7 then Varnish 4 can be installed (for example: <a href="http://repo.varnish-cache.org/redhat/varnish-4.0/el7/noarch/varnish-release/varnish-release-4.0-4.el7.noarch.rpm">http://repo.varnish-cache.org/redhat/varnish-4.0/el7/noarch/varnish-release/varnish-release-4.0-4.el7.noarch.rpm</a>)</p>

  <p>3) Make sure that EPEL repo is installed: <br />
  <code>yum install epel-release –y</code>
  (if you have followed my previous post then it is).</p>

  <p>4) <code>rpm --nosignature -i http://repo.varnish-cache.org/redhat/varnish-4.0/el7/noarch/varnish-release/varnish-release-4.0-4.el7.noarch.rpm</code> <br />
  (or whatever you choose from the Varnish repo site)</p>

  <p>5) <code>yum install varnish</code> <br />
  (so we can get varnish installed)</p>

  <p>6) After Varnish is successfully installed you can edit the <code>/etc/sysconfig/varnish</code> file if you want Varnish to run on a port other than 6081 (6082 for admin)</p>

  <p>7) edit /etc/varnish/default.vcl file and adjust section: </p>

  <p><code>backend default {
  .host = "x.x.x.x";
  .port = "8080";
  }</code></p>

  <p>Note: adjust x.x.x.x with your server ip address to where Apache is listening.</p>

  <p>At the end let's say what will be cached, in this case uncomment or add: sub vcl_recv { if (req.url ~ “.(png|gif|jpg|swf|css|js)$”) { return(lookup); } }</p>

  <p>set req.http.X-Forwarded-For = client.ip;</p>

  <p>}</p>

  <p>sub vcl_fetch { if (req.url ~ “.(png|gif|jpg|swf|css|js)$”) { unset beresp.http.set-cookie; } }</p>

  <p>Note: You can un-comment lines vclrecv { , sub vclpipe { , sub vclpass { , vclhash, vclhit and vclmiss if you wish additional things to be parsed.</p>

  <p>At the end push: <code>chkconfig varnish on &amp;&amp; service varnish start</code></p>

  <p>NOTE: Make sure that 6081 / 6082 is enabled within APF/CSF or whichever you choose. You can use <code>varnishstat</code> or <code>varnishtop</code> to check for next.</p>

  <p>Also, you can use commands like : </p>

  <p><code>lsof -i :6081</code></p>

  <p><code>lsof -i :6082</code></p>

  <p>To check if Varnish is listening and online.</p>

  <p>8) Now you can adjust (if you are using nDeploy Nginx version) /etc/nginx/conf.d/defaultserver.conf file and add proxypass / upstream to listen to 6081 port. If you want Varnish to be stand alone move Apache to port 8080 and then put Varnish on 80 port. If you want to enable Varnish selectively for some sites, please adjust the configuration file for the website at /etc/nginx/sites-enabled folder.</p>

  <p>That's all! Test your site using curl -I domain.com to see if Varnish will be displayed.</p>

  <p>Additional issues which could happen: If you have an issue with the IP address not being logged well, please install mod_rpaf and make sure the following line is added : <IfModule mod_rpaf-2.0.c></p>

  <p>RPAFenable On <br />
  RPAFsethostname On <br />
  RPAFproxy_ips 127.0.0.1 127.0.0.1 <my-public-ip> <br />
  RPAFheader X-Real-IP   &lt;--- From X-Forwarded-For  to X-Real-IP <br />
  </IfModule></p>

  <p>Restart Apache and that's all!</p>

  <p>Sometimes you need to purge the Varnish cache. If you need to, please push the following on your server: <br />
  <code>varnishadm -T 127.0.0.1:6082 -S /etc/varnish/secret “ban req.url ~ /”</code></p>

  <p>I hope this article is useful for you guys. Thank you for referring to this. I would appreciate your valuable comments and suggestions on this for further improvements. In next article, I will write about Memcached which is used for MySQL query caching.</p>


uuid: 6e66cc4a-b17e-4c17-8e53-ab86a7ccad35
title: Scalable VPN solution -- SoftEther VPN how-to install on CentOS 6 and 7
slug: vpn-how-to-get-started-and-install-on-centos-6-and-7-servers
image: /content/images/2017/06/softetherlogo.jpg
status: published
language: en_US
markdown:
  Today I will share how to create a VPN network and how to utilize it cost effectively or even for free. Recently, I've started to look at all of the possible solutions in this area and today I will mention two, which for me are ideal with regards to installation, maintenance, and optimal resource usage.

  What is a VPN?
  A VPN or Virtual Private Network is a method used to add security and privacy to private and public networks, like WiFi Hotspots and the Internet. VPNs are more commonly used by corporations to protect sensitive data. VPNs can provide functionality, security and/or network management benefits to the user. But they can also lead to new issues, and some VPN services, especially "free" ones, can actually violate their users' privacy by logging their usage and making it available without their consent or make money by selling the user's bandwidth to other users.

  Some VPNs allow employees to securely access a corporate intranet while located outside the office. Some can securely connect geographically separated offices of an organisation, creating one cohesive network. Individual Internet users can use some VPNs to secure their wireless transactions, to circumvent geo-restrictions and censorship, and/or to connect to proxy servers for the purpose of protecting personal identities and locations. But some Internet sites block access via known VPNs to prevent the circumvention of their geo-restrictions.

  A VPN is created by establishing a virtual point-to-point connection through the use of dedicated connections, virtual tunnelling protocols, or traffic encryption. A VPN available from the public Internet can provide some of the benefits of a wide area network (WAN). From a user perspective, the resources available within the private network can be accessed remotely or kept private.

  The first software that I tested was OpenVPN which comes from OpenVPN Technologies, Inc. James Yonan. The initial release was pushed on 13th, May 2001. They have regular updates and it declares itself as an open-source software application, but not all parts are "free". I will write more about that later.

  Looking for other VPN solutions I've run into the Softether VPN server application coming from the University of Tsukuba in Japan. Softether has long been a proprietary product under name PacketX and it has been open sourced just several years ago. That may be the reason why it is Windows oriented. The configuration GUI is Windows only and connecting from Linux clients requires extra work, but it also has Linux console which can be combined. Today we are going to cover both.

  We are testing this on plain CentOS 7 server, in my case KVM QEMU instance, but can be booted on anything really.

  First let's prepare system:
  1) `yum update`
  2) `yum -y groupinstall "Development Tools"`
  3) 
  ```
  yum -y install gcc zlib-devel openssl-devel readline-devel ncurses-devel wget tar dnsmasq net-tools iptables-services system-config-firewall-tui nano iptables-services
  ```

  Disable SELinux:
  ```
  sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
  ```

  Now we can start with server installation:
  4) cd /usr/local
  5) 
  ```
  wget -c http://www.softether-download.com/files/softether/v4.22-9634-beta-2016.11.27-tree/Linux/SoftEther_VPN_Server/64bit_-_Intel_x64_or_AMD64/softether-vpnserver-v4.22-9634-beta-2016.11.27-linux-x64-64bit.tar.gz
  ```
  (you can visit https://www.softether.org/5-download website to check for the latest version)

  6) extract archive within /usr/local/ folder:
  `tar -xvf softether-vpnserver-v4.22-9634-beta-2016.11.27-linux-x64-64bit.tar.gz`
  ![alt](/content/images/2017/06/softethervpn-1.png)

  7) cd /usr/local/vpnserver ; make
  ![alt](/content/images/2017/06/softethervpn-2.png)

  Select 1 and accept License Agreement.

  8) The developers of this software were really detailed and descriptive, so they have added everything at end of installation. If you get this at the end of your installation, it means that you have managed successfully to install SoftEther VPN application on your CentOS 7 server.

  --------------------------------------------------------------------
  The preparation of SoftEther VPN Server is completed!

  *** How to switch the display language of the SoftEther VPN Server Service ***
  SoftEther VPN Server supports the following languages:
    - Japanese
    - English
    - Simplified Chinese

  You can choose your preferred language of SoftEther VPN Server at any time.

  To switch the current language, open and edit the 'lang.config' file.

  *** How to start the SoftEther VPN Server Service ***

  Please execute './vpnserver start' to run the SoftEther VPN Server Background Service.
  And please execute './vpncmd' to run the SoftEther VPN Command-Line Utility to configure SoftEther VPN Server.

  Of course, you can use the VPN Server Manager GUI Application for Windows / Mac OS X on the other Windows / Mac OS X computers in order to configure the SoftEther VPN Server remotely.

  *** For Windows users ***
  You can download the SoftEther VPN Server Manager for Windows
  from the http://www.softether-download.com/ web site.
  This manager application helps you to completely and easily manage the VPN server services running in remote hosts.

  *** For Mac OS X users ***
  In April 2016 we released the SoftEther VPN Server Manager for Mac OS X.
  You can download it from the http://www.softether-download.com/ web site.
  VPN Server Manager for Mac OS X works perfectly as same as the traditional Windows versions. It helps you to completely and easily manage the VPN server services running in remote hosts.

  --------------------------------------------------------------------
  make[1]: Leaving directory `/usr/local/vpnserver'

  9) Next we need to make the init script for softether application because developers have not included into the install. Run nano /etc/init.d/vpnserver and make paste this script.

  ```
  #!/bin/sh
  ### BEGIN INIT INFO
  # Provides: vpnserver
  # Required-Start: $remote_fs $syslog
  # Required-Stop: $remote_fs $syslog
  # Default-Start: 2 3 4 5
  # Default-Stop: 0 1 6
  # Short-Description: Start daemon at boot time
  # Description: Enable Softether by daemon.
  ### END INIT INFO
  DAEMON=/usr/local/vpnserver/vpnserver
  LOCK=/var/lock/subsys/vpnserver
  TAP_ADDR=10.8.8.1

  test -x $DAEMON || exit 0
  case "$1" in
  start)
  $DAEMON start
  touch $LOCK
  sleep 1
  /sbin/ifconfig tap_soft $TAP_ADDR
  ;;
  stop)
  $DAEMON stop
  rm $LOCK
  ;;
  restart)
  $DAEMON stop
  sleep 3
  $DAEMON start
  sleep 1
  /sbin/ifconfig tap_soft $TAP_ADDR
  ;;
  *)
  echo "Usage: $0 {start|stop|restart}"
  exit 1
  esac
  exit 0
  ```

  7) Next step we need to make shell script that we just created before with executable rules, so it can be started properly during the boot process and manually.
  ```
  chmod +x /etc/init.d/vpnserver ; /etc/init.d/vpnserver start ; systemctl enable vpnserver
  ```

  ![alt](/content/images/2017/06/softethervpn-3.png)

  Note: Please ignore alert about tap_soft interface. It is added into the vpnserver bash script, and we will configure it later during this tutorial. 
  **In this tutorial we are going to use 10.8.8.0/24 as example range, and that is why TAP_ADDR=10.8.8.1 as server address. You can use anything you wish.**

  Moving over to the configuration part, we need to start vpncmd utility (Linux method), later I will add the easier way by installing SE VPN Server Manager Tool. Software is made in that way that everything can be configured from shell and GUI mode leaving to the end-user preferred method.

  `/usr/local/vpnserver/vpncmd`

  Press 1 to select "Management of VPN Server or VPN Bridge", and then when asked which server to configure, press enter and it will choose `localhost` where you just installed Softether application. Press Enter one more time to get access to the server as Administrator. 

  Next type:
  ```
  ServerPasswordSet
  ```
  to set the admin password for the server. In order to use softether, virtual hub needs to be created. We will create one named one.nemanja.io with the following command:
  ```
  HubCreate MOB
  ```
  You will be prompted to set the password, which you will use to administer a hub, without access to entire VPN server. Practically one VPN server can have unlimited Hubs connecting multiple customers, and that is why this software is so unique.

  Now we need to create a local bridge. That is more efficient way, comparing to other. There is also built-in SecureNAT which is easier to setup, but sadly it is resource intensive. We will go with setting up local bridge and tap device, note that with local bridge also DHCP server needs to be configured and installed which we are going to do bellow. Local bridge is created with the following command:
  ```
  BridgeCreate /DEVICE:"soft" /TAP:yes one.nemanja.io
  ```
  If TAP device creation fails with the message about insufficient privileges, you might want to check if your network controller is set in promiscuous mode. If you are using some popular Virtualization software like SolusVM/OnApp, make sure that Tun/Tap option is enabled for the VPS. I haven't had problems with Proxmox or with using Xen, so I believe this is enabled by default.

  Now we need to create a user for the one.nemanja.io virtual hub that we have previously created. Users are created with command `UserCreate` and you can view the list of users by command `UserList`. Users can be added to groups and each group can have different authentication mode/privileges, for example, Password, Certificate, RADIUS, NTLM and etc etc.

  **Configuring the Virtual Hub**

  Now we need to switch to hub one.nemanja.io
  `Hub one.nemanja.io`

  and create user
  `UserCreate test`
  <BR>
  We will keep it simple and use password `auth`, so use the following command
  ```
  UserPasswordSet test
  ```
  <BR>
  Now we need to setup L2TP/IPSec:
  VPN Server/one.nemanja.io> `IPsecEnable`
  IPsecEnable command - Enable or Disable IPsec VPN Server Function <BR>
  Enable L2TP over IPsec Server Function (yes / no): `yes` <BR>

  Enable Raw L2TP Server Function (yes / no): `yes` <BR>

  Enable EtherIP / L2TPv3 over IPsec Server Function (yes / no): `yes` <BR>

  Pre Shared Key for IPsec (Recommended: 9 letters at maximum): `secret`

  Default Virtual HUB in a case of omitting the HUB on the Username: `one.nemanja.io`

  The command completed successfully.

  That is it for the IPsec part, but we also want to have other protocols. For example OpenVPN and Microsoft protocols. Now we must use ServerCertRegenerate command to generate and register an SSL certificate for the server in order to be able to use it for the OpenVPN and Microsoft clients. The argument passed to the command must be your server IP address or FQDIN:
  ```
  ServerCertRegenerate <YOUR SERVER IP or FQDN>
  ```
  When new server certificate has been created, we need to save it to the file:
  ```
  ServerCertGet ~/cert.cer
  ```
  This certificate now can be distributed to your clients. We can now enable SSTP function with this command:
  ```
  SstpEnable yes
  ```
  And to enable OpenVPN:
  ```
  OpenVpnEnable yes /PORTS:1194
  ```
  Port for OpenVPN can be changed to your liking, but let's say that 1194 is standard. Then we need to create config for OpenVPN client like this
  ```
  OpenVpnMakeConfig ~/openvpn_config.zip
  ```

  **VPN over DNS and VPN over ICMP**

  Type Hub to return to administering entire VPN server and not just MOB hub.
  ```
  VPN Server/one.nemanja.io>Hub
  Hub command - Select Virtual Hub to Manage
  The Virtual Hub selection has been unselected.
  The command completed successfully.
  ```
  For maximal evasion of all blockages, we also need to enable VPN over ICMP and DNS:
  ```
  VpnOverIcmpDnsEnable /ICMP:yes /DNS:yes
  VpnOverIcmpDnsEnable command - Enable / Disable the VPN over ICMP / VPN over DNS Server Function
  The command completed successfully.
  ```
  Exit the vpncmd because we need to stop the vpnserver and setup dnsmasq. Dnsmasq is a Domain Name System (DNS) forwarder and Dynamic Host Configuration Protocol (DHCP) server for small computer networks, created as free software. Dnsmasq has low requirements for system resources, can run on Linux, BSDs, Android and OS X, and is included in most Linux distributions.

  OR

  We can use native CentOS 7 DHCP service and configure it. In case you don't have dhcp service installed, you can simply install it with:
  ```
  yum install dhcp
  ```

  Then:
  ```
  systemctl enable dhcpd &&
  systemctl start dhcpd
  ```

  Now we need to configure it:
  ```
  #
  # DHCP Server Configuration file.
  #   see /usr/share/doc/dhcp*/dhcpd.conf.example
  #   see dhcpd.conf(5) man page
  #
  option rfc3442-classless-static-routes code 121 = array of integer 8;
  option ms-classless-static-routes code 249 = array of integer 8;

  #host test  { hardware ethernet CA-80-D7; fixed-address 10.8.8.50; } # test line
  authoritative;

  subnet 10.8.8.0 netmask 255.255.255.0 {

          range 10.8.8.10 10.8.8.250;
          option subnet-mask 255.255.255.0;
          option broadcast-address 10.8.0.255;
          option rfc3442-classless-static-routes 24, 10, 8, 8, 10, 8, 8, 1;
          option ms-classless-static-routes 24, 10, 7, 7, 10, 7, 7, 1;

  }
  ```

  Short about info about this:
  1) range <-- Available IP address allocation for clients.
  2) option subnet-mask <-- default is 255.255.255.0, you can use anything which fits your network.
  3-4) option rfc3442-classless-static-routes and ms routes. The rfc3442-classless-static-routes option allows you to use classless network addresses http://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing (or CIDR) instead.

  CIDR requires a subnet mask to be explicitly stated, but the original DHCP option 33 doesn't have space for this. Therefore, this option (as defined in RFC 3442) simply enables a newer replacement DHCP option (option 121) which defines static routes using CIDR notation.

  Basically, if you need to issue static routes to your devices using DHCP and these static routes use CIDR then you need to enable this option.

  A static route can be used if you have split a network into to multiple smaller networks and need to inform each router about how traffic gets from one to another without using one of the many dynamic routing protocols available. You basically set up each router with a statement to the effect of "to get to network a.b.c.d, send traffic through f.g.h.i".

  If the route you set up in the router is classful, then you do not need to enable this option. However, if the routes are CIDR then you will need to enable this option. Fortunately, many home/cafe network use the 192.168.0.0 network with a subnet of 255.255.255.0 (or /24), which is a true Class-C network, therefore you can avoid this option.

  On the other hand, some home/cafe networks run on the 10.0.0.0 network. This is a Class-A network by default. If you are breaking this into many 10.0.x.0 subnets for example, then these will all be CIDR networks which mean you will need to enable this option.

  Here is a really useful article on pushing routes via dhcp service:
  http://thomasjaehnel.com/blog/2010/01/pushing-routes-via-dhcp.html

  Now let's continue further, and get back to the DNSMasq method:
  ```
  service vpnserver stop
  ```

  **DHCP server, forwarding and postrouting using DNSMasq**

  We need to edit /etc/dnsmasq.conf or use echo command to append needed lines to it. We will use latter option and while we are at it, we will also echo the ipv4_forwarding.conf
  ```
  echo interface=tap_soft >> /etc/dnsmasq.conf
  echo dhcp-range=tap_soft,192.168.7.50,192.168.7.90,12h >> /etc/dnsmasq.conf
  echo dhcp-option=tap_soft,3,192.168.7.1 >> /etc/dnsmasq.conf
  echo port=0 >> /etc/dnsmasq.conf
  echo dhcp-option=option:dns-server,8.8.8.8 >> /etc/dnsmasq.conf

  echo net.ipv4.ip_forward = 1 >> /etc/sysctl.d/ipv4_forwarding.conf
  ```
  Apply settings by running following:
  ```
  sysctl -n -e --system
  ```
  Check if it is applied:
  ```
  cat /proc/sys/net/ipv4/ip_forward
  ```
  It should show 1. If it shows 0, do this
  ```
  echo 1 > /proc/sys/net/ipv4/ip_forward
  ```
  Enable nat and postrouting:
  ```
  iptables -t nat -A POSTROUTING -s 10.8.8.0/24 -j SNAT --to-source [YOUR SERVER IP ADDRESS]

  iptables-save > /etc/sysconfig/iptables
  ```
  Restart vpn and dhcp servers with the folowing commands and enable them to start at every boot:

  ```
  service vpnserver start

  systemctl start dnsmasq

  systemctl enable dnsmasq

  chkconfig vpnserver on
  ```

  Now you can safely reboot your fresh VPN server and enjoy running it. If controlling your VPN server from the command line is too difficult, guys from Softether gone beyond and made beautiful software for Windows and MacOS. You can get the Softether VPN Manager software from the following address:
  http://www.softether-download.com/en.aspx?product=softether

  When you install software and load it, please click on "NEW SETTING" button. Next, fill up fields as shown bellow:

  ![alt](/content/images/2017/06/softethervpn-manager-1.png)

  After clicking on OK, a screen is going to look like this:
  ![alt](/content/images/2017/06/softethervpn-manager-2.png)

  From the next menu, you are able to control every aspect of your VPN server, as you do from the command line.
  ![alt](/content/images/2017/06/softethervpn-manager-3.png)

  I am going to describe closely each function on that screen:
  1) Manage Virtual Hub --> From there you can control each Hub.. add/remove/adjust users and etc.
  2) Online --> Power on Hub.
  3) Offline --> Shutdown Virtual Hub.
  4) View Status --> You can view status of Virtual Hub, like how many users are online, how many data is used (incoming and outgoing) and many other useful pieces of information.
  5) Create a Virtual Hub --> Boot up the new hub and configure the profile.
  6) Properties --> Change Hub properties (password, welcome text and other advanced options).
  7) Delete --> Remove Virtual Hub and all its settings.
  8) Management of listeners is probably going to be listed with 443,992,1194 and 5555 TCP ports. Here you can create the additional listening port if needed.
  9) Local Bridge Setting is reserved for the local bridge adapter which we have configured previously (tap_soft) which in our tutorial is listening on 10.8.8.1 IP address. You can from this menu create the additional bridge or redefine on how it is configured.
  10) Dynamic DNS settings is an amazing feature from SoftEther guys, which assigns a unique and permanent DNS hostname for this VPN server, for example:
  blablabla.softether.net
  11) Layer 3 Switch setting --> You can define Virtual Layer 3 switches between two or more virtual hubs operating on this VPN server to achieve routing between different IP Networks.
  12) VPN Azure settings --> NAT Traversal works with most of NATs and Firewalls, however, some restricted firewalls cannot pass NAT Traversal packets. In such a case do not give up. You can activate VPN Azure Relay Service on SoftEther VPN. After that, all VPN connecting requests from VPN Client or VPN Bridge will be relayed through the VPN Azure Cloud Servers operated by SoftEther VPN Project for free of charge. VPN Azure is the ultimate methods to penetrate any kinds of firewalls.
  The principle of VPN Azure is very simple. SoftEther VPN Server behind the firewall always keep a TCP-based connection toward a VPN Azure relay server. It is permitted automatically by the firewall because the direction of the initial connection is "from inside to outside". The firewall treats such packets as trusted. VPN Client or VPN Bridge requests the appropriate VPN Azure relay server to connect to the VPN Server. The VPN Azure relay server will relay the connection request to the SoftEther VPN Server. Then SoftEther VPN Server will establish the UDP-based VPN tunnel between the VPN Server and the VPN Client (or VPN Bridge). After the UDP-based link has been established the VPN relay server doesn't relay the payloads of VPN tunnel. More information here: https://www.softether.org/4-docs/2-howto/6.VPN_Server_Behind_NAT_or_Firewall/2.VPN_Azure
  13) IPSec/L2TP --> The IPsec VPN Server Function is disabled by default. Here is a useful doc page on how to configure IPSec/L2TP function properly:
  https://www.softether.org/4-docs/2-howto/9.L2TPIPsec_Setup_Guide_for_SoftEther_VPN_Server/1.Setup_L2TP%2F%2F%2F%2FIPsec_VPN_Server_on_SoftEther_VPN_Server
  14) OpenVPN --> Are you still using OpenVPN? SoftEther VPN has more ability, better performance and easy-configurable GUI-based management tools. SoftEther VPN has also the OpenVPN Server Clone Function so that any OpenVPN clients, including iPhone and Android, can connect to SoftEther VPN easily. Here is a useful documentation page on how to configure OpenVPN service properly:
  https://www.softether.org/4-docs/2-howto/7.Replacements_of_Legacy_VPNs/2.Replacements_of_OpenVPN

  More useful informaton and tutorials you can find on their amazing documentation page here:
  https://www.softether.org/4-docs/2-howto

  **Conclusion**

  That concludes installation and configuration of Softether VPN server. It is configured with Local Bridge for maximum performance, and now we only need to connect clients using different protocols. The Windows and Android ones are easy, for Windows you just head to the Softether site and download GUI client and connect (http://www.softether-download.com/en.aspx?product=softether). For Android, you don't need even that, you have VPN client built in, or you can download OpenVPN client from Google App store and configure it properly. On Linux OS (server) you can configure OpenVPN or any other known L2TP metho. Overall, from my tests so far, Softether has shown as decent free VPN solution software who by my personal opinion in several aspects is a way better solution than OpenVPN.

  If anyone needs help with installation and configuration of this software feel free to ping me at any time.

  Nemanja

html:
  <p>Today I will share how to create a VPN network and how to utilize it cost effectively or even for free. Recently, I've started to look at all of the possible solutions in this area and today I will mention two, which for me are ideal with regards to installation, maintenance, and optimal resource usage.</p>

  <p>What is a VPN? <br />
  A VPN or Virtual Private Network is a method used to add security and privacy to private and public networks, like WiFi Hotspots and the Internet. VPNs are more commonly used by corporations to protect sensitive data. VPNs can provide functionality, security and/or network management benefits to the user. But they can also lead to new issues, and some VPN services, especially "free" ones, can actually violate their users' privacy by logging their usage and making it available without their consent or make money by selling the user's bandwidth to other users.</p>

  <p>Some VPNs allow employees to securely access a corporate intranet while located outside the office. Some can securely connect geographically separated offices of an organisation, creating one cohesive network. Individual Internet users can use some VPNs to secure their wireless transactions, to circumvent geo-restrictions and censorship, and/or to connect to proxy servers for the purpose of protecting personal identities and locations. But some Internet sites block access via known VPNs to prevent the circumvention of their geo-restrictions.</p>

  <p>A VPN is created by establishing a virtual point-to-point connection through the use of dedicated connections, virtual tunnelling protocols, or traffic encryption. A VPN available from the public Internet can provide some of the benefits of a wide area network (WAN). From a user perspective, the resources available within the private network can be accessed remotely or kept private.</p>

  <p>The first software that I tested was OpenVPN which comes from OpenVPN Technologies, Inc. James Yonan. The initial release was pushed on 13th, May 2001. They have regular updates and it declares itself as an open-source software application, but not all parts are "free". I will write more about that later.</p>

  <p>Looking for other VPN solutions I've run into the Softether VPN server application coming from the University of Tsukuba in Japan. Softether has long been a proprietary product under name PacketX and it has been open sourced just several years ago. That may be the reason why it is Windows oriented. The configuration GUI is Windows only and connecting from Linux clients requires extra work, but it also has Linux console which can be combined. Today we are going to cover both.</p>

  <p>We are testing this on plain CentOS 7 server, in my case KVM QEMU instance, but can be booted on anything really.</p>

  <p>First let's prepare system: <br />
  1) <code>yum update</code> <br />
  2) <code>yum -y groupinstall "Development Tools"</code> <br />
  3)  </p>

  <pre><code>yum -y install gcc zlib-devel openssl-devel readline-devel ncurses-devel wget tar dnsmasq net-tools iptables-services system-config-firewall-tui nano iptables-services  
  </code></pre>

  <p>Disable SELinux:  </p>

  <pre><code>sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config  
  </code></pre>

  <p>Now we can start with server installation: <br />
  4) cd /usr/local <br />
  5)  </p>

  <pre><code>wget -c http://www.softether-download.com/files/softether/v4.22-9634-beta-2016.11.27-tree/Linux/SoftEther_VPN_Server/64bit_-_Intel_x64_or_AMD64/softether-vpnserver-v4.22-9634-beta-2016.11.27-linux-x64-64bit.tar.gz  
  </code></pre>

  <p>(you can visit <a href="https://www.softether.org/5-download">https://www.softether.org/5-download</a> website to check for the latest version)</p>

  <p>6) extract archive within /usr/local/ folder: <br />
  <code>tar -xvf softether-vpnserver-v4.22-9634-beta-2016.11.27-linux-x64-64bit.tar.gz</code>
  <img src="/content/images/2017/06/softethervpn-1.png" alt="alt" /></p>

  <p>7) cd /usr/local/vpnserver ; make <br />
  <img src="/content/images/2017/06/softethervpn-2.png" alt="alt" /></p>

  <p>Select 1 and accept License Agreement.</p>

  <p>8) The developers of this software were really detailed and descriptive, so they have added everything at end of installation. If you get this at the end of your installation, it means that you have managed successfully to install SoftEther VPN application on your CentOS 7 server.</p>

  <hr />

  <p>The preparation of SoftEther VPN Server is completed!</p>

  <p><strong>* How to switch the display language of the SoftEther VPN Server Service *</strong>
  SoftEther VPN Server supports the following languages: <br />
    - Japanese
    - English
    - Simplified Chinese</p>

  <p>You can choose your preferred language of SoftEther VPN Server at any time.</p>

  <p>To switch the current language, open and edit the 'lang.config' file.</p>

  <p><strong>* How to start the SoftEther VPN Server Service *</strong></p>

  <p>Please execute './vpnserver start' to run the SoftEther VPN Server Background Service. <br />
  And please execute './vpncmd' to run the SoftEther VPN Command-Line Utility to configure SoftEther VPN Server.</p>

  <p>Of course, you can use the VPN Server Manager GUI Application for Windows / Mac OS X on the other Windows / Mac OS X computers in order to configure the SoftEther VPN Server remotely.</p>

  <p><strong>* For Windows users *</strong>
  You can download the SoftEther VPN Server Manager for Windows <br />
  from the <a href="http://www.softether-download.com/">http://www.softether-download.com/</a> web site. <br />
  This manager application helps you to completely and easily manage the VPN server services running in remote hosts.</p>

  <p><strong>* For Mac OS X users *</strong>
  In April 2016 we released the SoftEther VPN Server Manager for Mac OS X. <br />
  You can download it from the <a href="http://www.softether-download.com/">http://www.softether-download.com/</a> web site. <br />
  VPN Server Manager for Mac OS X works perfectly as same as the traditional Windows versions. It helps you to completely and easily manage the VPN server services running in remote hosts.</p>

  <hr />

  <p>make[1]: Leaving directory `/usr/local/vpnserver'</p>

  <p>9) Next we need to make the init script for softether application because developers have not included into the install. Run nano /etc/init.d/vpnserver and make paste this script.</p>

  <p><pre><code>wget -c http://www.softether-download.com/files/softether/v4.22-9634-beta-2016.11.27-tree/Linux/SoftEther_VPN_Server/64bit_-_Intel_x64_or_AMD64/softether-vpnserver-v4.22-9634-beta-2016.11.27-linux-x64-64bit.tar.gz  
  </code></pre></p>

  <p>7) Next step we need to make shell script that we just created before with executable rules, so it can be started properly during the boot process and manually. <br />
  <code>tar -xvf softether-vpnserver-v4.22-9634-beta-2016.11.27-linux-x64-64bit.tar.gz</code></p>

  <p><img src="/content/images/2017/06/softethervpn-3.png" alt="alt" /></p>

  <p>Note: Please ignore alert about tap<em>soft interface. It is added into the vpnserver bash script, and we will configure it later during this tutorial. <br />
  <strong>In this tutorial we are going to use 10.8.8.0/24 as example range, and that is why TAP</em>ADDR=10.8.8.1 as server address. You can use anything you wish.</strong></p>

  <p>Moving over to the configuration part, we need to start vpncmd utility (Linux method), later I will add the easier way by installing SE VPN Server Manager Tool. Software is made in that way that everything can be configured from shell and GUI mode leaving to the end-user preferred method.</p>

  <p><code>/usr/local/vpnserver/vpncmd</code></p>

  <p>Press 1 to select "Management of VPN Server or VPN Bridge", and then when asked which server to configure, press enter and it will choose <code>localhost</code> where you just installed Softether application. Press Enter one more time to get access to the server as Administrator. </p>

  <p>Next type: <br />
  <code>/usr/local/vpnserver/vpncmd</code>
  to set the admin password for the server. In order to use softether, virtual hub needs to be created. We will create one named one.nemanja.io with the following command: <br />
  <code>localhost</code>
  You will be prompted to set the password, which you will use to administer a hub, without access to entire VPN server. Practically one VPN server can have unlimited Hubs connecting multiple customers, and that is why this software is so unique.</p>

  <p>Now we need to create a local bridge. That is more efficient way, comparing to other. There is also built-in SecureNAT which is easier to setup, but sadly it is resource intensive. We will go with setting up local bridge and tap device, note that with local bridge also DHCP server needs to be configured and installed which we are going to do bellow. Local bridge is created with the following command: <br />
  <code>UserCreate</code>
  If TAP device creation fails with the message about insufficient privileges, you might want to check if your network controller is set in promiscuous mode. If you are using some popular Virtualization software like SolusVM/OnApp, make sure that Tun/Tap option is enabled for the VPS. I haven't had problems with Proxmox or with using Xen, so I believe this is enabled by default.</p>

  <p>Now we need to create a user for the one.nemanja.io virtual hub that we have previously created. Users are created with command <code>UserCreate</code> and you can view the list of users by command <code>UserList</code>. Users can be added to groups and each group can have different authentication mode/privileges, for example, Password, Certificate, RADIUS, NTLM and etc etc.</p>

  <p><strong>Configuring the Virtual Hub</strong></p>

  <p>Now we need to switch to hub one.nemanja.io <br />
  <code>Hub one.nemanja.io</code></p>

  <p>and create user <br />
  <code>UserCreate test</code>
  <BR> <br />
  We will keep it simple and use password <code>auth</code>, so use the following command <br />
  <code>UserList</code>
  <BR> <br />
  Now we need to setup L2TP/IPSec: <br />
  VPN Server/one.nemanja.io> <code>IPsecEnable</code> <br />
  IPsecEnable command - Enable or Disable IPsec VPN Server Function <BR> <br />
  Enable L2TP over IPsec Server Function (yes / no): <code>yes</code> <BR></p>

  <p>Enable Raw L2TP Server Function (yes / no): <code>yes</code> <BR></p>

  <p>Enable EtherIP / L2TPv3 over IPsec Server Function (yes / no): <code>yes</code> <BR></p>

  <p>Pre Shared Key for IPsec (Recommended: 9 letters at maximum): <code>secret</code></p>

  <p>Default Virtual HUB in a case of omitting the HUB on the Username: <code>one.nemanja.io</code></p>

  <p>The command completed successfully.</p>

  <p>That is it for the IPsec part, but we also want to have other protocols. For example OpenVPN and Microsoft protocols. Now we must use ServerCertRegenerate command to generate and register an SSL certificate for the server in order to be able to use it for the OpenVPN and Microsoft clients. The argument passed to the command must be your server IP address or FQDIN:  </p>

  <pre><code>ServerCertRegenerate &lt;YOUR SERVER IP or FQDN&gt;  
  </code></pre>

  <p>When new server certificate has been created, we need to save it to the file:  </p>

  <pre><code>ServerCertGet ~/cert.cer  
  </code></pre>

  <p>This certificate now can be distributed to your clients. We can now enable SSTP function with this command:  </p>

  <pre><code>SstpEnable yes  
  </code></pre>

  <p>And to enable OpenVPN:  </p>

  <pre><code>OpenVpnEnable yes /PORTS:1194  
  </code></pre>

  <p>Port for OpenVPN can be changed to your liking, but let's say that 1194 is standard. Then we need to create config for OpenVPN client like this  </p>

  <pre><code>OpenVpnMakeConfig ~/openvpn_config.zip  
  </code></pre>

  <p><strong>VPN over DNS and VPN over ICMP</strong></p>

  <p>Type Hub to return to administering entire VPN server and not just MOB hub.  </p>

  <pre><code>VPN Server/one.nemanja.io&gt;Hub  
  Hub command - Select Virtual Hub to Manage  
  The Virtual Hub selection has been unselected.  
  The command completed successfully.  
  </code></pre>

  <p>For maximal evasion of all blockages, we also need to enable VPN over ICMP and DNS:  </p>

  <pre><code>VpnOverIcmpDnsEnable /ICMP:yes /DNS:yes  
  VpnOverIcmpDnsEnable command - Enable / Disable the VPN over ICMP / VPN over DNS Server Function  
  The command completed successfully.  
  </code></pre>

  <p>Exit the vpncmd because we need to stop the vpnserver and setup dnsmasq. Dnsmasq is a Domain Name System (DNS) forwarder and Dynamic Host Configuration Protocol (DHCP) server for small computer networks, created as free software. Dnsmasq has low requirements for system resources, can run on Linux, BSDs, Android and OS X, and is included in most Linux distributions.</p>

  <p>OR</p>

  <p>We can use native CentOS 7 DHCP service and configure it. In case you don't have dhcp service installed, you can simply install it with:  </p>

  <pre><code>yum install dhcp  
  </code></pre>

  <p>Then:  </p>

  <pre><code>systemctl enable dhcpd &amp;&amp;  
  systemctl start dhcpd  
  </code></pre>

  <p>Now we need to configure it:  </p>

  <pre><code>#
  # DHCP Server Configuration file.
  #   see /usr/share/doc/dhcp*/dhcpd.conf.example
  #   see dhcpd.conf(5) man page
  #
  option rfc3442-classless-static-routes code 121 = array of integer 8;  
  option ms-classless-static-routes code 249 = array of integer 8;

  #host test  { hardware ethernet CA-80-D7; fixed-address 10.8.8.50; } # test line
  authoritative;

  subnet 10.8.8.0 netmask 255.255.255.0 {

          range 10.8.8.10 10.8.8.250;
          option subnet-mask 255.255.255.0;
          option broadcast-address 10.8.0.255;
          option rfc3442-classless-static-routes 24, 10, 8, 8, 10, 8, 8, 1;
          option ms-classless-static-routes 24, 10, 7, 7, 10, 7, 7, 1;

  }
  </code></pre>

  <p>Short about info about this: <br />
  1) range &lt;-- Available IP address allocation for clients. <br />
  2) option subnet-mask &lt;-- default is 255.255.255.0, you can use anything which fits your network. <br />
  3-4) option rfc3442-classless-static-routes and ms routes. The rfc3442-classless-static-routes option allows you to use classless network addresses <a href="http://en.wikipedia.org/wiki/Classless">http://en.wikipedia.org/wiki/Classless</a><em>Inter-Domain</em>Routing (or CIDR) instead.</p>

  <p>CIDR requires a subnet mask to be explicitly stated, but the original DHCP option 33 doesn't have space for this. Therefore, this option (as defined in RFC 3442) simply enables a newer replacement DHCP option (option 121) which defines static routes using CIDR notation.</p>

  <p>Basically, if you need to issue static routes to your devices using DHCP and these static routes use CIDR then you need to enable this option.</p>

  <p>A static route can be used if you have split a network into to multiple smaller networks and need to inform each router about how traffic gets from one to another without using one of the many dynamic routing protocols available. You basically set up each router with a statement to the effect of "to get to network a.b.c.d, send traffic through f.g.h.i".</p>

  <p>If the route you set up in the router is classful, then you do not need to enable this option. However, if the routes are CIDR then you will need to enable this option. Fortunately, many home/cafe network use the 192.168.0.0 network with a subnet of 255.255.255.0 (or /24), which is a true Class-C network, therefore you can avoid this option.</p>

  <p>On the other hand, some home/cafe networks run on the 10.0.0.0 network. This is a Class-A network by default. If you are breaking this into many 10.0.x.0 subnets for example, then these will all be CIDR networks which mean you will need to enable this option.</p>

  <p>Here is a really useful article on pushing routes via dhcp service: <br />
  <a href="http://thomasjaehnel.com/blog/2010/01/pushing-routes-via-dhcp.html">http://thomasjaehnel.com/blog/2010/01/pushing-routes-via-dhcp.html</a></p>

  <p>Now let's continue further, and get back to the DNSMasq method:  </p>

  <pre><code>service vpnserver stop  
  </code></pre>

  <p><strong>DHCP server, forwarding and postrouting using DNSMasq</strong></p>

  <p>We need to edit /etc/dnsmasq.conf or use echo command to append needed lines to it. We will use latter option and while we are at it, we will also echo the ipv4_forwarding.conf  </p>

  <pre><code>echo interface=tap_soft &gt;&gt; /etc/dnsmasq.conf  
  echo dhcp-range=tap_soft,192.168.7.50,192.168.7.90,12h &gt;&gt; /etc/dnsmasq.conf  
  echo dhcp-option=tap_soft,3,192.168.7.1 &gt;&gt; /etc/dnsmasq.conf  
  echo port=0 &gt;&gt; /etc/dnsmasq.conf  
  echo dhcp-option=option:dns-server,8.8.8.8 &gt;&gt; /etc/dnsmasq.conf

  echo net.ipv4.ip_forward = 1 &gt;&gt; /etc/sysctl.d/ipv4_forwarding.conf  
  </code></pre>

  <p>Apply settings by running following:  </p>

  <pre><code>sysctl -n -e --system  
  </code></pre>

  <p>Check if it is applied:  </p>

  <pre><code>cat /proc/sys/net/ipv4/ip_forward  
  </code></pre>

  <p>It should show 1. If it shows 0, do this  </p>

  <pre><code>echo 1 &gt; /proc/sys/net/ipv4/ip_forward  
  </code></pre>

  <p>Enable nat and postrouting:  </p>

  <pre><code>iptables -t nat -A POSTROUTING -s 10.8.8.0/24 -j SNAT --to-source [YOUR SERVER IP ADDRESS]

  iptables-save &gt; /etc/sysconfig/iptables  
  </code></pre>

  <p>Restart vpn and dhcp servers with the folowing commands and enable them to start at every boot:</p>

  <pre><code>service vpnserver start

  systemctl start dnsmasq

  systemctl enable dnsmasq

  chkconfig vpnserver on  
  </code></pre>

  <p>Now you can safely reboot your fresh VPN server and enjoy running it. If controlling your VPN server from the command line is too difficult, guys from Softether gone beyond and made beautiful software for Windows and MacOS. You can get the Softether VPN Manager software from the following address: <br />
  <a href="http://www.softether-download.com/en.aspx?product=softether">http://www.softether-download.com/en.aspx?product=softether</a></p>

  <p>When you install software and load it, please click on "NEW SETTING" button. Next, fill up fields as shown bellow:</p>

  <p><img src="/content/images/2017/06/softethervpn-manager-1.png" alt="alt" /></p>

  <p>After clicking on OK, a screen is going to look like this: <br />
  <img src="/content/images/2017/06/softethervpn-manager-2.png" alt="alt" /></p>

  <p>From the next menu, you are able to control every aspect of your VPN server, as you do from the command line. <br />
  <img src="/content/images/2017/06/softethervpn-manager-3.png" alt="alt" /></p>

  <p>I am going to describe closely each function on that screen: <br />
  1) Manage Virtual Hub --> From there you can control each Hub.. add/remove/adjust users and etc. <br />
  2) Online --> Power on Hub. <br />
  3) Offline --> Shutdown Virtual Hub. <br />
  4) View Status --> You can view status of Virtual Hub, like how many users are online, how many data is used (incoming and outgoing) and many other useful pieces of information. <br />
  5) Create a Virtual Hub --> Boot up the new hub and configure the profile. <br />
  6) Properties --> Change Hub properties (password, welcome text and other advanced options). <br />
  7) Delete --> Remove Virtual Hub and all its settings. <br />
  8) Management of listeners is probably going to be listed with 443,992,1194 and 5555 TCP ports. Here you can create the additional listening port if needed. <br />
  9) Local Bridge Setting is reserved for the local bridge adapter which we have configured previously (tap<em>soft) which in our tutorial is listening on 10.8.8.1 IP address. You can from this menu create the additional bridge or redefine on how it is configured. <br />
  10) Dynamic DNS settings is an amazing feature from SoftEther guys, which assigns a unique and permanent DNS hostname for this VPN server, for example: <br />
  blablabla.softether.net <br />
  11) Layer 3 Switch setting --> You can define Virtual Layer 3 switches between two or more virtual hubs operating on this VPN server to achieve routing between different IP Networks. <br />
  12) VPN Azure settings --> NAT Traversal works with most of NATs and Firewalls, however, some restricted firewalls cannot pass NAT Traversal packets. In such a case do not give up. You can activate VPN Azure Relay Service on SoftEther VPN. After that, all VPN connecting requests from VPN Client or VPN Bridge will be relayed through the VPN Azure Cloud Servers operated by SoftEther VPN Project for free of charge. VPN Azure is the ultimate methods to penetrate any kinds of firewalls. <br />
  The principle of VPN Azure is very simple. SoftEther VPN Server behind the firewall always keep a TCP-based connection toward a VPN Azure relay server. It is permitted automatically by the firewall because the direction of the initial connection is "from inside to outside". The firewall treats such packets as trusted. VPN Client or VPN Bridge requests the appropriate VPN Azure relay server to connect to the VPN Server. The VPN Azure relay server will relay the connection request to the SoftEther VPN Server. Then SoftEther VPN Server will establish the UDP-based VPN tunnel between the VPN Server and the VPN Client (or VPN Bridge). After the UDP-based link has been established the VPN relay server doesn't relay the payloads of VPN tunnel. More information here: <a href="https://www.softether.org/4-docs/2-howto/6.VPN">https://www.softether.org/4-docs/2-howto/6.VPN</a></em>Server<em>Behind</em>NAT<em>or</em>Firewall/2.VPN<em>Azure <br />
  13) IPSec/L2TP --> The IPsec VPN Server Function is disabled by default. Here is a useful doc page on how to configure IPSec/L2TP function properly: <br />
  <a href="https://www.softether.org/4-docs/2-howto/9.L2TPIPsec">https://www.softether.org/4-docs/2-howto/9.L2TPIPsec</a></em>Setup<em>Guide</em>for<em>SoftEther</em>VPN<em>Server/1.Setup</em>L2TP%2F%2F%2F%2FIPsec<em>VPN</em>Server<em>on</em>SoftEther<em>VPN</em>Server <br />
  14) OpenVPN --> Are you still using OpenVPN? SoftEther VPN has more ability, better performance and easy-configurable GUI-based management tools. SoftEther VPN has also the OpenVPN Server Clone Function so that any OpenVPN clients, including iPhone and Android, can connect to SoftEther VPN easily. Here is a useful documentation page on how to configure OpenVPN service properly: <br />
  <a href="https://www.softether.org/4-docs/2-howto/7.Replacements">https://www.softether.org/4-docs/2-howto/7.Replacements</a><em>of</em>Legacy<em>VPNs/2.Replacements</em>of_OpenVPN</p>

  <p>More useful informaton and tutorials you can find on their amazing documentation page here: <br />
  <a href="https://www.softether.org/4-docs/2-howto">https://www.softether.org/4-docs/2-howto</a></p>

  <p><strong>Conclusion</strong></p>

  <p>That concludes installation and configuration of Softether VPN server. It is configured with Local Bridge for maximum performance, and now we only need to connect clients using different protocols. The Windows and Android ones are easy, for Windows you just head to the Softether site and download GUI client and connect (<a href="http://www.softether-download.com/en.aspx?product=softether">http://www.softether-download.com/en.aspx?product=softether</a>). For Android, you don't need even that, you have VPN client built in, or you can download OpenVPN client from Google App store and configure it properly. On Linux OS (server) you can configure OpenVPN or any other known L2TP metho. Overall, from my tests so far, Softether has shown as decent free VPN solution software who by my personal opinion in several aspects is a way better solution than OpenVPN.</p>

  <p>If anyone needs help with installation and configuration of this software feel free to ping me at any time.</p>

  <p>Nemanja</p>


uuid: 05d7036c-c2fb-4a92-bb22-7adbd4634f79
title: Tools
slug: tools
image: /content/images/2016/09/tools.png
status: published
language: en_US
markdown:
  Nunus
  https://github.com/nemke82/nunus
  ** Effective Stress test and Benchmark tool with Chart. Nunus is a lightweight stress test and benchmark tool designed for anyone who wishes to test (examine) capacity of their Website.

  NagClient
  https://github.com/nemke82/nagclient
  ** Multi platform Nagios client Installer. NagClient is tool designed for anyone who wishes to easily deploy Nagios application, NRPE Daemon and some recommended plugins on different Linux platforms and Cloud/VZ systems.

html:
  <p>Nunus <br />
  <a href="https://github.com/nemke82/nunus">https://github.com/nemke82/nunus</a> <br />
  ** Effective Stress test and Benchmark tool with Chart. Nunus is a lightweight stress test and benchmark tool designed for anyone who wishes to test (examine) capacity of their Website.</p>

  <p>NagClient <br />
  <a href="https://github.com/nemke82/nagclient">https://github.com/nemke82/nagclient</a> <br />
  ** Multi platform Nagios client Installer. NagClient is tool designed for anyone who wishes to easily deploy Nagios application, NRPE Daemon and some recommended plugins on different Linux platforms and Cloud/VZ systems.</p>


uuid: 6eb9ed44-52c3-4008-979e-92718dedd332
title: CentOS 7 issue with update or cPanel/WHM install
slug: centos-7-update-cpanel-install-issue
status: published
language: en_US
markdown:
  Patterns reported:
  CentOS 7 error: unpacking of archive failed on file /usr/lib/udev/rules.d/60-net.rules: cpio: rename

  OR

  cPanel/WHM installations (CentOS 7 SolusVM/OnApp template):
  2016-11-25 12:12:34  240 (DEBUG):       Updating   : initscripts-9.49.30-1.el7_2.3.x86_64                         1/2Error unpacking rpm package initscripts-9.49.30-1.el7_2.3.x86_64
  2016-11-25 12:12:34  240 (DEBUG):      
  2016-11-25 12:12:34  240 (DEBUG):     error: unpacking of archive failed on file /usr/lib/udev/rules.d/60-net.rules: cpio: rename
  2016-11-25 12:12:34  240 (DEBUG):       Verifying  : initscripts-9.49.30-1.el7_2.3.x86_64                         1/2initscripts-9.49.24-1.el7.x86_64 was supposed to be removed but is not!
  2016-11-25 12:12:34  240 (DEBUG):      
  2016-11-25 12:12:34  240 (DEBUG):       Verifying  : initscripts-9.49.24-1.el7.x86_64                             2/2 
  2016-11-25 12:12:34  240 (DEBUG):     
  2016-11-25 12:12:34  240 (DEBUG):     Failed:
  2016-11-25 12:12:34  240 (DEBUG):       initscripts.x86_64 0:9.49.24-1.el7   initscripts.x86_64 0:9.49.30-1.el7_2.3  
  2016-11-25 12:12:34  240 (DEBUG):     
  2016-11-25 12:12:34  240 (DEBUG):     Complete!
  2016-11-25 12:12:34  240 (ERROR):   - ssystem [EXIT_CODE] '/usr/bin/yum' exited with 1 (ignored)
  2016-11-25 12:12:34  240 (DEBUG):   - ssystem [END]
  2016-11-25 12:12:34  244 (FATAL): yum failed 6 times. The installation process cannot continue.
  Removing /root/installer.lock.

  *Solution:*

  Luckily, the solution is quite simple, we just need to remove the protection attribute using chattr. Run the following as root to remove the attribute on 60-net.rules preventing us from updating initscripts:

  chattr -i /usr/lib/udev/rules.d/60-net.rules

  That's it, try running yum update once more and the program will be updated without issue now.

  Again, if you are irritated with fact that cPanel/WHM decided to disable EA3 in new installs, the solution is before starting with 'sh latest' create the /etc/install_legacy_ea3_instead_of_ea4 file, and repeat 'sh latest' again.

html:
  <p>Patterns reported: <br />
  CentOS 7 error: unpacking of archive failed on file /usr/lib/udev/rules.d/60-net.rules: cpio: rename</p>

  <p>OR</p>

  <p>cPanel/WHM installations (CentOS 7 SolusVM/OnApp template): <br />
  2016-11-25 12:12:34  240 (DEBUG):       Updating   : initscripts-9.49.30-1.el7<em>2.3.x86</em>64                         1/2Error unpacking rpm package initscripts-9.49.30-1.el7<em>2.3.x86</em>64 <br />
  2016-11-25 12:12:34  240 (DEBUG): <br />
  2016-11-25 12:12:34  240 (DEBUG):     error: unpacking of archive failed on file /usr/lib/udev/rules.d/60-net.rules: cpio: rename <br />
  2016-11-25 12:12:34  240 (DEBUG):       Verifying  : initscripts-9.49.30-1.el7<em>2.3.x86</em>64                         1/2initscripts-9.49.24-1.el7.x86<em>64 was supposed to be removed but is not! <br />
  2016-11-25 12:12:34  240 (DEBUG): <br />
  2016-11-25 12:12:34  240 (DEBUG):       Verifying  : initscripts-9.49.24-1.el7.x86</em>64                             2/2 <br />
  2016-11-25 12:12:34  240 (DEBUG): <br />
  2016-11-25 12:12:34  240 (DEBUG):     Failed: <br />
  2016-11-25 12:12:34  240 (DEBUG):       initscripts.x86<em>64 0:9.49.24-1.el7   initscripts.x86</em>64 0:9.49.30-1.el7<em>2.3 <br />
  2016-11-25 12:12:34  240 (DEBUG): <br />
  2016-11-25 12:12:34  240 (DEBUG):     Complete! <br />
  2016-11-25 12:12:34  240 (ERROR):   - ssystem [EXIT</em>CODE] '/usr/bin/yum' exited with 1 (ignored) <br />
  2016-11-25 12:12:34  240 (DEBUG):   - ssystem [END] <br />
  2016-11-25 12:12:34  244 (FATAL): yum failed 6 times. The installation process cannot continue. <br />
  Removing /root/installer.lock.</p>

  <p><em>Solution:</em></p>

  <p>Luckily, the solution is quite simple, we just need to remove the protection attribute using chattr. Run the following as root to remove the attribute on 60-net.rules preventing us from updating initscripts:</p>

  <p>chattr -i /usr/lib/udev/rules.d/60-net.rules</p>

  <p>That's it, try running yum update once more and the program will be updated without issue now.</p>

  <p>Again, if you are irritated with fact that cPanel/WHM decided to disable EA3 in new installs, the solution is before starting with 'sh latest' create the /etc/install<em>legacy</em>ea3<em>instead</em>of_ea4 file, and repeat 'sh latest' again.</p>


uuid: 5fed7d97-3d17-4661-84a9-3f7a32f42f00
title: Your own Conference software -- Apache Open Meeting
slug: your-own-conference-software-apache-open-meeting
image: /content/images/2017/01/conference-room.jpg
status: published
language: en_US
markdown:
  Recently I have been testing some software for Presenting, Learning / Web Conferencing, and data sharing, but frankly none of them displayed good results. There are a few paid solutions, but my Open Source nature led me to test the Apache Open Meeting app. OpenMeetings is a software used for presenting, online training, web conferencing, collaborative whiteboard drawing, document editing, and user desktop sharing. The product is based on the OpenLaszlo RIA framework and Red5 media server, which in turn are based on some open source components. Communication takes place in virtual "meeting rooms" which may be set to different communication, security, and video quality modes. The recommended database engine for backend support is MySQL. The product can be set up as an installed server product, or used as a hosted service.
  Work on OpenMeetings began in 2006, and it has been downloaded over 250,000 times. OpenMeetings is available in 31 languages.
  Official site: http://openmeetings.apache.org/
  Working on all major operating systems, but today I will write the tutorial based on the latest 3.1.4 version on CentOS version 6 and 7 Linux (dedicated server / compatible with Xen/KVM if virtual).

  **1)** First we must modify Selinux level security for the installation: <BR>
  `yum install -y nano` <BR>
  `sudo nano /etc/selinux/config` <BR>

  …modify from: <BR>
  `SELINUX=enforcing` <BR>
  ...to
  `SELINUX=permissive`
  or if SELINUX not needed `SELINUX=disabled` <BR>

  **2)** Now we must update the operating system. Note: this is the recommended action, not required <BR>
  `yum update -y`
  <BR>
  For kernel changes and the new Selinux configuration to take effect, push:
  `reboot`
  <BR>

  **3)** Next will be to add Repositories. (you may get files to /root or /opt it does not matter)<BR>
  `yum install -y wget`

  == EPEL & Remi: == <BR>
  ```
  wget -c http://epel.mirror.nucleus.be/7/x86_64/e/epel-release-7-5.noarch.rpm
  ```
  ```
  wget http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
  ```
  `sudo rpm -Uvh remi-release-7*.rpm epel-release-7*.rpm` <BR>

  Enable Remi: <BR>
  `nano /etc/yum.repos.d/remi.repo`<BR>

  And modify (the first enabled):
  From enabled=0 to enabled=1 <BR>

  == ElRepo == <BR>
  ```
  rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
  ```
  ```
  rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
  ```

  == Nux == <BR>
  ```
  rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-
  5.el7.nux.noarch.rpm
  ```
  (copy as one line)
  <BR>

  Lastly, in this step we need to install **Adobe repo 64-bit x86_64** For Flash player. Push: <BR>

  ``` 
  rpm -ivh http://linuxdownload.adobe.com/adobe- release/adobe-release-x86_64-1.0-1.noarch.rpm 
  ``` 
  ```
  rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux
  ```
  `yum update -y`

  YYou can always confirm if the package was successfully installed using rpm -qa, for example:<BR>
  `rpm -qa | grep adobe`

  **4)** Installation of Oracle Java 1.8. Java 1.8 is necessary for OpenMeetings 3.1.4. Java 1.7 can be used only for the previous OM versions. We shall install Java 1.8 now. <BR>

  cd /opt<BR>

  Download the file:
  (All in one line only. 1ª and 2ª without space between them. A space to the 3ª)<BR>
  ```
  wget --no-cookies --no-check-certificate --header "Cookie:gpw_e24=http%3A%2F%2Fwww.oracle.com%2F;oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u101-b13/jdk-8u101-linux-x64.rpm"
  ```

  And then install it: 
  `rpm -ivh jdk-8u101-linux-x64.rpm`

  Maybe you have installed various versions of Java. We select the just installed Oracle Java:
  `update-alternatives --config java` <BR>

  And to see if the selected version is active:
  `java -version`

  Alternatively, you can simply run: <BR>
  `sudo yum install java-1.8.0-openjdk icedtea-web` <BR>

  And everything should work as expected, but it is recommended that you choose the latest 1.8 version.

  **5**) OpenMeetings will need LibreOffice to convert pdf, which can be uploaded through Dashboard. To install LibreOffice: <BR>
  `yum -y install libreoffice libreoffice-headless` <BR>

  **6**) Installation of necessary packages and libraries: <BR>
  ```
  yum install -y libjpeg libjpeg-devel ghostscript freetype freetype-devel unzip gcc gcc-c++ ncurses
  ncurses-devel make zlib zlib-devel libtool bison bison-devel openssl-devel bzip2 bzip2-devel fileroller git autoconf automake pkgconfig tomcat-native nmap vlc
  ```
  (Copy as one line with space)

  **7**) Now we are going to need ImageMagick, Sox and Swftools. ImageMagick, works with image filetypes jpg, png, gif, etc. Lets get that installed: <BR>
  `yum install -y ImageMagick giflib giflib-devel giflib-utils` <BR>

  Next is Sox, which deals with Sound. We will need to compile it with the latest version:<BR>
  `cd /opt`<BR>
  ```
  wget -c http://sourceforge.net/projects/sox/files/sox/14.4.2/sox-14.4.2.tar.gz
  ```
  `tar xzvf sox-14.4.2.tar.gz`<BR>
  `cd /opt/sox-14.4.2`<BR>
  `./configure`<BR>
  `make && make install`<BR>

  Swftools. LibreOffice converts the uploaded office files to pdf, and Swftools converts these pdfs to
  swf (flash file), which can later be presented on the whiteboard. Also convert jpg2swf, png2swf, gif2swf, and etc.
  Don´t compile the newer version, since they don't have pdf2swf, which is necessary. <BR>

  `cd /opt`<BR>
  ```
  wget -c http://www.swftools.org/swftools-2013-04-09-1007.tar.gz
  ```
  `tar xzvf swftools-2013-04-09-1007.tar.gz` <BR>
  `cd /opt/swftools-2013-04-09-1007` <BR>
  `./configure --libdir=/usr/lib --bindir=/usr/bin` <BR>
  `make && make install && cd /opt` <BR>

  **8**) OpenMeetings application requires Adobe Flash Player for rooms.
  `yum install -y flash-plugin` <BR>

  **9**) Jodconverter participates in the process to convert uploaded files. The Java OpenDocument Converter, converts documents between different office formats. It leverages OpenOffice.org, which arguably provides the best import/export filters for OpenDocument and Microsoft Office formats available today. Word to OpenDocument Text (odt); OpenDocument Text (odt) to Word and etc. <BR>
  `cd /opt`<BR>
  ```
  wget -c https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/jodconverter/jodconverter-core-3.0-beta-4-dist.zip
  ```
  `unzip jodconverter-core-3.0-beta-4-dist.zip`<BR>

  For, now we will leave it, and remember the path where was extracted.

  **10**) Next, we will need to install FFmpeg. FFmpeg is a free software project that produces libraries and programs for handling multimedia data. FFmpeg works with video. We will now install the necessary packages, libraries and vlc player which is needed to play the meeting recordings. <BR>
  ```
  yum install -y glibc alsa-lib-devel faac faac-devel faad2 faad2-devel gsm gsm-devel imlib2 imlib2-devel lame-devel vorbis-tools theora-tools libvpx-devel vlc autoconf automake cmake freetypedevel gcc gcc-c++ git libtool make mercurial nasm pkgconfig zlib-devel curl
  ```
  (copy as one line and execute on server)

  This ffmpeg compilation is based on this url, updated file versions 28-9-2016: <BR>
  https://trac.ffmpeg.org/wiki/CompilationGuide/Centos
  ```
  wget https://cwiki.apache.org/confluence/download/attachments/27838216/ffmpeg-centos2.sh
  ```
  `chmod a+rwx ffmpeg-centos2.sh` <BR>

  The installation can take up to 30 minutes to finish:
  `./ffmpeg-centos2.sh` <BR>
  When finished, please, go to step 11).

  **11**) Installation MariaDB data server. MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. It is notable for being led by the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle. You may use the default Oracle version or Percona, but that is up to you. <BR>

  `yum install -y mariadb-server` <BR>

  Now we are going to start MariaDB server (CentOS7):<BR>
  `systemctl start mariadb.service` <BR>
  `service mariadb start` <-- For CentOS 6<BR>

  Now we need to set password for Root. Please, modify new-password by your own. <BR>
  `mysqladmin -u root password new-password` <BR>

  Note: You may edit /root/.my.cnf and configure password less access for MySQL, or use -u root -p command. <BR>

  Make a database for OpenMeetings:
  `mysql -u root -p` <BR>

  MariaDB [(none)]> `CREATE DATABASE open314 DEFAULT CHARACTER SET 'utf8';` <BR>

  Now we create a user with all permission on this open314 database.
  MariaDB [(none)]> `GRANT ALL PRIVILEGES ON open314.* TO 'nemanja'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;` <BR>
  (Only one line with space between both)

   * open314 ............. name of the database
   * nemanja ............. user for that database
   * 123456 ...............password of that user

  You can change the data...but remember it! Later we´ll need it. Now we leave MariaDB:
  MariaDB [(none)]> `quit` <BR>

  **12**) Installation of OpenMeetings. <BR>
  We´ll install OpenMeetings in /opt/red5314. All the following information will be based in this
  directory. <BR>
  Call our folder of installation red5314
  Make the folder:
  `mkdir -p /opt/red5314`<BR>
  `cd /opt/red5314` <BR>

  Download the OpenMeetings installation archive:
  ```
  wget -c http://apache.rediris.es/openmeetings/3.1.4/bin/apache-openmeetings-3.1.4.zip
  ```
  `unzip apache-openmeetings-3.1.4.zip`

  Save the unloaded file to /opt:
  `mv apache-openmeetings-3.1.4.zip /opt`<BR>

  Unload and install the connector between OpenMeetings and MariaDB:
  `cd /opt`<BR>
  ```
  wget http://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.39/mysql-connector-java-
  5.1.39.jar
  ```

  Now copy it to where must be:
  ```
  cp /opt/mysql-connector-java-5.1.39.jar /opt/red5314/webapps/openmeetings/WEB-INF/lib
  ```

  Now we are going to configure OpenMeetings for our database in MariaDB, by adjusting few lines:<BR>
  ```
  nano /opt/red5313/webapps/openmeetings/WEB-INF/classes/META-INF/mysql_persistence.xml
  ```

  Modify in line 71:
  , Url=jdbc:mysql://localhost:3306/openmeetings_3_1?
  ...to
  , Url=jdbc:mysql://localhost:3306/open314?

  That is name of the database that we did initially (previously).

  On line 76, please modify:
  , Username=root
  ...to
  , Username=nemanja <BR>

  The database user that we did initially for the database. <BR>

  Modify in line 77:
  , Password=" /> <BR> to <BR>
  , Password=123456" /> <BR>

  It is the password that we did initially for the user "nemanja" in the database. So if initially you choose another name and password for the database, you will need to adjust them here. <BR>

  We should protect the access to the file:
  ```
  chmod 640 /opt/red5314/webapps/openmeetings/WEB-INF/classes/METAINF/mysql_persistence.xml
  ```

  **13**) Script to launch red5-OpenMeetings. <BR>
  We´ll download the script to run red5-OpenMeetings:

  `cd /opt`
  ```
  wget -c https://cwiki.apache.org/confluence/download/attachments/27838216/red5-3
  ```

  Copy it to the init.d folder:
  `cp red5-3 /etc/init.d/` <BR>

  Add execution permission:
  `chmod +x /etc/init.d/red5-3` <BR>

  If you made the installation in a different path to /opt/red5314, please edit the script and
  modify the line: <BR>
  RED5_HOME=/opt/red5314
  TO
  RED5_HOME=/your-path-installation <BR>
  (anyway review red5-3 file and make sure Paths are correctly adjusted)<BR>

  **14**) Now is the time to load the installation in your browser, and start configuring the application. First let's restart MariaDB (or any other MySQL service which you have installed): <BR>

  `systemctl restart mariadb.service` <BR>
  `service mariadb restart` <-- CentOS 6 <BR>

  Let's run red5-OpenMeetings service:
  `/etc/init.d/red5-3 start` <BR>

  Please wait until the text “clearSessionTable: 0”, is prompted. Then, open your favorite Web Browser and load:

  http://IP:5080/openmeetings/install <BR>

  Where IP is the real IP address of your server. Make sure that port 5080 is opened within the firewall, if any is used. A page similar to this one will appear:<BR>
  ![alt](https://repo.nemanja.io/images/lsjFiGF.png)<BR>

  Push the >> (button).  The next window will show the default database configuration with Derby, but we
  have deployed MySQL (MariaDB), so that needs to be switched (adjusted):
  ![alt](https://repo.nemanja.io/images/RWtlA6L.png)<BR>

  Click on > and proceed to the next menu.<BR>
  ![alt](https://repo.nemanja.io/images/rmRPCnu.png)<BR>

  Here you need to fill up basic stuff, like:

   Username = a-name ...this user will be administrator.
   Userpass = a-password ...for the previous user.

   Email = email-adress ...of the previous user.

   User Time Zone = country where is this server
   Name = example-openmeetings ...group name to choose.

  When the installation is finished, we can configure the rest of settings. <BR>

  Now go to bottom page and touch the '>>' button (double arrow). Will show this window:<BR>
  ![alt](https://repo.nemanja.io/images/0ogg9Px.png)<BR>

  Click the Finish button, and wait a few seconds until the tables populate. <BR>

  Next window will be this one: <BR>
  ![alt](https://repo.nemanja.io/images/tsQ8SyD.png)<BR>

  **NOTE**: Do not click on "Enter the Application" button. Open your terminal and push: <BR>
  `/etc/init.d/red5-3 restart` <BR>

  Now open browser and go to the:<BR>
  http://IP:5080/openmeetings <BR>

  Where IP is the real IP address of your server. Make sure that ports 1935 and 5080 are opened within firewall, if any are being used. A page will appear similar to this one:<BR>
  ![alt](https://repo.nemanja.io/images/zQMrQfU.png)<BR>

  You can enter the username and password which you have defined previously and enter your Apache Open Meeting Dashboard for the first time! Congrats.

  **15**) Important OpenMeetings's configuration:<BR>

  ![alt](https://repo.nemanja.io/images/SQn8ggm.png)<BR>

  Next important settings to be adjusted are as described bellow: <BR>
  ![alt](https://repo.nemanja.io/images/2yufQqV.png)<BR>

  Click on: swftools_path...and to the right in Value type: /usr/bin <BR>
  Click on: imagemagick_path...and to the right in Value type: /usr/bin <BR>
  Click on: sox_path...and to the right in Value type: /usr/local/bin <BR>
  Click on: ffmpeg_path...and to the right in Value type: /usr/local/bin <BR>
  Click on: office.path...and to the right in Value type: /usr/lib64/libreoffice <BR>
  Click on: jod.path...and to the right in Value type: /opt/jodconverter-core-3.0-beta-4/lib <BR>

  NOTE: Remember to save after each change (arrow number 3, in the up screenshot). In case the jod.path field does not exist, (which I have a seen few times), you will need to add new and configure the path as described previously.

  Your new virtual office is now configured and you can start configuring Rooms, Users and send invites. I have intentionally skipped SMTP configuration, because I believe that you already know how to configure an external SMTP user if needed (by default it is PHP Mailer). Soon I am planning to make an Ansible and Bash script installer for this amazing application, but until then feel free to follow my instructions. I would appreciate reviews, if possible.

html:
  <p>Recently I have been testing some software for Presenting, Learning / Web Conferencing, and data sharing, but frankly none of them displayed good results. There are a few paid solutions, but my Open Source nature led me to test the Apache Open Meeting app. OpenMeetings is a software used for presenting, online training, web conferencing, collaborative whiteboard drawing, document editing, and user desktop sharing. The product is based on the OpenLaszlo RIA framework and Red5 media server, which in turn are based on some open source components. Communication takes place in virtual "meeting rooms" which may be set to different communication, security, and video quality modes. The recommended database engine for backend support is MySQL. The product can be set up as an installed server product, or used as a hosted service. <br />
  Work on OpenMeetings began in 2006, and it has been downloaded over 250,000 times. OpenMeetings is available in 31 languages. <br />
  Official site: <a href="http://openmeetings.apache.org/">http://openmeetings.apache.org/</a> <br />
  Working on all major operating systems, but today I will write the tutorial based on the latest 3.1.4 version on CentOS version 6 and 7 Linux (dedicated server / compatible with Xen/KVM if virtual).</p>

  <p><strong>1)</strong> First we must modify Selinux level security for the installation: <BR>
  <code>yum install -y nano</code> <BR>
  <code>sudo nano /etc/selinux/config</code> <BR></p>

  <p>…modify from: <BR>
  <code>SELINUX=enforcing</code> <BR>
  ...to
  <code>SELINUX=permissive</code>
  or if SELINUX not needed <code>SELINUX=disabled</code> <BR></p>

  <p><strong>2)</strong> Now we must update the operating system. Note: this is the recommended action, not required <BR>
  <code>yum update -y</code>
  <BR> <br />
  For kernel changes and the new Selinux configuration to take effect, push: <br />
  <code>reboot</code>
  <BR></p>

  <p><strong>3)</strong> Next will be to add Repositories. (you may get files to /root or /opt it does not matter)<BR>
  <code>yum install -y wget</code></p>

  <p><mark> EPEL &amp; Remi: </mark> <BR></p>

  <pre><code>wget -c http://epel.mirror.nucleus.be/7/x86_64/e/epel-release-7-5.noarch.rpm  
  </code></pre>

  <pre><code>wget http://rpms.famillecollet.com/enterprise/remi-release-7.rpm  
  </code></pre>

  <p><code>sudo rpm -Uvh remi-release-7*.rpm epel-release-7*.rpm</code> <BR></p>

  <p>Enable Remi: <BR> <br />
  <code>nano /etc/yum.repos.d/remi.repo</code><BR></p>

  <p>And modify (the first enabled): <br />
  From enabled=0 to enabled=1 <BR></p>

  <p><mark> ElRepo </mark> <BR></p>

  <pre><code>rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org  
  </code></pre>

  <pre><code>rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm  
  </code></pre>

  <p><mark> Nux </mark> <BR></p>

  <pre><code>rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-  
  5.el7.nux.noarch.rpm  
  </code></pre>

  <p>(copy as one line)
  <BR></p>

  <p>Lastly, in this step we need to install <strong>Adobe repo 64-bit x86_64</strong> For Flash player. Push: <BR></p>

  <pre><code class="language- ">rpm -ivh http://linuxdownload.adobe.com/adobe- release/adobe-release-x86_64-1.0-1.noarch.rpm  
  </code></pre>

  <pre><code>rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux  
  </code></pre>

  <p><code>yum update -y</code></p>

  <p>YYou can always confirm if the package was successfully installed using rpm -qa, for example:<BR> <br />
  <code>rpm -qa | grep adobe</code></p>

  <p><strong>4)</strong> Installation of Oracle Java 1.8. Java 1.8 is necessary for OpenMeetings 3.1.4. Java 1.7 can be used only for the previous OM versions. We shall install Java 1.8 now. <BR></p>

  <p>cd /opt<BR></p>

  <p>Download the file: <br />
  (All in one line only. 1ª and 2ª without space between them. A space to the 3ª)<BR></p>

  <pre><code>wget --no-cookies --no-check-certificate --header "Cookie:gpw_e24=http%3A%2F%2Fwww.oracle.com%2F;oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u101-b13/jdk-8u101-linux-x64.rpm"  
  </code></pre>

  <p>And then install it: <br />
  <code>rpm -ivh jdk-8u101-linux-x64.rpm</code></p>

  <p>Maybe you have installed various versions of Java. We select the just installed Oracle Java: <br />
  <code>update-alternatives --config java</code> <BR></p>

  <p>And to see if the selected version is active: <br />
  <code>java -version</code></p>

  <p>Alternatively, you can simply run: <BR> <br />
  <code>sudo yum install java-1.8.0-openjdk icedtea-web</code> <BR></p>

  <p>And everything should work as expected, but it is recommended that you choose the latest 1.8 version.</p>

  <p><strong>5</strong>) OpenMeetings will need LibreOffice to convert pdf, which can be uploaded through Dashboard. To install LibreOffice: <BR>
  <code>yum -y install libreoffice libreoffice-headless</code> <BR></p>

  <p><strong>6</strong>) Installation of necessary packages and libraries: <BR></p>

  <pre><code>yum install -y libjpeg libjpeg-devel ghostscript freetype freetype-devel unzip gcc gcc-c++ ncurses  
  ncurses-devel make zlib zlib-devel libtool bison bison-devel openssl-devel bzip2 bzip2-devel fileroller git autoconf automake pkgconfig tomcat-native nmap vlc  
  </code></pre>

  <p>(Copy as one line with space)</p>

  <p><strong>7</strong>) Now we are going to need ImageMagick, Sox and Swftools. ImageMagick, works with image filetypes jpg, png, gif, etc. Lets get that installed: <BR>
  <code>yum install -y ImageMagick giflib giflib-devel giflib-utils</code> <BR></p>

  <p>Next is Sox, which deals with Sound. We will need to compile it with the latest version:<BR> <br />
  <code>cd /opt</code><BR></p>

  <pre><code>wget -c http://sourceforge.net/projects/sox/files/sox/14.4.2/sox-14.4.2.tar.gz  
  </code></pre>

  <p><code>tar xzvf sox-14.4.2.tar.gz</code><BR>
  <code>cd /opt/sox-14.4.2</code><BR>
  <code>./configure</code><BR>
  <code>make &amp;&amp; make install</code><BR></p>

  <p>Swftools. LibreOffice converts the uploaded office files to pdf, and Swftools converts these pdfs to <br />
  swf (flash file), which can later be presented on the whiteboard. Also convert jpg2swf, png2swf, gif2swf, and etc. <br />
  Don´t compile the newer version, since they don't have pdf2swf, which is necessary. <BR></p>

  <p><code>cd /opt</code><BR></p>

  <pre><code>wget -c http://www.swftools.org/swftools-2013-04-09-1007.tar.gz  
  </code></pre>

  <p><code>tar xzvf swftools-2013-04-09-1007.tar.gz</code> <BR>
  <code>cd /opt/swftools-2013-04-09-1007</code> <BR>
  <code>./configure --libdir=/usr/lib --bindir=/usr/bin</code> <BR>
  <code>make &amp;&amp; make install &amp;&amp; cd /opt</code> <BR></p>

  <p><strong>8</strong>) OpenMeetings application requires Adobe Flash Player for rooms.
  <code>yum install -y flash-plugin</code> <BR></p>

  <p><strong>9</strong>) Jodconverter participates in the process to convert uploaded files. The Java OpenDocument Converter, converts documents between different office formats. It leverages OpenOffice.org, which arguably provides the best import/export filters for OpenDocument and Microsoft Office formats available today. Word to OpenDocument Text (odt); OpenDocument Text (odt) to Word and etc. <BR>
  <code>cd /opt</code><BR></p>

  <pre><code>wget -c https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/jodconverter/jodconverter-core-3.0-beta-4-dist.zip  
  </code></pre>

  <p><code>unzip jodconverter-core-3.0-beta-4-dist.zip</code><BR></p>

  <p>For, now we will leave it, and remember the path where was extracted.</p>

  <p><strong>10</strong>) Next, we will need to install FFmpeg. FFmpeg is a free software project that produces libraries and programs for handling multimedia data. FFmpeg works with video. We will now install the necessary packages, libraries and vlc player which is needed to play the meeting recordings. <BR></p>

  <pre><code>yum install -y glibc alsa-lib-devel faac faac-devel faad2 faad2-devel gsm gsm-devel imlib2 imlib2-devel lame-devel vorbis-tools theora-tools libvpx-devel vlc autoconf automake cmake freetypedevel gcc gcc-c++ git libtool make mercurial nasm pkgconfig zlib-devel curl  
  </code></pre>

  <p>(copy as one line and execute on server)</p>

  <p>This ffmpeg compilation is based on this url, updated file versions 28-9-2016: <BR> <br />
  <a href="https://trac.ffmpeg.org/wiki/CompilationGuide/Centos">https://trac.ffmpeg.org/wiki/CompilationGuide/Centos</a>  </p>

  <pre><code>wget https://cwiki.apache.org/confluence/download/attachments/27838216/ffmpeg-centos2.sh  
  </code></pre>

  <p><code>chmod a+rwx ffmpeg-centos2.sh</code> <BR></p>

  <p>The installation can take up to 30 minutes to finish: <br />
  <code>./ffmpeg-centos2.sh</code> <BR>
  When finished, please, go to step 11).</p>

  <p><strong>11</strong>) Installation MariaDB data server. MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. It is notable for being led by the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle. You may use the default Oracle version or Percona, but that is up to you. <BR></p>

  <p><code>yum install -y mariadb-server</code> <BR></p>

  <p>Now we are going to start MariaDB server (CentOS7):<BR> <br />
  <code>systemctl start mariadb.service</code> <BR>
  <code>service mariadb start</code> &lt;-- For CentOS 6<BR></p>

  <p>Now we need to set password for Root. Please, modify new-password by your own. <BR> <br />
  <code>mysqladmin -u root password new-password</code> <BR></p>

  <p>Note: You may edit /root/.my.cnf and configure password less access for MySQL, or use -u root -p command. <BR></p>

  <p>Make a database for OpenMeetings: <br />
  <code>mysql -u root -p</code> <BR></p>

  <p>MariaDB [(none)]> <code>CREATE DATABASE open314 DEFAULT CHARACTER SET 'utf8';</code> <BR></p>

  <p>Now we create a user with all permission on this open314 database. <br />
  MariaDB [(none)]> <code>GRANT ALL PRIVILEGES ON open314.* TO 'nemanja'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;</code> <BR> <br />
  (Only one line with space between both)</p>

  <ul>
  <li>open314 ............. name of the database</li>
  <li>nemanja ............. user for that database</li>
  <li>123456 ...............password of that user</li>
  </ul>

  <p>You can change the data...but remember it! Later we´ll need it. Now we leave MariaDB: <br />
  MariaDB [(none)]> <code>quit</code> <BR></p>

  <p><strong>12</strong>) Installation of OpenMeetings. <BR>
  We´ll install OpenMeetings in /opt/red5314. All the following information will be based in this <br />
  directory. <BR> <br />
  Call our folder of installation red5314 <br />
  Make the folder: <br />
  <code>mkdir -p /opt/red5314</code><BR>
  <code>cd /opt/red5314</code> <BR></p>

  <p>Download the OpenMeetings installation archive:  </p>

  <pre><code>wget -c http://apache.rediris.es/openmeetings/3.1.4/bin/apache-openmeetings-3.1.4.zip  
  </code></pre>

  <p><code>unzip apache-openmeetings-3.1.4.zip</code></p>

  <p>Save the unloaded file to /opt: <br />
  <code>mv apache-openmeetings-3.1.4.zip /opt</code><BR></p>

  <p>Unload and install the connector between OpenMeetings and MariaDB: <br />
  <code>cd /opt</code><BR></p>

  <pre><code>wget http://repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.39/mysql-connector-java-  
  5.1.39.jar  
  </code></pre>

  <p>Now copy it to where must be:  </p>

  <pre><code>cp /opt/mysql-connector-java-5.1.39.jar /opt/red5314/webapps/openmeetings/WEB-INF/lib  
  </code></pre>

  <p>Now we are going to configure OpenMeetings for our database in MariaDB, by adjusting few lines:<BR>  </p>

  <pre><code>nano /opt/red5313/webapps/openmeetings/WEB-INF/classes/META-INF/mysql_persistence.xml  
  </code></pre>

  <p>Modify in line 71: <br />
  , Url=jdbc:mysql://localhost:3306/openmeetings<em>3</em>1?
  ...to
  , Url=jdbc:mysql://localhost:3306/open314?</p>

  <p>That is name of the database that we did initially (previously).</p>

  <p>On line 76, please modify: <br />
  , Username=root
  ...to
  , Username=nemanja <BR></p>

  <p>The database user that we did initially for the database. <BR></p>

  <p>Modify in line 77: <br />
  , Password=" /> <BR> to <BR>
  , Password=123456" /> <BR></p>

  <p>It is the password that we did initially for the user "nemanja" in the database. So if initially you choose another name and password for the database, you will need to adjust them here. <BR></p>

  <p>We should protect the access to the file:  </p>

  <pre><code>chmod 640 /opt/red5314/webapps/openmeetings/WEB-INF/classes/METAINF/mysql_persistence.xml  
  </code></pre>

  <p><strong>13</strong>) Script to launch red5-OpenMeetings. <BR>
  We´ll download the script to run red5-OpenMeetings:</p>

  <p><code>cd /opt</code></p>

  <pre><code>wget -c https://cwiki.apache.org/confluence/download/attachments/27838216/red5-3  
  </code></pre>

  <p>Copy it to the init.d folder: <br />
  <code>cp red5-3 /etc/init.d/</code> <BR></p>

  <p>Add execution permission: <br />
  <code>chmod +x /etc/init.d/red5-3</code> <BR></p>

  <p>If you made the installation in a different path to /opt/red5314, please edit the script and <br />
  modify the line: <BR> <br />
  RED5<em>HOME=/opt/red5314 <br />
  TO <br />
  RED5</em>HOME=/your-path-installation <BR> <br />
  (anyway review red5-3 file and make sure Paths are correctly adjusted)<BR></p>

  <p><strong>14</strong>) Now is the time to load the installation in your browser, and start configuring the application. First let's restart MariaDB (or any other MySQL service which you have installed): <BR></p>

  <p><code>systemctl restart mariadb.service</code> <BR>
  <code>service mariadb restart</code> &lt;-- CentOS 6 <BR></p>

  <p>Let's run red5-OpenMeetings service: <br />
  <code>/etc/init.d/red5-3 start</code> <BR></p>

  <p>Please wait until the text “clearSessionTable: 0”, is prompted. Then, open your favorite Web Browser and load:</p>

  <p><a href="http://IP:5080/openmeetings/install">http://IP:5080/openmeetings/install</a> <BR></p>

  <p>Where IP is the real IP address of your server. Make sure that port 5080 is opened within the firewall, if any is used. A page similar to this one will appear:<BR> <br />
  <img src="https://repo.nemanja.io/images/lsjFiGF.png" alt="alt" title="" /><BR></p>

  <p>Push the >> (button).  The next window will show the default database configuration with Derby, but we <br />
  have deployed MySQL (MariaDB), so that needs to be switched (adjusted): <br />
  <img src="https://repo.nemanja.io/images/RWtlA6L.png" alt="alt" title="" /><BR></p>

  <p>Click on > and proceed to the next menu.<BR> <br />
  <img src="https://repo.nemanja.io/images/rmRPCnu.png" alt="alt" title="" /><BR></p>

  <p>Here you need to fill up basic stuff, like:</p>

  <p>Username = a-name ...this user will be administrator.
   Userpass = a-password ...for the previous user.</p>

  <p>Email = email-adress ...of the previous user.</p>

  <p>User Time Zone = country where is this server
   Name = example-openmeetings ...group name to choose.</p>

  <p>When the installation is finished, we can configure the rest of settings. <BR></p>

  <p>Now go to bottom page and touch the '>>' button (double arrow). Will show this window:<BR> <br />
  <img src="https://repo.nemanja.io/images/0ogg9Px.png" alt="alt" title="" /><BR></p>

  <p>Click the Finish button, and wait a few seconds until the tables populate. <BR></p>

  <p>Next window will be this one: <BR> <br />
  <img src="https://repo.nemanja.io/images/tsQ8SyD.png" alt="alt" title="" /><BR></p>

  <p><strong>NOTE</strong>: Do not click on "Enter the Application" button. Open your terminal and push: <BR>
  <code>/etc/init.d/red5-3 restart</code> <BR></p>

  <p>Now open browser and go to the:<BR> <br />
  <a href="http://IP:5080/openmeetings">http://IP:5080/openmeetings</a> <BR></p>

  <p>Where IP is the real IP address of your server. Make sure that ports 1935 and 5080 are opened within firewall, if any are being used. A page will appear similar to this one:<BR> <br />
  <img src="https://repo.nemanja.io/images/zQMrQfU.png" alt="alt" title="" /><BR></p>

  <p>You can enter the username and password which you have defined previously and enter your Apache Open Meeting Dashboard for the first time! Congrats.</p>

  <p><strong>15</strong>) Important OpenMeetings's configuration:<BR></p>

  <p><img src="https://repo.nemanja.io/images/SQn8ggm.png" alt="alt" title="" /><BR></p>

  <p>Next important settings to be adjusted are as described bellow: <BR> <br />
  <img src="https://repo.nemanja.io/images/2yufQqV.png" alt="alt" title="" /><BR></p>

  <p>Click on: swftools<em>path...and to the right in Value type: /usr/bin <BR> <br />
  Click on: imagemagick</em>path...and to the right in Value type: /usr/bin <BR> <br />
  Click on: sox<em>path...and to the right in Value type: /usr/local/bin <BR> <br />
  Click on: ffmpeg</em>path...and to the right in Value type: /usr/local/bin <BR> <br />
  Click on: office.path...and to the right in Value type: /usr/lib64/libreoffice <BR> <br />
  Click on: jod.path...and to the right in Value type: /opt/jodconverter-core-3.0-beta-4/lib <BR></p>

  <p>NOTE: Remember to save after each change (arrow number 3, in the up screenshot). In case the jod.path field does not exist, (which I have a seen few times), you will need to add new and configure the path as described previously.</p>

  <p>Your new virtual office is now configured and you can start configuring Rooms, Users and send invites. I have intentionally skipped SMTP configuration, because I believe that you already know how to configure an external SMTP user if needed (by default it is PHP Mailer). Soon I am planning to make an Ansible and Bash script installer for this amazing application, but until then feel free to follow my instructions. I would appreciate reviews, if possible.</p>


uuid: 64e0324b-1598-46e5-930e-e706be339e80
title: How to create custom KVM template for SolusVM
slug: how-to-create-custom-kvm-template-for-solusvm
image: /content/images/2017/05/638917.png
status: published
language: en_US
markdown:
  Lately, I have been working with a lot of customised virtual machines based mostly on the KVM/XEN technology. The one really popular and easy to manage is from company SolusVM (recently purchased by OnApp) http://solusvm.com/. The most of the commercial hosting companies are using it nowadays. If you are deploying Master/Slave solution which KVM requires, you will find default templates on the following page here: https://tdn.solusvm.com/. But what if we want to create our own, custom KVM template with pre-installed software/configuration. In this article, I will describe how you can create your own custom KVM template designed for the SolusVM virtualization software.

  1) Download and boot CentOS 6 or 7 minimal iso files and upload to the /home/solusvm/kvm/iso folder on the MASTER SolusVM server.

  2) In SolusVM Dashboard go to Media --> Add ISO fill in all details, leaving PAE as 'off'
  ![alt](/content/images/2017/05/SolusVM-ISO.png)

  3) Go back to the Media section and click on the 'Media Sync' option. Select 'CREATE', scroll down to the bottom of the page and in the ISOS section select ISO files, on the left side you wish to synchronise and the nodes on the right side where files will be copied.

  ![alt](/content/images/2017/05/SolusVM-mediasync.png)

  You will then see:
  Successful! Sync job added

  You can go back, click on 'Media Sync' again and check the status of the synchronisation. Alternatively, you can log in to the KVM Slave node, and watch the curl process. It will show as something like this:

  ```
  curl -vvv --stderr /usr/local/solusvm/tmp/bc3x3ea4e26e5xe1af1408321416x956113a46x58763.log --insecure https://x.x.x.x:5656/9bxebcaxf01d29abf5xa6x39fc3ca6199ce7d9c7d2fd/CentOS-7-x86_64-Minimal-1611.iso -o /home/solusvm/xen/iso/CentOS-7-x86_64-Minimal-1611.iso
  ```

  Note: Make sure that the primary partition on the KVM Slave Hypervisors has enough space to store both ISO and fresh template data. By default that partition is 60Gb, but please check before pushing anything, to make sure.

  4) Go to 'Virtual server' tab --> Select 'Add new Server' and select KVM.
  5) Leave the 'Template' tab set to <none> and choose OS from <iso> tab. Type hostname and enter the proper MAC address (if needed). Leave 'Boot from network' unchecked.

  ![alt](/content/images/2017/05/solusvm-addvps.png)

  Click 'ADD'. On the next screen, you will be prompted to enter the VNC password, port number and IP address, but you can ignore that since you will finish the installation from within the main SolusVM dashboard which is already authenticated.

  6) Go back to 'Virtual Servers' --> 'List servers' and click on VNC to load the installation screen. Chrome seems to have some difficulties, so you can use Firefox, which works perfectly.

  ![alt](/content/images/2017/05/QEMU-VNC-ISO-Install.png)

  7) Select 'Language' and proceed. On the config screen, you can configure things like Date/Timezone, Keyboard, Security policies, but the most important part here is to click on the **INSTALLATION DESTINATION** and check **"I WILL CONFIGURE PARTITIONING"**. The reason for this is because if you leave it on the automatic option, it will configure LVM instead of IDE, and we need IDE partitioning so that later the /scripts/kvmtemplate script can configure the SolusVM template properly.

  ![alt](/content/images/2017/05/CentOS7-partitioning-SolusVM-ISO.png)

  Next you will be prompted with a "+" button down so you can start manually creating partitions. In my case, I want every custom cPanel installation to have 11GB for the "/" Root partition and 2GB for SWAP. You may adjust this, since the template will be Generation 2 anyway which gives the option to resize the partition for new virtual servers. This step is just to avoid LVM at all costs.

  ![alt](/content/images/2017/05/CentOS7-partitioning-SolusVM-ISO-1.png)

  8) Click on the 'NETWORK & HOSTNAME' option and enable it by selecting 'ON'.

  9) While the installation is running, you can set your Root password and create additional users (if needed). When SolusVM KVM template is created, the password will be changed to something random.

  10) When the OS installation complete, click 'REBOOT' and if everything is fine, you should get to the CentOS 7 login.

  11) Login as root and type your previously chosen password. Occasionally the network adapter is not properly added, so if when you type:

  ```
  ip addr show
  ```

  You see something like:

  ![alt](/content/images/2017/05/CentOS7-SolusVM-ISO-3.png)

  Go back to the SolusVM Dashboard and click 'Re-Configure' at the top of the page. Please wait for the system to finish. You will see on your screen. Virtual Server Re-Configured! System Files Changed:
  /etc/sysconfig/network
  /etc/sysconfig/network-scripts/ifcfg-eth0
  /etc/sysconfig/network-scripts/route-eth0
  /etc/resolv.conf
  /etc/hosts

  90% of the time after that you will be able to bring the IP address up and continue work from your local computer terminal. If not, log back into the VNC console and go to the **/etc/sysconfig/network-scripts** folder, and copy **ifcfg-ens3** to **ifcfg-ens3.bak** and then **cp ifcfg-eth0 to efcfg-ens3**. Click on 'reboot' from the VNC console, go back and type:

  ```
  ip addr show
  ```

  To see if the address is assigned properly. Always ping a site, I often do google.rs or google.com, which are safe.

  12) Log in as root from your local computer using SSH, port 22.

  13) type **yum update** to update the kernel and software to the latest version.

  14) type **reboot**

  15) Make adjustments on the server and type **halt** to power off the server.

  16) On the KVM Slave node where this instance is settled type:

  ```
  /scripts/kvmtemplate --mode=prescan --vmid=kvm108 --generation=2
  ```

  * Where kvm108 is real the KVM ID your server is using.

  Output will be similar to this:

  ```
  Name Type VFS Label MBR Size Parent UUID
  /dev/sda1 filesystem ext4 - - 11G - 7cad89f1-23d6-4a74-aa98-926c4626a6f9
  /dev/sda2 filesystem swap - - 2.0G - 3c13d689-bca8-4709-8ace-28a53fce7e96
  /dev/sda1 partition - - 83 11G /dev/sda -
  /dev/sda2 partition - - 82 2.0G /dev/sda -
  /dev/sda device - - - 13G - -
  ```

  From this, we now know that the primary partition is using /dev/sda1 and that SWAP must use /dev/sda2

  Note: This is very important because when you create the KVM Template from within the SolusVM dashboard you need to enter the proper partition number for the Resizable and SWAP partition. If you use QEMU-KVM (or virt-manager GUI) for running your virtual machines, you can specify a disk driver to be used for accessing the machine’s disk image. By default, IDE is selected, but VirtIO is a very popular choice as well. From my experience, VirtIO is giving better performance, but IDE is more stable, but this is my personal opinion, you may switch to the VirtIO during the CentOS 7 installation process.

  17) On the host node you will need to mount the virtual server's partitions:

  ```
  kpartx -a /dev/VOLUME_GROUP/KVMID_img
  mount /dev/mapper/VOLUME_GROUP-KVMID_img1 /mnt
  ```

  In my case that was:
  kpartx -a /dev/system/kvm108_img
  mount /dev/mapper/system-kvm108_img1 /mnt
  (you notice that we mounted _kvm1, well that is partition 1 or the space reserved for Root system)

  18) Jump to the /mnt/etc folder and edit fstab file. The file will look like:

  ```
  #
  # /etc/fstab
  # Created by anaconda on Sun May 7 13:24:19 2017
  #
  # Accessible filesystems, by reference, are maintained under '/dev/disk'
  # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
  #
  UUID=7cad89f1-23d6-4a74-aa98-926c4626a6f9 / ext4 defaults 1 1
  UUID=3c13d689-bca8-4709-8ace-28a53fce7e96 swap swap defaults 0 0
  ```

  You need to adjust it to:

  ```
  /dev/sda1 / ext4 defaults 1 1
  /dev/sda2 swap swap defaults 0 0
  ```

  19) Save file and then, let's remove bash history:

  ```
  rm -f /mnt/root/.bash_history
  ```

  20) Some distros need the following to get networking working (CentOS6 and older Debian versions):

  ```
  rm /mnt/etc/udev/rules.d/70-persistent-net.rules
  touch /mnt/etc/udev/rules.d/70-persistent-net.rules
  chattr +i /mnt/etc/udev/rules.d/70-persistent-net.rules
  ```

  21) Unmount the virtual server:

  ```
  cd ~
  umount /mnt
  ```

  22) Remove the mappings:

  ```
  kpartx -d /dev/VOLUME_GROUP/KVMID_img
  ```

  In my case:
  kpartx -d /dev/system/kvm108_img

  23) Now, everything is ready for the KVM template to be created:

  ```
  /scripts/kvmtemplate --mode=prepare --vmid=kvm108 --generation=2
  ```

  Once the source virtual server is ready to be packaged you simply run the package scripts. Example:

  ```
  /scripts/kvmtemplate --mode=package --vmid=kvm101 --generation=2
  ```

  It may take a while for it to complete. Once it's done you will be given a location and file name for the template. Example, in my case, is:

  ```
  Copy to destination and make sparse ...

  Sparsify operation completed with no errors. Before deleting the old 
  disk, carefully check that the target disk boots and works correctly.

  Creating template. Please wait...
  Template: /home/solusvm/kvm/template/template-3fa91718ef6628e568a4eb4eba7c2f083c888f3e.gz
  ```

  The next step is to rename the template. Example:

  ```
  cd /home/solusvm/kvm/template/
  mv template-3fa91718ef6628e568a4eb4eba7c2f083c888f3e.gz linux-centos-7-x86_64-custom-gen2-v1.gz
  ```

  or if it was a Windows template:
  ```
  cd /home/solusvm/kvm/template/
  mv template-149fee41d5801f96368c959a00dd6092a56ba031.gz windows-2008-server-r2-custom-x86_64-gen2-v1.gz
  ```
  Note: It is really important to delegate name of the template properly. You may freely follow my example naming it as custom-gen2, custom1-gen2... etc and then add proper descriptions within the SolusVM Main dashboard. I have tried making custom names, and that never worked.

  rsync gz archive and send it to the SolusVM Master node under the /home/solusvm/kvm/template/ folder.

  Note: Remove it on the current node after the archive is sent to the Master node. It will return back naturally when Media Sync is pushed.

  Finally, make sure when adding the template to SolusVM you specify the correct generation and partition to resize. Go back to the Media section and click on the 'Media Sync' option. Select CREATE, scroll down to the bottom where KVM Templates are stored and select a new template on the left side and the nodes on the right side where it needs to be copied (synchronised) -- (See Step 3. in this document)

  24) Wait again for the new template to get synced on the nodes. After that step, you can go to the Virtual Servers section and provision your first server with customised KVM template prepared for the SolusVM software.

  I hope this article will be helpful to anyone who wish to automate the vps deployment process and use QCOW2/KVM environment. I am starting soon http://vmtemplates.com website where I plan with really small amount to deploy many ready made templates for all type of virtual machines.

  Good luck!

html:
  <p>Lately, I have been working with a lot of customised virtual machines based mostly on the KVM/XEN technology. The one really popular and easy to manage is from company SolusVM (recently purchased by OnApp) <a href="http://solusvm.com/">http://solusvm.com/</a>. The most of the commercial hosting companies are using it nowadays. If you are deploying Master/Slave solution which KVM requires, you will find default templates on the following page here: <a href="https://tdn.solusvm.com/">https://tdn.solusvm.com/</a>. But what if we want to create our own, custom KVM template with pre-installed software/configuration. In this article, I will describe how you can create your own custom KVM template designed for the SolusVM virtualization software.</p>

  <p>1) Download and boot CentOS 6 or 7 minimal iso files and upload to the /home/solusvm/kvm/iso folder on the MASTER SolusVM server.</p>

  <p>2) In SolusVM Dashboard go to Media --> Add ISO fill in all details, leaving PAE as 'off' <br />
  <img src="/content/images/2017/05/SolusVM-ISO.png" alt="alt" /></p>

  <p>3) Go back to the Media section and click on the 'Media Sync' option. Select 'CREATE', scroll down to the bottom of the page and in the ISOS section select ISO files, on the left side you wish to synchronise and the nodes on the right side where files will be copied.</p>

  <p><img src="/content/images/2017/05/SolusVM-mediasync.png" alt="alt" /></p>

  <p>You will then see: <br />
  Successful! Sync job added</p>

  <p>You can go back, click on 'Media Sync' again and check the status of the synchronisation. Alternatively, you can log in to the KVM Slave node, and watch the curl process. It will show as something like this:</p>

  <pre><code>curl -vvv --stderr /usr/local/solusvm/tmp/bc3x3ea4e26e5xe1af1408321416x956113a46x58763.log --insecure https://x.x.x.x:5656/9bxebcaxf01d29abf5xa6x39fc3ca6199ce7d9c7d2fd/CentOS-7-x86_64-Minimal-1611.iso -o /home/solusvm/xen/iso/CentOS-7-x86_64-Minimal-1611.iso  
  </code></pre>

  <p>Note: Make sure that the primary partition on the KVM Slave Hypervisors has enough space to store both ISO and fresh template data. By default that partition is 60Gb, but please check before pushing anything, to make sure.</p>

  <p>4) Go to 'Virtual server' tab --> Select 'Add new Server' and select KVM. <br />
  5) Leave the 'Template' tab set to <none> and choose OS from <iso> tab. Type hostname and enter the proper MAC address (if needed). Leave 'Boot from network' unchecked.</p>

  <p><img src="/content/images/2017/05/solusvm-addvps.png" alt="alt" /></p>

  <p>Click 'ADD'. On the next screen, you will be prompted to enter the VNC password, port number and IP address, but you can ignore that since you will finish the installation from within the main SolusVM dashboard which is already authenticated.</p>

  <p>6) Go back to 'Virtual Servers' --> 'List servers' and click on VNC to load the installation screen. Chrome seems to have some difficulties, so you can use Firefox, which works perfectly.</p>

  <p><img src="/content/images/2017/05/QEMU-VNC-ISO-Install.png" alt="alt" /></p>

  <p>7) Select 'Language' and proceed. On the config screen, you can configure things like Date/Timezone, Keyboard, Security policies, but the most important part here is to click on the <strong>INSTALLATION DESTINATION</strong> and check <strong>"I WILL CONFIGURE PARTITIONING"</strong>. The reason for this is because if you leave it on the automatic option, it will configure LVM instead of IDE, and we need IDE partitioning so that later the /scripts/kvmtemplate script can configure the SolusVM template properly.</p>

  <p><img src="/content/images/2017/05/CentOS7-partitioning-SolusVM-ISO.png" alt="alt" /></p>

  <p>Next you will be prompted with a "+" button down so you can start manually creating partitions. In my case, I want every custom cPanel installation to have 11GB for the "/" Root partition and 2GB for SWAP. You may adjust this, since the template will be Generation 2 anyway which gives the option to resize the partition for new virtual servers. This step is just to avoid LVM at all costs.</p>

  <p><img src="/content/images/2017/05/CentOS7-partitioning-SolusVM-ISO-1.png" alt="alt" /></p>

  <p>8) Click on the 'NETWORK &amp; HOSTNAME' option and enable it by selecting 'ON'.</p>

  <p>9) While the installation is running, you can set your Root password and create additional users (if needed). When SolusVM KVM template is created, the password will be changed to something random.</p>

  <p>10) When the OS installation complete, click 'REBOOT' and if everything is fine, you should get to the CentOS 7 login.</p>

  <p>11) Login as root and type your previously chosen password. Occasionally the network adapter is not properly added, so if when you type:</p>

  <pre><code>ip addr show  
  </code></pre>

  <p>You see something like:</p>

  <p><img src="/content/images/2017/05/CentOS7-SolusVM-ISO-3.png" alt="alt" /></p>

  <p>Go back to the SolusVM Dashboard and click 'Re-Configure' at the top of the page. Please wait for the system to finish. You will see on your screen. Virtual Server Re-Configured! System Files Changed: <br />
  /etc/sysconfig/network
  /etc/sysconfig/network-scripts/ifcfg-eth0
  /etc/sysconfig/network-scripts/route-eth0
  /etc/resolv.conf
  /etc/hosts</p>

  <p>90% of the time after that you will be able to bring the IP address up and continue work from your local computer terminal. If not, log back into the VNC console and go to the <strong>/etc/sysconfig/network-scripts</strong> folder, and copy <strong>ifcfg-ens3</strong> to <strong>ifcfg-ens3.bak</strong> and then <strong>cp ifcfg-eth0 to efcfg-ens3</strong>. Click on 'reboot' from the VNC console, go back and type:</p>

  <pre><code>ip addr show  
  </code></pre>

  <p>To see if the address is assigned properly. Always ping a site, I often do google.rs or google.com, which are safe.</p>

  <p>12) Log in as root from your local computer using SSH, port 22.</p>

  <p>13) type <strong>yum update</strong> to update the kernel and software to the latest version.</p>

  <p>14) type <strong>reboot</strong></p>

  <p>15) Make adjustments on the server and type <strong>halt</strong> to power off the server.</p>

  <p>16) On the KVM Slave node where this instance is settled type:</p>

  <pre><code>/scripts/kvmtemplate --mode=prescan --vmid=kvm108 --generation=2
  </code></pre>

  <ul>
  <li>Where kvm108 is real the KVM ID your server is using.</li>
  </ul>

  <p>Output will be similar to this:</p>

  <pre><code>Name Type VFS Label MBR Size Parent UUID  
  /dev/sda1 filesystem ext4 - - 11G - 7cad89f1-23d6-4a74-aa98-926c4626a6f9
  /dev/sda2 filesystem swap - - 2.0G - 3c13d689-bca8-4709-8ace-28a53fce7e96
  /dev/sda1 partition - - 83 11G /dev/sda -
  /dev/sda2 partition - - 82 2.0G /dev/sda -
  /dev/sda device - - - 13G - -
  </code></pre>

  <p>From this, we now know that the primary partition is using /dev/sda1 and that SWAP must use /dev/sda2</p>

  <p>Note: This is very important because when you create the KVM Template from within the SolusVM dashboard you need to enter the proper partition number for the Resizable and SWAP partition. If you use QEMU-KVM (or virt-manager GUI) for running your virtual machines, you can specify a disk driver to be used for accessing the machine’s disk image. By default, IDE is selected, but VirtIO is a very popular choice as well. From my experience, VirtIO is giving better performance, but IDE is more stable, but this is my personal opinion, you may switch to the VirtIO during the CentOS 7 installation process.</p>

  <p>17) On the host node you will need to mount the virtual server's partitions:</p>

  <pre><code>kpartx -a /dev/VOLUME_GROUP/KVMID_img  
  mount /dev/mapper/VOLUME_GROUP-KVMID_img1 /mnt  
  </code></pre>

  <p>In my case that was: <br />
  kpartx -a /dev/system/kvm108<em>img <br />
  mount /dev/mapper/system-kvm108</em>img1 /mnt <br />
  (you notice that we mounted _kvm1, well that is partition 1 or the space reserved for Root system)</p>

  <p>18) Jump to the /mnt/etc folder and edit fstab file. The file will look like:</p>

  <pre><code>#
  # /etc/fstab
  # Created by anaconda on Sun May 7 13:24:19 2017
  #
  # Accessible filesystems, by reference, are maintained under '/dev/disk'
  # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
  #
  UUID=7cad89f1-23d6-4a74-aa98-926c4626a6f9 / ext4 defaults 1 1  
  UUID=3c13d689-bca8-4709-8ace-28a53fce7e96 swap swap defaults 0 0  
  </code></pre>

  <p>You need to adjust it to:</p>

  <pre><code>/dev/sda1 / ext4 defaults 1 1
  /dev/sda2 swap swap defaults 0 0
  </code></pre>

  <p>19) Save file and then, let's remove bash history:</p>

  <pre><code>rm -f /mnt/root/.bash_history  
  </code></pre>

  <p>20) Some distros need the following to get networking working (CentOS6 and older Debian versions):</p>

  <pre><code>rm /mnt/etc/udev/rules.d/70-persistent-net.rules  
  touch /mnt/etc/udev/rules.d/70-persistent-net.rules  
  chattr +i /mnt/etc/udev/rules.d/70-persistent-net.rules  
  </code></pre>

  <p>21) Unmount the virtual server:</p>

  <pre><code>cd ~  
  umount /mnt  
  </code></pre>

  <p>22) Remove the mappings:</p>

  <pre><code>kpartx -d /dev/VOLUME_GROUP/KVMID_img  
  </code></pre>

  <p>In my case: <br />
  kpartx -d /dev/system/kvm108_img</p>

  <p>23) Now, everything is ready for the KVM template to be created:</p>

  <pre><code>/scripts/kvmtemplate --mode=prepare --vmid=kvm108 --generation=2
  </code></pre>

  <p>Once the source virtual server is ready to be packaged you simply run the package scripts. Example:</p>

  <pre><code>/scripts/kvmtemplate --mode=package --vmid=kvm101 --generation=2
  </code></pre>

  <p>It may take a while for it to complete. Once it's done you will be given a location and file name for the template. Example, in my case, is:</p>

  <pre><code>Copy to destination and make sparse ...

  Sparsify operation completed with no errors. Before deleting the old  
  disk, carefully check that the target disk boots and works correctly.

  Creating template. Please wait...  
  Template: /home/solusvm/kvm/template/template-3fa91718ef6628e568a4eb4eba7c2f083c888f3e.gz  
  </code></pre>

  <p>The next step is to rename the template. Example:</p>

  <pre><code>cd /home/solusvm/kvm/template/  
  mv template-3fa91718ef6628e568a4eb4eba7c2f083c888f3e.gz linux-centos-7-x86_64-custom-gen2-v1.gz  
  </code></pre>

  <p>or if it was a Windows template:  </p>

  <pre><code>cd /home/solusvm/kvm/template/  
  mv template-149fee41d5801f96368c959a00dd6092a56ba031.gz windows-2008-server-r2-custom-x86_64-gen2-v1.gz  
  </code></pre>

  <p>Note: It is really important to delegate name of the template properly. You may freely follow my example naming it as custom-gen2, custom1-gen2... etc and then add proper descriptions within the SolusVM Main dashboard. I have tried making custom names, and that never worked.</p>

  <p>rsync gz archive and send it to the SolusVM Master node under the /home/solusvm/kvm/template/ folder.</p>

  <p>Note: Remove it on the current node after the archive is sent to the Master node. It will return back naturally when Media Sync is pushed.</p>

  <p>Finally, make sure when adding the template to SolusVM you specify the correct generation and partition to resize. Go back to the Media section and click on the 'Media Sync' option. Select CREATE, scroll down to the bottom where KVM Templates are stored and select a new template on the left side and the nodes on the right side where it needs to be copied (synchronised) -- (See Step 3. in this document)</p>

  <p>24) Wait again for the new template to get synced on the nodes. After that step, you can go to the Virtual Servers section and provision your first server with customised KVM template prepared for the SolusVM software.</p>

  <p>I hope this article will be helpful to anyone who wish to automate the vps deployment process and use QCOW2/KVM environment. I am starting soon <a href="http://vmtemplates.com">http://vmtemplates.com</a> website where I plan with really small amount to deploy many ready made templates for all type of virtual machines.</p>

  <p>Good luck!</p>


uuid: 4b05b013-4ac9-4659-9422-bbed60d45864
title: Step-by-step guide on how to install optional Python version on CentOS server
slug: step-by-step-guide-on-how-to-install-optional-python-version-on-centos-server
image: /content/images/2017/06/python-logo-3-6.gif
status: published
language: en_US
meta_title: Step-by-step guide on how to install optional Python version
markdown:
  Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. Python's simple, easy to learn syntax emphasises readability and therefore reduces the cost of program maintenance. Python supports modules and packages, which encourages program modularity and code reuse. The Python interpreter and the extensive standard library are available in source or binary form without charge for all the key platforms and can be freely distributed.

  By default on CentOS or any other popular Linux distribution Python, 2.6 version is installed.

  You can check the current version with the following command:

  ```
  ~]# python -V
  Python 2.6.6
  ```

  **Install optional Python version from IUS Community repository**
  <BR>
  To install Python 2.7 on the server, please do the following:

  Log in as root (superuser) and push:
  ```
  cd /root ; wget -c https://centos6.iuscommunity.org/ius-release.rpm ; rpm -ivh ius-release.rpm
  ```

  On CentOS 7:
  ```
  cd /root ; wget -c https://centos7.iuscommunity.org/ius-release.rpm ; rpm -ivh ius-release.rpm
  ```
  After IUS Community repositories are successfully installed (https://ius.io/GettingStarted/) you need to run the following commands:
  yum -y install python27 python27-devel python27-pip python27-setuptools python27-virtualenv --enablerepo=ius

  After packages get successfully installed, you can test if python2.7 and pip are working as expected, for example:
  ```
  [root@second ~]# python2.7 -V
  Python 2.7.13

  [root@second ~]# pip2.7
  Usage: 
  pip <command> [options]

  [root@second ~]# virtualenv-2.7
  You must provide a DEST_DIR
  Usage: virtualenv-2.7 [OPTIONS] DEST_DIR
  Options:
  --version show program's version number and exit
  -h, --help show this help message and exit
  -v, --verbose Increase verbosity.
  -q, --quiet Decrease verbosity.
  -p PYTHON_EXE, --python=PYTHON_EXE
  ```

  You may create an alias within the username by adjusting .bashrc file for python and virtualenv with the optional Python 2.7 version, for example:
  alias python='/usr/bin/python2.7'
  alias virtuelenv='/usr/bin/virtualenv-2.7'

  Repeat procedure In case you wish optional Python3+ version installed.

  Related articles
  https://ius.io/GettingStarted/
  https://community.letsencrypt.org/t/redhat-centos-6-x-users-need-python-2-7/2190

  <BR>
  **Install optional Python version from Software Collections (SCL) Repository**
  <BR>

  Software Collections give you the power to build, install, and use multiple versions of software on the same system, without affecting system-wide installed packages.

  To install Python 2.7 on CentOS server, please do the following:

  Install The Software Collections ( SCL ) Repository:
  ```
  yum install centos-release-scl
  ```
  Install python27 from SCL repository:
  ```
  yum install python27 -y
  ```
  Enable python27 with the following command:
  ```
  scl enable python27 bash
  ```
  Test active Python version:
  ```
  python -V
  ```
  The System should return "Python 2.7.X" Where X is the latest version of python in this branch.

  Related article:
  https://www.softwarecollections.org/en/

  <BR>
  **Install optional Python version using Anaconda**
  <BR>

  With Anaconda you can run multiple versions of Python in isolated environments, so choose the download with the Python version that you use more often, as that will be your default Python version.

  If you don't have the time or disk space for the entire distribution, try Miniconda (http://conda.pydata.org/miniconda.html) which contains only conda and Python. Then install just the individual packages you want through the conda command.

  1) Linux installer at https://www.continuum.io/downloads
  2) Run the provided bash command from above to start the installer:
  ```
  bash Anaconda3-4.4.0-Linux-x86_64.sh
  ```

  For Python 2.7 version:
  ```
  bash Anaconda2-4.4.0-Linux-x86_64.sh
  ```

  3) Agree to the license agreement
  4) Set the install location on /home somewhere, like 5) /home/anaconda3
  6) Add to bashrc when prompted.
  7) Exit current user and relog or source bashrc
  8) run "python -V" and you should see 3.6.1 (latest version)

  Anaconda is BSD licensed which gives you permission to use Anaconda commercially and for redistribution.

  Related article:
  https://www.continuum.io/downloads
  <BR>

  These are three known methods I've selected if you need optional Python versions on your server installed. Of course, you can compile, but who needs that when you can use different repositories to complete the task.

html:
  <p>Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together. Python's simple, easy to learn syntax emphasises readability and therefore reduces the cost of program maintenance. Python supports modules and packages, which encourages program modularity and code reuse. The Python interpreter and the extensive standard library are available in source or binary form without charge for all the key platforms and can be freely distributed.</p>

  <p>By default on CentOS or any other popular Linux distribution Python, 2.6 version is installed.</p>

  <p>You can check the current version with the following command:</p>

  <pre><code>~]# python -V
  Python 2.6.6  
  </code></pre>

  <p><strong>Install optional Python version from IUS Community repository</strong>
  <BR> <br />
  To install Python 2.7 on the server, please do the following:</p>

  <p>Log in as root (superuser) and push:  </p>

  <pre><code>cd /root ; wget -c https://centos6.iuscommunity.org/ius-release.rpm ; rpm -ivh ius-release.rpm  
  </code></pre>

  <p>On CentOS 7:  </p>

  <pre><code>cd /root ; wget -c https://centos7.iuscommunity.org/ius-release.rpm ; rpm -ivh ius-release.rpm  
  </code></pre>

  <p>After IUS Community repositories are successfully installed (<a href="https://ius.io/GettingStarted/">https://ius.io/GettingStarted/</a>) you need to run the following commands: <br />
  yum -y install python27 python27-devel python27-pip python27-setuptools python27-virtualenv --enablerepo=ius</p>

  <p>After packages get successfully installed, you can test if python2.7 and pip are working as expected, for example:  </p>

  <pre><code>[root@second ~]# python2.7 -V
  Python 2.7.13

  [root@second ~]# pip2.7
  Usage:  
  pip &lt;command&gt; [options]

  [root@second ~]# virtualenv-2.7
  You must provide a DEST_DIR  
  Usage: virtualenv-2.7 [OPTIONS] DEST_DIR  
  Options:  
  --version show program's version number and exit
  -h, --help show this help message and exit
  -v, --verbose Increase verbosity.
  -q, --quiet Decrease verbosity.
  -p PYTHON_EXE, --python=PYTHON_EXE
  </code></pre>

  <p>You may create an alias within the username by adjusting .bashrc file for python and virtualenv with the optional Python 2.7 version, for example: <br />
  alias python='/usr/bin/python2.7' <br />
  alias virtuelenv='/usr/bin/virtualenv-2.7'</p>

  <p>Repeat procedure In case you wish optional Python3+ version installed.</p>

  <p>Related articles <br />
  <a href="https://ius.io/GettingStarted/">https://ius.io/GettingStarted/</a> <br />
  <a href="https://community.letsencrypt.org/t/redhat-centos-6-x-users-need-python-2-7/2190">https://community.letsencrypt.org/t/redhat-centos-6-x-users-need-python-2-7/2190</a></p>

  <p><BR> <br />
  <strong>Install optional Python version from Software Collections (SCL) Repository</strong>
  <BR></p>

  <p>Software Collections give you the power to build, install, and use multiple versions of software on the same system, without affecting system-wide installed packages.</p>

  <p>To install Python 2.7 on CentOS server, please do the following:</p>

  <p>Install The Software Collections ( SCL ) Repository:  </p>

  <pre><code>yum install centos-release-scl  
  </code></pre>

  <p>Install python27 from SCL repository:  </p>

  <pre><code>yum install python27 -y  
  </code></pre>

  <p>Enable python27 with the following command:  </p>

  <pre><code>scl enable python27 bash  
  </code></pre>

  <p>Test active Python version:  </p>

  <pre><code>python -V  
  </code></pre>

  <p>The System should return "Python 2.7.X" Where X is the latest version of python in this branch.</p>

  <p>Related article: <br />
  <a href="https://www.softwarecollections.org/en/">https://www.softwarecollections.org/en/</a></p>

  <p><BR> <br />
  <strong>Install optional Python version using Anaconda</strong>
  <BR></p>

  <p>With Anaconda you can run multiple versions of Python in isolated environments, so choose the download with the Python version that you use more often, as that will be your default Python version.</p>

  <p>If you don't have the time or disk space for the entire distribution, try Miniconda (<a href="http://conda.pydata.org/miniconda.html">http://conda.pydata.org/miniconda.html</a>) which contains only conda and Python. Then install just the individual packages you want through the conda command.</p>

  <p>1) Linux installer at <a href="https://www.continuum.io/downloads">https://www.continuum.io/downloads</a> <br />
  2) Run the provided bash command from above to start the installer:  </p>

  <pre><code>bash Anaconda3-4.4.0-Linux-x86_64.sh  
  </code></pre>

  <p>For Python 2.7 version:  </p>

  <pre><code>bash Anaconda2-4.4.0-Linux-x86_64.sh  
  </code></pre>

  <p>3) Agree to the license agreement <br />
  4) Set the install location on /home somewhere, like 5) /home/anaconda3 <br />
  6) Add to bashrc when prompted. <br />
  7) Exit current user and relog or source bashrc <br />
  8) run "python -V" and you should see 3.6.1 (latest version)</p>

  <p>Anaconda is BSD licensed which gives you permission to use Anaconda commercially and for redistribution.</p>

  <p>Related article: <br />
  <a href="https://www.continuum.io/downloads">https://www.continuum.io/downloads</a> <br />
  <BR></p>

  <p>These are three known methods I've selected if you need optional Python versions on your server installed. Of course, you can compile, but who needs that when you can use different repositories to complete the task.</p>

meta_description:
  These are three known methods I've selected if you need optional Python versions on your server installed.


uuid: 3d05fc8a-dafa-4863-8284-37a6ff2040b2
title: How To Install Latest Nodejs and NPM on CentOS/RHEL 7/6
slug: how-to-install-latest-nodejs-and-npm-on-centos-rhel-7-6
image: /content/images/2017/07/nodejs-large-image.png
status: published
language: en_US
markdown:
  Node.js is a free open source platform built on Chrome's JavaScript runtime for easily building fast and scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices. More about NodeJS project you can find on their website here: https://nodejs.org/en/

  Today I am going to explain here briefly how to install latest NodeJS server with some packages (Grunt - http://gruntjs.com/installing-grunt).

  Step 1 – Add Node.js Yum Repository
  First, we will add a node.js yum repository in our system provided by the NodeJS official website. You also need development tools to build native addons to be installed on your system as described here https://nodejs.org/en/download/package-manager/

  Since we base on CentOS 6 and 7 version we will push following commands:

  Run as root on RHEL, CentOS or Fedora, for Node.js v6 LTS:
  ```
  curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -
  ```
  Alternatively for Node.js 8:
  ```
  curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -
  ```
  Alternatively for Node.js 0.10:
  ```
  curl --silent --location https://rpm.nodesource.com/setup | bash -
  ```

  Then install, as root:
  ```
  yum -y install nodejs
  ```
  Optional: install build tools

  To compile and install native add-ons from npm you may also need to install build tools:
  ```
  yum install gcc-c++ make
  ```
  or: yum groupinstall 'Development Tools' (which you can push on dev env server)

  Step 2 – Check Node.js and NPM Version
  After installing node.js application/server, verify and check the installed version. You can find more details about current version on the node.js official website.
  ```
  node -v
  ```
  Results is going to be displayed as:
  [root@second ~]# node -v <BR>
  v6.10.3

  Also, check the version of npm.
  ```
  npm -v
  ```
  [root@second ~]# npm -v <BR>
  3.10.10

  Step 3 – Create Demo Web Server (Optional)

  This is an optional step. If you want to test your node.js install. Let's create a web server with “Welcome Node.js” text. Create a file demo_server.js

  $ vim demo_server.js
  and add following content
  ```
  var http = require('http');
  http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Welcome Node.js');
  }).listen(3001, "<ENTER YOUR SERVER PUBLIC IP HERE>");
  console.log('Server running at http://<PUBLIC IP ADDRESS:3001/');
  ```
  Now start the web server using below command.
  ```
  node --debug demo_server.js
  ```
  debugger listening on port 5858
  Server running at http://PUBLIC-IP-ADDRESS:3001/

  The web server has been started on port 3001. Now access http://<WEB SERVER>:3001/ url in your browser. 

  Note: Make sure that port 3001 or whatever you wish to use is open on the server iptables.

  You may also require packages. Much like ruby or python, these are easy to install such as the following examples:
  ```
  npm install grunt
  ```
  More about grunt and how to install it you can read here:
  http://gruntjs.com/installing-grunt
  ```
  npm install grunt-cli
  ```
  More about grunt-cli and how to install it you can read here:
  http://gruntjs.com/using-the-cli

  After you install Node.js, you are ready to run Node.js applications. However, the exact steps to do this vary depending on the application configuration.

  Method #1: Use npm

  Many third-party and “production-ready” applications (such as Ghost) use the npm program to start the application, as shown by the following command:

  nohup npm start --production &
  The & places the command in the background, and the nohup command ensures that the application continues running even if you log out of the current terminal session.

  For this method to work, there must be a valid package.json file for the application. The package.json file contains project metadata that the npm program reads to determine how to start the application, manage its dependencies, and more.

  To view the official npm documentation for the package.json file, please visit https://docs.npmjs.com/files/package.json.

  Method #2: Run node directly

  For simple applications, or for any application that does not have a package.json file, you can run the node executable directly and specify the application filename. For example:

  nohup node my_app.js &
  However, you lose the benefits of using npm to manage the application.

  As above, the & places the command in the background, and the nohup command ensures that the application continues running even if you log out of the current terminal session.

  **STOPPING A NODE.JS APPLICATION**

  To stop a currently running Node.js application, type the following command:

  pkill node
  This command immediately stops all running Node.js applications.

  **INTEGRATING A NODE.JS APPLICATION WITH THE WEB SERVER**

  Depending on the type of Node.js application you are running, you may want to be able to access it using a web browser. If you are using Apache and even better cPanel/WHM server you can adjust .htaccess file. To do this, you need to select an unused port for the Node.js application to listen on, and then define server rewrite rules that redirect visitors to the application. The following steps demonstrate how to do this:

  In a text editor, add the following lines to the .htaccess file in the /home/username/public_html directory, where username represents your account username:
  ```
  RewriteEngine On
  RewriteRule ^$ http://127.0.0.1:XXXXX/ [P,L]
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteRule ^(.*)$ http://127.0.0.1:XXXXX/$1 [P,L]
  ```
  In both RewriteRule lines, replace XXXXX with the port on which your Node.js application listens.

  To run a Node.js application on a managed server, you must select an unused port, and the port number must be between 49152 and 65535 (inclusive).

  Save the changes to the .htaccess file, and then exit the text editor. Visitors to your web site are redirected to the Node.js application listening on the specified port.

  If your application fails to start, the port you chose may already be in use. Check the application log for error codes like EADDRINUSE that indicate the port is in use. If it is, select a different port number, update your application's configuration and the .htaccess file, and then try again.

  **MORE INFORMATION** <BR>
  For more information about Node.js, please visit http://nodejs.org.

html:
  <p>Node.js is a free open source platform built on Chrome's JavaScript runtime for easily building fast and scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices. More about NodeJS project you can find on their website here: <a href="https://nodejs.org/en/">https://nodejs.org/en/</a></p>

  <p>Today I am going to explain here briefly how to install latest NodeJS server with some packages (Grunt - <a href="http://gruntjs.com/installing-grunt">http://gruntjs.com/installing-grunt</a>).</p>

  <p>Step 1 – Add Node.js Yum Repository <br />
  First, we will add a node.js yum repository in our system provided by the NodeJS official website. You also need development tools to build native addons to be installed on your system as described here <a href="https://nodejs.org/en/download/package-manager/">https://nodejs.org/en/download/package-manager/</a></p>

  <p>Since we base on CentOS 6 and 7 version we will push following commands:</p>

  <p>Run as root on RHEL, CentOS or Fedora, for Node.js v6 LTS:  </p>

  <pre><code>curl --silent --location https://rpm.nodesource.com/setup_6.x | bash -  
  </code></pre>

  <p>Alternatively for Node.js 8:  </p>

  <pre><code>curl --silent --location https://rpm.nodesource.com/setup_8.x | bash -  
  </code></pre>

  <p>Alternatively for Node.js 0.10:  </p>

  <pre><code>curl --silent --location https://rpm.nodesource.com/setup | bash -  
  </code></pre>

  <p>Then install, as root:  </p>

  <pre><code>yum -y install nodejs  
  </code></pre>

  <p>Optional: install build tools</p>

  <p>To compile and install native add-ons from npm you may also need to install build tools:  </p>

  <pre><code>yum install gcc-c++ make  
  </code></pre>

  <p>or: yum groupinstall 'Development Tools' (which you can push on dev env server)</p>

  <p>Step 2 – Check Node.js and NPM Version <br />
  After installing node.js application/server, verify and check the installed version. You can find more details about current version on the node.js official website.  </p>

  <pre><code>node -v  
  </code></pre>

  <p>Results is going to be displayed as: <br />
  [root@second ~]# node -v <BR>
  v6.10.3</p>

  <p>Also, check the version of npm.  </p>

  <pre><code>npm -v  
  </code></pre>

  <p>[root@second ~]# npm -v <BR>
  3.10.10</p>

  <p>Step 3 – Create Demo Web Server (Optional)</p>

  <p>This is an optional step. If you want to test your node.js install. Let's create a web server with “Welcome Node.js” text. Create a file demo_server.js</p>

  <p>$ vim demo_server.js
  and add following content  </p>

  <pre><code>var http = require('http');  
  http.createServer(function (req, res) {  
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Welcome Node.js');
  }).listen(3001, "&lt;ENTER YOUR SERVER PUBLIC IP HERE&gt;");
  console.log('Server running at http://&lt;PUBLIC IP ADDRESS:3001/');  
  </code></pre>

  <p>Now start the web server using below command.  </p>

  <pre><code>node --debug demo_server.js  
  </code></pre>

  <p>debugger listening on port 5858 <br />
  Server running at <a href="http://PUBLIC-IP-ADDRESS:3001/">http://PUBLIC-IP-ADDRESS:3001/</a></p>

  <p>The web server has been started on port 3001. Now access http://<WEB SERVER>:3001/ url in your browser. </p>

  <p>Note: Make sure that port 3001 or whatever you wish to use is open on the server iptables.</p>

  <p>You may also require packages. Much like ruby or python, these are easy to install such as the following examples:  </p>

  <pre><code>npm install grunt  
  </code></pre>

  <p>More about grunt and how to install it you can read here: <br />
  <a href="http://gruntjs.com/installing-grunt">http://gruntjs.com/installing-grunt</a>  </p>

  <pre><code>npm install grunt-cli  
  </code></pre>

  <p>More about grunt-cli and how to install it you can read here: <br />
  <a href="http://gruntjs.com/using-the-cli">http://gruntjs.com/using-the-cli</a></p>

  <p>After you install Node.js, you are ready to run Node.js applications. However, the exact steps to do this vary depending on the application configuration.</p>

  <p>Method #1: Use npm</p>

  <p>Many third-party and “production-ready” applications (such as Ghost) use the npm program to start the application, as shown by the following command:</p>

  <p>nohup npm start --production &amp; <br />
  The &amp; places the command in the background, and the nohup command ensures that the application continues running even if you log out of the current terminal session.</p>

  <p>For this method to work, there must be a valid package.json file for the application. The package.json file contains project metadata that the npm program reads to determine how to start the application, manage its dependencies, and more.</p>

  <p>To view the official npm documentation for the package.json file, please visit <a href="https://docs.npmjs.com/files/package.json">https://docs.npmjs.com/files/package.json</a>.</p>

  <p>Method #2: Run node directly</p>

  <p>For simple applications, or for any application that does not have a package.json file, you can run the node executable directly and specify the application filename. For example:</p>

  <p>nohup node my_app.js &amp; <br />
  However, you lose the benefits of using npm to manage the application.</p>

  <p>As above, the &amp; places the command in the background, and the nohup command ensures that the application continues running even if you log out of the current terminal session.</p>

  <p><strong>STOPPING A NODE.JS APPLICATION</strong></p>

  <p>To stop a currently running Node.js application, type the following command:</p>

  <p>pkill node <br />
  This command immediately stops all running Node.js applications.</p>

  <p><strong>INTEGRATING A NODE.JS APPLICATION WITH THE WEB SERVER</strong></p>

  <p>Depending on the type of Node.js application you are running, you may want to be able to access it using a web browser. If you are using Apache and even better cPanel/WHM server you can adjust .htaccess file. To do this, you need to select an unused port for the Node.js application to listen on, and then define server rewrite rules that redirect visitors to the application. The following steps demonstrate how to do this:</p>

  <p>In a text editor, add the following lines to the .htaccess file in the /home/username/public_html directory, where username represents your account username:  </p>

  <pre><code>RewriteEngine On  
  RewriteRule ^$ http://127.0.0.1:XXXXX/ [P,L]  
  RewriteCond %{REQUEST_FILENAME} !-f  
  RewriteCond %{REQUEST_FILENAME} !-d  
  RewriteRule ^(.*)$ http://127.0.0.1:XXXXX/$1 [P,L]  
  </code></pre>

  <p>In both RewriteRule lines, replace XXXXX with the port on which your Node.js application listens.</p>

  <p>To run a Node.js application on a managed server, you must select an unused port, and the port number must be between 49152 and 65535 (inclusive).</p>

  <p>Save the changes to the .htaccess file, and then exit the text editor. Visitors to your web site are redirected to the Node.js application listening on the specified port.</p>

  <p>If your application fails to start, the port you chose may already be in use. Check the application log for error codes like EADDRINUSE that indicate the port is in use. If it is, select a different port number, update your application's configuration and the .htaccess file, and then try again.</p>

  <p><strong>MORE INFORMATION</strong> <BR>
  For more information about Node.js, please visit <a href="http://nodejs.org">http://nodejs.org</a>.</p>


uuid: 39cf8e29-2795-4e57-a32b-6d0f1c238625
title: How to install and configure -- The Reliable, High Performance TCP/HTTP Load Balancer - HA Proxy
slug: the-reliable-high-performance-tcp-http-load-balancer-ha-proxy
image: /content/images/2017/09/haproxy-2.png
status: published
language: en_US
markdown:
  Whether you are researching load
  balancer's for your own needs or the needs of your employer, you will definitely come across HAProxy. As such, you may be asking what it is and how it can benefit you or your company.

  When researching load balancers, you will find your options usually fall into one of two categories; Hardware based vs. Software Based. In the hardware realm, you will find options such as F5’s BIG-IP, Citrix NetScaler and Kemp Technologies that offer dedicated appliances that are running proprietary software. 

  On the opposite side of that, you have software based solutions where you are able to use commodity hardware that fits your needs independent of the load balancing software being used. In this realm,
  you will find solutions such as NGINX and HAProxy with the latter being the focus of this guide. 

  The goal of the team behind HAProxy is to provide a “free, very fast and reliable solution” for load-balancing TCP and HTTP-based applications. Because of this HAProxy is considered by many to be the de facto standard when it comes to software-based load balancing and is currently being used by sites such as GitHub, Reddit, Twitter and Tumblr to name a few.
  It has been designed to run on Linux, Solaris, FreeBSD, OpenBSD as well as AIX platforms. While it’s designed to run on most x86-64 hardware that has limited resources, it will perform best when provided enterprise-grade hardware such as 10+ Gig NIC’s and Xeon class CPU’s or similar.

  Installing HAProxy on CentOS 7. As a fast developing open source application HAProxy available for install in the CentOS default repositories might not be the latest release. To find out what version number is being offered through the official channels enter the following command.

  ```
  sudo yum info haproxy
  ```
  The output should look like this:
   
  ![alt](/content/images/2017/09/pic1.png)

  HAProxy has always three active stable versions of the releases, two of the latest versions in development plus a third older version that is still receiving critical updates. You can always check the currently newest stable version listed on the HAProxy website and then decide which version you wish to go with.

  In this guide, we will be installing the currently latest stable version of 1.7 (2017/08/18), which was not yet available in the standard repositories. Instead, you will need to install it from the source. But first, check that you have the prerequisites to download and compile the program.

  ```
  sudo yum install gcc pcre-static pcre-devel -y
  ```

  NOTE: It might happen that kernel-headers are missing. By default, the CentOS 7 cloud servers have kernel-headers disabled, so you'll need to install them manually by:
  ```
  sudo yum install -y kernel-headers --disableexcludes=all
  ```

  Download the source code with the command below. You can check if there is a newer version available at the HAProxy http://www.haproxy.org/#down

  ```
  wget http://www.haproxy.org/download/1.7/src/haproxy-1.7.9.tar.gz -O ~/haproxy.tar.gz
  ```

  Once the download is complete, extract the files using the command below.
  ```
  tar -xvf haproxy.tar.gz
  ```
  Change into the extracted source directory.
  ```
  cd haproxy-1.7.9
  ```
  Then compile the program for your system.
  ```
  make TARGET=linux2628
  ```
  And finally, install HAProxy itself.
  ```
  sudo make install
  ```
  The output should look like this:
  ![alt](/content/images/2017/09/pic2.png)

  With that done, HAProxy is now installed but requires some additional steps to get it operational. Continue below with setting up the software and services.

  **Setting up HAProxy for your server**

  Now, since we have choosen the "hard" way we need to add the following directories and the statistics file for HAProxy records.
  ```
  sudo mkdir -p /etc/haproxy
  sudo mkdir -p /var/lib/haproxy 
  sudo touch /var/lib/haproxy/stats
  ```

  Next, we will create a symbolic link for the binary to allow you to run HAProxy commands as a normal user.
  ```
  sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy
  ```
  If you want to add the proxy as a service to the system, copy the haproxy.init file from the examples to your /etc/init.d directory. Change the file permissions to make the script executable and then reload the systemd daemon.
  ```
  sudo cp ~/haproxy-1.7.9/examples/haproxy.init /etc/init.d/haproxy
  sudo chmod 755 /etc/init.d/haproxy
  sudo systemctl daemon-reload
  ```
  For general usage, it is also recommended to add a new user for HAProxy to be run under and assign password for it.
  ```
  sudo useradd -r haproxy
  passwd haproxy
  ```

  You can double check the installed version number with the following command.
  ```
  haproxy -v
  ```
  The output needs to look like this:
  ![alt](/content/images/2017/09/pic3.png)

  Lastly, the firewall on CentOS 7 is quite restrictive for this project by default. Use the following commands to allow the required services and reload the firewall. Whatever type of firewall you are using (built-in), iptables, csf/apf please open port 8181 TCP. We will use firewall-cmd now:
  ```
  sudo firewall-cmd --permanent --zone=public --add-service=http
  sudo firewall-cmd --permanent --zone=public --add-port=8181/tcp
  sudo firewall-cmd --reload
  ```
  Note: If you don't have firewalld installed you can install it with yum install firewalld then service firewalld restart and proceed with the commands above. Otherwise, you can use iptables as I said before.

  **Configuring the load balancer**
  How to Install HAProxy Load Balancer on CentOS
  /Tutorials /How to Install HAProxy Load Balancer on CentOS
   Tutorials
  Get started
  Try this guide out on UpCloud with our free trial!
  Load balancing is a common solution for distributing web applications horizontally across multiple hosts while providing the users with a single point of access to the service. HAProxy is one of the most popular open source load balancing software, which also offers high availability and proxy functionality.

  HAProxy aims to optimise resource usage, maximise throughput, minimise response time, and avoid overloading any single resource. It is available for install on many Linux distributions like CentOS 7 in this guide, but also on Debian 8 and Ubuntu 16 systems.



  HAProxy is particularly suited for very high traffic websites and is therefore often used to improve web service reliability and performance for multi-server configurations. This guide lays out the steps for setting up HAProxy as a load balancer on CentOS 7 to its own cloud host which then directs the traffic to your web servers.

  As a pre-requirement for the best results, you should have a minimum of two web servers and a server for the load balancer. The web servers need to be running at least the basic web service such as nginx or httpd to test out the load balancing between them.

  Contents

  Installing HAProxy CentOS 7

  Setting up HAProxy for your server

  Configuring the load balancer

  Load balancing at layer 4

  Different load balancing algorithms

  Configuring load balancing for layer 7

  Testing the setup

  Password protecting the statistics page

  Installing HAProxy CentOS 7

  As a fast developing open source application HAProxy available for install in the CentOS default repositories might not be the latest release. To find out what version number is being offered through the official channels enter the following command.

  sudo yum info haproxy
  HAProxy has always three active stable versions of the releases, two of the latest versions in development plus a third older version that is still receiving critical updates. You can always check the currently newest stable version listed on the HAProxy website and then decide which version you wish to go with.

  In this guide, we will be installing the currently latest stable version of 1.7, which was not yet available in the standard repositories. Instead, you will need to install it from the source. But first, check that you have the prerequisites to download and compile the program.

  sudo yum install gcc pcre-static pcre-devel -y
  Download the source code with the command below. You can check if there is a newer version available at the HAProxy download page.

  wget https://www.haproxy.org/download/1.7/src/haproxy-1.7.8.tar.gz -O ~/haproxy.tar.gz
  Once the download is complete, extract the files using the command below.

  tar xzvf ~/haproxy.tar.gz -C ~/
  Change into the extracted source directory.

  cd ~/haproxy-1.7.8
  Then compile the program for your system.

  make TARGET=linux2628
  And finally, install HAProxy itself.

  sudo make install
  With that done, HAProxy is now installed but requires some additional steps to get it operational. Continue below with setting up the software and services.

  Setting up HAProxy for your server

  Next, add the following directories and the statistics file for HAProxy records.

  sudo mkdir -p /etc/haproxy
  sudo mkdir -p /var/lib/haproxy 
  sudo touch /var/lib/haproxy/stats
  Create a symbolic link for the binary to allow you to run HAProxy commands as a normal user.

  sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy
  If you want to add the proxy as a service to the system, copy the haproxy.init file from the examples to your /etc/init.d directory. Change the file permissions to make the script executable and then reload the systemd daemon.

  sudo cp ~/haproxy-1.7.8/examples/haproxy.init /etc/init.d/haproxy
  sudo chmod 755 /etc/init.d/haproxy
  sudo systemctl daemon-reload
  For general usage, it is also recommended to add a new user for HAProxy to be run under.

  sudo useradd -r haproxy
  Afterwards, you can double check the installed version number with the following command.

  haproxy -v
  HA-Proxy version 1.7.8 2017/07/07
  Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>
  In this case, the version should be 1.7.8 like shown in the example output above.

  Lastly, the firewall on CentOS 7 is quite restrictive for this project by default. Use the following commands to allow the required services and reload the firewall.

  sudo firewall-cmd --permanent --zone=public --add-service=http
  sudo firewall-cmd --permanent --zone=public --add-port=8181/tcp
  sudo firewall-cmd --reload
  Configuring the load balancer

  Setting up HAProxy for load balancing is a quite straight forward process. Basically, all you need to do is tell HAProxy what kind of connections it should be listening for and where the connections should be relayed to.

  This is done by creating a configuration file /etc/haproxy/haproxy.cfg with the defining settings. You can read about the configuration options at HAProxy https://cbonte.github.io/haproxy-dconv/1.7/configuration.html in case you are interested to find out more.

  1) Load balancing at layer 4
  We can now start with the basic setup and configure haproxy.cfg file with the popular editor nano.
  ```
  nano /etc/haproxy/haproxy.cfg
  ```

  Add the following sections to the file. Replace the <server name> with what ever you want to call you servers on the statistics page and the <private/public IP> with the private IPs for the servers you wish to direct the web traffic to. You may network your servers using VPN, PrivateLAN or whatever datacenter provides:
  ```
  global
     log /dev/log local0
     log /dev/log local1 notice
     chroot /var/lib/haproxy
     stats timeout 30s
     user haproxy
     group haproxy
     daemon

  defaults
     log global
     mode http
     option httplog
     option dontlognull
     timeout connect 5000
     timeout client 50000
     timeout server 50000

  frontend http_front
     bind *:80
     stats uri /haproxy?stats
     default_backend http_back

  backend http_back
     balance roundrobin
     server <server name> <private IP>:80 check
     server <server name> <private IP>:80 check
  ```
  In my particular case, the haproxy.cfg file is looking like this:
  ![alt](/content/images/2017/09/pic4.png)

  Where I will route port 80 traffic to two other servers that I have already installed. This defines a layer 4 load balancer with a front-end name http_front listening to the port number 80, which then directs the traffic to the default backend named http_back. The additional stats URI /haproxy?stats enables the statistics page at that specified address.

  **Different load balancing algorithms**

  Configuring the servers in the backend section allows HAProxy to use these servers for load balancing according to the roundrobin algorithm whenever available.

  The balancing algorithms are used to decide which server at the backend each connection is transferred to. Some of the useful options include the following:

  - Roundrobin: Each server is used in turns according to their weights. This is the smoothest and fairest algorithm when the servers’ processing time remains equally distributed. This algorithm is dynamic, which allows server weights to be adjusted on the fly.

  - Leastconn: The server with the lowest number of connections is chosen. Round-robin is performed between servers with the same load. Using this algorithm is recommended with long sessions, such as LDAP, SQL, TSE, etc, but it is not very well suited for short sessions such as HTTP.

  - First: The first server with available connection slots receives the connection. The servers are chosen from the lowest numeric identifier to the highest, which defaults to the server’s position on the farm. Once a server reaches its maxconn value, the next server is used.

  - Source: The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request. This way the same client IP address will always reach the same server while the servers stay the same.

  In this article, I will do my best to explain the terminology and a few of the key terms and concepts you should understand when working with HAProxy. When working with load balancers, these are the key concepts that will apply to all solutions.

  2) Configuring load balancing for layer 7

  Another possibility is to configure the load balancer to work on layer 7, which is useful when parts of your web application are located on different hosts. This can be accomplished by conditioning the connection transfer for example by the URL. Definition - What does Layer 7 mean?

  Layer 7 refers to the seventh and topmost layer of the Open Systems Interconnect (OSI) Model known as the application layer. This is the highest layer which supports end-user processes and applications. Layer 7 identifies the communicating parties and the quality of service between them, considers privacy and user authentication, as well as identifies any constraints on the data syntax. This layer is wholly application-specific.

  Open the HAProxy configuration file with a text editor, nano in our case and:
  ```
  sudo nano /etc/haproxy/haproxy.cfg
  ```

  Then set the front and backend segments according to the example below. Once again change <server name> fields and <private-IP> with your real settings:
  ```
  frontend http_front
     bind *:80
     stats uri /haproxy?stats
     acl url_blog path_beg /blog
     use_backend blog_back if url_blog
     default_backend http_back

  backend http_back
     balance roundrobin
     server <server name> <private IP>:80 check
     server <server name> <private IP>:80 check

  backend blog_back
     server <server name> <private IP>:80 check
  ```

  Explanation of each term. The front end declares an ACL rule named url_blog that applies to all connections with paths that begin with /blog. Use_backend defines that connections matching the url_blog condition should be served by the backend named blog_back, while all other requests are handled by the default backend.

  At the backend side, the configuration sets up two server groups, http_back like before and the new one called blog_back that servers specifically connections to example.com/blog. This is very useful if you plan to divide Backend with Frontend like with Magento for example where there are many concurrent backend users doing a lot of changes.

  After making the configurations, save the file and restart HAProxy with the next command.
  ```
  sudo systemctl restart haproxy
  ```
  The output should look like this:

  ![alt](/content/images/2017/09/pic5.png)

  If you get any errors or warnings at start up, check the configuration for any mistypes and that you have created all the necessary files and folders, then try restarting again.

  **TESTING** <BR>
  Now we can test the setup and make sure this is working as expected. With the HAProxy configured and running, open your load balancer server’s public IP in a web browser and check that you get connected to your backend correctly. The parameter stats URI in the configuration enables the statistics page at the defined address.
  ```
  http://<load balancer public IP>/haproxy?stats
  ```

  When you load the statistics page and all of your servers are listed in green your configuration was successful!
  ![alt](/content/images/2017/09/pic6.png)

  In my case that is http://54.229.71.221/haproxy?stats and now when I visit just http://54.229.71.221 URL I can see the Nginx default page while pressing F5 or CTRL-R clearly HA Proxy is doing its job honoring "roundrobin" algorithm.
  ![alt](/content/images/2017/09/pic7.png)

  The statistics page contains some helpful information to keep track of your web hosts including up and down times and session counts. If a server is listed in red, check that the server is powered on and that you can ping it from the load balancer machine.

  In case your load balancer does not reply, check that HTTP connections are not getting blocked by the firewall. Also, confirm that HAProxy is running with the command below.
  ```
  sudo systemctl status haproxy
  ```

  **Password protecting the statistics page**
  Having the statistics page simply listed at the front end, however, is publicly open for anyone to view, which might not be such a good idea. Instead, you can set it up to its own port number by adding the example below to the end of your haproxy.cfg file. Replace the 'nemanja' and 'password' with something secure.
  ```
  listen stats
     bind *:8181
     stats enable
     stats uri /
     stats realm Haproxy\ Statistics
     stats auth username:password
  ```

  After adding the new listen group, remove the old reference to the stats uri from the frontend group. When done, save the file and restart HAProxy again.
  ```
  sudo systemctl restart haproxy
  ```
  Then open the load balancer again with the new port number, and log in with the username and password you set in the configuration file.

  http://load balancer ip-public:8181
  Check that your servers are still reporting all green and then open just the load balancer IP without any port numbers on your web browser.

  http://loadbalancer-ip-public/ OR use domain name instead.

  If your backend servers have at least slightly different landing pages you will notice that each time you reload the page you get the reply from a different host. You can try out different balancing algorithms in the configuration section or take a look at the full documentation page here https://cbonte.github.io/haproxy-dconv/

html:
  <p>Whether you are researching load <br />
  balancer's for your own needs or the needs of your employer, you will definitely come across HAProxy. As such, you may be asking what it is and how it can benefit you or your company.</p>

  <p>When researching load balancers, you will find your options usually fall into one of two categories; Hardware based vs. Software Based. In the hardware realm, you will find options such as F5’s BIG-IP, Citrix NetScaler and Kemp Technologies that offer dedicated appliances that are running proprietary software. </p>

  <p>On the opposite side of that, you have software based solutions where you are able to use commodity hardware that fits your needs independent of the load balancing software being used. In this realm, <br />
  you will find solutions such as NGINX and HAProxy with the latter being the focus of this guide. </p>

  <p>The goal of the team behind HAProxy is to provide a “free, very fast and reliable solution” for load-balancing TCP and HTTP-based applications. Because of this HAProxy is considered by many to be the de facto standard when it comes to software-based load balancing and is currently being used by sites such as GitHub, Reddit, Twitter and Tumblr to name a few. <br />
  It has been designed to run on Linux, Solaris, FreeBSD, OpenBSD as well as AIX platforms. While it’s designed to run on most x86-64 hardware that has limited resources, it will perform best when provided enterprise-grade hardware such as 10+ Gig NIC’s and Xeon class CPU’s or similar.</p>

  <p>Installing HAProxy on CentOS 7. As a fast developing open source application HAProxy available for install in the CentOS default repositories might not be the latest release. To find out what version number is being offered through the official channels enter the following command.</p>

  <pre><code>sudo yum info haproxy  
  </code></pre>

  <p>The output should look like this:</p>

  <p><img src="/content/images/2017/09/pic1.png" alt="alt" /></p>

  <p>HAProxy has always three active stable versions of the releases, two of the latest versions in development plus a third older version that is still receiving critical updates. You can always check the currently newest stable version listed on the HAProxy website and then decide which version you wish to go with.</p>

  <p>In this guide, we will be installing the currently latest stable version of 1.7 (2017/08/18), which was not yet available in the standard repositories. Instead, you will need to install it from the source. But first, check that you have the prerequisites to download and compile the program.</p>

  <pre><code>sudo yum install gcc pcre-static pcre-devel -y  
  </code></pre>

  <p>NOTE: It might happen that kernel-headers are missing. By default, the CentOS 7 cloud servers have kernel-headers disabled, so you'll need to install them manually by:  </p>

  <pre><code>sudo yum install -y kernel-headers --disableexcludes=all  
  </code></pre>

  <p>Download the source code with the command below. You can check if there is a newer version available at the HAProxy <a href="http://www.haproxy.org/#down">http://www.haproxy.org/#down</a></p>

  <pre><code>wget http://www.haproxy.org/download/1.7/src/haproxy-1.7.9.tar.gz -O ~/haproxy.tar.gz  
  </code></pre>

  <p>Once the download is complete, extract the files using the command below.  </p>

  <pre><code>tar -xvf haproxy.tar.gz  
  </code></pre>

  <p>Change into the extracted source directory.  </p>

  <pre><code>cd haproxy-1.7.9  
  </code></pre>

  <p>Then compile the program for your system.  </p>

  <pre><code>make TARGET=linux2628  
  </code></pre>

  <p>And finally, install HAProxy itself.  </p>

  <pre><code>sudo make install  
  </code></pre>

  <p>The output should look like this: <br />
  <img src="/content/images/2017/09/pic2.png" alt="alt" /></p>

  <p>With that done, HAProxy is now installed but requires some additional steps to get it operational. Continue below with setting up the software and services.</p>

  <p><strong>Setting up HAProxy for your server</strong></p>

  <p>Now, since we have choosen the "hard" way we need to add the following directories and the statistics file for HAProxy records.  </p>

  <pre><code>sudo mkdir -p /etc/haproxy  
  sudo mkdir -p /var/lib/haproxy  
  sudo touch /var/lib/haproxy/stats  
  </code></pre>

  <p>Next, we will create a symbolic link for the binary to allow you to run HAProxy commands as a normal user.  </p>

  <pre><code>sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy  
  </code></pre>

  <p>If you want to add the proxy as a service to the system, copy the haproxy.init file from the examples to your /etc/init.d directory. Change the file permissions to make the script executable and then reload the systemd daemon.  </p>

  <pre><code>sudo cp ~/haproxy-1.7.9/examples/haproxy.init /etc/init.d/haproxy  
  sudo chmod 755 /etc/init.d/haproxy  
  sudo systemctl daemon-reload  
  </code></pre>

  <p>For general usage, it is also recommended to add a new user for HAProxy to be run under and assign password for it.  </p>

  <pre><code>sudo useradd -r haproxy  
  passwd haproxy  
  </code></pre>

  <p>You can double check the installed version number with the following command.  </p>

  <pre><code>haproxy -v  
  </code></pre>

  <p>The output needs to look like this: <br />
  <img src="/content/images/2017/09/pic3.png" alt="alt" /></p>

  <p>Lastly, the firewall on CentOS 7 is quite restrictive for this project by default. Use the following commands to allow the required services and reload the firewall. Whatever type of firewall you are using (built-in), iptables, csf/apf please open port 8181 TCP. We will use firewall-cmd now:  </p>

  <pre><code>sudo firewall-cmd --permanent --zone=public --add-service=http  
  sudo firewall-cmd --permanent --zone=public --add-port=8181/tcp  
  sudo firewall-cmd --reload  
  </code></pre>

  <p>Note: If you don't have firewalld installed you can install it with yum install firewalld then service firewalld restart and proceed with the commands above. Otherwise, you can use iptables as I said before.</p>

  <p><strong>Configuring the load balancer</strong>
  How to Install HAProxy Load Balancer on CentOS <br />
  /Tutorials /How to Install HAProxy Load Balancer on CentOS
   Tutorials
  Get started <br />
  Try this guide out on UpCloud with our free trial! <br />
  Load balancing is a common solution for distributing web applications horizontally across multiple hosts while providing the users with a single point of access to the service. HAProxy is one of the most popular open source load balancing software, which also offers high availability and proxy functionality.</p>

  <p>HAProxy aims to optimise resource usage, maximise throughput, minimise response time, and avoid overloading any single resource. It is available for install on many Linux distributions like CentOS 7 in this guide, but also on Debian 8 and Ubuntu 16 systems.</p>

  <p>HAProxy is particularly suited for very high traffic websites and is therefore often used to improve web service reliability and performance for multi-server configurations. This guide lays out the steps for setting up HAProxy as a load balancer on CentOS 7 to its own cloud host which then directs the traffic to your web servers.</p>

  <p>As a pre-requirement for the best results, you should have a minimum of two web servers and a server for the load balancer. The web servers need to be running at least the basic web service such as nginx or httpd to test out the load balancing between them.</p>

  <p>Contents</p>

  <p>Installing HAProxy CentOS 7</p>

  <p>Setting up HAProxy for your server</p>

  <p>Configuring the load balancer</p>

  <p>Load balancing at layer 4</p>

  <p>Different load balancing algorithms</p>

  <p>Configuring load balancing for layer 7</p>

  <p>Testing the setup</p>

  <p>Password protecting the statistics page</p>

  <p>Installing HAProxy CentOS 7</p>

  <p>As a fast developing open source application HAProxy available for install in the CentOS default repositories might not be the latest release. To find out what version number is being offered through the official channels enter the following command.</p>

  <p>sudo yum info haproxy <br />
  HAProxy has always three active stable versions of the releases, two of the latest versions in development plus a third older version that is still receiving critical updates. You can always check the currently newest stable version listed on the HAProxy website and then decide which version you wish to go with.</p>

  <p>In this guide, we will be installing the currently latest stable version of 1.7, which was not yet available in the standard repositories. Instead, you will need to install it from the source. But first, check that you have the prerequisites to download and compile the program.</p>

  <p>sudo yum install gcc pcre-static pcre-devel -y <br />
  Download the source code with the command below. You can check if there is a newer version available at the HAProxy download page.</p>

  <p>wget <a href="https://www.haproxy.org/download/1.7/src/haproxy-1.7.8.tar.gz">https://www.haproxy.org/download/1.7/src/haproxy-1.7.8.tar.gz</a> -O ~/haproxy.tar.gz <br />
  Once the download is complete, extract the files using the command below.</p>

  <p>tar xzvf ~/haproxy.tar.gz -C ~/ <br />
  Change into the extracted source directory.</p>

  <p>cd ~/haproxy-1.7.8 <br />
  Then compile the program for your system.</p>

  <p>make TARGET=linux2628 <br />
  And finally, install HAProxy itself.</p>

  <p>sudo make install <br />
  With that done, HAProxy is now installed but requires some additional steps to get it operational. Continue below with setting up the software and services.</p>

  <p>Setting up HAProxy for your server</p>

  <p>Next, add the following directories and the statistics file for HAProxy records.</p>

  <p>sudo mkdir -p /etc/haproxy <br />
  sudo mkdir -p /var/lib/haproxy <br />
  sudo touch /var/lib/haproxy/stats <br />
  Create a symbolic link for the binary to allow you to run HAProxy commands as a normal user.</p>

  <p>sudo ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy <br />
  If you want to add the proxy as a service to the system, copy the haproxy.init file from the examples to your /etc/init.d directory. Change the file permissions to make the script executable and then reload the systemd daemon.</p>

  <p>sudo cp ~/haproxy-1.7.8/examples/haproxy.init /etc/init.d/haproxy <br />
  sudo chmod 755 /etc/init.d/haproxy <br />
  sudo systemctl daemon-reload <br />
  For general usage, it is also recommended to add a new user for HAProxy to be run under.</p>

  <p>sudo useradd -r haproxy <br />
  Afterwards, you can double check the installed version number with the following command.</p>

  <p>haproxy -v <br />
  HA-Proxy version 1.7.8 2017/07/07 <br />
  Copyright 2000-2017 Willy Tarreau <a href="m&#97;i&#108;&#x74;&#x6f;:&#x77;i&#x6c;&#x6c;&#121;&#x40;&#x68;a&#x70;r&#x6f;&#120;&#121;&#x2e;&#x6f;&#114;&#x67;">&#x77;i&#x6c;&#x6c;&#121;&#x40;&#x68;a&#x70;r&#x6f;&#120;&#121;&#x2e;&#x6f;&#114;&#x67;</a> <br />
  In this case, the version should be 1.7.8 like shown in the example output above.</p>

  <p>Lastly, the firewall on CentOS 7 is quite restrictive for this project by default. Use the following commands to allow the required services and reload the firewall.</p>

  <p>sudo firewall-cmd --permanent --zone=public --add-service=http <br />
  sudo firewall-cmd --permanent --zone=public --add-port=8181/tcp <br />
  sudo firewall-cmd --reload <br />
  Configuring the load balancer</p>

  <p>Setting up HAProxy for load balancing is a quite straight forward process. Basically, all you need to do is tell HAProxy what kind of connections it should be listening for and where the connections should be relayed to.</p>

  <p>This is done by creating a configuration file /etc/haproxy/haproxy.cfg with the defining settings. You can read about the configuration options at HAProxy <a href="https://cbonte.github.io/haproxy-dconv/1.7/configuration.html">https://cbonte.github.io/haproxy-dconv/1.7/configuration.html</a> in case you are interested to find out more.</p>

  <p>1) Load balancing at layer 4 <br />
  We can now start with the basic setup and configure haproxy.cfg file with the popular editor nano.  </p>

  <pre><code>nano /etc/haproxy/haproxy.cfg  
  </code></pre>

  <p>Add the following sections to the file. Replace the <server name> with what ever you want to call you servers on the statistics page and the <private/public IP> with the private IPs for the servers you wish to direct the web traffic to. You may network your servers using VPN, PrivateLAN or whatever datacenter provides:  </p>

  <pre><code>global  
     log /dev/log local0
     log /dev/log local1 notice
     chroot /var/lib/haproxy
     stats timeout 30s
     user haproxy
     group haproxy
     daemon

  defaults  
     log global
     mode http
     option httplog
     option dontlognull
     timeout connect 5000
     timeout client 50000
     timeout server 50000

  frontend http_front  
     bind *:80
     stats uri /haproxy?stats
     default_backend http_back

  backend http_back  
     balance roundrobin
     server &lt;server name&gt; &lt;private IP&gt;:80 check
     server &lt;server name&gt; &lt;private IP&gt;:80 check
  </code></pre>

  <p>In my particular case, the haproxy.cfg file is looking like this: <br />
  <img src="/content/images/2017/09/pic4.png" alt="alt" /></p>

  <p>Where I will route port 80 traffic to two other servers that I have already installed. This defines a layer 4 load balancer with a front-end name http<em>front listening to the port number 80, which then directs the traffic to the default backend named http</em>back. The additional stats URI /haproxy?stats enables the statistics page at that specified address.</p>

  <p><strong>Different load balancing algorithms</strong></p>

  <p>Configuring the servers in the backend section allows HAProxy to use these servers for load balancing according to the roundrobin algorithm whenever available.</p>

  <p>The balancing algorithms are used to decide which server at the backend each connection is transferred to. Some of the useful options include the following:</p>

  <ul>
  <li><p>Roundrobin: Each server is used in turns according to their weights. This is the smoothest and fairest algorithm when the servers’ processing time remains equally distributed. This algorithm is dynamic, which allows server weights to be adjusted on the fly.</p></li>
  <li><p>Leastconn: The server with the lowest number of connections is chosen. Round-robin is performed between servers with the same load. Using this algorithm is recommended with long sessions, such as LDAP, SQL, TSE, etc, but it is not very well suited for short sessions such as HTTP.</p></li>
  <li><p>First: The first server with available connection slots receives the connection. The servers are chosen from the lowest numeric identifier to the highest, which defaults to the server’s position on the farm. Once a server reaches its maxconn value, the next server is used.</p></li>
  <li><p>Source: The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request. This way the same client IP address will always reach the same server while the servers stay the same.</p></li>
  </ul>

  <p>In this article, I will do my best to explain the terminology and a few of the key terms and concepts you should understand when working with HAProxy. When working with load balancers, these are the key concepts that will apply to all solutions.</p>

  <p>2) Configuring load balancing for layer 7</p>

  <p>Another possibility is to configure the load balancer to work on layer 7, which is useful when parts of your web application are located on different hosts. This can be accomplished by conditioning the connection transfer for example by the URL. Definition - What does Layer 7 mean?</p>

  <p>Layer 7 refers to the seventh and topmost layer of the Open Systems Interconnect (OSI) Model known as the application layer. This is the highest layer which supports end-user processes and applications. Layer 7 identifies the communicating parties and the quality of service between them, considers privacy and user authentication, as well as identifies any constraints on the data syntax. This layer is wholly application-specific.</p>

  <p>Open the HAProxy configuration file with a text editor, nano in our case and:  </p>

  <pre><code>sudo nano /etc/haproxy/haproxy.cfg  
  </code></pre>

  <p>Then set the front and backend segments according to the example below. Once again change <server name> fields and <private-IP> with your real settings:  </p>

  <pre><code>frontend http_front  
     bind *:80
     stats uri /haproxy?stats
     acl url_blog path_beg /blog
     use_backend blog_back if url_blog
     default_backend http_back

  backend http_back  
     balance roundrobin
     server &lt;server name&gt; &lt;private IP&gt;:80 check
     server &lt;server name&gt; &lt;private IP&gt;:80 check

  backend blog_back  
     server &lt;server name&gt; &lt;private IP&gt;:80 check
  </code></pre>

  <p>Explanation of each term. The front end declares an ACL rule named url<em>blog that applies to all connections with paths that begin with /blog. Use</em>backend defines that connections matching the url<em>blog condition should be served by the backend named blog</em>back, while all other requests are handled by the default backend.</p>

  <p>At the backend side, the configuration sets up two server groups, http<em>back like before and the new one called blog</em>back that servers specifically connections to example.com/blog. This is very useful if you plan to divide Backend with Frontend like with Magento for example where there are many concurrent backend users doing a lot of changes.</p>

  <p>After making the configurations, save the file and restart HAProxy with the next command.  </p>

  <pre><code>sudo systemctl restart haproxy  
  </code></pre>

  <p>The output should look like this:</p>

  <p><img src="/content/images/2017/09/pic5.png" alt="alt" /></p>

  <p>If you get any errors or warnings at start up, check the configuration for any mistypes and that you have created all the necessary files and folders, then try restarting again.</p>

  <p><strong>TESTING</strong> <BR>
  Now we can test the setup and make sure this is working as expected. With the HAProxy configured and running, open your load balancer server’s public IP in a web browser and check that you get connected to your backend correctly. The parameter stats URI in the configuration enables the statistics page at the defined address.  </p>

  <pre><code>http://&lt;load balancer public IP&gt;/haproxy?stats  
  </code></pre>

  <p>When you load the statistics page and all of your servers are listed in green your configuration was successful! <br />
  <img src="/content/images/2017/09/pic6.png" alt="alt" /></p>

  <p>In my case that is <a href="http://54.229.71.221/haproxy?stats">http://54.229.71.221/haproxy?stats</a> and now when I visit just <a href="http://54.229.71.221">http://54.229.71.221</a> URL I can see the Nginx default page while pressing F5 or CTRL-R clearly HA Proxy is doing its job honoring "roundrobin" algorithm. <br />
  <img src="/content/images/2017/09/pic7.png" alt="alt" /></p>

  <p>The statistics page contains some helpful information to keep track of your web hosts including up and down times and session counts. If a server is listed in red, check that the server is powered on and that you can ping it from the load balancer machine.</p>

  <p>In case your load balancer does not reply, check that HTTP connections are not getting blocked by the firewall. Also, confirm that HAProxy is running with the command below.  </p>

  <pre><code>sudo systemctl status haproxy  
  </code></pre>

  <p><strong>Password protecting the statistics page</strong>
  Having the statistics page simply listed at the front end, however, is publicly open for anyone to view, which might not be such a good idea. Instead, you can set it up to its own port number by adding the example below to the end of your haproxy.cfg file. Replace the 'nemanja' and 'password' with something secure.  </p>

  <pre><code>listen stats  
     bind *:8181
     stats enable
     stats uri /
     stats realm Haproxy\ Statistics
     stats auth username:password
  </code></pre>

  <p>After adding the new listen group, remove the old reference to the stats uri from the frontend group. When done, save the file and restart HAProxy again.  </p>

  <pre><code>sudo systemctl restart haproxy  
  </code></pre>

  <p>Then open the load balancer again with the new port number, and log in with the username and password you set in the configuration file.</p>

  <p><a href="http://load">http://load</a> balancer ip-public:8181 <br />
  Check that your servers are still reporting all green and then open just the load balancer IP without any port numbers on your web browser.</p>

  <p><a href="http://loadbalancer-ip-public/">http://loadbalancer-ip-public/</a> OR use domain name instead.</p>

  <p>If your backend servers have at least slightly different landing pages you will notice that each time you reload the page you get the reply from a different host. You can try out different balancing algorithms in the configuration section or take a look at the full documentation page here <a href="https://cbonte.github.io/haproxy-dconv/">https://cbonte.github.io/haproxy-dconv/</a></p>


uuid: 53db9ff2-6e65-4f06-9386-5c4b48b78e94
title: HTTPS and SSH on same port
slug: https-and-ssh-on-same-port
image: /content/images/2017/11/sslh-1024x422.png
status: published
language: en_US
markdown:
  Some Internet specialist organizations and corporate organizations may have blocked the greater part of the ports, and permitted just a couple of particular ports, for example, port 80 and 443 to fix their security. In such cases, we must choose between limited options, yet utilize the same port for different projects, say the HTTPS Port 443, which is once in a while blocked. Here is the place SSLH, an SSL/SSH multiplexer, comes in offer assistance. It will tune in for approaching associations on a port 443. To put this all the more just, SSLH enables us to run a few projects/benefits on port 443 on a Linux framework. In this way, you can utilize both SSL and SSH utilizing a same port in the meantime. On the off chance that you at any point been in a circumstance where most ports are obstructed by the firewalls, you can utilize SSLH to get to your remote server.

  This brief tutorial describes how to share the same port for https, ssh using SSLH in Unix-like operating systems.

  **SSLH – Share The Same Port For HTTPS, SSH, And OpenVPN
  Install SSLH**

  SSLH is packaged for most Linux distributions, so you can install it using the default package managers.

  On Arch Linux and derivatives like Antergos, Manjaro Linux, install it using Pacman as shown below.
  ```
  sudo pacman -S sslh
  ```
  On Debian, Ubuntu, and derivatives, run:
  ```
  sudo apt-get install sslh
  ```

  On RHEL, CentOS, you need to add EPEL repository and then install SSLH as shown below.

  ```
  sudo yum install epel-release
  sudo yum install sslh
  ```

  On Fedora:
  ```
  sudo dnf install sslh
  ```

  The output should look like this:

  ![alt](/content/images/2017/11/1.png)

  **Configure Apache or Nginx webservers**
  <BR>
  As you already know, Apache and Nginx webservers will listen on all network interfaces (i.e 0.0.0.0:443) by default. We need to change this setting to tell the webserver to listen on the localhost interface only (i.e 127.0.0.1:443 or localhost:443).

  To do so, edit the webserver (nginx or apache) configuration file and find the following line:
  ```
  listen 443 ssl;
  ```
  And, change it to:
  ```
  listen 127.0.0.1:443 ssl;
  ```
  If you’re using Virutalhosts in Apache, make sure you have changed that it too.
  ```
  VirtualHost 127.0.0.1:443
  ```
  Save and close the config files. Do not restart the services. We haven’t finished yet.

  - Configure SSLH 

  Once you have made the webservers to listen on the local interface only, edit SSLH config file:
  ```
  nano /etc/sslh.cfg
  ```

  You may configure various settings here, but the most important adjustment is to put the section with your external address name to "0.0.0.0" which means that service is going to listen to all network addresses on the same server. Other than that you can configure user running and other things. Here is how my config file on test CentOS 7 server looks like:
  ```
  ~]# cat /etc/sslh.cfg 
  # This is a basic configuration file that should provide
  # sensible values for "standard" setup.

  verbose: false;
  foreground: true;
  inetd: false;
  numeric: false;
  transparent: false;
  timeout: 2;
  user: "root";


  # Change hostname with your external address name.
  listen:
  (
      { host: "0.0.0.0"; port: "443"; }
  );

  protocols:
  (
       { name: "ssh"; service: "ssh"; host: "malizeleni1.mylabserver.com"; port: "22"; },
       { name: "openvpn"; host: "malizeleni1.mylabserver.com"; port: "1194"; },
       { name: "xmpp"; host: "malizeleni1.mylabserver.com"; port: "5222"; },
       { name: "http"; host: "malizeleni1.mylabserver.com"; port: "80"; },
       { name: "ssl"; host: "malizeleni1.mylabserver.com"; port: "443"; log_level: 0; },
       { name: "anyprot"; host: "malizeleni1.mylabserver.com"; port: "443"; }
  );
  ```

  Save and close the file.

  Now enable sslh service and start it:
  ```
  systemctl enable sslh
  ```
  And start it:
  ```
  systemctl start sslh
  ```

  You may use lsof command to see what is now listening on the 443 port:
  ![alt](/content/images/2017/11/2.png)

  And now you can test and connect to your server and use Multiplexer service:

  ```
  > ssh -p 443 user@malizeleni1.mylabserver.com
  user@malizeleni1.mylabserver.com's password: 
  Last login: Sun Nov 12 20:44:07 2017 from cable-xx-2x0-2x4-2x.dynamic.sbb.rs
  [user@malizeleni1 ~]$ hostname
  malizeleni1.mylabserver.com
  [user@malizeleni1 ~]$
  ```

  You are now able to access the remote server via SSH even if the default SSH port 22 is blocked. As you see in the above example, I have used the https port 443 for SSH connection. Also, we can use the same port 443 for OpenVPN connections too. For more details, check the project’s website URL given below.

  Alternatively, you can adjust/swap your Nginx and/or Apache to alternative port (for ex 4433 or whatever) and then adjust the sslh.cfg file with that port, for example:
  ```
  { name: "ssl"; host: "localhost"; port: "4433"; log_level: 0; },
  ```

  You may done the same with any other service/port and/or protocol and use this Multiplexer in that purpose.

  Cheers!

  Resource:
  SSLH website (http://www.rutschle.net/sslh)

  Hope this article helps you to organize more secure environment within your corporate and/or private network, or at least to override when using public Wi-Fi hotspots.

html:
  <p>Some Internet specialist organizations and corporate organizations may have blocked the greater part of the ports, and permitted just a couple of particular ports, for example, port 80 and 443 to fix their security. In such cases, we must choose between limited options, yet utilize the same port for different projects, say the HTTPS Port 443, which is once in a while blocked. Here is the place SSLH, an SSL/SSH multiplexer, comes in offer assistance. It will tune in for approaching associations on a port 443. To put this all the more just, SSLH enables us to run a few projects/benefits on port 443 on a Linux framework. In this way, you can utilize both SSL and SSH utilizing a same port in the meantime. On the off chance that you at any point been in a circumstance where most ports are obstructed by the firewalls, you can utilize SSLH to get to your remote server.</p>

  <p>This brief tutorial describes how to share the same port for https, ssh using SSLH in Unix-like operating systems.</p>

  <p><strong>SSLH – Share The Same Port For HTTPS, SSH, And OpenVPN
  Install SSLH</strong></p>

  <p>SSLH is packaged for most Linux distributions, so you can install it using the default package managers.</p>

  <p>On Arch Linux and derivatives like Antergos, Manjaro Linux, install it using Pacman as shown below.  </p>

  <pre><code>sudo pacman -S sslh  
  </code></pre>

  <p>On Debian, Ubuntu, and derivatives, run:  </p>

  <pre><code>sudo apt-get install sslh  
  </code></pre>

  <p>On RHEL, CentOS, you need to add EPEL repository and then install SSLH as shown below.</p>

  <pre><code>sudo yum install epel-release  
  sudo yum install sslh  
  </code></pre>

  <p>On Fedora:  </p>

  <pre><code>sudo dnf install sslh  
  </code></pre>

  <p>The output should look like this:</p>

  <p><img src="/content/images/2017/11/1.png" alt="alt" /></p>

  <p><strong>Configure Apache or Nginx webservers</strong>
  <BR> <br />
  As you already know, Apache and Nginx webservers will listen on all network interfaces (i.e 0.0.0.0:443) by default. We need to change this setting to tell the webserver to listen on the localhost interface only (i.e 127.0.0.1:443 or localhost:443).</p>

  <p>To do so, edit the webserver (nginx or apache) configuration file and find the following line:  </p>

  <pre><code>listen 443 ssl;  
  </code></pre>

  <p>And, change it to:  </p>

  <pre><code>listen 127.0.0.1:443 ssl;  
  </code></pre>

  <p>If you’re using Virutalhosts in Apache, make sure you have changed that it too.  </p>

  <pre><code>VirtualHost 127.0.0.1:443  
  </code></pre>

  <p>Save and close the config files. Do not restart the services. We haven’t finished yet.</p>

  <ul>
  <li>Configure SSLH </li>
  </ul>

  <p>Once you have made the webservers to listen on the local interface only, edit SSLH config file:  </p>

  <pre><code>nano /etc/sslh.cfg  
  </code></pre>

  <p>You may configure various settings here, but the most important adjustment is to put the section with your external address name to "0.0.0.0" which means that service is going to listen to all network addresses on the same server. Other than that you can configure user running and other things. Here is how my config file on test CentOS 7 server looks like:  </p>

  <pre><code>~]# cat /etc/sslh.cfg 
  # This is a basic configuration file that should provide
  # sensible values for "standard" setup.

  verbose: false;  
  foreground: true;  
  inetd: false;  
  numeric: false;  
  transparent: false;  
  timeout: 2;  
  user: "root";


  # Change hostname with your external address name.
  listen:  
  (
      { host: "0.0.0.0"; port: "443"; }
  );

  protocols:  
  (
       { name: "ssh"; service: "ssh"; host: "malizeleni1.mylabserver.com"; port: "22"; },
       { name: "openvpn"; host: "malizeleni1.mylabserver.com"; port: "1194"; },
       { name: "xmpp"; host: "malizeleni1.mylabserver.com"; port: "5222"; },
       { name: "http"; host: "malizeleni1.mylabserver.com"; port: "80"; },
       { name: "ssl"; host: "malizeleni1.mylabserver.com"; port: "443"; log_level: 0; },
       { name: "anyprot"; host: "malizeleni1.mylabserver.com"; port: "443"; }
  );
  </code></pre>

  <p>Save and close the file.</p>

  <p>Now enable sslh service and start it:  </p>

  <pre><code>systemctl enable sslh  
  </code></pre>

  <p>And start it:  </p>

  <pre><code>systemctl start sslh  
  </code></pre>

  <p>You may use lsof command to see what is now listening on the 443 port: <br />
  <img src="/content/images/2017/11/2.png" alt="alt" /></p>

  <p>And now you can test and connect to your server and use Multiplexer service:</p>

  <pre><code>&gt; ssh -p 443 user@malizeleni1.mylabserver.com
  user@malizeleni1.mylabserver.com's password:  
  Last login: Sun Nov 12 20:44:07 2017 from cable-xx-2x0-2x4-2x.dynamic.sbb.rs  
  [user@malizeleni1 ~]$ hostname
  malizeleni1.mylabserver.com  
  [user@malizeleni1 ~]$
  </code></pre>

  <p>You are now able to access the remote server via SSH even if the default SSH port 22 is blocked. As you see in the above example, I have used the https port 443 for SSH connection. Also, we can use the same port 443 for OpenVPN connections too. For more details, check the project’s website URL given below.</p>

  <p>Alternatively, you can adjust/swap your Nginx and/or Apache to alternative port (for ex 4433 or whatever) and then adjust the sslh.cfg file with that port, for example:  </p>

  <pre><code>{ name: "ssl"; host: "localhost"; port: "4433"; log_level: 0; },
  </code></pre>

  <p>You may done the same with any other service/port and/or protocol and use this Multiplexer in that purpose.</p>

  <p>Cheers!</p>

  <p>Resource: <br />
  SSLH website (<a href="http://www.rutschle.net/sslh">http://www.rutschle.net/sslh</a>)</p>

  <p>Hope this article helps you to organize more secure environment within your corporate and/or private network, or at least to override when using public Wi-Fi hotspots.</p>


uuid: 4411619b-9933-44d9-a9dd-e09d21a38035
title: Quality server monitoring solution using NetData/Prometheus/Grafana
slug: quality-server-monitoring-solution-using-netdata-prometheus-grafana
image: /content/images/2018/12/prometheus-monitoring-1.jpg
status: published
language: en_US
markdown:
  Before a while, I had written a post about monitoring a single Linux system. It was being done by Nagios and its magic. I am still a huge Nagios lover and fan, so this stays of course.

  Recently, I've been discovered NetData application (https://github.com/netdata/netdata). 

  Netdata is distributed, real-time, performance and health monitoring for systems and applications. It is a highly optimized monitoring agent you install on all your systems and containers.

  Netdata provides unparalleled insights, in real-time, of everything happening on the systems it runs (including web servers, databases, applications), using highly interactive web dashboards. It can run autonomously, without any third party components, or it can be integrated into existing monitoring toolchains (Prometheus, Graphite, OpenTSDB, Kafka, Grafana, etc).

  Netdata is fast and efficient, designed to permanently run on all systems (physical & virtual servers, containers, IoT devices), without disrupting their core function.

  Netdata is free, open-source software and it currently runs on Linux, FreeBSD, and MacOS.

  In this blog post, I will describe how I have added multiple cPanel based servers with NetData to Prometheus and then to the Grafana tool.

  Prometheus is a distributed monitoring system which offers a very simple set up along with a robust data model. Recently netdata added support for Prometheus. 

  I’m going to quickly show you how to install both netdata and Prometheus on the client and server. We can then use grafana pointed at Prometheus to obtain long-term metrics netdata offers. 

  There is a number of ways to install netdata according to Installation (https://docs.netdata.cloud/packaging/installer/#installation).

  The suggested way of installing the latest netdata and keep it upgrade automatically. Using one line installation:

  bash <(curl -Ss https://my-netdata.io/kickstart.sh)

  Since I am using xtendWeb application alongside cPanel, I can use the following bash script safely:

  **CLIENT SIDE INSTALLATION** <BR>
  1) /opt/nDeploy/scripts/easy_netdata_setup.sh
  2) edit /opt/netdata/netdata-config/netdata.conf
  bind=0.0.0.0.0:19999

  (you can here assign internal network IP address, VPN address or anything else)

  3) If you are using CSF as a frontend firewall please do the following:

  csf.allow:
  ```
  tcp|in|d=19999|s=10.8.8.1
  tcp|out|d=19999|s=10.8.8.1
  ```
  csf.deny:
  ```
  tcp|in|d=19999|d=0.0.0.0/0 # do not delete
  ```
  (that do not delete is important comment because CSF is not going to remove it when doing cronjob/updates). What we did here is that we have whitelisted originated Prometheus/Grafana server, and blocked everything else, of course in the CSF style. You can use native IPtables to accomplish this, firewalld or any other popular firewall tools.

  At this point, we should have netdata listening on port 19999. Attempt to take your browser here:
  ```
  http://your.netdata.ip:19999
  ```
  (replace your.netdata.ip with the IP or hostname of the server running netdata), or http://your.netdata.ip/netdata if you are using XtendWeb netdata bash script to test it. This is everything needed to be installed on the client server, VPS, docker or anywhere else.


  **SERVER SIDE INSTALLATION (PROMETHEUS/GRAFANA)**

  -- Installing Prometheus -- <BR>
  In order to install Prometheus, we are going to introduce our own systemd startup script along with an example of prometheus.yaml configuration. Prometheus needs to be pointed to your server at a specific target url for it to scrape netdata’s api. Prometheus is always a pull model meaning netdata is the passive client within this architecture. Prometheus always initiates the connection with netdata.

  1) Download Prometheus:
  ```
  wget -O /tmp/prometheus-2.3.2.linux-amd64.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz
  ```

  2) Create a Prometheus system user:
  ```
  sudo useradd -r prometheus
  ```
  3) Create a Prometheus directory:
  ```
  sudo mkdir /opt/prometheus
  sudo chown prometheus:prometheus /opt/prometheus
  ```

  4) Untar prometheus directory:
  ```
  sudo tar -xvf /tmp/prometheus-2.3.2.linux-amd64.tar.gz -C /opt/prometheus --strip=1
  ```

  -- prometheus.yml adjustments -- <BR>
  We will use the following prometheus.yml file. Save it at `/opt/prometheus/prometheus.yml`.

  Make sure to replace your.netdata.ip with the IP or hostname of the host running netdata.

  ```
  [root@one ~]# cat /opt/prometheus/prometheus.yml 
  # my global config
  global:
    scrape_interval:     5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
    evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.
    # scrape_timeout is set to the global default (10s).

    # Attach these labels to any time series or alerts when communicating with
    # external systems (federation, remote storage, Alertmanager).
    external_labels:
        monitor: 'codelab-monitor'

  # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
  rule_files:
    # - "first.rules"
    # - "second.rules"

  scrape_configs:

    - job_name: 'netdata-hostname'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP 
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname:19999']

    - job_name: 'netdata-hostname0'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP 
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname0:19999']


    - job_name: 'netdata-hostname1'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname1:19999']

    - job_name: 'netdata-hostname2'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname2:19999']
  ```

  Note: Key points here are job-name and targets. As Prometheus Yaml structure says, you can combine multiple targets into single job-name, or you can split them as I've stated in my working example here which added 4 active hostnames (servers).

  -- Optional: nodes.yml -- <BR>
  The following is completely optional, it will enable Prometheus to generate alerts from some NetData sources. Tweak the values to your own needs. We will use the following nodes.yml file below. Save it at /opt/prometheus/nodes.yml, and add a - “nodes.yml” entry under the rule_files: section in the example prometheus.yml file above.

  ```
  groups:
  - name: nodes

    rules:
    - alert: node_high_cpu_usage_70
      expr: avg(rate(netdata_cpu_cpu_percentage_average{dimension="idle"}[1m])) by (job) > 70
      for: 1m
      annotations:
        description: '{{ $labels.job }} on ''{{ $labels.job }}'' CPU usage is at {{ humanize $value }}%.'
        summary: CPU alert for container node '{{ $labels.job }}'

    - alert: node_high_memory_usage_70
      expr: 100 / sum(netdata_system_ram_MB_average) by (job) 
        * sum(netdata_system_ram_MB_average{dimension=~"free|cached"}) by (job) < 30
      for: 1m
      annotations:
        description: '{{ $labels.job }} memory usage is {{ humanize $value}}%.'
        summary: Memory alert for container node '{{ $labels.job }}'

    - alert: node_low_root_filesystem_space_20
      expr: 100 / sum(netdata_disk_space_GB_average{family="/"}) by (job)
        * sum(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}) by (job) < 20
      for: 1m
      annotations:
        description: '{{ $labels.job }} root filesystem space is {{ humanize $value}}%.'
        summary: Root filesystem alert for container node '{{ $labels.job }}'

    - alert: node_root_filesystem_fill_rate_6h
      expr: predict_linear(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}[1h], 6 * 3600) < 0
      for: 1h
      labels:
        severity: critical
      annotations:
        description: Container node {{ $labels.job }} root filesystem is going to fill up in 6h.
        summary: Disk fill alert for Swarm node '{{ $labels.job }}'
  ```

  -- Create prometheus service -- <BR>
  ```
  [Unit]
  Description=Prometheus Server
  AssertPathExists=/opt/prometheus

  [Service]
  Type=simple
  WorkingDirectory=/opt/prometheus
  User=prometheus
  Group=prometheus
  ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --log.level=info
  ExecReload=/bin/kill -SIGHUP $MAINPID
  ExecStop=/bin/kill -SIGINT $MAINPID

  [Install]
  WantedBy=multi-user.target
  ```

  Start Prometheus service and enable it:
  ```
  sudo systemctl start prometheus
  sudo systemctl enable prometheus
  ```

  Prometheus should now start and listen on port 9090. Attempt to head there with your browser.

  If everything is working correctly when you fetch http://your.prometheus.ip:9090 you will see a ‘Status’ tab. Click this and click on ‘targets’ We should see the netdata host as a scraped target.

  This is my example (screenshot):
  ![alt](/content/images/2018/12/targets-prometheus.png)


  Note: As I've said before you can limit access to Prometheus using firewall service as actively it does not have any authentication built-in.

  == GRAFANA INSTALLATION AND INTEGRATION ==

  What is Grafana?
  Grafana is software for data visualization & Monitoring with support for Graphite, InfluxDB, Prometheus, Elasticsearch and many more databases.

  More details about Grafana project you can read here:
  https://grafana.com/

  To install grafana on the server side, please do the following:
  1) `curl -s https://packagecloud.io/install/repositories/grafana/stable/script.rpm.sh | sudo bash`
  2) `sudo yum install grafana-5.4.2-1.x86_64`
  3) `systemctl start grafana-server`
  4) `systemctl status grafana-server`
  5) `systemctl enable grafana-server`

  At this point you should have active grafana listening on default port 3000, in my example the GUI access is:
  ```
  http://10.8.8.1:3000
  ```

  Initially default username/password for Grafana is admin/admin. After first entry, you will be asked to change it to something else. When you are logged inside, we need to add Prometheus as Data source.

  To do that:
  - Navigate to Configuration --> Data Sources
  ![alt](/content/images/2018/12/date-source-grafana.png)

  Click on the ADD DATA SOURCES, SELECT PROMETHEUS and fill up these fields carefully:
  ![alt](/content/images/2018/12/data-source-grafana1.png)
  In my example the Prometheus server is assigned as 10.8.8.1:9090, but you can use public, internal or vpn IP address as I've stated before.

  Next, you need to navigate to http://10.8.8.1:3000/dashboard/import (or + sign at left sidebar then IMPORT)

  ![alt](/content/images/2018/12/data-source-grafana2.png)

  There are few popular NetData ready made Dashboards using Prometheus. I've picked up https://grafana.com/dashboards/7107. You can simply import that one and start using it, or you can easily create your own Dashboard instead.

  Now that everything is installed you can safely navigate to the Dashboard and enjoy. I will paste here a few active examples from monitoring my 4 servers:

  ![alt](/content/images/2018/12/data-source-grafana3.png)
  ![alt](/content/images/2018/12/data-source-grafana4.png)
  ![alt](/content/images/2018/12/data-source-grafana5.png)

  And the one I especially like is CPU app time grap where you can setup Alert to get notified on Email or even Better SLACK!
  ![alt](/content/images/2018/12/data-source-grafana6.png)

  The Grafana Slack integration is nicely explained on the http://docs.grafana.org/alerting/notifications/#slack page, so please just follow it and enjoy.

  Conclusion:
  We have today installed netdata metric gather tool on four separate servers, installed prometheus/grafana on the separate server which will be used to collect data and alert us if resource usage on instances become unstable or beyond defined levels. I hope this article helps anyone with needs to monitor multiple servers at the same time.

  Reference articles:
  https://docs.netdata.cloud/backends/walkthrough/
  https://github.com/netdata/netdata
  https://grafana.com/dashboards/7107
  http://docs.grafana.org/alerting/notifications/#slack

html:
  <p>Before a while, I had written a post about monitoring a single Linux system. It was being done by Nagios and its magic. I am still a huge Nagios lover and fan, so this stays of course.</p>

  <p>Recently, I've been discovered NetData application (<a href="https://github.com/netdata/netdata">https://github.com/netdata/netdata</a>). </p>

  <p>Netdata is distributed, real-time, performance and health monitoring for systems and applications. It is a highly optimized monitoring agent you install on all your systems and containers.</p>

  <p>Netdata provides unparalleled insights, in real-time, of everything happening on the systems it runs (including web servers, databases, applications), using highly interactive web dashboards. It can run autonomously, without any third party components, or it can be integrated into existing monitoring toolchains (Prometheus, Graphite, OpenTSDB, Kafka, Grafana, etc).</p>

  <p>Netdata is fast and efficient, designed to permanently run on all systems (physical &amp; virtual servers, containers, IoT devices), without disrupting their core function.</p>

  <p>Netdata is free, open-source software and it currently runs on Linux, FreeBSD, and MacOS.</p>

  <p>In this blog post, I will describe how I have added multiple cPanel based servers with NetData to Prometheus and then to the Grafana tool.</p>

  <p>Prometheus is a distributed monitoring system which offers a very simple set up along with a robust data model. Recently netdata added support for Prometheus. </p>

  <p>I’m going to quickly show you how to install both netdata and Prometheus on the client and server. We can then use grafana pointed at Prometheus to obtain long-term metrics netdata offers. </p>

  <p>There is a number of ways to install netdata according to Installation (<a href="https://docs.netdata.cloud/packaging/installer/#installation">https://docs.netdata.cloud/packaging/installer/#installation</a>).</p>

  <p>The suggested way of installing the latest netdata and keep it upgrade automatically. Using one line installation:</p>

  <p>bash &lt;(curl -Ss <a href="https://my-netdata.io/kickstart.sh">https://my-netdata.io/kickstart.sh</a>)</p>

  <p>Since I am using xtendWeb application alongside cPanel, I can use the following bash script safely:</p>

  <p><strong>CLIENT SIDE INSTALLATION</strong> <BR>
  1) /opt/nDeploy/scripts/easy<em>netdata</em>setup.sh <br />
  2) edit /opt/netdata/netdata-config/netdata.conf <br />
  bind=0.0.0.0.0:19999</p>

  <p>(you can here assign internal network IP address, VPN address or anything else)</p>

  <p>3) If you are using CSF as a frontend firewall please do the following:</p>

  <p>csf.allow:  </p>

  <pre><code>tcp|in|d=19999|s=10.8.8.1  
  tcp|out|d=19999|s=10.8.8.1  
  </code></pre>

  <p>csf.deny:  </p>

  <pre><code>tcp|in|d=19999|d=0.0.0.0/0 # do not delete  
  </code></pre>

  <p>(that do not delete is important comment because CSF is not going to remove it when doing cronjob/updates). What we did here is that we have whitelisted originated Prometheus/Grafana server, and blocked everything else, of course in the CSF style. You can use native IPtables to accomplish this, firewalld or any other popular firewall tools.</p>

  <p>At this point, we should have netdata listening on port 19999. Attempt to take your browser here:  </p>

  <pre><code>http://your.netdata.ip:19999  
  </code></pre>

  <p>(replace your.netdata.ip with the IP or hostname of the server running netdata), or <a href="http://your.netdata.ip/netdata">http://your.netdata.ip/netdata</a> if you are using XtendWeb netdata bash script to test it. This is everything needed to be installed on the client server, VPS, docker or anywhere else.</p>

  <p><strong>SERVER SIDE INSTALLATION (PROMETHEUS/GRAFANA)</strong></p>

  <p>-- Installing Prometheus -- <BR>
  In order to install Prometheus, we are going to introduce our own systemd startup script along with an example of prometheus.yaml configuration. Prometheus needs to be pointed to your server at a specific target url for it to scrape netdata’s api. Prometheus is always a pull model meaning netdata is the passive client within this architecture. Prometheus always initiates the connection with netdata.</p>

  <p>1) Download Prometheus:  </p>

  <pre><code>wget -O /tmp/prometheus-2.3.2.linux-amd64.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz  
  </code></pre>

  <p>2) Create a Prometheus system user:  </p>

  <pre><code>sudo useradd -r prometheus  
  </code></pre>

  <p>3) Create a Prometheus directory:  </p>

  <pre><code>sudo mkdir /opt/prometheus  
  sudo chown prometheus:prometheus /opt/prometheus  
  </code></pre>

  <p>4) Untar prometheus directory:  </p>

  <pre><code>sudo tar -xvf /tmp/prometheus-2.3.2.linux-amd64.tar.gz -C /opt/prometheus --strip=1  
  </code></pre>

  <p>-- prometheus.yml adjustments -- <BR>
  We will use the following prometheus.yml file. Save it at <code>/opt/prometheus/prometheus.yml</code>.</p>

  <p>Make sure to replace your.netdata.ip with the IP or hostname of the host running netdata.</p>

  <pre><code>[root@one ~]# cat /opt/prometheus/prometheus.yml 
  # my global config
  global:  
    scrape_interval:     5s # Set the scrape interval to every 5 seconds. Default is every 1 minute.
    evaluation_interval: 5s # Evaluate rules every 5 seconds. The default is every 1 minute.
    # scrape_timeout is set to the global default (10s).

    # Attach these labels to any time series or alerts when communicating with
    # external systems (federation, remote storage, Alertmanager).
    external_labels:
        monitor: 'codelab-monitor'

  # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
  rule_files:  
    # - "first.rules"
    # - "second.rules"

  scrape_configs:

    - job_name: 'netdata-hostname'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP 
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname:19999']

    - job_name: 'netdata-hostname0'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP 
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname0:19999']


    - job_name: 'netdata-hostname1'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname1:19999']

    - job_name: 'netdata-hostname2'

      metrics_path: '/api/v1/allmetrics'
      params:
        # format: prometheus | prometheus_all_hosts
        # You can use `prometheus_all_hosts` if you want Prometheus to set the `instance` to your hostname instead of IP
        format: [prometheus_all_hosts]
        #
        # sources: as-collected | raw | average | sum | volume
        # default is: average
        #source: [as-collected]
        #
        # server name for this prometheus - the default is the client IP
        # for netdata to uniquely identify it
        #server: ['prometheus1']
      honor_labels: true

      static_configs:
        - targets: ['hostname2:19999']
  </code></pre>

  <p>Note: Key points here are job-name and targets. As Prometheus Yaml structure says, you can combine multiple targets into single job-name, or you can split them as I've stated in my working example here which added 4 active hostnames (servers).</p>

  <p>-- Optional: nodes.yml -- <BR>
  The following is completely optional, it will enable Prometheus to generate alerts from some NetData sources. Tweak the values to your own needs. We will use the following nodes.yml file below. Save it at /opt/prometheus/nodes.yml, and add a - “nodes.yml” entry under the rule_files: section in the example prometheus.yml file above.</p>

  <pre><code>groups:  
  - name: nodes

    rules:
    - alert: node_high_cpu_usage_70
      expr: avg(rate(netdata_cpu_cpu_percentage_average{dimension="idle"}[1m])) by (job) &gt; 70
      for: 1m
      annotations:
        description: '{{ $labels.job }} on ''{{ $labels.job }}'' CPU usage is at {{ humanize $value }}%.'
        summary: CPU alert for container node '{{ $labels.job }}'

    - alert: node_high_memory_usage_70
      expr: 100 / sum(netdata_system_ram_MB_average) by (job) 
        * sum(netdata_system_ram_MB_average{dimension=~"free|cached"}) by (job) &lt; 30
      for: 1m
      annotations:
        description: '{{ $labels.job }} memory usage is {{ humanize $value}}%.'
        summary: Memory alert for container node '{{ $labels.job }}'

    - alert: node_low_root_filesystem_space_20
      expr: 100 / sum(netdata_disk_space_GB_average{family="/"}) by (job)
        * sum(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}) by (job) &lt; 20
      for: 1m
      annotations:
        description: '{{ $labels.job }} root filesystem space is {{ humanize $value}}%.'
        summary: Root filesystem alert for container node '{{ $labels.job }}'

    - alert: node_root_filesystem_fill_rate_6h
      expr: predict_linear(netdata_disk_space_GB_average{family="/",dimension=~"avail|cached"}[1h], 6 * 3600) &lt; 0
      for: 1h
      labels:
        severity: critical
      annotations:
        description: Container node {{ $labels.job }} root filesystem is going to fill up in 6h.
        summary: Disk fill alert for Swarm node '{{ $labels.job }}'
  </code></pre>

  <p>-- Create prometheus service -- <BR></p>

  <pre><code>[Unit]
  Description=Prometheus Server  
  AssertPathExists=/opt/prometheus

  [Service]
  Type=simple  
  WorkingDirectory=/opt/prometheus  
  User=prometheus  
  Group=prometheus  
  ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --log.level=info  
  ExecReload=/bin/kill -SIGHUP $MAINPID  
  ExecStop=/bin/kill -SIGINT $MAINPID

  [Install]
  WantedBy=multi-user.target  
  </code></pre>

  <p>Start Prometheus service and enable it:  </p>

  <pre><code>sudo systemctl start prometheus  
  sudo systemctl enable prometheus  
  </code></pre>

  <p>Prometheus should now start and listen on port 9090. Attempt to head there with your browser.</p>

  <p>If everything is working correctly when you fetch <a href="http://your.prometheus.ip:9090">http://your.prometheus.ip:9090</a> you will see a ‘Status’ tab. Click this and click on ‘targets’ We should see the netdata host as a scraped target.</p>

  <p>This is my example (screenshot): <br />
  <img src="/content/images/2018/12/targets-prometheus.png" alt="alt" /></p>

  <p>Note: As I've said before you can limit access to Prometheus using firewall service as actively it does not have any authentication built-in.</p>

  <p><mark> GRAFANA INSTALLATION AND INTEGRATION </mark></p>

  <p>What is Grafana? <br />
  Grafana is software for data visualization &amp; Monitoring with support for Graphite, InfluxDB, Prometheus, Elasticsearch and many more databases.</p>

  <p>More details about Grafana project you can read here: <br />
  <a href="https://grafana.com/">https://grafana.com/</a></p>

  <p>To install grafana on the server side, please do the following: <br />
  1) <code>curl -s https://packagecloud.io/install/repositories/grafana/stable/script.rpm.sh | sudo bash</code> <br />
  2) <code>sudo yum install grafana-5.4.2-1.x86_64</code> <br />
  3) <code>systemctl start grafana-server</code> <br />
  4) <code>systemctl status grafana-server</code> <br />
  5) <code>systemctl enable grafana-server</code></p>

  <p>At this point you should have active grafana listening on default port 3000, in my example the GUI access is:  </p>

  <pre><code>http://10.8.8.1:3000  
  </code></pre>

  <p>Initially default username/password for Grafana is admin/admin. After first entry, you will be asked to change it to something else. When you are logged inside, we need to add Prometheus as Data source.</p>

  <p>To do that: <br />
  - Navigate to Configuration --> Data Sources
  <img src="/content/images/2018/12/date-source-grafana.png" alt="alt" /></p>

  <p>Click on the ADD DATA SOURCES, SELECT PROMETHEUS and fill up these fields carefully: <br />
  <img src="/content/images/2018/12/data-source-grafana1.png" alt="alt" />
  In my example the Prometheus server is assigned as 10.8.8.1:9090, but you can use public, internal or vpn IP address as I've stated before.</p>

  <p>Next, you need to navigate to <a href="http://10.8.8.1:3000/dashboard/import">http://10.8.8.1:3000/dashboard/import</a> (or + sign at left sidebar then IMPORT)</p>

  <p><img src="/content/images/2018/12/data-source-grafana2.png" alt="alt" /></p>

  <p>There are few popular NetData ready made Dashboards using Prometheus. I've picked up <a href="https://grafana.com/dashboards/7107">https://grafana.com/dashboards/7107</a>. You can simply import that one and start using it, or you can easily create your own Dashboard instead.</p>

  <p>Now that everything is installed you can safely navigate to the Dashboard and enjoy. I will paste here a few active examples from monitoring my 4 servers:</p>

  <p><img src="/content/images/2018/12/data-source-grafana3.png" alt="alt" />
  <img src="/content/images/2018/12/data-source-grafana4.png" alt="alt" />
  <img src="/content/images/2018/12/data-source-grafana5.png" alt="alt" /></p>

  <p>And the one I especially like is CPU app time grap where you can setup Alert to get notified on Email or even Better SLACK! <br />
  <img src="/content/images/2018/12/data-source-grafana6.png" alt="alt" /></p>

  <p>The Grafana Slack integration is nicely explained on the <a href="http://docs.grafana.org/alerting/notifications/#slack">http://docs.grafana.org/alerting/notifications/#slack</a> page, so please just follow it and enjoy.</p>

  <p>Conclusion: <br />
  We have today installed netdata metric gather tool on four separate servers, installed prometheus/grafana on the separate server which will be used to collect data and alert us if resource usage on instances become unstable or beyond defined levels. I hope this article helps anyone with needs to monitor multiple servers at the same time.</p>

  <p>Reference articles: <br />
  <a href="https://docs.netdata.cloud/backends/walkthrough/">https://docs.netdata.cloud/backends/walkthrough/</a> <br />
  <a href="https://github.com/netdata/netdata">https://github.com/netdata/netdata</a> <br />
  <a href="https://grafana.com/dashboards/7107">https://grafana.com/dashboards/7107</a> <br />
  <a href="http://docs.grafana.org/alerting/notifications/#slack">http://docs.grafana.org/alerting/notifications/#slack</a></p>


uuid: e6186978-bd52-43e1-80ee-fc4c14d5be23
title: Install Kubernetes/Docker on Scaleway provider + review
slug: install-kubernetes-docker-on-scaleway-provider-review
status: draft
language: en_US
markdown:
  Hello,

  With the recent changes in the Web Hosting industry, I wanted to give it a try and test myself with the popular Microservice software. For start, I've picked to test Scaleway.com provider which recently merge with online.net provider where I've tested few customers and deployed their work/business and applications. 

  root@controller-106:/data# hdparm -W1 /dev/sda

  /dev/sda:
   setting drive write-caching to 1 (on)
   write-caching =  1 (on)
  root@controller-106:/data# dd if=/dev/zero of=/data/testfile bs=1G count=1 oflag=direct
  1+0 records in
  1+0 records out
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.77861 s, 225 MB/s
  root@controller-106:/data# hdparm -W0 /dev/sda

  /dev/sda:
   setting drive write-caching to 0 (off)
   write-caching =  0 (off)
  root@controller-106:/data# dd if=/dev/zero of=/data/testfile bs=1G count=1 oflag=direct
  1+0 records in
  1+0 records out
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.26656 s, 171 MB/s
  root@controller-106:/data# 

  root@controller-106:/data# hdparm -t --direct /dev/sda

  /dev/sda:
   Timing O_DIRECT disk reads: 782 MB in  3.00 seconds = 260.43 MB/sec

html:
  <p>Hello,</p>

  <p>With the recent changes in the Web Hosting industry, I wanted to give it a try and test myself with the popular Microservice software. For start, I've picked to test Scaleway.com provider which recently merge with online.net provider where I've tested few customers and deployed their work/business and applications. </p>

  <p>root@controller-106:/data# hdparm -W1 /dev/sda</p>

  <p>/dev/sda:
   setting drive write-caching to 1 (on)
   write-caching =  1 (on)
  root@controller-106:/data# dd if=/dev/zero of=/data/testfile bs=1G count=1 oflag=direct <br />
  1+0 records in <br />
  1+0 records out <br />
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.77861 s, 225 MB/s <br />
  root@controller-106:/data# hdparm -W0 /dev/sda</p>

  <p>/dev/sda:
   setting drive write-caching to 0 (off)
   write-caching =  0 (off)
  root@controller-106:/data# dd if=/dev/zero of=/data/testfile bs=1G count=1 oflag=direct <br />
  1+0 records in <br />
  1+0 records out <br />
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.26656 s, 171 MB/s <br />
  root@controller-106:/data# </p>

  <p>root@controller-106:/data# hdparm -t --direct /dev/sda</p>

  <p>/dev/sda:
   Timing O_DIRECT disk reads: 782 MB in  3.00 seconds = 260.43 MB/sec</p>


uuid: cd39ba43-89dd-4005-ab7b-5a58a912f4a4
title: How to install Virtlib/KVM/QEMU on CentOS 7 / RHEL 7 Server
slug: how-to-install-virtlib-kvm-qemu-on-centos-7-rhel-7-server
image: /content/images/2019/04/Virtual_network_default_network_overview.jpg
status: published
language: en_US
markdown:
  Hello folks. Recently I was hired by my good friend to migrate his data from the old server which was running from 2016. Back then we installed OpenVZ platform with some freeware GUI and Webmin and deployed several Virtual machines on it. It was/is working perfectly, but I had to move it to something newer as technology grows. He rented a server from a known provider in Europe called Hetzner AG. The server seems to be powerful with Intel Skylake processor and NVME drives. I was pretty impressed until I've realized that I can't install Debian/Ubuntu on it and simply move forward with Proxmox as freeware KVM/Xen solution so I had to think of different ways to accomplish my mission. Started reading about the virsh tool which is using libvirt as a base (https://libvirt.org/).

  The virsh program is the main interface for managing virsh guest domains. The program can be used to create, pause, and shutdown domains. It can also be used to list current domains. Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). It is free software available under the GNU Lesser General Public License. Virtualization of the Linux Operating System means the ability to run multiple instances of Operating Systems concurrently on a single hardware system where the basic resources are driven by a Linux instance. The library aims at providing a long term stable C API . It currently supports Xen, QEmu, KVM , LXC , OpenVZ, VirtualBox and VMware ESX .

  First started with minimal CentOS 7.6 installation. Depending on the Datacenter used that can be achieved on many ways. Hetzner has their auto installer so this part was almost automatized. 

  **Step [1] - Checking hardware compatibility -**

  Next thing we need to make sure that Virtualization Technology (VT) is enabled in your server’s BIOS.

  To test from shell:
  ```
  [root@CentOS-76-64-minimal ~]# lscpu | grep Virtualization
  Virtualization:        VT-x
  [root@CentOS-76-64-minimal ~]# hostname
  CentOS-76-64-minimal
  [root@CentOS-76-64-minimal ~]# date
  Mon Apr 22 06:16:43 CEST 2019
  ```
  If you get the empty response you will probably need to ask your provider to get that option enabled or if available do that by yourself on site, using IPMI or etc.

  Step [2] - YUM and installing packages -

  ```
  yum update
  ```
  Restart server and then push:

  ```
  yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
  ```

  Start and enable the libvirtd service:
  ```
  systemctl enable libvirtd
  ```
  ```
  systemctl start libvirtd
  ```

  **Step [2] - Verify kvm installation -**

  Make sure KVM module loaded using lsmod command and grep command:
  ```
  lsmod | grep -i kvm
  ```

  Valid output is:
  ```
  [root@CentOS-76-64-minimal ~]# lsmod | grep -i kvm
  kvm_intel             183705  38 
  kvm                   615914  1 kvm_intel
  irqbypass              13503  7 kvm
  ```

  **Step [3] - Configure bridged networking** - <BR>
  By default DHCP based network bridge configured by libvirtd. Well, what is the bridge?

  A bridge is a device that separates two or more network segments within one logical network (e.g. a single IP-subnet).

  A bridge is usually placed between two separate groups of computers that talk with each other, but not that much with the computers in the other group. A good example of this is to consider a cluster of Macintoshes and a cluster of Unix machines. Both of these groups of machines tend to be quite chatty amongst themselves, and the traffic they produce on the network causes collisions for the other machines who are trying to speak to one another.

  The job of the bridge is to examine the destination of the data packets one at a time and decide whether or not to pass the packets to the other side of the Ethernet segment. The result is a faster, quieter network with fewer collisions.

  The bridging code decides whether to bridge data or to drop it not by looking at the protocol type (IP, IPX, NetBEUI), but by looking at the MAC-address unique to each NIC.

  Important: It's vital to understand that a bridge is neither a router nor a firewall. Spoken in simple term a bridge behaves like a network switch (i.e. Layer 2 Switch), making it a transparent network component (which is not absolutely true, but nearly).

  You can verify that with the following commands:
  ```
  brctl show
  ```
  ```
  virsh net-list
  ```

  Output:
  ```
  [root@CentOS-76-64-minimal ~]# brctl show
  bridge name	bridge id		STP enabled	interfaces
  br0		8000.b42e99481ead	no		eno1
  							vnet0
  							vnet1
  							vnet2
  virbr0		8000.525400535849	yes		virbr0-nic

  [root@CentOS-76-64-minimal ~]# virsh net-list
   Name                 State      Autostart     Persistent
  ----------------------------------------------------------
   default              active     yes           yes
  ```

  Now, I already have some virtual machines on this server so it is normal to have a br0 device. Otherwise, it is perfectly fine to have virbr0 or "Virtual Bridge 0" interface is used for NAT (Network Address Translation). It is provided by the libvirt library, and virtual environments sometimes use it to connect to the outside network.

  All VMs (guest machine) only have network access to other VMs on the same server. A private network 192.168.122.0/24 created for you. Verify it:
  ```
  virsh net-dumpxml default
  ```

  The normal output is:
  ```
  [root@CentOS-76-64-minimal ~]# virsh net-dumpxml default
  <network>
    <name>default</name>
    <uuid>0fc9b244-06f5-42d4-a7d1-09fe4bc6f14e</uuid>
    <forward mode='nat'>
      <nat>
        <port start='1024' end='65535'/>
      </nat>
    </forward>
    <bridge name='virbr0' stp='on' delay='0'/>
    <mac address='52:54:00:53:58:49'/>
    <ip address='192.168.122.1' netmask='255.255.255.0'>
      <dhcp>
        <range start='192.168.122.2' end='192.168.122.254'/>
      </dhcp>
    </ip>
  </network>
  ```
  Note: You can change/adjust this range to whatever you like by editing network ranges or default MAC address (if needed) by using net-edit command. Here is a useful doc page from libvirt site on how you can use it: https://libvirt.org/sources/virshcmdref/html/sect-net-edit.html

  If you want your VMs available to other servers on your LAN, set up a network bridge on the server that connected to your LAN. Now, this part is critical. 

  1) First what I personally advise is to disable NetworkManager. It usually overwrites the file within /etc/sysconfig/network-scripts which we need to manually modify so we actually don't need that service on our kvm/qemu host.

  ```
  systemctl stop NetworkManager
  ```
  ```
  systemctl disable NetworkManager
  ```

  Now, cd into /etc/sysconfig/network-scripts folder and simple list of files.

  ```
  [root@CentOS-76-64-minimal network-scripts]# ls
  ifcfg-br0   ifdown-bnep  ifdown-ipv6  ifdown-routes  ifup-aliases  ifup-ippp  ifup-plusb   ifup-sit          network-functions
  ifcfg-eno1  ifdown-eth   ifdown-isdn  ifdown-sit     ifup-bnep     ifup-ipv6  ifup-post    ifup-tunnel       network-functions-ipv6
  ifcfg-lo    ifdown-ib    ifdown-post  ifdown-tunnel  ifup-eth      ifup-isdn  ifup-ppp     ifup-wireless     route-br0
  ifdown      ifdown-ippp  ifdown-ppp   ifup           ifup-ib       ifup-plip  ifup-routes  init.ipv6-global
  ```

  In my example with Hetzner server, we had ifcfg-eno1 interface which needs to be modified. Based on my experiences I've chosen MAC Driven Linux bridge method which I am going to explain soon.

  ```
  [root@CentOS-76-64-minimal network-scripts]# cat ifcfg-eno1 
  ### Hetzner Online GmbH installimage
  # device: eth0
  DEVICE=eno1
  BOOTPROTO=static
  HWADDR=b4:2e:99:48:1e:ad
  ONBOOT=yes
  BRIDGE=br0
  ```

  And edited/created ifcfg-br0 file with following content:
  ```
  [root@CentOS-76-64-minimal network-scripts]# cat ifcfg-br0 
  DEVICE=br0
  TYPE="Bridge"
  BOOTPROTO=static
  IPADDR=78.46.89.244
  NETMASK=255.255.255.224
  SCOPE="peer 78.46.89.225"
  ONBOOT=yes
  DELAY=0
  ```

  Vital points here:
   - HWADDR describes real MAC so you need to enter real MAC within eno1 (or eth0) network configuration file. <BR>
   - BRIDGE=br0 is added so Bridge can communicate with the outside world through the eno1 port. <BR>
   - SCOPE is added which is peer to the gateway port. In the case of Hetzner AG DC, it was needed. <BR>

  And lastly from the networking part, there is route-br0 file configured in this way:
  ```
  [root@CentOS-76-64-minimal network-scripts]# cat route-br0 
  ### Hetzner Online GmbH installimage

  ADDRESS0=0.0.0.0
  NETMASK0=0.0.0.0
  GATEWAY0=78.46.89.225
  ```
  Note: Again, there is a gateway0 part which you need to adjust as per your data center settings.

  **Step [4] - Create your first virtual machine -**

  I am going to create a CentOS 7.x VM. First, grab CentOS 7.x latest ISO image using the wget command:

  ```
  cd /var/lib/libvirt/boot/
  ```
  ```
  wget -c https://mirrors.edge.kernel.org/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-Minimal-1810.iso
  ```
  Note: You can visit https://mirrors.edge.kernel.org/centos link with browser to select desired CentOS7 version or any other you wish to install.

  In my next example, as done on the live/production server I am going to deploy the first virtual machine. 

  ```
  virt-install --virt-type=kvm --name server.radio-uzivo.com --ram 32768 --vcpus=16 --os-variant=centos7.0 --cdrom=/var/lib/libvirt/boot/CentOS-7-x86_64-Minimal-1810.iso --network=bridge=br0,model=virtio --graphics vnc --disk path=/var/lib/libvirt/images/centos7.qcow2,size=300,bus=virtio,format=qcow2
  ```
  Note: To elaborate. This is a 32gb ram virtual machine with 16 CPUs and 300gb of HDD.
  --ram **32768** <BR>
  --vcpus=**16**
  path=/var/lib/libvirt/images/centos7.qcow2,size=**300**,bus=virtio,format=qcow2

  Rest is really self-explanatory and can be modified as per specific needs. 

  While pushed that command, keep the screen (Ssh terminal running), open new tab and let's see what VNC port is assigned, so we can finish this installation from VNC GUI style.

  ```
  virsh dumpxml server.radio-uzivo.com | grep vnc 
  ```

  The desired output is:
  ```
  [root@CentOS-76-64-minimal network-scripts]# virsh dumpxml server.radio-uzivo.com | grep vnc
      <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
  ```

  Now, that we know these facts you can sign off from the same terminal, load another one and login to server using SSH Tunnel:

  ```
  ssh root@x.x.x.x -L 5900:127.0.0.1:5900
  ```
  Note: where x.x.x.x needs to be the real IP address of your's server.

  Keep the terminal running (1st and 2nd), load favorite VNC Viewer tool (I am using TigerVNC on Linux, but you can load any), and connect to the GUI interface using 127.0.0.1:5900 address.

  Initial screen (example):
  ![alt](/content/images/2019/04/Screenshot_2019-04-22_07-06-06.png)

  Using mouse please finish partitioning and networking part and to define dummy/plain password which you can use later to configure networking within the virtual machine. Usually, it is fine to put 123456 as a password at this point so you can easily type it on the next reboot. Finish installation and reboot the virtual machine (server).

  Login as root user from VNC interface, and let's configure networking there

  Network (on the virtual machine):
  1) systemctl stop NetworkManager
  2) systemctl disable NetworkManager
  3) [root@server network-scripts]# cat ifcfg-eth0 
  TYPE="Ethernet"
  BOOTPROTO="none"
  DEFROUTE="yes"
  DEVICE="eth0"
  NAME="eth0"
  IPADDR="x.x.x.x"
  PREFIX="24"
  GATEWAY="x.x.x.x"
  HWADDR="00:50:56:00:96:2E"
  NETMASK="255.255.255.224"
  (example - get MAC and additional IP data from Hetzner panel or your vendor/provider)

  Note:
  it is critical that you enter valid MAC address, IPADDR, and GATEWAY.

  Swap back to the Host SSH terminal. We will now configure MAC address there as well:

  ```
  virsh edit server.radio-uzivo.com
  ```

  Adjust real MAC address and save it (it is a vi editor so basically press : and x)

  Once you are back on shell (#), please execute:
  ```
  virsh destroy server.radio-uzivo.com
  ```
  ```
  virsh start server.radio-uzivo.com
  ```

  At this point, you may open a new ssh tab and test additional IP address by ping tool. If you get a response, you can now safely ssh into virtual machine from your local computer, configure new root password and finish installation of other applications needed.

  Free addon (tip). My good friend (https://apawel.me) who is experienced sysadmin recommended that before starting with libvirt and qemu installation grab a valid centos qemu ev package from http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm address. In this case the yum command line needs to be changed to:
  ```
  yum install qemu-kvm-ev virt-install libvirt cifs-utils
  ```

  Besides that, his recommendation is to enable KSM if you plan to overuse (oversell) hardware:
  ```
  echo 1 > /sys/kernel/mm/ksm/run
  ```
  Note: place line in the /etc/rc.local to make it permanent.

  ```
  echo 'vm.swappiness = 10' > /etc/sysctl.d/00-swappiness.conf
  ```

  In the next article we will focus on how to deploy virtual machines using Cloud-Init (https://cloudinit.readthedocs.io/en/latest/) method, so please stay tuned. I hope this article helps :) Cheers.

html:
  <p>Hello folks. Recently I was hired by my good friend to migrate his data from the old server which was running from 2016. Back then we installed OpenVZ platform with some freeware GUI and Webmin and deployed several Virtual machines on it. It was/is working perfectly, but I had to move it to something newer as technology grows. He rented a server from a known provider in Europe called Hetzner AG. The server seems to be powerful with Intel Skylake processor and NVME drives. I was pretty impressed until I've realized that I can't install Debian/Ubuntu on it and simply move forward with Proxmox as freeware KVM/Xen solution so I had to think of different ways to accomplish my mission. Started reading about the virsh tool which is using libvirt as a base (<a href="https://libvirt.org/">https://libvirt.org/</a>).</p>

  <p>The virsh program is the main interface for managing virsh guest domains. The program can be used to create, pause, and shutdown domains. It can also be used to list current domains. Libvirt is a C toolkit to interact with the virtualization capabilities of recent versions of Linux (and other OSes). It is free software available under the GNU Lesser General Public License. Virtualization of the Linux Operating System means the ability to run multiple instances of Operating Systems concurrently on a single hardware system where the basic resources are driven by a Linux instance. The library aims at providing a long term stable C API . It currently supports Xen, QEmu, KVM , LXC , OpenVZ, VirtualBox and VMware ESX .</p>

  <p>First started with minimal CentOS 7.6 installation. Depending on the Datacenter used that can be achieved on many ways. Hetzner has their auto installer so this part was almost automatized. </p>

  <p><strong>Step [1] - Checking hardware compatibility -</strong></p>

  <p>Next thing we need to make sure that Virtualization Technology (VT) is enabled in your server’s BIOS.</p>

  <p>To test from shell:  </p>

  <pre><code>[root@CentOS-76-64-minimal ~]# lscpu | grep Virtualization
  Virtualization:        VT-x  
  [root@CentOS-76-64-minimal ~]# hostname
  CentOS-76-64-minimal  
  [root@CentOS-76-64-minimal ~]# date
  Mon Apr 22 06:16:43 CEST 2019  
  </code></pre>

  <p>If you get the empty response you will probably need to ask your provider to get that option enabled or if available do that by yourself on site, using IPMI or etc.</p>

  <p>Step [2] - YUM and installing packages -</p>

  <pre><code>yum update  
  </code></pre>

  <p>Restart server and then push:</p>

  <pre><code>yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install  
  </code></pre>

  <p>Start and enable the libvirtd service:  </p>

  <pre><code>systemctl enable libvirtd  
  </code></pre>

  <pre><code>systemctl start libvirtd  
  </code></pre>

  <p><strong>Step [2] - Verify kvm installation -</strong></p>

  <p>Make sure KVM module loaded using lsmod command and grep command:  </p>

  <pre><code>lsmod | grep -i kvm  
  </code></pre>

  <p>Valid output is:  </p>

  <pre><code>[root@CentOS-76-64-minimal ~]# lsmod | grep -i kvm
  kvm_intel             183705  38  
  kvm                   615914  1 kvm_intel  
  irqbypass              13503  7 kvm  
  </code></pre>

  <p><strong>Step [3] - Configure bridged networking</strong> - <BR>
  By default DHCP based network bridge configured by libvirtd. Well, what is the bridge?</p>

  <p>A bridge is a device that separates two or more network segments within one logical network (e.g. a single IP-subnet).</p>

  <p>A bridge is usually placed between two separate groups of computers that talk with each other, but not that much with the computers in the other group. A good example of this is to consider a cluster of Macintoshes and a cluster of Unix machines. Both of these groups of machines tend to be quite chatty amongst themselves, and the traffic they produce on the network causes collisions for the other machines who are trying to speak to one another.</p>

  <p>The job of the bridge is to examine the destination of the data packets one at a time and decide whether or not to pass the packets to the other side of the Ethernet segment. The result is a faster, quieter network with fewer collisions.</p>

  <p>The bridging code decides whether to bridge data or to drop it not by looking at the protocol type (IP, IPX, NetBEUI), but by looking at the MAC-address unique to each NIC.</p>

  <p>Important: It's vital to understand that a bridge is neither a router nor a firewall. Spoken in simple term a bridge behaves like a network switch (i.e. Layer 2 Switch), making it a transparent network component (which is not absolutely true, but nearly).</p>

  <p>You can verify that with the following commands:  </p>

  <pre><code>brctl show  
  </code></pre>

  <pre><code>virsh net-list  
  </code></pre>

  <p>Output:  </p>

  <pre><code>[root@CentOS-76-64-minimal ~]# brctl show
  bridge name    bridge id       STP enabled interfaces  
  br0        8000.b42e99481ead   no      eno1  
                              vnet0
                              vnet1
                              vnet2
  virbr0        8000.525400535849   yes     virbr0-nic

  [root@CentOS-76-64-minimal ~]# virsh net-list
   Name                 State      Autostart     Persistent
  ----------------------------------------------------------
   default              active     yes           yes
  </code></pre>

  <p>Now, I already have some virtual machines on this server so it is normal to have a br0 device. Otherwise, it is perfectly fine to have virbr0 or "Virtual Bridge 0" interface is used for NAT (Network Address Translation). It is provided by the libvirt library, and virtual environments sometimes use it to connect to the outside network.</p>

  <p>All VMs (guest machine) only have network access to other VMs on the same server. A private network 192.168.122.0/24 created for you. Verify it:  </p>

  <pre><code>virsh net-dumpxml default  
  </code></pre>

  <p>The normal output is:  </p>

  <pre><code>[root@CentOS-76-64-minimal ~]# virsh net-dumpxml default
  &lt;network&gt;  
    &lt;name&gt;default&lt;/name&gt;
    &lt;uuid&gt;0fc9b244-06f5-42d4-a7d1-09fe4bc6f14e&lt;/uuid&gt;
    &lt;forward mode='nat'&gt;
      &lt;nat&gt;
        &lt;port start='1024' end='65535'/&gt;
      &lt;/nat&gt;
    &lt;/forward&gt;
    &lt;bridge name='virbr0' stp='on' delay='0'/&gt;
    &lt;mac address='52:54:00:53:58:49'/&gt;
    &lt;ip address='192.168.122.1' netmask='255.255.255.0'&gt;
      &lt;dhcp&gt;
        &lt;range start='192.168.122.2' end='192.168.122.254'/&gt;
      &lt;/dhcp&gt;
    &lt;/ip&gt;
  &lt;/network&gt;  
  </code></pre>

  <p>Note: You can change/adjust this range to whatever you like by editing network ranges or default MAC address (if needed) by using net-edit command. Here is a useful doc page from libvirt site on how you can use it: <a href="https://libvirt.org/sources/virshcmdref/html/sect-net-edit.html">https://libvirt.org/sources/virshcmdref/html/sect-net-edit.html</a></p>

  <p>If you want your VMs available to other servers on your LAN, set up a network bridge on the server that connected to your LAN. Now, this part is critical. </p>

  <p>1) First what I personally advise is to disable NetworkManager. It usually overwrites the file within /etc/sysconfig/network-scripts which we need to manually modify so we actually don't need that service on our kvm/qemu host.</p>

  <pre><code>systemctl stop NetworkManager  
  </code></pre>

  <pre><code>systemctl disable NetworkManager  
  </code></pre>

  <p>Now, cd into /etc/sysconfig/network-scripts folder and simple list of files.</p>

  <pre><code>[root@CentOS-76-64-minimal network-scripts]# ls
  ifcfg-br0   ifdown-bnep  ifdown-ipv6  ifdown-routes  ifup-aliases  ifup-ippp  ifup-plusb   ifup-sit          network-functions  
  ifcfg-eno1  ifdown-eth   ifdown-isdn  ifdown-sit     ifup-bnep     ifup-ipv6  ifup-post    ifup-tunnel       network-functions-ipv6  
  ifcfg-lo    ifdown-ib    ifdown-post  ifdown-tunnel  ifup-eth      ifup-isdn  ifup-ppp     ifup-wireless     route-br0  
  ifdown      ifdown-ippp  ifdown-ppp   ifup           ifup-ib       ifup-plip  ifup-routes  init.ipv6-global  
  </code></pre>

  <p>In my example with Hetzner server, we had ifcfg-eno1 interface which needs to be modified. Based on my experiences I've chosen MAC Driven Linux bridge method which I am going to explain soon.</p>

  <pre><code>[root@CentOS-76-64-minimal network-scripts]# cat ifcfg-eno1 
  ### Hetzner Online GmbH installimage
  # device: eth0
  DEVICE=eno1  
  BOOTPROTO=static  
  HWADDR=b4:2e:99:48:1e:ad  
  ONBOOT=yes  
  BRIDGE=br0  
  </code></pre>

  <p>And edited/created ifcfg-br0 file with following content:  </p>

  <pre><code>[root@CentOS-76-64-minimal network-scripts]# cat ifcfg-br0 
  DEVICE=br0  
  TYPE="Bridge"  
  BOOTPROTO=static  
  IPADDR=78.46.89.244  
  NETMASK=255.255.255.224  
  SCOPE="peer 78.46.89.225"  
  ONBOOT=yes  
  DELAY=0  
  </code></pre>

  <p>Vital points here: <br />
   - HWADDR describes real MAC so you need to enter real MAC within eno1 (or eth0) network configuration file. <BR>
   - BRIDGE=br0 is added so Bridge can communicate with the outside world through the eno1 port. <BR>
   - SCOPE is added which is peer to the gateway port. In the case of Hetzner AG DC, it was needed. <BR></p>

  <p>And lastly from the networking part, there is route-br0 file configured in this way:  </p>

  <pre><code>[root@CentOS-76-64-minimal network-scripts]# cat route-br0 
  ### Hetzner Online GmbH installimage

  ADDRESS0=0.0.0.0  
  NETMASK0=0.0.0.0  
  GATEWAY0=78.46.89.225  
  </code></pre>

  <p>Note: Again, there is a gateway0 part which you need to adjust as per your data center settings.</p>

  <p><strong>Step [4] - Create your first virtual machine -</strong></p>

  <p>I am going to create a CentOS 7.x VM. First, grab CentOS 7.x latest ISO image using the wget command:</p>

  <pre><code>cd /var/lib/libvirt/boot/  
  </code></pre>

  <pre><code>wget -c https://mirrors.edge.kernel.org/centos/7.6.1810/isos/x86_64/CentOS-7-x86_64-Minimal-1810.iso  
  </code></pre>

  <p>Note: You can visit <a href="https://mirrors.edge.kernel.org/centos">https://mirrors.edge.kernel.org/centos</a> link with browser to select desired CentOS7 version or any other you wish to install.</p>

  <p>In my next example, as done on the live/production server I am going to deploy the first virtual machine. </p>

  <pre><code>virt-install --virt-type=kvm --name server.radio-uzivo.com --ram 32768 --vcpus=16 --os-variant=centos7.0 --cdrom=/var/lib/libvirt/boot/CentOS-7-x86_64-Minimal-1810.iso --network=bridge=br0,model=virtio --graphics vnc --disk path=/var/lib/libvirt/images/centos7.qcow2,size=300,bus=virtio,format=qcow2  
  </code></pre>

  <p>Note: To elaborate. This is a 32gb ram virtual machine with 16 CPUs and 300gb of HDD. <br />
  --ram <strong>32768</strong> <BR>
  --vcpus=<strong>16</strong>
  path=/var/lib/libvirt/images/centos7.qcow2,size=<strong>300</strong>,bus=virtio,format=qcow2</p>

  <p>Rest is really self-explanatory and can be modified as per specific needs. </p>

  <p>While pushed that command, keep the screen (Ssh terminal running), open new tab and let's see what VNC port is assigned, so we can finish this installation from VNC GUI style.</p>

  <pre><code>virsh dumpxml server.radio-uzivo.com | grep vnc  
  </code></pre>

  <p>The desired output is:  </p>

  <pre><code>[root@CentOS-76-64-minimal network-scripts]# virsh dumpxml server.radio-uzivo.com | grep vnc
      &lt;graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'&gt;
  </code></pre>

  <p>Now, that we know these facts you can sign off from the same terminal, load another one and login to server using SSH Tunnel:</p>

  <pre><code>ssh root@x.x.x.x -L 5900:127.0.0.1:5900  
  </code></pre>

  <p>Note: where x.x.x.x needs to be the real IP address of your's server.</p>

  <p>Keep the terminal running (1st and 2nd), load favorite VNC Viewer tool (I am using TigerVNC on Linux, but you can load any), and connect to the GUI interface using 127.0.0.1:5900 address.</p>

  <p>Initial screen (example): <br />
  <img src="/content/images/2019/04/Screenshot_2019-04-22_07-06-06.png" alt="alt" /></p>

  <p>Using mouse please finish partitioning and networking part and to define dummy/plain password which you can use later to configure networking within the virtual machine. Usually, it is fine to put 123456 as a password at this point so you can easily type it on the next reboot. Finish installation and reboot the virtual machine (server).</p>

  <p>Login as root user from VNC interface, and let's configure networking there</p>

  <p>Network (on the virtual machine): <br />
  1) systemctl stop NetworkManager <br />
  2) systemctl disable NetworkManager <br />
  3) [root@server network-scripts]# cat ifcfg-eth0 <br />
  TYPE="Ethernet" <br />
  BOOTPROTO="none" <br />
  DEFROUTE="yes" <br />
  DEVICE="eth0" <br />
  NAME="eth0" <br />
  IPADDR="x.x.x.x" <br />
  PREFIX="24" <br />
  GATEWAY="x.x.x.x" <br />
  HWADDR="00:50:56:00:96:2E" <br />
  NETMASK="255.255.255.224" <br />
  (example - get MAC and additional IP data from Hetzner panel or your vendor/provider)</p>

  <p>Note: <br />
  it is critical that you enter valid MAC address, IPADDR, and GATEWAY.</p>

  <p>Swap back to the Host SSH terminal. We will now configure MAC address there as well:</p>

  <pre><code>virsh edit server.radio-uzivo.com  
  </code></pre>

  <p>Adjust real MAC address and save it (it is a vi editor so basically press : and x)</p>

  <p>Once you are back on shell (#), please execute:  </p>

  <pre><code>virsh destroy server.radio-uzivo.com  
  </code></pre>

  <pre><code>virsh start server.radio-uzivo.com  
  </code></pre>

  <p>At this point, you may open a new ssh tab and test additional IP address by ping tool. If you get a response, you can now safely ssh into virtual machine from your local computer, configure new root password and finish installation of other applications needed.</p>

  <p>Free addon (tip). My good friend (<a href="https://apawel.me">https://apawel.me</a>) who is experienced sysadmin recommended that before starting with libvirt and qemu installation grab a valid centos qemu ev package from <a href="http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm">http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/centos-release-qemu-ev-1.0-1.el7.noarch.rpm</a> address. In this case the yum command line needs to be changed to:  </p>

  <pre><code>yum install qemu-kvm-ev virt-install libvirt cifs-utils  
  </code></pre>

  <p>Besides that, his recommendation is to enable KSM if you plan to overuse (oversell) hardware:  </p>

  <pre><code>echo 1 &gt; /sys/kernel/mm/ksm/run  
  </code></pre>

  <p>Note: place line in the /etc/rc.local to make it permanent.</p>

  <pre><code>echo 'vm.swappiness = 10' &gt; /etc/sysctl.d/00-swappiness.conf  
  </code></pre>

  <p>In the next article we will focus on how to deploy virtual machines using Cloud-Init (<a href="https://cloudinit.readthedocs.io/en/latest/">https://cloudinit.readthedocs.io/en/latest/</a>) method, so please stay tuned. I hope this article helps :) Cheers.</p>


uuid: 7a232600-0d98-465f-a911-510b657b92e7
title: How to fix Send a friend Magento 2 module for email relay compatibility
slug: how-to-fix-send-a-friend-magento-2-module-for-relay-compatibility
image: /content/images/2019/06/amity_refer_a_friend.png
status: published
language: en_US
markdown:
  To expand the awareness of your brand, Magento 2 allows inviting products to friends via email by setting up the Refer Email to a Friend in the configuration. For the easier way, Email a Friend link appears along with an envelope icon instantly on the product page. Just clicking on the link, customers are navigated to the Email to a Friend page, then complete all information of the email. It is possible to send to multiple recipients; however, to avoid the spam, you should limit the number of sharing products in one hour and the number of recipients per sent email.

  One of the major bottlenecks here is that Magento recommends use of the SMTP module to successfully send and inbox inviting email, but that never works. Simply, the module-send-friend is designed to place the FROM with the Customer account or Guest account (if enabled) details. Even with all valid SMTP settings (details) this is most likely going to fail when sending emails to any popular email exchange provider like Google, Yahoo, Hotmail, Yandex or similar. Mostly experienced System administrators prevent this kind of FROM cloaking. To make this things work I've did one dirty trick, and yes I know Magento developers are going to say "Never edit Core files", but as this is just an experiment and not permanent solution I don't care.

  I've figured that module is controlled from vendor/magento/module-send-friend/Model/SendFriend.php and that lines 179 and 180 controls this:

  ```
  $sender = [
  'name' => $this->_escaper->escapeHtml($this->getSender()->getName()),
  'email' => $this->_escaper->escapeHtml($this->getSender()->getEmail()),
          ];
  ```

  In my example I just edited these lines with the following:
  ```
  $sender = [
  'name' => $this->_escaper->escapeHtml($this->getSender()->getEmail()),
  'email' => 'no-reply@magentocommand.ml',
  //$this->_escaper->escapeHtml($this->getSender()->getEmail()),
          ];
  ```

  To explain. I've swapped the 'name' and 'email' places since it is not necessarry to have Name field repeated twice. You can freely test this functionality of my adjusted code on my Demo/Test page https://magentocommand.ml/hanna-sweater.html by clicking on the Email icon here:

  ![alt](/content/images/2019/06/Screenshot_2019-06-06_11-04-45.png)

  If you are using AmazonSES or any other popular service which enforce the policy that the FROM sender needs to be used as a domain name used this would be extremely helpful to avoid usage of the SMTP module. As I've explained previously mostly SMTP module part fails because SPF/DKIM/DMARC fails.

  Sample error:
  2019-06-05 17:41:29 1hYZuj-00003b-PW ** x@xxx.xxx R=send_via_ses T=ses_smtp H=xxxxx.com X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no: SMTP error from remote mail server after MAIL FROM:<> SIZE=3446: 501 Invalid MAIL FROM address provided

  Once module tweak is added:
  2019-06-05 17:49:32 1hYa2W-00003k-Fp => xx@xxx.com R=send_via_ses T=ses_smtp H=xxxx.com X=TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256 CV=no A=ses_login C="250 Ok 0100016b28c2a272-2250b0bf-d310-4e2a-b8c6-615259231294-000000"
  2019-06-05 17:49:32 1hYa2W-00003k-Fp Completed


  **How to setup Refer Email to a Friend in Magento 2** <BR>

  Follow this article to learn how to setup Refer Email to a Friend in Magento 2:

  On the Admin Panel, Stores > Settings > Configuration.

  On the left panel, Catalog > Email to a Friend.
  Open the Email Templates section,

  ![alt](/content/images/2019/06/how-setup-refer-email-friend-in-magento-2-email-template.png)


  For Email to a Friend Email Template, you can read here:
  https://devdocs.magento.com/guides/v2.3/frontend-dev-guide/templates/template-email.html

  The end results are working finally without SMTP module and as expected:
  ![alt](/content/images/2019/06/Screenshot_2019-06-06_11-20-13.png)

  The potential customer now can reply safely to the same or click on the link from Email template and browse the store.

  **ADVANCED WAY:**
  <BR>
  Let's admit that the code adjusted in the first place was the most easiest and probably not that recommended by general Magento standards so let's try now to pull the **trans_email/ident_support/email** data which is defined within Stores --> Configuration --> Store Email Address area, so I have edited two files:

  1) vendor/magento/module-send-friend/Helper/Data.php by adding:
  ```
  public function getConfig($config_path)
     {
         return $this->scopeConfig->getValue(
             $config_path,
             \Magento\Store\Model\ScopeInterface::SCOPE_STORE
         );
     }
  ```

  2) vendor/magento/module-send-friend/Model/SendFriend.php
  ```
  'email' => $this->_sendfriendData->getConfig('trans_email/ident_support/email'),
  ```
  (line 184)

  **THE ULTIMATE FINAL WAY** <BR>(ready for contribution submit to Magento):

  1) Edit vendor/magento/module-send-friend/Helper/Data.php and just after const CHECK_COOKIE =0; add:
  ```
  const XML_PATH_SENDER_EMAIL = 'trans_email/ident_support/email';

   /**
      * Get Sender Email
      *
      * @param int $store
      * @return string
      */

      public function getSenderEmail($store = null)
      {
          return $this->scopeConfig->getValue(
              self::XML_PATH_SENDER_EMAIL,
              \Magento\Store\Model\ScopeInterface::SCOPE_STORE,
              $store
          );
      }
  ```

  2) Edit vendor/magento/module-send-friend/Model/SendFriend.php
  ```
  'email' => $this->_sendfriendData->getSenderEmail(),
  ```
  (line 184)

  Hope this workaround helps.

html:
  <p>To expand the awareness of your brand, Magento 2 allows inviting products to friends via email by setting up the Refer Email to a Friend in the configuration. For the easier way, Email a Friend link appears along with an envelope icon instantly on the product page. Just clicking on the link, customers are navigated to the Email to a Friend page, then complete all information of the email. It is possible to send to multiple recipients; however, to avoid the spam, you should limit the number of sharing products in one hour and the number of recipients per sent email.</p>

  <p>One of the major bottlenecks here is that Magento recommends use of the SMTP module to successfully send and inbox inviting email, but that never works. Simply, the module-send-friend is designed to place the FROM with the Customer account or Guest account (if enabled) details. Even with all valid SMTP settings (details) this is most likely going to fail when sending emails to any popular email exchange provider like Google, Yahoo, Hotmail, Yandex or similar. Mostly experienced System administrators prevent this kind of FROM cloaking. To make this things work I've did one dirty trick, and yes I know Magento developers are going to say "Never edit Core files", but as this is just an experiment and not permanent solution I don't care.</p>

  <p>I've figured that module is controlled from vendor/magento/module-send-friend/Model/SendFriend.php and that lines 179 and 180 controls this:</p>

  <pre><code>$sender = [
  'name' =&gt; $this-&gt;_escaper-&gt;escapeHtml($this-&gt;getSender()-&gt;getName()),  
  'email' =&gt; $this-&gt;_escaper-&gt;escapeHtml($this-&gt;getSender()-&gt;getEmail()),  
          ];
  </code></pre>

  <p>In my example I just edited these lines with the following:  </p>

  <pre><code>$sender = [
  'name' =&gt; $this-&gt;_escaper-&gt;escapeHtml($this-&gt;getSender()-&gt;getEmail()),  
  'email' =&gt; 'no-reply@magentocommand.ml',  
  //$this-&gt;_escaper-&gt;escapeHtml($this-&gt;getSender()-&gt;getEmail()),
          ];
  </code></pre>

  <p>To explain. I've swapped the 'name' and 'email' places since it is not necessarry to have Name field repeated twice. You can freely test this functionality of my adjusted code on my Demo/Test page <a href="https://magentocommand.ml/hanna-sweater.html">https://magentocommand.ml/hanna-sweater.html</a> by clicking on the Email icon here:</p>

  <p><img src="/content/images/2019/06/Screenshot_2019-06-06_11-04-45.png" alt="alt" /></p>

  <p>If you are using AmazonSES or any other popular service which enforce the policy that the FROM sender needs to be used as a domain name used this would be extremely helpful to avoid usage of the SMTP module. As I've explained previously mostly SMTP module part fails because SPF/DKIM/DMARC fails.</p>

  <p>Sample error: <br />
  2019-06-05 17:41:29 1hYZuj-00003b-PW ** x@xxx.xxx R=send<em>via</em>ses T=ses<em>smtp H=xxxxx.com X=TLS1.2:ECDHE</em>RSA<em>AES</em>256<em>GCM</em>SHA384:256 CV=no: SMTP error from remote mail server after MAIL FROM:&lt;> SIZE=3446: 501 Invalid MAIL FROM address provided</p>

  <p>Once module tweak is added: <br />
  2019-06-05 17:49:32 1hYa2W-00003k-Fp => xx@xxx.com R=send<em>via</em>ses T=ses<em>smtp H=xxxx.com X=TLS1.2:ECDHE</em>RSA<em>AES</em>256<em>GCM</em>SHA384:256 CV=no A=ses_login C="250 Ok 0100016b28c2a272-2250b0bf-d310-4e2a-b8c6-615259231294-000000" <br />
  2019-06-05 17:49:32 1hYa2W-00003k-Fp Completed</p>

  <p><strong>How to setup Refer Email to a Friend in Magento 2</strong> <BR></p>

  <p>Follow this article to learn how to setup Refer Email to a Friend in Magento 2:</p>

  <p>On the Admin Panel, Stores > Settings > Configuration.</p>

  <p>On the left panel, Catalog > Email to a Friend. <br />
  Open the Email Templates section,</p>

  <p><img src="/content/images/2019/06/how-setup-refer-email-friend-in-magento-2-email-template.png" alt="alt" /></p>

  <p>For Email to a Friend Email Template, you can read here: <br />
  <a href="https://devdocs.magento.com/guides/v2.3/frontend-dev-guide/templates/template-email.html">https://devdocs.magento.com/guides/v2.3/frontend-dev-guide/templates/template-email.html</a></p>

  <p>The end results are working finally without SMTP module and as expected: <br />
  <img src="/content/images/2019/06/Screenshot_2019-06-06_11-20-13.png" alt="alt" /></p>

  <p>The potential customer now can reply safely to the same or click on the link from Email template and browse the store.</p>

  <p><strong>ADVANCED WAY:</strong>
  <BR> <br />
  Let's admit that the code adjusted in the first place was the most easiest and probably not that recommended by general Magento standards so let's try now to pull the <strong>trans<em>email/ident</em>support/email</strong> data which is defined within Stores --> Configuration --> Store Email Address area, so I have edited two files:</p>

  <p>1) vendor/magento/module-send-friend/Helper/Data.php by adding:  </p>

  <pre><code>public function getConfig($config_path)  
     {
         return $this-&gt;scopeConfig-&gt;getValue(
             $config_path,
             \Magento\Store\Model\ScopeInterface::SCOPE_STORE
         );
     }
  </code></pre>

  <p>2) vendor/magento/module-send-friend/Model/SendFriend.php  </p>

  <pre><code>'email' =&gt; $this-&gt;_sendfriendData-&gt;getConfig('trans_email/ident_support/email'),  
  </code></pre>

  <p>(line 184)</p>

  <p><strong>THE ULTIMATE FINAL WAY</strong> <BR>(ready for contribution submit to Magento):</p>

  <p>1) Edit vendor/magento/module-send-friend/Helper/Data.php and just after const CHECK_COOKIE =0; add:  </p>

  <pre><code>const XML_PATH_SENDER_EMAIL = 'trans_email/ident_support/email';

   /**
      * Get Sender Email
      *
      * @param int $store
      * @return string
      */

      public function getSenderEmail($store = null)
      {
          return $this-&gt;scopeConfig-&gt;getValue(
              self::XML_PATH_SENDER_EMAIL,
              \Magento\Store\Model\ScopeInterface::SCOPE_STORE,
              $store
          );
      }
  </code></pre>

  <p>2) Edit vendor/magento/module-send-friend/Model/SendFriend.php  </p>

  <pre><code>'email' =&gt; $this-&gt;_sendfriendData-&gt;getSenderEmail(),  
  </code></pre>

  <p>(line 184)</p>

  <p>Hope this workaround helps.</p>


uuid: d05ed308-dcb9-4fbc-86f8-cdd639295b5c
title: Backup and restore data (full and partially) on ProxMox based server
slug: backup-and-restore-data-full-and-partially-on-proxmox-based-server
image: /content/images/2019/07/backup.jpg
status: published
language: en_US
markdown:
  Hello,

  In this article, I will present how to organize Backups and restores on Proxmox based server with KVM/Qemu virtual machines installed. Every experienced sysadmin understands the need for quality and valid backups in case of disasters. So far, the only functional thing on the market is to extract data to a remote server. This can be done in various ways, but today I will present here how I did on one of my Proxmox based servers. The distance (remote) server is using C14 (https://www.online.net/en/storage/c14-cold-storage) which supports Rsync and FTP only connections. 

  For start, make sure that sshfs is installed and configured:
  ```
  apt-get install sshfs
  ```

  Create dir, for example,/c14backup/ and then mount new filesystem using sshfs fuse:
  ```
  sshfs -p 48843 c14ssh@xxxxxxxxx.buffer.c14.io:/buffer/ /c14backup/
  ```
  Explanation:
  -p is ssh port used.
  c14ssh is ssh username.
  From @ is SSH hostname split by
  :/buffer/ is the remote directory where data is going to be uploaded

  And, /c14backup/ is a local folder here where we are going to mount new filesystem.

  To check if everything is well mounted, please type:
  ```
  df -h
  ```

  In the output you should see SSH2FS FUSE filesystem, example:
  ![alt](/content/images/2019/07/Screenshot_2019-07-07_10-40-44.png)

  Now, that this is finished, let's jump to the Proxmox panel and create new storage. 

  Click on DATACENTER --> Storage --> Add --> Directory

  ![alt](/content/images/2019/07/Screenshot_2019-07-07_10-45-01.png)

  Explanation:
  ID: (enter a preferred ID, can be anything really)
  Directory: (Enter the freshly mounted SSH2FS FUSE filesystem located within /c14backup/ folder)
  Content: Open that tab and select everything offered.
  Max Backups: (Select preferred, by default, is 1 and  values should be raised)

  Next, you can safely open QEMU/KVM virtual instance, click on the BACKUP option, select storage as ID "c14" or whatever you wish and click on the BACKUP button further.

  ![alt](/content/images/2019/07/Screenshot_2019-07-07_10-49-23.png)

  At this point, you can safely explore /c14backup/ folder and see that dumps are created within /c14backup/dump/ directory. Your data is safely stored within remote server **while keeping QEMU/KVM virtual machine alive.**

  Of course, this can be automated from Terminal without need of GUI interface. Reference is https://pve.proxmox.com/wiki/Backup_and_Restore article with example:

  Use snapshot mode (no downtime) and non-default dump directory.

  ```
  vzdump 777 --dumpdir /mnt/backup --mode snapshot
  ```

  You can adjust /mnt/backup to /c14backup/ safely where 777 is QEMU/KVM ID.

  There are some tools with C14 OnlineSAS provider where you can create a bucket from the command line. Please check https://github.com/scaleway/c14-cli tool for further pieces of information.

  **Restore from LZO backup**

  Okay, now that we are having data on the remote server and they are safely residing there. I chose to do this via the command line, instead of the web interface.

  First of, make sure that there’s enough space on the server where you are going to restore. It will need to hold the vma file as well as the extracted image.. You can do that safely on FUSE SSH2FS remotely mounted filesystem, but ideally, because of the speed of local hard drive, you will wish to do that on the real hard drive.

  The command used is lzop:(https://www.lzop.org/lzop_man.php)

  lzop -d vzdump-qemu-xxxxxx.vma.lzo

  This will extract the vma file from the lzo archive. Change the file name to whatever your actual file name is. Note: Make sure you have again enough space. The differences between original lzo file and extracted can be greater.

  ![alt](/content/images/2019/07/Screenshot_2019-07-07_14-14-11.png)

  Next, once file extracted we are going to use vma tool (https://pve.proxmox.com/wiki/VMA) to extract the image from it.

  Example:
  vma extract vzdump-qemu-xxxx.vma -v /data/nemanja

  Note: Folder nemanja is going to be created automatically.

  This will extract the VM image file as well as the configuration to the directory /data/nemanja. You can change this path to whatever is appropriate. The directory restore will be automatically created.
  You can now mount this disk to the new QEMU/KVM virtual machine or mount a loopback device to extract data from it partially by using losetup tool.

  Linux supports a special block device called the loop device, which maps a normal file onto a virtual block device. This allows for the file to be used as a “virtual file system” inside another file. With Linux, it’s possible to create a file-system inside a single file. These storage devices are available as device files such as /dev/device_name.

  cd into the /data/nemanja folder. You will see disk-drive-scsi0.raw file or with the format original filesystem is using (COW2 for example).

  Next step is to create a loop device with the file. Use the command “losetup” to create a loop device “loop0”

  ```
  losetup -fP disk-drive-scsi0.raw
  ```

  Explanation:
  -f – find the first unused loop device. If a file argument is present, use this device. Otherwise, print its name.
  -P – force kernel to scan the partition table on a newly created loop device.

  To print the loop device generated using the above command use “losetup -a”. You can type losetup -l to list it as well.

  Now let's mount it:
  ```
  mkdir mount
  ```

  ```
  mount -o loop /dev/loop0p3 /data/nemanja/mount/ 
  ```

  Explanation:
  -o loop is to mount a loopback device
  - /dev/loop0 
  p1 (boot init)
  p2 (is used for swap)
  p3 (is usually root /home partition)

  ![alt](/content/images/2019/07/Screenshot_2019-07-07_17-21-25.png)

  Extract files from it rsync or whatever. Before the disk image can be detached again from the loop device, the mounted filesystem needs to be unmounted. Depending on whether the disk image is from a single partition or an entire disk, the device to be unmounted needs to be given, like in the mount command.

  ````
  umount /dev/loop0p3
  ```

  With the filesystem unmounted, the disk image can be detached from the loop device.

  $ sudo losetup -d /dev/loop0

  With the “-d” option, losetup is instructed to detach the specified loop device.

  Hope this article helps.

html:
  <p>Hello,</p>

  <p>In this article, I will present how to organize Backups and restores on Proxmox based server with KVM/Qemu virtual machines installed. Every experienced sysadmin understands the need for quality and valid backups in case of disasters. So far, the only functional thing on the market is to extract data to a remote server. This can be done in various ways, but today I will present here how I did on one of my Proxmox based servers. The distance (remote) server is using C14 (<a href="https://www.online.net/en/storage/c14-cold-storage">https://www.online.net/en/storage/c14-cold-storage</a>) which supports Rsync and FTP only connections. </p>

  <p>For start, make sure that sshfs is installed and configured:  </p>

  <pre><code>apt-get install sshfs  
  </code></pre>

  <p>Create dir, for example,/c14backup/ and then mount new filesystem using sshfs fuse:  </p>

  <pre><code>sshfs -p 48843 c14ssh@xxxxxxxxx.buffer.c14.io:/buffer/ /c14backup/  
  </code></pre>

  <p>Explanation: <br />
  -p is ssh port used.
  c14ssh is ssh username. <br />
  From @ is SSH hostname split by <br />
  :/buffer/ is the remote directory where data is going to be uploaded</p>

  <p>And, /c14backup/ is a local folder here where we are going to mount new filesystem.</p>

  <p>To check if everything is well mounted, please type:  </p>

  <pre><code>df -h  
  </code></pre>

  <p>In the output you should see SSH2FS FUSE filesystem, example: <br />
  <img src="/content/images/2019/07/Screenshot_2019-07-07_10-40-44.png" alt="alt" /></p>

  <p>Now, that this is finished, let's jump to the Proxmox panel and create new storage. </p>

  <p>Click on DATACENTER --> Storage --> Add --> Directory</p>

  <p><img src="/content/images/2019/07/Screenshot_2019-07-07_10-45-01.png" alt="alt" /></p>

  <p>Explanation: <br />
  ID: (enter a preferred ID, can be anything really) <br />
  Directory: (Enter the freshly mounted SSH2FS FUSE filesystem located within /c14backup/ folder) <br />
  Content: Open that tab and select everything offered. <br />
  Max Backups: (Select preferred, by default, is 1 and  values should be raised)</p>

  <p>Next, you can safely open QEMU/KVM virtual instance, click on the BACKUP option, select storage as ID "c14" or whatever you wish and click on the BACKUP button further.</p>

  <p><img src="/content/images/2019/07/Screenshot_2019-07-07_10-49-23.png" alt="alt" /></p>

  <p>At this point, you can safely explore /c14backup/ folder and see that dumps are created within /c14backup/dump/ directory. Your data is safely stored within remote server <strong>while keeping QEMU/KVM virtual machine alive.</strong></p>

  <p>Of course, this can be automated from Terminal without need of GUI interface. Reference is <a href="https://pve.proxmox.com/wiki/Backup">https://pve.proxmox.com/wiki/Backup</a><em>and</em>Restore article with example:</p>

  <p>Use snapshot mode (no downtime) and non-default dump directory.</p>

  <pre><code>vzdump 777 --dumpdir /mnt/backup --mode snapshot  
  </code></pre>

  <p>You can adjust /mnt/backup to /c14backup/ safely where 777 is QEMU/KVM ID.</p>

  <p>There are some tools with C14 OnlineSAS provider where you can create a bucket from the command line. Please check <a href="https://github.com/scaleway/c14-cli">https://github.com/scaleway/c14-cli</a> tool for further pieces of information.</p>

  <p><strong>Restore from LZO backup</strong></p>

  <p>Okay, now that we are having data on the remote server and they are safely residing there. I chose to do this via the command line, instead of the web interface.</p>

  <p>First of, make sure that there’s enough space on the server where you are going to restore. It will need to hold the vma file as well as the extracted image.. You can do that safely on FUSE SSH2FS remotely mounted filesystem, but ideally, because of the speed of local hard drive, you will wish to do that on the real hard drive.</p>

  <p>The command used is lzop:(<a href="https://www.lzop.org/lzop_man.php">https://www.lzop.org/lzop_man.php</a>)</p>

  <p>lzop -d vzdump-qemu-xxxxxx.vma.lzo</p>

  <p>This will extract the vma file from the lzo archive. Change the file name to whatever your actual file name is. Note: Make sure you have again enough space. The differences between original lzo file and extracted can be greater.</p>

  <p><img src="/content/images/2019/07/Screenshot_2019-07-07_14-14-11.png" alt="alt" /></p>

  <p>Next, once file extracted we are going to use vma tool (<a href="https://pve.proxmox.com/wiki/VMA">https://pve.proxmox.com/wiki/VMA</a>) to extract the image from it.</p>

  <p>Example: <br />
  vma extract vzdump-qemu-xxxx.vma -v /data/nemanja</p>

  <p>Note: Folder nemanja is going to be created automatically.</p>

  <p>This will extract the VM image file as well as the configuration to the directory /data/nemanja. You can change this path to whatever is appropriate. The directory restore will be automatically created. <br />
  You can now mount this disk to the new QEMU/KVM virtual machine or mount a loopback device to extract data from it partially by using losetup tool.</p>

  <p>Linux supports a special block device called the loop device, which maps a normal file onto a virtual block device. This allows for the file to be used as a “virtual file system” inside another file. With Linux, it’s possible to create a file-system inside a single file. These storage devices are available as device files such as /dev/device_name.</p>

  <p>cd into the /data/nemanja folder. You will see disk-drive-scsi0.raw file or with the format original filesystem is using (COW2 for example).</p>

  <p>Next step is to create a loop device with the file. Use the command “losetup” to create a loop device “loop0”</p>

  <pre><code>losetup -fP disk-drive-scsi0.raw  
  </code></pre>

  <p>Explanation: <br />
  -f – find the first unused loop device. If a file argument is present, use this device. Otherwise, print its name.
  -P – force kernel to scan the partition table on a newly created loop device.</p>

  <p>To print the loop device generated using the above command use “losetup -a”. You can type losetup -l to list it as well.</p>

  <p>Now let's mount it:  </p>

  <pre><code>mkdir mount  
  </code></pre>

  <pre><code>mount -o loop /dev/loop0p3 /data/nemanja/mount/  
  </code></pre>

  <p>Explanation: <br />
  -o loop is to mount a loopback device
  - /dev/loop0 
  p1 (boot init) <br />
  p2 (is used for swap) <br />
  p3 (is usually root /home partition)</p>

  <p><img src="/content/images/2019/07/Screenshot_2019-07-07_17-21-25.png" alt="alt" /></p>

  <p>Extract files from it rsync or whatever. Before the disk image can be detached again from the loop device, the mounted filesystem needs to be unmounted. Depending on whether the disk image is from a single partition or an entire disk, the device to be unmounted needs to be given, like in the mount command.</p>

  <pre><code class="language-`">umount /dev/loop0p3  
  </code></pre>

  <p>With the filesystem unmounted, the disk image can be detached from the loop device.</p>

  <p>$ sudo losetup -d /dev/loop0</p>

  <p>With the “-d” option, losetup is instructed to detach the specified loop device.</p>

  <p>Hope this article helps.</p>


uuid: 6005d1ad-eee3-4e69-9acd-785d4de7376f
title: Speed up Magento 2 JavaScript delivery using baler method
slug: optimize-magento-2-store-using-baler-method
image: /content/images/2019/10/magentoBundlingRealWorld.png
status: published
language: en_US
markdown:
  While working at the current job I see many Magento stores are affected by performance issues. Design and a lot of different functionalities can look great, but all that does not matter if you cannot deliver fast rendering to the end-customer who is, by the way, the potential buyer. There are so many different vendors focusing on proper JavaScript bundling, but recently I've tested https://github.com/magento/baler, and I wanted here to share my insights.

  In short, baler is an AMD (https://requirejs.org/) module bundler and preloader for Magento 2 stores.

  **How Does it Work?** <BR>
  baler has two different strategies for optimizing JavaScript delivery

  **Core Bundle** <BR>
  The "core" bundle contains the code that baler can determine is critical to the first render of the page. This includes the Magento core libraries, along with some theme-specific and feature-specific code.

  **Graph Preloading** <BR>
  The JavaScript included in the "core" bundle only includes dependencies that can be statically-analyzed from a theme's requirejs-config.js. However, many parts of the Magento front-end are controlled by widgets that are specified using a declarative notation in .phtml files.

  This is where Graph Preloading comes in. When baler is run, it crawls the file system and determines which .phtml are eligible to be used in a specific area (frontend/adminhtml/base). These templates are then analyzed for any AMD module dependencies.

  When a graph of dependencies has been collected for each .phtml file, the lists are flattened and deduped against the "core" bundle. Then, when a shopper requests a page of your store, preload tags are injected that instruct the browser to immediately begin fetching necessary dependencies.

  I've tested on latest Magento 2.3.3 CE version, and you can check page load results here https://magentocommand.ml

  It does what it says in manual, creating one core-bundle.js containing all possible aspects from core side which can be bundled safely.

  Example:
  https://magentocommand.ml/static/version1571521932/frontend/Magento/luma/en_US/balerbundles/core-bundle.js

  **What is JavaScript bundling?** <BR>
  **Javascript** bundling is a technique that groups separate files in order to reduce the number of HTTP requests that are required to load a page. Bundling is commonly used in today’s “module-based” development where some functionalities are basically split into Modules (roughly explained). 

  So let me show today how I did it on Fresh Magento 2.3.3 CE version with default Luma theme.

  1) We need to clone current Baler repository:
  ```
  git clone https://github.com/magento/baler.git
  ```
  2) You can cd into baler folder and install it using npm
  ```
  npm install
  ```
  3) Optionally, if you don't wish to get anything installed on your local server there is already ready made binary file within bin/ folder named baler.

  4) Make sure that you have node.js version >= 10.12.0 installed.
  ```
  ~/baler/bin$ ./baler 
  baler requires a version of node.js >= 10.12.0. You're currently using v8.16.0
  ```

  This can be fixed easily by downloading latest Node version:
  ```
  wget -c https://nodejs.org/dist/v10.14.1/node-v10.14.1-linux-x64.tar.gz
  ```
  Then let's unpack archive:
  ```
  tar -xvf node-v10.14.1-linux-x64.tar.gz
  ```

  Once extracted we can export active PATH to new Node installation:
  ```
  export PATH=/srv/baler/node-v10.14.1-linux-x64/bin:$PATH
  ```
  Mine direct path was /srv/baler/node-v10.14.1-linux-x64/bin but you can use pwd command to get real path and configure it.

  In case you receive the following error:
  ```
   ./baler --help

  Error: Cannot find module '../dist/cli/initializeCLI'
  at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)
  at Function.Module._load (internal/modules/cjs/loader.js:507:25)
  at Module.require (internal/modules/cjs/loader.js:637:17)
  at require (internal/modules/cjs/helpers.js:22:18)
  at Object.<anonymous> (/srv/baler/bin/baler:39:1)
  at Module._compile (internal/modules/cjs/loader.js:689:30)
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)
  at Module.load (internal/modules/cjs/loader.js:599:32)
  at tryModuleLoad (internal/modules/cjs/loader.js:538:12)
  at Function.Module._load (internal/modules/cjs/loader.js:530:3)
  ```

  Fix is to run:
  ```
  npm run build
  ```

  To test once this is installed:
  ```
  ~/baler/bin$ ./baler --help
  Usage
    $ baler <command> [options]

    Commands
      build --theme Vendor/name
      graph --theme Vendor/name

    Examples
      Optimize all eligible themes
      $ baler build

      Optimize multiple themes
      $ baler build --theme Magento/foo --theme Magento/bar

      Generate Dependency Graph
      $ baler graph --theme Magento/luma
  ```

  Next jump to the Magento 2 docroot and run it:
  ```
  :~/public_html$ /srv/baler/bin/baler 
  ✔ Collected theme/module data 155ms
  ✔ (Magento/luma) Created core bundle file 1.2s
  ✔ (Magento/luma) Minified core bundle and RequireJS 17.6s

  Optimization is done, but stats have not been implemented in the CLI yet
  ```

  Clear Magento and Redis (if installed) cache:
  ```
  ~/public_html$ n98-magerun2 cache:clean
  config cache cleaned
  layout cache cleaned
  block_html cache cleaned
  collections cache cleaned
  reflection cache cleaned
  db_ddl cache cleaned
  compiled_config cache cleaned
  eav cache cleaned
  customer_notification cache cleaned
  config_integration cache cleaned
  config_integration_api cache cleaned
  google_product cache cleaned
  full_page cache cleaned
  config_webservice cache cleaned
  translate cache cleaned
  vertex cache cleaned
  ```
  ```
  ~/public_html$ n98-magerun2 cache:flush
  redis-cli flushconfig cache flushed
  layout cache flushed
  block_html cache flushed
  collections cache flushed
  reflection cache flushed
  db_ddl cache flushed
  compiled_config cache flushed
  eav cache flushed
  customer_notification cache flushed
  config_integration cache flushed
  config_integration_api cache flushed
  google_product cache flushed
  full_page cache flushed
  config_webservice cache flushed
  translate cache flushed
  vertex cache flushed
  ```
  ```
  ~/public_html$ redis-cli flushall
  OK
  ```
  (this one is optional if Redis used)


  Next, we need to install Magento 2 module which is going to handle bundlings:
  ```
  composer config repositories.baler vcs https://github.com/adifucan/m2-baler
  ```

  Then run:
  ```
  composer require magento/module-baler:dev-master
  ```

  Activate module:
  ```
  n98-magerun2 module:enable Magento_Baler
  ```

  Run deployment process (setup:upgrade ; setup:di:compile and setup:static-content:deploy)
  ```
  n98-magerun2 setup:upgrade
  ```
  ```
  n98-magerun2 setup:di:compile
  ```
  ```
  n98-magerun2 setup:static-content:deploy
  ```

  Clear Magento/Redis cache like we did it upper and then push the following:
  ```
  php bin/magento config:set dev/js/enable_baler_js_bundling 1
  ```

  Run the baler once again:
  ```
  $ /srv/baler/bin/baler build 
  ✔ Collected theme/module data 156ms
  ✔ (Magento/luma) Created core bundle file 1.2s
  ✔ (Magento/luma) Minified core bundle and RequireJS 17.8s

  Optimization is done, but stats have not been implemented in the CLI yet
  ```

  Then we need to disable Magento 2 built in JavaScript Minify/Merge/Bundling:
  ```
  php bin/magento config:set dev/js/minify_files 0
  ```
  ```
  php bin/magento config:set dev/js/enable_js_bundling 0
  ```
  ```
  php bin/magento config:set dev/js/merge_files 0
  ```

  Clear Magento/Redis cache once again (yeah I know boring...), load frontpage and test... 

  **Addon:** <BR>
  While this is still in Alpha testing, there are certain things that needs to be patched. You can read them here https://github.com/DrewML/baler/issues/6 

  If you just want to apply the fixes, see the patch in core here https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574

  That's it! Most pages in your storefront should now be faster than they were before.

  Hope this article helps. I strongly encourage you before optimizing your store to check https://drewml.com/posts/improving-javascript-delivery-in-magento2/ excellent blog post about what is Baler and what is the current project state.

html:
  <p>While working at the current job I see many Magento stores are affected by performance issues. Design and a lot of different functionalities can look great, but all that does not matter if you cannot deliver fast rendering to the end-customer who is, by the way, the potential buyer. There are so many different vendors focusing on proper JavaScript bundling, but recently I've tested <a href="https://github.com/magento/baler">https://github.com/magento/baler</a>, and I wanted here to share my insights.</p>

  <p>In short, baler is an AMD (<a href="https://requirejs.org/">https://requirejs.org/</a>) module bundler and preloader for Magento 2 stores.</p>

  <p><strong>How Does it Work?</strong> <BR>
  baler has two different strategies for optimizing JavaScript delivery</p>

  <p><strong>Core Bundle</strong> <BR>
  The "core" bundle contains the code that baler can determine is critical to the first render of the page. This includes the Magento core libraries, along with some theme-specific and feature-specific code.</p>

  <p><strong>Graph Preloading</strong> <BR>
  The JavaScript included in the "core" bundle only includes dependencies that can be statically-analyzed from a theme's requirejs-config.js. However, many parts of the Magento front-end are controlled by widgets that are specified using a declarative notation in .phtml files.</p>

  <p>This is where Graph Preloading comes in. When baler is run, it crawls the file system and determines which .phtml are eligible to be used in a specific area (frontend/adminhtml/base). These templates are then analyzed for any AMD module dependencies.</p>

  <p>When a graph of dependencies has been collected for each .phtml file, the lists are flattened and deduped against the "core" bundle. Then, when a shopper requests a page of your store, preload tags are injected that instruct the browser to immediately begin fetching necessary dependencies.</p>

  <p>I've tested on latest Magento 2.3.3 CE version, and you can check page load results here <a href="https://magentocommand.ml">https://magentocommand.ml</a></p>

  <p>It does what it says in manual, creating one core-bundle.js containing all possible aspects from core side which can be bundled safely.</p>

  <p>Example: <br />
  <a href="https://magentocommand.ml/static/version1571521932/frontend/Magento/luma/en_US/balerbundles/core-bundle.js">https://magentocommand.ml/static/version1571521932/frontend/Magento/luma/en_US/balerbundles/core-bundle.js</a></p>

  <p><strong>What is JavaScript bundling?</strong> <BR>
  <strong>Javascript</strong> bundling is a technique that groups separate files in order to reduce the number of HTTP requests that are required to load a page. Bundling is commonly used in today’s “module-based” development where some functionalities are basically split into Modules (roughly explained). </p>

  <p>So let me show today how I did it on Fresh Magento 2.3.3 CE version with default Luma theme.</p>

  <p>1) We need to clone current Baler repository:  </p>

  <pre><code>git clone https://github.com/magento/baler.git  
  </code></pre>

  <p>2) You can cd into baler folder and install it using npm  </p>

  <pre><code>npm install  
  </code></pre>

  <p>3) Optionally, if you don't wish to get anything installed on your local server there is already ready made binary file within bin/ folder named baler.</p>

  <p>4) Make sure that you have node.js version >= 10.12.0 installed.  </p>

  <pre><code>~/baler/bin$ ./baler 
  baler requires a version of node.js &gt;= 10.12.0. You're currently using v8.16.0  
  </code></pre>

  <p>This can be fixed easily by downloading latest Node version:  </p>

  <pre><code>wget -c https://nodejs.org/dist/v10.14.1/node-v10.14.1-linux-x64.tar.gz  
  </code></pre>

  <p>Then let's unpack archive:  </p>

  <pre><code>tar -xvf node-v10.14.1-linux-x64.tar.gz  
  </code></pre>

  <p>Once extracted we can export active PATH to new Node installation:  </p>

  <pre><code>export PATH=/srv/baler/node-v10.14.1-linux-x64/bin:$PATH  
  </code></pre>

  <p>Mine direct path was /srv/baler/node-v10.14.1-linux-x64/bin but you can use pwd command to get real path and configure it.</p>

  <p>In case you receive the following error:  </p>

  <pre><code> ./baler --help

  Error: Cannot find module '../dist/cli/initializeCLI'  
  at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15)  
  at Function.Module._load (internal/modules/cjs/loader.js:507:25)  
  at Module.require (internal/modules/cjs/loader.js:637:17)  
  at require (internal/modules/cjs/helpers.js:22:18)  
  at Object.&lt;anonymous&gt; (/srv/baler/bin/baler:39:1)  
  at Module._compile (internal/modules/cjs/loader.js:689:30)  
  at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10)  
  at Module.load (internal/modules/cjs/loader.js:599:32)  
  at tryModuleLoad (internal/modules/cjs/loader.js:538:12)  
  at Function.Module._load (internal/modules/cjs/loader.js:530:3)  
  </code></pre>

  <p>Fix is to run:  </p>

  <pre><code>npm run build  
  </code></pre>

  <p>To test once this is installed:  </p>

  <pre><code>~/baler/bin$ ./baler --help
  Usage  
    $ baler &lt;command&gt; [options]

    Commands
      build --theme Vendor/name
      graph --theme Vendor/name

    Examples
      Optimize all eligible themes
      $ baler build

      Optimize multiple themes
      $ baler build --theme Magento/foo --theme Magento/bar

      Generate Dependency Graph
      $ baler graph --theme Magento/luma
  </code></pre>

  <p>Next jump to the Magento 2 docroot and run it:  </p>

  <pre><code>:~/public_html$ /srv/baler/bin/baler 
  ✔ Collected theme/module data 155ms
  ✔ (Magento/luma) Created core bundle file 1.2s
  ✔ (Magento/luma) Minified core bundle and RequireJS 17.6s

  Optimization is done, but stats have not been implemented in the CLI yet  
  </code></pre>

  <p>Clear Magento and Redis (if installed) cache:  </p>

  <pre><code>~/public_html$ n98-magerun2 cache:clean
  config cache cleaned  
  layout cache cleaned  
  block_html cache cleaned  
  collections cache cleaned  
  reflection cache cleaned  
  db_ddl cache cleaned  
  compiled_config cache cleaned  
  eav cache cleaned  
  customer_notification cache cleaned  
  config_integration cache cleaned  
  config_integration_api cache cleaned  
  google_product cache cleaned  
  full_page cache cleaned  
  config_webservice cache cleaned  
  translate cache cleaned  
  vertex cache cleaned  
  </code></pre>

  <pre><code>~/public_html$ n98-magerun2 cache:flush
  redis-cli flushconfig cache flushed  
  layout cache flushed  
  block_html cache flushed  
  collections cache flushed  
  reflection cache flushed  
  db_ddl cache flushed  
  compiled_config cache flushed  
  eav cache flushed  
  customer_notification cache flushed  
  config_integration cache flushed  
  config_integration_api cache flushed  
  google_product cache flushed  
  full_page cache flushed  
  config_webservice cache flushed  
  translate cache flushed  
  vertex cache flushed  
  </code></pre>

  <pre><code>~/public_html$ redis-cli flushall
  OK  
  </code></pre>

  <p>(this one is optional if Redis used)</p>

  <p>Next, we need to install Magento 2 module which is going to handle bundlings:  </p>

  <pre><code>composer config repositories.baler vcs https://github.com/adifucan/m2-baler  
  </code></pre>

  <p>Then run:  </p>

  <pre><code>composer require magento/module-baler:dev-master  
  </code></pre>

  <p>Activate module:  </p>

  <pre><code>n98-magerun2 module:enable Magento_Baler  
  </code></pre>

  <p>Run deployment process (setup:upgrade ; setup:di:compile and setup:static-content:deploy)  </p>

  <pre><code>n98-magerun2 setup:upgrade  
  </code></pre>

  <pre><code>n98-magerun2 setup:di:compile  
  </code></pre>

  <pre><code>n98-magerun2 setup:static-content:deploy  
  </code></pre>

  <p>Clear Magento/Redis cache like we did it upper and then push the following:  </p>

  <pre><code>php bin/magento config:set dev/js/enable_baler_js_bundling 1  
  </code></pre>

  <p>Run the baler once again:  </p>

  <pre><code>$ /srv/baler/bin/baler build 
  ✔ Collected theme/module data 156ms
  ✔ (Magento/luma) Created core bundle file 1.2s
  ✔ (Magento/luma) Minified core bundle and RequireJS 17.8s

  Optimization is done, but stats have not been implemented in the CLI yet  
  </code></pre>

  <p>Then we need to disable Magento 2 built in JavaScript Minify/Merge/Bundling:  </p>

  <pre><code>php bin/magento config:set dev/js/minify_files 0  
  </code></pre>

  <pre><code>php bin/magento config:set dev/js/enable_js_bundling 0  
  </code></pre>

  <pre><code>php bin/magento config:set dev/js/merge_files 0  
  </code></pre>

  <p>Clear Magento/Redis cache once again (yeah I know boring...), load frontpage and test... </p>

  <p><strong>Addon:</strong> <BR>
  While this is still in Alpha testing, there are certain things that needs to be patched. You can read them here <a href="https://github.com/DrewML/baler/issues/6">https://github.com/DrewML/baler/issues/6</a> </p>

  <p>If you just want to apply the fixes, see the patch in core here <a href="https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574">https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574</a></p>

  <p>That's it! Most pages in your storefront should now be faster than they were before.</p>

  <p>Hope this article helps. I strongly encourage you before optimizing your store to check <a href="https://drewml.com/posts/improving-javascript-delivery-in-magento2/">https://drewml.com/posts/improving-javascript-delivery-in-magento2/</a> excellent blog post about what is Baler and what is the current project state.</p>


uuid: e23dc0e4-3b9d-4e2f-944d-5dba83c30c4c
title: Project legacy work
slug: project-legacy-work
image: /content/images/2019/12/1545346423_virtualbox6.jpg
status: published
language: en_US
markdown:
  Hello,

  Recently I've been doing something really outside of my regular interests and activities, and it was really exciting so I wanted to share the story. My dad and mom are proud owners of a small gift shop. Mainly they produce gift items by themself now, but 10 years ago they were having multiple gift shops, warehouses, and selling the entire line of imported products mainly from China and Turkey. Back then I was in charge of the IT department. Their accounting/retail software is Unix based relational database system called Progress, now called OpenEdge Progress (https://documentation.progress.com/output/ua/OpenEdge_latest/index.html#page/gsdev/openedge-rdbms.html). To have at least some GUI with that terminal-based Unix system I've made two desktop computers basically networked together through LAN where one was Unix server hosting database and the other one was Windows XP hosting other applications. It was really stable solution, but to be able to host one monitor, keyboard and mouse I've had to use KVM Cable [2x PS/2 plug, VGA plug - 1x VGA plug, USB 1.1 connector A] 3 m Grey Digitus, something like well explained here https://en.wikipedia.org/wiki/KVM_switch

  So basically the retail POS system required 2 Desktop computers networked with a lot of cables. The system was working stable for more than 2 decades (20 years yep!), and now I've decided to make some enhancements.

  What I did is:
  1) Created ISO file using a Linux dd tool (https://www.geeksforgeeks.org/dd-command-linux/). dd is a command-line utility for Unix and Unix-like operating systems whose primary purpose is to convert and copy files. Since my fork on one of the older laptops (Acer Travelmate) had RedHat 9 Linux on it with the functional database I've plugged USB disk to it, created ISO file and saved ISO on the USB disk.

  Command used was:
  dd if=/dev/hda of=linux.iso bs=2048

  “bs” specifies the block size for both the input and output files. So dd uses 2048bytes as a block size in the above command.

  The process looked like this:
  ![alt](/content/images/2019/12/IMG_20191222_175626.jpg)

  When data was copied, I've powered down my old laptop computer and booted newer Acer Travelmate Aspire 5520 with 4gb of RAM and new SSD disk installed on Windows 10 operating system.

  The next thing was to copy linux.iso file from USB Disk to the Windows laptop and install VirtualBox (https://www.virtualbox.org/wiki/Downloads) latest version on it. 

  1) Open command prompt
  2) Navigate to the c:\Program Files\VirtualBox folder

  In a Terminal window on your run the following command:
  ```
  VBoxManage convertfromraw linux.iso linux.vdmk
  ```
  note: Make sure that you copy linux.iso to the Virtualbox folder or use full paths when converting or how you desire.

  Once that is done and vdmk file created visit VirtualBox manager icon and start creating Virtual Machine. My example requires RedHat type system 32bit so I have configured that way, but theoretically, you may use the same to convert any standard Linux distribution.

  There are two critical parts to make it work:
  ![alt](/content/images/2019/12/Screenshot_1.png)
  ![alt](/content/images/2019/12/Screenshot_2.png)

  Since the project required that Virtual Machine communicates with COM1 port and Prolific USB to Serial device I've had to enable it using the above settings. The reason for the Serial COM1 requirement is because the Serbian government enforces usage of "special" thermal printers who have specific chips which they use to track how many bills are printed and how much revenue your shop had.

  ![alt](/content/images/2019/12/IMG-61cac33ef9ef7be5b30f5f64bdec01da-V.jpg)

  The fun part to make this work on RedHat is actually to setup Serial correctly. The serial hardware often has only a limited number of IRQs it can be set at. Also, you don't want IRQ conflicts. So there may not be much of a choice. Your PC may normally come with ttyS0 and ttyS2 at IRQ 4, and ttyS1 and ttyS3 at IRQ 3. Looking at /proc/interrupts will show which IRQs are being used by programs currently running. You likely don't want to use one of these. Before IRQ 5 was used for sound cards, it was often used for a serial port.

  To get this adapter installed on Windows 10 OS you need to visit http://www.prolific.com.tw/US/ShowProduct.aspx?p_id=225&pcid=41 download and install drivers.

  Command used:
  ```
  /sbin/setserial /dev/ttyS0 irq 3
  ```

  To test if data is going from my Freshly booted Redhat Virtual machine I've used following command from Terminal:
  ```
  echo -ne "Nemanja is King" > /dev/ttyS0
  ```

  To test I have downloaded and installed https://www.eltima.com/com-port-monitor-download.html tool on the windows machine to monitor output and to make sure that COM1 serial from Virtual Machine back to Windows and USB adapter is working as expected.

  Video at the end of this article explains:
  https://www.youtube.com/watch?v=HTyDuOy_rzM

  Two systems in one:
  ![alt](/content/images/2019/12/IMG-4fde8cebd3d5f5d6eb1c0df5e1ad7c98-V.jpg)

  <div class="tenor-gif-embed" data-postid="5717550" data-share-method="host" data-width="100%" data-aspect-ratio="1.3863636363636365"><a href="https://tenor.com/view/cat-gif-5717550">Cat GIF</a> from <a href="https://tenor.com/search/cat-gifs">Cat GIFs</a></div><script type="text/javascript" async src="https://tenor.com/embed.js"></script>

  A bit unfinished story here... I will send another video/pictures to demonstrate how this looks in realtime (real shop) working with real Thermal POS printer.

  Thank you for reading this.
  Nemanja

html:
  <p>Hello,</p>

  <p>Recently I've been doing something really outside of my regular interests and activities, and it was really exciting so I wanted to share the story. My dad and mom are proud owners of a small gift shop. Mainly they produce gift items by themself now, but 10 years ago they were having multiple gift shops, warehouses, and selling the entire line of imported products mainly from China and Turkey. Back then I was in charge of the IT department. Their accounting/retail software is Unix based relational database system called Progress, now called OpenEdge Progress (<a href="https://documentation.progress.com/output/ua/OpenEdge">https://documentation.progress.com/output/ua/OpenEdge</a><em>latest/index.html#page/gsdev/openedge-rdbms.html). To have at least some GUI with that terminal-based Unix system I've made two desktop computers basically networked together through LAN where one was Unix server hosting database and the other one was Windows XP hosting other applications. It was really stable solution, but to be able to host one monitor, keyboard and mouse I've had to use KVM Cable [2x PS/2 plug, VGA plug - 1x VGA plug, USB 1.1 connector A] 3 m Grey Digitus, something like well explained here <a href="https://en.wikipedia.org/wiki/KVM">https://en.wikipedia.org/wiki/KVM</a></em>switch</p>

  <p>So basically the retail POS system required 2 Desktop computers networked with a lot of cables. The system was working stable for more than 2 decades (20 years yep!), and now I've decided to make some enhancements.</p>

  <p>What I did is: <br />
  1) Created ISO file using a Linux dd tool (<a href="https://www.geeksforgeeks.org/dd-command-linux/">https://www.geeksforgeeks.org/dd-command-linux/</a>). dd is a command-line utility for Unix and Unix-like operating systems whose primary purpose is to convert and copy files. Since my fork on one of the older laptops (Acer Travelmate) had RedHat 9 Linux on it with the functional database I've plugged USB disk to it, created ISO file and saved ISO on the USB disk.</p>

  <p>Command used was: <br />
  dd if=/dev/hda of=linux.iso bs=2048</p>

  <p>“bs” specifies the block size for both the input and output files. So dd uses 2048bytes as a block size in the above command.</p>

  <p>The process looked like this: <br />
  <img src="/content/images/2019/12/IMG_20191222_175626.jpg" alt="alt" /></p>

  <p>When data was copied, I've powered down my old laptop computer and booted newer Acer Travelmate Aspire 5520 with 4gb of RAM and new SSD disk installed on Windows 10 operating system.</p>

  <p>The next thing was to copy linux.iso file from USB Disk to the Windows laptop and install VirtualBox (<a href="https://www.virtualbox.org/wiki/Downloads">https://www.virtualbox.org/wiki/Downloads</a>) latest version on it. </p>

  <p>1) Open command prompt <br />
  2) Navigate to the c:\Program Files\VirtualBox folder</p>

  <p>In a Terminal window on your run the following command:  </p>

  <pre><code>VBoxManage convertfromraw linux.iso linux.vdmk  
  </code></pre>

  <p>note: Make sure that you copy linux.iso to the Virtualbox folder or use full paths when converting or how you desire.</p>

  <p>Once that is done and vdmk file created visit VirtualBox manager icon and start creating Virtual Machine. My example requires RedHat type system 32bit so I have configured that way, but theoretically, you may use the same to convert any standard Linux distribution.</p>

  <p>There are two critical parts to make it work: <br />
  <img src="/content/images/2019/12/Screenshot_1.png" alt="alt" />
  <img src="/content/images/2019/12/Screenshot_2.png" alt="alt" /></p>

  <p>Since the project required that Virtual Machine communicates with COM1 port and Prolific USB to Serial device I've had to enable it using the above settings. The reason for the Serial COM1 requirement is because the Serbian government enforces usage of "special" thermal printers who have specific chips which they use to track how many bills are printed and how much revenue your shop had.</p>

  <p><img src="/content/images/2019/12/IMG-61cac33ef9ef7be5b30f5f64bdec01da-V.jpg" alt="alt" /></p>

  <p>The fun part to make this work on RedHat is actually to setup Serial correctly. The serial hardware often has only a limited number of IRQs it can be set at. Also, you don't want IRQ conflicts. So there may not be much of a choice. Your PC may normally come with ttyS0 and ttyS2 at IRQ 4, and ttyS1 and ttyS3 at IRQ 3. Looking at /proc/interrupts will show which IRQs are being used by programs currently running. You likely don't want to use one of these. Before IRQ 5 was used for sound cards, it was often used for a serial port.</p>

  <p>To get this adapter installed on Windows 10 OS you need to visit <a href="http://www.prolific.com.tw/US/ShowProduct.aspx?p_id=225&amp;pcid=41">http://www.prolific.com.tw/US/ShowProduct.aspx?p_id=225&amp;pcid=41</a> download and install drivers.</p>

  <p>Command used:  </p>

  <pre><code>/sbin/setserial /dev/ttyS0 irq 3
  </code></pre>

  <p>To test if data is going from my Freshly booted Redhat Virtual machine I've used following command from Terminal:  </p>

  <pre><code>echo -ne "Nemanja is King" &gt; /dev/ttyS0  
  </code></pre>

  <p>To test I have downloaded and installed <a href="https://www.eltima.com/com-port-monitor-download.html">https://www.eltima.com/com-port-monitor-download.html</a> tool on the windows machine to monitor output and to make sure that COM1 serial from Virtual Machine back to Windows and USB adapter is working as expected.</p>

  <p>Video at the end of this article explains: <br />
  <a href="https://www.youtube.com/watch?v=HTyDuOy_rzM">https://www.youtube.com/watch?v=HTyDuOy_rzM</a></p>

  <p>Two systems in one: <br />
  <img src="/content/images/2019/12/IMG-4fde8cebd3d5f5d6eb1c0df5e1ad7c98-V.jpg" alt="alt" /></p>

  <p><div class="tenor-gif-embed" data-postid="5717550" data-share-method="host" data-width="100%" data-aspect-ratio="1.3863636363636365"><a href="https://tenor.com/view/cat-gif-5717550">Cat GIF</a> from <a href="https://tenor.com/search/cat-gifs">Cat GIFs</a></div><script type="text/javascript" async src="https://tenor.com/embed.js"></script></p>

  <p>A bit unfinished story here... I will send another video/pictures to demonstrate how this looks in realtime (real shop) working with real Thermal POS printer.</p>

  <p>Thank you for reading this. <br />
  Nemanja</p>


uuid: 7a53e2df-6ffe-44f9-9d4d-9df0107c7058
title: How to test/check headers behind CDN on Kubernetes/Docker Infrastructure
slug: how-to-test-check-headers-behind-cdn-on-kubernetes-docker-infrastructure
image: /content/images/2020/01/curl-logo.svg
status: published
language: en_US
markdown:
  Since this one is too hard to test by direct browser visit (you can see only 502 error from CloudFront or any other popular CDN provider), you can always test it directly from customer's SSH container, by the following pattern:

  curl -kI --header 'Host: domain.com' 'https://nginx/lifetime-fitness-sample/ma-ny-lifeguard.html'

  Original link for that one was:
  https://domain.com/lifetime-fitness-sample/ma-ny-lifeguard.html

  You can figure that we injected Host: method to the curl command and then called Nginx hostname directly.

  But what if the link is hidden behind certain username/password protection or authentication? 

  Not a problem at all. Please watch my video on how that can be achieved.

  Please watch the video here:
  https://vimeo.com/386681722

  Addon:
  For Windows users use the copy as cURL (cmd) option:
  ![alt](/content/images/2020/01/curl-windows.png)

  Ref article:
  https://curl.haxx.se/docs/manpage.html

html:
  <p>Since this one is too hard to test by direct browser visit (you can see only 502 error from CloudFront or any other popular CDN provider), you can always test it directly from customer's SSH container, by the following pattern:</p>

  <p>curl -kI --header 'Host: domain.com' '<a href="https://nginx/lifetime-fitness-sample/ma-ny-lifeguard.html">https://nginx/lifetime-fitness-sample/ma-ny-lifeguard.html</a>'</p>

  <p>Original link for that one was: <br />
  <a href="https://domain.com/lifetime-fitness-sample/ma-ny-lifeguard.html">https://domain.com/lifetime-fitness-sample/ma-ny-lifeguard.html</a></p>

  <p>You can figure that we injected Host: method to the curl command and then called Nginx hostname directly.</p>

  <p>But what if the link is hidden behind certain username/password protection or authentication? </p>

  <p>Not a problem at all. Please watch my video on how that can be achieved.</p>

  <p>Please watch the video here: <br />
  <a href="https://vimeo.com/386681722">https://vimeo.com/386681722</a></p>

  <p>Addon: <br />
  For Windows users use the copy as cURL (cmd) option: <br />
  <img src="/content/images/2020/01/curl-windows.png" alt="alt" /></p>

  <p>Ref article: <br />
  <a href="https://curl.haxx.se/docs/manpage.html">https://curl.haxx.se/docs/manpage.html</a></p>


uuid: e04eca4b-d83e-4101-9ffa-36f53ed46388
title: Speed up Magento 2 page load rendering using Magepack method
slug: speed-up-magento-2-page-load-rendering-using-magepack-method
image: /content/images/2020/04/image-1.png
status: published
language: en_US
markdown:
  Hello again!

  It's been a while since I haven't posted something. I wanted to share how-to and my personal experiences using advanced javascript bundling methods. This time we will focus on https://www.npmjs.com/package/magepack tool which is built on experiences gained with Advanced JavaScript bundling guide (https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html) and Baler (https://nemanja.io/optimize-magento-2-store-using-baler-method/) to provide the best of both worlds - ease of use and superior performance.

  Here are the requirements for Magepack to work:

  1) You need Node.js version 10 or higher installed.
  2) You need to have mixins.js module patched. I mainly apply the fix from https://github.com/integer-net/magento2-requirejs-bundling here.
  3) If you are using Magento 2.3.3 or lower, you need jquery.cookie module shim (patch provided and explained here). (optional) - https://github.com/magento/baler/issues/6
  Magepack Magento module installed. - https://github.com/magesuite/magepack-magento

  For test purposes, I've applied https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-perf-data.html to fresh Magento 2.3.4 CE using Medium profile.

  ```
  php bin/magento setup:perf:generate-fixtures ./setup/performance-toolkit/profiles/ce/medium.xml
  ```

  Results on first (cold cache load) are:
  ![alt](/content/images/2020/04/DeepinScreenshot_select-area_20200427164716.png)

  Then we need to proceed to Step 2. and apply fix for minins.js file:
  ```
  composer require cweagans/composer-patches
  ```
  You can always override PHP memory issues if they display with following command:
  ```
  /usr/local/bin/php -dmemory_limit=20000M /usr/local/bin/composer require cweagans/composer-patches
  ```
  update composer.json file with the data required as per your version, then push:
  ```
  /usr/local/bin/php -dmemory_limit=20000M /usr/local/bin/composer update
  ```

  Note: It is possible that one } is missing based on https://github.com/integer-net/magento2-requirejs-bundling document but you will be warned if missing:
  ```
    [Seld\JsonLint\ParsingException]               
    "./composer.json" does not contain valid JSON  
    Parse error on line 101:                       
    ...  }        }    }                           
    -------------------^                           
    Expected one of: '}', ',' 
  ```

  Example of composer.json extra area addon for Magento 2.3.4 CE version:
  ```
  "extra": {
          "magento-force": "override",
          "composer-exit-on-patch-failure": true,
          "patches": {
              "magento/magento2-base": {
                  "Refactor JavaScript mixins module": "https://raw.githubusercontent.com/integer-net/magento2-requirejs-bundling/master/patches/composer/M234/github-pr-25587-base.diff"
              }
          }
  }
      }
  ```

  Output:
  ![alt](/content/images/2020/04/DeepinScreenshot_select-area_20200427172222.png)

  Push setup:upgrade, setup:di:compile and setup:static-content:deploy

  Next, as per Step 4. we need to install https://github.com/magesuite/magepack-magento Magento 2 module:
  ```
  /usr/local/bin/php -dmemory_limit=20000M /usr/local/bin/composer require creativestyle/magesuite-magepack
  ```

  Now that 4 steps are finished we can start using this tool.

  Install with npm:
  npm install -g magepack

  If you receive following error on screen:
  ![alt](/content/images/2020/04/DeepinScreenshot_select-area_20200427182528.png)

  This can be fixed easily by downloading latest Node version:
  ```
  wget -c https://nodejs.org/dist/latest-v14.x/node-v14.0.0-linux-x64.tar.gz
  ```

  Then let's unpack the archive:
  ```
  tar -xvf node-v14.0.0-linux-x64.tar.gz
  ```

  Once extracted we can export active PATH to new Node installation:
  ```
  export PATH=/srv/node-v14.0.0-linux-x64/bin:$PATH
  ```

  Test:
  ![alt](/content/images/2020/04/DeepinScreenshot_select-area_20200427183654.png)

  The alternative way is to use NPM Manager, but this method works so we will continue using like in the Baler (https://nemanja.io/optimize-magento-2-store-using-baler-method/) case.

  Go back to the Magento 2 document root folder and re-run npm install -g magepack module again.

  The output should be like this:
  ![alt](/content/images/2020/04/DeepinScreenshot_select-area_20200427184036.png)

  The vendor How-To explain installation with Yarn. I will skip that part, but feel free to use it as well.

  Test magepack tool:
  ```
  $ magepack 
  Usage: magepack [generate|bundle] <options...>

  Options:
    -v, --version       Output the current version.
    -h, --help          Show this command summary.

  Commands:
    generate [options]  Generate optimization configuration based on given page URLs.
    bundle [options]    Bundle JavaScript files using given configuration file.
  ```

  Generating bundler configuration
  The first step is to run the generation against the existing, working shop. You can do it on any machine with access to the target shop, even your own computer. The goal here is to collect all of the RequireJS dependencies needed for a certain type of page layout. Currently, following bundles are prepared:

  cms containing modules needed by CMS pages.
  category containing modules needed by category pages.
  product containing modules needed by product pages.
  checkout containing modules needed by cart and checkout pages.
  In addition, there is the common bundle created by extracting all modules needed by each of above and loaded on every page.

  Running the generator:
  ```
  magepack generate --cms-url="{{CMS_PAGE_URL}}" --category-url="{{CATEGORY_PAGE_URL}}" --product-url="{{PRODUCT_PAGE_URL}}"
  ```

  There are 3 required options you need to pass:

  --cms-url - URL to one of CMS pages (e.g. homepage).
  --category-url - URL to one of the category pages.
  --product-url - URL to one of the product pages.

  Note: Megepack will use the given product page, add this product to the cart and visit both cart and checkout pages to collect dependencies.

  Running the above command will generate magepack.config.js file, where you can find each of the prepared bundles with the list of modules that will be included in them.

  1) Navigate to Magento 2 docroot and run the following command (example):
  ```
  magepack generate --cms-url="https://magentocommand.ml/" --category-url="https://magentocommand.ml/catalog/category/view/s/jackets/id/314/" --product-url="https://magentocommand.ml/proteus-fitness-jackshirt.html"
  ```

  The correct output is following:
  ![alt](/content/images/2020/05/DeepinScreenshot_select-area_20200512193656.png)

  **Bundling** <BR>
  Once you have generated bundler configuration, the next step would be to trigger the actual optimization after static content deploy stage has finished by running the following in shop root directory:

  magepack bundle
  This command will iterate over each deployed locale (excluding Magento/blank) and prepare bundles for each of them.

  Output:
  ![alt](/content/images/2020/05/DeepinScreenshot_select-area_20200512195452.png)

  Enabling
  Once you made sure Magepack Magento module is installed, what is left is to enable it via admin panel under Stores->Configuration->Advanced->Developer or CLI:

  ```
  php bin/magento config:set dev/js/enable_magepack_js_bundling 1
  ```

  and clearing the cache:
  ```
  php bin/magento cache:clean
  ```
  ```
  php bin/magento cache:flush
  ```

  If Redis/Varnish used you may need to purge it as well. And In the end, enjoy the fastest bundling method ever for Magento 2 frontends.

  Frontpage got only 36 requests on default Luma theme with no additional modules installed:
  ![alt](/content/images/2020/05/DeepinScreenshot_select-area_20200512200130.png)

  To know if bundler is working or not you will notice 3 Javascript files loaded:
  ![alt](/content/images/2020/05/DeepinScreenshot_select-area_20200512200221.png)

  Now scroll to the screenshot we made without Magepack installed.

  I hope this article was helpful. Happy Advanced bundling!

html:
  <p>Hello again!</p>

  <p>It's been a while since I haven't posted something. I wanted to share how-to and my personal experiences using advanced javascript bundling methods. This time we will focus on <a href="https://www.npmjs.com/package/magepack">https://www.npmjs.com/package/magepack</a> tool which is built on experiences gained with Advanced JavaScript bundling guide (<a href="https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html">https://devdocs.magento.com/guides/v2.3/performance-best-practices/advanced-js-bundling.html</a>) and Baler (<a href="https://nemanja.io/optimize-magento-2-store-using-baler-method/">https://nemanja.io/optimize-magento-2-store-using-baler-method/</a>) to provide the best of both worlds - ease of use and superior performance.</p>

  <p>Here are the requirements for Magepack to work:</p>

  <p>1) You need Node.js version 10 or higher installed. <br />
  2) You need to have mixins.js module patched. I mainly apply the fix from <a href="https://github.com/integer-net/magento2-requirejs-bundling">https://github.com/integer-net/magento2-requirejs-bundling</a> here. <br />
  3) If you are using Magento 2.3.3 or lower, you need jquery.cookie module shim (patch provided and explained here). (optional) - <a href="https://github.com/magento/baler/issues/6">https://github.com/magento/baler/issues/6</a> <br />
  Magepack Magento module installed. - <a href="https://github.com/magesuite/magepack-magento">https://github.com/magesuite/magepack-magento</a></p>

  <p>For test purposes, I've applied <a href="https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-perf-data.html">https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-perf-data.html</a> to fresh Magento 2.3.4 CE using Medium profile.</p>

  <pre><code>php bin/magento setup:perf:generate-fixtures ./setup/performance-toolkit/profiles/ce/medium.xml  
  </code></pre>

  <p>Results on first (cold cache load) are: <br />
  <img src="/content/images/2020/04/DeepinScreenshot_select-area_20200427164716.png" alt="alt" /></p>

  <p>Then we need to proceed to Step 2. and apply fix for minins.js file:  </p>

  <pre><code>composer require cweagans/composer-patches  
  </code></pre>

  <p>You can always override PHP memory issues if they display with following command:  </p>

  <pre><code>/usr/local/bin/php -dmemory_limit=20000M /usr/local/bin/composer require cweagans/composer-patches
  </code></pre>

  <p>update composer.json file with the data required as per your version, then push:  </p>

  <pre><code>/usr/local/bin/php -dmemory_limit=20000M /usr/local/bin/composer update
  </code></pre>

  <p>Note: It is possible that one } is missing based on <a href="https://github.com/integer-net/magento2-requirejs-bundling">https://github.com/integer-net/magento2-requirejs-bundling</a> document but you will be warned if missing:  </p>

  <pre><code>  [Seld\JsonLint\ParsingException]               
    "./composer.json" does not contain valid JSON  
    Parse error on line 101:                       
    ...  }        }    }                           
    -------------------^                           
    Expected one of: '}', ',' 
  </code></pre>

  <p>Example of composer.json extra area addon for Magento 2.3.4 CE version:  </p>

  <pre><code>"extra": {
          "magento-force": "override",
          "composer-exit-on-patch-failure": true,
          "patches": {
              "magento/magento2-base": {
                  "Refactor JavaScript mixins module": "https://raw.githubusercontent.com/integer-net/magento2-requirejs-bundling/master/patches/composer/M234/github-pr-25587-base.diff"
              }
          }
  }
      }
  </code></pre>

  <p>Output: <br />
  <img src="/content/images/2020/04/DeepinScreenshot_select-area_20200427172222.png" alt="alt" /></p>

  <p>Push setup:upgrade, setup:di:compile and setup:static-content:deploy</p>

  <p>Next, as per Step 4. we need to install <a href="https://github.com/magesuite/magepack-magento">https://github.com/magesuite/magepack-magento</a> Magento 2 module:  </p>

  <pre><code>/usr/local/bin/php -dmemory_limit=20000M /usr/local/bin/composer require creativestyle/magesuite-magepack
  </code></pre>

  <p>Now that 4 steps are finished we can start using this tool.</p>

  <p>Install with npm: <br />
  npm install -g magepack</p>

  <p>If you receive following error on screen: <br />
  <img src="/content/images/2020/04/DeepinScreenshot_select-area_20200427182528.png" alt="alt" /></p>

  <p>This can be fixed easily by downloading latest Node version:  </p>

  <pre><code>wget -c https://nodejs.org/dist/latest-v14.x/node-v14.0.0-linux-x64.tar.gz  
  </code></pre>

  <p>Then let's unpack the archive:  </p>

  <pre><code>tar -xvf node-v14.0.0-linux-x64.tar.gz  
  </code></pre>

  <p>Once extracted we can export active PATH to new Node installation:  </p>

  <pre><code>export PATH=/srv/node-v14.0.0-linux-x64/bin:$PATH  
  </code></pre>

  <p>Test: <br />
  <img src="/content/images/2020/04/DeepinScreenshot_select-area_20200427183654.png" alt="alt" /></p>

  <p>The alternative way is to use NPM Manager, but this method works so we will continue using like in the Baler (<a href="https://nemanja.io/optimize-magento-2-store-using-baler-method/">https://nemanja.io/optimize-magento-2-store-using-baler-method/</a>) case.</p>

  <p>Go back to the Magento 2 document root folder and re-run npm install -g magepack module again.</p>

  <p>The output should be like this: <br />
  <img src="/content/images/2020/04/DeepinScreenshot_select-area_20200427184036.png" alt="alt" /></p>

  <p>The vendor How-To explain installation with Yarn. I will skip that part, but feel free to use it as well.</p>

  <p>Test magepack tool:  </p>

  <pre><code>$ magepack 
  Usage: magepack [generate|bundle] &lt;options...&gt;

  Options:  
    -v, --version       Output the current version.
    -h, --help          Show this command summary.

  Commands:  
    generate [options]  Generate optimization configuration based on given page URLs.
    bundle [options]    Bundle JavaScript files using given configuration file.
  </code></pre>

  <p>Generating bundler configuration <br />
  The first step is to run the generation against the existing, working shop. You can do it on any machine with access to the target shop, even your own computer. The goal here is to collect all of the RequireJS dependencies needed for a certain type of page layout. Currently, following bundles are prepared:</p>

  <p>cms containing modules needed by CMS pages. <br />
  category containing modules needed by category pages. <br />
  product containing modules needed by product pages. <br />
  checkout containing modules needed by cart and checkout pages. <br />
  In addition, there is the common bundle created by extracting all modules needed by each of above and loaded on every page.</p>

  <p>Running the generator:  </p>

  <pre><code>magepack generate --cms-url="{{CMS_PAGE_URL}}" --category-url="{{CATEGORY_PAGE_URL}}" --product-url="{{PRODUCT_PAGE_URL}}"  
  </code></pre>

  <p>There are 3 required options you need to pass:</p>

  <p>--cms-url - URL to one of CMS pages (e.g. homepage).
  --category-url - URL to one of the category pages.
  --product-url - URL to one of the product pages.</p>

  <p>Note: Megepack will use the given product page, add this product to the cart and visit both cart and checkout pages to collect dependencies.</p>

  <p>Running the above command will generate magepack.config.js file, where you can find each of the prepared bundles with the list of modules that will be included in them.</p>

  <p>1) Navigate to Magento 2 docroot and run the following command (example):  </p>

  <pre><code>magepack generate --cms-url="https://magentocommand.ml/" --category-url="https://magentocommand.ml/catalog/category/view/s/jackets/id/314/" --product-url="https://magentocommand.ml/proteus-fitness-jackshirt.html"  
  </code></pre>

  <p>The correct output is following: <br />
  <img src="/content/images/2020/05/DeepinScreenshot_select-area_20200512193656.png" alt="alt" /></p>

  <p><strong>Bundling</strong> <BR>
  Once you have generated bundler configuration, the next step would be to trigger the actual optimization after static content deploy stage has finished by running the following in shop root directory:</p>

  <p>magepack bundle <br />
  This command will iterate over each deployed locale (excluding Magento/blank) and prepare bundles for each of them.</p>

  <p>Output: <br />
  <img src="/content/images/2020/05/DeepinScreenshot_select-area_20200512195452.png" alt="alt" /></p>

  <p>Enabling <br />
  Once you made sure Magepack Magento module is installed, what is left is to enable it via admin panel under Stores->Configuration->Advanced->Developer or CLI:</p>

  <pre><code>php bin/magento config:set dev/js/enable_magepack_js_bundling 1  
  </code></pre>

  <p>and clearing the cache:  </p>

  <pre><code>php bin/magento cache:clean  
  </code></pre>

  <pre><code>php bin/magento cache:flush  
  </code></pre>

  <p>If Redis/Varnish used you may need to purge it as well. And In the end, enjoy the fastest bundling method ever for Magento 2 frontends.</p>

  <p>Frontpage got only 36 requests on default Luma theme with no additional modules installed: <br />
  <img src="/content/images/2020/05/DeepinScreenshot_select-area_20200512200130.png" alt="alt" /></p>

  <p>To know if bundler is working or not you will notice 3 Javascript files loaded: <br />
  <img src="/content/images/2020/05/DeepinScreenshot_select-area_20200512200221.png" alt="alt" /></p>

  <p>Now scroll to the screenshot we made without Magepack installed.</p>

  <p>I hope this article was helpful. Happy Advanced bundling!</p>


uuid: 2d283d7b-a073-4e39-88b4-105552d68689
title: Increase SEO results using prerender method
slug: increase-seo-results-using-prerender-method
image: /content/images/2020/06/DeepinScreenshot_select-area_20200614215128.png
status: published
language: en_US
markdown:
  Hello,

  More often we hear that website dropped on popular search engines because of slowness. These things happen when we serve uncached content to the Search engine Bots while visiting pages and we get penalized because of that. Recently, I have been testing one amazing service called prerender.io

  Prerender is a node server that uses Headless Chrome to render HTML, screenshots, PDFs, and HAR files out of any web page. The Prerender server listens for an http request, takes the URL and loads it in Headless Chrome, waits for the page to finish loading by waiting for the network to be idle, and then returns your content.

  ![alt](/content/images/2020/06/image01.png)

  Google recommends that you use Prerender.io in their docs (https://developers.google.com/search/docs/guides/dynamic-rendering), so your site is guaranteed to be crawled correctly by Google & other search engines.

  It is fast, simple to use, and really straightforward. Vendor came with two solutions about how the application can be used:

  1) By using prerender.io servers as middleware.
  2) By installing your own server, and using it as a middleware in your's fleet.

  Let's start with first. Visit https://prerender.io website and register. Once that is done, you will be logged to their Control panel area.

  ![alt](/content/images/2020/06/image--20-.png)

  In https://prerender.io/documentation very detailed documentation pages vendor mention various ways to implement middleware between your application and this service, but since today we will make implementation for Magento 2 hosted on Nginx Web server, we will focus on that part.

  https://prerender.io/documentation/install-middleware#nginx
  Says that https://gist.github.com/thoop/8165802 having best way, but here is specific Nginx code that I've used to make this service integrated into present (functional) Nginx config file used for Magento 2 application

  ```
          set $prerender 0;
          if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
              set $prerender 1;
          }
          if ($args ~ "_escaped_fragment_") {
              set $prerender 1;
          }
          if ($http_user_agent ~ "Prerender") {
              set $prerender 0;
          }
          if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
              set $prerender 0;
          }
          
          #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
          resolver 8.8.8.8;
   
          if ($prerender = 1) {
          return 503;
          }

  error_page 500 502 503 504 =200 @prerender;

  location @prerender {            
          proxy_set_header X-Prerender-Token <ENTER TOKEN HERE>;
              #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
              set $prerender "service.prerender.io";
              rewrite .* /$scheme://$host$request_uri? break;
              proxy_pass http://$prerender;
          }
  ```

  From this point when above Nginx conf is added, you are redirecting traffic for Google and all bots mentioned in the example to service.prerender.io service. You may simulate traffic by executing following command from any popular SSH terminal:

  ```
  curl -IkL -A "Googlebot" https://magentocommand.ml
  ```

  The output would be 200 OK:
  ```
  > curl -IkL -A "Googlebot" https://magentocommand.ml
  HTTP/2 200 
  content-type: text/html;charset=UTF-8
  content-length: 178959
  vary: Accept-Encoding
  server: nginx
  date: Sun, 14 Jun 2020 19:38:58 GMT
  vary: Accept-Encoding
  access-control-allow-origin: *
  x-cache: Miss from cloudfront
  via: 1.1 27aa7ec4f54edf4b2fd5fffda84693a0.cloudfront.net (CloudFront)
  x-amz-cf-pop: SOF50-C1
  x-amz-cf-id: fmPB6u2Cqhv06tGTBvgaFDFaW9aJ8eGjGnvlFT2tEGtqgrCHrjdSJw==
  ```

  Visit https://prerender.io/ panel in your browser click on the Crawl stats at the top area (https://prerender.io/crawl-stats) and you will see simulated hit:

  ![alt](/content/images/2020/06/DeepinScreenshot_select-area_20200614214029.png)

  Within their panel, you may also view how RAW HTML page looks like by clicking on the Cached Pages option at the top menu.

  To be honest, if you are having a small website with not many SEF pages, the vendor offers 250 cached pages completely free, then next tier is 20,000 cached pages for $15.00 USD per month.

  There are several more small details about the implementation, but I did my best to make this as short as possible. We will jump now to 2nd method hosting your own prerender.io application.

  I've installed a very small Cloud Server with German data center Hetzner AG because it is cheap, stable, and close to my location, but ideally, you can install this application on any platform.

  1) As mentioned before, we will use Hetzner Cloud as our dedicated server. This guide was written using an instance with the Ubuntu image, but with a few little adjustments, it should also work with any popular Linux operating system.

  Regarding the instance type, I suggest starting with something small, get some metrics, and then scale if needed, in the test I've rented CX21 type who comes with 2 VCPU, 4Gb of RAM and 40Gb of NVME disk space. It should be okay to start off with. Finally, configure the security group to allow SSH connections and custom TCP inbounds on port 3000 as that is where your prerender service will work (you can choose your own port here). I usually do the /etc/hosts.allow method allowing only certain IP to gain access. You can go with a popular Firewalls like firewalld or something inbound from Data Center (or Cloud) provider.

  2) Setting up the environment:

  Connect to the instance and install the needed tools. To get the node, I recommend using nvm and choosing the latest LTS version.

  https://github.com/nvm-sh/nvm#git-install

  ```
  apt-get update && apt-get install git
  ```

  ```
  cd ~/ && git clone https://github.com/nvm-sh/nvm.git .nvm
  ```

  ```
  cd ~/.nvm && git checkout v0.35.3
  ```

  Now add these lines to your ~/.bashrc, ~/.profile, or ~/.zshrc file to have it automatically sourced upon login: (you may have to add to more than one of the above files)

  ```
  export NVM_DIR="$HOME/.nvm"
  [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
  [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion
  ```

  Sign off as SSH user, and log in again.

  At the moment of writing this article latest Node/NVM was 12.18.0 (LTS: Erbium), so we will install this one:
  ```
  nvm install 12.18.0
  ```

  Output:
  ![alt](/content/images/2020/06/DeepinScreenshot_select-area_20200614223002.png)

  After nvm and node, we need to get forever, which will allow us to run the prerender server continuously in the background.

  https://github.com/foreversd/forever

  ```
  npm install forever -g
  ```

  Output:
  ![alt](/content/images/2020/06/DeepinScreenshot_select-area_20200614223237.png)

  Next, we need to fetch the prerender source. To do this, clone the GitHub repo and run an npm install to get its dependencies.

  ```
  git clone https://github.com/prerender/prerender.git
  ```

  Next, we need to install Google Chrome.
  ```
  wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
  ```
  Installing packages on Ubuntu requires Sudo privileges. Make sure you are running the following command as a user with Sudo privileges.

  Install the Google Chrome .deb package by typing:
  ```
  apt install ./google-chrome-stable_current_amd64.deb
  ```

  Time to add first flag and modify /root/prerender/server.js file with the following code:
  ```
  var server = prerender({
    chromeFlags: ['--no-sandbox', '--headless', '--disable-gpu', '--remote-debugging-port=9222', '--hide-scrollbars']
  });
  ```

  The default one defaults to:
  ```
  var server = prerender({});
  ```

  A lot of other useful customizations can be added in this block based on https://github.com/prerender/prerender instructions.

  To test if all is fine, run following from /root/prerender/ directory:
  ```
  node server.js
  ```

  Output:
  ![alt](/content/images/2020/06/DeepinScreenshot_select-area_20200614230613.png)

  To make it running uninterrupted, we can use forever tool that we have previously installed and run this as follows:
  ```
  forever start /root/prerender/server.js
  ```

  Output:
  root@prerender-01:~/prerender# forever start /root/prerender/server.js 
  warn:    --minUptime not set. Defaulting to: 1000ms
  warn:    --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms
  info:    Forever processing file: /root/prerender/server.js

  You may check if the port is listening by executing lsof tool:
  ```
  lsof -i :3000
  ```

  Output:
  root@prerender-01:~/prerender# lsof -i :3000 
  COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
  node    9198 root   18u  IPv6  38348      0t0  TCP *:3000 (LISTEN)

  Now, that we are having Prerender server listening on port 3000, we can safely create systemd service so it can boot safely again in case of difficulties.

  Navigate to /etc/systemd/system folder and create prerender.service file with the following content:

  ```
  [Unit]
  Description=Prerender Service

  [Service]
  User=root
  WorkingDirectory=/root/prerender
  ExecStart=/root/.nvm/versions/node/v12.18.0/bin/node /root/prerender/server.js

  [Install]
  WantedBy=multi-user.target
  ```

  Then execute:
  systemctl enable prerender
  systemctl start prerender

  Check status now with systemctl status pretender. The output should be like this one:
  ![alt](/content/images/2020/06/DeepinScreenshot_select-area_20200614233539.png)

  That's all! Now you can create a DNS zone in your's external entity and start using Prerender service on your Web server by implementing the Middleware method from the start of this article.

  To just test out of the box if Prerender service is working you may execute the following:
  ```
  curl http://localhost:3000/render?url=https://magentocommand.ml
  ```

  One of the Alternative to track pages better is to install Nginx on the Prerender server and then using location / simply proxypass traffic to port 3000 while keeping that port enabled only to the localhost. That way, you may filter who is able and who is not to access Prerender service by using ports 80 or even more advanced with 443 (SSL).

  The next one would be to configure Prerender on port 80 (why not!) and use it. To do that add one line to the /etc/systemd/system/prerender.service file under [Service] tab:
  ```
  Environment=NODE_ENV=production PORT=80
  ```

  Optionally, you may add some Cache settings and Number of workers. To be honest, I am testing this on 2 CPU Virtual Private server so I had to limit number of workers to single to prevent OOM failures. To make caching even possible you need to install https://github.com/prerender/prerender-memory-cache plugin.

  ```
  Environment=NODE_ENV=production CACHE_MAXSIZE=100000
  Environment=NODE_ENV=production CACHE_TTL=432000
  Environment=NODE_ENV=production PRERENDER_NUM_WORKERS=1
  ```

  The complete file should look like this:
  ```
  [Unit]
  Description=Prerender Service

  [Service]
  User=root
  WorkingDirectory=/root/prerender
  Environment=NODE_ENV=production PORT=80
  Environment=NODE_ENV=production CACHE_MAXSIZE=100000
  Environment=NODE_ENV=production CACHE_TTL=432000
  Environment=NODE_ENV=production PRERENDER_NUM_WORKERS=1
  ExecStart=/root/.nvm/versions/node/v12.18.0/bin/node /root/prerender/server.js

  [Install]
  WantedBy=multi-user.target

  ```

  Extra addon. Generating HTML snapshots is a resource-intensive process, so some sort of caching strategy should be used to improve the performance. Prerender.io comes with several different caching plugins, but we will use the one based on Redis, as it offers scalability and a simple cache expiration mechanism. Redis can be built from source, but the pre-built version is sufficient for the task at hand.

  ```
  sudo apt-get update
  ```
  ```
  sudo apt-get upgrade
  ```
  ```
  sudo apt-get -y install redis-server
  ```

  Open /etc/redis/redis.conf file and make configuration is configured properly. For security measures I just uncomment bind line to bind to only 127.0.0.1 IP address or let's say localhost. Rest details can be adjusted per requirements/needs.

  Download the Redis cache plugin and install it. Currently, it caches the pages for one day and then expires them. This can be overridden by specifying the env variable process.env.PAGE_TTL in seconds. To never expire, you should set the PAGE_TTL variable to 0.

  ```
  npm install prerender-redis-cache --save
  ```

  Change the server.js to use the redis cache plugin:
  ```
  server.use(require('prerender-redis-cache'));
  ```

  Needs to be added here:
  ![alt](/content/images/2020/06/DeepinScreenshot_select-area_20200615132313.png)

  Good to add (optional) is access.log feature and keep the access logs - it can be useful for debugging and other maintenance tasks.

  https://github.com/unDemian/prerender-access-log
  ```
  npm install prerender-access-log --save
  ```

  Initialize the plugin in the server.js. You also need to configure the access log settings. Here is the finished version of the server.js:
  ```
  #!/usr/bin/env node
  var prerender = require('./lib');

  var server = prerender({
    accessLog: {
          // Check out the file-stream-rotator docs for parameters
          fileStreamRotator: {
              filename: '/var/log/prerender/access-%DATE%.log',
              frequency: 'daily',
              date_format: 'YYYY-MM-DD',
              verbose: false
          },

          // Check out the morgan docs for the available formats
          morgan: {
              format: 'combined'
          }
      },
  chromeFlags: ['--no-sandbox', '--headless', '--disable-gpu', '--remote-debugging-port=9222', '--hide-scrollbars']
  });

  server.use(prerender.sendPrerenderHeader());
  // server.use(prerender.blockResources());
  server.use(prerender.removeScriptTags());
  server.use(prerender.httpHeaders());
  server.use(require('prerender-memory-cache'));
  server.use(require('prerender-redis-cache'));
  server.use(require('prerender-access-log'));

  server.start();
  ```

  Best way to see if Redis started adding keys is to use redis-cli tool:
  ```
  root@prerender-01:~/prerender# redis-cli
  127.0.0.1:6379> keys *
  1) "https://magentocommand.ml/catalog/category/view/s/jackets/id/323/?climate=8057&color=7903&material=8008"
  2) "https://magentocommand.ml/"
  3) "https://magentocommand.ml/catalog/category/view/s/bottoms/id/322/?climate=8064&eco_collection=0&erin_recommends=0&material=8003&new=1&pattern=8049&style_bottom=7960"
  ```

  Now test page a few times and see how it renders super fast!

  I hope that you will find this post useful for your JavaScript SEO usage scenarios. As always, please do not hesitate to send me your comments and questions. I will be happy to help. Good luck!

html:
  <p>Hello,</p>

  <p>More often we hear that website dropped on popular search engines because of slowness. These things happen when we serve uncached content to the Search engine Bots while visiting pages and we get penalized because of that. Recently, I have been testing one amazing service called prerender.io</p>

  <p>Prerender is a node server that uses Headless Chrome to render HTML, screenshots, PDFs, and HAR files out of any web page. The Prerender server listens for an http request, takes the URL and loads it in Headless Chrome, waits for the page to finish loading by waiting for the network to be idle, and then returns your content.</p>

  <p><img src="/content/images/2020/06/image01.png" alt="alt" /></p>

  <p>Google recommends that you use Prerender.io in their docs (<a href="https://developers.google.com/search/docs/guides/dynamic-rendering">https://developers.google.com/search/docs/guides/dynamic-rendering</a>), so your site is guaranteed to be crawled correctly by Google &amp; other search engines.</p>

  <p>It is fast, simple to use, and really straightforward. Vendor came with two solutions about how the application can be used:</p>

  <p>1) By using prerender.io servers as middleware. <br />
  2) By installing your own server, and using it as a middleware in your's fleet.</p>

  <p>Let's start with first. Visit <a href="https://prerender.io">https://prerender.io</a> website and register. Once that is done, you will be logged to their Control panel area.</p>

  <p><img src="/content/images/2020/06/image--20-.png" alt="alt" /></p>

  <p>In <a href="https://prerender.io/documentation">https://prerender.io/documentation</a> very detailed documentation pages vendor mention various ways to implement middleware between your application and this service, but since today we will make implementation for Magento 2 hosted on Nginx Web server, we will focus on that part.</p>

  <p><a href="https://prerender.io/documentation/install-middleware#nginx">https://prerender.io/documentation/install-middleware#nginx</a> <br />
  Says that <a href="https://gist.github.com/thoop/8165802">https://gist.github.com/thoop/8165802</a> having best way, but here is specific Nginx code that I've used to make this service integrated into present (functional) Nginx config file used for Magento 2 application</p>

  <pre><code>        set $prerender 0;
          if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
              set $prerender 1;
          }
          if ($args ~ "_escaped_fragment_") {
              set $prerender 1;
          }
          if ($http_user_agent ~ "Prerender") {
              set $prerender 0;
          }
          if ($uri ~* "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf|dat|dmg|iso|flv|m4v|torrent|ttf|woff|svg|eot)") {
              set $prerender 0;
          }

          #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
          resolver 8.8.8.8;

          if ($prerender = 1) {
          return 503;
          }

  error_page 500 502 503 504 =200 @prerender;

  location @prerender {  
          proxy_set_header X-Prerender-Token &lt;ENTER TOKEN HERE&gt;;
              #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
              set $prerender "service.prerender.io";
              rewrite .* /$scheme://$host$request_uri? break;
              proxy_pass http://$prerender;
          }
  </code></pre>

  <p>From this point when above Nginx conf is added, you are redirecting traffic for Google and all bots mentioned in the example to service.prerender.io service. You may simulate traffic by executing following command from any popular SSH terminal:</p>

  <pre><code>curl -IkL -A "Googlebot" https://magentocommand.ml  
  </code></pre>

  <p>The output would be 200 OK:  </p>

  <pre><code>&gt; curl -IkL -A "Googlebot" https://magentocommand.ml
  HTTP/2 200  
  content-type: text/html;charset=UTF-8  
  content-length: 178959  
  vary: Accept-Encoding  
  server: nginx  
  date: Sun, 14 Jun 2020 19:38:58 GMT  
  vary: Accept-Encoding  
  access-control-allow-origin: *  
  x-cache: Miss from cloudfront  
  via: 1.1 27aa7ec4f54edf4b2fd5fffda84693a0.cloudfront.net (CloudFront)  
  x-amz-cf-pop: SOF50-C1  
  x-amz-cf-id: fmPB6u2Cqhv06tGTBvgaFDFaW9aJ8eGjGnvlFT2tEGtqgrCHrjdSJw==  
  </code></pre>

  <p>Visit <a href="https://prerender.io/">https://prerender.io/</a> panel in your browser click on the Crawl stats at the top area (<a href="https://prerender.io/crawl-stats">https://prerender.io/crawl-stats</a>) and you will see simulated hit:</p>

  <p><img src="/content/images/2020/06/DeepinScreenshot_select-area_20200614214029.png" alt="alt" /></p>

  <p>Within their panel, you may also view how RAW HTML page looks like by clicking on the Cached Pages option at the top menu.</p>

  <p>To be honest, if you are having a small website with not many SEF pages, the vendor offers 250 cached pages completely free, then next tier is 20,000 cached pages for $15.00 USD per month.</p>

  <p>There are several more small details about the implementation, but I did my best to make this as short as possible. We will jump now to 2nd method hosting your own prerender.io application.</p>

  <p>I've installed a very small Cloud Server with German data center Hetzner AG because it is cheap, stable, and close to my location, but ideally, you can install this application on any platform.</p>

  <p>1) As mentioned before, we will use Hetzner Cloud as our dedicated server. This guide was written using an instance with the Ubuntu image, but with a few little adjustments, it should also work with any popular Linux operating system.</p>

  <p>Regarding the instance type, I suggest starting with something small, get some metrics, and then scale if needed, in the test I've rented CX21 type who comes with 2 VCPU, 4Gb of RAM and 40Gb of NVME disk space. It should be okay to start off with. Finally, configure the security group to allow SSH connections and custom TCP inbounds on port 3000 as that is where your prerender service will work (you can choose your own port here). I usually do the /etc/hosts.allow method allowing only certain IP to gain access. You can go with a popular Firewalls like firewalld or something inbound from Data Center (or Cloud) provider.</p>

  <p>2) Setting up the environment:</p>

  <p>Connect to the instance and install the needed tools. To get the node, I recommend using nvm and choosing the latest LTS version.</p>

  <p><a href="https://github.com/nvm-sh/nvm#git-install">https://github.com/nvm-sh/nvm#git-install</a></p>

  <pre><code>apt-get update &amp;&amp; apt-get install git  
  </code></pre>

  <pre><code>cd ~/ &amp;&amp; git clone https://github.com/nvm-sh/nvm.git .nvm  
  </code></pre>

  <pre><code>cd ~/.nvm &amp;&amp; git checkout v0.35.3  
  </code></pre>

  <p>Now add these lines to your ~/.bashrc, ~/.profile, or ~/.zshrc file to have it automatically sourced upon login: (you may have to add to more than one of the above files)</p>

  <pre><code>export NVM_DIR="$HOME/.nvm"  
  [ -s "$NVM_DIR/nvm.sh" ] &amp;&amp; \. "$NVM_DIR/nvm.sh"  # This loads nvm
  [ -s "$NVM_DIR/bash_completion" ] &amp;&amp; \. "$NVM_DIR/bash_completion"  # This loads nvm bash_completion
  </code></pre>

  <p>Sign off as SSH user, and log in again.</p>

  <p>At the moment of writing this article latest Node/NVM was 12.18.0 (LTS: Erbium), so we will install this one:  </p>

  <pre><code>nvm install 12.18.0  
  </code></pre>

  <p>Output: <br />
  <img src="/content/images/2020/06/DeepinScreenshot_select-area_20200614223002.png" alt="alt" /></p>

  <p>After nvm and node, we need to get forever, which will allow us to run the prerender server continuously in the background.</p>

  <p><a href="https://github.com/foreversd/forever">https://github.com/foreversd/forever</a></p>

  <pre><code>npm install forever -g  
  </code></pre>

  <p>Output: <br />
  <img src="/content/images/2020/06/DeepinScreenshot_select-area_20200614223237.png" alt="alt" /></p>

  <p>Next, we need to fetch the prerender source. To do this, clone the GitHub repo and run an npm install to get its dependencies.</p>

  <pre><code>git clone https://github.com/prerender/prerender.git  
  </code></pre>

  <p>Next, we need to install Google Chrome.  </p>

  <pre><code>wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb  
  </code></pre>

  <p>Installing packages on Ubuntu requires Sudo privileges. Make sure you are running the following command as a user with Sudo privileges.</p>

  <p>Install the Google Chrome .deb package by typing:  </p>

  <pre><code>apt install ./google-chrome-stable_current_amd64.deb  
  </code></pre>

  <p>Time to add first flag and modify /root/prerender/server.js file with the following code:  </p>

  <pre><code>var server = prerender({  
    chromeFlags: ['--no-sandbox', '--headless', '--disable-gpu', '--remote-debugging-port=9222', '--hide-scrollbars']
  });
  </code></pre>

  <p>The default one defaults to:  </p>

  <pre><code>var server = prerender({});  
  </code></pre>

  <p>A lot of other useful customizations can be added in this block based on <a href="https://github.com/prerender/prerender">https://github.com/prerender/prerender</a> instructions.</p>

  <p>To test if all is fine, run following from /root/prerender/ directory:  </p>

  <pre><code>node server.js  
  </code></pre>

  <p>Output: <br />
  <img src="/content/images/2020/06/DeepinScreenshot_select-area_20200614230613.png" alt="alt" /></p>

  <p>To make it running uninterrupted, we can use forever tool that we have previously installed and run this as follows:  </p>

  <pre><code>forever start /root/prerender/server.js  
  </code></pre>

  <p>Output: <br />
  root@prerender-01:~/prerender# forever start /root/prerender/server.js <br />
  warn:    --minUptime not set. Defaulting to: 1000ms <br />
  warn:    --spinSleepTime not set. Your script will exit if it does not stay up for at least 1000ms <br />
  info:    Forever processing file: /root/prerender/server.js</p>

  <p>You may check if the port is listening by executing lsof tool:  </p>

  <pre><code>lsof -i :3000  
  </code></pre>

  <p>Output: <br />
  root@prerender-01:~/prerender# lsof -i :3000 <br />
  COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME <br />
  node    9198 root   18u  IPv6  38348      0t0  TCP *:3000 (LISTEN)</p>

  <p>Now, that we are having Prerender server listening on port 3000, we can safely create systemd service so it can boot safely again in case of difficulties.</p>

  <p>Navigate to /etc/systemd/system folder and create prerender.service file with the following content:</p>

  <pre><code>[Unit]
  Description=Prerender Service

  [Service]
  User=root  
  WorkingDirectory=/root/prerender  
  ExecStart=/root/.nvm/versions/node/v12.18.0/bin/node /root/prerender/server.js

  [Install]
  WantedBy=multi-user.target  
  </code></pre>

  <p>Then execute: <br />
  systemctl enable prerender <br />
  systemctl start prerender</p>

  <p>Check status now with systemctl status pretender. The output should be like this one: <br />
  <img src="/content/images/2020/06/DeepinScreenshot_select-area_20200614233539.png" alt="alt" /></p>

  <p>That's all! Now you can create a DNS zone in your's external entity and start using Prerender service on your Web server by implementing the Middleware method from the start of this article.</p>

  <p>To just test out of the box if Prerender service is working you may execute the following:  </p>

  <pre><code>curl http://localhost:3000/render?url=https://magentocommand.ml  
  </code></pre>

  <p>One of the Alternative to track pages better is to install Nginx on the Prerender server and then using location / simply proxypass traffic to port 3000 while keeping that port enabled only to the localhost. That way, you may filter who is able and who is not to access Prerender service by using ports 80 or even more advanced with 443 (SSL).</p>

  <p>The next one would be to configure Prerender on port 80 (why not!) and use it. To do that add one line to the /etc/systemd/system/prerender.service file under [Service] tab:  </p>

  <pre><code>Environment=NODE_ENV=production PORT=80  
  </code></pre>

  <p>Optionally, you may add some Cache settings and Number of workers. To be honest, I am testing this on 2 CPU Virtual Private server so I had to limit number of workers to single to prevent OOM failures. To make caching even possible you need to install <a href="https://github.com/prerender/prerender-memory-cache">https://github.com/prerender/prerender-memory-cache</a> plugin.</p>

  <pre><code>Environment=NODE_ENV=production CACHE_MAXSIZE=100000  
  Environment=NODE_ENV=production CACHE_TTL=432000  
  Environment=NODE_ENV=production PRERENDER_NUM_WORKERS=1  
  </code></pre>

  <p>The complete file should look like this:  </p>

  <pre><code>[Unit]
  Description=Prerender Service

  [Service]
  User=root  
  WorkingDirectory=/root/prerender  
  Environment=NODE_ENV=production PORT=80  
  Environment=NODE_ENV=production CACHE_MAXSIZE=100000  
  Environment=NODE_ENV=production CACHE_TTL=432000  
  Environment=NODE_ENV=production PRERENDER_NUM_WORKERS=1  
  ExecStart=/root/.nvm/versions/node/v12.18.0/bin/node /root/prerender/server.js

  [Install]
  WantedBy=multi-user.target
  </code></pre>

  <p>Extra addon. Generating HTML snapshots is a resource-intensive process, so some sort of caching strategy should be used to improve the performance. Prerender.io comes with several different caching plugins, but we will use the one based on Redis, as it offers scalability and a simple cache expiration mechanism. Redis can be built from source, but the pre-built version is sufficient for the task at hand.</p>

  <pre><code>sudo apt-get update  
  </code></pre>

  <pre><code>sudo apt-get upgrade  
  </code></pre>

  <pre><code>sudo apt-get -y install redis-server  
  </code></pre>

  <p>Open /etc/redis/redis.conf file and make configuration is configured properly. For security measures I just uncomment bind line to bind to only 127.0.0.1 IP address or let's say localhost. Rest details can be adjusted per requirements/needs.</p>

  <p>Download the Redis cache plugin and install it. Currently, it caches the pages for one day and then expires them. This can be overridden by specifying the env variable process.env.PAGE<em>TTL in seconds. To never expire, you should set the PAGE</em>TTL variable to 0.</p>

  <pre><code>npm install prerender-redis-cache --save  
  </code></pre>

  <p>Change the server.js to use the redis cache plugin:  </p>

  <pre><code>server.use(require('prerender-redis-cache'));  
  </code></pre>

  <p>Needs to be added here: <br />
  <img src="/content/images/2020/06/DeepinScreenshot_select-area_20200615132313.png" alt="alt" /></p>

  <p>Good to add (optional) is access.log feature and keep the access logs - it can be useful for debugging and other maintenance tasks.</p>

  <p><a href="https://github.com/unDemian/prerender-access-log">https://github.com/unDemian/prerender-access-log</a>  </p>

  <pre><code>npm install prerender-access-log --save  
  </code></pre>

  <p>Initialize the plugin in the server.js. You also need to configure the access log settings. Here is the finished version of the server.js:  </p>

  <pre><code>#!/usr/bin/env node
  var prerender = require('./lib');

  var server = prerender({  
    accessLog: {
          // Check out the file-stream-rotator docs for parameters
          fileStreamRotator: {
              filename: '/var/log/prerender/access-%DATE%.log',
              frequency: 'daily',
              date_format: 'YYYY-MM-DD',
              verbose: false
          },

          // Check out the morgan docs for the available formats
          morgan: {
              format: 'combined'
          }
      },
  chromeFlags: ['--no-sandbox', '--headless', '--disable-gpu', '--remote-debugging-port=9222', '--hide-scrollbars']  
  });

  server.use(prerender.sendPrerenderHeader());  
  // server.use(prerender.blockResources());
  server.use(prerender.removeScriptTags());  
  server.use(prerender.httpHeaders());  
  server.use(require('prerender-memory-cache'));  
  server.use(require('prerender-redis-cache'));  
  server.use(require('prerender-access-log'));

  server.start();  
  </code></pre>

  <p>Best way to see if Redis started adding keys is to use redis-cli tool:  </p>

  <pre><code>root@prerender-01:~/prerender# redis-cli  
  127.0.0.1:6379&gt; keys *  
  1) "https://magentocommand.ml/catalog/category/view/s/jackets/id/323/?climate=8057&amp;color=7903&amp;material=8008"  
  2) "https://magentocommand.ml/"  
  3) "https://magentocommand.ml/catalog/category/view/s/bottoms/id/322/?climate=8064&amp;eco_collection=0&amp;erin_recommends=0&amp;material=8003&amp;new=1&amp;pattern=8049&amp;style_bottom=7960"  
  </code></pre>

  <p>Now test page a few times and see how it renders super fast!</p>

  <p>I hope that you will find this post useful for your JavaScript SEO usage scenarios. As always, please do not hesitate to send me your comments and questions. I will be happy to help. Good luck!</p>


