# posts

uuid: c377414a-d033-460f-ad99-f9fa69ba57c7
title: Disable Category Permissions in Magento
slug: disable-category-permissions-in-magento
image: /content/images/2020/07/DeepinScreenshot_select-area_20200707200800.png
status: published
language: en_US
markdown:
  As of version 2.3.5 Commerce Edition, Magento implemented new feature called Category Permissions (https://docs.magento.com/user-guide/catalog/category-permissions.html).

  Category access can be limited to specific customer groups, or restricted entirely. You can control the display of product prices, and determine which customer groups can add products to the cart, and specify the landing page.

  Error displayed when category page is restricted:
  ```
  [2020-07-07 17:23:06] report.CRITICAL: You may need more permissions to access this category. {"exception":"[object] (Magento\\Framework\\Exception\\LocalizedException(code: 0): You may need more permissions to access this category. at /srv/xxxxxxxxx/releases/20200707154013/vendor/magento/module-catalog-permissions/Observer/ApplyCategoryPermissionObserver.php:113)"} []
  [2020-07-07 17:23:06] report.CRITICAL: You may need more permissions to access this category. {"exception":"[object] (Magento\\Framework\\Exception\\LocalizedException(code: 0): You may need more permissions to access this category. at /srv/xxxxxxxxxx/releases/20200707154013/vendor/magento/module-catalog-permissions/Observer/ApplyCategoryPermissionObserver.php:113)"} []
  ```

  How To Disable Category Permissions:
  ```
  php bin/magento config:set catalog/magento_catalogpermissions/enabled "0" --lock-config
  ```

  Or add following to the env.php or config.php files:
  ```
  'system' => [
          'default' => [
              'catalog' => [
                  'magento_catalogpermissions' => [
                      'enabled' => '0'
                  ]
              ]
          ]
      ]
  ```

  For security reasons, there is no on / off switch available from the Magento admin panel (otherwise a potential attacker with Magento admin access could disable it easily). As such it must be done from the command line.

  Hope this helps. Good luck!

html:
  <p>As of version 2.3.5 Commerce Edition, Magento implemented new feature called Category Permissions (<a href="https://docs.magento.com/user-guide/catalog/category-permissions.html">https://docs.magento.com/user-guide/catalog/category-permissions.html</a>).</p>

  <p>Category access can be limited to specific customer groups, or restricted entirely. You can control the display of product prices, and determine which customer groups can add products to the cart, and specify the landing page.</p>

  <p>Error displayed when category page is restricted:  </p>

  <pre><code>[2020-07-07 17:23:06] report.CRITICAL: You may need more permissions to access this category. {"exception":"[object] (Magento\\Framework\\Exception\\LocalizedException(code: 0): You may need more permissions to access this category. at /srv/xxxxxxxxx/releases/20200707154013/vendor/magento/module-catalog-permissions/Observer/ApplyCategoryPermissionObserver.php:113)"} []
  [2020-07-07 17:23:06] report.CRITICAL: You may need more permissions to access this category. {"exception":"[object] (Magento\\Framework\\Exception\\LocalizedException(code: 0): You may need more permissions to access this category. at /srv/xxxxxxxxxx/releases/20200707154013/vendor/magento/module-catalog-permissions/Observer/ApplyCategoryPermissionObserver.php:113)"} []
  </code></pre>

  <p>How To Disable Category Permissions:  </p>

  <pre><code>php bin/magento config:set catalog/magento_catalogpermissions/enabled "0" --lock-config  
  </code></pre>

  <p>Or add following to the env.php or config.php files:  </p>

  <pre><code>'system' =&gt; [  
          'default' =&gt; [
              'catalog' =&gt; [
                  'magento_catalogpermissions' =&gt; [
                      'enabled' =&gt; '0'
                  ]
              ]
          ]
      ]
  </code></pre>

  <p>For security reasons, there is no on / off switch available from the Magento admin panel (otherwise a potential attacker with Magento admin access could disable it easily). As such it must be done from the command line.</p>

  <p>Hope this helps. Good luck!</p>


uuid: 27ea427b-2bbd-4254-9b8b-253d040985ef
title: Enable RabbitMQ on Magento 2 for additional consumers
slug: enable-rabbitmq-on-magento-2-for-additional-consumer
image: /content/images/2020/11/0_qlbJbYSQLwrmtpNF.png
status: published
language: en_US
markdown:
  Hello everyone,

  It has been a while since my last post. We all know RabbitMQ service is present in the Magento community for some time but with recent versions, it became standard for any serious production store equally or not even the same as ElasticSearch. So what RabbitMQ is? RabbitMQ is open-source message-broker software that originally implemented the Advanced Message Queuing Protocol and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol, MQ Telemetry Transport, and other protocols.

  Message queues provide an asynchronous communications mechanism in which the sender and the receiver of a message do not contact each other. Nor do they need to communicate with the message queue at the same time. When a sender places a message onto a queue, it is stored until the recipient receives them.

  In Magento, the Message Queue Framework (MQF) is a fully-functional system that allows a module to publish messages to queues. It also creates consumers to receive them asynchronously. The MQF primarily uses RabbitMQ as the messaging broker, which provides a scalable platform for sending and receiving messages. It also includes a mechanism for storing undelivered messages. RabbitMQ is based on the Advanced Message Queuing Protocol (AMQP) 0.9.1 specification.

  A basic message queue system can also be set up without using RabbitMQ. In this system, MySQL adapter stores messages in the database. Three database tables (`queue`, `queue_message`, and `queue_message_status`) manage the message queue workload. Cron jobs ensure the consumers are able to receive messages. This solution is not very scalable. RabbitMQ should be used whenever possible.

  In a MySQL adapter environment, when a message is published to multiple queues, create a single record in queue_message and multiple records in each `queue_message_status`: one for each queue. (A join on the `queue`, `queue_message`, and `queue_message_status` tables is required).

  By default, only `async.operations.all` consumer is used by RabbitMQ service, but what is happening with the rest of consumers?

  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201103232910.png)

  Magento DevDocs pages on https://devdocs.magento.com/guides/v2.4/extension-dev-guide/message-queues/message-queues.html#change-message-queue-from-mysql-to-amqp address explain with an example of how this can be accomplished by adjusting `env.php` file configuration that allows you to redefine the adapter for a topic.

  ```
  'queue' => [
      'topics' => [
          'product_action_attribute.update' => [
              'publisher' => 'amqp-magento'
          ]
      ],
      'config' => [
          'publishers' => [
              'product_action_attribute.update' => [
                  'connections' => [
                      'amqp' => [
                          'name' => 'amqp',
                          'exchange' => 'magento',
                          'disabled' => false
                      ],
                      'db' => [
                          'name' => 'db',
                          'disabled' => true
                      ]
                  ]
              ]
          ]
      ],
      'consumers' => [
          'product_action_attribute.update' => [
              'connection' => 'amqp',
          ],
      ],
  ],
  ```

  With small tweaks correct code that needs to be added to the env.php file became following:
  ```
  'queue' => [
          'amqp' => [
              'host' => '127.0.0.1',
              'port' => '5672',
              'user' => 'guest',
              'password' => 'guest',
              'virtualhost' => '/'
          ],
          'consumers_wait_for_messages' => 1,
          'topics' => [
          'product_action_attribute.update' => [
              'publisher' => 'amqp-magento'
          ]
      ],
      'config' => [
          'publishers' => [
              'product_action_attribute.update' => [
                  'connections' => [
                      'amqp' => [
                          'name' => 'amqp',
                          'exchange' => 'magento',
                          'disabled' => false
                      ],
                      'db' => [
                          'name' => 'db',
                          'disabled' => true
                      ]
                  ]
              ]
          ]
      ],
      'consumers' => [
          'product_action_attribute.update' => [
              'connection' => 'amqp',
          ],
      ],
  ],
  ```
  - You may notice that we have just updated block with correct amqp values used for Magento 2 to connect to the RabbitMQ service.

  The most important note when converting MySQL adapter to AMQP is that each time added you need to push setup:upgrade because that is when Magento 2 creates correct Queue in RabbitMQ and test it. You can use rabbitmqctl tool or Web interface plugin to check when new queue is created.

  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201103233504.png)

  Well, this was easy... we just copy/paste from Magento 2 dev docs example and surprisingly it works! Now, let's try to add the one mostly used when there is an external integration requirement, exportProcessor consumer used when products are exported.

  Module in charge is Magento 2 core module named ImportExport located in the app/code/Magento/ImportExport/ directory. We will open 3 files that instruct which adapter is used (MySQL or Amqp).

  ```
  queue_consumer.xml
  queue_publisher.xml
  queue_topology.xml
  ```
  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201103234446.png)
  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201103234603.png)
  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201103234644.png)

  Based on these pieces of information we can now create variables for the env.php file:

  ```
      'queue' => [
          'amqp' => [
              'host' => '127.0.0.1',
              'port' => '5672',
              'user' => 'guest',
              'password' => 'guest',
              'virtualhost' => '/'
          ],
          'consumers_wait_for_messages' => 1,
          'topics' => [
              'import_export.export' => [
                  'publisher' => 'amqp-magento'
              ]
          ],
          'config' => [
              'publishers' => [
                  'import_export.export' => [
                      'connections' => [
                          'amqp' => [
                              'name' => 'amqp',
                              'exchange' => 'magento',
                              'disabled' => false
                          ],
                          'db' => [
                              'name' => 'db',
                              'disabled' => true
                          ]
                      ]
                  ]
              ]
          ],
          'consumers' => [
              'exportProcessor' => [
                  'connection' => 'amqp'
              ]
          ]
      ],
  ```
  Once again we need to run setup:upgrade command so Magento 2 can generate a new queue named "export". Usually, this prompts at the end of setup:upgrade

  ```
  2020-11-03 23:05:34.999 [info] <0.8343.0> closing AMQP connection <0.8343.0> (127.0.0.1:38132 -> 127.0.0.1:5672, vhost: '/', user: 'guest')
  ```

  Push rabbitmqctl or web interface and you should see a new queue created:
  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201104000724.png)

  Now, time to test and compare results. Based on the https://devdocs.magento.com/guides/v2.4/config-guide/cli/config-cli-subcommands-perf-data.html article we will generate data for a Medium profile store with 24,000 products.

  ```
  $ php bin/magento setup:perf:generate-fixtures /workspace/magento2gitpod/setup/performance-toolkit/profiles/ce/small.xml
  Generating a profile with the following params:
   |- Admin Users: 50
   |- Websites: 3
   |- Store Groups Count: 3
   |- Store Views Count: 3
   |- Categories: 300
   |- Attribute Sets (Default): 3
   |- Attribute Sets (Extra): 30
   |- Simple products: 24000
   |- Configurable products: 640
   |- Product images: 1000, 3 per product
   |- Customers: 2000
   |- Cart Price Rules: 20
   |- Catalog Price Rules: 20
   |- Coupon Codes: 20
   |- Orders: 50000
  Config Changes...  done in 00:00:00
  Generating admin users...  done in 00:00:07
  Generating websites, stores and store views...  done in 00:00:04
  Generating categories...  done in 00:00:58
  Generating attribute sets...  done in 00:00:24
  Generating simple products...  done in 00:05:46
  Generating configurable EAV variations...  done in 00:00:00
  Generating bundle products...  done in 00:00:00
  Generating configurable products...  done in 00:02:30
  Generating images...  done in 00:03:52
  Generating customer groups...  done in 00:00:00
  Generating customers...  done in 00:01:06
  Generating cart price rules...  done in 00:00:19
  Generating catalog price rules...  done in 00:00:18
  Generating tax rates...  done in 00:07:07
  Generating tax rules...  done in 00:00:00
  Generating coupon codes...  done in 00:00:00
  Generating orders...  done in 00:08:10
  Indexers Mode Changes...  done in 00:00:01
  Design Config Grid index has been rebuilt successfully in 00:00:00
  Customer Grid index has been rebuilt successfully in 00:00:02
  Category Products index has been rebuilt successfully in 00:00:21
  Product Categories index has been rebuilt successfully in 00:00:00
  Catalog Rule Product index has been rebuilt successfully in 00:00:56
  Product EAV index has been rebuilt successfully in 00:00:22
  Stock index has been rebuilt successfully in 00:00:19
  Product Price index has been rebuilt successfully in 00:00:36
  Catalog Product Rule index has been rebuilt successfully in 00:00:00
  exception: Catalog Search index is locked by another reindex process. Skipping.
  Total execution time: 00:33:34
  ```

  Since we are not running Crons here, we need to start exportProcessor consumer which detects when export is initiated and start creating file. We will also monitor RabbitMQ and see when messages are processed.

  ```
  $ n98-magerun2 queue:consumer:start exportProcessor
  ```

  Next is to jump to the Dashboard System --> Export area and initiate export for all products.

  From this point, you can watch the RabbitMQ Web interface or from CLI

  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201104005107.png)

  There are several useful options
  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201104005216.png)

  Plus you can watch and see that the exportProcessor process is using a lot of CPU memory:
  ![alt](/content/images/2020/11/DeepinScreenshot_select-area_20201104005310.png)

  Timings are clearly in favor of RabbitMQ. You can use https://github.com/nemke82/magento2gitpod fully functional Magento 2 setup and test with RabbitMQ and MySQL from your local computer using any popular browser.

  Hope this article helps. Cheers.

html:
  <p>Hello everyone,</p>

  <p>It has been a while since my last post. We all know RabbitMQ service is present in the Magento community for some time but with recent versions, it became standard for any serious production store equally or not even the same as ElasticSearch. So what RabbitMQ is? RabbitMQ is open-source message-broker software that originally implemented the Advanced Message Queuing Protocol and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol, MQ Telemetry Transport, and other protocols.</p>

  <p>Message queues provide an asynchronous communications mechanism in which the sender and the receiver of a message do not contact each other. Nor do they need to communicate with the message queue at the same time. When a sender places a message onto a queue, it is stored until the recipient receives them.</p>

  <p>In Magento, the Message Queue Framework (MQF) is a fully-functional system that allows a module to publish messages to queues. It also creates consumers to receive them asynchronously. The MQF primarily uses RabbitMQ as the messaging broker, which provides a scalable platform for sending and receiving messages. It also includes a mechanism for storing undelivered messages. RabbitMQ is based on the Advanced Message Queuing Protocol (AMQP) 0.9.1 specification.</p>

  <p>A basic message queue system can also be set up without using RabbitMQ. In this system, MySQL adapter stores messages in the database. Three database tables (<code>queue</code>, <code>queue_message</code>, and <code>queue_message_status</code>) manage the message queue workload. Cron jobs ensure the consumers are able to receive messages. This solution is not very scalable. RabbitMQ should be used whenever possible.</p>

  <p>In a MySQL adapter environment, when a message is published to multiple queues, create a single record in queue_message and multiple records in each <code>queue_message_status</code>: one for each queue. (A join on the <code>queue</code>, <code>queue_message</code>, and <code>queue_message_status</code> tables is required).</p>

  <p>By default, only <code>async.operations.all</code> consumer is used by RabbitMQ service, but what is happening with the rest of consumers?</p>

  <p><img src="/content/images/2020/11/DeepinScreenshot_select-area_20201103232910.png" alt="alt" /></p>

  <p>Magento DevDocs pages on <a href="https://devdocs.magento.com/guides/v2.4/extension-dev-guide/message-queues/message-queues.html#change-message-queue-from-mysql-to-amqp">https://devdocs.magento.com/guides/v2.4/extension-dev-guide/message-queues/message-queues.html#change-message-queue-from-mysql-to-amqp</a> address explain with an example of how this can be accomplished by adjusting <code>env.php</code> file configuration that allows you to redefine the adapter for a topic.</p>

  <pre><code>'queue' =&gt; [  
      'topics' =&gt; [
          'product_action_attribute.update' =&gt; [
              'publisher' =&gt; 'amqp-magento'
          ]
      ],
      'config' =&gt; [
          'publishers' =&gt; [
              'product_action_attribute.update' =&gt; [
                  'connections' =&gt; [
                      'amqp' =&gt; [
                          'name' =&gt; 'amqp',
                          'exchange' =&gt; 'magento',
                          'disabled' =&gt; false
                      ],
                      'db' =&gt; [
                          'name' =&gt; 'db',
                          'disabled' =&gt; true
                      ]
                  ]
              ]
          ]
      ],
      'consumers' =&gt; [
          'product_action_attribute.update' =&gt; [
              'connection' =&gt; 'amqp',
          ],
      ],
  ],
  </code></pre>

  <p>With small tweaks correct code that needs to be added to the env.php file became following:  </p>

  <pre><code>'queue' =&gt; [  
          'amqp' =&gt; [
              'host' =&gt; '127.0.0.1',
              'port' =&gt; '5672',
              'user' =&gt; 'guest',
              'password' =&gt; 'guest',
              'virtualhost' =&gt; '/'
          ],
          'consumers_wait_for_messages' =&gt; 1,
          'topics' =&gt; [
          'product_action_attribute.update' =&gt; [
              'publisher' =&gt; 'amqp-magento'
          ]
      ],
      'config' =&gt; [
          'publishers' =&gt; [
              'product_action_attribute.update' =&gt; [
                  'connections' =&gt; [
                      'amqp' =&gt; [
                          'name' =&gt; 'amqp',
                          'exchange' =&gt; 'magento',
                          'disabled' =&gt; false
                      ],
                      'db' =&gt; [
                          'name' =&gt; 'db',
                          'disabled' =&gt; true
                      ]
                  ]
              ]
          ]
      ],
      'consumers' =&gt; [
          'product_action_attribute.update' =&gt; [
              'connection' =&gt; 'amqp',
          ],
      ],
  ],
  </code></pre>

  <ul>
  <li>You may notice that we have just updated block with correct amqp values used for Magento 2 to connect to the RabbitMQ service.</li>
  </ul>

  <p>The most important note when converting MySQL adapter to AMQP is that each time added you need to push setup:upgrade because that is when Magento 2 creates correct Queue in RabbitMQ and test it. You can use rabbitmqctl tool or Web interface plugin to check when new queue is created.</p>

  <p><img src="/content/images/2020/11/DeepinScreenshot_select-area_20201103233504.png" alt="alt" /></p>

  <p>Well, this was easy... we just copy/paste from Magento 2 dev docs example and surprisingly it works! Now, let's try to add the one mostly used when there is an external integration requirement, exportProcessor consumer used when products are exported.</p>

  <p>Module in charge is Magento 2 core module named ImportExport located in the app/code/Magento/ImportExport/ directory. We will open 3 files that instruct which adapter is used (MySQL or Amqp).</p>

  <pre><code>queue_consumer.xml  
  queue_publisher.xml  
  queue_topology.xml  
  </code></pre>

  <p><img src="/content/images/2020/11/DeepinScreenshot_select-area_20201103234446.png" alt="alt" />
  <img src="/content/images/2020/11/DeepinScreenshot_select-area_20201103234603.png" alt="alt" />
  <img src="/content/images/2020/11/DeepinScreenshot_select-area_20201103234644.png" alt="alt" /></p>

  <p>Based on these pieces of information we can now create variables for the env.php file:</p>

  <pre><code>    'queue' =&gt; [
          'amqp' =&gt; [
              'host' =&gt; '127.0.0.1',
              'port' =&gt; '5672',
              'user' =&gt; 'guest',
              'password' =&gt; 'guest',
              'virtualhost' =&gt; '/'
          ],
          'consumers_wait_for_messages' =&gt; 1,
          'topics' =&gt; [
              'import_export.export' =&gt; [
                  'publisher' =&gt; 'amqp-magento'
              ]
          ],
          'config' =&gt; [
              'publishers' =&gt; [
                  'import_export.export' =&gt; [
                      'connections' =&gt; [
                          'amqp' =&gt; [
                              'name' =&gt; 'amqp',
                              'exchange' =&gt; 'magento',
                              'disabled' =&gt; false
                          ],
                          'db' =&gt; [
                              'name' =&gt; 'db',
                              'disabled' =&gt; true
                          ]
                      ]
                  ]
              ]
          ],
          'consumers' =&gt; [
              'exportProcessor' =&gt; [
                  'connection' =&gt; 'amqp'
              ]
          ]
      ],
  </code></pre>

  <p>Once again we need to run setup:upgrade command so Magento 2 can generate a new queue named "export". Usually, this prompts at the end of setup:upgrade</p>

  <pre><code>2020-11-03 23:05:34.999 [info] &lt;0.8343.0&gt; closing AMQP connection &lt;0.8343.0&gt; (127.0.0.1:38132 -&gt; 127.0.0.1:5672, vhost: '/', user: 'guest')  
  </code></pre>

  <p>Push rabbitmqctl or web interface and you should see a new queue created: <br />
  <img src="/content/images/2020/11/DeepinScreenshot_select-area_20201104000724.png" alt="alt" /></p>

  <p>Now, time to test and compare results. Based on the <a href="https://devdocs.magento.com/guides/v2.4/config-guide/cli/config-cli-subcommands-perf-data.html">https://devdocs.magento.com/guides/v2.4/config-guide/cli/config-cli-subcommands-perf-data.html</a> article we will generate data for a Medium profile store with 24,000 products.</p>

  <pre><code>$ php bin/magento setup:perf:generate-fixtures /workspace/magento2gitpod/setup/performance-toolkit/profiles/ce/small.xml
  Generating a profile with the following params:  
   |- Admin Users: 50
   |- Websites: 3
   |- Store Groups Count: 3
   |- Store Views Count: 3
   |- Categories: 300
   |- Attribute Sets (Default): 3
   |- Attribute Sets (Extra): 30
   |- Simple products: 24000
   |- Configurable products: 640
   |- Product images: 1000, 3 per product
   |- Customers: 2000
   |- Cart Price Rules: 20
   |- Catalog Price Rules: 20
   |- Coupon Codes: 20
   |- Orders: 50000
  Config Changes...  done in 00:00:00  
  Generating admin users...  done in 00:00:07  
  Generating websites, stores and store views...  done in 00:00:04  
  Generating categories...  done in 00:00:58  
  Generating attribute sets...  done in 00:00:24  
  Generating simple products...  done in 00:05:46  
  Generating configurable EAV variations...  done in 00:00:00  
  Generating bundle products...  done in 00:00:00  
  Generating configurable products...  done in 00:02:30  
  Generating images...  done in 00:03:52  
  Generating customer groups...  done in 00:00:00  
  Generating customers...  done in 00:01:06  
  Generating cart price rules...  done in 00:00:19  
  Generating catalog price rules...  done in 00:00:18  
  Generating tax rates...  done in 00:07:07  
  Generating tax rules...  done in 00:00:00  
  Generating coupon codes...  done in 00:00:00  
  Generating orders...  done in 00:08:10  
  Indexers Mode Changes...  done in 00:00:01  
  Design Config Grid index has been rebuilt successfully in 00:00:00  
  Customer Grid index has been rebuilt successfully in 00:00:02  
  Category Products index has been rebuilt successfully in 00:00:21  
  Product Categories index has been rebuilt successfully in 00:00:00  
  Catalog Rule Product index has been rebuilt successfully in 00:00:56  
  Product EAV index has been rebuilt successfully in 00:00:22  
  Stock index has been rebuilt successfully in 00:00:19  
  Product Price index has been rebuilt successfully in 00:00:36  
  Catalog Product Rule index has been rebuilt successfully in 00:00:00  
  exception: Catalog Search index is locked by another reindex process. Skipping.  
  Total execution time: 00:33:34  
  </code></pre>

  <p>Since we are not running Crons here, we need to start exportProcessor consumer which detects when export is initiated and start creating file. We will also monitor RabbitMQ and see when messages are processed.</p>

  <pre><code>$ n98-magerun2 queue:consumer:start exportProcessor
  </code></pre>

  <p>Next is to jump to the Dashboard System --> Export area and initiate export for all products.</p>

  <p>From this point, you can watch the RabbitMQ Web interface or from CLI</p>

  <p><img src="/content/images/2020/11/DeepinScreenshot_select-area_20201104005107.png" alt="alt" /></p>

  <p>There are several useful options <br />
  <img src="/content/images/2020/11/DeepinScreenshot_select-area_20201104005216.png" alt="alt" /></p>

  <p>Plus you can watch and see that the exportProcessor process is using a lot of CPU memory: <br />
  <img src="/content/images/2020/11/DeepinScreenshot_select-area_20201104005310.png" alt="alt" /></p>

  <p>Timings are clearly in favor of RabbitMQ. You can use <a href="https://github.com/nemke82/magento2gitpod">https://github.com/nemke82/magento2gitpod</a> fully functional Magento 2 setup and test with RabbitMQ and MySQL from your local computer using any popular browser.</p>

  <p>Hope this article helps. Cheers.</p>


uuid: d28a7758-b6d7-4bb0-8b5e-b20b10177ddd
title: Speed up Magento 2 page load with just-in-time preloading
slug: speed-up-magento-2-page-load-with-just-in-time-preloading
image: /content/images/2021/02/DeepinScreenshot_select-area_20210215233807.png
status: published
language: en_US
markdown:
  Hello,

  It's been a while since my last post and first of all dear reader, I hope this blog post finds you well. This pandemia (Covid-19) left an undoubtful impact on all of us. Today, I would like to show you one super easy trick to speed up Magento 2 page load using the just-in-time preloading technique.

  It is based on https://instant.page/ which is a free Open source project of ~200 lines https://github.com/instantpage/instant.page. More technical details on how this feature works you can read on https://instant.page/tech page, and I strongly encourage you to host application on your own server or better Cloudflare Workers (https://workers.cloudflare.com/) and distribute content based on the visitor location.

  Preloading is done with following concept:
  ```
  <link rel="prefetch" href="url">
  ```

  Integration with Magento 2 is really simple. Navigate to Dashboard --> Content --> Design --> Configuration and under Footer --> Miscellaneous HTML paste the code from https://instant.page/ website.

  ![alt](/content/images/2021/02/DeepinScreenshot_select-area_20210215233112.png)

  Like I said above it is strongly recommended for production stores that you host this application on your own servers and use your own domain (preferably subdomain of the main domain used).

  That's all! Clear Magento 2 caches and if Redis/Varnish used and test.

  Cheating the brain
  The human brain perceives actions taking less than 100 ms as instant.  As a result, instant.page makes your pages feel instant even on 3G (assuming your pages are fast to render).

  Easy on your server and your user’s data plan
  Pages are preloaded only when there’s a good chance that a user will visit them, and only the HTML is preloaded, being respectful of your users’ and servers’ bandwidth and CPU.

  It uses passive event listeners and requestIdleCallback so that your pages stay smooth. It respects data saver mode. It’s 1 kB and loads after everything else. And it’s free and open-source (MIT license).

  There is also a nice Chrome extension doing the same:
  https://chrome.google.com/webstore/detail/fasterchrome/nmgpnfccjfjhdenioncabecepjcmdnjg

  Hope this article was helpful. Kudos to author and great idea https://github.com/instantpage/instant.page and all contributors involved.

html:
  <p>Hello,</p>

  <p>It's been a while since my last post and first of all dear reader, I hope this blog post finds you well. This pandemia (Covid-19) left an undoubtful impact on all of us. Today, I would like to show you one super easy trick to speed up Magento 2 page load using the just-in-time preloading technique.</p>

  <p>It is based on <a href="https://instant.page/">https://instant.page/</a> which is a free Open source project of ~200 lines <a href="https://github.com/instantpage/instant.page">https://github.com/instantpage/instant.page</a>. More technical details on how this feature works you can read on <a href="https://instant.page/tech">https://instant.page/tech</a> page, and I strongly encourage you to host application on your own server or better Cloudflare Workers (<a href="https://workers.cloudflare.com/">https://workers.cloudflare.com/</a>) and distribute content based on the visitor location.</p>

  <p>Preloading is done with following concept:  </p>

  <pre><code>&lt;link rel="prefetch" href="url"&gt;  
  </code></pre>

  <p>Integration with Magento 2 is really simple. Navigate to Dashboard --> Content --> Design --> Configuration and under Footer --> Miscellaneous HTML paste the code from <a href="https://instant.page/">https://instant.page/</a> website.</p>

  <p><img src="/content/images/2021/02/DeepinScreenshot_select-area_20210215233112.png" alt="alt" /></p>

  <p>Like I said above it is strongly recommended for production stores that you host this application on your own servers and use your own domain (preferably subdomain of the main domain used).</p>

  <p>That's all! Clear Magento 2 caches and if Redis/Varnish used and test.</p>

  <p>Cheating the brain <br />
  The human brain perceives actions taking less than 100 ms as instant.  As a result, instant.page makes your pages feel instant even on 3G (assuming your pages are fast to render).</p>

  <p>Easy on your server and your user’s data plan <br />
  Pages are preloaded only when there’s a good chance that a user will visit them, and only the HTML is preloaded, being respectful of your users’ and servers’ bandwidth and CPU.</p>

  <p>It uses passive event listeners and requestIdleCallback so that your pages stay smooth. It respects data saver mode. It’s 1 kB and loads after everything else. And it’s free and open-source (MIT license).</p>

  <p>There is also a nice Chrome extension doing the same: <br />
  <a href="https://chrome.google.com/webstore/detail/fasterchrome/nmgpnfccjfjhdenioncabecepjcmdnjg">https://chrome.google.com/webstore/detail/fasterchrome/nmgpnfccjfjhdenioncabecepjcmdnjg</a></p>

  <p>Hope this article was helpful. Kudos to author and great idea <a href="https://github.com/instantpage/instant.page">https://github.com/instantpage/instant.page</a> and all contributors involved.</p>


uuid: 1d33a36d-018c-4764-a182-4d54377a9f8b
title: Install Gitpod Self hosted using Vagrant and VirtualBox
slug: install-gitpod-selft-hosted-using-vagrant-and-virtualbox
status: draft
language: en_US
markdown:
  Hello,

  With the high demand for quality test tools, I've come with one solid solution.

  Prerequirements:
  - Git <BR>
  - VirtualBox (minimal 6.0) <BR>
  - Vagrant

  For the start step let's clone https://github.com/nemke82/cka_lab.git demo-example repository which you can use to deploy Gitpod Self-Hosted solution (https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/) on your local desktop computer or server.

  By default, we deploy Kubernetes master with 2 worker nodes. You can extend that by editing https://github.com/nemke82/cka_lab/blob/master/Vagrantfile#L22 line with the following:
  ```
  (1..3).each do |i|
  ```
  For 3 worker nodes and etc...

  Then edit https://github.com/nemke82/cka_lab/blob/master/build_cluster.sh file and extend command for Kubernetes join cluster:
  ```
  vagrant ssh node2 -c "$join_cmd"
  ```
  node3, node4, node5 (depends on the resources RAM/CPU you have). At the moment of writing of this article, max supported version that needs to be used in the Self-Hosted Gitpod version is 1.17 therefore we need to edit the https://github.com/nemke82/cka_lab/blob/master/provision/base.sh#L18 file and adjust to the following:
  ```
  apt-get install -y kubelet=1.16.15-00 kubeadm=1.16.15-00 kubectl=1.16.15-00
  ```
  ** IMPORTANT NOTE ** <BR>
  You can always execute the following command to get the desired version:
  ```
  curl -s https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages | grep Version | awk '{print $2}'
  ```

  Gitpod is going to support newer versions in the next releases, so that area is a subject of change. My repository example is based on the Debian Bruster version. You can adjust https://github.com/nemke82/cka_lab/blob/master/Vagrantfile#L6 Vagrantfile and pick any desired Debian version like either Debian or Ubuntu but if you wish to go RHEL way you can but that requires a little bit more modifications in this and few other files.

  When everything is ready with files, navigate to the cloned directory and execute:
  ```
  ./build_cluster.sh
  ```

  Login to master using the following command:
  ```
  vagrant ssh master
  ```

  First check if all nodes are online and joined local Kubernetes cluster:
  ```
  vagrant@master:~$ kubectl get nodes
  NAME     STATUS   ROLES                  AGE     VERSION
  master   Ready    control-plane,master   18m     v1.20.4
  node1    Ready    <none>                 4m15s   v1.20.4
  node2    Ready    <none>                 4m5s    v1.20.4
  node3    Ready    <none>                 3m55s   v1.20.4
  node4    Ready    <none>                 3m45s   v1.20.4
  node5    Ready    <none>                 3m34s   v1.20.4
  ```

  Various ways for storage here. I've picked to install local path provisioned to provide option for Dynamic provisioning. Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the user configuration, the Local Path Provisioner will create hostPath based persistent volume on the node automatically. It utilizes the features introduced by Kubernetes Local Persistent Volume feature, but make it a simpler solution than the built-in local volume feature in Kubernetes.
  https://github.com/rancher/local-path-provisioner

  ```
  kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
  ```

  Result:
  ![alt](/content/images/2021/03/DeepinScreenshot_select-area_20210307132702.png)

  Then make this storage as default one:
  ```
  kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  ```

  We need to add Ephemeral storage as well, create a file with following content:
  ```
  # https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6184#note_283677909
  apiVersion: v1
  kind: LimitRange
  metadata:
    name: limit-ephemeral-storage
  spec:
    limits:
    - default:
        ephemeral-storage: "5Gi"
      defaultRequest:
        ephemeral-storage: "300Mi"
      type: Container
  ```
  save it as default-ephemeral-storage.yml file name and execute it with following command:
  ```
  kubectl apply -f default-ephemeral-storage.yml
  ```

  By following https://www.gitpod.io/docs/self-hosted/latest/install/configure-ingress/#using-lets-encrypt-to-generate-https-certificates guidelines we can generate Let's Encrypt SSL certificate and add secretes to the Kubernetes cluster and Gitpod application. Now, that Gitpod requires actual real domain name this step is required.

  Add TLS cert to kubernetes secrets
  ```
  kubectl create secret tls tls-certs --cert=/home/vagrant/nemanja/secrets/https-certificates/fullchain.pem --key=/home/vagrant/nemanja/secrets/https-certificates/privkey.pem
  ```

  Next, let's install Helm:
  https://helm.sh/docs/intro/install/

  Helm now has an installer script that will automatically grab the latest version of Helm and install it locally.

  You can fetch that script and then execute it locally. It's well documented so that you can read through it and understand what it is doing before you run it.

  ```
  $ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
  $ chmod 700 get_helm.sh
  $ ./get_helm.sh
  ```
  ![alt](/content/images/2021/03/DeepinScreenshot_select-area_20210306192410.png)

  In this step we need to install WeaveNet (https://www.weave.works/) module:
  ```
  kubectl apply -f https://raw.githubusercontent.com/weaveworks/weave/master/prog/weave-kube/weave-daemonset-k8s-1.9.yaml
  ```
  Or substitute for particular Kubernetes cluster used:
  ```
  kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"
  ```

  Gitpod schedules two kinds of workloads: the Gitpod installation itself (which we refer to as meta) and the workspaces. Ideally both types of workloads run on seperate nodes to make makes scaling easier. Depending on your cluster size that may not be feasible though. Either way, you need two node labels in your cluster:

  gitpod.io/workload_meta=true which marks the “meta” nodes and
  gitpod.io/workload_workspace=true which marks the workspace nodes.
  If you want to “mix ‘n match”, i.e., don’t separate the nodes, you can simply run:
  ```
  kubectl label node --all gitpod.io/workload_meta=true gitpod.io/workload_workspace=true
  ```

  At this point everything should be ready for installation of Gitpod application to our cluster.

  Based on the https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/ article we will add gitpod repository and install it.

  ```
  vagrant@master:~$ helm repo add gitpod.io https://charts.gitpod.io
  "gitpod.io" has been added to your repositories
  ```

  ```
  helm install gitpod gitpod.io/gitpod
  ```

  Run **kubectl get pods** and verify that all pods are in state **RUNNING**. If some are not, please see the Troubleshooting Guide (https://www.gitpod.io/docs/self-hosted/latest/install/troubleshooting/)

  If you are using this way most likely you will need to create single values.custom.yaml file with following content:
  ```
  components:
    wsDaemon:
      containerRuntime:
        nodeRoots:
        - /var/run/docker/containerd
  ```

  You can also do:
  KUBE_EDITOR="nano" kubectl edit daemonsets.apps ws-daemon

  And adjust type: Socket area to the following:
  ```
  path: /var/run/docker/containerd/containerd.sock
            type: Socket
  ```

  Then run the following command to apply the changes.:
  ```
  helm upgrade --install -f values.custom.yaml gitpod gitpod.io/gitpod
  ```

html:
  <p>Hello,</p>

  <p>With the high demand for quality test tools, I've come with one solid solution.</p>

  <p>Prerequirements: <br />
  - Git <BR>
  - VirtualBox (minimal 6.0) <BR>
  - Vagrant</p>

  <p>For the start step let's clone <a href="https://github.com/nemke82/cka_lab.git">https://github.com/nemke82/cka_lab.git</a> demo-example repository which you can use to deploy Gitpod Self-Hosted solution (<a href="https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/">https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/</a>) on your local desktop computer or server.</p>

  <p>By default, we deploy Kubernetes master with 2 worker nodes. You can extend that by editing <a href="https://github.com/nemke82/cka_lab/blob/master/Vagrantfile#L22">https://github.com/nemke82/cka_lab/blob/master/Vagrantfile#L22</a> line with the following:  </p>

  <pre><code>(1..3).each do |i|
  </code></pre>

  <p>For 3 worker nodes and etc...</p>

  <p>Then edit <a href="https://github.com/nemke82/cka">https://github.com/nemke82/cka</a><em>lab/blob/master/build</em>cluster.sh file and extend command for Kubernetes join cluster:  </p>

  <pre><code>vagrant ssh node2 -c "$join_cmd"  
  </code></pre>

  <p>node3, node4, node5 (depends on the resources RAM/CPU you have). At the moment of writing of this article, max supported version that needs to be used in the Self-Hosted Gitpod version is 1.17 therefore we need to edit the <a href="https://github.com/nemke82/cka_lab/blob/master/provision/base.sh#L18">https://github.com/nemke82/cka_lab/blob/master/provision/base.sh#L18</a> file and adjust to the following:  </p>

  <pre><code>apt-get install -y kubelet=1.16.15-00 kubeadm=1.16.15-00 kubectl=1.16.15-00  
  </code></pre>

  <p><em>* IMPORTANT NOTE *</em> <BR>
  You can always execute the following command to get the desired version:  </p>

  <pre><code>curl -s https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages | grep Version | awk '{print $2}'  
  </code></pre>

  <p>Gitpod is going to support newer versions in the next releases, so that area is a subject of change. My repository example is based on the Debian Bruster version. You can adjust <a href="https://github.com/nemke82/cka_lab/blob/master/Vagrantfile#L6">https://github.com/nemke82/cka_lab/blob/master/Vagrantfile#L6</a> Vagrantfile and pick any desired Debian version like either Debian or Ubuntu but if you wish to go RHEL way you can but that requires a little bit more modifications in this and few other files.</p>

  <p>When everything is ready with files, navigate to the cloned directory and execute:  </p>

  <pre><code>./build_cluster.sh
  </code></pre>

  <p>Login to master using the following command:  </p>

  <pre><code>vagrant ssh master  
  </code></pre>

  <p>First check if all nodes are online and joined local Kubernetes cluster:  </p>

  <pre><code>vagrant@master:~$ kubectl get nodes  
  NAME     STATUS   ROLES                  AGE     VERSION  
  master   Ready    control-plane,master   18m     v1.20.4  
  node1    Ready    &lt;none&gt;                 4m15s   v1.20.4  
  node2    Ready    &lt;none&gt;                 4m5s    v1.20.4  
  node3    Ready    &lt;none&gt;                 3m55s   v1.20.4  
  node4    Ready    &lt;none&gt;                 3m45s   v1.20.4  
  node5    Ready    &lt;none&gt;                 3m34s   v1.20.4  
  </code></pre>

  <p>Various ways for storage here. I've picked to install local path provisioned to provide option for Dynamic provisioning. Local Path Provisioner provides a way for the Kubernetes users to utilize the local storage in each node. Based on the user configuration, the Local Path Provisioner will create hostPath based persistent volume on the node automatically. It utilizes the features introduced by Kubernetes Local Persistent Volume feature, but make it a simpler solution than the built-in local volume feature in Kubernetes. <br />
  <a href="https://github.com/rancher/local-path-provisioner">https://github.com/rancher/local-path-provisioner</a></p>

  <pre><code>kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml  
  </code></pre>

  <p>Result: <br />
  <img src="/content/images/2021/03/DeepinScreenshot_select-area_20210307132702.png" alt="alt" /></p>

  <p>Then make this storage as default one:  </p>

  <pre><code>kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'  
  </code></pre>

  <p>We need to add Ephemeral storage as well, create a file with following content:  </p>

  <pre><code># https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6184#note_283677909
  apiVersion: v1  
  kind: LimitRange  
  metadata:  
    name: limit-ephemeral-storage
  spec:  
    limits:
    - default:
        ephemeral-storage: "5Gi"
      defaultRequest:
        ephemeral-storage: "300Mi"
      type: Container
  </code></pre>

  <p>save it as default-ephemeral-storage.yml file name and execute it with following command:  </p>

  <pre><code>kubectl apply -f default-ephemeral-storage.yml  
  </code></pre>

  <p>By following <a href="https://www.gitpod.io/docs/self-hosted/latest/install/configure-ingress/#using-lets-encrypt-to-generate-https-certificates">https://www.gitpod.io/docs/self-hosted/latest/install/configure-ingress/#using-lets-encrypt-to-generate-https-certificates</a> guidelines we can generate Let's Encrypt SSL certificate and add secretes to the Kubernetes cluster and Gitpod application. Now, that Gitpod requires actual real domain name this step is required.</p>

  <p>Add TLS cert to kubernetes secrets  </p>

  <pre><code>kubectl create secret tls tls-certs --cert=/home/vagrant/nemanja/secrets/https-certificates/fullchain.pem --key=/home/vagrant/nemanja/secrets/https-certificates/privkey.pem  
  </code></pre>

  <p>Next, let's install Helm: <br />
  <a href="https://helm.sh/docs/intro/install/">https://helm.sh/docs/intro/install/</a></p>

  <p>Helm now has an installer script that will automatically grab the latest version of Helm and install it locally.</p>

  <p>You can fetch that script and then execute it locally. It's well documented so that you can read through it and understand what it is doing before you run it.</p>

  <pre><code>$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
  $ chmod 700 get_helm.sh
  $ ./get_helm.sh
  </code></pre>

  <p><img src="/content/images/2021/03/DeepinScreenshot_select-area_20210306192410.png" alt="alt" /></p>

  <p>In this step we need to install WeaveNet (<a href="https://www.weave.works/">https://www.weave.works/</a>) module:  </p>

  <pre><code>kubectl apply -f https://raw.githubusercontent.com/weaveworks/weave/master/prog/weave-kube/weave-daemonset-k8s-1.9.yaml  
  </code></pre>

  <p>Or substitute for particular Kubernetes cluster used:  </p>

  <pre><code>kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"  
  </code></pre>

  <p>Gitpod schedules two kinds of workloads: the Gitpod installation itself (which we refer to as meta) and the workspaces. Ideally both types of workloads run on seperate nodes to make makes scaling easier. Depending on your cluster size that may not be feasible though. Either way, you need two node labels in your cluster:</p>

  <p>gitpod.io/workload<em>meta=true which marks the “meta” nodes and <br />
  gitpod.io/workload</em>workspace=true which marks the workspace nodes. <br />
  If you want to “mix ‘n match”, i.e., don’t separate the nodes, you can simply run:  </p>

  <pre><code>kubectl label node --all gitpod.io/workload_meta=true gitpod.io/workload_workspace=true  
  </code></pre>

  <p>At this point everything should be ready for installation of Gitpod application to our cluster.</p>

  <p>Based on the <a href="https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/">https://www.gitpod.io/docs/self-hosted/latest/install/install-on-kubernetes/</a> article we will add gitpod repository and install it.</p>

  <pre><code>vagrant@master:~$ helm repo add gitpod.io https://charts.gitpod.io  
  "gitpod.io" has been added to your repositories
  </code></pre>

  <pre><code>helm install gitpod gitpod.io/gitpod  
  </code></pre>

  <p>Run <strong>kubectl get pods</strong> and verify that all pods are in state <strong>RUNNING</strong>. If some are not, please see the Troubleshooting Guide (<a href="https://www.gitpod.io/docs/self-hosted/latest/install/troubleshooting/">https://www.gitpod.io/docs/self-hosted/latest/install/troubleshooting/</a>)</p>

  <p>If you are using this way most likely you will need to create single values.custom.yaml file with following content:  </p>

  <pre><code>components:  
    wsDaemon:
      containerRuntime:
        nodeRoots:
        - /var/run/docker/containerd
  </code></pre>

  <p>You can also do: <br />
  KUBE_EDITOR="nano" kubectl edit daemonsets.apps ws-daemon</p>

  <p>And adjust type: Socket area to the following:  </p>

  <pre><code>path: /var/run/docker/containerd/containerd.sock  
            type: Socket
  </code></pre>

  <p>Then run the following command to apply the changes.:  </p>

  <pre><code>helm upgrade --install -f values.custom.yaml gitpod gitpod.io/gitpod  
  </code></pre>


uuid: b60f4c9b-3706-4a4f-9e42-486b0afaeedf
title: Run VS Code IDE editor on any Web Hosting plafrom
slug: run-vs-code-ide-editor-on-any-web-hosting-plafrom
image: /content/images/2021/06/DeepinScreenshot_select-area_20210601231237-1.png
status: published
language: en_US
markdown:
  Hello,

  It's been a while since my last blog post, and today I would like to show you a neat way to temporarily or permanently boot VS Code IDE editor in your browser tab on any popular Web Hosting platform. This is useful if you need to hook on the random server and troubleshoot without the need to copy the codebase to an external server or on your local desktop or laptop computer.

  Let's start. First, you need to connect to the SSH terminal on your Web Hosting server (account). Most of the popular Web Hosting providers provide this nowadays.

  Basically, to boot VS Code IDE Editor we will follow https://github.com/cdr/code-server/blob/main/docs/install.md#standalone-releases instructions and boot standalone version.

  This can be used as a one-liner from a bash shell:
  ```
  VERSION=$(wget -qO- 'https://github.com/cdr/code-server/releases/latest' | grep -o 'tag/[v.0-9]*' | awk -F/ '{print substr($2,2)}' | tail -1) && mkdir -p ~/.local/lib ~/.local/bin && curl -fL https://github.com/cdr/code-server/releases/download/v$VERSION/code-server-$VERSION-linux-amd64.tar.gz | tar -C ~/.local/lib -xz && mv ~/.local/lib/code-server-$VERSION-linux-amd64 ~/.local/lib/code-server-$VERSION &&
  ln -s ~/.local/lib/code-server-$VERSION/bin/code-server ~/.local/bin/code-server && PATH="~/.local/bin:$PATH"
  ```

  When you get this finished type code-server just for init so config files are created then CTRL-C to terminate the process. By default, it reads on 8080 port which is probably occupied by Web server Nginx or Apache or similar.

  The config file is settled within the following path:
  ```
  ~/.config/code-server/config.yaml
  ```

  Output:
  ```
  www@svc-magentocommand-ml:~$ cat ~/.config/code-server/config.yaml
  bind-addr: 127.0.0.1:3333
  auth: password
  password: 7627d80b434561a3c2e0814b4
  cert: false
  ```
  Note: I usually put 3333 port and copy password somewhere since we will need it later.

  Once the config file is changed you can start the process again on the screen, using & at the end as a background process, but if you have a deeper level of access you could create systemd or supervisord service.

  The last part is to decide how you wish to connect. I prefer suggesting SSH Tunnel, but you can pick whatever you wish.

  For example, good output when you login using SSH tunnel to your's instance:
  ```
  www@svc-magentocommand-ml:~$ PATH="~/.local/bin:$PATH"
  www@svc-magentocommand-ml:~$ code-server
  [2021-06-01T21:09:59.200Z] info  code-server 3.10.2 387b12ef4ca404ffd39d84834e1f0776e9e3c005
  [2021-06-01T21:09:59.202Z] info  Using user-data-dir ~/.local/share/code-server
  [2021-06-01T21:09:59.212Z] info  Using config file ~/.config/code-server/config.yaml
  [2021-06-01T21:09:59.212Z] info  HTTP server listening on http://127.0.0.1:3333 
  [2021-06-01T21:09:59.212Z] info    - Authentication is enabled
  [2021-06-01T21:09:59.213Z] info      - Using password from ~/.config/code-server/config.yaml
  [2021-06-01T21:09:59.213Z] info    - Not serving HTTPS 
  ```

  Navigate to your favorite browser and open 127.0.0.1:3333

  ![alt](/content/images/2021/06/DeepinScreenshot_select-area_20210601231059.png)

  Enter the password from config.yaml file and enjoy!

  ![alt](/content/images/2021/06/DeepinScreenshot_select-area_20210601231237.png)

  Hope this article was helpful. In the next article, I will write little bit about how to debug code using Symfony var_dump server and combine that with VS Code IDE editor.

  Cheers!

html:
  <p>Hello,</p>

  <p>It's been a while since my last blog post, and today I would like to show you a neat way to temporarily or permanently boot VS Code IDE editor in your browser tab on any popular Web Hosting platform. This is useful if you need to hook on the random server and troubleshoot without the need to copy the codebase to an external server or on your local desktop or laptop computer.</p>

  <p>Let's start. First, you need to connect to the SSH terminal on your Web Hosting server (account). Most of the popular Web Hosting providers provide this nowadays.</p>

  <p>Basically, to boot VS Code IDE Editor we will follow <a href="https://github.com/cdr/code-server/blob/main/docs/install.md#standalone-releases">https://github.com/cdr/code-server/blob/main/docs/install.md#standalone-releases</a> instructions and boot standalone version.</p>

  <p>This can be used as a one-liner from a bash shell:  </p>

  <pre><code>VERSION=$(wget -qO- 'https://github.com/cdr/code-server/releases/latest' | grep -o 'tag/[v.0-9]*' | awk -F/ '{print substr($2,2)}' | tail -1) &amp;&amp; mkdir -p ~/.local/lib ~/.local/bin &amp;&amp; curl -fL https://github.com/cdr/code-server/releases/download/v$VERSION/code-server-$VERSION-linux-amd64.tar.gz | tar -C ~/.local/lib -xz &amp;&amp; mv ~/.local/lib/code-server-$VERSION-linux-amd64 ~/.local/lib/code-server-$VERSION &amp;&amp;  
  ln -s ~/.local/lib/code-server-$VERSION/bin/code-server ~/.local/bin/code-server &amp;&amp; PATH="~/.local/bin:$PATH"  
  </code></pre>

  <p>When you get this finished type code-server just for init so config files are created then CTRL-C to terminate the process. By default, it reads on 8080 port which is probably occupied by Web server Nginx or Apache or similar.</p>

  <p>The config file is settled within the following path:  </p>

  <pre><code>~/.config/code-server/config.yaml
  </code></pre>

  <p>Output:  </p>

  <pre><code>www@svc-magentocommand-ml:~$ cat ~/.config/code-server/config.yaml  
  bind-addr: 127.0.0.1:3333  
  auth: password  
  password: 7627d80b434561a3c2e0814b4  
  cert: false  
  </code></pre>

  <p>Note: I usually put 3333 port and copy password somewhere since we will need it later.</p>

  <p>Once the config file is changed you can start the process again on the screen, using &amp; at the end as a background process, but if you have a deeper level of access you could create systemd or supervisord service.</p>

  <p>The last part is to decide how you wish to connect. I prefer suggesting SSH Tunnel, but you can pick whatever you wish.</p>

  <p>For example, good output when you login using SSH tunnel to your's instance:  </p>

  <pre><code>www@svc-magentocommand-ml:~$ PATH="~/.local/bin:$PATH"  
  www@svc-magentocommand-ml:~$ code-server  
  [2021-06-01T21:09:59.200Z] info  code-server 3.10.2 387b12ef4ca404ffd39d84834e1f0776e9e3c005
  [2021-06-01T21:09:59.202Z] info  Using user-data-dir ~/.local/share/code-server
  [2021-06-01T21:09:59.212Z] info  Using config file ~/.config/code-server/config.yaml
  [2021-06-01T21:09:59.212Z] info  HTTP server listening on http://127.0.0.1:3333 
  [2021-06-01T21:09:59.212Z] info    - Authentication is enabled
  [2021-06-01T21:09:59.213Z] info      - Using password from ~/.config/code-server/config.yaml
  [2021-06-01T21:09:59.213Z] info    - Not serving HTTPS 
  </code></pre>

  <p>Navigate to your favorite browser and open 127.0.0.1:3333</p>

  <p><img src="/content/images/2021/06/DeepinScreenshot_select-area_20210601231059.png" alt="alt" /></p>

  <p>Enter the password from config.yaml file and enjoy!</p>

  <p><img src="/content/images/2021/06/DeepinScreenshot_select-area_20210601231237.png" alt="alt" /></p>

  <p>Hope this article was helpful. In the next article, I will write little bit about how to debug code using Symfony var_dump server and combine that with VS Code IDE editor.</p>

  <p>Cheers!</p>


uuid: 1a7b638e-95b0-43f1-b01d-1acaade406df
title: Speed Magento 2 Indexer optimization
slug: speed-magento-2-indexing-with-multi-thread-option
image: /content/images/2021/09/image-1.png
status: published
language: en_US
meta_title: Indexer optimization
markdown:
  Hello,

  The biggest bottleneck when you are using Magento 2 application is when your database grows. A huge number of products, orders, invoices, quotes, attributes, categories all can make application resources demanding. On top of that, the application was designed as a "single" threaded by default :( which adds slowness even more. To fix these parts and make Magento 2 indexing process faster again we will talk today about something which is not new in Magento 2 (started with 2.2.6 version), but somehow silently announced and still not used by default for unknown reasons. Many indexers (like catalog price for example) are now scoped and multithreaded.

  Magento 2 explained really nicely on https://devdocs.magento.com/guides/v2.4/extension-dev-guide/indexer-optimization.html page how to optimize Indexes by adjust batch sizes or translated the amount of data processed per each run. Besides that and configuring scopes which was explained a long time ago on https://devdocs.magento.com/guides/v2.2/config-guide/cli/config-cli-subcommands-index.html#config-cli-subcommands-index-reindex-parallel page, we can question ourselves what if we have many CPU cores available but still process single? Right... :(

  The best fix is to edit the env.php file and add the following line:
  ```
  'MAGE_INDEXER_THREADS_COUNT' => (int)`nproc`,
  ```
  This is going to fit your deployment to check how many CPU cores are available and use them all where possible.

  I have tested on fresh Magento 2.4.3 installation using Performance Fixtures (medium.xml) and here are results:

  ```
    Magento System Information  
                                

  +------------------+----------------------------------+
  | name             | value                            |
  +------------------+----------------------------------+
  | Name             | Magento                          |
  | Version          | dev-2.4-develop                  |
  | Edition          | Community                        |
  | Root             | /workspace/magento2gitpod        |
  | Application Mode | default                          |
  | Session          | redis                            |
  | Crypt Key        | 63fc9be49098a73ac467066d8c798b98 |
  | Install Date     | Wed, 29 Sep 2021 18:12:03 +0000  |
  | Cache Backend    | Cm_Cache_Backend_File            |
  | Vendors          | Magento                          |
  | Attribute Count  | 393                              |
  | Customer Count   | 2000                             |
  | Category Count   | 302                              |
  | Product Count    | 40000                            |
  +------------------+----------------------------------+
  ```

  Without:
  ```
  real    4m22.295s
  user    0m54.003s
  sys     0m6.056s
  ```

  With option added to env.php file:
  ```
  real    2m38.801s
  user    0m59.741s
  sys     0m7.201s
  ```

  Entire logic is grabbed from the Magento Indexer module or specific Model/ProcessManager.php file

  ![alt](/content/images/2021/09/DeepinScreenshot_select-area_20210930112325.png)

  Then checked if possible to raise Threads here:
  ![alt](/content/images/2021/09/DeepinScreenshot_select-area_20210930112443.png)

  **Conclusion:** <BR>
  Today we have learned that Magento 2 indexing can be optimized from many aspects. If multiple websites are used then it can be scoped by website type. You can also set dimensions per customer group especially useful for B2B type of store, but there is also a way to speed up things by executing indexing in parallel by raising the number of threads that your server allows from nproc variable from Linux.

  Hope this will help someone struggling. Wishing you all the best until the next article. Salute o/

html:
  <p>Hello,</p>

  <p>The biggest bottleneck when you are using Magento 2 application is when your database grows. A huge number of products, orders, invoices, quotes, attributes, categories all can make application resources demanding. On top of that, the application was designed as a "single" threaded by default :( which adds slowness even more. To fix these parts and make Magento 2 indexing process faster again we will talk today about something which is not new in Magento 2 (started with 2.2.6 version), but somehow silently announced and still not used by default for unknown reasons. Many indexers (like catalog price for example) are now scoped and multithreaded.</p>

  <p>Magento 2 explained really nicely on <a href="https://devdocs.magento.com/guides/v2.4/extension-dev-guide/indexer-optimization.html">https://devdocs.magento.com/guides/v2.4/extension-dev-guide/indexer-optimization.html</a> page how to optimize Indexes by adjust batch sizes or translated the amount of data processed per each run. Besides that and configuring scopes which was explained a long time ago on <a href="https://devdocs.magento.com/guides/v2.2/config-guide/cli/config-cli-subcommands-index.html#config-cli-subcommands-index-reindex-parallel">https://devdocs.magento.com/guides/v2.2/config-guide/cli/config-cli-subcommands-index.html#config-cli-subcommands-index-reindex-parallel</a> page, we can question ourselves what if we have many CPU cores available but still process single? Right... :(</p>

  <p>The best fix is to edit the env.php file and add the following line:  </p>

  <pre><code>'MAGE_INDEXER_THREADS_COUNT' =&gt; (int)`nproc`,  
  </code></pre>

  <p>This is going to fit your deployment to check how many CPU cores are available and use them all where possible.</p>

  <p>I have tested on fresh Magento 2.4.3 installation using Performance Fixtures (medium.xml) and here are results:</p>

  <pre><code>  Magento System Information  


  +------------------+----------------------------------+
  | name             | value                            |
  +------------------+----------------------------------+
  | Name             | Magento                          |
  | Version          | dev-2.4-develop                  |
  | Edition          | Community                        |
  | Root             | /workspace/magento2gitpod        |
  | Application Mode | default                          |
  | Session          | redis                            |
  | Crypt Key        | 63fc9be49098a73ac467066d8c798b98 |
  | Install Date     | Wed, 29 Sep 2021 18:12:03 +0000  |
  | Cache Backend    | Cm_Cache_Backend_File            |
  | Vendors          | Magento                          |
  | Attribute Count  | 393                              |
  | Customer Count   | 2000                             |
  | Category Count   | 302                              |
  | Product Count    | 40000                            |
  +------------------+----------------------------------+
  </code></pre>

  <p>Without:  </p>

  <pre><code>real    4m22.295s  
  user    0m54.003s  
  sys     0m6.056s  
  </code></pre>

  <p>With option added to env.php file:  </p>

  <pre><code>real    2m38.801s  
  user    0m59.741s  
  sys     0m7.201s  
  </code></pre>

  <p>Entire logic is grabbed from the Magento Indexer module or specific Model/ProcessManager.php file</p>

  <p><img src="/content/images/2021/09/DeepinScreenshot_select-area_20210930112325.png" alt="alt" /></p>

  <p>Then checked if possible to raise Threads here: <br />
  <img src="/content/images/2021/09/DeepinScreenshot_select-area_20210930112443.png" alt="alt" /></p>

  <p><strong>Conclusion:</strong> <BR>
  Today we have learned that Magento 2 indexing can be optimized from many aspects. If multiple websites are used then it can be scoped by website type. You can also set dimensions per customer group especially useful for B2B type of store, but there is also a way to speed up things by executing indexing in parallel by raising the number of threads that your server allows from nproc variable from Linux.</p>

  <p>Hope this will help someone struggling. Wishing you all the best until the next article. Salute o/</p>

meta_description:
  speed up things by executing indexing in parallel by raising the number of threads that your server allows from nproc variable from Linux.


uuid: 5eb6b7c1-a168-48ca-8b93-aa488dfa2faa
title: Convert REST API Web endpoints from Synchronous to Asynchronous in Magento 2
slug: convert-rest-api-web-endpoints-from-synchronous-to-asynchronous-in-magento-2
image: /content/images/2021/10/1_6nlfVAitkAztuSrJGwPdbQ.jpeg
status: published
language: en_US
markdown:
  Hello my friends,

  Today I am going to teach you how to convert any Magento 2 REST Web endpoint from Synchronous to Asynchronous method in Magento 2. 

  An asynchronous web endpoint intercepts messages to a Web API and writes them to the message queue. Each time the system accepts such an API request, it generates a UUID identifier. Magento includes this UUID when it adds the message to the queue. Then, a consumer reads the messages from the queue and executes them one by one.

  This method requires you to have async.operations.all constantly running so keep that in mind to edit env.php file and remove async.operations.all from consumer_runners basically removing this one that handles asynchronous and bulk API messages to be executed as a Magento 2 Cron task.

  ```
  php bin/magento queue:consumers:start async.operations.all 
  ```
  Note: You can configure supervisors or systemyd service to run this as a Service with the config to restart on the first minute after midnight to avoid known issues with certain Memory leaks which we are not going to talk about much today.

  The route to all asynchronous calls contains the prefix /async, added before /V1 of a standard synchronous endpoint.

  As a Demo, I have installed the latest Magento 2.4.3 version. You can use well known Gitpod.io service and my example for the Dev environment on the https://github.com/nemke82/magento2gitpod following link.

  First of all, make sure that your app/etc/env.php file is configured with correct RabbitMQ credentials since we are going to utilize REST endpoint asynchronous way using rabbitmq broker.

  Navigate to the Dashboard, and under System --> Integrations and add a new integration.

  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007215553.png)

  Then click on the Activate button:
  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007215625.png)

  Copy keys, tokens, and secrets to a safe place and then click the done button:
  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007215819.png)

  Next, you can log in to the RabbitMQ GUI to check if messages are being sent and if async.operations.all queue exists. This is how it looks like when everything is operational and ready:
  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007220103.png)

  Now is a good time to test POST requests. In my example, I have used the famous/known tool Postman and the task is to create a demo Customer. This article https://devdocs.magento.com/guides/v2.4/rest/tutorials/orders/order-create-customer.html guided me an example with the correct Payload to send a request to my Demo Magento 2 installation.

  So URL with POST selected is:
  https://URL-TO-MAGENTO/rest/all/V1/customers

  OAuth 1.0 and entered credentials:
  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007220652.png)

  With following body payload:
  ```
  {
    "customer": {
      "email": "async-jdoe1@example.com",
      "firstname": "Jane-async1",
      "lastname": "Doe-async1",
      "addresses": [
        {
          "defaultShipping": true,
          "defaultBilling": true,
          "firstname": "Jane",
          "lastname": "Doe",
          "region": {
            "regionCode": "NY",
            "region": "New York",
            "regionId": 43
          },
          "postcode": "10755",
          "street": [
            "123 Oak Ave"
          ],
          "city": "Purchase",
          "telephone": "512-555-1111",
          "countryId": "US"
        }
      ]
    },
    "password": "Password1"
  }
  ```

  Now the first test is using Synchronous way, executed and it went well:
  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007221040.png)

  But still, nothing is going Async way, we just sent a REST API call to Magento and it is done. Now let's try to change the POST URL

  From:
  https://URL-OF-MAGENTO/rest/all/V1/customers

  To:
  https://URL-OF-MAGENTO/rest/all/async/V1/customers

  And execute:
  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007221415.png)

  ![alt](/content/images/2021/10/DeepinScreenshot_select-area_20211007222229.png)

  ```
  $ n98-magerun2 customer:list
  +----+--------------+------------+-------------------------+---------+---------------------+
  | id | firstname    | lastname   | email                   | website | created_at          |
  +----+--------------+------------+-------------------------+---------+---------------------+
  | 1  | Jane-async1  | Doe-async1 | async-jdoe1@example.com | 1       | 2021-10-07 20:10:10 |
  | 2  | Jane-async12 | Doe-async2 | async-jdoe2@example.com | 1       | 2021-10-07 20:13:07 |
  +----+--------------+------------+-------------------------+---------+---------------------+
  ```

  Exactly how the https://devdocs.magento.com/guides/v2.4/rest/asynchronous-web-endpoints.html article advertises. It changed behavior from Synchronous to Asynchronous and went directly to RabbitMQ and got consumed. I've tested the time of execution between them and the Asynchronous way is a lot faster and it does not lock Database with MySQL queries at all. My idea is to push a variable and try to create 1000 customers using the Async way and compare it with the Synchronous way. The next step is to create a module that is going to convert the REST API endpoint or simply create a Composer patch by patching a Magento 2 Core module. Just make sure that the RabbitMQ broker is always ready and constantly running.

html:
  <p>Hello my friends,</p>

  <p>Today I am going to teach you how to convert any Magento 2 REST Web endpoint from Synchronous to Asynchronous method in Magento 2. </p>

  <p>An asynchronous web endpoint intercepts messages to a Web API and writes them to the message queue. Each time the system accepts such an API request, it generates a UUID identifier. Magento includes this UUID when it adds the message to the queue. Then, a consumer reads the messages from the queue and executes them one by one.</p>

  <p>This method requires you to have async.operations.all constantly running so keep that in mind to edit env.php file and remove async.operations.all from consumer_runners basically removing this one that handles asynchronous and bulk API messages to be executed as a Magento 2 Cron task.</p>

  <pre><code>php bin/magento queue:consumers:start async.operations.all  
  </code></pre>

  <p>Note: You can configure supervisors or systemyd service to run this as a Service with the config to restart on the first minute after midnight to avoid known issues with certain Memory leaks which we are not going to talk about much today.</p>

  <p>The route to all asynchronous calls contains the prefix /async, added before /V1 of a standard synchronous endpoint.</p>

  <p>As a Demo, I have installed the latest Magento 2.4.3 version. You can use well known Gitpod.io service and my example for the Dev environment on the <a href="https://github.com/nemke82/magento2gitpod">https://github.com/nemke82/magento2gitpod</a> following link.</p>

  <p>First of all, make sure that your app/etc/env.php file is configured with correct RabbitMQ credentials since we are going to utilize REST endpoint asynchronous way using rabbitmq broker.</p>

  <p>Navigate to the Dashboard, and under System --> Integrations and add a new integration.</p>

  <p><img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007215553.png" alt="alt" /></p>

  <p>Then click on the Activate button: <br />
  <img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007215625.png" alt="alt" /></p>

  <p>Copy keys, tokens, and secrets to a safe place and then click the done button: <br />
  <img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007215819.png" alt="alt" /></p>

  <p>Next, you can log in to the RabbitMQ GUI to check if messages are being sent and if async.operations.all queue exists. This is how it looks like when everything is operational and ready: <br />
  <img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007220103.png" alt="alt" /></p>

  <p>Now is a good time to test POST requests. In my example, I have used the famous/known tool Postman and the task is to create a demo Customer. This article <a href="https://devdocs.magento.com/guides/v2.4/rest/tutorials/orders/order-create-customer.html">https://devdocs.magento.com/guides/v2.4/rest/tutorials/orders/order-create-customer.html</a> guided me an example with the correct Payload to send a request to my Demo Magento 2 installation.</p>

  <p>So URL with POST selected is: <br />
  <a href="https://URL-TO-MAGENTO/rest/all/V1/customers">https://URL-TO-MAGENTO/rest/all/V1/customers</a></p>

  <p>OAuth 1.0 and entered credentials: <br />
  <img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007220652.png" alt="alt" /></p>

  <p>With following body payload:  </p>

  <pre><code>{
    "customer": {
      "email": "async-jdoe1@example.com",
      "firstname": "Jane-async1",
      "lastname": "Doe-async1",
      "addresses": [
        {
          "defaultShipping": true,
          "defaultBilling": true,
          "firstname": "Jane",
          "lastname": "Doe",
          "region": {
            "regionCode": "NY",
            "region": "New York",
            "regionId": 43
          },
          "postcode": "10755",
          "street": [
            "123 Oak Ave"
          ],
          "city": "Purchase",
          "telephone": "512-555-1111",
          "countryId": "US"
        }
      ]
    },
    "password": "Password1"
  }
  </code></pre>

  <p>Now the first test is using Synchronous way, executed and it went well: <br />
  <img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007221040.png" alt="alt" /></p>

  <p>But still, nothing is going Async way, we just sent a REST API call to Magento and it is done. Now let's try to change the POST URL</p>

  <p>From: <br />
  <a href="https://URL-OF-MAGENTO/rest/all/V1/customers">https://URL-OF-MAGENTO/rest/all/V1/customers</a></p>

  <p>To: <br />
  <a href="https://URL-OF-MAGENTO/rest/all/async/V1/customers">https://URL-OF-MAGENTO/rest/all/async/V1/customers</a></p>

  <p>And execute: <br />
  <img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007221415.png" alt="alt" /></p>

  <p><img src="/content/images/2021/10/DeepinScreenshot_select-area_20211007222229.png" alt="alt" /></p>

  <pre><code>$ n98-magerun2 customer:list
  +----+--------------+------------+-------------------------+---------+---------------------+
  | id | firstname    | lastname   | email                   | website | created_at          |
  +----+--------------+------------+-------------------------+---------+---------------------+
  | 1  | Jane-async1  | Doe-async1 | async-jdoe1@example.com | 1       | 2021-10-07 20:10:10 |
  | 2  | Jane-async12 | Doe-async2 | async-jdoe2@example.com | 1       | 2021-10-07 20:13:07 |
  +----+--------------+------------+-------------------------+---------+---------------------+
  </code></pre>

  <p>Exactly how the <a href="https://devdocs.magento.com/guides/v2.4/rest/asynchronous-web-endpoints.html">https://devdocs.magento.com/guides/v2.4/rest/asynchronous-web-endpoints.html</a> article advertises. It changed behavior from Synchronous to Asynchronous and went directly to RabbitMQ and got consumed. I've tested the time of execution between them and the Asynchronous way is a lot faster and it does not lock Database with MySQL queries at all. My idea is to push a variable and try to create 1000 customers using the Async way and compare it with the Synchronous way. The next step is to create a module that is going to convert the REST API endpoint or simply create a Composer patch by patching a Magento 2 Core module. Just make sure that the RabbitMQ broker is always ready and constantly running.</p>


uuid: 7e109b4d-b22f-4edf-8ef4-011280621d1c
title: Certified Kubernetes Administrator
slug: certified-kubernetes-administrator
status: draft
language: en_US
markdown:
  Due to the nature of my work and engagements in the Web Hosting industry, I always wanted to get in touch with Container orchestration applications. My road always ends with demonstrating the skills, knowledge, and competencies to perform the responsibilities of a Kubernetes Administrator. Earners demonstrated proficiency in Application Lifecycle Management, Installation, Configuration & Validation, Core Concepts, Networking, Scheduling, Security, Cluster Maintenance, Logging / Monitoring, Storage, and Troubleshooting.

  What I want to share today is my road.

  ![alt](/content/images/2021/11/DeepinScreenshot_select-area_20211104134956.png)

html:
  <p>Due to the nature of my work and engagements in the Web Hosting industry, I always wanted to get in touch with Container orchestration applications. My road always ends with demonstrating the skills, knowledge, and competencies to perform the responsibilities of a Kubernetes Administrator. Earners demonstrated proficiency in Application Lifecycle Management, Installation, Configuration &amp; Validation, Core Concepts, Networking, Scheduling, Security, Cluster Maintenance, Logging / Monitoring, Storage, and Troubleshooting.</p>

  <p>What I want to share today is my road.</p>

  <p><img src="/content/images/2021/11/DeepinScreenshot_select-area_20211104134956.png" alt="alt" /></p>


uuid: 7d4909ac-265f-4dbf-b9e1-a42c508ec4ed
title: Warm up pages using Trafilatura Web Scraping tool
slug: warm-up-pages-using-trafilatura-web-scraping-tool-2
image: /content/images/2021/12/asger.jpg
status: published
language: en_US
markdown:
  Hello,

  I wanted to share today how you can use this very user-friendly web scraping tool and warm-up pages on your website or store.

  $ python3 --version <BR>
  Python 3.8.6 # version 3.6 or higher is fine

  ```
  pip3 install trafilatura
  ```

  There are two ways to generate a list of links.

  The following examples return lists of links. If --list is absent the pages that have been found are directly retrieved, processed, and returned in the chosen output format (per default: TXT and standard output).

  run link discovery through a sitemap for sitemaps.org and store the resulting links in a file
  ```
  trafilatura --sitemap "https://www.sitemaps.org/" --list > mylinks.txt
  ```
  using an already known sitemap URL
  ```
  trafilatura --sitemap "https://www.sitemaps.org/sitemap.xml" --list
  ```
  ==Substitute sitemaps.org address with your website or store==

  Next, when links are ready we can start and crawl website pages. We will use a popular curl (https://linux.die.net/man/1/curl) tool combined with parallel (https://linux.die.net/man/1/parallel) to get things done faster.

  ```
  pip3 install trafilatura
  sudo apt-get update -y
  sudo apt-get install -y parallel
  trafilatura --sitemap "https://website-address.com/" --list > mylinks.txt;
  rm -f config;
  cat mylinks.txt | while read f; do echo -e "curl -s -o /dev/null $f" >> config; done;
  curl -K config
  ```

  2nd faster way using parallel:
  ```
  rm -f config; echo 'curl -s -o /dev/null $1' > config; chmod a+rwx config; cat mylinks.txt | parallel -I% -j 5 ./config %&
  ```

  You can combine and make a single liner and run this as a cron task, or you can check/test the Docker image I've created just for this and run it as an image or configure it as Kubernetes pod as a scheduled task.

  As a free gift at the end of this article please check/test instructions from https://github.com/nemke82/asger-cache-warm my GitHub repository and use that as a start point in your Trafilatura and Parallel journey. Good luck!

html:
  <p>Hello,</p>

  <p>I wanted to share today how you can use this very user-friendly web scraping tool and warm-up pages on your website or store.</p>

  <p>$ python3 --version <BR>
  Python 3.8.6 # version 3.6 or higher is fine</p>

  <pre><code>pip3 install trafilatura  
  </code></pre>

  <p>There are two ways to generate a list of links.</p>

  <p>The following examples return lists of links. If --list is absent the pages that have been found are directly retrieved, processed, and returned in the chosen output format (per default: TXT and standard output).</p>

  <p>run link discovery through a sitemap for sitemaps.org and store the resulting links in a file  </p>

  <pre><code>trafilatura --sitemap "https://www.sitemaps.org/" --list &gt; mylinks.txt  
  </code></pre>

  <p>using an already known sitemap URL  </p>

  <pre><code>trafilatura --sitemap "https://www.sitemaps.org/sitemap.xml" --list  
  </code></pre>

  <p><mark>Substitute sitemaps.org address with your website or store</mark></p>

  <p>Next, when links are ready we can start and crawl website pages. We will use a popular curl (<a href="https://linux.die.net/man/1/curl">https://linux.die.net/man/1/curl</a>) tool combined with parallel (<a href="https://linux.die.net/man/1/parallel">https://linux.die.net/man/1/parallel</a>) to get things done faster.</p>

  <pre><code>pip3 install trafilatura  
  sudo apt-get update -y  
  sudo apt-get install -y parallel  
  trafilatura --sitemap "https://website-address.com/" --list &gt; mylinks.txt;  
  rm -f config;  
  cat mylinks.txt | while read f; do echo -e "curl -s -o /dev/null $f" &gt;&gt; config; done;  
  curl -K config  
  </code></pre>

  <p>2nd faster way using parallel:  </p>

  <pre><code>rm -f config; echo 'curl -s -o /dev/null $1' &gt; config; chmod a+rwx config; cat mylinks.txt | parallel -I% -j 5 ./config %&amp;  
  </code></pre>

  <p>You can combine and make a single liner and run this as a cron task, or you can check/test the Docker image I've created just for this and run it as an image or configure it as Kubernetes pod as a scheduled task.</p>

  <p>As a free gift at the end of this article please check/test instructions from <a href="https://github.com/nemke82/asger-cache-warm">https://github.com/nemke82/asger-cache-warm</a> my GitHub repository and use that as a start point in your Trafilatura and Parallel journey. Good luck!</p>


uuid: 9a306e97-c855-432c-9fda-26bd9c47a9fb
title: Use Terser to minify all JavaScript assets in Magento 2
slug: use-terser-to-minify-all-javascript-assets
status: published
language: en_US
markdown:
  Hello,

  This is probably my last blog post in 2021. This year was really dynamic and interesting. While I was checking the Performance and SEO score for one of the clients hosted I noticed that Google is having new interesting recommendations for minifying all Javascript assets.

  ![alt](/content/images/2021/12/DeepinScreenshot_select-area_20211227222024.png)

  Terser is the JavaScript minifier. It processes JavaScript files as well as the compiled output from other languages like CoffeeScript and TypeScript and transpilers like Babel.

  https://github.com/terser/terser

  First, make sure you have installed the latest version of node.js (You may need to restart your computer after this step).

  From NPM for use as a command-line app:
  ```
  npm install terser -g
  ```

  Output:
  ```
  $ node -v
  v17.3.0

  $ npm install terser -g

  added 6 packages, and audited 7 packages in 687ms

  found 0 vulnerabilities

  $ whereis terser
  terser: /usr/local/nvm/versions/node/v17.3.0/bin/terser
  ```

  Usage is really simple:
  https://github.com/terser/terser#command-line-usage

  Terser can take multiple input files. It's recommended that you pass the input files first, then pass the options. Terser will parse input files in sequence and apply any compression options. The files are parsed in the same global scope, that is, a reference from a file to some variable/function declared in another file will be matched properly.

  If no input file is specified, Terser will read from STDIN.

  If you wish to pass your options before the input files, separate the two with a double dash to prevent input files from being used as option arguments:
  ```
  terser --compress --mangle -- input.js
  ```
  In Magento 2 specifically, we can implement the following after setup:static-content:deploy (Static asset) deployment is processed, but before that, we need to know the path of our Theme directory.

  For example, let's try now to bundle the default Luma theme on a specific path for en_US language:
  ```
  THEME_FOLDER=('/workspace/magento2gitpod/pub/static/frontend/Magento/luma/en_US')
  ```

  ![alt](/content/images/2021/12/terser1.png)

  Now, let's bundle files with the following line:
  ```
  find ${THEME_FOLDER[@]} \( -name '*.js' -not -name '*.min.js' -not -name 'requirejs-bundle-config.js' \) -exec /usr/local/nvm/versions/node/v17.3.0/bin/terser \{} -c -m reserved=['$','jQuery','define','require','exports'] -o \{} \;
  ```
  Note: we skipped some of the patterns like *.min.js and requirejs-bundle-config.js, but if you are using MagePack or any other popular Bundling tool try to skip their Core bundle files generated. For example, for Baler tool:

  -not -name 'core-bundle.js' 

  You can test and Minify them as well, but my advice is to test this before adding this to your Production environment.

  ![alt](/content/images/2021/12/terser2.png)

  The last step is to clear Magento 2 cache and Redis if used and test.

  Run audit test again, let's say on https://web.dev/measure/ website and compare results before/after

  ![alt](/content/images/2021/12/terser3.png)

  Disable built-in Minify mechanism for Javascript minify and integrate this tool and how to convert your Theme into your Deployment process and enjoy the performance boost.

  \o/ Hope this was helpful.

html:
  <p>Hello,</p>

  <p>This is probably my last blog post in 2021. This year was really dynamic and interesting. While I was checking the Performance and SEO score for one of the clients hosted I noticed that Google is having new interesting recommendations for minifying all Javascript assets.</p>

  <p><img src="/content/images/2021/12/DeepinScreenshot_select-area_20211227222024.png" alt="alt" /></p>

  <p>Terser is the JavaScript minifier. It processes JavaScript files as well as the compiled output from other languages like CoffeeScript and TypeScript and transpilers like Babel.</p>

  <p><a href="https://github.com/terser/terser">https://github.com/terser/terser</a></p>

  <p>First, make sure you have installed the latest version of node.js (You may need to restart your computer after this step).</p>

  <p>From NPM for use as a command-line app:  </p>

  <pre><code>npm install terser -g  
  </code></pre>

  <p>Output:  </p>

  <pre><code>$ node -v
  v17.3.0

  $ npm install terser -g

  added 6 packages, and audited 7 packages in 687ms

  found 0 vulnerabilities

  $ whereis terser
  terser: /usr/local/nvm/versions/node/v17.3.0/bin/terser  
  </code></pre>

  <p>Usage is really simple: <br />
  <a href="https://github.com/terser/terser#command-line-usage">https://github.com/terser/terser#command-line-usage</a></p>

  <p>Terser can take multiple input files. It's recommended that you pass the input files first, then pass the options. Terser will parse input files in sequence and apply any compression options. The files are parsed in the same global scope, that is, a reference from a file to some variable/function declared in another file will be matched properly.</p>

  <p>If no input file is specified, Terser will read from STDIN.</p>

  <p>If you wish to pass your options before the input files, separate the two with a double dash to prevent input files from being used as option arguments:  </p>

  <pre><code>terser --compress --mangle -- input.js  
  </code></pre>

  <p>In Magento 2 specifically, we can implement the following after setup:static-content:deploy (Static asset) deployment is processed, but before that, we need to know the path of our Theme directory.</p>

  <p>For example, let's try now to bundle the default Luma theme on a specific path for en_US language:  </p>

  <pre><code>THEME_FOLDER=('/workspace/magento2gitpod/pub/static/frontend/Magento/luma/en_US')  
  </code></pre>

  <p><img src="/content/images/2021/12/terser1.png" alt="alt" /></p>

  <p>Now, let's bundle files with the following line:  </p>

  <pre><code>find ${THEME_FOLDER[@]} \( -name '*.js' -not -name '*.min.js' -not -name 'requirejs-bundle-config.js' \) -exec /usr/local/nvm/versions/node/v17.3.0/bin/terser \{} -c -m reserved=['$','jQuery','define','require','exports'] -o \{} \;  
  </code></pre>

  <p>Note: we skipped some of the patterns like *.min.js and requirejs-bundle-config.js, but if you are using MagePack or any other popular Bundling tool try to skip their Core bundle files generated. For example, for Baler tool:</p>

  <p>-not -name 'core-bundle.js' </p>

  <p>You can test and Minify them as well, but my advice is to test this before adding this to your Production environment.</p>

  <p><img src="/content/images/2021/12/terser2.png" alt="alt" /></p>

  <p>The last step is to clear Magento 2 cache and Redis if used and test.</p>

  <p>Run audit test again, let's say on <a href="https://web.dev/measure/">https://web.dev/measure/</a> website and compare results before/after</p>

  <p><img src="/content/images/2021/12/terser3.png" alt="alt" /></p>

  <p>Disable built-in Minify mechanism for Javascript minify and integrate this tool and how to convert your Theme into your Deployment process and enjoy the performance boost.</p>

  <p>\o/ Hope this was helpful.</p>

image:
  /content/images/2021/12/68747470733a2f2f7465727365722e6f72672f696d672f7465727365722d62616e6e65722d6c6f676f2e706e67.png


uuid: 29384b00-1e8d-49b0-89cf-9f064eff6df6
title: L2 caching in Magento
slug: l2-caching-in-magento
image: /content/images/2022/03/cache.jpg
status: published
language: en_US
markdown:
  Hello my friends,

  It's been a while since my last post. I wanted today to share what L2 caching in Magento is and what are two ways you can utilize it. 

  So what L2 caching is?
  In really simple words to reduce network congestion and bandwidth usage between microservices, this mechanism is implemented using \\Magento\\Framework\\Cache\\Backend\\RemoteSynchronizedCache class. 

  Magento stores the hashed data version in Redis, with the suffix ‘:hash’ appended to the regular key. In case of an outdated local cache, the data is transferred to the local machine with a cache adapter.

  Magento or Devdocs gave really nice example on how to modify or replace the existing cache section in the app/etc/env.php file (https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html), so basically it does what it does stores local backend to RAM, but what I want to share todate is that you can load balance between two Redis environments instead, let's say we can make one locally hosted and one on another node using well network.

  ```
  'cache' => [
          'frontend' => [
              'default' => [
                  'backend' => '\\Magento\\Framework\\Cache\\Backend\\RemoteSynchronizedCache',
                  'backend_options' => [
                      'remote_backend' => '\\Magento\\Framework\\Cache\\Backend\\Redis',
                      'remote_backend_options' => [
                          'persistent' => 0,
                          'server' => 'redis-somewhere-in-the-cloud',
                          'database' => '1',
                          'port' => '6379',
                          'password' => '',
                          'compress_data' => '1'
                      ],
                      'local_backend' => '\\Magento\\Framework\\Cache\\Backend\\Redis',
                      'local_backend_options' => [
                          'persistent' => 0,
                          'server' => '127.0.0.1',
                          'database' => '3',
                          'port' => '6379',
                          'password' => '',
                          'compress_data' => '1'
                      ],
                      'use_stale_cache' => false
                  ],
                  'frontend_options' => [
                      'write_control' => false
                  ],
                  'id_prefix' => 'd01_'
              ],
              'page_cache' => [
                  'id_prefix' => 'd01_',
                  'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                  'backend_options' => [
                      'server' => '127.0.0.1',
                      'database' => '2',
                      'port' => '6379',
                      'password' => '',
                      'compress_data' => '0',
                      'compression_lib' => ''
                  ]
              ]
          ],
          'allow_parallel_generation' => false
      ],
  ```

  In my example using https://github.com/nemke82/magento2gitpod I've made a tests and indeed data is spread between two services.

  Keyspace
  db0:keys=1,expires=1,avg_ttl=11052265
  **db1:keys=148,expires=42,avg_ttl=786241446** <-- 1st Redis
  **db3:keys=62,expires=21,avg_ttl=821121720** <-- Failover Redis environment <BR>
  gitpod /workspace/magento2gitpod $ redis-cli info

  Magento recommends using Redis for remote caching (\Magento\Framework\Cache\Backend\Redis) and Cm_Cache_Backend_File for the local caching of data in shared memory, using 'local_backend_options' => ['cache_dir' => '/dev/shm/'], but as you can see we can use Redis remote or locally hosted instead and bring other benefits from it.

  One of them is:
  https://devdocs.magento.com/guides/v2.4/config-guide/redis/redis-pg-cache.html#redis-preload-feature

  and

  https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html#stale-cache-options

  Hope this article helps. Good luck optimizing!

html:
  <p>Hello my friends,</p>

  <p>It's been a while since my last post. I wanted today to share what L2 caching in Magento is and what are two ways you can utilize it. </p>

  <p>So what L2 caching is? <br />
  In really simple words to reduce network congestion and bandwidth usage between microservices, this mechanism is implemented using \Magento\Framework\Cache\Backend\RemoteSynchronizedCache class. </p>

  <p>Magento stores the hashed data version in Redis, with the suffix ‘:hash’ appended to the regular key. In case of an outdated local cache, the data is transferred to the local machine with a cache adapter.</p>

  <p>Magento or Devdocs gave really nice example on how to modify or replace the existing cache section in the app/etc/env.php file (<a href="https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html">https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html</a>), so basically it does what it does stores local backend to RAM, but what I want to share todate is that you can load balance between two Redis environments instead, let's say we can make one locally hosted and one on another node using well network.</p>

  <pre><code>'cache' =&gt; [  
          'frontend' =&gt; [
              'default' =&gt; [
                  'backend' =&gt; '\\Magento\\Framework\\Cache\\Backend\\RemoteSynchronizedCache',
                  'backend_options' =&gt; [
                      'remote_backend' =&gt; '\\Magento\\Framework\\Cache\\Backend\\Redis',
                      'remote_backend_options' =&gt; [
                          'persistent' =&gt; 0,
                          'server' =&gt; 'redis-somewhere-in-the-cloud',
                          'database' =&gt; '1',
                          'port' =&gt; '6379',
                          'password' =&gt; '',
                          'compress_data' =&gt; '1'
                      ],
                      'local_backend' =&gt; '\\Magento\\Framework\\Cache\\Backend\\Redis',
                      'local_backend_options' =&gt; [
                          'persistent' =&gt; 0,
                          'server' =&gt; '127.0.0.1',
                          'database' =&gt; '3',
                          'port' =&gt; '6379',
                          'password' =&gt; '',
                          'compress_data' =&gt; '1'
                      ],
                      'use_stale_cache' =&gt; false
                  ],
                  'frontend_options' =&gt; [
                      'write_control' =&gt; false
                  ],
                  'id_prefix' =&gt; 'd01_'
              ],
              'page_cache' =&gt; [
                  'id_prefix' =&gt; 'd01_',
                  'backend' =&gt; 'Magento\\Framework\\Cache\\Backend\\Redis',
                  'backend_options' =&gt; [
                      'server' =&gt; '127.0.0.1',
                      'database' =&gt; '2',
                      'port' =&gt; '6379',
                      'password' =&gt; '',
                      'compress_data' =&gt; '0',
                      'compression_lib' =&gt; ''
                  ]
              ]
          ],
          'allow_parallel_generation' =&gt; false
      ],
  </code></pre>

  <p>In my example using <a href="https://github.com/nemke82/magento2gitpod">https://github.com/nemke82/magento2gitpod</a> I've made a tests and indeed data is spread between two services.</p>

  <p>Keyspace <br />
  db0:keys=1,expires=1,avg<em>ttl=11052265 <br />
  <strong>db1:keys=148,expires=42,avg</em>ttl=786241446</strong> &lt;-- 1st Redis
  <strong>db3:keys=62,expires=21,avg_ttl=821121720</strong> &lt;-- Failover Redis environment <BR>
  gitpod /workspace/magento2gitpod $ redis-cli info</p>

  <p>Magento recommends using Redis for remote caching (\Magento\Framework\Cache\Backend\Redis) and Cm<em>Cache</em>Backend<em>File for the local caching of data in shared memory, using 'local</em>backend<em>options' => ['cache</em>dir' => '/dev/shm/'], but as you can see we can use Redis remote or locally hosted instead and bring other benefits from it.</p>

  <p>One of them is: <br />
  <a href="https://devdocs.magento.com/guides/v2.4/config-guide/redis/redis-pg-cache.html#redis-preload-feature">https://devdocs.magento.com/guides/v2.4/config-guide/redis/redis-pg-cache.html#redis-preload-feature</a></p>

  <p>and</p>

  <p><a href="https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html#stale-cache-options">https://devdocs.magento.com/guides/v2.4/config-guide/cache/two-level-cache.html#stale-cache-options</a></p>

  <p>Hope this article helps. Good luck optimizing!</p>


uuid: b76be9d7-2d3c-43b3-92bb-a3042f623dd1
title: Manage remote Kubernetes clusters with kubectl and Lens
slug: manage-remote-kubernetes-clusters-with-kubectl-and-lens
image: /content/images/2022/05/00c71841fc9d2334f53b68a74877865c.png
status: published
language: en_US
markdown:
  Hello,

  This article will explain how to access the Kubernetes cluster remotely using the SSH Tunnel mechanism. 

  The most popular tool to manage Kubernetes clusters from CLI is called kubectl. The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.

  Kubernetes.io page https://kubernetes.io/docs/tasks/tools/ explains how you can install it on your desired Operating system, but today I will focus on something that I use as a Desktop, Fedora Linux.

  Before you begin, please note that you must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.24 client can communicate with v1.23, v1.24, and v1.25 control planes. Using the latest compatible version of kubectl helps avoid unforeseen issues.

  What vendor page says https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/ you can simply use curl and download a specific version.

  In my example, I've had to install the 1.15.1 version to match the Kubernetes cluster running.

  ```
  curl -LO https://dl.k8s.io/release/v1.15.1/bin/linux/amd64/kubectl
  ```

  Next, we can copy kubectl to /usr/sbin or /usr/bin but we can also edit ~/.bashrc and add an alias as a workaround:
  ```
  alias kubectl='/home/nemke/kubectl'
  ```

  We can test the tool, it should list basic helper commands. Next, we need to create .kube directory and copy over the config file from the Kubernetes master or worker node.
  ```
  mkdir -p ~/.kube
  ```

  You can use rsync,SCP, or similar or simply copy/paste the config file and leave it under ~/.kube/config path.

  Edit the config file and adjust server: section with the following:
  ```
  server: https://127.0.0.1:6443
  ```

  Under - cluster: line we need to add:
  ```
  insecure-skip-tls-verify: true
  ```

  And comment certificate-authority-data: if your cluster has it.

  Picture of config file how it should look like:
  ![alt](/content/images/2022/05/DeepinScreenshot_select-area_20220513110022.png)

  Login to the Kubernetes master or worker node and grab the Cluster IP address from /etc/hosts file. You can use kubectl tool as well to get that info, but let's use this way.

  Now it's time to setup SSH tunnel, this is working example:
  ```
  ssh -f username@IP-OF-SERVER -L 6443:CLUSTER-IP:6443 -N
  ```

  When executed it will return back. That's it! Time to test...

  ![alt](/content/images/2022/05/DeepinScreenshot_select-area_20220513110521.png)

  You are now managing Kubernetes cluster from your local environment. Good luck!

  Let's now test to put everything under UI. I've selected Lens because I think this IDE Kubernetes editor is brilliant when you need to troubleshoot and analyzing cluster events or simply put multiple container logs and monitor them. Lens – the Kubernetes IDE — is fruit of a Mirantis-sponsored open source project. Available for Linux, Mac, and Windows, Lens gives you a powerful interface and toolkit for managing, visualizing, and interacting with multiple Kubernetes clusters, while remaining always in proper context.

  Visit https://k8slens.dev/, download your desired application, and continue.

  When you log in and browse Clusters you will see your's right there listed:
  ![alt](/content/images/2022/05/DeepinScreenshot_select-area_20220513110523.png)

  Click on it and enjoy analyzing clusters and logging in to the nodes. You will still have the help of the terminal where kubectl runs and you can execute commands:
  ![alt](/content/images/2022/05/DeepinScreenshot_select-area_20220513111043.png)

  Final toughts/conclusions...
  ![alt](/content/images/2022/05/its-a-kubernetes-cluster-i-know-this-1.jpg)

  I hope this article was helpful. Using this method you can add multiple Kubernetes Clusters on your Desktop environment and manage it successfully. Good luck!

html:
  <p>Hello,</p>

  <p>This article will explain how to access the Kubernetes cluster remotely using the SSH Tunnel mechanism. </p>

  <p>The most popular tool to manage Kubernetes clusters from CLI is called kubectl. The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.</p>

  <p>Kubernetes.io page <a href="https://kubernetes.io/docs/tasks/tools/">https://kubernetes.io/docs/tasks/tools/</a> explains how you can install it on your desired Operating system, but today I will focus on something that I use as a Desktop, Fedora Linux.</p>

  <p>Before you begin, please note that you must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.24 client can communicate with v1.23, v1.24, and v1.25 control planes. Using the latest compatible version of kubectl helps avoid unforeseen issues.</p>

  <p>What vendor page says <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/">https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/</a> you can simply use curl and download a specific version.</p>

  <p>In my example, I've had to install the 1.15.1 version to match the Kubernetes cluster running.</p>

  <pre><code>curl -LO https://dl.k8s.io/release/v1.15.1/bin/linux/amd64/kubectl  
  </code></pre>

  <p>Next, we can copy kubectl to /usr/sbin or /usr/bin but we can also edit ~/.bashrc and add an alias as a workaround:  </p>

  <pre><code>alias kubectl='/home/nemke/kubectl'  
  </code></pre>

  <p>We can test the tool, it should list basic helper commands. Next, we need to create .kube directory and copy over the config file from the Kubernetes master or worker node.  </p>

  <pre><code>mkdir -p ~/.kube  
  </code></pre>

  <p>You can use rsync,SCP, or similar or simply copy/paste the config file and leave it under ~/.kube/config path.</p>

  <p>Edit the config file and adjust server: section with the following:  </p>

  <pre><code>server: https://127.0.0.1:6443  
  </code></pre>

  <p>Under - cluster: line we need to add:  </p>

  <pre><code>insecure-skip-tls-verify: true  
  </code></pre>

  <p>And comment certificate-authority-data: if your cluster has it.</p>

  <p>Picture of config file how it should look like: <br />
  <img src="/content/images/2022/05/DeepinScreenshot_select-area_20220513110022.png" alt="alt" /></p>

  <p>Login to the Kubernetes master or worker node and grab the Cluster IP address from /etc/hosts file. You can use kubectl tool as well to get that info, but let's use this way.</p>

  <p>Now it's time to setup SSH tunnel, this is working example:  </p>

  <pre><code>ssh -f username@IP-OF-SERVER -L 6443:CLUSTER-IP:6443 -N  
  </code></pre>

  <p>When executed it will return back. That's it! Time to test...</p>

  <p><img src="/content/images/2022/05/DeepinScreenshot_select-area_20220513110521.png" alt="alt" /></p>

  <p>You are now managing Kubernetes cluster from your local environment. Good luck!</p>

  <p>Let's now test to put everything under UI. I've selected Lens because I think this IDE Kubernetes editor is brilliant when you need to troubleshoot and analyzing cluster events or simply put multiple container logs and monitor them. Lens – the Kubernetes IDE — is fruit of a Mirantis-sponsored open source project. Available for Linux, Mac, and Windows, Lens gives you a powerful interface and toolkit for managing, visualizing, and interacting with multiple Kubernetes clusters, while remaining always in proper context.</p>

  <p>Visit <a href="https://k8slens.dev/">https://k8slens.dev/</a>, download your desired application, and continue.</p>

  <p>When you log in and browse Clusters you will see your's right there listed: <br />
  <img src="/content/images/2022/05/DeepinScreenshot_select-area_20220513110523.png" alt="alt" /></p>

  <p>Click on it and enjoy analyzing clusters and logging in to the nodes. You will still have the help of the terminal where kubectl runs and you can execute commands: <br />
  <img src="/content/images/2022/05/DeepinScreenshot_select-area_20220513111043.png" alt="alt" /></p>

  <p>Final toughts/conclusions... <br />
  <img src="/content/images/2022/05/its-a-kubernetes-cluster-i-know-this-1.jpg" alt="alt" /></p>

  <p>I hope this article was helpful. Using this method you can add multiple Kubernetes Clusters on your Desktop environment and manage it successfully. Good luck!</p>


uuid: 9fb9f7ab-843d-44d6-9950-a10091cd40af
title: Use CSSNano and Postcss to Minify CSS in Magento 2
slug: use-cssnano-and-postcss-to-minify-css-in-magento-2
image: /content/images/2022/06/11719785413_bea9ccd83a.jpg
status: published
language: en_US
markdown:
  Hello,

  Today I want to show how to use another tremendous external tool to Minify Theme's CSS files, skipping Magento 2 built-in (Core) functionality.

  Minification is taking some code and using various methods to reduce its size on disk. Unlike techniques such as gzip, which preserve the original semantics of the CSS file and are therefore lossless, minification is an inherently lossy process, where values can be replaced with smaller equivalents or selectors merged, for example.

  The final result of a minification step is that the resulting code will behave the same as the original file, but some parts will be altered to reduce the size as much as possible.

  Combining gzip compression with minification leads to the best reduction in file size, but don't just take our word for it. 

  Vendor suggests trying out css-size, a module especially created to measure CSS size before & after minification.
  https://npmjs.org/package/css-size

  cssnano tool is one such minifier, which is written in Node.js. It's a PostCSS plugin that you can add to your build process, to ensure that the resulting stylesheet is as small as possible for a production environment.

  To get started as per vendor https://cssnano.co/docs/getting-started/ page we will install cssnano and postcss CLI tools.

  ```
  npm install --save-dev postcss cssnano cssnano-cli postcss-cli 
  ```

  You can do it with -g for global purposes
  ```
  npm install --save-dev postcss cssnano cssnano-cli postcss-cli -g
  ```

  Now that tool is installed you can type **whereis** command to get the path of postcss tool

  ```
  whereis postcss
  ```

  Once you have done this, you will need to configure cssnano by creating a **postcss.config.js** file in the root of your project. This should contain cssnano as well as any other plugins that you might want for your project; as an example:

  ```
  module.exports = {
      plugins: [
          require('cssnano')({
              preset: 'default',
          }),
      ],
  };
  ```
  Read more about presets at https://cssnano.co/docs/presets there are tons of other options available but for Magento 2 specific I have tested with the "default" preset.

  Next, we will get the **THEME folder** path scripted into the bash variable
  ```
  THEME_FOLDER=('/workspace/magento2gitpod/pub/static/frontend/THEME/porto/en_US')
  ```
  Note: make sure you enter the correct path to your deployed theme folder in the pub/ directory

  Time to push command and replace current CSS files with the minified version of them using the same file names.

  ```
  find ${THEME_FOLDER[@]} \( -name '*.css' -not -name '*.min.css' -not -name '*.json' -not -name '*.js' \) -exec /usr/local/nvm/versions/node/v16.14.0/bin/postcss --replace \{} \;
  ```
  Note: You see, I've used the nvm (Node Version Manager) tool and a specific v16.14.0 version. Tool vendors claim that you can use anything after version 10 safely.

  Time to measure before/after :)

  ```
  npm install -g css-size
  ```

  Pick one random CSS file before running it, for example:
  ![alt](/content/images/2022/06/2022-06-19_12-19.png)

  Auspicious results :) but okay, some files give more significant differences some do not. With the Advanced preset, you can remove comments, which is good in the production environment to save more space and reduce CSS sizes.

  Here you can find detailed instructions on what CSSnano Advanced preset offers:
  https://cssnano.co/docs/advanced-transforms/
  https://cssnano.co/docs/what-are-optimisations/

  Since the tool is pure CLI you can integrate it within any deployment CI/CD process and run it always after pushing setup:static-content:deploy command from the native Magento tool.

  Happy optimizing!

html:
  <p>Hello,</p>

  <p>Today I want to show how to use another tremendous external tool to Minify Theme's CSS files, skipping Magento 2 built-in (Core) functionality.</p>

  <p>Minification is taking some code and using various methods to reduce its size on disk. Unlike techniques such as gzip, which preserve the original semantics of the CSS file and are therefore lossless, minification is an inherently lossy process, where values can be replaced with smaller equivalents or selectors merged, for example.</p>

  <p>The final result of a minification step is that the resulting code will behave the same as the original file, but some parts will be altered to reduce the size as much as possible.</p>

  <p>Combining gzip compression with minification leads to the best reduction in file size, but don't just take our word for it. </p>

  <p>Vendor suggests trying out css-size, a module especially created to measure CSS size before &amp; after minification. <br />
  <a href="https://npmjs.org/package/css-size">https://npmjs.org/package/css-size</a></p>

  <p>cssnano tool is one such minifier, which is written in Node.js. It's a PostCSS plugin that you can add to your build process, to ensure that the resulting stylesheet is as small as possible for a production environment.</p>

  <p>To get started as per vendor <a href="https://cssnano.co/docs/getting-started/">https://cssnano.co/docs/getting-started/</a> page we will install cssnano and postcss CLI tools.</p>

  <pre><code>npm install --save-dev postcss cssnano cssnano-cli postcss-cli  
  </code></pre>

  <p>You can do it with -g for global purposes  </p>

  <pre><code>npm install --save-dev postcss cssnano cssnano-cli postcss-cli -g  
  </code></pre>

  <p>Now that tool is installed you can type <strong>whereis</strong> command to get the path of postcss tool</p>

  <pre><code>whereis postcss  
  </code></pre>

  <p>Once you have done this, you will need to configure cssnano by creating a <strong>postcss.config.js</strong> file in the root of your project. This should contain cssnano as well as any other plugins that you might want for your project; as an example:</p>

  <pre><code>module.exports = {  
      plugins: [
          require('cssnano')({
              preset: 'default',
          }),
      ],
  };
  </code></pre>

  <p>Read more about presets at <a href="https://cssnano.co/docs/presets">https://cssnano.co/docs/presets</a> there are tons of other options available but for Magento 2 specific I have tested with the "default" preset.</p>

  <p>Next, we will get the <strong>THEME folder</strong> path scripted into the bash variable  </p>

  <pre><code>THEME_FOLDER=('/workspace/magento2gitpod/pub/static/frontend/THEME/porto/en_US')  
  </code></pre>

  <p>Note: make sure you enter the correct path to your deployed theme folder in the pub/ directory</p>

  <p>Time to push command and replace current CSS files with the minified version of them using the same file names.</p>

  <pre><code>find ${THEME_FOLDER[@]} \( -name '*.css' -not -name '*.min.css' -not -name '*.json' -not -name '*.js' \) -exec /usr/local/nvm/versions/node/v16.14.0/bin/postcss --replace \{} \;  
  </code></pre>

  <p>Note: You see, I've used the nvm (Node Version Manager) tool and a specific v16.14.0 version. Tool vendors claim that you can use anything after version 10 safely.</p>

  <p>Time to measure before/after :)</p>

  <pre><code>npm install -g css-size  
  </code></pre>

  <p>Pick one random CSS file before running it, for example: <br />
  <img src="/content/images/2022/06/2022-06-19_12-19.png" alt="alt" /></p>

  <p>Auspicious results :) but okay, some files give more significant differences some do not. With the Advanced preset, you can remove comments, which is good in the production environment to save more space and reduce CSS sizes.</p>

  <p>Here you can find detailed instructions on what CSSnano Advanced preset offers: <br />
  <a href="https://cssnano.co/docs/advanced-transforms/">https://cssnano.co/docs/advanced-transforms/</a> <br />
  <a href="https://cssnano.co/docs/what-are-optimisations/">https://cssnano.co/docs/what-are-optimisations/</a></p>

  <p>Since the tool is pure CLI you can integrate it within any deployment CI/CD process and run it always after pushing setup:static-content:deploy command from the native Magento tool.</p>

  <p>Happy optimizing!</p>


uuid: 17dc330e-3d47-48dd-9e77-a823b0ede154
title: Manage Remote Kubernetes clusters with VS Code IDE editor
slug: manage-remote-kubernetes-clusters-with-vs-code-ide-editor
image: /content/images/2022/07/capa-kubernetes-400x280.png
status: published
language: en_US
markdown:
  Hello,

  Today I want to show how you can use your local VS Code IDE editor to manage Remote Kubernetes Clusters and connect to running containers, browse local files/folders, and works with MySQL, Redis, ElasticSearch, Debug, and a lot more!

  I will skip the part where I wrote one of the possible methods how to connect your kubectl tool to the Remote Kubernetes cluster running in the Cloud or Bare metal servers, and that is already described briefly in the https://nemanja.io/manage-remote-kubernetes-clusters-with-kubectl-and-lens/ article, so please open it now if you need to set up that parts on your local computer.

  Please visit the following URL to set up the Remote Kubernetes cluster https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools address or Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.

  ```
  ext install ms-kubernetes-tools.vscode-kubernetes-tools
  ```

  Once the module is installed, you can click on the left sidebar on the icon and start exploring.

  ![alt](/content/images/2022/07/2022-07-30_11-36.png)

  You will notice that Cluster is kubernetes-admin@kubernetes. That is the one you set up using my tutorial with the SSH Tunnel mechanism. This extension has very big documentation of what you can actually do with it from your VS Code editor. Feel free to review https://github.com/vscode-kubernetes-tools/vscode-kubernetes-tools URL. I will cover just the basics of what you need, for example, to connect/attach to a container within a namespace, and one of the best features is "Attach Visual Studio Code" which remotely connects you to the specific container in a separate VS Code window, and you can work there same as you do locally and even install extensions on that container which can be used again next time.

  Press CTRL+SHIFT+P keys, then type **kubernetes**
  ![alt](/content/images/2022/07/2022-07-30_11-45.png)

  Select **Kubernetes: Use Namespace** option and start searching your namespace name, which I assume you already know. You can also explore all namespaces (slower if you have a lot of them) and then search from that area simply by typing keywords.

  Once the specific namespace is selected, we can explore everything that Kubernetes offers, like Deployments, Workloads, Secretes, and ConfigMaps. I like this because you can edit something within the YAML file, and with a single click on X, it will check, save, and Kubernetes is going to restart to the latest version. You can also schedule this change for any time that is suitable, close your browser and enjoy your day while the task is going to be scheduled in the Kubernetes.

  Deployments --> Workload --> Pods you can simply SSH into any available container within POD, and you will be asked if there are multiple within a single Kubernetes pod.

  ![alt](/content/images/2022/07/2022-07-30_11-54.png)

  From the same area right, click, you can do other things like grab logs, describe pod/container or watch it actively while troubleshooting start/stop into a separate window.

  Next super feature of this extension is that you can actually "Attach" VS Code IDE server directly into any Kubernetes Pod/Container and remotely connect to it, working from your computer the same way as you do locally. I find this super useful because of many aspects, but I will repeat here few:

  - Search is amazing, fast, and accurate
  - You can edit any file and parse them with many useful VS Code extensions like a parser, tabnine, or similar
  - Smart log extension https://marketplace.visualstudio.com/items?itemName=mbehr1.smart-log
  - MySQL/ElasticSearch/Redis browser/editor https://marketplace.visualstudio.com/items?itemName=cweijan.vscode-mysql-client2

  ![alt](/content/images/2022/07/2022-07-30_12-04.png)

  You can install anything useful that will help you develop, troubleshoot/debug or test just now with the aspect that you can work from the same place and operate with the Kubernetes cluster.

  I hope this article was helpful. Happy DevOpsing :)

html:
  <p>Hello,</p>

  <p>Today I want to show how you can use your local VS Code IDE editor to manage Remote Kubernetes Clusters and connect to running containers, browse local files/folders, and works with MySQL, Redis, ElasticSearch, Debug, and a lot more!</p>

  <p>I will skip the part where I wrote one of the possible methods how to connect your kubectl tool to the Remote Kubernetes cluster running in the Cloud or Bare metal servers, and that is already described briefly in the <a href="https://nemanja.io/manage-remote-kubernetes-clusters-with-kubectl-and-lens/">https://nemanja.io/manage-remote-kubernetes-clusters-with-kubectl-and-lens/</a> article, so please open it now if you need to set up that parts on your local computer.</p>

  <p>Please visit the following URL to set up the Remote Kubernetes cluster <a href="https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools">https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools</a> address or Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.</p>

  <pre><code>ext install ms-kubernetes-tools.vscode-kubernetes-tools  
  </code></pre>

  <p>Once the module is installed, you can click on the left sidebar on the icon and start exploring.</p>

  <p><img src="/content/images/2022/07/2022-07-30_11-36.png" alt="alt" /></p>

  <p>You will notice that Cluster is kubernetes-admin@kubernetes. That is the one you set up using my tutorial with the SSH Tunnel mechanism. This extension has very big documentation of what you can actually do with it from your VS Code editor. Feel free to review <a href="https://github.com/vscode-kubernetes-tools/vscode-kubernetes-tools">https://github.com/vscode-kubernetes-tools/vscode-kubernetes-tools</a> URL. I will cover just the basics of what you need, for example, to connect/attach to a container within a namespace, and one of the best features is "Attach Visual Studio Code" which remotely connects you to the specific container in a separate VS Code window, and you can work there same as you do locally and even install extensions on that container which can be used again next time.</p>

  <p>Press CTRL+SHIFT+P keys, then type <strong>kubernetes</strong> <br />
  <img src="/content/images/2022/07/2022-07-30_11-45.png" alt="alt" /></p>

  <p>Select <strong>Kubernetes: Use Namespace</strong> option and start searching your namespace name, which I assume you already know. You can also explore all namespaces (slower if you have a lot of them) and then search from that area simply by typing keywords.</p>

  <p>Once the specific namespace is selected, we can explore everything that Kubernetes offers, like Deployments, Workloads, Secretes, and ConfigMaps. I like this because you can edit something within the YAML file, and with a single click on X, it will check, save, and Kubernetes is going to restart to the latest version. You can also schedule this change for any time that is suitable, close your browser and enjoy your day while the task is going to be scheduled in the Kubernetes.</p>

  <p>Deployments --> Workload --> Pods you can simply SSH into any available container within POD, and you will be asked if there are multiple within a single Kubernetes pod.</p>

  <p><img src="/content/images/2022/07/2022-07-30_11-54.png" alt="alt" /></p>

  <p>From the same area right, click, you can do other things like grab logs, describe pod/container or watch it actively while troubleshooting start/stop into a separate window.</p>

  <p>Next super feature of this extension is that you can actually "Attach" VS Code IDE server directly into any Kubernetes Pod/Container and remotely connect to it, working from your computer the same way as you do locally. I find this super useful because of many aspects, but I will repeat here few:</p>

  <ul>
  <li>Search is amazing, fast, and accurate</li>
  <li>You can edit any file and parse them with many useful VS Code extensions like a parser, tabnine, or similar</li>
  <li>Smart log extension <a href="https://marketplace.visualstudio.com/items?itemName=mbehr1.smart-log">https://marketplace.visualstudio.com/items?itemName=mbehr1.smart-log</a></li>
  <li>MySQL/ElasticSearch/Redis browser/editor <a href="https://marketplace.visualstudio.com/items?itemName=cweijan.vscode-mysql-client2">https://marketplace.visualstudio.com/items?itemName=cweijan.vscode-mysql-client2</a></li>
  </ul>

  <p><img src="/content/images/2022/07/2022-07-30_12-04.png" alt="alt" /></p>

  <p>You can install anything useful that will help you develop, troubleshoot/debug or test just now with the aspect that you can work from the same place and operate with the Kubernetes cluster.</p>

  <p>I hope this article was helpful. Happy DevOpsing :)</p>


uuid: 79d06b11-651e-4319-9dee-9542bb7247da
title: Tools to analyze Redis databases
slug: tools-to-analyze-redis-databases
image: /content/images/2022/09/1_77Vo1RFQ-5DcLKdeHbb2-A.png
status: published
language: en_US
markdown:
  Hello,

  Today I would like to explain a simple trick of how you can search and analyze any data stored in Redis and then export it in JSON format to filter better. 

  You can get the dump if you export data on the production node/server.rdb file from Redis root folder (CONFIG GET DIR), or if Persistance mode is not enabled (something ephemeral), you can extract data using the https://github.com/r043v/rdd tool.

  RDB is a dump. You can call save to force an Rdb. It will be stored in the DB filename setting you have, or dump.rdb in the current working directory if that setting is missing.

  More Info: http://redis.io/topics/persistence

  Now that rdb dump is ready on your local computer or any other device/server, you can start by installing Redis-Commander.

  What is Redis Commander?
  Redis-Commander is a node.js web application used to view, edit, and manage a Redis Database.

  More Info: http://joeferner.github.io/redis-commander/

  Super simple to get going with NPM:
  ```
  npm install -g redis-commander
  ```
  Start it with:
  ```
  redis-commander
  ```
  Then point your browser to the local computer's address in the console. By default, access with browser at http://127.0.0.1:8081 address.

  If you are using any Remote IDE environment like Gitpod (https://gitpod.io), you can get the address exposed when Redis-commander service starts.

  Make sure you have Redis installed. I will not write about how to install the Redis server with cli tools but make sure it started on your local computer or remote device/server.

  To check if Redis Commander works, load UI, and you must see it connected to your Redis server.
  ![alt](/content/images/2022/09/2022-09-08_17-32.png)

  If you wish to start the server with the Redis dump you exported from your server, yes, you can do it stop this one and start the server again with the following command:
  ```
  /usr/bin/redis-server --port 6379 --dbfilename backup_of_master.rdb
  ```
  ** my redis-server is at /usr/bin path, but you can type `whereis redis-server` to see where it is installed.

  You can start analyzing data. There is also a helper who can help you with certain commands. One of the best functionalities here is that we can now export data in JSON format and analyze values much better using sublime or any other popular editor or even Online using https://jsoneditoronline.org/classic/index.html#left=local.sugeve website. Neat!

  2nd popular application to analyze data from Redis is https://resp.app/

  Good luck, happy troubleshooting!

html:
  <p>Hello,</p>

  <p>Today I would like to explain a simple trick of how you can search and analyze any data stored in Redis and then export it in JSON format to filter better. </p>

  <p>You can get the dump if you export data on the production node/server.rdb file from Redis root folder (CONFIG GET DIR), or if Persistance mode is not enabled (something ephemeral), you can extract data using the <a href="https://github.com/r043v/rdd">https://github.com/r043v/rdd</a> tool.</p>

  <p>RDB is a dump. You can call save to force an Rdb. It will be stored in the DB filename setting you have, or dump.rdb in the current working directory if that setting is missing.</p>

  <p>More Info: <a href="http://redis.io/topics/persistence">http://redis.io/topics/persistence</a></p>

  <p>Now that rdb dump is ready on your local computer or any other device/server, you can start by installing Redis-Commander.</p>

  <p>What is Redis Commander? <br />
  Redis-Commander is a node.js web application used to view, edit, and manage a Redis Database.</p>

  <p>More Info: <a href="http://joeferner.github.io/redis-commander/">http://joeferner.github.io/redis-commander/</a></p>

  <p>Super simple to get going with NPM:  </p>

  <pre><code>npm install -g redis-commander  
  </code></pre>

  <p>Start it with:  </p>

  <pre><code>redis-commander  
  </code></pre>

  <p>Then point your browser to the local computer's address in the console. By default, access with browser at <a href="http://127.0.0.1:8081">http://127.0.0.1:8081</a> address.</p>

  <p>If you are using any Remote IDE environment like Gitpod (<a href="https://gitpod.io">https://gitpod.io</a>), you can get the address exposed when Redis-commander service starts.</p>

  <p>Make sure you have Redis installed. I will not write about how to install the Redis server with cli tools but make sure it started on your local computer or remote device/server.</p>

  <p>To check if Redis Commander works, load UI, and you must see it connected to your Redis server. <br />
  <img src="/content/images/2022/09/2022-09-08_17-32.png" alt="alt" /></p>

  <p>If you wish to start the server with the Redis dump you exported from your server, yes, you can do it stop this one and start the server again with the following command:  </p>

  <pre><code>/usr/bin/redis-server --port 6379 --dbfilename backup_of_master.rdb
  </code></pre>

  <p>** my redis-server is at /usr/bin path, but you can type <code>whereis redis-server</code> to see where it is installed.</p>

  <p>You can start analyzing data. There is also a helper who can help you with certain commands. One of the best functionalities here is that we can now export data in JSON format and analyze values much better using sublime or any other popular editor or even Online using <a href="https://jsoneditoronline.org/classic/index.html#left=local.sugeve">https://jsoneditoronline.org/classic/index.html#left=local.sugeve</a> website. Neat!</p>

  <p>2nd popular application to analyze data from Redis is <a href="https://resp.app/">https://resp.app/</a></p>

  <p>Good luck, happy troubleshooting!</p>


uuid: 76e7f02f-38c3-445e-bfc2-e741bcb60d99
title: Magento 2 Debug Varnish cache in production mode
slug: magento-2-debug-varnish-cache-in-production-mode
image: /content/images/2022/09/1511065546varnish-hit-1024x416.gif
status: published
language: en_US
markdown:
  Hello,

  By default, Magento 2 offers to debug Varnish if pages are in MISS or HIT mode when Developer mode is enabled. You can debug the store in two ways in the Production mode.

  1) **varnishlog**
  varnishlog is a CLI tool written to read the output from the Varnish service in format order. You can print output on the terminal, parse it to other formats, stream outside of the container or service, or similar. For more details about the varnishlog tool, you can read the following https://varnish-cache.org/docs/trunk/reference/varnishlog.html address.

  To limit output only to your IP address, we can use the following CLI command:
  ```
  varnishlog -q "ReqHeader eq 'X-Real-IP: 178.223.40.243'"
  ```
  Note: Update your real IP address. You can use X-Real-IP or X-Forward-For depending on your Upstream and Webserver settings.

  The standard output has a lot of information, but mostly it starts with:
  ```
  << Request  >>
  ```
  Then it has a timestamp, ReqStart, ReqMethod (GET, POST...), ReqURL, and ReqProtocol, but the longest part is ReqHeader and RespHeader section. varnishlog is not the tool you want to collect logs to store. However, it’s the perfect choice to obtain debugging data thanks to the indecent amount of data it produces.

  The one I find important to debug the Varnish cache is:
  ```
  VCL_call
  ```

  It can end up as DELIVER (not cached) or HIT (cached). You can read Varnish documentation pages [Varnish Output](https://book.varnish-software.com/4.0/chapters/Examining_Varnish_Server_s_Output.html) on how many other options can be used. Typically followed by VCL_call Deliver is RespHeader Age: 0

  ![alt](/content/images/2022/09/2022-09-16_17-56.png)
  ![alt](/content/images/2022/09/2022-09-16_17-55.png)

  2) If you find this hard and not efficient enough (yeah, I know the browser/inspect header sounds better), we can do the following to debug and make it easier.

  - Under *sub vcl_deliver {* section comment few lines.

  ```
   //if (resp.http.X-Magento-Debug) {
          if (resp.http.x-varnish ~ " ") {
              set resp.http.X-Magento-Cache-Debug = "HIT";
          } else {
              set resp.http.X-Magento-Cache-Debug = "MISS";
          }
      //} else {
         // # unset resp.http.Age;
      //}
  ```
  Note: by default, it's without //

  Save the VCL file, test, and reload your Varnish cache. You will see there is a new RespHeader added on page load.

  ![alt](/content/images/2022/09/2022-09-16_21-46.png)

  What is **x-magento-cache-debug** you can read here:
  https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/cache/varnish/config-varnish-final.html

  That's it! Now you follow this simple trick, leave your store in the Production mode, and still debug your Varnish cache if pages are HIT or MISS.

  Good luck!

html:
  <p>Hello,</p>

  <p>By default, Magento 2 offers to debug Varnish if pages are in MISS or HIT mode when Developer mode is enabled. You can debug the store in two ways in the Production mode.</p>

  <p>1) <strong>varnishlog</strong> <br />
  varnishlog is a CLI tool written to read the output from the Varnish service in format order. You can print output on the terminal, parse it to other formats, stream outside of the container or service, or similar. For more details about the varnishlog tool, you can read the following <a href="https://varnish-cache.org/docs/trunk/reference/varnishlog.html">https://varnish-cache.org/docs/trunk/reference/varnishlog.html</a> address.</p>

  <p>To limit output only to your IP address, we can use the following CLI command:  </p>

  <pre><code>varnishlog -q "ReqHeader eq 'X-Real-IP: 178.223.40.243'"  
  </code></pre>

  <p>Note: Update your real IP address. You can use X-Real-IP or X-Forward-For depending on your Upstream and Webserver settings.</p>

  <p>The standard output has a lot of information, but mostly it starts with:  </p>

  <pre><code>&lt;&lt; Request  &gt;&gt;  
  </code></pre>

  <p>Then it has a timestamp, ReqStart, ReqMethod (GET, POST...), ReqURL, and ReqProtocol, but the longest part is ReqHeader and RespHeader section. varnishlog is not the tool you want to collect logs to store. However, it’s the perfect choice to obtain debugging data thanks to the indecent amount of data it produces.</p>

  <p>The one I find important to debug the Varnish cache is:  </p>

  <pre><code>VCL_call  
  </code></pre>

  <p>It can end up as DELIVER (not cached) or HIT (cached). You can read Varnish documentation pages <a href="https://book.varnish-software.com/4.0/chapters/Examining_Varnish_Server_s_Output.html">Varnish Output</a> on how many other options can be used. Typically followed by VCL_call Deliver is RespHeader Age: 0</p>

  <p><img src="/content/images/2022/09/2022-09-16_17-56.png" alt="alt" />
  <img src="/content/images/2022/09/2022-09-16_17-55.png" alt="alt" /></p>

  <p>2) If you find this hard and not efficient enough (yeah, I know the browser/inspect header sounds better), we can do the following to debug and make it easier.</p>

  <ul>
  <li>Under <em>sub vcl_deliver {</em> section comment few lines.</li>
  </ul>

  <pre><code> //if (resp.http.X-Magento-Debug) {
          if (resp.http.x-varnish ~ " ") {
              set resp.http.X-Magento-Cache-Debug = "HIT";
          } else {
              set resp.http.X-Magento-Cache-Debug = "MISS";
          }
      //} else {
         // # unset resp.http.Age;
      //}
  </code></pre>

  <p>Note: by default, it's without //</p>

  <p>Save the VCL file, test, and reload your Varnish cache. You will see there is a new RespHeader added on page load.</p>

  <p><img src="/content/images/2022/09/2022-09-16_21-46.png" alt="alt" /></p>

  <p>What is <strong>x-magento-cache-debug</strong> you can read here: <br />
  <a href="https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/cache/varnish/config-varnish-final.html">https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/cache/varnish/config-varnish-final.html</a></p>

  <p>That's it! Now you follow this simple trick, leave your store in the Production mode, and still debug your Varnish cache if pages are HIT or MISS.</p>

  <p>Good luck!</p>


uuid: 64b06404-c96d-428f-b983-453f69f084c5
title: Invalid host header PWA Studio
slug: invalid-host-header-pwa-studio
image: /content/images/2022/10/pwa.jpeg
status: published
language: en_US
markdown:
  Hello,

  Recently, I discovered the issue when trying to start the PWA Studio development environment.

  **Invalid host header**

  command used: yarn watch

  ![alt](/content/images/2022/10/2022-10-25_13-17.png)

  To fix this, you can add the following to the node_modules/@magento/pwa-buildpack/lib/WebpackTools/PWADevServer.js


  ```
  disableHostCheck: true,
  ```

  ![alt](/content/images/2022/10/2022-10-25_13-19.png)

  Save the file, start yarn watch again and enjoy development that can refresh content without compiling.

  I hope this helps.

html:
  <p>Hello,</p>

  <p>Recently, I discovered the issue when trying to start the PWA Studio development environment.</p>

  <p><strong>Invalid host header</strong></p>

  <p>command used: yarn watch</p>

  <p><img src="/content/images/2022/10/2022-10-25_13-17.png" alt="alt" /></p>

  <p>To fix this, you can add the following to the node_modules/@magento/pwa-buildpack/lib/WebpackTools/PWADevServer.js</p>

  <pre><code>disableHostCheck: true,  
  </code></pre>

  <p><img src="/content/images/2022/10/2022-10-25_13-19.png" alt="alt" /></p>

  <p>Save the file, start yarn watch again and enjoy development that can refresh content without compiling.</p>

  <p>I hope this helps.</p>


uuid: 22fdfb65-ce40-4bcc-ac56-901ab577be00
title: Split Read and Writes in Magento 2 using AWS RDS and ProxySQL
slug: split-read-and-writes-using-aws-rds-and-proxy
image: /content/images/2023/03/master-slave-replication-1.jpg
status: published
language: en_US
markdown:
  Hello,

  I have been exploring options lately to scale MySQL and tested this solution. In my tests, I am using small db.t2.medium AWS RDS cluster with Write and Read replica. 

  For test purposes following components were used:
  - AWS account setup of RDS with public access with IP-only limitation. <BR>
  - Magento 2 gitpod instance (https://github.com/nemke82/magento2gitpod) started. <BR>
  - Modified m2-install.sh file to connect to Remote AWS RDS environment and install fresh Magento 2 install with Performance fixtures data (small). <BR>
  - ProxySQL installed on the Gitpod environment and configured

  Before you start, you have to create a test database. In my example:
  ```
  mysql -u nemanja -h nemanja-instance-1.cusgvuriflp3.us-east-1.rds.amazonaws.com -pRandomPassword123 -e 'create database nemanja;'
  ```

  Then we can add privileges to that database:
  ```
  GRANT ALL PRIVILEGES ON nemanja.* TO 'nemanja'@'%';
  FLUSH PRIVILEGES;
  exit;
  ```

  Modify m2-install.sh (if you are testing on the Magento 2 Gitpod platform) file with the new MySQL hostname endpoint, username, and password and execute the installation.

  Once the installation of fresh Magento 2 is ready and data is in your AWS RDS instance, we will proceed with ProxySQL configuration.

  To set up ProxySQL to split SELECT queries between an AWS RDS master instance and its read replicas, follow these steps:

  **Install ProxySQL** <BR>
  Install ProxySQL on an EC2 instance or a server within your VPC, so it has access to your RDS instances.

  URL: https://proxysql.com/documentation/installing-proxysql/

  **Configure ProxySQL**<BR>
  Edit the ProxySQL configuration file, usually located at /etc/proxysql.cnf, and configure it according to your needs. An example configuration is shown below:

  ```
  datadir="/var/lib/proxysql"
  admin_variables=
  {
    admin_credentials="admin:admin"
    mysql_ifaces="0.0.0.0:6032"
  }
  mysql_variables=
  {
    threads=4
    max_connections=2048
    default_query_delay=0
    default_query_timeout=36000000
    have_compress=true
    poll_timeout=2000
    interfaces="0.0.0.0:6033"
    default_schema="information_schema"
    stacksize=1048576
    server_version="5.7.27"
    connect_timeout_server=3000
    monitor_history=600000
    monitor_connect_interval=60000
    monitor_ping_interval=10000
    ping_interval_server_msec=10000
    ping_timeout_server=200
    commands_stats=true
    sessions_sort=true
  }
  ```

  **Start the ProxySQL service:**
  ```
  sudo systemctl start proxysql
  ```

  Next task is to Configure RDS instances in ProxySQL.

  Let's access the ProxySQL admin interface:
  ```
  mysql -u admin -p -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin> '
  ```
  Default password is "admin" for Fresh ProxySQL installation.


  **Add your RDS master and read replica instances:** <BR>
  ```
  -- Add master instance
  INSERT INTO mysql_servers(hostgroup_id, hostname, port)
  VALUES (0, 'your_master_instance_endpoint', 3306);

  -- Add read replica instance
  INSERT INTO mysql_servers(hostgroup_id, hostname, port)
  VALUES (1, 'your_read_replica_instance_endpoint', 3306);

  LOAD MYSQL SERVERS TO RUNTIME;
  SAVE MYSQL SERVERS TO DISK;
  ```
  Here we start with one master and 1 read replica endpoint. We can scale later and add more easily.

  **Set up query rules** <BR>

  The next step is to create rules to route SELECT queries to the read replica:

  ```
  INSERT INTO mysql_query_rules (rule_id, active, match_pattern, destination_hostgroup, apply)
  VALUES (1, 1, '^SELECT.*FOR UPDATE$', 0, 1),
         (2, 1, '^SELECT', 1, 1);             

  LOAD MYSQL QUERY RULES TO RUNTIME;
  SAVE MYSQL QUERY RULES TO DISK;
  ```
  -- Send 'SELECT ... FOR UPDATE' to the master <BR>
  -- Send other SELECTs to the read replica <BR>
  You can extend this with anything you think could benefit route SELECT queries to the read replica.

  **Configure user credentials**<BR>

  Add the database user credentials to ProxySQL:
  ```
  INSERT INTO mysql_users(username, password, default_hostgroup)
  VALUES ('your_username', 'your_password', 0);

  LOAD MYSQL USERS TO RUNTIME;
  SAVE MYSQL USERS TO DISK;
  ```
  We will use the same username and password we used to define when creating the AWS RDS cluster. I advise using anything except the "admin" username, as that is the default username for ProxySQL, which we could change later, but is not crucial in the current proof-of-concept demo.

  **Monitor and adjust**<BR>
  Monitor ProxySQL's performance and adjust settings as needed. You can add more read replicas and change the load balancing settings for better performance.

  Start by adjusting the env.php file. In the 'host' field, enter:
  ```
  127.0.0.1:6033
  ```
  We could experiment here by adding Socket instead of a loopback which I think should optimize this area better.

  Next, adjust the 'username' and 'password' fields with the one you created above.

  Now, your application should connect to ProxySQL on port 6033 instead of directly connecting to the RDS instances. ProxySQL will route SELECT queries to the read replica and other write queries to the master instance.

  **-- Horizontally scaling Read replicas --** <BR>
  To add more read replicas and adjust the load balancing settings for better performance, follow these steps:

  **Add more read replicas in AWS RDS** <BR>
  First, create additional read replicas of your master RDS instance from the AWS Management Console or the AWS CLI.

  **Add read replicas to ProxySQL**<BR>
  Connect to the ProxySQL admin interface:
  ```
  mysql -u admin -p -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin> '
  ```

  Add the new read replica instances to ProxySQL, assigning them to the same hostgroup (1 in this example):
  ```
  INSERT INTO mysql_servers(hostgroup_id, hostname, port)
  VALUES (1, 'your_additional_read_replica_instance_endpoint_1', 3306),
         (1, 'your_additional_read_replica_instance_endpoint_2', 3306),
         ...;

  LOAD MYSQL SERVERS TO RUNTIME;
  SAVE MYSQL SERVERS TO DISK;
  ```
  -- Add additional read replica instances

  **Configure load balancing settings**<BR>
  By default, ProxySQL uses the round-robin algorithm to balance the load among the read replicas. You can adjust the weight of each server to control the traffic distribution. A higher weight means more traffic will be sent to that server.

  To modify the weight of a read replica, update the weight column in the mysql_servers table:

  ```
  UPDATE mysql_servers SET weight = new_weight
  WHERE hostgroup_id = 1 AND hostname = 'your_read_replica_instance_endpoint';

  LOAD MYSQL SERVERS TO RUNTIME;
  SAVE MYSQL SERVERS TO DISK;
  ```
  Here we can say for read_replica_instance_endpoint. We will put 50, then next one, we can put 50 if two read replicas are added. Overall that makes them split traffic even. Monitor the performance of your read replicas and adjust their weights accordingly. You can use the ProxySQL stats schema to gather information about traffic distribution and query performance.

  ![alt](/content/images/2023/03/2023-03-26_20-01.png)

  For example, to view the traffic distribution among the read replicas, you can run:
  ```
  select * from stats_mysql_connection_pool;
  ```

  You can see several queries sent and other useful data:
  ![alt](/content/images/2023/03/2023-03-26_20-03.png)

  Adjust the read replica weights based on the performance metrics and your application's requirements. Continue to monitor and fine-tune the load balancing settings to achieve optimal performance.

  Remember that adding more read replicas may increase costs but can help distribute read traffic and improve performance for read-heavy workloads. Ensure you monitor your instances' resource usage to balance performance and price. All the above can be partially or fully automated with a systemd or cron task checking the state of the AWS RDS server and then scaling up replicas based on selected patterns.

  We can also enable ProxySQL UI and watch stats in the browser.
  Ref article: https://proxysql.com/documentation/http-web-server/

  Connect to the ProxySQL admin interface:
  ```
  mysql -u admin -p -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin> '
  ```

  Then:
  ```
  SET admin-web_enabled='true';
  LOAD ADMIN VARIABLES TO RUNTIME;
  ```

  I usually use SSH tunnel to connect and get to the UI, for example:
  ```
  ssh -p <SSHport> -L 6080:127.0.0.1:6080 <SSHuser>@<SSHhost>
  ```
  Then you can open up your favorite browser and visit https://127.0.0.1:6080
  username and password by default are admin:admin

  Where are problems here? <BR>
  A typical issue with Indexer is the following:
  ```
  Product Price index process error during indexation process:
  SQLSTATE[Y0000]: <<Unknown error>>: 9006 ProxySQL Error: connection is locked to hostgroup 0 but trying to reach hostgroup 1, query was: SELECT `i`.`entity_id`, `o`.`option_id` FROM `catalog_product_index_price_temp` AS `i`
   INNER JOIN `catalog_product_entity` AS `e` ON e.entity_id = i.entity_id
   INNER JOIN `catalog_product_option` AS `o` ON o.product_id = e.entity_id
  Catalog Search index process error during indexation process:
  SQLSTATE[Y0000]: <<Unknown error>>: 9006 ProxySQL Error: connection is locked to hostgroup 0 but trying to reach hostgroup 1, query was: SELECT `indexer_state`.* FROM `indexer_state` WHERE (`indexer_state`.`indexer_id`='catalogsearch_fulltext')
  ```

  **To fix:** <BR>
  As is explained in the documentation <BR>
  https://proxysql.com/documentation/global-variables/mysql-variables/#mysql-set_query_lock_on_hostgroup

  So to resolve this issue on the single primary server on Group replication, try to do this.

  Log into ProxySql admin monitor.
  ```
  set mysql-set_query_lock_on_hostgroup=0;
  load mysql variables to runtime;
  save mysql variables to disk;
  ```

  Not mandatory, but the ProxySql service can be restarted to check if variables are saved on disk. Every setting here was done with "hot-reload". The advantage of ProxySQL service is that we can use proxysql.cnf file and add them as default instead.

  Stress testing this just phase one. I will update you with additional how time goes on. Using the tool **mysqlslap** <BR>(https://dev.mysql.com/doc/refman/8.0/en/mysqlslap.html), I've tried to stress/load test this with 100 concurrent users sending some generic SELECT query to see how READS are spread across the board. It was to select all Quote Items:
  ```
  gitpod /workspace/magento2gitpod $ mysqlslap -unemanja -P6033 -p -h127.0.0.1  --concurrency=100 --iterations=20 --create-schema=nemanja --query="SELECT * from quote_item" --verbose
  Enter password: 
  ```
  **Benchmark** <BR>
          The average number of seconds to run all queries: 0.369 seconds<BR>
          Minimum number of seconds to run all queries: 0.218 seconds <BR>
          Maximum number of seconds to run all queries: 1.144 seconds <BR>
          Number of clients running queries: 100 <BR>
          The average number of queries per client: 1

  Next, Benchmark using the same query was a lot faster since the query entered Query Cache, which ProxySQL provides by default even if AWS RDS is 8.0.26 used.

  **Benchmark** <BR>
          The average number of seconds to run all queries: 0.292 seconds <BR>
          Minimum number of seconds to run all queries: 0.184 seconds <BR>
          Maximum number of seconds to run all queries: 0.494 seconds <BR>
          Number of clients running queries: 100 <BR>
          The average number of queries per client: 1

  You can load separate terminals or screens (monitor) and watch how SELECT queries from the same Load/Stress test are spread across two Read replicas that I've created in this Concept Proof demo:
  ```
  watch -n 1 "mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e 'select * from stats_mysql_connection_pool;'"
  ```

  I hope this article was helpful. Good luck!

html:
  <p>Hello,</p>

  <p>I have been exploring options lately to scale MySQL and tested this solution. In my tests, I am using small db.t2.medium AWS RDS cluster with Write and Read replica. </p>

  <p>For test purposes following components were used: <br />
  - AWS account setup of RDS with public access with IP-only limitation. <BR>
  - Magento 2 gitpod instance (<a href="https://github.com/nemke82/magento2gitpod">https://github.com/nemke82/magento2gitpod</a>) started. <BR>
  - Modified m2-install.sh file to connect to Remote AWS RDS environment and install fresh Magento 2 install with Performance fixtures data (small). <BR>
  - ProxySQL installed on the Gitpod environment and configured</p>

  <p>Before you start, you have to create a test database. In my example:  </p>

  <pre><code>mysql -u nemanja -h nemanja-instance-1.cusgvuriflp3.us-east-1.rds.amazonaws.com -pRandomPassword123 -e 'create database nemanja;'  
  </code></pre>

  <p>Then we can add privileges to that database:  </p>

  <pre><code>GRANT ALL PRIVILEGES ON nemanja.* TO 'nemanja'@'%';  
  FLUSH PRIVILEGES;  
  exit;  
  </code></pre>

  <p>Modify m2-install.sh (if you are testing on the Magento 2 Gitpod platform) file with the new MySQL hostname endpoint, username, and password and execute the installation.</p>

  <p>Once the installation of fresh Magento 2 is ready and data is in your AWS RDS instance, we will proceed with ProxySQL configuration.</p>

  <p>To set up ProxySQL to split SELECT queries between an AWS RDS master instance and its read replicas, follow these steps:</p>

  <p><strong>Install ProxySQL</strong> <BR>
  Install ProxySQL on an EC2 instance or a server within your VPC, so it has access to your RDS instances.</p>

  <p>URL: <a href="https://proxysql.com/documentation/installing-proxysql/">https://proxysql.com/documentation/installing-proxysql/</a></p>

  <p><strong>Configure ProxySQL</strong><BR>
  Edit the ProxySQL configuration file, usually located at /etc/proxysql.cnf, and configure it according to your needs. An example configuration is shown below:</p>

  <pre><code>datadir="/var/lib/proxysql"  
  admin_variables=  
  {
    admin_credentials="admin:admin"
    mysql_ifaces="0.0.0.0:6032"
  }
  mysql_variables=  
  {
    threads=4
    max_connections=2048
    default_query_delay=0
    default_query_timeout=36000000
    have_compress=true
    poll_timeout=2000
    interfaces="0.0.0.0:6033"
    default_schema="information_schema"
    stacksize=1048576
    server_version="5.7.27"
    connect_timeout_server=3000
    monitor_history=600000
    monitor_connect_interval=60000
    monitor_ping_interval=10000
    ping_interval_server_msec=10000
    ping_timeout_server=200
    commands_stats=true
    sessions_sort=true
  }
  </code></pre>

  <p><strong>Start the ProxySQL service:</strong></p>

  <pre><code>sudo systemctl start proxysql  
  </code></pre>

  <p>Next task is to Configure RDS instances in ProxySQL.</p>

  <p>Let's access the ProxySQL admin interface:  </p>

  <pre><code>mysql -u admin -p -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin&gt; '  
  </code></pre>

  <p>Default password is "admin" for Fresh ProxySQL installation.</p>

  <p><strong>Add your RDS master and read replica instances:</strong> <BR></p>

  <pre><code>-- Add master instance
  INSERT INTO mysql_servers(hostgroup_id, hostname, port)  
  VALUES (0, 'your_master_instance_endpoint', 3306);

  -- Add read replica instance
  INSERT INTO mysql_servers(hostgroup_id, hostname, port)  
  VALUES (1, 'your_read_replica_instance_endpoint', 3306);

  LOAD MYSQL SERVERS TO RUNTIME;  
  SAVE MYSQL SERVERS TO DISK;  
  </code></pre>

  <p>Here we start with one master and 1 read replica endpoint. We can scale later and add more easily.</p>

  <p><strong>Set up query rules</strong> <BR></p>

  <p>The next step is to create rules to route SELECT queries to the read replica:</p>

  <pre><code>INSERT INTO mysql_query_rules (rule_id, active, match_pattern, destination_hostgroup, apply)  
  VALUES (1, 1, '^SELECT.*FOR UPDATE$', 0, 1),  
         (2, 1, '^SELECT', 1, 1);             

  LOAD MYSQL QUERY RULES TO RUNTIME;  
  SAVE MYSQL QUERY RULES TO DISK;  
  </code></pre>

  <p>-- Send 'SELECT ... FOR UPDATE' to the master <BR>
  -- Send other SELECTs to the read replica <BR>
  You can extend this with anything you think could benefit route SELECT queries to the read replica.</p>

  <p><strong>Configure user credentials</strong><BR></p>

  <p>Add the database user credentials to ProxySQL:  </p>

  <pre><code>INSERT INTO mysql_users(username, password, default_hostgroup)  
  VALUES ('your_username', 'your_password', 0);

  LOAD MYSQL USERS TO RUNTIME;  
  SAVE MYSQL USERS TO DISK;  
  </code></pre>

  <p>We will use the same username and password we used to define when creating the AWS RDS cluster. I advise using anything except the "admin" username, as that is the default username for ProxySQL, which we could change later, but is not crucial in the current proof-of-concept demo.</p>

  <p><strong>Monitor and adjust</strong><BR>
  Monitor ProxySQL's performance and adjust settings as needed. You can add more read replicas and change the load balancing settings for better performance.</p>

  <p>Start by adjusting the env.php file. In the 'host' field, enter:  </p>

  <pre><code>127.0.0.1:6033  
  </code></pre>

  <p>We could experiment here by adding Socket instead of a loopback which I think should optimize this area better.</p>

  <p>Next, adjust the 'username' and 'password' fields with the one you created above.</p>

  <p>Now, your application should connect to ProxySQL on port 6033 instead of directly connecting to the RDS instances. ProxySQL will route SELECT queries to the read replica and other write queries to the master instance.</p>

  <p><strong>-- Horizontally scaling Read replicas --</strong> <BR>
  To add more read replicas and adjust the load balancing settings for better performance, follow these steps:</p>

  <p><strong>Add more read replicas in AWS RDS</strong> <BR>
  First, create additional read replicas of your master RDS instance from the AWS Management Console or the AWS CLI.</p>

  <p><strong>Add read replicas to ProxySQL</strong><BR>
  Connect to the ProxySQL admin interface:  </p>

  <pre><code>mysql -u admin -p -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin&gt; '  
  </code></pre>

  <p>Add the new read replica instances to ProxySQL, assigning them to the same hostgroup (1 in this example):  </p>

  <pre><code>INSERT INTO mysql_servers(hostgroup_id, hostname, port)  
  VALUES (1, 'your_additional_read_replica_instance_endpoint_1', 3306),  
         (1, 'your_additional_read_replica_instance_endpoint_2', 3306),
         ...;

  LOAD MYSQL SERVERS TO RUNTIME;  
  SAVE MYSQL SERVERS TO DISK;  
  </code></pre>

  <p>-- Add additional read replica instances</p>

  <p><strong>Configure load balancing settings</strong><BR>
  By default, ProxySQL uses the round-robin algorithm to balance the load among the read replicas. You can adjust the weight of each server to control the traffic distribution. A higher weight means more traffic will be sent to that server.</p>

  <p>To modify the weight of a read replica, update the weight column in the mysql_servers table:</p>

  <pre><code>UPDATE mysql_servers SET weight = new_weight  
  WHERE hostgroup_id = 1 AND hostname = 'your_read_replica_instance_endpoint';

  LOAD MYSQL SERVERS TO RUNTIME;  
  SAVE MYSQL SERVERS TO DISK;  
  </code></pre>

  <p>Here we can say for read<em>replica</em>instance_endpoint. We will put 50, then next one, we can put 50 if two read replicas are added. Overall that makes them split traffic even. Monitor the performance of your read replicas and adjust their weights accordingly. You can use the ProxySQL stats schema to gather information about traffic distribution and query performance.</p>

  <p><img src="/content/images/2023/03/2023-03-26_20-01.png" alt="alt" /></p>

  <p>For example, to view the traffic distribution among the read replicas, you can run:  </p>

  <pre><code>select * from stats_mysql_connection_pool;  
  </code></pre>

  <p>You can see several queries sent and other useful data: <br />
  <img src="/content/images/2023/03/2023-03-26_20-03.png" alt="alt" /></p>

  <p>Adjust the read replica weights based on the performance metrics and your application's requirements. Continue to monitor and fine-tune the load balancing settings to achieve optimal performance.</p>

  <p>Remember that adding more read replicas may increase costs but can help distribute read traffic and improve performance for read-heavy workloads. Ensure you monitor your instances' resource usage to balance performance and price. All the above can be partially or fully automated with a systemd or cron task checking the state of the AWS RDS server and then scaling up replicas based on selected patterns.</p>

  <p>We can also enable ProxySQL UI and watch stats in the browser. <br />
  Ref article: <a href="https://proxysql.com/documentation/http-web-server/">https://proxysql.com/documentation/http-web-server/</a></p>

  <p>Connect to the ProxySQL admin interface:  </p>

  <pre><code>mysql -u admin -p -h 127.0.0.1 -P 6032 --prompt='ProxySQLAdmin&gt; '  
  </code></pre>

  <p>Then:  </p>

  <pre><code>SET admin-web_enabled='true';  
  LOAD ADMIN VARIABLES TO RUNTIME;  
  </code></pre>

  <p>I usually use SSH tunnel to connect and get to the UI, for example:  </p>

  <pre><code>ssh -p &lt;SSHport&gt; -L 6080:127.0.0.1:6080 &lt;SSHuser&gt;@&lt;SSHhost&gt;  
  </code></pre>

  <p>Then you can open up your favorite browser and visit <a href="https://127.0.0.1:6080">https://127.0.0.1:6080</a> <br />
  username and password by default are admin:admin</p>

  <p>Where are problems here? <BR> <br />
  A typical issue with Indexer is the following:  </p>

  <pre><code>Product Price index process error during indexation process:  
  SQLSTATE[Y0000]: &lt;&lt;Unknown error&gt;&gt;: 9006 ProxySQL Error: connection is locked to hostgroup 0 but trying to reach hostgroup 1, query was: SELECT `i`.`entity_id`, `o`.`option_id` FROM `catalog_product_index_price_temp` AS `i`  
   INNER JOIN `catalog_product_entity` AS `e` ON e.entity_id = i.entity_id
   INNER JOIN `catalog_product_option` AS `o` ON o.product_id = e.entity_id
  Catalog Search index process error during indexation process:  
  SQLSTATE[Y0000]: &lt;&lt;Unknown error&gt;&gt;: 9006 ProxySQL Error: connection is locked to hostgroup 0 but trying to reach hostgroup 1, query was: SELECT `indexer_state`.* FROM `indexer_state` WHERE (`indexer_state`.`indexer_id`='catalogsearch_fulltext')  
  </code></pre>

  <p><strong>To fix:</strong> <BR>
  As is explained in the documentation <BR> <br />
  <a href="https://proxysql.com/documentation/global-variables/mysql-variables/#mysql-set">https://proxysql.com/documentation/global-variables/mysql-variables/#mysql-set</a><em>query</em>lock<em>on</em>hostgroup</p>

  <p>So to resolve this issue on the single primary server on Group replication, try to do this.</p>

  <p>Log into ProxySql admin monitor.  </p>

  <pre><code>set mysql-set_query_lock_on_hostgroup=0;  
  load mysql variables to runtime;  
  save mysql variables to disk;  
  </code></pre>

  <p>Not mandatory, but the ProxySql service can be restarted to check if variables are saved on disk. Every setting here was done with "hot-reload". The advantage of ProxySQL service is that we can use proxysql.cnf file and add them as default instead.</p>

  <p>Stress testing this just phase one. I will update you with additional how time goes on. Using the tool <strong>mysqlslap</strong> <BR>(<a href="https://dev.mysql.com/doc/refman/8.0/en/mysqlslap.html">https://dev.mysql.com/doc/refman/8.0/en/mysqlslap.html</a>), I've tried to stress/load test this with 100 concurrent users sending some generic SELECT query to see how READS are spread across the board. It was to select all Quote Items:  </p>

  <pre><code>gitpod /workspace/magento2gitpod $ mysqlslap -unemanja -P6033 -p -h127.0.0.1  --concurrency=100 --iterations=20 --create-schema=nemanja --query="SELECT * from quote_item" --verbose  
  Enter password:  
  </code></pre>

  <p><strong>Benchmark</strong> <BR>
          The average number of seconds to run all queries: 0.369 seconds<BR>
          Minimum number of seconds to run all queries: 0.218 seconds <BR>
          Maximum number of seconds to run all queries: 1.144 seconds <BR>
          Number of clients running queries: 100 <BR>
          The average number of queries per client: 1</p>

  <p>Next, Benchmark using the same query was a lot faster since the query entered Query Cache, which ProxySQL provides by default even if AWS RDS is 8.0.26 used.</p>

  <p><strong>Benchmark</strong> <BR>
          The average number of seconds to run all queries: 0.292 seconds <BR>
          Minimum number of seconds to run all queries: 0.184 seconds <BR>
          Maximum number of seconds to run all queries: 0.494 seconds <BR>
          Number of clients running queries: 100 <BR>
          The average number of queries per client: 1</p>

  <p>You can load separate terminals or screens (monitor) and watch how SELECT queries from the same Load/Stress test are spread across two Read replicas that I've created in this Concept Proof demo:  </p>

  <pre><code>watch -n 1 "mysql -u admin -padmin -h 127.0.0.1 -P 6032 -e 'select * from stats_mysql_connection_pool;'"  
  </code></pre>

  <p>I hope this article was helpful. Good luck!</p>


# settings

uuid: d81c2c90-abd8-474a-aff4-1562e5ac0322
key: navigation
type: blog
value:
  [{"label":"Home","url":"/"},{"label":"About Me","url":"/about-me/"},{"label":"My Resume and LinkedIn","url":"/resume/"},{"label":"Command line help","url":"/command-line-help/"},{"label":"Tools","url":"/tools/"}]


