The solving problems series is just a way to share small ideas, experiences and outcomes of solving daily problems as I go. I wonder if some tips or experiences shared can help you build better what you are working on right now.

During the last months, I have been migrating an important PHP service to ECS Fargate along with the runtime upgrade. The service is composed of a lot of parts and we have been architecting the migration so the operation causes no downtime to customers, even when they are over four different continents and many time zones.

One very important part of the service is already running in production for some months with success. We are preparing the next service.

For the migration plan, we deployed infrastructure ahead of starting moving traffic, planned to daily incremental traffic switch, like 5, 10, 25, 50, 75, and close monitoring. Also prepared a second plan to avoid rollback in case some performance issue arises. While monitoring we created backlog tickets with the observability outcomes.

During migration phases prepare yourself beforehand for the initial (1%, or 5%) traffic switch, so you can catch quickly those hidden use cases that only happen in production and act quickly. If you do so, other phases are just a matter of watching how scaling works.

Using containers (of course Kubernetes is a great alternative) is a fantastic opportunity to upgrade PHP runtimes efficiently at the same time where we use a much better platform that helps with delivery and developer experiences. The very first and most important step I recommend is to review how you deal with your secret and environment variables. This is pivotal for the success of a smooth migration.

We can expect that those type of applications has a fair amount of cron jobs associated with them. This is a great opportunity to follow the old saying "use the right tool for the right problem" and my suggestion would be to rewrite it, turning it into Lambda or Step Functions, as applicable to each of what the cron job is doing. This is closer to what and how a job should run.

It happens that not always we can start refactoring right away, and then I can say that my experiences with Event Bridge Scheduler triggering ECS tasks (previously cron jobs) are great. They are interestingly cheap alternatives while waiting for the refactoring project to take over. Don't take this as your permanent solution though, because it is not just right and a waste of resources and couple the cron job too much with parts of the application not really related.

We were reviewing the backlog and observability results of the last service. As we could prioritise and execute some backlog tickets, the dashboard and metrics highlighted that we had some room to review scaling and resource thresholds. We changed them carefully, resulting in a bill ~50% cheaper, CPU and memory resource stable and no performance degradation.

Some notes:

• Investing in test automation is good for your developer experience, site reliability and revenue; also a great support for technology improvements
• It is worth taking a look at the ALBRequestCountPerTarget metric if you have CPU-heavy processes as you can better control how ECS will handle scale policies, avoiding peak of CPU where the CPU average metric is not enough for scaling

Links:

• https://dev.to/lpossamai/smooth-sailing-from-aws-ec2-to-ecs-a-comprehensive-migration-guide-2dci
• https://www.php.net/
• https://docs.aws.amazon.com/eventbridge/latest/userguide/using-eventbridge-scheduler.html
• https://docs.aws.amazon.com/AmazonECS/latest/userguide/what-is-fargate.html
• https://aws.amazon.com/ecs/
• https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-scaling-target-tracking.html
• https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_PredefinedMetricSpecification.html
• https://aws.amazon.com/lambda/

This is why, when I’m called for a performance problem (or for an audit), my first take is to look at the size of the data compared to the size of the indexes. If you store more indexes than data for a transactional workload, that’s bad. The worst I’ve seen was a database with 12 times more indexes stored on disk than data! Of course, it was a transactional workload… Would you buy a cooking book with 10 pages of recipes and 120 pages of indexes at the end of the book?

The problem with indexes is that each time you write (insert, update, delete), you will have to write to the indexes too! That can become very costly in resources and time.

Functional Classes

A place for everything, and everything in its place.

What is a class? According to the dictionary a class is:

A set, collection, group, or configuration containing members regarded as having certain attributes or traits in common; a kind or category.

The Simple Class

I work in many legacy code bases, and in fact, I’ve made it a big part of my career. I love diving into big monoliths that have grown out of proportion and tidying them up. One of the best parts of that work is rewriting a God class into a collection of small reusable classes. Let’s take a look at what makes a simple class great.

The economics of clean code

Code smarter. Code balanced. That is OK to have some debt. But pay them off quickly.

PHP é à quinta-feira – 50 dicas sobre desempenho e segurança | Peopleware

Howto access ext3 partition from Windows | Ubuntu Geek

This tutorial will allow you to access your ext3 partition under Windows, using Sun VirtualBox and Ubuntu. The tutorial is pretty long due to the images, but they explain things easier sometimes (they are not just meaningless screenshots) (forgive me for your scroll button )

Upload Pie - The Simple Image Sharing Tool

Sharing files with expiration dates

Mantis Bug Tracker

Gerenciamento de erros

TIOBE Software: Tiobe Index

Index of languages being used around the world.

Busca de CEP em PHP, Ajax, PHP, ASP, Java, Python, Flash, XML, C#, Ruby

Color Fading Menu with jQuery | CSS-Tricks

Increase your internet speed with Namebench | Ubuntu Geek

Guia de referência de comando em PT-BR | Ubuntu Dicas

Uma lista com os comandos mais usados no linux em português brasileiro. É possível salvar os comandos em algum imagem para que fique como um papel de parede.

Blog do Márcio d’Ávila » Fraude Surpreendo - Proteja seus dados pessoais

RootSudo - Ubuntu Brasil

Ripando e Gerando DVDs no Linux de forma simples « jmmwrite – simples e direto

I am also keeping this page for a reference to my future self because we cannot rely too much on our memory, and that will be a good reference I want to re-visit. I will make constant updates on this page.

Use objects with declared properties over array

Arrays have a larger footprint to avoid constant memory pointers reassignments. It then reserves large amounts of memory when more elements or indexes are added.

Be careful to self-referencing that would prevent garbage collector from work

Garbage collector is working as expected when the internal reference count (how may times a value is used) reaches zero:

$x = "foobar";    // refcount = 1
$y = $x;            // refcount = 2
unset($x);      // refcount = 1
unset($y);      // refcount = 0 -> garbage collector will be happy ==> Destroy!

But self-referencing can be tricky:

$x = [];            // refcount = 1
$x[0] =& $x;    // refcount = 2
unset($x);      // refcount = 1
                    // It will never come to zero due to cycle

The cycle collector will eventually destroy it, but it will hang on memory for a while anyway.

Sprintf vs double/single quote concatenation

A very common use case is string concatenation or interpolation when you want to add a variable into a static string. It is interesting to note that:

If you have PHP < 7.4, use double-quote interpolation or single quote concatenation over sprintf function.

<?php 

$this->start($loop);

ob_start();

for ($i = 0; $i < $this->loop; ++$i) {
    print 'Lorem '.$i.' ipsum dolor sit amet, consectetur adipiscing elit. Proin malesuada, nisl sit amet congue blandit';
}

ob_end_clean();

return $this->end();

If you have PHP greater than 7.4, use sprintf:

<?php 

$this->start($loop);

for ($i = 0; $i < $this->loop; ++$i) {
    $value = sprintf('Lorem %s ipsum dolor sit amet, consectetur adipiscing elit. Proin malesuada, nisl sit amet congue blandit', $i);
}

return $this->end();

• https://php.lito.com.es/test/compare/SprintfTest/PrintSingleQuoteCodeConcatTest
• https://php.lito.com.es/test/compare/SprintfTest/PrintDoubleQuoteCodeTest

PHP Benchmarking

PHPBench.com was constructed as a way to open people's eyes to the fact that not every PHP code snippet will run at the same speed. You may be surprised at the results that this page generates, but that is ok. This page was also created so that you would be able to find discovery in these statistics and then maybe re-run these tests in your own server environment to play around with this idea yourself, by using the code examples (these code examples are automatically generated and as the code in my .php files change, so do they).

PHP benchmarks and optimizations

Collection of tests and benchmarks for common operations in PHP. Tests run on several versions of PHP. There is an option to compare different solutions for the same problem to compare performances between them, such as checking values with isset against !empty.

It has never been so easy to document your things!

usql

A universal command-line interface for PostgreSQL, MySQL, Oracle Database, SQLite3, Microsoft SQL Server, and many other databases including NoSQL and non-relational databases!

Agendando tarefas com o Cron para Node

O Cron para Node é um pacote npm que nos permite fazer o agendamento de tarefas baseado em uma regra de tempo. Ele é baseado no Cron do Linux e seu funcionamento segue a mesma linha. Com ele é possível definir uma função para ser executada de tempos em tempos, ou seja, ela será agendada para ser executada dentro do Node. É uma maneira bastante eficaz para tarefas repetitivas que precisam rodar em segundo plano, como o envio de notificação, backup de banco de dados, entre outras.

Howto: use one VCL per domain

The Varnish Configuration Language (VCL), I'm sure you know already, is the source of Varnish versatility: by only enforcing the protocol flow and leaving the business logic to the user, Varnish can be easily configured to do things far beyond caching.

However, because the logic of websites is generally focused around hosts, and the VCL thinks in terms of processing steps, configuration may sometimes a bit odd, with the need to place safeguards around your code to ensure that logic for one host isn't applied to another one.

It works, but it can be tedious and unwieldy, so today we are going to have a look at how we can silo our VCL per website to achieve better maintainability.

Understanding the 8 Fallacies of Distributed Systems

Are you working on a distributed system? Microservices, Web APIs, SOA, web server, application server, database server, cache server, load balancer - if these describe components in your system's design, then the answer is yes. Distributed systems are comprised of many computers that coordinate to achieve a common goal.

More than 20 years ago Peter Deutsch and James Gosling defined the 8 fallacies of distributed computing. These are false assumptions that many developers make about distributed systems. These are usually proven wrong in the long run, leading to hard to fix bugs.

PostgreSQL Tuning: Key Things to Drive Performance

Performance is one of the key requirements in software architecture design, and has been the focus of PostgreSQL developers since its beginnings

Illuminate your career

If you are a developer, this article is for you.

5 Things You Have Never Done with a REST Specification

How to to Backup Linux with Snapshots

While working on different web projects I have accumulated a large pool of tools and services to facilitate the work of developers, system administrators and DevOps
One of the first challenges, that every developer faces at the end of each project is backup configuration and maintenance of media files, UGC, databases, application and servers' data (e.g. configuration files).

Awesome PHP

A curated list of amazingly awesome PHP libraries, resources and shiny things.

There’s a lot of information out there on how to configure PostgreSQL, on the importance of backups and testing them, etc.

But what about the server you run PostgreSQL on? We tend to pay a lot less attention to tools that you won’t need unless something breaks. But it’s worth taking some time to do so now, because that’s time you won’t have when your server is down and you’re in a rush.

SQL Feature Comparison

This comparison focuses on SQL features that can be used in SQL statements or self-contained SQL scripts that don't require additional software (e.g. a compiler) to be usable. Features for database administration or deployment are also not the focus of this comparison.

Building the DOM faster: speculative parsing, async, defer and preload

In 2017, the toolbox for making sure your web page loads fast includes everything from minification and asset optimization to caching, CDNs, code splitting and tree shaking. However, you can get big performance boosts with just a few keywords and mindful code structuring, even if you’re not yet familiar with the concepts above and you’re not sure how to get started.

How we tweaked Postgres upsert performance to be 2-3* faster than MongoDB

As we all know, relational databases are fine if you’re dealing with small amounts of data but for web-scale high performance high inserts speed and masses of queries per second, NoSQL is what you need. At least, that’s the conventional wisdom/hype surrounding NoSQL databases such as MongoDB. However as we’ve recently discovered this is very wrong indeed.

Painful Varnish mistakes

This post was initially titled "Top 6 Varnish mistakes", to echo Espen's blog. Even though his material is three years old, the content is still relevant. Plus, there's a healthy colleague competition going on here, and I can't just mimic Espen if I hope to beat him, so I had to do something different.

NULL in SQL: Indicating the Absence of Data

O método mais rápido para melhorar o desempenho de qualquer Servidor de Aplicações Web PHP usando MySQL ou PostgreSQL

Getting first and last values per group

Every so often someone needs solution to getting first (or couple of first) values for given column. Or last. For some cases (when there is not many groups) you can use recursive queries. But it's not always the best choice. Let's try to implement first() and last() aggregates, so these could be easily used by anybody.

Roadmap to becoming a web developer in 2017

Promises are very simple once you get your head around them, but there are a few gotchas that can leave you with your head scratching. Here are a few that got me.

PG Phriday: Why Postgres

Generic HTTP Error Handling in AngularJS

Lately during development at one of our clients, Ravello Systems, we decided we wanted better HTTP error handling.

Basically, our perfect solution would have generic handlers for errors, and most calls in the code will not have to do any special work for handling errors. This means that things like authentication problems, server unavailability issues, etc. will be handled in one place — like adding a generic “something went wrong” modal.

The Fastest Method to Evaluate Tune the Performance of Any PHP Web Application Server using MySQL or PostgreSQL

In the Web development world, we often have the problem of choosing the right server to use in the production environment of a Web application.

Maybe we need to buy a new server to handle the expected load, or maybe the customer wants to deploy in an existing server.

In any case, if after deploying and running the application it will show poor performance, then we need to ask the team what we can do to make the application faster or use a better server.

Therefore we need to determine if the application is performing well. Read this article to learn how to quickly determine the performance of an application on the current server.

Postgres 9.6 Features

PostgreSQL is now on version 16. Check the release notes

The Definitive Guide to DateTime Manipulation

As a software developer, you can’t run away from date manipulation. Almost every app a developer builds will have some component where date/time needs to be obtained from the user, stored in a database, and displayed back to the user.

collect-exec.sh – My personal OS report

The script collects a lot of information about the running system and save the output of each commands in a text file, and saves copies of important files in a directory named files. At the end of the script everything is compressed with tar in the global directory.

Faster PostgreSQL Counting

Everybody counts, but not always quickly. This article is a close look into how PostgreSQL optimizes counting. If you know the tricks there are ways to count rows orders of magnitude faster than you do already.

A function that would take two JSONB objects in PostgreSQL, and return how the left-hand side differs from the right-hand side.

Deadlocks in PostgreSQL

Compreenda os tipos de lock existentes no PostgreSQL, como ocorre um deadlock e pesquisar o causador do lock.

PG Phriday: COPY and Alternative Import Methods

Alternativa para cópia de conteúdo de arquivos para o banco de dados.

Always Do This #4: Put stats_temp_directory on a memory file system

The PostgreSQL statistics collector generates a lot of very important statistics about the state of the database. If it’s not working, autovacuum doesn’t work, among other problems. But it does generate a lot of write activity, and by default, that goes back onto the database volume.

Instead, always set statstempdirectory to point to a RAM disk (which has to be owned by the postgres user, with 0600 permissions). The statistics are written back to the database on shutdown, so in normal operations, you won’t lose anything on a reboot. (You’ll lose the most recent statistics on a crash, but you will anyway; the statistics are reset on recovery operations, including restart from a crash.)

This can substantially cut down the amount of write I/O the main database storage volume has to receive, and it’s free!

Putting stats_temp_directory on a ramdisk

When statistics are not generated, we can have, among other problems, the halt of the autovacuum execution. As a consequence of the problems caused by the interruption of statistics collection, large spikes in writing activity end up occurring, which overloads server utilization. Changing the stat_temp_directory setting can prevent this.

Há tabelas que possuem dados com os quais você não se importa de perdê-los. São situações de dados transientes, como áreas de dados passageiros, tabelas temporárias persistentes, tabelas com dados crus de importação. Por quê não aproveitar o fato do PostgreSQL oferecer a opção de ser UNLOGGED? Ainda mais porque pode-se evitar usar recursos do servidor desnecessariamente.

Check your pg_dump compression levels

Ao realizar backups do banco PostgreSQL, há muitas situações em que encontramos uma sobrecarga inesperada e o nível de compressão escolhido para fazer o backup pode ter ação direta sobre isso. Como a compressão nem sempre é tão importante, não esquecer este detalhe pode poupar incômodos desnecessários em operações de backup que não as rotineiras.

How to install an Opensource VPN Server on Linux

Instalação de VPN própria para assegurar o controle do tráfego em conexões.

Filtrando e validando dados no PHP com filter_var()

Entrada de dados é uma característica de quase a totalidade dos sistema ou sites. É indispensável, para segurança dos dados, filtrar esta entrada a fim de evitar invasões, roubo de dados ou inconsistência. No PHP, aprenda a fazer isso usando filter_var().

FFmpeg no Ubuntu: veja como instalar esse pacote no 14.04/14.10 via repositório

Trabalhando com logs no PostgreSQL

Dicas de configurações de log em servidores PostgreSQL. As informações contidas em logs são essenciais em muitos problemas e importantes para a saúde da aplicação e do sistema de banco de dados.