Categorias
Banco de dados PostGreSQL

Recover and replace win1252 content to utf-8 for a PostgreSQL database

I shouldn't be the first to need to export data from a PostgreSQL database installed on Windows with a win1252 code to a database with UTF-8 code (in my case, on a Linux server).

It is not enough to transform the transfer file to UTF-8, as the characters of win1252 (left double quote, right double quote, single quote and dash) will be there, with a weird value in your database.

In my experience, I had to import data as-is and afterwards perform an update using the function to rewrite data for UTF-8 characters correctly.

The following code examples are for: 1 - transform to HTML characters; 2 - transform to single characters.

HTML:

-- DROP FUNCTION substitui_win1252_html(texto);

CREATE OR REPLACE FUNCTION substitui_win1252_html(texto text)
    RETURNS text AS
$$
BEGIN

    texto := replace(replace(replace(replace(replace(texto, '’', '’'), '“', '“'), '”', '”'), '•','•'), '–', '–')::text; 
    RETURN texto;

END;
$$
LANGUAGE plpgsql;

Simple:

-- DROP FUNCTION substitui_win1252(texto);

CREATE OR REPLACE FUNCTION substitui_win1252(texto text)
    RETURNS text AS
$$
BEGIN

    texto := replace(replace(replace(replace(replace(texto, '’', ''''), '“', '"'), '”', '"'), '•','•'), '–', '-')::text;  
    RETURN texto;

END;
$$
LANGUAGE plpgsql;

Don't worry about the squares that you might see. If you copy it to a good text editor, you will their actual value.

Categorias
Linux Operational Systems Technology

Arch and my Thinkpad P14s

Here I will list some of the items I used to have Arch running with the minimum I needed for my ThinkPad P14s. I have experienced a seamless experience with my old ThinkPad Carbon x1.

I relate this to the Realtek 8852AE, an 802.11ax device used by P14s, that does not work out-the-box as the one used by x1, although faster.

Hardware

Software

Categorias
PHP

Introducing value objects in PHP

Domain-Driven Design (DDD) is a software design philosophy with one crucial concept: the structure and language of software code (class names, class methods, class variables) should match the business domain. To attend to this concept, DDD presents Value Objects, which, in practice, represents an object similar to a primitive type but should be modelled after the domain's business rules.

What does it mean?

(Checkout code at https://github.com/rafaelbernard/blog-value-objects)

Value Objects are first described in Evans' Domain-Driven Design book, and further explained in Smith and Lerman's Domain-Driven Design Fundamentals course. It is an immutable type that is distinguishable only by the state of its properties. Unlike an Entity1, a class type with a unique identifier and remains distinct even if its properties are otherwise identical, two Value Objects with the same properties can be considered equal.

We would use Value Object classes to represent a type strictly and encapsulate the validation rules of that type. Think of age or a card from a poker deck. They can seem sufficiently represented by primitive types such as integer and string, but in reality, they are managed by strict business rules2. For example, you would like to ensure that a particular age value is a non-negative integer. Or a picked card has a value not greater than 10, not 1, i.e., there is a range of allowed numbers or unique letters combined within four allowed suits.

To translate it to code, you could think of a Person class as a guest application to map a person's name and age. Usually, we would say a name is a string and age is an int.

class Person
{
    private string $name;
    private int $age;

    public function __constructor(string $name, int $age)
    {
        $this->name = $name;
        $this->age = $age;
    }
}

Although we can think that the specification above is good and we are just instantiating a Person object with all the required attributes, we may have some problems with the implementation because negative values are also int and would be allowed:

$personOk = new Person('John Doe', 18);
$notARealPerson = new Person('Benjamin Button', -18); // -18 year? no way!

The code above will not fail and is very common that some business logic is performed to assert age will never be negative:

// if from a form
$age = (int) $_POST['age'];

if ($age < 0) {
    throw new \Exception('Age could not be negative');
}

// but you need to copy the check everywhere a Person class is used. What if someone overlooks it?
$person = new Person('John Doe', $age);

When using Value Objects, you leverage your class with the exact type you need, correctly applying business logic. I will give more details later, but a hint of how the Person class would be with a Value Object:

use App\Domain;

class Person
{
    private string $name;
    private Age $age;

    public function __construct(string $name, Age $age)
    {
        $this->name = $name;
        $this->age = $age;
    }
}

And now we are always sure that Person objects will have a valid age. Bear with me and understand how Age Value Object class should look.

Samples of Value Objects

You may have already realized that we have many cases where we need similar things on every application. They are used to validate patterns to an expected format, a set of possible values to simulate an enum (only present on PHP 8) or to extend this set of values to be validated against some more rules.

I will be using PHP 7.4 compatible code in this blog. If you have applications using older versions, changing them should not be too complicated. Let me know in the comments if you need samples about how to write to an older version.

Some examples:

Validating a format:

Our Age class should be implemented as:

<?php

namespace App\Domain;

class Age
{
    private int $value;

    /**
     * @param int $value
     */
    public function __construct(int $value)
    {
        if (!$value < 0) {
            throw new \UnexpectedValueException('Negative numbers are not a valid age.');
        }

        $this->value = $value;
    }

    public function value(): int
    {
        return $this->value;
    }
}

Or a richer example with card suits:

<?php

namespace App\Domain;

class CardSuit
{
    const HEARTS = 'H';
    const DIAMONDS = 'D';
    const CLUBS = 'C';
    const SPADES = 'S';

    const SUITS = [
        self::HEARTS,
        self::DIAMONDS,
        self::CLUBS,
        self::SPADES,
    ];

    private string $value;

    public function __construct(string $value)
    {
        if (!in_array($value, self::SUITS)) {
            throw new \InvalidArgumentException("`$value` is not a valid card suit.");
        }

        $this->value = $value;
    }

    public function __toString()
    {
        return $this->value;
    }

    public function value(): string
    {
        return $this->value;
    }
}

Validating a set and its rules:

Observe rules when checking grade with isAlumni() from a given ClassYear. Ideally, we would have a Grade class, but I kept it more straightforward to simplify understanding the check, and I have a similar example with poker cards classes above.

<?php

namespace App\Domain;

class ClassYear
{
    private int $value;

    /**
     * @param int $value
     */
    public function __construct(int $value)
    {
        // For class year, our business rules is from 1901-2999
        if (!preg_match('/^((19\d{2})|(2)\d{3})$/', $value)) {
            throw new \InvalidArgumentException('Invalid year');
        }

        $this->value = $value;
    }

    public static function fromNow(): self
    {
        return new self(date('Y'));
    }

    public function value(): int
    {
        return $this->value;
    }

    public function isAlumni(): bool
    {
        return date('Y') - $this->value >= 13;
    }
}

Enum-like validation:

Observe that it encapsulates some Enum values, but also a simple rule (isGreaterThan), and it enriches the type with a simple business rule support that the application can also use.

<?php

namespace App\Domain;

class CardRank
{
    const ACE = 'A';
    const KING = 'K';
    const QUEEN = 'Q';
    const JACK = 'J';
    const TEN = 'X';
    const NINE = '9';
    const EIGHT = '8';
    const SEVEN = '7';
    const SIX = '6';
    const FIVE = '5';
    const FOUR = '4';
    const THREE = '3';
    const TWO = '2';

    const RANKS = [
        self::ACE,
        self::KING,
        self::QUEEN,
        self::JACK,
        self::TEN,
        self::NINE,
        self::EIGHT,
        self::SEVEN,
        self::SIX,
        self::FIVE,
        self::FOUR,
        self::THREE,
        self::TWO
    ];

    const WEIGHTS = [
        self::ACE => 20,
        self::KING => 13,
        self::QUEEN => 12,
        self::JACK => 11,
        self::TEN => 10,
        self::NINE => 9,
        self::EIGHT => 8,
        self::SEVEN => 7,
        self::SIX => 6,
        self::FIVE => 5,
        self::FOUR => 4,
        self::THREE => 3,
        self::TWO => 2
    ];

    private string $value;

    public function __construct(string $value)
    {
        if (!in_array($value, self::RANKS)) {
            throw new \InvalidArgumentException("`$value` is not a valid card rank.");
        }

        $this->value = $value;
    }

    public function __toString()
    {
        return $this->value;
    }

    public function value(): string
    {
        return $this->value;
    }

    public function weight()
    {
        return self::WEIGHTS[$this->value];
    }

    public function isGreaterThan(CardRank $cardRank): bool
    {
        return $this->weight() > $cardRank->weight();
    }

    // Some static helper functions can be created to be more readable

    public static function two(): CardRank
    {
        return new self(self::TWO);
    }

    public static function ace(): CardRank
    {
        return new self(self::ACE);
    }
}

When to use it

Create and use the Value Objects whenever you see that it fits "encapsulate the business rules for a given type" and "it represents an object similar to a primitive type". This will directly understand the expected type of code and instantly validate capabilities.

Some examples:

// a
$person = new Person('John Doe', new Age(20));
public function saveGuest(Person $person);

// b
$currentUserClassYear = ClassYear::fromNow();
if (!$currentUserClassYear->isAlumni()) {
    // do something for a non-alumni user
}

// c
$pickedCard = new Card(new CardSuit(CardSuit::SPADES), new CardRank(CardRank::ACE));

// d
$aceSpade = new Card(CardSuit::spades(), CardRank::ace());
$twoSpade = new Card(CardSuit::spades(), CardRank::two());
if ($aceSpace->isGreaterThan($twoSpade)) {
    // do something when greater, such as sum the weight to count points
}

A code-base that uses Value Objects avoids repetition code to validate a type that is not represented by native types, improves readability, and keeps consistency about business rules (no one could overlook or forget to "copy the validation code check"). Value Objects has a built-in validation, making it superior to validation classes and creating a relationship with your business domain rules. It then keeps the code clean and lean.

You can see all related code at https://github.com/rafaelbernard/blog-value-objects.

Glossary

  1. Entity: An entity may be defined as a thing capable of an independent existence that can be uniquely identified. An entity is an abstraction from the complexities of a domain. When we speak of an entity, we normally speak of some aspect of the real world that can be distinguished from other aspects of the real world. [(from Wikipedia)]
  2. Business rule: A business rule defines or constrains some aspect of business and always resolves to either true or false. Business rules are intended to assert business structure or to control or influence the behavior of the business.
  3. Object-oriented programming (OOP) is a programming paradigm based on the concept of "objects", which can contain data and code: data in the form of fields (often known as attributes or properties), and code, in the form of procedures (often known as methods).
Categorias
PHP

PHP 8.1: more on new in initializers

I could not agree more with Brent when he says concerning the "new in initializers"[1] feature:

PHP 8.1 adds a feature that might seem like a small detail, but one that I think will have a significant day-by-day impact on many people.

When I see this new feature, lots of places that use Dependency Injection[3] come to my focus as candidates to be impacted, such as application or infrastructure service classes. As a result, we will write a much cleaner and leaner code without giving up on good practices to write modular, maintainable and testable software.

The Dependency Inversion Principle[4] gives us decoupling powers. But we know that many classes will receive the same concrete implementation most of (if not all) the time.

So this is very common to see some variation of this code:

$someDependencyToBeInjected = FactoryClass::create();
$someService = new SomeServiceClass($someDependencyToBeInjected);

Important note: I will ignore for now Service Containers and frameworks features that deal with service instantiation, auto wiring, etc.

Think of a database query service class: you depend on a connection object. Every time you need to instantiate your database service class, you need to prepare the connection dependency and inject it at the service class. Database connections are a great example when you use the same concrete implementation more than 90% of the time.

The same applies to a Service class that you use to handle business logic and depends on QueryService and CommandHandler interfaces to do its job.

Before PHP 8.1 we have code like this:

// service class to apply business logic
// most standard
class DefaultLeadRecordService implements LeadRecordService
{
    public function __construct(
        private LeadQueryService $queryService,
        private LeadCommandHandler $commandHandler
    ) {
    }
}

// infrastructure class to match the interface -- a DBAL concrete class
// sneakily allowing a "default" value, but also open to Dependency Injection
// but not that great
class DbalLeadQueryService implements LeadQueryService
{
    public function __construct(private ?Connection $connection = null)
    {
        if (!$this->connection) {
            $this->connection = Core::getConnection();
        }
    }
}

// instantiation would be something like -- given you have $connection already instantiated
$connection =  \Doctrine\DBAL\DriverManager::getConnection($connectionParams, $config);

$service = new \Blog\Application\DefaultLeadRecordService(
    new \Blog\Infrastructure\DbalLeadQueryService($connection),
    new \Blog\Infrastructure\DbalLeadCommandHandler($connection),
);

// if we allow construct get default value
$service = new \Blog\Application\DefaultLeadRecordService(
    new \Blog\Infrastructure\DbalLeadQueryService(),
    new \Blog\Infrastructure\DbalLeadCommandHandler(),
);

While on PHP 8.1, you will be able to write it like so:

class DefaultLeadRecordService implements LeadRecordService
{
    public function __construct(
        private LeadQueryService $queryService = new DbalLeadQueryService(),
        private LeadCommandHandler $commandHandler = new DbalLeadCommandHandler()
    ) {
    }
}

// we see there is still room for new features here
// still not that great
class DbalLeadQueryService implements LeadQueryService
{
    public function __construct(private ?Connection $connection = null)
    {
        // waiting when new initializers feature allows static function as default parameters
        if (!$this->connection) {
            $this->connection = Core::getConnection();
        }
    }
}

$service = new \Blog\Application\DefaultLeadRecordService();

One liner! That saves a lot of typing, and the code remains very well structured. This is the type of significant impact we will have on our day-to-day work. We will write a more straightforward, robust and meaningful code, and we will ship features faster with high-quality code.

If you want to see a full implementation of this, check the code at https://github.com/rafaelbernard/blog-php-81-new-initializers

Test, our faithful friend

Writing tests is a must-have for any repository where quality is a requirement. However, the "New in initializers" feature does not force us to give up on a complete suite of tests. We still have all powers of unit or integration tests.

For application code, we would write unit tests and all the expectations for concrete dependencies:

<?php

namespace Test\Unit\Blog\Application;

use Blog\Application\DefaultLeadRecordService;
use Blog\Domain\LeadCommandHandler;
use Blog\Domain\LeadQueryService;
use PHPUnit\Framework\MockObject\MockObject;
use Test\TestCase;

class DefaultLeadRecordServiceTest extends TestCase
{
    private const EMAIL = '[email protected]';

    private LeadQueryService|MockObject $leadQueryServiceMock;
    private LeadCommandHandler|MockObject $leadCommandHandlerMock;

    private DefaultLeadRecordService $service;

    protected function setUp(): void
    {
        parent::setUp();

        $this->leadQueryServiceMock = $this->getMockBuilder(LeadQueryService::class)->getMock();
        $this->leadCommandHandlerMock = $this->getMockBuilder(LeadCommandHandler::class)->getMock();

        $this->service = new DefaultLeadRecordService($this->leadQueryServiceMock, $this->leadCommandHandlerMock);
    }

    public function testCanAdd()
    {
        $this->leadQueryServiceMock
            ->expects(self::once())
            ->method('getByEmail')
            ->with(self::EMAIL)
            ->willReturn(false);

        $this->leadCommandHandlerMock
            ->expects(self::once())
            ->method('add')
            ->with(self::EMAIL)
            ->willReturn(1);

        $result = $this->service->add(self::EMAIL);

        self::assertEquals(1, $result);
    }

    public function testAddExistentReturnsFalse()
    {
        $this->leadQueryServiceMock
            ->expects(self::once())
            ->method('getByEmail')
            ->with(self::EMAIL)
            ->willReturn(['email' => self::EMAIL]);

        $this->leadCommandHandlerMock
            ->expects(self::never())
            ->method('add');

        $result = $this->service->add(self::EMAIL);

        self::assertFalse($result);
    }

    public function testCanGetAll()
    {
        $unsorted = [
            ['email' => '[email protected]'],
            ['email' => '[email protected]'],
            ['email' => '[email protected]'],
        ];

        $this->leadQueryServiceMock
            ->expects(self::once())
            ->method('getAll')
            ->willReturn($unsorted);

        $fetched = $this->service->getAll();

        $expected = $unsorted;
        asort($expected);

        self::assertEquals($expected, $fetched);
    }
}

Integration tests can be written for infrastructure code. For instance, we can use an SQLite database file to assert the logic for database operations.

Be aware that I am creating an SQLite temp database file on-demand for each test execution with $this->databaseFilePath = '/tmp/test-' . time(); and, thanks to the Dbal library, we can be confident that operations could work for any database.

-> It is highly recommended that, as an alternative, create a container with a seeded database that is compatible with your production database system.

<?php

namespace Test\Integration\Blog\Infrastructure;

use Blog\Infrastructure\DbalLeadQueryService;
use Doctrine\DBAL\Connection;
use Faker\Factory;
use Faker\Generator;
use Test\TestCase;

class DbalLeadQueryServiceTest extends TestCase
{
    private string $databaseFilePath;

    private Generator $faker;
    private Connection $connection;

    private DbalLeadQueryService $service;

    public function testCanGetAll()
    {
        $this->addEmail($email1 = $this->faker->email());
        $this->addEmail($email2 = $this->faker->email());
        $this->addEmail($email3 = $this->faker->email());

        $expected = [
            ['email' => $email1],
            ['email' => $email2],
            ['email' => $email3],
        ];

        $fetched = $this->service->getAll();

        self::assertEquals($expected, $fetched);
    }

    protected function setUp(): void
    {
        parent::setUp();

        $this->faker = Factory::create();

        $this->createLeadTable();

        $this->service = new DbalLeadQueryService($this->connection());
    }

    protected function tearDown(): void
    {
        parent::tearDown();

        $this->dropDatabase();
    }

    private function connection(): Connection
    {
        if (!isset($this->connection)) {
            $this->databaseFilePath = '/tmp/test-' . time();

            $config = new \Doctrine\DBAL\Configuration();
            $connectionParams = [
                'url' => "sqlite:///{$this->databaseFilePath}",
            ];

            $this->connection = DriverManager::getConnection($connectionParams, $config);
        }

        return $this->connection;
    }

    private function dropDatabase()
    {
        @unlink($this->databaseFilePath);
    }

    private function createLeadTable(): void
    {
        $this->connection()->executeQuery('CREATE TABLE IF NOT EXISTS leads ( email VARCHAR )');
    }

    private function addEmail(string $email): int
    {
        return $this->connection()->insert('leads', ['email' => $email]);
    }
}

Conclusion

PHP is evolving very quickly, with new features that enable more quality software, help developers and is even more committed to the fact that most of the web run flavours of PHP code. New features improve readability, software architecture, test coverage and performance. Those are all proof of a mature and live language.

Upgrade to PHP 8.1 and use "new in initializers" as soon as possible. You will not regret it.

If there is something you want to discuss more, let me know in the comments.

Links:

  1. New in initializers RFC
  2. Road to PHP 8.1
  3. Dependency Injection
  4. Dependency inversion principle
  5. Interface segregation principle
  6. Solid relevance
  7. SOLID principles
Categorias
PHP Programação Tropeçando

Tropeçando 103

What is Domain-Driven Design (DDD)

A definition of DDD as a software design discipline

How to refactor without overtime and missed deadlines

A lot of software engineers, including myself, are passionate about code quality. This striving for a well-shaped codebase, while getting things done could cost one quite a few hours and nerves, though. I'm constantly looking for ways to achieve these two goals without significant trade-offs. Stand by for the current state.

How to test a PHP app? PHP unit testing and more

Do you really need to create tests? Of course, there are many reasons to do so – improved quality of software, decreased risks while making changes in the code, identifying errors, checking business requirements, improving security…I could go on and on with that. The point is – tests do make a difference.

Application Modernization Isn’t Just Fighting Legacy Tech

When radical innovations were rare, businesses could afford to treat application modernization as a sporadic reaction to change. A decade ago, most organizations modernized only when they were compelled to.

However, in the era of open-source and continuous innovation, modernization can’t be an isolated, one-off project. Businesses need to embrace a culture that celebrates change to thrive in the digital age. According to a report by F5, the past year has witnessed 133% growth in application modernization.

Responsible tech playbook

As technology becomes more central to peoples' lives, and to what businesses do, and how they succeed, the ethics of technology must come into sharper focus.

Despite technology becoming a critical part of what enterprizes do, it's not always clear how to approach and apply technology in an ethical or responsible way.

The Responsible tech playbook is a collection of tools, methods, and frameworks that help you to assess, model and mitigate values and risks of the software you are creating with a special emphasis on the impact of your work on the individual and society.

Categorias
PHP Programação

PHP Memory Usage and Performance Improvements Tips

Memory usage and performance improvements make everybody happier, from end-user to cloud and infrastructure engineers. And they are all right, and this is an optimization that we should try to achieve as much as possible.

I am also keeping this page for a reference to my future self because we cannot rely too much on our memory, and that will be a good reference I want to re-visit. I will make constant updates on this page.

Use objects with declared properties over array

Arrays have a larger footprint to avoid constant memory pointers reassignments. It then reserves large amounts of memory when more elements or indexes are added.

Image for array vs object memory usage

Be careful to self-referencing that would prevent garbage collector from work

Garbage collector is working as expected when the internal reference count (how may times a value is used) reaches zero:

$x = "foobar";    // refcount = 1
$y = $x;            // refcount = 2
unset($x);      // refcount = 1
unset($y);      // refcount = 0 -> garbage collector will be happy ==> Destroy!

But self-referencing can be tricky:

$x = [];            // refcount = 1
$x[0] =& $x;    // refcount = 2
unset($x);      // refcount = 1
                    // It will never come to zero due to cycle

The cycle collector will eventually destroy it, but it will hang on memory for a while anyway.

Sprintf vs double/single quote concatenation

A very common use case is string concatenation or interpolation when you want to add a variable into a static string. It is interesting to note that:

If you have PHP < 7.4, use double-quote interpolation or single quote concatenation over sprintf function.

<?php 

$this->start($loop);

ob_start();

for ($i = 0; $i < $this->loop; ++$i) {
    print 'Lorem '.$i.' ipsum dolor sit amet, consectetur adipiscing elit. Proin malesuada, nisl sit amet congue blandit';
}

ob_end_clean();

return $this->end();

If you have PHP greater than 7.4, use sprintf:

<?php 

$this->start($loop);

for ($i = 0; $i < $this->loop; ++$i) {
    $value = sprintf('Lorem %s ipsum dolor sit amet, consectetur adipiscing elit. Proin malesuada, nisl sit amet congue blandit', $i);
}

return $this->end();

PHP Benchmarking

PHPBench.com was constructed as a way to open people's eyes to the fact that not every PHP code snippet will run at the same speed. You may be surprised at the results that this page generates, but that is ok. This page was also created so that you would be able to find discovery in these statistics and then maybe re-run these tests in your own server environment to play around with this idea yourself, by using the code examples (these code examples are automatically generated and as the code in my .php files change, so do they).

PHP benchmarks and optimizations

Collection of tests and benchmarks for common operations in PHP. Tests run on several versions of PHP. There is an option to compare different solutions for the same problem to compare performances between them, such as checking values with isset against !empty.

Categorias
PostGreSQL

PostgreSQL – pg_upgrade from 10 to 12

I have some PostgreSQL databases running pretty well but we need to keep our software updated. This is a mandatory practice for a high-quality service. Those servers are running version 10 and they need to be upgraded to version 12. I have used pg_dump / pg_restore strategy for a long time, but this time I would rather use pg_upgrade.

Let's dive into how to do it.

Table of contents:

  1. Install
  2. InitDB
  3. Check upgrade consistency
  4. Set locale
  5. Upgrade
  6. Configurations

Install

The package postgresql12-server contains everything needed to run the server, but my databases use some extensions [1], then I will add postgresql12-devel and postgresql12-contrib to be able to compile and to install the extensions.

yum install postgresql12-server postgresql12-devel postgresql12-contrib

InitDB

After installation we need to setup new server with initdb:

~% /usr/pgsql-12/bin/postgresql-12-setup initdb
Initializing database … OK

Check upgrade consistency

We need to check compatibility. Turn to postgres user (su - postgres) and run the command:

~% /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin --new-bindir=/usr/pgsql-12/bin --old-datadir=/var/lib/pgsql/10/data --new-datadir=/var/lib/pgsql/12/data --check

Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               fatal

Your installation contains tables declared WITH OIDS, which is not supported
anymore. Consider removing the oid column using
    ALTER TABLE ... SET WITHOUT OIDS;
A list of tables with the problem is in the file:
    tables_with_oids.txt

Failure, exiting

As you may see, I got a fatal error, indicating that the upgrade is not possible. In my case, tables with OIDs are the culprit. In your case could be something else. In any case, we need to fix before upgrading.

I fixed tables removing OIDs on mentioned tables. And ran check again:

Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

*Clusters are compatible*

Yay!

Set locale

There is a tricky configuration that is not detected by pg_upgrade check but it is very important to me. I use C locale on my databases [2], then I need to perform an extra step. If this is your case, you may follow the same steps applying yours.

I need to stop postgresql10 and start postgresql12:

systemctl stop postgresql-10.service
systemctl start postgresql-12.service

Then I run locale change at my template1 then locale will be enabled when my database will be upgraded.

UPDATE pg_database SET datcollate='C', datctype='C' WHERE datname='template1';

And stop again: systemctl stop postgresql-12.service to be ready to upgrade.

Upgrade

Upgrade command is the same that we run before, without --check flag.

~% /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin --new-bindir=/usr/pgsql-12/bin --old-datadir=/var/lib/pgsql/10/data --new-datadir=/var/lib/pgsql/12/data

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Consider running analyze_new_cluster. Optional but nice to have.

vacuumdb: processing database "mydb": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "mydb": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "mydb": Generating default (full) optimizer statistics
vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
vacuumdb: processing database "template1": Generating default (full) optimizer statistics

Done

Configurations

Before deleting your old cluster, remember to get some of your configurations.

mv /var/lib/pgsql/12/data/pg_hba.conf /var/lib/pgsql/12/data/pg_hba.conf.new
cp /var/lib/pgsql/10/data/pg_hba.conf /var/lib/pgsql/12/data/

I am not a big fan of writing directly to postgresql.conf file. Instead, I keep configuration files under version control and include the directory where those files are deployed. I treat them as code, then it becomes easier to maintain and manage.

Another advantage is that I don't have any mess about config file differences when a new version arises. I am automatically using new default configurations, my customized setting are loaded and I can quickly address any incompatibility caused by migration without touching the original conf file.

Let's go for it:

# Add settings for extensions here
include_dir = '/var/lib/pgsql/conf.d'   # include files ending in '.conf' from

Then you may delete your old cluster. 🙂

Links:

  1. Extensions: https://www.postgresql.org/docs/current/contrib.html
  2. Locale: https://www.postgresql.org/docs/current/locale.html
Categorias
PHP Programação Technology

PHP Test Coverage Using Bitbucket and Codacy

Wikipedia:

In computer science, code coverage is a measure used to describe the degree to which the source code of a program is tested by a particular test suite. A program with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a program with low code coverage.

Testing is an unavoidable process for building a trustful software. Unfortunately, in PHP world we have a massive number of legacy software still running today that are very valuable but born in an age where testing was skipped for various reasons.

As today we are refactoring those untested systems into tested ones or we are creating new projects already focusing on having tests, we can go one step further and measure the code coverage, leveraging bug protection and code quality.

You can use these steps for a legacy project, a new project, a well-covered project, a poorly covered project; no matter the state of your project.

We are considering a PHP project using Bitbucket Pipelines as our CI and Codacy for monitoring our test coverage reports but the main concepts could be easily used when using other tools.

Table of contents:

  1. Dependencies installation
  2. PHPUnit configuration
  3. Set up Codacy project API token
  4. Pipelines configuration

1. Dependencies installation

Use composer to install dependencies:

composer require --dev phpunit/php-code-coverage codacy/coverage

Installation results would be similar to:

Using version ^1.4 for codacy/coverage
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 3 installs, 0 updates, 0 removals
  - Installing symfony/process (v5.0.4): Downloading (100%)
  - Installing gitonomy/gitlib (v1.2.0): Downloading (100%)
  - Installing codacy/coverage (1.4.3): Downloading (100%)
Writing lock file
Generating autoload files
ocramius/package-versions:  Generating version class...
ocramius/package-versions: ...done generating version class

2. PHPUnit configuration

We need to configure at least whitelist and logging sections. They are required to code coverage.

Whitelist is the section that determines which files will be considered as your available code and how existent tests cover this code.

As I want that all my code loaded and analyzes by PHPUnit, I will set processUncoveredFilesFromWhitelist. Considering that all my code is under ./src folder:

<phpunit>
    <!-- ... -->
    <filter>
        <!-- ... -->
        <whitelist processUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./src</directory>
        </whitelist>
    </filter>
</phpunit>

Logging is where we configure logging of the test execution. Clover configuration is enough for now:

<phpunit>
    <!-- ... -->
    <logging>
      <log type="coverage-clover" target="/tmp/coverage.xml"/>
    </logging>
</phpunit>

You should now run your tests locally to ensure that you can fix everything that will be analyzed by PHPUnit. All errors have to be fixed. One example that might appear:

Fatal error: Interface 'InterfaceClass' not found in /var/www/src/Example/Application/ClassService.php on line 5

Oops.

<?php

namespace Example;

class ClassService implements InterfaceClass {
    /* */
}

I have a class extending from another but I missed the import for the parent class.

<?php

namespace Example;

use Example\Domain\InterfaceClass;

class ClassService implements InterfaceClass {
    /* */
}

And now I am good. After fixing all errors that might appear (and discovering some dead classes...), we test results and report generation message:

OK (10 tests, 17 assertions)

Generating code coverage report in Clover XML format ... done [5.71 seconds]

This has already configured code coverage. We will use Codacy as a tool to keep track of code coverage status, representing them in a beautiful dashboard and some other tools such as a check for new pull requests.

3. Set up Codacy project API token

For sending coverage results to Codacy, we need the project API token. This is located at Settings > Integrations tab.

If there is already a code, you can use it. Otherwise, generate one. A project token would look like something as:

a9564ebc3289b7a14551baf8ad5ec60a // not real

We will use this as an environment variable at Bitbucket. At your project in Bitbucket, go to Configurations > Pipelines > Repository Variables. In my case, I used:

Name: CODACY_PROJECT_TOKEN
Value: a9564ebc3289b7a14551baf8ad5ec60a

I want the value securely encrypted. Then I mark "Secure".

Right now we have Codacy token and the value enabled to use as an environment variable at our pipeline.

4. Pipelines configuration

For Pipelines now you should provide API token as an environment variable:

variables:
  - CODACY_PROJECT_TOKEN: $CODACY_PROJECT_TOKEN

Enable xdebug:

  - pecl install xdebug-2.9.2 && docker-php-ext-enable xdebug

And execute codacycoverage to send saved report to Codacy:

  - src/vendor/bin/codacycoverage clover /tmp/coverage.xml

Considering one of my legacy projects that I am adding code coverage, this could be my unit test step:

  • using alpine
  • source code (whitelisted) at site/src
  • logfile generated at /tmp/unit-clover.xml
  • g++ gcc make git php7-dev are required to install and enable xdebug
    - step: &step-unit-tests
        name: unit tests
        image: php:7.2-alpine
        variables:
          - CODACY_PROJECT_TOKEN: $CODACY_PROJECT_TOKEN
        caches:
          - composer
        script:
          - apk add --no-cache g++ gcc make git php7-dev
          - pecl install xdebug-2.9.2 && docker-php-ext-enable xdebug
          - site/src/vendor/bin/phpunit -c tests/Unit/phpunit.xml
          - site/src/vendor/bin/codacycoverage clover /tmp/unit-clover.xml

After being successfully executed by pipelines, you may see the results at your Codacy dashboard.

Now code with love.
And coverage.


Useful links:

Categorias
Programação

Environment Variables in Angular

Need to use different values depending on the environment you’re in? If you’re building an app that needs to use API host URLs depending on the environment, you may do it easily in Angular using the environmen.ts file.

We are considering Angular 8+ apps for this article.

Angular CLI projects already use a production environment variable to enable production mode when in the production environment at main.ts:

if (environment.production) {
  enableProdMode();
}

And you'll also notice that by default in the src/environment folder you have an environment file for development and one for production. Let's use this feature to allow us to use different API host URL depending if we're in development or production mode:

environment.ts:

export const environment = {
  production: false,
  apiHost: https://api.local.com
}

environment.prod.ts:

export const environment = {
  production: true,
  apiHost: https://api.production-url.com
};

And in our app.component.ts all we have to do in order to access the variable is the following:

import { Component } from '@angular/core';
import { environment } from '../environments/environment';

@Component({ ... })
export class AppComponent {
  apiHost: string = environment.apiHost;
}

Now in development mode the apiHost variable resolves to https://api.local.com and in production resolves to https://api.production-url.com. You may run ng build --prod and check.

Detecting Development Mode

Angular also provides us with an utility function called isDevMode that makes it easy to check if the app in running in dev mode:

import { Component, OnInit, isDevMode } from '@angular/core';

@Component({ ... })
export class AppComponent implements OnInit {
  ngOnInit() {
    if (isDevMode()) {
      console.log('Development!');
    } else {
      console.log('Cool. Production!');
    }
  }
}

Adding a Staging Environment

To add a new environment in Angular projects a new entry to configuration property should be added at angular.json file. Let's add a staging environment for example. Note that production property already exists.

"configurations": {
  "production": {
    "optimization": true,
    "outputHashing": "all",
    "sourceMap": false,
    "extractCss": true,
    "namedChunks": false,
    "aot": true,
    "extractLicenses": true,
    "vendorChunk": false,
    "buildOptimizer": true,
    "fileReplacements": [
      {
        "replace": "src/environments/environment.ts",
        "with": "src/environments/environment.prod.ts"
      }
    ]
  },
  "staging": {
    "optimization": true,
    "outputHashing": "all",
    "sourceMap": false,
    "extractCss": true,
    "namedChunks": false,
    "aot": true,
    "extractLicenses": true,
    "vendorChunk": false,
    "buildOptimizer": true,
    "fileReplacements": [
      {
        "replace": "src/environments/environment.ts",
        "with": "src/environments/environment.stating.ts"
      }
    ]
  }

And now we can add a staging environment file and suddenly be and build the project with ng build --configuration=staging on our CI (or deploy process) to deploy on staging environment:

environment.staging.ts

export const environment = {
  production: false,
  apiHost: https://staging.host.com
};
Categorias
PHP

High-performance Fibonacci numbers generator in PHP

Based on the article High-performance Fibonacci numbers generator in Go I wrote my version using PHP. Despite the differences between PHP and Go architectures reflected in response times, we can face a huge performance difference when using an optimized function. We may notice that we can have the same results, but the quality of the written code can change lots of things.

Recursive approach

function fibonacci(int $n):int {
  if ($n <= 1) {
    return $n;
  }

  return fibonacci($n-1) + fibonacci($n-2);
}

Benchmark and test

function test_fibonacci() {
  $data = [
    [0,0], [1,1], [2,1], [3,2], [4,3], [5,5], [6,8], [10,55], [42,267914296]
  ];

  foreach($data as $test) {
    $result = fibonacci($test[0]);
    if ($result !== $test[1]) {
      throw new \UnexpectedValueException("Error Processing Request. N: {$test[0]}, got: {$result}, expected: {$test[1]}", 1);
    }
  }

  echo "Tests - Success.".PHP_EOL;
}

/**
  * From https://gist.github.com/blongden/2352583
  */
function benchmark($x)
{
    $start = $t = microtime(true);
    $total = $c = $loop = 0;
    while (true) {
        $x();
        $c++;
        $now = microtime(true);
        if ($now - $t > 1) {
            $loop++;
            $total += $c;
            list($t, $c) = array(microtime(true), 0);
        }
        if ($now - $start > 2) {
            return round($total / $loop);
        }
    }
}
Benchmark 10 run: 163,754/sec or 0.0061067210571955ms/op
Benchmark 20 run: 1,351/sec or 0.74019245003701ms/op

As we can see, calculations of 20 Fibonacci numbers takes 123 times longer than 10 Fibonacci numbers. Not well performed at all! The explanation can be found in the linked article.

Sequential approach

function fibonacci_tuned(int $n):float {
  if ($n <= 1) {
    return $n;
  }

  $n2 = 0;
  $n1 = 1;

  for ($i = 2; $i < $n; $i++) {
    $n2_ = $n2;
    $n2 = $n1;
    $n1 = ($n1 + $n2_);
  }

  return $n2 + $n1;
}

function test_fibonacci_tuned() {
  $data = [
    [0,0], [1,1], [2,1], [3,2], [4,3], [5,5], [6,8], [10,55], [42,267914296]
  ];

  foreach($data as $test) {
    $result = fibonacci_tuned($test[0]);
    $float_test_value = (float) $test[1];
    if ($result !== $float_test_value) {
      throw new \UnexpectedValueException("Error Processing Request. N: {$test[0]}, got: {$result}, expected: {$float_test_value}", 1);
    }
  }

  echo "Tests - Success.".PHP_EOL;
}

Results:

Benchmark 10 tuned run: 3,345,999/sec or 0.00029886440492062ms/op
Benchmark 20 tuned run: 2,069,100/sec or 0.00048330191870862ms/op

As a much better scenario, calculate 20 numbers takes almost 2 times longer than 10 numbers. Makes sense. And performs well!

Considering the two approaches, the recursive approach runs 10 Fibonacci numbers operations 20 times longer than sequential one and 1,824 times longer for 20 Fibonacci numbers.

Fibonacci implementation in PHP can be found at https://github.com/rafaelbernard/php-fibonacci.