Tropeçando 98

(Operating Lambda: Debugging code – Part 1)[https://aws.amazon.com/blogs/compute/operating-lambda-debugging-code-part-1/]
(Operating Lambda: Debugging configurations – Part 2)[https://aws.amazon.com/blogs/compute/operating-lambda-debugging-configurations-part-2/]
(Operating Lambda: Debugging configurations – Part 3)[https://aws.amazon.com/blogs/compute/operating-lambda-debugging-integrations-part-3/

In the Operating Lambda series, I cover important topics for developers, architects, and systems administrators who are managing AWS Lambda-based applications. This three-part series discusses core debugging concepts for Lambda-based applications.

(Those pesky pull request reviews)[https://jessitron.com/2021/03/27/those-pesky-pull-request-reviews/]

They’re everywhere. In Slack: “hey, can I get a review on this?” In email: “Your review is requested!” In JIRA: “8 user stories In-Progress” (but code-complete). In your repository: 5 open pull requests. They’re slowing your delivery. They’re interrupting your developers.

How can we get people to review pull requests faster??

(Operating Lambda: Using CloudWatch Logs Insights)[https://aws.amazon.com/blogs/compute/operating-lambda-using-cloudwatch-logs-insights/]

In the Operating Lambda series, I cover important topics for developers, architects, and systems administrators who are managing AWS Lambda-based applications. This three-part series discusses monitoring and observability for Lambda-based applications and covers:

  • Using Amazon CloudWatch, CloudWatch Logs Insights, and AWS X-Ray to apply monitoring
    across services.
  • How existing monitoring concepts apply to Lambda-based applications.
  • Troubleshooting application issues in an example walkthrough.
    This post explains how to use CloudWatch Logs Insights in your serverless applications.

(CDK Lambda Deployment takes about a minute - how about sub second Function Code Deployment?)[https://aws-blog.de/2021/04/cdk-lambda-deployment-takes-about-a-minute-how-about-sub-second-function-code-deployment.html]

Creation of Lambda infrastructure with the CDK is really powerful. Updating the Function code is really slow. Here is a fix for that to get to a sub-second Lambda function deployment time.

(Best practices for developing cloud applications with AWS CDK)[https://aws.amazon.com/blogs/devops/best-practices-for-developing-cloud-applications-with-aws-cdk/]

In this post, we discuss strategies for organizing the development of complex cloud applications with large teams, using the AWS Cloud Development Kit (AWS CDK) as a central technology. AWS CDK allows developers and administrators to define their cloud applications using a familiar programming language, such as TypeScript, Python, Java, or C#. Applications are organized into stages, stacks, and constructs, which allows for modular design techniques in both runtime logic (such as AWS Lambda code or containerized services) and infrastructure components such as Amazon Simple Storage Service (Amazon S3) buckets, Amazon Relational Database Service (Amazon RDS) databases, and network infrastructure.

Tropeçando 97

The Missing Guide to AWS API Gateway Access Logs

In this post, we’re continuing the deep dive on API Gateway. Here, we’ll be looking at API Gateway access logging. Access logging can save your bacon when debugging a gnarly API Gateway issue, but you need to understand some nuance before you can use it correctly. We’ll dig into the details here so that you’ll be logging like Paul Bunyan in no time.

Ready for changes with Hexagonal Architecture

Netflix considerations related to the decisions of using Hexagonal Architecture

OAuth Patterns and Anti-Patterns RefCard

Modern, more secure recommendations continue to replace some of OAuth's original elements as the protocol evolves. Securing access to APIs and other resources and data effectively under OAuth 2.0 requires first learning the components and tools it involves.

Can auto_explain (with timing) have low overhead?

Some benchmarks for use of auto_explain. It seems great to be enabled on production.

Here we’ll be looking into the overhead of auto_explain. Many places warn about its overhead, but I’ve found concrete information difficult to come by.

Setting up SSL authentication for PostgreSQL

PostgreSQL is a secure database and we want to keep it that way. It makes sense, then, to consider SSL to encrypt the connection between client and server. This posting will help you to set up SSL authentication for PostgreSQL properly, and hopefully also to understand some background information to make your database more secure.

Tropeçando 96

Refinement Code Review

With the right environment, I can look a bit of code written six months ago, see some problems with how it's written, and quickly fix them. This may be because this code was flawed when it was written, or that changes in the code base since led to the code no longer being quite right. Whichever the cause, the important thing is to fix problems as soon as they start getting in our way. As soon as I have an understanding about the code that wasn't immediately apparent from reading it, I have the responsibility to (as Ward Cunningham so wonderfully said) take that understanding out of my head and put it into the code. That way the next reader won't have to work so hard.

After all, many problems that code reviews seek to remedy are problems that only become problems when the code is read in the future.

MONITORING REPLICATION: PG_STAT_REPLICATION

PostgreSQL replication (synchronous and asynchronous replication) is one of the most widespread features in the database community. Nowadays, people are building high-availability clusters or use replication to create read-only replicas to spread out the workload. What is important to note here is that if you are using replication, you must make sure that your clusters are properly monitored.

The purpose of this post is to explain some of the fundamentals, to make sure that your PostgreSQL clusters stay healthy.

POSTGRESQL: WHAT IS A CHECKPOINT?

Check some information about checkpoints, mix and max wal size, how they operate toghether and what they play on our day-to-day database workload.

Checkpoints are a core concept in PostgreSQL. However, many people don’t know what they actually are, nor do they understand how to tune checkpoints to reach maximum efficiency. This post will explain both checkpoints and checkpoint tuning, and will hopefully shed some light on these vital database internals.

Boos your user defined functions in PostgreSQL

Using the RDBMS only to store data is restricting the full potential of the database systems, which were designed for server-side processing and provide other options besides being a data container. Some of these options are stored procedures and functions that allow the user to write server-side code, using the principle of bringing computation to data, avoiding large datasets round trips and taking advantage of server resources. PostgreSQL allows programming inside the database since the beginning, with User Defined Functions (UDFs). These functions can be written in several languages like SQL, PL/pgsql, PL/Python, PL/Perl, and others. But the most common are the first two mentioned: SQL and PL/pgsql. However, there may be “anti-patterns” in your code within functions and they can affect performance. This blog will show the reader some simple tips, examples and explanations about increasing performance in server-side processing with User Defined Functions in PostgreSQL. It is also important to clarify that the intention of this post isn’t to discuss whether Business Logic should be placed, but only how you can take advantage of the resources of the database server.

The career-changing art of reading the docs

Don’t wait for knowledge to find you through years of inefficient trial and error. Go get it. And the most convenient, comprehensive place to grab it was there in front of you all along.

Read the docs.

Tropeçando 95

Windy - TailwindCSS

Transform every element on any website into Tailwind CSS

AWS Workshops

This website lists workshops created by the teams at Amazon Web Services (AWS). Workshops are hands-on events designed to teach or introduce practical skills, techniques, or concepts which you can use to solve business problems.

Well-Architected AWS

The Well-Architected framework has been developed to help cloud architects build the most secure, high-performing, resilient, and efficient infrastructure possible for their applications. This framework provides a consistent approach for customers and partners to evaluate architectures, and provides guidance to help implement designs that will scale with your application needs over time.

This repository contains documentation and code in the format of hands-on labs to help you learn, measure, and build using architectural best practices. The labs are categorized into levels, where 100 is introductory, 200/300 is intermediate and 400 is advanced.

Fallacies of distributed computing

false assumptions that programmers new to distributed applications invariably make.

Understanding DynamoDB Condition Expressions

Some use cases to understand this powerful yet misunderstood feature of DynamoDB. There are also examples of bad use of it.

Tropeçando 94

https://extendsclass.com/

ExtendsClass provides tools directly usable in a browser. It saves you from having to install add-ons to your browser in order to add features.

You have at your disposal syntax validators, code formatters, testers, HTTP clients, mock server, but also a SQLite browser.

These are small and easy-to-use tools that can help when you do not want to install software on your workstation.

Solid Relevance

More topics that highlighs the importance of SOLID concepts. How they are key to develop a solid application.

Kubernetes: ClusterIP vs NodePort vs LoadBalancer, Services, and Ingress — an overview with examples

Dockerfile best practices

Writing production-worthy Dockerfiles is, unfortunately, not as simple as you would imagine. Most Docker images in the wild fail here, and even professionals often[1] get[2] this[2] wrong[3].

This repository has best-practices for writing Dockerfiles that I (@slimsag) have quite painfully learned over the years both from my personal projects and from my work @sourcegraph. This is all guidance, not a mandate - there may sometimes be reasons to not do what is described here, but if you don't know then this is probably what you should be doing.

How to Make Your Code Reviewer Fall in Love with You

When people talk about code reviews, they focus on the reviewer. But the developer who writes the code is just as important to the review as the person who reads it. There’s scarcely any guidance on preparing your code for review, so authors often screw up this process out of sheer ignorance.

This article describes best practices for participating in a code review when you’re the author. In fact, by the end of this post, you’re going to be so good at sending out your code for review that your reviewer will literally fall in love with you.

Tropeçando 93

CQRS Is an Anti-Pattern for DDD

Are you interested in new ways to build better software systems? If you work with distributed systems or build any kind of web application, you most likely have heard of the new trends like using Domain-Driven Design with Event-Sourcing and Command Query Responsibility Segregation (CQRS). Well, they are not exactly brand new. However, they are now becoming increasingly popular.

nip.io/

Dead simple wildcard DNS for any IP Address

Stop editing your /etc/hosts file with custom hostname and IP address mappings.

The Tighten Test: 12 Steps to a Better Team

Twenty years ago today, Joel Spolsky (who later co-founded Stack Overflow) published The Joel Test: 12 Steps to Better Code listing 12 metrics for rating the quality of a software development team. The premise is simple: you get 1 point for each “yes” answer, for a total score of up to 12 points.

Architecture decision records, also known as ADRs, are a great way to document how and why a decision was reached within a codebase. We’ve started to adopt them within the mobile team here at GitHub, documenting decisions that affect the iOS codebase and Android codebase, as well as decisions that affect both mobile clients.

ADRs are not the most common within open source codebases, but have gained more popularity since ~2017 within long-lived, “evolutionary” codebases like those in more enterprise-y settings.

So why write an ADR? Why spend time documenting something when a decision has already been made?

https://free-for.dev/

This is a list of software (SaaS, PaaS, IaaS, etc.) and other offerings that have free tiers for developers.

Tropeçando 92

Mocking/stubbing the current Date in Jest tests

This post goes through multiple approaches to mocking, stubbing and spying on the date constructor using Jest.

Don't get stuck

A history about the importance of know your time and to not get stuck.

For this special history guide, we are going to take a trip back in time to see where the seed of Linux was planted — namely via the Unix systems of the early 1970s and how it has progressed through the modern day. Though most are completely unaware of the enormous impact that Unix-like operating systems have planted on our society, understanding its storied history can allow us to realize why the Unix model has lived on far longer and become more successful than any other operating system architecture (and philosophy) in existence.

tmpmail - A temporary email right from your terminal

tmpmail is a command line utility that allows you to create a temporary email address and receive emails to the temporary email address. It uses 1secmail's API to receive the emails.

EXPLAIN docs

After a lot of time looking at query plans, we’re still coming across new operation types, fields, and terminology. Many of these terms were tricky to look up and understand, so we decided to share descriptions and useful links for many of the most common in a glossary.

Tropeçando 91

Introducing the MDN Web Docs Front-end developer learning pathway

The MDN Web Docs Learning Area (LA) was first launched in 2015, with the aim of providing a useful counterpart to the regular MDN reference and guide material. MDN had traditionally been aimed at web professionals, but we were getting regular feedback that a lot of our audience found MDN too difficult to understand, and that it lacked coverage of basic topics.
https://developer.mozilla.org/en-US/docs/Learn

SQL Optimizations in PostgreSQL: IN vs EXISTS vs ANY/ALL vs JOIN

This is one of the most common questions asked by developers who write SQL queries against the PostgreSQL database. There are multiple ways in which a sub select or lookup can be framed in a SQL statement. PostgreSQL optimizer is very smart at optimizing queries, and many of the queries can be rewritten/transformed for better performance.

Expose

Expose is a beautiful, open source, tunnel application that allows you to share your local websites with others via the internet.

Combining event sourcing and stateful systems

As Brent stated: In this two-part series, my colleague Freek and I will discuss the architecture of a project we're working on. We will share our insights and answers to problems we encountered along the way. This part will be about the design of the system, while Freek's part will look at the concrete implementation.

Let's set the scene.

Does scrum ruin great engineers or are you doing it wrong?

A question on Stack Overflow’s Software Engineering site caught our attention recently. It tries to come to terms with the impact of scrum on developers' ability to do a great job. The claim is a bold one: Scrum is turning good developers into average ones. Could that be true?

Tropeçando 90

Auto-restart a crashed service in systemd

Systemd allows you to configure a service so that it automatically restarts in case it’s crashed.

My Personal Best Practices For Using LaunchDarkly Feature Flags

The Most-Neglected Postgres Feature?

log_line_prefix should be the most-neglected postgres feature. Overused and mis-configured. The author talk about his finding, the great use and some tips for log_line_prefix configuration. This feature is very powerful on PostgreSQL.

Logical Replication Between PostgreSQL and MongoDB

A decoder plugin to enable logical replication from a PostgreSQL (as publisher) to MongoDB (as subscriber).

Awesome-Compose: Application samples for project development kickoff

A curated list of Docker Compose samples.

These samples provide a starting point for how to integrate different services using a Compose file and to manage their deployment with Docker Compose.

PostgreSQL – pg_upgrade from 10 to 12

I have some PostgreSQL databases running pretty well but we need to keep our software updated. This is a mandatory practice for a high-quality service. Those servers are running version 10 and they need to be upgraded to version 12. I have used pg_dump / pg_restore strategy for a long time, but this time I would rather use pg_upgrade.

Let's dive into how to do it.

Table of contents:

  1. Install
  2. InitDB
  3. Check upgrade consistency
  4. Set locale
  5. Upgrade
  6. Configurations

Install

The package postgresql12-server contains everything needed to run the server, but my databases use some extensions [1], then I will add postgresql12-devel and postgresql12-contrib to be able to compile and to install the extensions.

yum install postgresql12-server postgresql12-devel postgresql12-contrib

InitDB

After installation we need to setup new server with initdb:

~% /usr/pgsql-12/bin/postgresql-12-setup initdb
Initializing database … OK

Check upgrade consistency

We need to check compatibility. Turn to postgres user (su - postgres) and run the command:

~% /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin --new-bindir=/usr/pgsql-12/bin --old-datadir=/var/lib/pgsql/10/data --new-datadir=/var/lib/pgsql/12/data --check

Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               fatal

Your installation contains tables declared WITH OIDS, which is not supported
anymore. Consider removing the oid column using
    ALTER TABLE ... SET WITHOUT OIDS;
A list of tables with the problem is in the file:
    tables_with_oids.txt

Failure, exiting

As you may see, I got a fatal error, indicating that the upgrade is not possible. In my case, tables with OIDs are the culprit. In your case could be something else. In any case, we need to fix before upgrading.

I fixed tables removing OIDs on mentioned tables. And ran check again:

Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

*Clusters are compatible*

Yay!

Set locale

There is a tricky configuration that is not detected by pg_upgrade check but it is very important to me. I use C locale on my databases [2], then I need to perform an extra step. If this is your case, you may follow the same steps applying yours.

I need to stop postgresql10 and start postgresql12:

systemctl stop postgresql-10.service
systemctl start postgresql-12.service

Then I run locale change at my template1 then locale will be enabled when my database will be upgraded.

UPDATE pg_database SET datcollate='C', datctype='C' WHERE datname='template1';

And stop again: systemctl stop postgresql-12.service to be ready to upgrade.

Upgrade

Upgrade command is the same that we run before, without --check flag.

~% /usr/pgsql-12/bin/pg_upgrade --old-bindir=/usr/pgsql-10/bin --new-bindir=/usr/pgsql-12/bin --old-datadir=/var/lib/pgsql/10/data --new-datadir=/var/lib/pgsql/12/data

Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for tables WITH OIDS                               ok
Checking for invalid "sql_identifier" user columns          ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

Consider running analyze_new_cluster. Optional but nice to have.

vacuumdb: processing database "mydb": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)
vacuumdb: processing database "mydb": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)
vacuumdb: processing database "mydb": Generating default (full) optimizer statistics
vacuumdb: processing database "postgres": Generating default (full) optimizer statistics
vacuumdb: processing database "template1": Generating default (full) optimizer statistics

Done

Configurations

Before deleting your old cluster, remember to get some of your configurations.

mv /var/lib/pgsql/12/data/pg_hba.conf /var/lib/pgsql/12/data/pg_hba.conf.new
cp /var/lib/pgsql/10/data/pg_hba.conf /var/lib/pgsql/12/data/

I am not a big fan of writing directly to postgresql.conf file. Instead, I keep configuration files under version control and include the directory where those files are deployed. I treat them as code, then it becomes easier to maintain and manage.

Another advantage is that I don't have any mess about config file differences when a new version arises. I am automatically using new default configurations, my customized setting are loaded and I can quickly address any incompatibility caused by migration without touching the original conf file.

Let's go for it:

# Add settings for extensions here
include_dir = '/var/lib/pgsql/conf.d'   # include files ending in '.conf' from

Then you may delete your old cluster. 🙂

Links:

  1. Extensions: https://www.postgresql.org/docs/current/contrib.html
  2. Locale: https://www.postgresql.org/docs/current/locale.html