PHP Execution Stats

Sometimes you want to know how long a script took and how much memory it consumed.

Run Time and Memory used can also be useful if tracked overtime, for example in:

• cron/cli scripts
• web requests
• api requests

While tools such as New Relic (paid), DataDog (paid), NetData (opensource), Prometheus (opensource) could be used, sometimes a simpler local solution is all that is needed.

Here is a simple Trait to extend your classes with

<?php

namespace App\Traits;

trait Stats
{
    private $timer_start;
    private $timer_finish;

    public function statsTimerStart()
    {
        $this->timer_start = microtime(true);
    }

    public function statsTimerFinish()
    {
        $this->timer_finish = microtime(true);
    }

    public function statsTimerRuntime()
    {
        if (empty($this->timer_finish)) {
            $this->statsTimerFinish();
        }

        $runtime = $this->timer_finish - $this->timer_start;
        return gmdate('H:i:s', $runtime);
    }

    public function statsMemoryUsed()
    {
        $memory_used = memory_get_peak_usage(true);

        return $this->statsFormatBytes($memory_used);
    }

    public function statsFormatBytes(int $bytes, int $precision = 2)
    {
        $units = ['B', 'KB', 'MB', 'GB', 'TB', 'PB'];
        $unit_index = 0;

        while ($bytes > 1024) {
            $bytes /= 1024;
            $unit_index++;
        }
        return round($bytes, $precision) . $units[$unit_index];
    }

} 

Basic usage:

$this->statsTimerStart()

.. do stuff ..

log 'stuff processed in ' . $this->statsTimerRuntime() . ' using ' . $this->statsMemoryUsed();

github gist

-End of Document-

Thanks for reading

Slim PHP from the command line

Slim is a PHP micro framework that helps you quickly write simple yet powerful web applications and APIs.
https://www.slimframework.com/

Slim can also be used to run scripts or commands from the command line, also known as the cli.

While libraries such as symfony/console or even slim/console can be used, they can be rather cumbersome ie heavy when all you want to do is run a command and have access to your existing code and services in Slim.

The basic flow of a Slim API call could be envisioned as:
    uri -> route -> action -> use query params

From the command line, there is no uri.
But maybe we could take the cli arguments and map that to a uri, and thus a route.

Perhaps such as
> php cli.php /cli/dostuff

Hmm, /cli/dostuff sure looks like a uri, right?
That can be mapped a Slim route nicely.

How about
> php cli.php /cli/dostuff?verbose=1&dryrun=1

Calling 

$request->getQueryParams() 

should return
verbose = 1
dryrun = 1

But what if we use arguments in a more common cli fashion?
> php cli.php /cli/dostuff verbose=1 dryrun=1

Hmm, verbose and dryrun still look like query string params
Would be nice if calling 

$request->getQueryParams() 

would return
verbose = 1
dryrun = 1

But what if we use arguments in a more typical cli fashion?
> php cli.php /cli/dostuff -verbose -dryrun

Hmm, verbose and dryrun do not look like query string params, but what if they were mapped to a key, such as argv?
Would be nice if calling 

$request->getQueryParams() 

would return
argv = [-verbose, -dryrun]

Well, for those changes and more, create a new file, named something such as runrundorun.php or maybe just cli.php. Place the new file below public, in the project root, at the same level as public, src, vendor, etc.

In cli.php, we will map the command line arguments to a uri, so Slim can test against routes, and then bootstrap Slim as normal.

cli.php

<?php

if (PHP_SAPI != 'cli') {
    exit("CLI only");
}

if (empty($argv) || count($argv) < 2) {
    exit("Missing route for CLI");
}

// remove calling script
array_shift($argv);

// get route + params from 1st argument
$uri = array_shift($argv);

// group routes by /cli/
if (strpos($uri, '/cli/') !== 0) {
    // handle os shell quirks
    // windows git bash
    if (strpos($uri, 'C:/Program Files/Git/cli/') === 0) {
        $uri = str_replace('C:/Program Files/Git/cli/', '/cli/', $uri);
    } else {
        echo "uri: " . $uri . PHP_EOL;
        exit("CLI Route must start with /cli/");
    }
}

// get any more arguments
if (!empty($argv)) {
    $additional = '';
    foreach ($argv as $arg) {
        if (strpos($arg, '=') !== false) {
            // r=1 d=10
            $additional .= '&' . $arg;
        } else {
            // normal args -r 10, store as argv
            $additional .= '&argv[]=' . $arg;
        }
    }
    if (strpos($uri, '?') === false) {
        $uri .= '?';
    }
    $uri .= $additional;
}
// set uri based on cli arguments
$_SERVER['REQUEST_URI'] = $uri;

// normal Slim app, routes_cli_*, ActionCli
(require __DIR__ . '/cfg/bootstrap.php');
GitHub gist

So now just add a route for /cli/dostuff and call your code!

> php [/app/]cli.php /uri/mapped/to/route?param1=v1 argv1=v1 argv0 

Extra:

Now, if you are running cli commands, you probably do not want errors returned as html.
You can add a customer error handler and return just text or maybe something with a little structure, such as json.  See the example in the official Slim docs to add a ErrorHandler to return json.
https://www.slimframework.com/docs/v4/objects/application.html

-End of Document-

Thanks for reading

Cypress configuration and organization

Cypress is a next generation front end testing tool built for the modern web.

Instead of using an external listener, such as Selenium, which can be 'flaky' at times waiting for responses, Cypress has been built to run in the browser so it can more accurately monitor and react to requests.  More information can be found at Cypress.io 

The following will install Cypress, give some configuration and organization recommendations, and show how to persist sessions.  Writing the tests is up to you! (Write your first test)

Install

In the base of your application, make a tests directory, and a cypress directory, as you may have or end up using other test suites

> mkdir tests/cypress

> cd tests/cypress

Simple npm install

> npm install cypress --save-dev

run Cypress

> npx cypress open

Official docs Install Cypress

Organization

While the install of Cypress creates an example skeleton directory 

test/cypress/cypress

It is recommended to create your own directory 'just in case' a npm update decides to do 'something' with those skeleton directories.

create an app or unique name under 

tests/cypress/[app_abbrev]

You can copy from 

tests/cypress/cypress

or create the directories:

• integration 

where you write and run tests  

• fixtures 

static data for testing

• plugins 

enable you to modify or extend Cypress

• screenshots 

if taken, storage

• support

loaded automatically before your test files ie global tests configuration

• commands

additional grouping of common executed test steps

• callbacks

additional grouping of callbacks which can modify the behaviors of tests

Configuration

Under 

tests/cypress

create three config files,

cypress.json

cypress.env.json

cypress.env.json.example

cypress.json contains global configuration related to Cypress

Add your [app_abbrev] location to the config:

{

    "fixturesFolder": "[app_abbrev]/fixtures",

    "integrationFolder": "[app_abbrev]/integration",

    "pluginsFile": "[app_abbrev]/plugins",

    "screenshotsFolder": "[app_abbrev]/screenshots",

    "supportFile": "[app_abbrev]/support"

}

cypress.env.json contains environment dependent configuration, such as user names for logins

You should create and maintain a 

cypress.env.json.example 

with the available config options too

An example config:

{

    "web_base_url": "http://app.dev.localhost:8088/",

    "login_username": "yourtestuser@dev.localhost",

    "login_password": "randchars"

}

Note, while Cypress does have a baseUrl config option that can be added to cypress.json, doing so does not allow the url to change per environment/developer/tester.  If you are using a defined centralized test environment, or defined containers, then this should not be an issue.  But to allow the url the app uses during testing to vary between environment/developer/tester, you can add and use your own base url by adding it to cypress.env.json

So instead of 

Cypress.config().baseUrl

You would use

Cypress.env("web_base_url")

Support

The index.js in support is called on every run of a test.  

This is where you can add global tests configuration and behaviors

Note, for easier maintenance, try to keep functionality to one file.

Update

tests/cypress/[app_abbrev]/support/index.js

to contain

import './commands/login_ui';

import './callbacks/stop_on_failure';

import './callbacks/preserve_session';

Support Commands

An example of a common executed test step may be to log in to your app.

Login UI

support/commands/login_ui.js

Create and add the minimal steps to log into your app, which may look similar to:

// https://on.cypress.io/custom-commands

Cypress.Commands.add("login_ui", (email, password) => {

    // minimal info to login via ui

    cy.visit(Cypress.env("web_base_url"));

    cy.url().should("include", "[app_abbrev]");

    let el = cy.get('#login_form [name="username"]');

    el.type(Cypress.env("login_username"));

    el = cy.get('#login_form [name="password"]');

    el.type(Cypress.env("login_password"));

    el = cy.get('#login_submit');

    el.click();

});

Note, instead of using your apps ui to log in for every test, you should create a token or api access to expedite the test

Now you can call the command using one consistent statement

describe("Select that Awesome Thing Test", () => {

    describe("Can Login", () => {

        it("Can login", () => {

            cy.login_ui(Cypress.env("login_username"), Cypress.env("login_password"));

        });

    });

});

Support Callbacks/Behaviors

Stop on Failure:

support/callbacks/stop_on_failure.js

When on step of a test fails, it will often cause the next steps to fail.  

So fail early so the problem can be found quicker.

Create and add

// after each test

// stop test after first failure

afterEach(function () {

    if (this.currentTest.state === 'failed') {

        Cypress.runner.stop()

    }

});

Preserve Sessions:

support/callbacks/preserve_session.js

All tests are supposed to be isolated, so Cypress will often clean up your cookies.

While it would be nice if the cleanup happens after the first test, or the last test in a suite, the clean up will happen after a few tests, which can log you out of your app and make it seem like your app or the test are broken.

To persists your session, which is often stored in a session cookie, create and add:

// once before all tests

// preserve session cookie so don't get 'randomly' logged out after several specs

before(function () {

    Cypress.Cookies.defaults({

        preserve: ['your_sessionid_name']

    });

});

Package.json

While you can run Cypress via

> npx cypress open

You can also add a more common alias to your package.json

  "scripts": {

    "test": "npx cypress open"

  },

And run Cypress via

> npm run test

.gitignore
Update your .gitignore
/tests/cypress/node_modules
/tests/cypress/cypress
/tests/cypress/cypress.env.json

Hopefully the above information helps you setup and use Cypress in a more enjoyable and useful fashion.

-End of Document-

Thanks for reading

AWS S3 Lifecycle delete rules

By putting a cleanup Lifecycle rule in place on your S3 buckets, you may be able to potentially save costs and increase LIST performance.

"Incomplete Multipart Uploads – S3’s multipart upload feature accelerates the uploading of large objects by allowing you to split them up into logical parts that can be uploaded in parallel.  If you initiate a multipart upload but never finish it, the in-progress upload occupies some storage space and will incur storage charges. However, these uploads are not visible when you list the contents of a bucket and (until today’s release) had to be explicitly removed.

Expired Object Delete Markers – S3’s versioning feature allows you to preserve, retrieve, and restore every version of every object stored in a versioned bucket. When you delete a versioned object, a delete marker is created. If all previous versions of the object subsequently expire, an expired object delete marker is left. These markers do not incur storage charges. However, removing unneeded delete markers can improve the performance of S3’s LIST operation."

Source: https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/

To add a cleanup Lifecycle rule:

• Log into the Amazon S3 web console

• Select your S3 bucket

• Select Management

• Select Add lifecycle rule

• Enter a name such as 

'Delete incomplete multipart upload and Delete previous versions'

• Skip Transitions for now

Transitions allow you to move storage to slower locations at a reduced cost

https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html

• Expiration

• Delete Previous versions after 365 days

You can choose shorter periods such as 7 days or 30 days if you don’’t have a use case for retrieving prior S3 versions.

You will still have the current version, which is usually all you want, but deleting previous versions can help with costs and S3 LIST performance.

• Clean up incomplete multipart uploads after 7 days

If you do not have any automated processes that may re-try uploads, you could choose 1 day

• Review

Agree to the 'scary' this applies to all objects in bucket

Note, if you have S3 objects (uploads) which require different policies, you may find it easier to manage by creating a S3 bucket per policy.

You now have some basic cleanup of your S3 bucket(s) configured.

-End of Document-

Thanks for reading

PHP code quality using Code Sniffer

about

PHP CodeSniffer is an essential development tool that ensures your code remains clean and consistent. It can also help prevent some common semantic errors made by developers.   It is a set of two PHP scripts; the main phpcs script that tokenizes PHP, JavaScript and CSS files to detect violations of a defined coding standard, and a second phpcbf script to automatically correct coding standard violations. 

https://github.com/squizlabs/PHP_CodeSniffer 

goal

More consistent indentation, spacing, and formatting.

So your team will have less conflicts from git, and a more consistent code base to maintain and grow.

install

install via composer

> php composer.phar require --dev squizlabs/php_codesniffer:3.*

or edit your composer.json and add

    "require-dev": {

        "squizlabs/php_codesniffer": "3.*"

    }

and then 

> composer install

configure

create a config file where you composer.json is located

phpcs.xml

example of PHP CodeSniffer's phpcs.xml

https://github.com/squizlabs/PHP_CodeSniffer/blob/master/phpcs.xml.dist

But that is overly verbose.  A simpler config using PSR12 as a base rule, plus adding in your app dirs, excluding some shared libs/dirs, and some rules exclusions due to your app might be:

https://www.php-fig.org/psr/psr-12/

<?xml version="1.0"?>

<ruleset name="PHP_CodeSniffer">

    <description>PHP Code Sniffer configuration file.</description>

    <!-- https://github.com/squizlabs/PHP_CodeSniffer -->

    <!-- check all these dirs/files -->

    <file>app</file>

    <file>bin</file>

    <file>cfg</file>

    <file>src</file>

    <!-- but don't check these -->

    <exclude-pattern>composer$</exclude-pattern>

    <exclude-pattern>public$</exclude-pattern>

    <!-- phpcs argument options -->

    <arg name="basepath" value="./"/>

    <arg name="colors"/>

    <arg name="tab-width" value="4"/>

    <arg name="extensions" value="php,js,css"/>

    <!-- how many files to check at once -->

    <arg name="parallel" value="10"/>

    <!-- base rule: set to PSR12-->

    <!-- https://www.php-fig.org/psr/psr-12/ -->

    <!-- https://github.com/squizlabs/PHP_CodeSniffer/wiki/Customisable-Sniff-Properties -->

    <rule ref="PSR12">

<!-- add any exclusions here -->

    </rule>

    <!-- Don't hide tokenizer exceptions -->

    <rule ref="Internal.Tokenizer.Exception">

        <type>error</type>

    </rule>

    <!-- require 4 spaces, css -->

    <rule ref="Squiz.CSS.Indentation">

        <properties>

            <property name="indent" value="4" />

        </properties>

    </rule>

    <!-- lines can be lineLimit chars long (warnings), errors at absoluteLineLimit chars -->

    <rule ref="Generic.Files.LineLength">

        <properties>

            <!-- 120 is PSR12; cannot be 0; large for sql, arrays -->

            <property name="lineLimit" value="360"/>

            <!-- 0 to not show as error -->

            <property name="absoluteLineLimit" value="0"/>

        </properties>

    </rule>

    <!-- ban some functions -->

    <rule ref="Generic.PHP.ForbiddenFunctions">

        <properties>

            <property name="forbiddenFunctions" type="array">

                <element key="sizeof" value="count"/>

                <element key="delete" value="unset"/>

                <element key="print" value="echo"/>

                <element key="is_null" value="null"/>

                <element key="create_function" value="null"/>

            </property>

        </properties>

    </rule>

</ruleset>

GitHub gist

Now setup PHP CodeSniffer for some ides

https://www.jetbrains.com/help/idea/using-php-code-sniffer.html

https://www.twilio.com/blog/set-up-php-codesniffer-local-development-sublime-text-php-storm-vs-code

results

A large amount of the corrections will probably be a mix of tabs vs spaces (use 4 spaces).

If auto correct a whole file, recheck it for unintended indentation (and fix), and functionality.

Only auto correct what you will verify and test ie not the whole app.

Commit auto corrects separately from fixes, so can see fixes in git diffs easier.

Your team should now have less conflicts from git, and a more consistent code base to maintain and grow.

-End of Document-

Thanks for reading

Git Feedback Branches

Most Git workflows do not address picking what is released. Once a feature branch has been merged back to 'development', it is in the release pipe, pending QA and Business validation. Once in 'development', code/branches cannot easily or arbitrarily be plucked out to go directly to production as the code is often intermingled with other branches. But the code changes in 'development' can be manually re-coded (Git patches help) into a hotfix branch from 'master'/'production' with the risk of not being QA-ed.

The 'Git single branch strategy' primarily removes the pain point of the conflicts between 'master' and 'development', while providing a clearer history of production releases, and an easier rollback with switching branches.

Generally picking what to release is largely mitigated by how, and the order in which Tickets are chosen. However, the release pipe can be slowed down by a Ticket/branch needing time to fix, or be validated by QA or Business. To remove the blocking branch, depending on the changes, the feature could be hidden, or removed if a small change, or more likely wait for the fix(es) and QA.
Hopefully the upfront choosing of Tickets and quality of specifications somewhat mitigates the blockers.

Some Git strategies to mitigate blockers in the release pipe, which are caused by Tickets that often require feedback once seen.

 1) Put less in the release pipe: (Less is more)

If limit releases to one branch per release, then there is no re-picking once in 'development'. Basically the other feature branches would queue up waiting to be picked and for merge to 'development' and QA-ed. Which leads to the pros and cons of Staging branches.

 2) Staging branches:

Another process to maybe help with picking branches for release is to not merge feature branches back to 'development' until picked. The feature branches could be deployed to their own directory (devsite.com/branch/123-shortdesc) be QA-ed, reviewed by Business, then if ok, merged to 'development'. Then when decided to go to production, everything currently in 'development' is QA-ed again, fixes added via branch updates or a new branch, and then released via 'master'/'production'.
Note, if after being merged to 'development', production release decisions change, well, then we are back to the same problem of doing hotfixes, or hiding not ready functionality, or waiting for the fixes.
Also this approach can be a burden on the developers: fixes enhancements, code cleanups, won't be seen or utilized until the branch is picked, and those fixes/enhancements might be required or desired for another branch, thus duplication of code which probably means conflicts later. Before merging the feature branch to 'development', 'development' would need to be merged back to the feature branch to handle any changes or conflicts in the branch, so the developer can test again before merging to 'development'. And as the code won't be fresh on the developers mind, there is a higher risk of mistakes to be made during the merge to 'development'.

pros:

• able to preview branches before release
• able to choose branches for release

cons:

• more work for Business and/or QA as the merged branches in ‘development’ still need to be reviewed
• if decisions change to remove a branch or hotfix a branch to production before qa, same problems
• more burden on developers, potential conflicts, developing the branch twice: once orig, then later (days, weeks) when picked
• dev-ops + some app work to make 'their own directory' happen

Committing often, merging often seems to be better for code quality.

3) Feedback branches, Preview branches: (A hybrid of Staging branches)
For branches which require Business or early QA feedback, after development is done, but before QA or merging to 'development', push the branch to a preview location (devsite.com/branch/123-shortdesc). There it can be previewed for one or two days, before being merged to 'development' and moved to QA; required feedback branches should not be held for a long time, else the cons of Staged branches may become apparent. The branches that require feedback should be marked as such before development. Every branch should not be marked as requires feedback, only a few should be.

pros:

• able to preview branches marked as feedback before release
• should reduce fixes needed when in 'development' for QA
• as only a day or two delay, no large time incurred burden on developers

cons:

• does not allow changing order of released branches
• more work for Business and/or QA as the merged branches in ‘development’ still need to be reviewed
• if decisions change to remove a branch or hotfix a branch to production before QA, same problems
• dev-ops work + some app work to make 'preview location' happen

Hopefully some useful Git strategies when dealing with Tickets that often require feedback once seen.

-End of Document-

Thanks for reading

Git single branch strategy

A common Git workflow has two main branches, 'development' and 'master'. With the common workflow of creating feature branches from ‘development’, merging the feature branch back to ‘development’, and then for a release to production, merging 'development' to 'master'. Non-trivial conflicts can occur during the merge to 'master' when same/similar changes are made in both 'master' and 'development', or 'development' has had some necessary reverts or other changes.

Instead of having two main or long lived branches, another idea for a Git workflow is to use only one main branch, then create a branch for production pulls.
aka 'trunk-based development workflow' or 'single branch strategy'

Think of the one main branch as the 'development' branch

• flow for development would be the same:
• branch from 'development' for a Ticket, merge request to 'development' when done
• when want to do a release to production, instead of a merge request to 'master'
• create a new 'production' branch at the last or desired 'development' commit
• when want to do another release to production
• rename current 'production' to 'production-date' ie 'production-20200531'
• create a new 'production' branch at the last or desired 'development' commit
• 'master' branch would then be unused and eventually removed
• current purpose of 'master' is: 
• release this code, which will be the purpose of 'production'
• 'development' is never merged into 'production'
• overtime, like feature branches, old 'production-date' branches can be deleted

pros:

• never any conflicts between 'master' eg 'production' and 'development', as only one main branch
• hotfixes can be applied to current 'production' branch without worry of later conflicts
• 'production-date' can be used as a rollback for code in case a release is non usable/broken and code related

cons:

• no 'master' branch; but 'production' and 'development' are more explicit
• no merge request to 'master'; 'production' and 'production-*' can be marked as a protected branches in GitLab
• slightly different

GitLab CI/CD:

• should still work at creation of 'production',
• and maybe the renaming of 'production-date' and creation of 'production' could be part of the GitLab CI/CD

Tags:
An alternative to creating 'production' branches would be to create 'production' tags. If a hotfix is needed, then create a branch from the tag. But just keeping everything a branch simplifies: GitLab CI/CD, Git UIs, merge request for hotfix, etc

And yes, the one branch to rule them all could be named the default Git repo branch name of ‘master’. Or 'sam', or whatever your group agrees upon, and you can tell others.

-End of Document-

Thanks for reading