CI/CD optimization and Pest 5

The problem

Recently we experienced huge waiting times in between new pull requests, merging to the main branch and releasing new versions, all because our tests were taking ~50 minutes to complete.

Now you might be thinking, alright you just have a lot of tests, or you just have slow machines that are running the tests. Well maybe, but we didn’t really want to jump to that conclusion straight off the bat, because who likes to increase their maintenance costs without first trying to fix things without “throwing money”.

Step 1: Update dependencies

First we decided to upgrade to Pest v5, since we were still running on Pest v3. Pest v5 introduces a lot of new cool features like the tia engine, time balanced sharding, agent plugin and browser testing. All of these features are really really good and I highly recommend that you use them in your PHP projects.

Step 2: Clean up Unit and Feature tests

Next off we used Drift to migrate our syntax from PHPUnit to Pest and Rector to clean up our code a bit. These 2 things weren’t really necessary but since we were changing a lot of code why not keep it up to standard as well.

Step 3: Refactoring E2E tests

We saved a lot of time with this step. Pretty much all of our E2E tests were doing a full login process and went to the dashboard after that, from where they navigated to the page where they needed to be using the sidebar navigation of our app. We fixed this by logging in once and keeping the session alive for every other test, and also removing the part where they manually navigate to the desired page. Oh and another thing. We had a lot of places where we used waitForTimeout which would hardcode a specific amount time that needs to be waited. This wasn’t really the best idea since some action could’ve already completed and we we were still waiting a few seconds for no reason.

Step 4: Parallel workers and compact logs

Since we are running our CI pipeline through GitHub workers, we have 2 cores which we can use, which lets us use the --parallel --processes=2 tag. This runs the tests in parallel which basically reduces the time it takes for the tests to finish by ~50%. Alongside this we started using --compact which made the logs much smaller (only logs failed tests) and this decreased the running time by a lot because they didn’t have to write out every single thing which usually isn’t necessary in CI.

Step 5: Building the Docker image once and reusing it

Since our E2E tests needed to build the Docker image first, and we had 3 scenarios for which they had to run (Chromium, Firefox and Pest) it didn’t make sense to wait for each of them to build an image first. So we created another step where the image is built first and then these scenarios pull that image we saved on billing time here more than we did on waiting time.

Results

With all of this we managed to reduce the total waiting time by ~50% and our billed minutes by ~25%. This is a great result for us because since then we are waiting for about 17 minutes in total per PR. And we even saved some money by reducing the billed time 🙂.

Metric	Old PR	New PR	Difference
Billed minutes	168 min	125 min	−43 min (−25.6%)
Raw summed runtime	157m 50s	112m 24s	−45m 26s (−28.8%)
End-to-end wait	37m 57s	18m 31s	−19m 26s (−51.2%)

What else we considered

We also considered sharding and it reduced our waiting time by ~50% but we decided not to go with it since it increased our billing time by ~60% and we didn’t really see the need for that since the optimizations that we made were already good enough for us. These are the results when using sharding in comparison with another PR.

Metric	Old PR	New PR	Difference
Executed jobs	24	43	+19 (+79.2%)
Billed minutes	142 min	208 min	+66 min (+46.5%)
Raw summed runtime	2h 11m 58s	3h 07m 46s	+55m 48s (+42.3%)
End-to-end wait	21m 48s	10m 53s	−10m 55s (−50.1%)

Another thing that we would like to start using is Pest browser testing which would basically remove the need for separate Playwright tests. This still uses Playwright underneath but the syntax is a lot cleaner and is one less thing to remember in the future. Example of the syntax:

Playwright:

test('may welcome the user', async ({ page }) => {
    await page.goto('/');

    await expect(page.getByText('Welcome')).toBeVisible();
});

Pest:

it('may welcome the user', function () {
    $page = visit('/');

    $page->assertSee('Welcome');
});

This is a really small example but I’m sure you can imagine how much it helps when you have a 500-line test and hundreds of them.

Final thoughts

These changes cut our end-to-end waiting time roughly in half while reducing our billed minutes by about 25%. The biggest takeaway was that faster CI does not always require more powerful runners—sometimes removing unnecessary work and making better use of the available resources is enough. But we are happy with what we’ve managed to achieve.