The Challenge of Long Test Execution Times
In test automation, reducing execution time without compromising results is crucial, especially as test suites grow in large projects with continuous testing needs. This can become a bottleneck when executing all tests.
This article explores how to leverage GitHub Actions and Playwright’s built-in sharding capabilities to run TypeScript tests in parallel, significantly reducing overall execution time.
I’ve shared the solution presented in this article in my Playwright Typescript example project.
Key Differences from Python Implementation:
- Native Sharding Support: Unlike the Python version that required pytest-split, Playwright for TypeScript has built-in sharding capabilities. This eliminates the need for additional libraries to split tests.
- Simplified Artifact Handling: Allure Playwright for TypeScript automatically includes artifacts in the Allure report, streamlining the process and reducing the need for manual artifact management.
Implementing the Solution
The solution code can be found here.
jobs:
setup-matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- id: set-matrix
run: |
count=${{ github.event.inputs.parallelism || 2 }}
matrix=$(seq -s ',' 1 $count)
echo "matrix=$(jq -cn --argjson groups "[$matrix]" '{group: $groups}')" >> $GITHUB_OUTPUTThe setup-matrix job dynamically creates a matrix based on the specified number of parallel executions, allowing flexible scaling of our test infrastructure.
nightly-test:
needs: setup-matrix
timeout-minutes: 15
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.46.0-jammy
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.setup-matrix.outputs.matrix) }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- name: Cache node_modules
uses: actions/cache@v4
id: cache-node-modules
with:
path: node_modules
key: modules-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Run Playwright tests
run: |
xvfb-run npx playwright test ${{ github.event.inputs.test_command || '--grep-invert "devRun"' }} --shard=${{ matrix.group }}/${{ github.event.inputs.parallelism || 2 }}
- name: Upload test results and artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results-${{ matrix.group }}
path: |
test-results/
allure-results
retention-days: 7
The nightly-test job is where the actual test execution occurs,
this job uses the dynamic matrix to run tests in parallel.
The fail-fast:
false setting in the matrix strategy prevents the entire job from failing as soon as one matrix configuration fails.
This means that all test shards will continue to execute, even if one or more fail.
This job uses a pre-configured Docker image that comes with Playwright and all necessary browsers pre-installed.
This significantly speeds up the test setup process
by eliminating the need to run the npx playwright install --with-deps
command,
which can be time-consuming, especially in CI environments.
Playwright’s built-in sharding capability is leveraged to split tests across multiple machines, dramatically reducing overall execution time. The sharding is configured dynamically based on the input parallelism. After test execution, we upload the Allure results, The uploaded artifacts set the stage for the merge-reports job, where we’ll consolidate results and determine the overall test suite status.
merge-reports:
needs: nightly-test
if: always()
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download all test results
uses: actions/download-artifact@v4
with:
path: artifacts
- name: Merge Allure Results
run: |
mkdir -p allure-results
for dir in artifacts/test-results-*/allure-results; do
cp -r $dir/* allure-results/
done
- name: Link Git Information And Browser Version To Allure Report
working-directory: allure-results
if: always()
run: |
git config --global --add safe.directory "$GITHUB_WORKSPACE"
{
echo BUILD_URL=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
echo GIT_BRANCH=${{ github.head_ref || github.ref_name }}
echo GIT_COMMIT_ID=${{ github.sha }}
echo GIT_COMMIT_MESSAGE="$(git show -s --format=%s HEAD)"
echo GIT_COMMIT_AUTHOR_NAME="$(git show -s --format='%ae' HEAD)"
echo GIT_COMMIT_TIME="$(git show -s --format=%ci HEAD)"
} >> environment.properties
- name: Generate Allure Report
uses: simple-elf/allure-report-action@master
if: always()
id: allure-report
with:
allure_results: allure-results
allure_report: allure-report
gh_pages: gh-pages
allure_history: allure-history
- name: Deploy Report To Github Pages
if: always()
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: allure-historyThis merge-reports job is designed to consolidate test results from multiple runs and generate a comprehensive Allure report.
It runs after the nightly-test job and is executed even if previous jobs fail.
The process begins by checking out the repository code and downloading all artifacts from previous jobs. It combines all Allure results into a single directory. To provide context, it adds important environment information to the Allure report, including Git details (like branch, commit ID, commit message, author, and time) and the Chrome browser version used for testing.
The job then generates the Allure report from these merged results.
Finally, the generated Allure report is deployed to GitHub Pages. This makes the report easily accessible to team members and stakeholders, allowing them to view the test results without needing to download or generate the report locally.
Conclusion
By leveraging GitHub Actions and Playwright’s native sharding capabilities, we’ve created an efficient parallel testing framework for TypeScript projects. This approach significantly reduces execution time, provides comprehensive reporting through Allure, and integrates seamlessly with GitHub Pages for easy access to test results.
The simplified artifact handling and built-in sharding make this TypeScript implementation more streamlined than its Python counterpart. It offers a powerful solution for teams looking to optimize their test automation workflows.
By continually refining your test automation strategy in this way, you can ensure faster feedback cycles and maintain high-quality software delivery. The result is a more responsive, efficient testing process that can keep pace with modern development needs.
Happy testing!