Introduction
When working with Playwright for automated testing, especially in CI/CD environments, generating structured test reports is crucial for debugging and maintaining test quality. Allure Reports provide a detailed overview of test executions. However, when tests are executed in parallel using the GitHub Actions matrix strategy, merging multiple Allure results into a single report can be cumbersome.
A recently published GitHub Action, Allure Results Combiner and Publisher, simplifies this process, allowing seamless merging of test results across matrix jobs and automatic publishing to GitHub Pages. In this article, I’ll discuss how I leveraged this GitHub Action to optimize my Playwright test reporting workflow. I’ve shared the solution presented in this article in my Playwright Typescript example project.
The Challenge: Merging Allure Reports in CI/CD
When running Playwright tests in GitHub Actions using a matrix strategy, each shard generates separate Allure result files. These must be combined to create a unified HTML report. Previously, I manually implemented steps to:
- Download Allure results from multiple jobs.
- Merge them using bash scripts.
- Generate a final report.
- Deploy the report to GitHub Pages.
This approach, while functional, was cumbersome and error-prone.
What I Had to Do Before
Previously, my Nightly Regression Test Workflow looked like this:
name: Nightly regression tests
on:
schedule:
- cron: '0 0 * * *'
workflow_dispatch:
inputs:
test_command:
description: 'Custom test command'
required: true
default: '--grep-invert "@devRun"'
type: string
parallelism:
description: 'Number of machines to split tests'
required: false
default: 2
type: number
jobs:
setup-matrix:
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- id: set-matrix
run: |
count=${{ github.event.inputs.parallelism || 2 }}
matrix=$(seq -s ',' 1 $count)
echo "matrix=$(jq -cn --argjson groups "[${matrix}]" '{group: $groups}')" >> $GITHUB_OUTPUT
nightly-test:
needs: setup-matrix
if: always()
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/playwright:v1.50.1-jammy
env:
BASE_URL: ${{ vars.BASE_URL }}
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.setup-matrix.outputs.matrix) }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- name: Cache node_modules
uses: actions/cache@v4
id: cache-node-modules
with:
path: node_modules
key: modules-${{ hashFiles('package-lock.json') }}
- name: Install dependencies
if: steps.cache-node-modules.outputs.cache-hit != 'true'
run: npm ci
- name: Run Playwright tests
run: |
npx playwright test ${{ github.event.inputs.test_command || '--grep-invert "@devRun"' }} --shard=${{ matrix.group }}/${{ github.event.inputs.parallelism || 2 }}
- name: Upload test results and artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results-${{ matrix.group }}
path: |
test-results/
allure-results
retention-days: 7
merge-reports:
needs: nightly-test
if: always()
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download all test results
uses: actions/download-artifact@v4
with:
path: artifacts
- name: Merge Allure Results
run: |
mkdir -p allure-results
for dir in artifacts/test-results-*/allure-results; do
cp -r $dir/* allure-results/
done
- name: Link Git Information And Browser Version To Allure Report
working-directory: allure-results
run: |
git config --global --add safe.directory "$GITHUB_WORKSPACE"
{
echo BUILD_URL=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
echo GIT_BRANCH=${{ github.head_ref || github.ref_name }}
echo GIT_COMMIT_ID=$(git rev-parse HEAD)
echo GIT_COMMIT_MESSAGE="$(git show -s --format=%s HEAD)"
echo GIT_COMMIT_AUTHOR_NAME="$(git show -s --format='%ae' HEAD)"
echo GIT_COMMIT_TIME="$(git show -s --format=%ci HEAD)"
} >> environment.properties
- name: Generate Allure Report
uses: simple-elf/allure-report-action@master
if: always()
id: allure-report
with:
allure_results: allure-results
allure_report: allure-report
gh_pages: gh-pages
allure_history: allure-history
- name: Deploy Report To GitHub Pages
if: always()
uses: peaceiris/actions-gh-pages@v4
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: allure-history
While this worked, it required multiple manual steps and redundant scripting.
How the New Approach Simplified My Workflow
The solution code can be found here.
By integrating Allure Results Combiner and Publisher, I was able to:
- Eliminate the need for separate , , and steps.
- Automate Allure report generation in one step.
- Automatically publish test history to GitHub Pages.
- Enhance debugging with linked commit and environment details.
merge-reports:
needs: nightly-test
if: always()
runs-on: ubuntu-latest
steps:
- name: Merge and Publish Allure Report
uses: Valiantsin2021/allure-shard-results-publish@1.0.6
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
add-env: 'true'This significantly reduced the complexity and maintenance overhead and improved the reliability of test reporting.
Conclusion
The Allure Results Combiner and Publisher GitHub Action is a game-changer for Playwright test automation. It simplifies test reporting, improves debugging efficiency, and ensures a structured and accessible historical record of test executions.
If you’re using Playwright with GitHub Actions, I highly recommend incorporating this action into your workflow to enhance your reporting capabilities.
Happy testing!