playwright - How to make sure one suite results don't overwrite the other ones? - Stack Overflow

admin管理员组

文章数量:1122846

I have a couple batch files which kick off certain sets of tests at different times throughout the day:

1. Full regression (over 1000 tests) - kicks off at 9PM
   • npx playwright test --grep "@daily" --reporter=junit
2. Smoke Tests - which kick off at 7AM and run every hour on the hour
   • npx playwright test --grep "@smoke" --reporter=".\tests\Logging\splunkreporter.ts"

With over 1000 tests full regression, using playwright's --last-failed flag is a huge help, due to test flakiness.

The problem is, once the smoke tests kick off at 7AM, that overwrites everything in the test-results directory including the .last-run.json file, the results file, and all the folders with video, etc... And therefore the results from the full regression are gone.

I hadn't considered this problem until both were finally running; especially since the smoke tests use a custom reporter which publish the test results to a splunk server.

Is there a good way to disable overwriting previous run logs in our test-results directory when using a customer reporter? Or does anyone else have any other recommendations to handle this situation?

I have a couple batch files which kick off certain sets of tests at different times throughout the day:

1. Full regression (over 1000 tests) - kicks off at 9PM
   • npx playwright test --grep "@daily" --reporter=junit
2. Smoke Tests - which kick off at 7AM and run every hour on the hour
   • npx playwright test --grep "@smoke" --reporter=".\tests\Logging\splunkreporter.ts"

With over 1000 tests full regression, using playwright's --last-failed flag is a huge help, due to test flakiness.

The problem is, once the smoke tests kick off at 7AM, that overwrites everything in the test-results directory including the .last-run.json file, the results file, and all the folders with video, etc... And therefore the results from the full regression are gone.

I hadn't considered this problem until both were finally running; especially since the smoke tests use a custom reporter which publish the test results to a splunk server.

Is there a good way to disable overwriting previous run logs in our test-results directory when using a customer reporter? Or does anyone else have any other recommendations to handle this situation?

• playwright
• playwright-test

Share Improve this question Follow edited Nov 23, 2024 at 14:59 Vishal Aggarwal 4,06722 gold badges2020 silver badges3434 bronze badges asked Nov 22, 2024 at 12:59 Andy GiebelAndy Giebel 28133 silver badges1818 bronze badges 1

• There is nothing wrong with this question. Please don't suggest to close without any strong reasons.We are users community , we should help each other. – Vishal Aggarwal Commented Nov 23, 2024 at 15:01

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 4

I had a similar situation where we were smoke testing multiple web sites and wanted to group the results together. I ended up just redirecting the results to a different folder that was unique for each web site. I did this by setting the environment variable PLAYWRIGHT_HTML_REPORT (which sets the output directory for the HTML reporter), before each web sites had its tests run.

https://playwright.dev/docs/test-reporters#junit-reporter shows a list of all the environment variables you can set/change for jUnit. Some of them deal with output.

Configure Suite Results to Save in Different Folders

We can configure our test results to be saved in different folders for each suite using the outputDir property in your test configuration under different project configurations. Here's how we can do it:

import { defineConfig} from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'suite1',
      testDir: './tests/suite1',
      outputDir: './results/suite1', // Results for suite1
    },
    {
      name: 'suite2',
      testDir: './tests/suite2',
      outputDir: './results/suite2', // Results for suite2
    },
  ]
 
});

and can run a specific suite like below:

npx playwright test --project=suite1

本文标签： playwrightHow to make sure one suite results don39t overwrite the other onesStack Overflow