Azure release pipeline to Publish Test Results without rerunning tests - Stack Overflow

admin管理员组

文章数量:1289496

I'm trying to create a release pipeline that allows me to automatically update the test results in a Test plan. I run the tests through nunit-console-runner in a build pipeline. I have already successfully built a build pipeline that runs tests and reads off results. This is the YAML for that pipeline:

pool:
  name: YourAgentPoolName
  demands:
  - msbuild
  - visualstudio
  - vstest

variables:
  BuildConfiguration: 'Release'
  BuildPlatform: 'any cpu'

steps:
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk 7.0.x'
  inputs:
    version: 7.0.x

- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: NuGetCommand@2
  displayName: 'NuGet restore'

- task: VSBuild@1
  displayName: 'Build solution **\*.sln'
  inputs:
    msbuildArgs: '/p:Configuration=$(BuildConfiguration) /p:Platform="$(BuildPlatform)" /p:OutputPath=$(Build.ArtifactStagingDirectory)'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    clean: true

- script: |
   set PATH=%PATH%;C:\ProgramData\chocolatey\bin
         choco install nunit-console-runner -y
  displayName: 'Install nunit-console-runner'

- script: |
       if exist "C:\Program Files (x86)\NUnit\nunit-console\nunit3-console.exe" (
         echo ✅ NUnit is installed correctly.
       ) else (
         echo ❌ NUnit is NOT installed! Check the installation.
         exit 1
       )
  displayName: 'Verify NUnit Installation'

- script: |
   echo "Checking for test DLL in build output..."
   dir $(Build.ArtifactStagingDirectory) /s /b | findstr "YourTestProject.dll"
  displayName: 'Verify DLL Location'

- script: |
   cd "C:\Program Files (x86)\NUnit\nunit-console"
   nunit3-console.exe $(Build.ArtifactStagingDirectory)\YourTestProject.dll --where "$(FeaturesToTest)"
  displayName: 'Run NUnit Tests'
  continueOnError: true

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: '**/TestResult.xml'
    searchFolder: '$(Build.SourcesDirectory)'
    mergeTestResults: true

- task: PublishPipelineArtifact@1
  displayName: 'Publish Pipeline Artifact'
  inputs:
    targetPath: '$(Build.SourcesDirectory)'
    artifact: XmlLogsArtifact

I want to make a release pipeline that makes it where you can go to a Test Plan, click on the tests you want to update the status of, click

Execute > Run With Options > Automated tests using release stage

to update the results. This is the YAML for each task in my release pipeline so far.

steps:
- task: DownloadPipelineArtifact@2
  displayName: 'Download Test Artifacts'
  inputs:
    artifactName: XmlLogsArtifact
    targetPath: '$(System.ArtifactsDirectory)'
steps:
- task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell.Xpirit-Vsts-Build-InlinePowershell.InlinePowershell@1
  displayName: 'What are certain things'
  inputs:
    Script: |
     Write-Host "System.ArtifactsDirectory: $(System.ArtifactsDirectory)"
     Write-Host "Agent.TempDirectory: $(Agent.TempDirectory)"
     Write-Host "System.DefaultWorkingDiretory: $(System.DefaultWorkingDirectory)"
     Get-ChildItem -Path "$(System.ArtifactsDirectory)" -Recurse
steps:
- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Visual Studio Test Platform Installer'
steps:
- task: VSTest@2
  displayName: 'VsTest - testAssemblies'
  inputs:
    testAssemblyVer2: '**/Trintech.Tests.Cadency.dll'
    searchFolder: '$(Build.ArtifactStagingDirectory)'
    resultsFolder: TestResult.xml
    runOnlyImpactedTests: false
    runTestsInIsolation: true
  continueOnError: true
steps:
- task: PublishTestResults@2
  displayName: 'Publish Test Results **/TestResult.xml'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: '**/TestResult.xml'
    searchFolder: '$(System.ArtifactsDirectory)'
    mergeTestResults: true
    failTaskOnMissingResultsFile: true

I've had the release pipeline deploy successfully, but it will fail at the "Validating stage" when it runs. The error it currently gives is "The selected stage does not have the right version or settings of the Visual Studio Test task to run tests."

Do I need the VsTest task at all? I thought no, because I don't want to rerun the same tests twice. But the only time I've had this work was when I was doing that. When I remove that task, I get the same error. Please help.

I'm trying to create a release pipeline that allows me to automatically update the test results in a Test plan. I run the tests through nunit-console-runner in a build pipeline. I have already successfully built a build pipeline that runs tests and reads off results. This is the YAML for that pipeline:

pool:
  name: YourAgentPoolName
  demands:
  - msbuild
  - visualstudio
  - vstest

variables:
  BuildConfiguration: 'Release'
  BuildPlatform: 'any cpu'

steps:
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk 7.0.x'
  inputs:
    version: 7.0.x

- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: NuGetCommand@2
  displayName: 'NuGet restore'

- task: VSBuild@1
  displayName: 'Build solution **\*.sln'
  inputs:
    msbuildArgs: '/p:Configuration=$(BuildConfiguration) /p:Platform="$(BuildPlatform)" /p:OutputPath=$(Build.ArtifactStagingDirectory)'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    clean: true

- script: |
   set PATH=%PATH%;C:\ProgramData\chocolatey\bin
         choco install nunit-console-runner -y
  displayName: 'Install nunit-console-runner'

- script: |
       if exist "C:\Program Files (x86)\NUnit.\nunit-console\nunit3-console.exe" (
         echo ✅ NUnit is installed correctly.
       ) else (
         echo ❌ NUnit is NOT installed! Check the installation.
         exit 1
       )
  displayName: 'Verify NUnit Installation'

- script: |
   echo "Checking for test DLL in build output..."
   dir $(Build.ArtifactStagingDirectory) /s /b | findstr "YourTestProject.dll"
  displayName: 'Verify DLL Location'

- script: |
   cd "C:\Program Files (x86)\NUnit.\nunit-console"
   nunit3-console.exe $(Build.ArtifactStagingDirectory)\YourTestProject.dll --where "$(FeaturesToTest)"
  displayName: 'Run NUnit Tests'
  continueOnError: true

- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: '**/TestResult.xml'
    searchFolder: '$(Build.SourcesDirectory)'
    mergeTestResults: true

- task: PublishPipelineArtifact@1
  displayName: 'Publish Pipeline Artifact'
  inputs:
    targetPath: '$(Build.SourcesDirectory)'
    artifact: XmlLogsArtifact

I want to make a release pipeline that makes it where you can go to a Test Plan, click on the tests you want to update the status of, click

Execute > Run With Options > Automated tests using release stage

to update the results. This is the YAML for each task in my release pipeline so far.

steps:
- task: DownloadPipelineArtifact@2
  displayName: 'Download Test Artifacts'
  inputs:
    artifactName: XmlLogsArtifact
    targetPath: '$(System.ArtifactsDirectory)'
steps:
- task: petergroenewegen.PeterGroenewegen-Xpirit-Vsts-Build-InlinePowershell.Xpirit-Vsts-Build-InlinePowershell.InlinePowershell@1
  displayName: 'What are certain things'
  inputs:
    Script: |
     Write-Host "System.ArtifactsDirectory: $(System.ArtifactsDirectory)"
     Write-Host "Agent.TempDirectory: $(Agent.TempDirectory)"
     Write-Host "System.DefaultWorkingDiretory: $(System.DefaultWorkingDirectory)"
     Get-ChildItem -Path "$(System.ArtifactsDirectory)" -Recurse
steps:
- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Visual Studio Test Platform Installer'
steps:
- task: VSTest@2
  displayName: 'VsTest - testAssemblies'
  inputs:
    testAssemblyVer2: '**/Trintech.Tests.Cadency.dll'
    searchFolder: '$(Build.ArtifactStagingDirectory)'
    resultsFolder: TestResult.xml
    runOnlyImpactedTests: false
    runTestsInIsolation: true
  continueOnError: true
steps:
- task: PublishTestResults@2
  displayName: 'Publish Test Results **/TestResult.xml'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: '**/TestResult.xml'
    searchFolder: '$(System.ArtifactsDirectory)'
    mergeTestResults: true
    failTaskOnMissingResultsFile: true

I've had the release pipeline deploy successfully, but it will fail at the "Validating stage" when it runs. The error it currently gives is "The selected stage does not have the right version or settings of the Visual Studio Test task to run tests."

Do I need the VsTest task at all? I thought no, because I don't want to rerun the same tests twice. But the only time I've had this work was when I was doing that. When I remove that task, I get the same error. Please help.

• azure
• azure-pipelines
• azure-pipelines-release-pipeline

Share Improve this question Follow asked Feb 19 at 21:24 ColeCole 3522 silver badges88 bronze badges 3

• Hi @Cole, Good day. Have you got a chance to check the suggested options to run VSTest in my answer below? Hope it can resolve your concerns in this post. – Alvin Zhao - MSFT Commented Feb 25 at 2:17
• @AlvinZhao-MSFT Hey Alvin, sorry it took so long to respond. Conceptually, this makes sense. I've had the release pipeline work running tests seperate from the build pipeline before, so no reason I can see why this shouldn't work. Trying to do it this way, with VsTest only in the release pipeline. Issue right now is that it for some reason isn't able to find any of the tests, whether I set it to Test Assemblies, Test Run, or Test Plan. This could be completely unrelated. I just haven't figured out why it doesn't work yet. – Cole Commented Mar 3 at 20:52
• What was the error and what is your current workflow look like? How did your build pipeline publish the test assemblies and did your release pipeline fail to find the test assemblies in the release pipeline? Which path were those test assemblies downloaded into? – Alvin Zhao - MSFT Commented Mar 4 at 1:25

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

If your expectation is to Run Automated tests from test plans, then Yes, you need to use the VSTest task in the environment(stage) of your release pipeline.

The functionality to Run with options of Automated tests using release stage will validates the selected stage to ensure the Visual Studio Test task is present and has valid settings. The property testSelector: testRun of the VSTest is one of those required settings.

As another prerequisite, you also need to Associate automated tests with test cases, so that the VSTest task can first acknowledge the test case(s) in the test plan that you intend to run and then find the corresponding associated build assembly/assemblies downloaded as pipeline artifacts into searchFolder.

Regarding your current workflow, if possible, you may consider moving the steps to run tests from the build pipeline into the VSTest step of the release pipeline to avoid duplicate tests.

Please note that the above feature is still a manually triggered test workflow using release stage. To automatically trigger the release pipeline to pickup tests in your test plan, you can use the 3rd testSelector: testPlan, which also requires you to associate test assemblies with test cases in the first place. This release also populates the execution history of each test case in the test plan, whose associated assembly is tested and can be triggered automatically with CD trigger enabled.

本文标签： Azure release pipeline to Publish Test Results without rerunning testsStack Overflow