Jenkins
Space shortcuts
Child pages
  • VsTestRunner Plugin
Skip to end of metadata
Go to start of metadata

Plugin Information

View VSTest Runner on the plugin site for more information.

This plugin allow you to execute test using VsTest command line tool.

Description

This plugin allow you to execute test using VsTest command line tool.
VSTest are test classes for .NET framework which are integrated into Visual Studio.

Configuration

System configuration

  1. Open the system configuration page "Manage Jenkins→Global Tool Configuration"
  2. Enter the path to the VsTest command line client, that should be used by Jenkins.
  • The path location is as instance: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe.

Job configuration

name

description

VsTest Version

Select VsTest Version.

Test Files

Specify the path to your VsTest compiled assemblies.
You can specify multiple test assemblies by separating them with new-line or space.

Settings File

Run tests with additional settings such as data collectors.

Test Name

Run tests with names that match the provided values.
To provide multiple values, separate them by commas.

Test Case Filter

Run tests that match the given expression.
<Expression> is of the format <property>=<value>VsTestRunner Plugin.

Enable Code Coverage

Enables data diagnostic adapter CodeCoverage in the test run.

InIsolation

Runs the tests in an isolated process.

Use Vsix Extensions

This makes vstest.console.exe process use or skip the VSIX extensions installed (if any) in the test run.

Use VS 2017 or LaterThis makes changes to the arguments passed to the VS Test Agent in order to ensure compatibility with the latest version.

Platform

Target platform architecture to be used for test execution.

.NET Framework Version

Target .NET Framework version to be used for test execution.

Specify a logger for test results.

Specify a logger for test results. For example, to log results into a Visual Studio Test Results File (TRX) use /Logger:trx.

Command Line Arguments

This is a whitespace separated list of command line arguments you can specify. These can be the same as if you were to run VsTest from the command line.

Fail build on test failure

Fail build on test failure

Changelog

Version 1.0.5 (10/24/2017)
  • To be updated with description of changes and links to the resolved issues.
Version 1.0.4 (08/31/2014)
Version 1.0.3 (10/21/2013)
Version 1.0.2 (07/24/2013)
  • Supports Jenkins variable.
    I can be used in the following items.
    • Test Files
    • Settings File
    • Test Name
    • Test Case Filter
    • Command Line Arguments
  • I Fixed a bug that was not used by the "Command Line Arguments".
Version 1.0.1 (03/09/2013)

Mod default value.

  • Plaform Field: None
  • .NET Framework Version Field: None
  • Specify a logger for test results. Field: trx
Version 1.0.0 (01/29/2013)

Initial release.

34 Comments

  1. Cedric Bertolasio

    Is there a way to use wildcards in the test file name?

    eg "DataCenter\\SolutionItems\\UnitTests\\**
    *Tests.dll"

    or do i need to know the exact path to every unit test dll?

    1. Calvin Kong

      There is a bug which was fixed but not released.

      1. Anne Moroney

        Is there going to be a new release? It seems like there are quite a few commits since last year.
        Without a new release, one could still compile the source and put it in the plugsins folder, right?
        thank you!

        1. Erik Goughnour

          1. There is a new release (but please wait for 1.0.6).
          2. I plan on increasing the cadence of releases.
          3. Yes you can use any valid .HPI that compiles, though I wouldn't recommend it in every case.

  2. Qa Automation Team

    After a test run, Is there a way to access the total of run tests, the number of failed and number of passed?
    I am looking for this in order to add to a notification e-mail.

    For instance: $BUILD_NUMBER $BUILD_STATUS. Total tests: $VSTEST_TOTAL. Passed: $VSTEST_PASSED. Failed: $VSTEST_FAILED

    Is there any way to do this?

    Thanks,

    QA Automation Team

  3. Ana Franco

    Hi, Im trying to access the environment variable VSTEST_RESULT_TRX to check the name of the generated trx file to publish using the MS Test Plugin but its not working, I try to do an echo on the variable and checked the jenkins environment variables used and I can't find it.

    Do I need to install something else? the new trx files are being created its just the variables that don't seem to work.

    Thanks

     

  4. Kevin Tighe

    Hi,

    The 1.05 release broke this plugin for me (using Jenkins 2.86).  I can no longer see the vstest tool configuration section, and the build steps have disappeared from my builds.

    1. Joao Miranda

      In the LTS version too (2.73)

    2. Erik Goughnour

      Thanks.
      Working on 1.0.6.

    3. John Chen

      Also broke for us

    4. Mukul Garg

      1.0.5 also broke for me. So i downgraded to 1.0.4

  5. Erik Goughnour

    1. Thanks for the feedback.  If I don't know it's broken then I can't fix it.
    2. For now please revert to the previous release.
    3. I am working on the next release currently.  

      Please make sure I am aware of your current issues–particularly the use case. I will try to test out any scenarios that are failing–but I will have to know exactly what is failing in order to do so.

    By the way I am hoping to have pipeline and block support.

    I'd like to be able to eliminate most of the pain of configuring the plugin (and maintaining related pipeline scripts) by simplifying the groovy syntax and making most or all of the necessary following steps selectable via options (or at least callable within the block).  

     

  6. Mukul Garg

    Hi,

    Can i use wild card character in Test Files?

  7. David Schob

    Looks to me like the latest version of Visual Studio 2017 comes with a vstest.console.exe that now enforces case sensitivity on its command line arguments.  Example: vstest runner passes "framework45."  Up until a recent version of VS2017, this was fine.  Now it complains with:

    "Invalid .Net Framework version:framework45. Please give the fullname of the TargetFramework. Other supported .Net Framework versions are Framework35, Framework40, Framework45 and FrameworkCore10."

    My work around:

  8. Jens Schumacher

    Hi.

    I just updated to 1.0.7 from 1.0.4 and now the test runner now insists it can't find my test files.

    "no files matching the pattern"

    One of the file paths is %WORKSPACE%\125-20\Common\UnitTest\bin\x86\Debug\Unitek.UniLock.Common.CommonClassLib.Tests.dll

    I have attempted without %WORKSPACE% and with the aboslute path to the file. none of it works in 1.0.7
    Downgrading to 1.0.4 and it works fine.

    1. ben bourner

      Confirmed, I have the same problem, it seems the latest version 1.0.7 cannot expand environment variables? Downgrading to 1.0.4 resolves the issue. Apparently author is not bothered that he's broken things, judging by lack of response to you or anybody below.

  9. Jose Fariña

    Hello,


    Seems that the version 1.0.7 fixed the issue that makes that you cant see the plugin in the build steps. But it has a new one, everytime I restart the jenkins, the vstest.console installation folder got reseted, so i need to add a new installation folder.


    Regards,
    José

    1. Jens Schumacher

      I have that problem too. Every reboot and the folder is reset and jenkins can't find vstest.... argh...

    2. Vincent Ooi

      Encountered same issue, restart jenkins, vstest configurations reset to default.

    3. Richard Landfair

      The same problem exists for me. The Global Tool Configuration settings use the defaults after restarting Jenkins.

  10. Scott P

    1.0.7 puts a PREFIX SLASH in front of the test files. Please see below. Because of this prefix SLASH, it can no longer find the test files. HELP. I will see if I can downgrade. 1.05 worked well.

    BLAH. The slashes can't be placed in this textbox....

    PLEASE remove the SLASH from the front of the code files. I am also using the D drive, instead of the C drive, so I am not sure if thats an issue here or not.

     

    1. Mario Giammarco

      Hello,

      have you found a solution?

      IT is absolutely blocking for me!

      Thanks,

      Mario

  11. Vincent Ooi

    Using 1.0.7, when configure VSTest, requires to put the directory in double quote "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
    Else, it will have error like below, Console Error:

    ****************************************************

    Path To VSTest.console.exe: C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
Executing VSTest: cmd.exe /C C:\Windows\TEMP\vstest2560868472995098884.bat && exit %ERRORLEVEL%
[DeviceProgrammedParameter] $ cmd.exe /C C:\Windows\TEMP\vstest2560868472995098884.bat && exit %ERRORLEVEL%

D:\JenkinsWorkspace\DeviceProgrammedParameter>C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "MY.Test/bin/Debug/My.Test.dll" /Enablecodecoverage /UseVsixExtensions:false /Logger:trx 
'C:\Program' is not recognized as an internal or external command,
operable program or batch file.

    ****************************************************

    1. Mario Giammarco

      I have the same problem, solved with same workaround. Version 1.0.7

  12. Thomas Klammer

    Updated to 1.0.7 from 1.0.4: "no files matching the pattern"

    Regardless what I'm doing I'm always getting this error message

    1. Thomas Klammer

      You aren't really interested in solving this problem, aren't you?

      1. ben bourner

        I also have this problem. Author does not seem bothered. Downgrading to 1.0.4 solves it. Stuff like this makes me want to move away from Jenkins - it's impossible to update any plugin without risking everything breaking.

  13. Mario Giammarco

    I have opened a ticket in jenkins google group https://groups.google.com/forum/?nomobile=true#!searchin/jenkinsci-users/vstestrunner%7Csort:date/jenkinsci-users/5g7Hxqwk61k/z3LO9sF_CQAJ

     

    Please go there and reply to show that there is interest for the plugin.

  14. Joao Chagas

    Hello guys, I need help as well... In my tests show this mensage:

     

     

    Warning: Using Isolation mode to run tests as required by effective Platform:X64 and .Net Framework:Framework45 settings for test run. Use the /inIsolation parameter to suppress this warning.

    Starting test discovery, please wait...

    Warning: A total of 7 tests were discovered but no test matches the specified selection criteria(CriarPAC). Use right value(s) and try again.

    Path To VSTest.console.exe: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe

    Executing VSTest: cmd.exe /C C:\Users\ADMINI~1\AppData\Local\Temp\vstest4061933848113271387.bat && exit %ERRORLEVEL%

    [Teste Web] $ cmd.exe /C C:\Users\ADMINI~1\AppData\Local\Temp\vstest4061933848113271387.bat && exit %ERRORLEVEL%

     

    I'm used the plugin version 1.0.4, Windowns 10, Selenium with Visual Studio

  15. Debbie Wright

    We're having the same issue as "Thomas Klammer" "no files matching the pattern" "No test source files were specified."

    using version 1.0.7 of this plugin, Vstest console 2017, .net framework 4.7.1

    I've tried multiple version of the file pattern as seen below

    17:27:27 Path To VSTest.console.exe: "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
    17:27:27 no files matching the pattern %WORKSPACE%\src\UnitTests\obj\Release\UnitTests.dll
    17:27:27 ${WORKSPACE}\src\iNetControlTests\obj\Release\iNetControlTests.dll
    17:27:27 "%WORKSPACE%\src\UnitTests\obj\Release\UnitTests.dll"
    17:27:27 "${WORKSPACE}\src\iNetControlTests\obj\Release\iNetControlTests.dll"
    17:27:27 C:\Jenkins\workspace\TestWindows\src\DomainModelTests\obj\Release\DomainModelTests.dll
    17:27:27 "C:\Jenkins\workspace\TestWindows\src\iNetAdminConsoleTests\obj\Release\iNetAdminConsoleTests.dll"
    17:27:27 Executing VSTest: cmd.exe /C C:\Windows\TEMP\vstest2305357646525754546.bat && exit %ERRORLEVEL%
    17:27:27 [TestWindows] $ cmd.exe /C C:\Windows\TEMP\vstest2305357646525754546.bat && exit %ERRORLEVEL%
    17:27:27
    17:27:27 C:\Jenkins\workspace\TestWindows>"C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /Settings:"C:\Jenkins\workspace\TestWindows\src\CodeCoverage.runsettings" /Platform:x64 /Logger:trx
    17:27:27 Microsoft (R) Test Execution Command Line Tool Version 15.8.0
    17:27:27 Copyright (c) Microsoft Corporation. All rights reserved.
    17:27:27
    17:27:27 No test source files were specified.
    17:27:27 Build step 'Run unit tests with VSTest.console' changed build result to UNSTABLE
    17:27:28 Finished: UNSTABLE

    1. ben bourner

      Also have this problem, looks like you were trying as me to see if somehow it had changed to accepted ${ENVIRONMENT} instead of %ENVIRONMENT% vars. Nope. It won't expand any of them. Author does not reply. I'm getting tired of fragile Jenkins plugins that cannot be updated without huge risk. Downgrading to 1.0.4 solves the problem.

  16. Stephen Leung

    There is no option to downgrade to 1.0.4 in latest Jenkins.  Using 1.08 plugin.  Any other workarounds or fixes?

    Path To VSTest.console.exe: "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\\Extensions\TestPlatform\vstest.console.exe"
no files matching the pattern TestCases.dll
Executing VSTest: cmd.exe /C C:\WINDOWS\TEMP\vstest6330007923803257574.bat && exit %ERRORLEVEL%
[adxchrome] $ cmd.exe /C C:\WINDOWS\TEMP\vstest6330007923803257574.bat && exit %ERRORLEVEL%

C:\Program Files (x86)\Jenkins\workspace\adxchrome>"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\\Extensions\TestPlatform\vstest.console.exe" /InIsolation /Platform:x64 /Logger:trx 
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

No test source files were specified.
Build step 'Run unit tests with VSTest.console' changed build result to UNSTABLE
Finished: UNSTABLE

  17. Alex Cardos

    There is a workaround for this and it would be to Execute Windows Batch Command with parameters similar to:

    "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" TestFolder\bin\x86\Release\Test.dll

    1. Stephen Leung

      Thanks, this workaround runs successfully for me.  I was able to append /Logger:trx to the command line and use the Post Build Publish MS test result to create a report.