Plugin Information |
---|
View VSTest Runner on the plugin site for more information. |
Description
This plugin allow you to execute test using VsTest command line tool.
VSTest are test classes for .NET framework which are integrated into Visual Studio.
Configuration
System configuration
- Open the system configuration page "Manage Jenkins→Global Tool Configuration"
- Enter the path to the VsTest command line client, that should be used by Jenkins.
- The path location is as instance: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe.
Job configuration
name | description |
---|---|
VsTest Version | Select VsTest Version. |
Test Files | Specify the path to your VsTest compiled assemblies. |
Settings File | Run tests with additional settings such as data collectors. |
Test Name | Run tests with names that match the provided values. |
Test Case Filter | Run tests that match the given expression. |
Enable Code Coverage | Enables data diagnostic adapter CodeCoverage in the test run. |
InIsolation | Runs the tests in an isolated process. |
Use Vsix Extensions | This makes vstest.console.exe process use or skip the VSIX extensions installed (if any) in the test run. |
Use VS 2017 or Later | This makes changes to the arguments passed to the VS Test Agent in order to ensure compatibility with the latest version. |
Platform | Target platform architecture to be used for test execution. |
.NET Framework Version | Target .NET Framework version to be used for test execution. |
Specify a logger for test results. | Specify a logger for test results. For example, to log results into a Visual Studio Test Results File (TRX) use /Logger:trx. |
Command Line Arguments | This is a whitespace separated list of command line arguments you can specify. These can be the same as if you were to run VsTest from the command line. |
Fail build on test failure | Fail build on test failure |
Changelog
Version 1.0.5 (10/24/2017)
- To be updated with description of changes and links to the resolved issues.
Version 1.0.4 (08/31/2014)
- Make trx and coverage files paths available to further build steps in VSTestRunner plugin. (JENKINS-22450)(Pull Requests #2)
Version 1.0.3 (10/21/2013)
- Changed to extend BuildStepDescriptor. (Pull Requests #1)
Version 1.0.2 (07/24/2013)
- Supports Jenkins variable.
I can be used in the following items.- Test Files
- Settings File
- Test Name
- Test Case Filter
- Command Line Arguments
- I Fixed a bug that was not used by the "Command Line Arguments".
Version 1.0.1 (03/09/2013)
Mod default value.
- Plaform Field: None
- .NET Framework Version Field: None
- Specify a logger for test results. Field: trx
Version 1.0.0 (01/29/2013)
Initial release.
34 Comments
Unknown User (cbertolasio)
Is there a way to use wildcards in the test file name?
eg "DataCenter\\SolutionItems\\UnitTests\\**
*Tests.dll"
or do i need to know the exact path to every unit test dll?
Unknown User (hungwunfai)
There is a bug which was fixed but not released.
Unknown User (annetheagile)
Is there going to be a new release? It seems like there are quite a few commits since last year.
Without a new release, one could still compile the source and put it in the plugsins folder, right?
thank you!
Unknown User (egoughnour)
Unknown User (qa_automation2015)
After a test run, Is there a way to access the total of run tests, the number of failed and number of passed?
I am looking for this in order to add to a notification e-mail.
For instance: $BUILD_NUMBER $BUILD_STATUS. Total tests: $VSTEST_TOTAL. Passed: $VSTEST_PASSED. Failed: $VSTEST_FAILED
Is there any way to do this?
Thanks,
QA Automation Team
Unknown User (afrancoc2000)
Hi, Im trying to access the environment variable VSTEST_RESULT_TRX to check the name of the generated trx file to publish using the MS Test Plugin but its not working, I try to do an echo on the variable and checked the jenkins environment variables used and I can't find it.
Do I need to install something else? the new trx files are being created its just the variables that don't seem to work.
Thanks
Unknown User (ktighe)
Hi,
The 1.05 release broke this plugin for me (using Jenkins 2.86). I can no longer see the vstest tool configuration section, and the build steps have disappeared from my builds.
Unknown User (joaomiranda)
In the LTS version too (2.73)
Unknown User (egoughnour)
Thanks.
Working on 1.0.6.
Unknown User (johntiger1)
Also broke for us
Unknown User (gargmukku07)
1.0.5 also broke for me. So i downgraded to 1.0.4
Unknown User (egoughnour)
Please make sure I am aware of your current issues–particularly the use case. I will try to test out any scenarios that are failing–but I will have to know exactly what is failing in order to do so.
By the way I am hoping to have pipeline and block support.
I'd like to be able to eliminate most of the pain of configuring the plugin (and maintaining related pipeline scripts) by simplifying the groovy syntax and making most or all of the necessary following steps selectable via options (or at least callable within the block).
Unknown User (gargmukku07)
Hi,
Can i use wild card character in Test Files?
Unknown User (daves)
Looks to me like the latest version of Visual Studio 2017 comes with a vstest.console.exe that now enforces case sensitivity on its command line arguments. Example: vstest runner passes "framework45." Up until a recent version of VS2017, this was fine. Now it complains with:
"Invalid .Net Framework version:framework45. Please give the fullname of the TargetFramework. Other supported .Net Framework versions are Framework35, Framework40, Framework45 and FrameworkCore10."
My work around:
Unknown User (gensplejs)
Hi.
I just updated to 1.0.7 from 1.0.4 and now the test runner now insists it can't find my test files.
"no files matching the pattern"
One of the file paths is %WORKSPACE%\125-20\Common\UnitTest\bin\x86\Debug\Unitek.UniLock.Common.CommonClassLib.Tests.dll
I have attempted without %WORKSPACE% and with the aboslute path to the file. none of it works in 1.0.7
Downgrading to 1.0.4 and it works fine.
Unknown User (benbo)
Confirmed, I have the same problem, it seems the latest version 1.0.7 cannot expand environment variables? Downgrading to 1.0.4 resolves the issue. Apparently author is not bothered that he's broken things, judging by lack of response to you or anybody below.
Unknown User (jose_farina)
Hello,
Seems that the version 1.0.7 fixed the issue that makes that you cant see the plugin in the build steps. But it has a new one, everytime I restart the jenkins, the vstest.console installation folder got reseted, so i need to add a new installation folder.
Regards,
José
Unknown User (gensplejs)
I have that problem too. Every reboot and the folder is reset and jenkins can't find vstest.... argh...
Unknown User (vincentsk)
Encountered same issue, restart jenkins, vstest configurations reset to default.
Unknown User (rlandfair)
The same problem exists for me. The Global Tool Configuration settings use the defaults after restarting Jenkins.
Unknown User (spoiledtechie)
1.0.7 puts a PREFIX SLASH in front of the test files. Please see below. Because of this prefix SLASH, it can no longer find the test files. HELP. I will see if I can downgrade. 1.05 worked well.
BLAH. The slashes can't be placed in this textbox....
PLEASE remove the SLASH from the front of the code files. I am also using the D drive, instead of the C drive, so I am not sure if thats an issue here or not.
Unknown User (mgiammarco)
Hello,
have you found a solution?
IT is absolutely blocking for me!
Thanks,
Mario
Unknown User (vincentsk)
****************************************************
****************************************************
Unknown User (mgiammarco)
I have the same problem, solved with same workaround. Version 1.0.7
Unknown User (thomasklammer)
Updated to 1.0.7 from 1.0.4: "no files matching the pattern"
Regardless what I'm doing I'm always getting this error message
Unknown User (thomasklammer)
You aren't really interested in solving this problem, aren't you?
Unknown User (benbo)
I also have this problem. Author does not seem bothered. Downgrading to 1.0.4 solves it. Stuff like this makes me want to move away from Jenkins - it's impossible to update any plugin without risking everything breaking.
Unknown User (mgiammarco)
I have opened a ticket in jenkins google group https://groups.google.com/forum/?nomobile=true#!searchin/jenkinsci-users/vstestrunner%7Csort:date/jenkinsci-users/5g7Hxqwk61k/z3LO9sF_CQAJ
Please go there and reply to show that there is interest for the plugin.
Unknown User (joao)
Hello guys, I need help as well... In my tests show this mensage:
Warning: Using Isolation mode to run tests as required by effective Platform:X64 and .Net Framework:Framework45 settings for test run. Use the /inIsolation parameter to suppress this warning.
Starting test discovery, please wait...
Warning: A total of 7 tests were discovered but no test matches the specified selection criteria(CriarPAC). Use right value(s) and try again.
Path To VSTest.console.exe: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe
Executing VSTest: cmd.exe /C C:\Users\ADMINI~1\AppData\Local\Temp\vstest4061933848113271387.bat && exit %ERRORLEVEL%
[Teste Web] $ cmd.exe /C C:\Users\ADMINI~1\AppData\Local\Temp\vstest4061933848113271387.bat && exit %ERRORLEVEL%
I'm used the plugin version 1.0.4, Windowns 10, Selenium with Visual Studio
Unknown User (dwright)
We're having the same issue as "Thomas Klammer" "no files matching the pattern" "No test source files were specified."
using version 1.0.7 of this plugin, Vstest console 2017, .net framework 4.7.1
I've tried multiple version of the file pattern as seen below
17:27:27 Path To VSTest.console.exe: "C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
17:27:27 no files matching the pattern %WORKSPACE%\src\UnitTests\obj\Release\UnitTests.dll
17:27:27 ${WORKSPACE}\src\iNetControlTests\obj\Release\iNetControlTests.dll
17:27:27 "%WORKSPACE%\src\UnitTests\obj\Release\UnitTests.dll"
17:27:27 "${WORKSPACE}\src\iNetControlTests\obj\Release\iNetControlTests.dll"
17:27:27 C:\Jenkins\workspace\TestWindows\src\DomainModelTests\obj\Release\DomainModelTests.dll
17:27:27 "C:\Jenkins\workspace\TestWindows\src\iNetAdminConsoleTests\obj\Release\iNetAdminConsoleTests.dll"
17:27:27 Executing VSTest: cmd.exe /C C:\Windows\TEMP\vstest2305357646525754546.bat && exit %ERRORLEVEL%
17:27:27 [TestWindows] $ cmd.exe /C C:\Windows\TEMP\vstest2305357646525754546.bat && exit %ERRORLEVEL%
17:27:27
17:27:27 C:\Jenkins\workspace\TestWindows>"C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /Settings:"C:\Jenkins\workspace\TestWindows\src\CodeCoverage.runsettings" /Platform:x64 /Logger:trx
17:27:27 Microsoft (R) Test Execution Command Line Tool Version 15.8.0
17:27:27 Copyright (c) Microsoft Corporation. All rights reserved.
17:27:27
17:27:27 No test source files were specified.
17:27:27 Build step 'Run unit tests with VSTest.console' changed build result to UNSTABLE
17:27:28 Finished: UNSTABLE
Unknown User (benbo)
Also have this problem, looks like you were trying as me to see if somehow it had changed to accepted ${ENVIRONMENT} instead of %ENVIRONMENT% vars. Nope. It won't expand any of them. Author does not reply. I'm getting tired of fragile Jenkins plugins that cannot be updated without huge risk. Downgrading to 1.0.4 solves the problem.
Unknown User (xstepl)
There is no option to downgrade to 1.0.4 in latest Jenkins. Using 1.08 plugin. Any other workarounds or fixes?
Unknown User (alexcardos)
There is a workaround for this and it would be to Execute Windows Batch Command with parameters similar to:
"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" TestFolder\bin\x86\Release\Test.dll
Unknown User (xstepl)
Thanks, this workaround runs successfully for me. I was able to append /Logger:trx to the command line and use the Post Build Publish MS test result to create a report.