Child pages
  • MSTestRunner Plugin
Skip to end of metadata
Go to start of metadata

View Jenkins MSTestRunner plugin on the plugin site for more information.

This plugin allow you to execute test using MSTest command line tool.

Description

This plugin allow you to execute test using MSTest command line tool.

MSTest are test classes for .NET framework which are integrated into Visual Studio. You should use a plugin to process the result files, I recommend using MSTest Jenkins Plugin. You can read my post about how to setup Jenkins with MSTest in my blog.

Configuration

To configure a MSTest installation, go to Manage Jenkins -> Configure System (or Manage Jenkins -> Global Tool Configuration in Jenkins 2.8, possibly earlier) and add a MSTest installation. Name is mandatory. If Path to MSTest is left blank, the default is MSTest.exe.

MSTestRunner can be used as a build step. The screenshot below shows the configuration parameters available where the Test Files and the Result File Name are mandatory. The MSTest Version is not mandatory. If the default is used, the plugin will look for MSTest.exe, so ideally it should be in the path.

For further configuration tips, see each field's inline help.

Version History

Version 1.3.0 (25/05/2016)

Version 1.2.0 (09/12/2015)

  • JENKINS-28869 - Fixes problem with newline tokennization (thanks to Thomas Blitz)
  • JENKINS-29389 - Quotation marks for /testcategories argument, must beused only when multiple category was specifed (thanks to Nikolas Falco)
  • JENKINS-29391 - add support to full path in testFiles (thanks to Nikolas Falco)
  • JENKINS-30292 - Not visible in Conditional BuildStep Plugin (thanks to Ido Ran)

Version 1.1.0 (06/04/2014)

  • Add Continue On Failed Test checkbox which allows to report success even when test have failed (JENKINS-20367)

Version 1.0.1 (06/06/2013)

  • Use workspace path instead of module path to support multi-module build (#2 by blazee)

Version 1.0.0 (31/05/2013)

  • Path to test containers (DLLs) are allowed to have space in them.
  • Test containers can now only be separated with new lines because they are allowed to have space in them.
  • The license of this plugin is now MIT.

Version 0.2.1 (09/14/2012)

  • Fix the bug in which the flag /noisolation was not added by default.
  • Add new option in MSTest config (Manage Jenkins -> Configure System -> MSTest) name Omit Noisolation. Checking this option will omit the /noisolation flag from the call to MSTest.

Version 0.2 (09/02/2012)

  • Add parameter for running MSTest with or without isolation (thanks to roicp #1)

Notice
Due to update in version 0.2 the plugin do not add /noisolation flag by default which break MSTest on some systems.

To work around this problem until then either downgrade to 0.1 or add /noisolation flag in Manage Jenkins -> Configure System -> MSTest -> Default Paramters.

You can upgrade to version 0.2.1 which fix the problem.

Version 0.1 (01/15/2012)

  • Initial release 

20 Comments

  1. Nice plugin.  Simple and to the point.  Really helped me cleanup some jobs by removing awkward shell calls to build MSTest calls.

    I'm still having to have one shell call just prior to the MSTest run to manually delete the prior runs *.trx output files.  MSTest will not overwrite so if the old file is there the job fails.

    Would it be reasonable to extend this plugin to optionally delete the output file if a specific name is specified?

  2. Is there a way to set the PATH or the present working directory before MSTest is run. I tried the EnvInject plugin to modify the PATH before the MSTest ran but that didn't seem to work.

    1. What is the problem? Why do you want to set the PATH?

    2. I mean you could just add a Powershell or batch/command prompt build step before the MSTest and do $Env:Path or something

  3. It would be nice to have a link to a Download page

    1. You are welcome to download it via Jenkins Plugins page. 

      Also you can see the source on github. 

      Ido

  4. It would be nice to have a link to a Download page

  5. When i have failing tests i get my build marked as a failure rather than unstable. It seems the MSTestRunner marks the build as a failure. Any ideas?

    this in my output:169/177 test(s) Passed, 8 Failed

    Summary
    -------
    Test Run Failed.
    Failed 8
    Passed 169
    -----------
    Total 177
    Results file: D:\Jenkins\JOB\workspace\mstest-result.trx
    Test Settings: Default Test Settings
    Build step 'Run unit tests with MSTest' marked build as failure

    1. Also, I was checking the help text for the MSTest setup in jenkins, it says:

      Specify the path to your msbuild executable. This will default to 'msbuild.exe'. It is usually located at C:\WINDOWS\Microsoft.NET\Framework[version]\MSBuild.exe.
      Specify default paramaters to pass to all MSBuild tasks.

      I assume this is wrong?

      1. Hi, I've develop MSTestRunner plugin.

        About the fact that failed test fail the build - this is by design. If the test do not pass the whole build is considered failed.

        It is possible to change this behavior to make it configurable, will that be sufficient for you?

        Abou the help text, I've start my project from MSBuild plugin so I may have forgot to replace the help text in the setup area. It will be fixed.

        Thank you,

        Ido.

        1. Yeah having that as an option would be great!

        2. Hi, will be a new version about unstable configurable option?

          Thanks.

          1. What you are asking is to have a configuration option that will decide if a test fail then the whole build should be either mark as failed or as unstable?

              1. Sorry for the very long delay.

                I realize that your request is right but on the wrong plugin.

                MSTestRunner is executing the test, not analyzing the results.

                MSTest plugin by Antonio Marques* *is the right place to put such option because there the analyze happen.

                I'll try to do the change there.

                1. Hi Ido Ran,

                  i want to thank you for that useful plugin but while digging in MSTestPublisher code i ended up with this piece of code:

                  catch (AbortException e) {

                  if(build.getResult()==Result.FAILURE) return true; // most likely a build failed before it gets to the test phase.

                  // don't report confusing error message.

                  listener.getLogger().println(e.getMessage());

                  build.setResult(Result.FAILURE); return true;

                  These are the only two FAILUREs forced from the publisher plugin.

                  The build looks already in FAIL state from test-running plugin.

                  The publisher plugin seems to keep the test-runner plugin failure as-is (and it looks ok to me)

                  Am i wrong?

                  I expect test runner just to run test, collecting results and decide how to handle them is business of the publisher plugin.

                  Full code:

                  https://svn.jenkins-ci.org/trunk/hudson/plugins/mstest/src/main/java/hudson/plugins/mstest/MSTestPublisher.java

  6. Thanks for your effort in creating this plugin. I noticed one problem, first the result file is deleted then the build fails because its not there. Any workaround to this?

    Delete old result file file:/C:/.jenkins/jobs/JenkinsTest-1/workspace/TestResults/List1.trx
    FATAL: null
    java.lang.NullPointerException
    at org.jenkinsci.plugins.MsTestBuilder.perform(MsTestBuilder.java:141)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814)
    at hudson.model.Build$BuildExecution.build(Build.java:199)
    at hudson.model.Build$BuildExecution.doRun(Build.java:160)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:593)
    at hudson.model.Run.execute(Run.java:1567)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:237)

    1. I noticed I had the same problem when using a windows cmd script so the issue it not with the plugin. There seem to be an issue when using a static name for the .trx file.

  7. Hello, I have actually problem to run my ServiceStack unit tests. I discovered that is problem with parameters order generated by plugin.

    When I open cmd and run my unit tests like this:

    "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\MSTest.exe" /testcontainer:[test.dll] /category:[category] /resultsfile:[resultfile] /testsettings:[file settings]
    all works fine, but when I'm trying run my tests by plugin, he generate this line like this:
    "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\MSTest.exe" /resultsfile:[resultfile] /noisolation /testsettings:[file settings] /category:[category] /testcontainer:[test.dll]
    my all tests are failed in half second because something in background don't load my app.config or something else with ServiceStack license. As you can see order of parameters are differents, maybe it's issue?


    MSDN said that /testcontainer must be specified before using /category.

  8. Configuration of the Path to MSTest field in Jenkin's Global Tool configuration screen (v2.89.3) gives a misleading error message of:

     

    C:\PROGRA~2\MICROS~2.0\Common7\IDE\mstest.exe is not a directory on the Jenkins master (but perhaps it exists on some agents)

     

    It does not matter if we use the normal name (with spaces and parens) or the short name (windows 8.3 name) shown above. The error is always displayed.

     

    Removing "\mstest.exe" from the end removes the warning. However, if we remove the name from the end then the mstest build step fails with a message such as:

     

    [Hello World] $ "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\" /resultsfile:Results.trx /noisolation "/testcontainer:C:\Users\John\.jenkins\workspace\Hello World\HelloWorldTests\bin\Debug\HelloWorldTests.dll"
    Access is denied
    FATAL: MSTest command execution failed
    java.io.IOException: Cannot run program "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\" (in directory "C:\Users\John\.jenkins\workspace\Hello World"): CreateProcess error=5, Access is denied
    	at java.lang.ProcessBuilder.start(Unknown Source)
    	at hudson.Proc$LocalProc.<init>(Proc.java:249)
    	at hudson.Proc$LocalProc.<init>(Proc.java:218)
    	at hudson.Launcher$LocalLauncher.launch(Launcher.java:930)
    	at hudson.Launcher$ProcStarter.start(Launcher.java:450)
    	at hudson.Launcher$ProcStarter.join(Launcher.java:461)
    	at org.jenkinsci.plugins.MsTestBuilder.perform(MsTestBuilder.java:244)
    	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
    	at hudson.model.Build$BuildExecution.build(Build.java:206)
    	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
    	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
    	at hudson.model.Run.execute(Run.java:1724)
    	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    	at hudson.model.ResourceController.execute(ResourceController.java:97)
    	at hudson.model.Executor.run(Executor.java:421)
    Caused by: java.io.IOException: CreateProcess error=5, Access is denied
    	at java.lang.ProcessImpl.create(Native Method)
    	at java.lang.ProcessImpl.<init>(Unknown Source)
    	at java.lang.ProcessImpl.start(Unknown Source)
    	... 16 more
    Build step 'Run unit tests with MSTest' marked build as failure
    [MSTEST-PLUGIN] INFO processing test results in file(s) Results.trx
    ERROR: Step ‘Publish MSTest test result report’ failed: [MSTEST-PLUGIN] No MSTest TRX test report files were found. Configuration error?
    Finished: FAILURE 

     

    The trailng "\mstest.exe" suffix is needed. This is differrent from the MSBuild plugin. Ignore the warning from Jenkins.

     

Write a comment…