Child pages
  • MSTest Plugin
Skip to end of metadata
Go to start of metadata

Plugin Information

View MSTest on the plugin site for more information.

This plugin converts MSTest TRX test reports into JUnit XML reports so it can be integrated with Jenkin's JUnit features. This plugin converts the .coveragexml files found in the project workspace to the EMMA format.

You can use MSTestRunner plugin or VsTestRunner plugin to run the test and use this plugin to process the results.

 

The MSTest plugin analyzes the test execution reports (TRX) files generated by mstest and vstest.console. These files include a test execution summary, and detailed data about what happened during the tests execution. Along with the test execution records, vstest or mstest may also include a link towards the code coverage data collected during the tests execution. These coverage files have a binary, proprietary format, and you have to convert them to XML before that the MSTest publisher is invoked.

This plugin converts the test records to the JUnit format, and adds them to the build report. The plugin also searches for the link to the code coverage data. If it is present, and it contains valuable data, the plugin converts it to the Emma format by means of an ad-hoc XSL transformation.

If you want to show the coverage data on the build report, it is up to you to add the Emma plugin to your Jenkins instance, and to add the corresponding post-build action to your build. The coverage reports will be somewhere in the build workspace, and their name will match the pattern: `emma\coverage.xml`. Apart from the Emma plugin, which is rather outdated, probably also other plugins can complete the same task (for example, the JaCoCo plugin).

Pipeline Support

The MSTest plugin supports the pipeline plugin (and/or Jenkinsfile build definitions). The plugin call be called with an instruction like

step([$class: 'MSTestPublisher', testResultsFile:"**/*.trx", failOnError: true, keepLongStdio: true])

Or with its shortest alternative:

mstest testResultsFile:"**/*.trx", keepLongStdio: true

 

Code Coverage Support

Since the code coverage data has a binary, proprietary format, and that the tools capable of handling them are released under a proprietary license along with the develipment environments, you will have to perform the data conversion yourself. Here's how.

Code Coverage Data Conversion

To convert the binary VSTest.Console output to the Microsoft CoverageDS XML format, you may use one of the prebuilt applications referenced in the V0.14 release notes below, or you can build the following converter application:

CoverageCoverter.exe
    class Program
    {
        static int Main(string[] args)
        {
            if ( args.Length != 2)
            {
                Console.WriteLine("Coverage Convert - reads VStest binary code coverage data, and outputs it in XML format.");
                Console.WriteLine("Usage:  ConverageConvert <sourcefile> <destinationfile>");
                return 1;
            }

            CoverageInfo info;
            string path;
            try
            {
                path = System.IO.Path.GetDirectoryName(args[0]);
                info = CoverageInfo.CreateFromFile(args[0], new string[] { path }, new string[] { });
            }
            catch (Exception e)
            {
                Console.WriteLine("Error opening coverage data: {0}",e.Message);
                return 1;
            }

            CoverageDS data = info.BuildDataSet();

            try
            {
                data.WriteXml(args[1]);
            }
            catch (Exception e)
            {

                Console.WriteLine("Error writing to output file: {0}", e.Message);
                return 1;
            }

            return 0;
        }
    }

The CoverageDS and CoverageInfo types are being exposed by the Microsoft.VisualStudio.Coverage.Analysis.dll (official documentation: https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.coverage.analysis.coverageds.aspx, article explaining how to use these assemblies with a nice step-by-step: https://blogs.msdn.microsoft.com/phuene/2009/12/01/programmatic-coverage-analysis-in-visual-studio-2010/

The following Powershell build step will find one binary coverage data file in your workspace and convert it to XML format, assuming you use the default TestResults directory. (Modify as necessary to handle multiple coverage files)

Powershell build step for CoverageDS XML Conversion
$generatedCoverageFile = $(get-ChildItem -Path .\TestResults -Recurse -Include *coverage)[0]
CoverageConverter $generatedCoverageFile TestResults\vstest.coveragexml

Can I use Microsoft's CodeCoverage.exe for data conversion?

Microsoft supplies C:\Program Files (x86)\Microsoft Visual Studio 1X.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe with Visual Studio. But unfortunately this tool uses an XML format distinct from the CoverageDS format used by vstest and mstest, and is not compatible with the MSTest plugin. If you are interested in this topic, and you've some experience with XSL, there are a couple of transforms to convert back and forth in the github repository.

Change Log

Version 0.20 (September 1st, 2017)

Version 0.19 (September 1st, 2015)

  • Support for web tests (contacted by email, by Peter Barnes. No Jira issue has been opened)
  • Let the users still using Java 1.6 to continue using the plugin [JENKINS-29032]
  • Mark the inconclusive tests as 'skipped' [JENKINS-29316]

Version 0.18 (May 12th, 2015) --- !!! Java 1.7 is required !!!

  • Add support for "Retain long standard output/error" [JENKINS-28281]. The default value for this option is false. If you're automating the creation of your jobs, simply specify keepLongStdio=on as a parameter of your query. Any other value than 'on' will set this option to false.
  • Add localized messages for it, pt-BR, fr
  • Cumulated code coverage filename: vstest.coveragexml
  • Add default values for test result pattern and "fail if no result file is found": */.trx and true.

Version 0.17 (May 4th, 2015) --- !!! Java 1.7 is required !!!

  • Add a checkbox to ignore missing TRX files (Thanks Christopher Bush, pull request #7). The pull request contains also a way to automate job creation using the REST API. So, if you're automating the creation of your jobs, just specify failOnError=on to enable this feature. Any other value than 'on' will set this option to false.
  • Fix the code coverage calculations (Thanks junshanxu, pull request #6): a sum over all the nodes is better than using the value of the first node only.

Version 0.16 (Apr 14th, 2015) --- !!! Java 1.7 is required !!!

  • Show the code coverage graph for coveragexml files (one of the two XSD, the one produced by vstest)

Version 0.15 (Apr 14th, 2015) --- !!! Java 1.7 is required !!!

  • Improve support for data driven tests (Thanks, Darryl Melander: pull request #6)
  • Preserve charsets while fixing TRX files (JENKINS-23531, reopened by JitinJohn@MS)

Version 0.14 (Apr 1st, 2015)

  • Support for output/stdout messages (JENKINS-19384)
  • Drop invalid XML entities (JENKINS-23531). MSTest allows writing XML entities corresponding to invalid XML characters. These XML entities generate exceptions while being parsed by Java parsers. For me, it's still unclear if such entities are standard or not. However, to avoid these exceptions, the mstest parser simply drops them. These entities normally correspond to non printable characters.
  • Support for .coveragexml files. The coverage data present in these files is being transformed in an EMMA coverage report. Today, you can try to generate vscoveragexml files using https://github.com/gredman/CI.MSBuild.Tasks or https://github.com/yasu-s/CoverageConverter.

Version 0.13 (Mar 18, 2015)

  • Support for ignored tests (JENKINS_27469)
  • Support for data driven tests (JENKINS-8193, JENKINS-4075)
  • Support for timed out tests (JENKINS-11332)
  • Support for TextMessages (JENKINS-17506)
  • Improved processing for tests whose @outcome is not set
  • Stacktraces are now shown as stacktraces, and error messages as error messages

Version 0.12 (Mar 12, 2015)

  • Convert MS XML code coverage reports in emma coverage reports, and show them.
  • Fix: the tests for which the outcome is 'error' (or missing, with an error message or a stack trace) will be reported as junit errors.

Version 0.11 (Jan 17, 2015)

  • Support vstest TRX format
  • Support environment variables as target (vstestrunner-plugin exports the full path to the TRX as environment variable)

Version 0.7 (Jun 17, 2011)

Version 0.6 (Feb 11, 2010)

  • Fixed issue JENKINS-3906: Durations greater than 59s
  • Fixed issue JENKINS-4632: MSTest plugin does not parse Visual Studio 2010 results

Version 0.5 (Feb 6, 2010)

  • Update code for more recent Hudson

Version 0.4 (Jun 16, 2009)

  • Fixed the AbortException issue
  • Added i18n support
  • Added Brazilian portuguese localization

Version 0.3

  • Indentifies test's class using the ExecutionId variable

Version 0.2

  • Fixed a problem to identify namespace and class name from the TestMethod tag
  • Changed JUnit test report file name

Version 0.1

  • Initial Release

31 Comments

  1. Unknown User (john.bolton@quest.com)

    Feature request - please add the capability to process multiple report (*.TRX) files. Thank you.

    1. Unknown User (mdulfer)

      I'd really appreciate the ability to process multiple trx files too, thanks.

    2. Unknown User (alea)

      Handling multiple files would be really handy a-la TRXMerge would be fantastic!

    3. Unknown User (kunom)

      As a workaround, you can invoke MSTest with multiple assemblies to test at once, so that you don't end with multiple .TRX files.

      1. Unknown User (hector.insausti@gmail.com)

        That doen't work if you need to merge tests that need to be run with different configurations... TRXMerge only joins the test results, but Hudson doen't accept that file because some references are missed.

        1. Unknown User (jamiet)

          Hi,
          I'm glad I'm not the only one encountering this. We too need the ability to specify multiple .trx files, the reason being that we need to run different configurations for each of our test containers. Multiple configs means multiple calls to mstest which means multiple .trx files.

          Wildcards would be a perfectly adequate solution by the way! (i.e. TestResults*.trx)

          Other than that, love the plugin, really appreciate the time spent building it. Thanks very much.

          Regards
          Jamie

    4. I find myself wanting this too.  I have some MSTtest tests I run, but I also have some Silverlight unit tests that I run with StatLight.  I want all test results to be merged.

  2. Unknown User (gregoryboissinot)

    Do you plan to make the MSTest plugin an extension of the xUnit plugin

  3. Unknown User (lenkor)

    Is it possible to add columns(like test description) to Test Results table?

  4. I get the following error when trying to use the plugin:

    Processing tests results in file(s) TestResults.trx
    FATAL: No MSTest test report files were found.
    FATAL: No MSTest TRX test report files were found. Configuration error?
    Finished: FAILURE
    

    The file is present and is a proper result file created by the commandline version of MSTest from Visual Studio 2008.

    Regards,
    Lars

    1. Do you have the same behavior with the xUnit plugin?

  5. Hi,

    I've wrote a plugin to run MSTest tests, it is stored at github - https://github.com/jenkinsci/mstestrunner-plugin

    I think it will be helpful to combine both of our plugins to create one complete plugin for run and parse MSTest tests.

    My email is ido.ran@gmail.com

  6. I am using the plugin to publish test results from trx file. I can see a table with no. of tests run and passed. But I cannot get the list of all run test cases.

    From the top level I can see results for package (root) as the table, but, when I get into the package, I don't see the class or the test cases, like I see with jUnit.

    Is it possible to access the jUnit xml converted from trx?

  7. When I have MSTest set to a dll that does not have any tests yet it does not create a trx file and this plugin fails the build due to no file.  Is there a way I can make it optional, like use the trx file if you find it, otherwise don't fail the build?  

  8. Unknown User (nnkfws333)

    I noticed when it automatically adds the parameter "/noisolation". Is there anyway to remove this? It's driving me mad!

  9. The report doesn't seem to take into account timeouts. Is there a way of adding these to the failure report?

  10. my unit tests seem to run fine, but the jenkins output is

    Processing tests results in file(s)  TestResults.trx
    TestResults.trx
    Recording test results
    None of the test reports contained any result

    and i get no unit tests graph on my project page!

    i run the mstest like this

    C:\ci\mstest\mstest.exe /noisolation /testcontainer:test\xxx\bin\Release\SomeTest.dll /testcontainer:test\UtilitiesTest\bin\Release\UtilitiesTest.dll /resultsfile:TestResults.trx
    

    I dont find anywhere why this is the case, what do i have to do to get the graph?

    Here is one example output; I shortened it down to just a few lines per section

    <?xml version="1.0" encoding="UTF-8"?>
    <TestRun id="4b571fa5-71e2-45fb-83a6-e97c422092dd" name="SYSTEM@SERVER_NAME 2014-09-17 18:26:12" runUser="NT AUTHORITY\SYSTEM" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
      <TestSettings name="Default Test Settings" id="abe77a9e-cdcc-475a-8073-c5b194e33b03">
        <Deployment userDeploymentRoot="C:\Program Files (x86)\Jenkins\jobs\ProjectName\workspace" useDefaultDeploymentRoot="false" runDeploymentRoot="SYSTEM_SERVER_NAME 2014-09-17 18_26_12" />
        <Execution>
          <TestTypeSpecific />
          <AgentRule name="Execution Agents">
          </AgentRule>
        </Execution>
        <Properties />
      </TestSettings>
      <Times creation="2014-09-17T18:26:12.2593446-07:00" queuing="2014-09-17T18:26:12.8687197-07:00" start="2014-09-17T18:26:13.0562172-07:00" finish="2014-09-17T18:26:13.9780954-07:00" />
      <ResultSummary outcome="Completed">
        <Counters total="24" executed="24" passed="24" error="0" failed="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
      </ResultSummary>
      <TestDefinitions>
        <UnitTest name="TestExampleBon2TruncatedHash" storage="test\utilitiestest\bin\release\lion.utilitiestest.dll" id="f32ca728-ccba-c189-0fc8-ecf706cc5764">
          <Execution timeOut="1800000" id="75078514-d10b-4492-95be-e7383132acb2" />
          <TestMethod codeBase="C:/Program Files (x86)/Jenkins/jobs/LionEntertainment/workspace/test/UtilitiesTest/bin/Release/Lion.UtilitiesTest.dll" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="Lion.UtilitiesTest.HashGeneratorTest, Lion.UtilitiesTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" name="TestExampleBon2TruncatedHash" />
        </UnitTest>
        <UnitTest name="TestExampleBon2Hash" storage="test\utilitiestest\bin\release\lion.utilitiestest.dll" id="e5054321-1c82-20b7-adad-2295ccc86630">
          <Execution timeOut="1800000" id="8946ac7c-d36d-4023-a7b3-0080ec0325dc" />
          <TestMethod codeBase="C:/Program Files (x86)/Jenkins/jobs/LionEntertainment/workspace/test/UtilitiesTest/bin/Release/Lion.UtilitiesTest.dll" adapterTypeName="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestAdapter, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" className="Lion.UtilitiesTest.HashGeneratorTest, Lion.UtilitiesTest, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null" name="TestExampleBon2Hash" />
        </UnitTest>
    </TestDefinitions>
      <TestLists>
        <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
        <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
      </TestLists>
      <TestEntries>
        <TestEntry testId="c927370f-4fdd-8a31-9a1f-1d5cd81175eb" executionId="d2923e16-dfd1-42cc-9390-c53f2bdcfa02" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
        <TestEntry testId="d3e46b42-9570-5dac-fe49-42004f7ed1ea" executionId="ad69e558-e772-49cf-84ad-5a3aeff166d9" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
     </TestEntries>
      <Results>
        <UnitTestResult executionId="d2923e16-dfd1-42cc-9390-c53f2bdcfa02" testId="c927370f-4fdd-8a31-9a1f-1d5cd81175eb" testName="TestCheckLockOnCreateOneTimeCard" computerName="EMPIRE" duration="00:00:00.4369766" startTime="2014-09-17T18:26:13.1187206-07:00" endTime="2014-09-17T18:26:13.6499697-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="d2923e16-dfd1-42cc-9390-c53f2bdcfa02">
        </UnitTestResult>
        <UnitTestResult executionId="ad69e558-e772-49cf-84ad-5a3aeff166d9" testId="d3e46b42-9570-5dac-fe49-42004f7ed1ea" testName="TestCreateNormalCardForOtc" computerName="EMPIRE" duration="00:00:00.0034263" startTime="2014-09-17T18:26:13.6499697-07:00" endTime="2014-09-17T18:26:13.6499697-07:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="ad69e558-e772-49cf-84ad-5a3aeff166d9">
        </UnitTestResult>
     </Results>
    </TestRun>
    
    
  11. Hi,

    After my tests runned (via the  vs2013 exe vstest.console.exe ) and the "a.trx" file is created, my jenkins should "publish a test report with MSTest".

    So I give the TestResults\a.trx file as parameter of the plugin but I get an error .

    Here are the logs :

     ...Total tests: 15. Passed: 15. Failed: 0. Skipped: 0.
    Test Run Successful.
    Test execution time: 0,3350 Seconds
    MSTest: Processing tests results in file(s) TestResults\a.trx
    MSTest: C:\Program Files (x86)\Jenkins\workspace\tepp\TestResults\a.trx
    mstest xml coverage report file not found: C:\Program Files (x86)\Jenkins\workspace\tepp\TestResults\mstest-coverage.xml
    
    1. However, it is able to process the code coverage report  if we keep the report file as $

      Unknown macro: {workspace}

      \emma\coverage.xml.

    2. Have you tried to generate the coveragexml report using the hint on the wiki page above?

      The coverage data present in these files is being transformed in an EMMA coverage report. Today, you can try to generate vscoveragexml files using https://github.com/gredman/CI.MSBuild.Tasks orhttps://github.com/yasu-s/CoverageConverter.

  12. I'm also getting the mstest xml coverage report not found when the file is exactly where its saying its not. I execute the performance monitor manually, launch the unit tests and then shutdown the performance monitor. MSTEST XML coverage report file not found: E:\jenkins\jobs\Informatics\workspace\Source\Informatics\Testing\UnitTests\mstest-coverage.xml
    MSTEST XML coverage report file not found: E:\jenkins\jobs\Informatics\workspace\Source\Informatics\Testing\UnitTests\NoDependency_TestResults.coveragexml

    I do not have the Emma coverage plugin in my post build steps. However in my project area I have the Coverage Trend link which goes to nothing but a broken image. After I run the tests I copy the existing coverage file to both of the files that mstest is looking for but still it reports as the file is not found.

    1. Have you tried to generate the coveragexml report using the hint on the wiki page above?

      The coverage data present in these files is being transformed in an EMMA coverage report. Today, you can try to generate vscoveragexml files using https://github.com/gredman/CI.MSBuild.Tasks orhttps://github.com/yasu-s/CoverageConverter.

  13. I spent hours trying to get the coverage data to work properly with this plugin, because it is poorly documented and the responses to previous questions are abysmal.  But I was finally successful and thought I would provide some help for others.

    Q:  What file format does the MSTest plugin expect for its coverage input?

    A:  The file should be in Emma XML format.  I don't know where this is specified, but there is an example here

    Q:  What file name does the MSTest plugin expect the coverage report to have?

    A:  The coverage file must be located in a directory named "emma" (anywhere in your workspace) and must be named coverage.xml.  (**/emma/coverage.xml)

    Q:  How can I create this file with modern (e.g. 2013, 2015) versions of Visual Studio?

    A:  You're going to need some tools:

    • Download and install the VSTestRunner Plugin
    • Download MSXSL from Microsoft
    • Download the MSTest-To-Emma XSL tranform from the Jenkins wiki
    • Either download one of the transformation tools linked by Ivo Bellin Salarin above, or build your own. I built my own (CoverageConverter.exe) using the following code:
      CoverageCoverter.exe
          class Program
          {
              static int Main(string[] args)
              {            
                  if ( args.Length != 2)
                  {
                      Console.WriteLine("Coverage Convert - reads VStest binary code coverage data, and outputs it in XML format.");
                      Console.WriteLine("Usage:  ConverageConvert <sourcefile> <destinationfile>");
                      return 1;
                  }
      
                  CoverageInfo info;
                  string path;
                  try
                  {
                      path = System.IO.Path.GetDirectoryName(args[0]);
                      info = CoverageInfo.CreateFromFile(args[0], new string[] { path }, new string[] { });
                  }
                  catch (Exception e)
                  {
                      Console.WriteLine("Error opening coverage data: {0}",e.Message);
                      return 1;
                  }
                  
                  CoverageDS data = info.BuildDataSet();
      
                  try
                  {
                      data.WriteXml(args[1]);
                  }
                  catch (Exception e)
                  {
      
                      Console.WriteLine("Error writing to output file: {0}", e.Message);
                      return 1;
                  }
                  
                  return 0;
              }
          }
      

    I used the following approach:

    • Install the VSTestRunner plugin
    • Add a build step to Run Unit Tests with VSTest.console
    • Configure that step to Enable Code Coverage
    • Add a Windows Powershell build step to convert the binary output of VSTest.Console to the Emma XML format.
      Powershell build step
      if (test-path emma) { remove-item emma -recurse -force }
      new-item -type container emma | out-null
      $generatedCoverageFile = $(get-ChildItem -Path .\TestResults -Recurse -Include *coverage)[0]
      CoverageConverter $generatedCoverageFile temp.xml
      msxsl temp.xml "C:\path\to\MSTestCoverageToEmma.xsl" -o emma\coverage.xml
      remove-item temp.xml
      
    • Add a Publish MSTest test result report Post-build Action

    This worked for me, hope it helps someone else

    1. Thanks Tom!

      that's a great documentation.

      Would you please add it to the wiki, with a little precision: if your converter saves the xml file next to the trx, and names it vstest.coveragexml or $NameOfTheTrxWithoutExtension.coveragexml, the mstest plugin will do the rest. i.e. the commands msxsl temp.xml "C:\path\to\MSTestCoverageToEmma.xsl" -o emma\coverage.xml
      remove-item temp.xml

      are not needed.

      Ciao,
      Ivo

  14. Hi,

    I am using MSTestRunner in a parameterized build and I defined a parameter(RF) for Test Categories. The problem is that the parameter is not converted in to the value where the job step is executed.

    This is the parameter configuration:

    This is the MSTest configuration :

    This is the output:

    Any suggestions?

    1. Wrong plugin! You should contact the mstest runner plugin owner, and ask her/him to take into account environment variables.

  15. Hello,

    I'm using pipeline with few stages.

    Having job to Publish MSTest test results (MSTest Plugin) and when having failed tests the job ended UNSTABLE.

    Why failed tests cause job ended UNSTABLE?

    How can I force it to end SUCCESSFUL?

    Best Regards, Moshe

    1. Are you sure that you were using the MSTest plugin in a pipeline in April? Perhaps you're talking about the XUnit plugin?

  16. The test report does not show total duration by Class after selecting a package.

    1. Ciao Jitin, how are you?

      If you have a TRX, even one where you've replaced all the assembly/company/classes with $trash$, I'll have a look at it.

  17. I've configured this plugin to generate emma/coverage.xml. Now how do I feed that into the Jacoco plugin?

Write a comment…