Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Wiki Markup
{jenkins-plugin-info:pluginId=mstest}
Excerpt

This plugin converts MSTest TRX test reports into JUnit XML reports so it can be integrated with Jenkin's JUnit features. This plugin converts the .coveragexml files found in the project workspace to the EMMA format.

You can use MSTestRunner plugin or VsTestRunner plugin to run the test and use this plugin to process the results.

Code Coverage Support

Internally, the MSTest plugin uses code coverage information in the Emma XML file format, but it will automatically convert data in the Microsoft CoverageDS XML format to the Emma format. When using the Microsoft CoverageDS XML format, the coverage file must be named vstest.coveragexml and must be located in the same directory as the test results (.TRX) file. This is usually %workspace%\TestResults

In the unlikely case that you are somehow generating files in the Emma XML format, the coverage file must be named emma\coverage.xml and may be present anywhere in the workspace (in a directory named emma, as shown here)

Code Coverage Data Generation

The VSTestRunner Plugin allows you to specify a build step that includes code coverage generation. This plugin is usually configured to use the Microsoft VSTest.Console.exe application, which generates coverage information in a binary format must be converted to XML format before it can be used by the MSTest Plugin. This conversion is described below. 

The MSTest plugin analyzes the test execution reports (TRX) files generated by mstest and vstest.console. These files include a test execution summary, and detailed data about what happened during the tests execution. Along with the test execution records, vstest or mstest may also include a link towards the code coverage data collected during the tests execution. These coverage files have a binary, proprietary format, and you have to convert them to XML before that the MSTest publisher is invoked.

This plugin converts the test records to the JUnit format, and adds them to the build report. The plugin also searches for the link to the code coverage data. If it is present, and it contains valuable data, the plugin converts it to the Emma format by means of an ad-hoc XSL transformation.

If you want to show the coverage data on the build report, it is up to you to add the Emma plugin to your Jenkins instance, and to add the corresponding post-build action to your build. The coverage reports will be somewhere in the build workspace, and their name will match the pattern: `emma\coverage.xml`. Apart from the Emma plugin, which is rather outdated, probably also other plugins can complete the same task (for example, the JaCoCo plugin).

Pipeline Support

The MSTest plugin supports the pipeline plugin (and/or Jenkinsfile build definitions). The plugin call be called with an instruction like

step([$class: 'MSTestPublisher', testResultsFile:"**/*.trx", failOnError: true, keepLongStdio: true])

Or with its shortest alternative:

mstest testResultsFile:"**/*.trx", keepLongStdio: true

 

Code Coverage Support

Since the code coverage data has a binary, proprietary format, and that the tools capable of handling them are released under a proprietary license along with the develipment environments, you will have to perform the data conversion yourself. Here's how.

Code Coverage Data Conversion

...

Code Block
titleCoverageCoverter.exe
     class Program
    {
        static int Main(string[] args)
        {
            if ( args.Length != 2)
            {
                Console.WriteLine("Coverage Convert - reads VStest binary code coverage data, and outputs it in XML format.");
                Console.WriteLine("Usage:  ConverageConvert <sourcefile> <destinationfile>");
                return 1;
            }

            CoverageInfo info;
            string path;
            try
            {
                path = System.IO.Path.GetDirectoryName(args[0]);
                info = CoverageInfo.CreateFromFile(args[0], new string[] { path }, new string[] { });
            }
            catch (Exception e)
            {
                Console.WriteLine("Error opening coverage data: {0}",e.Message);
                return 1;
            }

            CoverageDS data = info.BuildDataSet();

            try
            {
                data.WriteXml(args[1]);
            }
            catch (Exception e)
            {

                Console.WriteLine("Error writing to output file: {0}", e.Message);
                return 1;
            }

            return 0;
        }
    }

The CoverageDS and CoverageInfo types are being exposed by the Microsoft.VisualStudio.Coverage.Analysis.dll (official documentation: https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.coverage.analysis.coverageds.aspx, article explaining how to use these assemblies with a nice step-by-step: https://blogs.msdn.microsoft.com/phuene/2009/12/01/programmatic-coverage-analysis-in-visual-studio-2010/

The following Powershell build step will find one binary coverage data file in your workspace and convert it to XML format, assuming you use the default TestResults directory. (Modify as necessary to handle multiple coverage files)

Code Block
titlePowershell build step for CoverageDS XML Conversion

$generatedCoverageFile = $(get-ChildItem -Path .\TestResults -Recurse -Include *coverage)[0]
CoverageConverter $generatedCoverageFile TestResults\vstest.coveragexml

In every case, the MSTest plugin will find this file and convert it to the EMMA format.

If you have enabled the property 'Enable Display of Code Coverage Results' in the plugin configuration section, the MSTest plugin will display a code coverage trend/code coverage report via the EMMA plugin.

Can I use Microsoft's CodeCoverage.exe for data conversion?

Microsoft supplies C:\Program Files (x86)\Microsoft Visual Studio 1X.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe with Visual Studio. This does not produce But unfortunately this tool uses an XML format distinct from the CoverageDS XML format used by vstest and mstest, and is not compatible with the MSTest plugin. If you are interested in this topic, and you've some experience with XSL, there are a couple of transforms to convert back and forth in the github repository.

Change Log

Version 0.20 (September 1st, 2017)

Version 0.19 (September 1st, 2015)

  • Support for web tests (contacted by email, by Peter Barnes. No Jira issue has been opened)
  • Let the users still using Java 1.6 to continue using the plugin [JENKINS-29032]
  • Mark the inconclusive tests as 'skipped' [JENKINS-29316]

...

  • Add a checkbox to ignore missing TRX files (Thanks Christopher Bush, pull request  #7). The pull request contains also a way to automate job creation using the REST API. So, if you're automating the creation of your jobs, just specify failOnError=on to enable this feature. Any other value than 'on' will set this option to false.
  • Fix the code coverage calculations (Thanks junshanxu, pull request #6): a sum over all the nodes is better than using the value of the first node only.

...