Plugin Information |
---|
Distribution of this plugin has been suspended due to unresolved security vulnerabilities, see below. |
The current version of this plugin may not be safe to use. Please review the following warnings before use:
This plugin is up for adoption. Want to help improve this plugin? Click here to learn more!
This plugin allows to copy a set of files, from a location somewhere on the master node, to jobs' workspaces. It also allows to copy files back from the workspaces of jobs located on a slave node to their workspaces on the master one.
About this plugin
This plugin allows:
- To copy, from a location somewhere on the master node, to the workspace of a job being built, a set of files that are required for the good execution of the job. This is useful, for example, when not all the build scripts (Ant build files, etc.) are in the SCM.
- To copy, from the slave node running the job, back to the master node, a set of files that are required for some post-processing (e.g. test results).
User guide
Copy files into the job's workspace before building
To activate this plugin for a given job, simply check the Copy files into the job's workspace before building checkbox in the Build Environment section of the job. You then get the following fields:
- Files to copy is where you define the files that have to be copied to the workspace of the job before the build actually takes place. If the files to copy have previously been copied (in a previous build), they are overwritten.
- Paths are relative to is used to define where the files specified in Files to copy come from. There are three choices (cf. screenshot below). Using
HUDSON_HOME/userContent
is highly recommended.
The Advanced section offers the following additional fields:
- Files to exclude from the copy allows to define which files, that have been taken into account by the Files to copy field, must not be copied.
- Flatten directories is used to flatten the directory structure, if applicable.
- Include Ant's default excludes is used to specify that Ant's default excludes (e.g.
.svn
folders, etc.) must also be copied.
Kindly refer to the inline help for more information.
Copy files back to master node
To activate this plugin for a given job, simply check the Copy files back to the job's workspace on the master node checkbox in the Post-build Actions section of the job. You then get the same two fields as for the Copy files to slave node before building section (note that label in the screenshot is old):
Version history
Version 1.4.4 (02/23/2015)
- Fixed ISSUE#25346 which addresses the workspace issue where files copied back to master are not in correct path
Version 1.4.3 (09/28/2013)
- Fixed ISSUE#18899 by adding an option to select whether to run copying files back to master after result is finalised or along with other post build tasks. (See the updated screen shot above)
- Plugin upgraded to Jenkins 1.424
- Sorry about versions 1.4.1 and 1.4.2, they failed while performing release as I am new to jenkins,maven and git.
Version 1.4 (01/17/2012)
- Implemented JENKINS-10691: Added a new Override destination folder field to override destination folder on master node
- As part of JENKINS-12281, added a new section in the main Jenkins configuration screen to specify an additional Paths are relative to option
- Fixed JENKINS-12281: It is now possible to remove
HUDSON_HOME
from the list of base locations for copy to slave operations (from the new section in the main Jenkins configuration screen) - Fixed a typo in the help message of the Files to copy field
- Plugin upgraded to Jenkins 1.409
Version 1.3 (03/02/2011)
- New Paths are relative to option offering more possibilities to specify the root location of files to copy (three possibilities:
HUDSON_HOME/userContent
,HUDSON_HOME
or the jobs' workspace) - Refactored UI to emphasize the new Paths are relative to field
- Changed labels for easier understanding
Version 1.2.3 (12/13/2010)
- New Flatten directories option (see JENKINS-8220)
- Fixed JENKINS-8274: An InvalidClassException was thrown when using copy-to-slave from Linux to Windows
- Fixed JENKINS-8155: It's now possible to copy to an AIX, HP-UX or Linux S/390 node
Version 1.2.2 (11/11/2010)
- Fixed JENKINS-7021: It is now possible to copy files relatively to
HUDSON_HOME
rather than the workspace of jobs - Fixed JENKINS-7999: Ant's default excludes are now taken into account
- Plugin upgraded to Hudson 1.375
Version 1.2.1 (09/01/2010)
- Fixed JENKINS-6045
Version 1.2 (03/08/2010)
- The use of the plugin is no more restricted to free-style projects
Version 1.1 (02/01/2010)
- New Copy back to master notifier which allows copying data back to the master once the build ends
Version 1.0 (10/27/2009)
- Initial release
51 Comments
Unknown User (mdmillerus@gmail.com)
I need a little help (more information) to get this plug-in working. I have two machines, an AIX Hudson master and a Windows machine running XP as the Hudson slave. The slave agent is installed as a service. It works great. I actually have the slave machine building my Java docs. So the setup works, but I am unable to copy any files from the master machine to slave using this plug-in. Here's what I do:
Step 4 is when I start running into trouble. When I specify "/test/build.xml" the edit box complains it can't find the file. I've verified the file exists. I even copy-pasted the path from the console window to make sure it's not a typo. Hudson edit box says it can't find the file. Ok, fine, I ignore it and run the job anyway. The job reports success... but there is no file copied over, and I don't see anything in the job console output showing a file being copied over, but the plug-in also doesn't complain that it's not finding any files. I'm using plug-in 1.2. I'm using Hudson 1.348.
The help text for each option is confusing. The help says:
"When a build takes place on a slave node, copies a set of files from the master node to this slave node. This is useful, for example, when some files required for the build to take place are not present in the SCM and are located direclty in the project's workspace.
Of course, no copy will take place if the build is run on the master node."
"Comma- or space-separated list of patterns of files/directories to be copied from the master node to the slave one. See the @includes of Ant fileset for the exact format. The base directory is the workspace.
E.g.: *.xml,*/.properties"
The part that's confusing me is that when a job is tied to a slave node the "workspace" is now on the slave machine, not local to the master, so any file paths using the workspace as the base wouldn't work since the files I need to copy aren't on the slave, obviously. I'm thinking this is just bad text, or maybe I'm missing something obvious about the wording.
So here is what I do know.
If you could provide any insight as to what I'm doing wrong I'd really appreciate it. I've systematically tried just about everything I can think of and have been completely unsuccessful at getting the plug-in to work for me.
I appreciate the help.
Mark
Unknown User (vlatombe)
I agree with the previous comment, I have no clue about where the files to copy to the slave are taken from.
Unknown User (rseguy)
When you activate this plugin for job A, then the Files to copy field is expected to contain a set of folders/files that are relative to the workspace of the job (there is a link in to this workspace in the inline help), that is, that are relative to the folder
<HUDSON_HOME>/jobs/A/workspace
(as of Hudson 1.312). E.g., if the files to copy are named*.xml
and are located in<HUDSON_HOME>/commonFiles
, then the field must be set to../../../commonFiles/*.xml
.Unknown User (vlatombe)
Hello,
thanks for the precisions, I have been able to copy a file that sits in the workspace. However, couldn't manage to copy an xml file that is inside HUDSON_HOME. In my case, I've tried using ../../../clean.xml (the file sits in HUDSON_HOME) and it doesn't seem to be copied to the slave.
Furthermore, even when it works, it could be useful to get a log of what was copied (at least the number of files). Currently, I get nothing in the output.
Unknown User (rseguy)
Try to manually create the
workspace
folder on the master, it should help I think. For the next release I will add the creation of the workspace to avoid this pbm.The plugin has some logging capabilities, you need to set the log level to FINEST.
Unknown User (vlatombe)
Ok, I now have the logs, but I still cannot reference any file that is in parent directories. Thanks for the help anyway.
Unknown User (rseguy)
That's weird... May you send me the details of you config (Hudson version and job's
config.xml
) on my mail so that I can investigate on that?Unknown User (carl.quinn@gmail.com)
I am using Hudson 1.353 and the latest version 1.2 of this plugin and I also see a failure in its ability to get to files above the jobs workspace. Even with the workspace manually created.
Is it possible that Hudson core is preventing it from accessing files above the workspace? Or maybe this plugin is somehow being executed on the slave and not the master?
I am trying to use this plugin to push a build artifact from one job built on a local slave, to another job that builds on an EC2 instance which doesn't have access to our network. This plugin seemed like just the right tool to do the push...
Unknown User (rseguy)
The 1.2.2 release adds a new option which allows specifying the files relatively to
HUDSON_HOME
rather than the workspace. HTH.Unknown User (gabrielbaez)
It would be awesome if this plugin cloned the work space from the master to a slave for building.
For example in my environment we cant install clearcase on linux machines and there are some applications that need to be build from a linux environment. My hudson install it's on a windows server, if I could checkout the code from the windows hudson and then send that to the slave that would be great!
Unknown User (rseguy)
You should take a look at the Clone Workspace SCM Plugin for that.
Unknown User (gaccardo@slb.com)
I'm having exactly the same issue as Vincent Latombe. I can specify ** as the files to copy to the slave, and get every
file in the workspace. However, if I specify "../**" or any relative path, I get nothing. I'm running Hudson 1.355 on a
Windows 2003 system.
Any assistance resolving this issue would be appreciated.
Unknown User (rseguy)
Kindly open an issue in JIRA for that an assign it to the proper component (you may need to request through the dev mailing list for the copy-to-slave component to be created, I don't think it's been created yet).
Unknown User (fredg)
I opened a bug for this: http://issues.jenkins-ci.org/browse/JENKINS-7021
Please vote for it.
Unknown User (rseguy)
Fixed in 1.2.2: It adds a new option which allows specifying the files relatively to
HUDSON_HOME
rather than the workspace.Unknown User (crasierwind@hotmail.com)
Hi all:
what's the execution order between copy-to-slave and groovyo-postbuild plugin ?
I want to run copy-to-slave first,then groovy-postbuild ,but since I install groovy-postbuild plugin ,then install copy-to-slave plugin ,Hudson always run groovy-postbuild first.
somebody help me ? :(
Hudson version: 1.383
copy-to-slave: 1.2
groovy-postbuild: 1.3
Unknown User (rseguy)
Hi,
What is your exact requirement? I guess that your need it to process the file(s) that the build has produced using some Groovy stuff. If yes, couldn't you do it on the slave side before using the "Copy back to master" option?
I ask that because the order of the execution of the plugins can't be guaranteed unless some special care is taken at the source code level. But, I won't implement that in the copy-to-slave plugin since the requirements will never be the same depending on users.
Unknown User (hbockelmann)
Hey you, I want to use this pretty cool plugin to copy some generally needed scripts from master to several slaves.
The files are located in $HUDSON_HOME/userContent/* and the upload to slaves work quite good (at least on my Linux slaves ... there are some problems on AIX slaves btw).
What I now see is that on the slaves the files get copies to $WORKSPACE and the relative path from the master is preserved, ie. on the slave a new directory userContent is created. I think it would be a fine option for the plugin to be able to set the path on the slave (relative to $WORKSPACE) within the job configuration.
Unknown User (rseguy)
Hi Hendrik,
I understand your need, but it raises a question (which is why I didn't implement such a functionality yet): How the handling of several different paths to copy (e.g.
userContent/*/, myScripts/*/
) would be done? In case you copy just one folder, then ok, it can be easily implemented in the plugin by adding some kind of amv
command. But in case of several folders, that's far more complicated than that.What about having a "flatten" option which would just, well, flatten everything in the workspace directory?
Unknown User (hbockelmann)
Hey Romain, thanx for quick response. imho it would be a nice option if I could set a "flatten" flag that ensures all files from different directories on master to be copied directly into $WORKSPACE of the slave.
Maybe there could be a dropdown-menu which says: 'keep directory structure of master' or 'copy all files to $WORKPACE'
Unknown User (rseguy)
Ok, let's go for the "flatten" option. Please open an issue in the JIRA issue tracker, it'll be automatically assigned to myself and I'll be sure to not lose track of it (as I won't do it today).
Unknown User (prenaud76)
This plugin is interesting. However, in the case of a maven2 project I noticed it runs after the pre-build steps, which is too late for what I would need it to. Is there a way to bump its execution order to occur before the pre-build steps?
Unknown User (rseguy)
I'm afraid the answer is no...
Unknown User (prenaud76)
Hi Romain,
But you do realize that it makes this plugin rather useless for maven2 jobs, do you? Or should it say: it makes it useless for maven2 jobs who need the files copied by the plugin during the pre-build steps.
If it is really impossible to run the plugin before the maven pre-build steps as you indicated, would you be able to suggest an alternate path to do what I need to do? In essence, the pre-build step need to run on files copied by the copy-to-slave plugin (thus files purposely not in SCM). Otherwise, it seems that my only choice is to abandon using this plugin, and that would not be my first choice. This is a nice plugin that fits well in a build process, but executing at the wrong time.
Unknown User (rseguy)
Hi Patrick,
One first thing, can you confirm I'm right if I say that what brings pre- and post-build steps to your Maven jobs is the M2 Extra Steps plugin? I'll assume the answer is yes.
This plugin adds pre-build steps by using the notion of build wrapper. The Copy To Slave plugin does exactly the same. The bad point with build wrappers is that there's currently no way to control the order in which they are executed (although this could actually be done by altering the behavior of Hudson core). That's why my answer is no, and that's also why versions 1.0 and 1.1 of the plugin were purposely not usable for Maven jobs.
Unknown User (prenaud76)
Salut Romain,
OK, now I understand! Tx very much for the explanation. Is it in your plans to upgrade the plugin eventually so we could use it for maven jobs that way? No rush, just asking...
Tx again!
Unknown User (rseguy)
I'm not a Maven user + Hudson is currently under heavy changes with the move to Maven 3, so I don't feel I can do this by myself.
EDIT: Cf. comment below.
Unknown User (rseguy)
Patrick, I really don't know why I didn't think about that before, but what if I propose the copy feature through a build step in addition to the current build wrapper? I think it could fit the need you describe + I should be able to easily implement it.
If you're ok with that, please open a JIRA issue and I'll do the work (I don't give a planning of course, I'll implement that on my spare time).
Unknown User (dtka)
I can copy singular files, but not directories. This is my tree:
The console informs me as following:
but nothing has been copied.
It does work if I copy the readme.txt file in that directory. What am I missing?
Unknown User (rseguy)
Try with
testdir/**/*
. This specifies that you want the files inside testdir to be copied, recursively.Unknown User (dtka)
Thank you for a quick reply!
I would like to copy only a subdirectory of workspace directory in my example above. Can I do that so that <HUDSON_HOME>/userContent/workspace/subdir will be copied to <JOB_WORKSPACE_SLAVE>/subdir?
When I am specifying workspace/subdir/**/* I am getting the following folder structure on the target: <JOB_WORKSPACE_SLAVE>/workspace/subdir.
Unknown User (rseguy)
That's technically not possible to do that (nor implement that) in the plugin. You'll need to do some Shell or Ant scripting.
Unknown User (pfalcon)
Isn't anyone, including author, sees that plugin is a security hazard? Jenkins allows to have access separation, where only users with admin permissions haev access to global config. However, with this plugin installed, any user who has job create/edit permission can fetch Jenkins' config.xml, and that may contain sensitive data, like EC2 credentials for example.
That's why carefully written plugins split there config in 2 parts: global config where arbitrary paths, credentials, etc. are set, job config, where usually user can just select a choice from predefined set using a dropdown.
Suggestions for alleviating immediate security threat: remove "HUDSON_HOME" option, replace it with "HUDSON_HOME/copyToSlave" option. The latter option is still required to preserve generality of plugin, as "HUDSON_HOME/userContent" is not suitable for all content, as it is publicly accessible via http://<jenkins>/userContent. "Job's workspace on master" on its own is not enough either, as for the case of hundreds of jobs all requiring same prerequisites, it's very cumbersome to configure each job's workspace.
And well, this assumes that you didn't try to patch Ant's file pattern code which by all means should prohibit ".." syntax to access files outside of the specified prefix, because otherwise this plugin can be used to access any file on master's filesystem which is very big security risk.
Unknown User (rseguy)
I'm aware of that and I agree, but I just don't have time to work on that. Don't hesitate to submit patch (and create JIRA issues), I'll review and incorporate it.
Out of scope: If you're concerned about security, there are many other places in Jenkins that you should check. For example, some path validation mechanisms (I'm referring to these tips that may appear below fields when keying some wrong values) allow to list the content of the master filesystem. Also, the Groovy console is a threat. Etc. Jenkins has to be used carefully, by trusted users if security is required.
Btw, shouldn't sensitive credentials be stored in a vault (e.g. a keystore) rather than in a plain file?
Unknown User (pfalcon)
Thanks for quick reply and agreeing that this is an issue (I may imagine that folks using Jenkins behind firewalls with 2-3 people in teams wouldn't even consider it such, but Jenkins is more than that).
Well, I'm not sure what usecase would be for something exactly *in* HUDSON_HOME. Per comments above, people sometimes want to push result of some other job. Well, that calls for adding HUDSON_HOME/jobs . That's not ideal either (can bypass project-based matrix security, etc.), but having HUDSON_HOME is just immediate security issue.
Having global/per-job config split is of course the ideal solution, but I'm personally not so knowledgeable about Jenkins and plugins to do that ;-(.
I bet. Jenkins core at least gets regular updates and attention, but plugins may indeed cause issues. But the only way to deal with that IMHO is to take such issues seriously, not thinking "well, that unlikely will apply in practice". Because first you use it with team of 2-3 behind the firewall, then give access contractors from another part of the world, then to customers/community...
Well, yeah, one of selling point of Jenkins may also become its curse. Some plugins attentively offer both "paste" and "set path of a file" fields for such data, but some go with just "paste" ones w/o any choice (and yes, that's an issue too).
So, I opened https://issues.jenkins-ci.org/browse/SECURITY-21 for this, and would for sure post patch for just disabling HUDSON_HOME. Not sure if I could go further though ;-(.
Unknown User (rseguy)
I don't have access to this issue (only you and KK have, I guess). May you please cancel it and open another one in the JENKINS project? Thanks.
Unknown User (pfalcon)
Happy New Year! Sorry for the delay due to holidays. Resubmitted as https://issues.jenkins-ci.org/browse/JENKINS-12281 . There's also a fork on github which implements a simple measure (removing HUDSON_HOME from choices): https://github.com/pfalcon/copy-to-slave-plugin
Unknown User (rseguy)
I've attached to the JIRA issue a version of the plugin (1.4-SNAPSHOT) which implements the fix. Can you give a try please? Thanks.
Unknown User (pfalcon)
Very nice, thanks for quick turnaround and adding an admin section for configuration. Looking forward to a release soon!
Unknown User (miked)
Would be great to use this also as a build step and not only pre or post build.
This is useful for cases like you want to copy the build log file to the slave and do something with it, or for example, i had an issue where i needed to view scm-polling.log file on the slave, and it appears only on the master. this file only updates after build starts so i couldnt use the plugin to copy it.
Unknown User (rseguy)
Please create a JIRA issue for that.
Unknown User (mako)
Hi,
I've found a bug with permissions, I've tried to copy .sh script to workspace and plugin copied it as text file however I've made it executable (chmod 755)
Unknown User (rseguy)
See JENKINS-8249.
Unknown User (sangi0688)
Hi,
I can see only options like copy files from master to slave before building and slave to master after build.
Is there a way to copy files like build.xml which is generated after the build from master to slave?
Thanks,
Sangeetha.
Unknown User (joshuakwan)
I was trying this plug-in with copying the default readme.txt file to a slave but hit this error:
Building remotely on 10.8.26.213 in workspace /opt/jenkins/workspace/Test_deploy_build_to_slave
copy-to-slave Copying 'readme.txt', excluding nothing, from 'file:/C:/Users/Administrator/.jenkins/userContent/' on the master to 'file:/opt/jenkins/workspace/Test_deploy_build_to_slave/' on '10.8.26.213'.
FATAL: null
java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Unknown Source)
at java.util.regex.Matcher.reset(Unknown Source)
at java.util.regex.Matcher.<init>(Unknown Source)
at java.util.regex.Pattern.matcher(Unknown Source)
at hudson.plugins.xshell.XShellBuilder.perform(XShellBuilder.java:78)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1502)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
Any idea on this? Thanks a lot!
Unknown User (kalpsat)
I am trying to copy a file from another master job's workspace(upstream) to slave(note both master and slave are windows boxes)
Files to copy specified as "../jobs/testproj/TestData/TestData.xml"
Paths are relative to selected as "$JENKINS_HOME/userContent"copy-to-slave Copying '../jobs/testproj/TestData/TestData.xml', excluding nothing, from 'file:/C:/.jenkins/userContent/' on the master to 'file:/C:/Jenkins/Slave/workspace/testslaveproj/workspace/' on '<slave machine>l'.
This is not working.
but readme.txt copied fine from userContent directory. what am i doing wrong?
Also, want to specify the user input variable in the Files to Copy like ../jobs/testproj/TestData/$file.xml, this does not work unless i specify a quotes around "$file", but it also adds quotes around the $file value. is it possible to use job variable in Files to Copy section?
Unknown User (tspmap)
I'm running into a problem with a Jenkins server no Windows. The specified file is not being copied to the slave. I think it has to do with spaces in my Jenkins home dir. Notice the %20 substrings in the Jenkins home directory:copy-to-slave Copying 'logParsingRules/.', excluding nothing, from 'file:/C:/Program%20Files%20(x86)/Jenkins/userContent/' on the master to 'file:/users/tparker/jenkins-slave/workspace/SHAPE-linux/' on 'estragon'.
Unknown User (idaband)
I think I missed something.
I have project A. I didn't use the standard workspace. So... Jenkins is install on the C Drive... but project A's workspace is D:\blaa
Now i'm making a new project (job) Project B which will run remote. I need it to copy the files from d:\blaa to -> remote:\ Project B
I'm copying nunit dll's that were produced in one project and trying to farm out nunit to run on a bunch of computers...(long story)
Anyone have thoughts?
Unknown User (swift)
Hello I am using Copy to slave plugin for copying file from slave to master. I have android slave running and copying *.apk to master.
so where to see my file has been copied ? can anyone help me ?
Thanks.
Unknown User (yrsurya)
HI,
I am able to copy workspace from Docker executor to master for regular jobs but is there any way how to configure this plugin for pipeline jobs
please let me know if any solution for this
Thanks
Unknown User (adityachs)
@Romain Seguy, @ Sergey Shelomentsev,
Thanks for this plug-in. I was looking for such a nice plugin. I am facing some difficulties in using this plugin. When I use the suggested Jenkins-UserContent folder, the copy is working like charm, but if I provide a custom location on the same machine, it is not working.
See the below out put :
But it didn't copied any files
You can see the bext.txt on the home directory
My requirement is :
Jenkins runs on X machine (Linux)
Code/artifacts on Y machine (Windows, Jenkins is not building these artifacts)
Deploy to Z machine (windows).... So, I am planning to use this plugin for copying files from Y to Z. So, if this plugin cant access directories other than Jenkins userContent Folder, I fail to achieve my plan.
Please help.