Child pages
  • Jenkins Agent on DalvikVM in Android

Due to some maintenance issues, this service has been switched in read-only mode, you can find more information about the why

and how to migrate your plugin documentation in this blogpost

Skip to end of metadata
Go to start of metadata

Please Help

This is a work in progress. Please help by adding a comment, or making contributions or edits to this wiki doc. Please also see the jenkinsci-dev mailing list thread.

Overview

While compiling code on an Android device does not make any sense, driving tests on Android devices does. The Android Emulator Plugin by Christopher Orr exploits multi configuration jobs to test Android applications on many different screen sizes and configurations. Unfortunately, the Emulator only emulates ARMv5 and it lacks a real GPU. This means that some software must be tested on bare-metal hardware. Running the agent jar on a host PC machine to communicate with USB devices introduces undesired complexity and, as some of you may know, the Android Debug Bridge has some issues, especially while under a stressful load.
Currently our solution employs multiple pc agents each with multiple usb adb connections to many devices. The pc agents send commands to the phones over a tcp connection established using busybox, netcat, and bash. To prevent more than one job using the same phone at a time the devices are locked as a "busy resource" using an external web application.
This configuration ads more complexity and maintenance than is needed by duplicating much of the jenkins distributed execution core. Ideally we could leverage all of the work that has already been implemented in Jenkins by running the agent jar directly on the Android device (or emulator).

Approach

Convert the agent jar to Dalvik bytecode and launch it from adb shell or include it in an Android application and launch it from an Android Service. This worked swimmingly until the agent attempted to download java classes from the master using a RemoteClassLoader.

Issues

Agent downloads classes dynamically

A different set of classes can always be remotely loaded by any client. Some classes come from plugin configurations.

Dalvik bytecode

A major issue is that Android does not run native java bytecode. All bytecode must be converted to Dalvik bytecode before being executed on the Android device. The Apache Felix Project has an excellent article about deploying the OSGi client to Android which details how to convert java bytecode to dex bytecode at compile time. We need to be able to do this at runtime.

Not the whole Runtime

The following packages are not supported in Android:

  • java.applet
  • java.awt
  • java.beans
  • java.lang.management
  • java.rmi
  • javax.accessibility
  • javax.activity
  • javax.imageio
  • javax.management
  • javax.naming
  • javax.print
  • javax.rmi
  • javax.security.auth.kerberos
  • javax.security.auth.spi
  • javax.security.sasl
  • javax.swing
  • javax.transaction
  • javax.xml (except javax.xml.parsers)
  • org.ietf.*
  • org.omg.*
  • org.w3c.dom.* (sub-packages)
    From http://www.zdnet.com/blog/burnette/java-vs-android-apis/504. Fortunately there are work-arounds to this issue.

Our Own Custom RemoteClassLoader

It seems that the most reliable way to run the agent jar on an Android device is to create an overloaded ClassLoader that converts the java bytecode to Dalvik bytecode before attempting to load it.
This question has been asked before on different ways to load java bytecode into DalvikVM at runtime. It seems that the Dalvik team does have it on their radar but as Jesse Wilson explains in his Stack Overflow post:

The Dalvik team would like to build a first-class runtime code generation library. We're tracking the feature request as Android bug 6322. Unfortunately, we have a very long list of performance and correctness issues, so I can't give you a timeline for when we'll spend time on this issue.

There are some alternatives, but they will all take some work:

  • Run your application on a standard JVM and exercise all runtime code generation there. Dump the .class files from memory to files, and then run dx on those files. If you're quite sophisticated, you could integrate all of this work into your build.
  • Include the open source dx tool as a project library, and execute it programatically from within your application, possibly in your application's classloader. This will bloat your application's binary.

The Implementation

I like the second option and dx.jar comes with the Android SDK. Upon reviewing the source it appears that most of the work is done by com.android.dx.dex.cf.CfTranslator.translate(). There is some example code that converts java classes to dex at runtime over at Jythonroid: https://jythonroid.googlecode.com/svn-history/r34/branches/Jythonroid/src/org/python/debug/FixMe.java

It seems to me that the most elegant solution would be to update hudson.remoting.RemoteClassLoader to do a System.getProperty() check against java.vm.name to check for "Dalvik" and then insert the branch to the conversion logic at runtime.

  • No labels

3 Comments

  1. Unknown User (cristiano)

    Hi, I like the possibility to have an "android-native" Jenkins agent, it would be simplify the management of attached devices without having to rely on in-between nodes.

    While ardently waiting for it to come, I am trying to achieve it in the meantime by using an OpenJDK build for Android, which can happily launch the Jenkins agent.

    I am not sure this is the right place to ask for help on this topic... I hope it's fine (big grin)

    My target is a HiKey 970 with Android. Jenkins can deploy and launch the agent but, unfortunately, after the Jenkins master sees that the agent is online and tries to do something with it, the JVM on the target crashes:

    [...]
    Slave.jar version: 3.10.2
    This is a Unix agent
    Evacuated stdout
    Agent successfully connected and online
    # To suppress the following error report, specify this argument
    # after -XX: or in .hotspotrc:  SuppressErrorAt=/os_linux.cpp:4505
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  Internal Error (/Users/akikazu/workspace/Projects/OpenJDK/openjdk/hotspot/src/os/linux/vm/os_linux.cpp:4505), pid=12324, tid=12340
    #  assert(rc == 0) failed: clock_gettime is expected to return 0 code
    #
    # JRE version: OpenJDK Runtime Environment (9.0) (fastdebug build 9-internal+0-adhoc.akikazu.openjdk)
    # Java VM: OpenJDK Server VM (fastdebug 9-internal+0-adhoc.akikazu.openjdk, mixed mode, serial gc, linux-)
    # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    #
    # An error report file with more information is saved as:
    # /data/workspace/hls/hs_err_pid12324.log
    #
    # If you would like to submit a bug report, please visit:
    #   http://bugreport.java.com/bugreport/crash.jsp
    #
    Current thread is 12340
    Dumping core ...
    Aborted 
    ERROR: Connection terminated
    java.io.EOFException
    	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2638)
    	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113)
    	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
    	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
    	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
    	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
    	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
    Caused: java.io.IOException: Unexpected termination of the channel
    	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
    Slave JVM has terminated. Exit code=134
    [05/09/19 13:48:09] [SSH] Connection closed.


    Has anyone gone through the same already? Or is there anyone who could help?

    Thanks in advance!

    1. Unknown User (slide_o_mix)

      I think you would probably be better served by posting to the mailing list. Also, Jenkins won't run with JDK9, you would need JDK8.


      1. Unknown User (cristiano)

        Thanks! I'll do.