Palladio Build Infrastructure

From SDQ-Wiki
Jump to: navigation, search

Documentation restructuring

This page is both, unstructured and outdated. The following items should be addressed:

* Start with integration scenarios and describe each by example
* Describe the Palladio Build Infrastructure Buckminster including the federated rmaps
* Describe the parameters provided by the Palladio Buckminster rmaps
* Describe how to create a new build job on the SDQ jenkins build server
* Describe migration to GitHub and building GitHub projects on Jenkins
* Describe the central Palladio Update sites (for releases and for nightly builds)
* Best would be to have a systematic central entry page - best would be to have this one
* Have subpages for the detailed information outlined above

Jenkins

We use a common infrastructure to build the palladio core, addons as well as additional palladio standalone components, such as the workflow engine or ginpex. The Jenkins webfrontend can be accessed at: https://sdqweb.ipd.kit.edu/jenkins

Palladio Build Infrastructure/Troubleshooting

Integrating New Projects

Build Tools

Tools we use to setup and run our builds and perform quality checks.

Buckminster

Buckminster Install in Eclipse

To use Buckminster inside your Eclipse IDE, you should install it from this update site: http://download.eclipse.org/tools/buckminster/updates-4.2

You will need not all of the provided features but at lease the Buckminster Core, PDE Support, and subversive (or subclipse) SVN support. Note that you should have installed the svn support (including svn connectors) before you try to run a buckminster process.

Buckminster provides a feature called materialize a build project. We setted up our builds so that one can checkout a buckminster project, open the included cquery file with the buckminster editor and click materialize to install the dependencies and checkout the development projects to start working with the projects source code.

Continuous RCP builds with Buckminster

While Buckminster does not directly have a feature for automating the creation of binary/product RCP builds, it is possible to achieve this using Ant Build Scripts that then execute the necessary tasks on the p2 director. Please refer to http://www.lorenzobettini.it/2013/01/building-an-eclipse-rcp-product-with-buckminster/ for a tutorial on this.

It is highly recommended not to use Eclipse's .target platform definition mechanism as it offers little to no control over versioning etc. of the plugins and features materialized to the target platform. Instead, create a Buckminster project dedicated to materializing the Target Platform: http://www.lorenzobettini.it/2012/11/materializing-and-provisioning-your-target-platform-as-local-p2-site-with-buckminster/

Troubleshooting

Problems with Binary Builds

When trying to figure out what causes problems with integrated ant scripts it is recommended to increase ant's log level to debug. To do this in a Jenkins build job, set the Equinox launcher arguments of the 'Run Buckminster' task to --loglevel info,ant=info .

If the binary build does not execute successfully and results in an error message similar to the following, the likely cause are conflicting or outdated version requirements in the target platform materialisation.

[ant] Installing eu.cactosfp7.cactoscale.cdomodelgenerator.updater.product 1.0.0.
[ant] Installation failed.
[ant] Cannot complete the install because one or more required items could not be found.
[ant]  Software being installed: CDO Model Generator 1.0.0 (eu.cactosfp7.cactoscale.cdomodelgenerator.updater.product 1.0.0)
[ant]  Missing requirement: Core Runtime 3.10.0.v20140318-2214 (org.eclipse.core.runtime 3.10.0.v20140318-2214) requires 'bundle org.eclipse.equinox.common [3.6.100,4.0.0)' but it could not be found
[ant]  Cannot satisfy dependency:
[ant]   From: eu.cactosfp7.cactoscale.cdomodelgenerator.feature 1.0.0.201503241905 (eu.cactosfp7.cactoscale.cdomodelgenerator.feature.feature.group 1.0.0.201503241905)
[ant]   To: eu.cactosfp7.infrastructuremodels 1.0.0
[ant]  Cannot satisfy dependency:
[ant]   From: CDO Model Generator 1.0.0 (eu.cactosfp7.cactoscale.cdomodelgenerator.updater.product 1.0.0)
[ant]   To: eu.cactosfp7.cactoscale.cdomodelgenerator.feature.feature.group [1.0.0.201503241905]
[ant]  Cannot satisfy dependency:
[ant]   From: CACTOS Infrastructure Models 1.0.0.201503111454 (eu.cactosfp7.infrastructuremodels 1.0.0.201503111454)
[ant]   To: bundle org.eclipse.core.runtime [3.10.0,4.0.0)
[ant] Cannot complete the install because one or more required items could not be found.
[ant] Software being installed: CDO Model Generator 1.0.0 (eu.cactosfp7.cactoscale.cdomodelgenerator.updater.product 1.0.0)
[ant] Missing requirement: Core Runtime 3.10.0.v20140318-2214 (org.eclipse.core.runtime 3.10.0.v20140318-2214) requires 'bundle org.eclipse.equinox.common [3.6.100,4.0.0)' but it could not be found
[ant] Cannot satisfy dependency:
[ant] Cannot satisfy dependency:
[ant] Cannot satisfy dependency:

This issue cannot be resolved or avoided by introducing version constraints in the Target Platform content instructions defined in the *.cspec file of the Buckminster target platform project. Instead, it is necessary to introduce an advisor node to the *.cquery of the target platform project:

<cq:advisorNode namePattern="org.eclipse.equinox.common*" componentType="osgi.bundle" useMaterialization="false" useTargetPlatform="false" useWorkspace="false"/>

Make sure that the bundle or feature that was not materialized (in this case bundle org.eclipse.equinox.common is part of your *.cspec as it will no longer be materialized to the target platform based on dependencies. This advisor node prevents the use of any features or bundles as part of the target platform materialization process that were not explicitly declared in the *.cspec.

Referenced plugins are not resolved on the deployed p2 site

Check the content.xml contained in the content.jar of the update site to see if all update sites that should be visited are specified here. If not, make sure you did not specify the referenced site's URL to use the HTTPS protocol. URLs need to be specified to follow the HTTP protocol, otherwise they will go missing somewhere in the construction of the p2 site.

JUnit Tests

JUnit tests and JUnit Plug-In tests are automatically run in every build by Jenkins. To ensure that the tests are detected by the build script, an Eclipse launch configuration file ending on .tests.launch needs to be present in the root folder of the project. In every build, a shell script (create_test_launch_scripts.sh) is run prior to the actual build which scans every directory in the workspace for files ending on .tests.launch. Note that subdirectories of projects are not checked! The script then generates an EMMA command for every launch script and stores the command in a text file. This textfile is imported into a buckminster build step later on. The test results are recorded and published automatically on the Jenkins frontend. The latest test results can be found at https://sdqweb.ipd.kit.edu/jenkins/job/Palladio%20Core/

Emma Code Coverage and ANTLR
Emma and ANTLR generated code are not smoothly compatible. To handle unit tests which run against code partly generated with ANTLr, it is recommended to exclude this generated code from the code coverage detection. It is also recommended to place the ANTLR generated code in a specific package and/or source directory to be able to exclude only this generated code from beeing checked by emmma. A typical exception thrown if this is not taken into account looks like:

java.lang.NoSuchMethodError: com.vladium.emma.rt.RT.S([[ZLjava/lang/String;J)V

Note: it is not yet checked how this code can be excluded. A possible way to check is the usage of the following parameter described in the Buckminster documentation:

-D «key»[=«value»] , --define «key»[=«value»]
Defines a property as a key=value pair. The value may include ANT-style expansion constructs
that will be expanded using both system properties and other properties that has been set. 

Launch Configuration Collecting Script

Below, the build shell script located on the build server to search all launch configuration files and build the comprehensive buckminster command script is shown:

#!/bin/bash
##
## Shell Script Buckinster Test Commando generation
##
## Author: Benjamin
## Last-Modification-Date: 2014-05-14
##
## The script generates a file with emma execution commandos to be executed
## by the jenkins buckminster plugin
##
## A naming convention is used searching for files ending with .tats.launch
## For details see: https://sdqweb.ipd.kit.edu/wiki/Palladio_Build_Infrastructure#JUnit_Tests
##
## Launch configuration files are searched recusively. However, the emma commando refers to the
## files using the direct parent and the file name only. This is necessary, because the emma commando
## searches for a matching bundle not a relative path. Thus, the launch configuration file's parent
## directory should match the bundle id.
## According to our conding conventions, projects should be named according to plugin id's and
## launch configurations should be placed in the project's root directory. Thus this constrained should
## be satisfied.
##

if [ -e build_command.txt ]
then
        cat /dev/null > build_command.txt
else
        touch build_command.txt
fi
find -name *.tests.launch |
while read filename
do
 directory=`dirname $filename | xargs -I{} basename {}`
 name=`basename $filename`
 echo "emma -m -l '$directory/$name' -o 'output/junit/$name.xml' --xml 'output/emma/coverage.xml'\n" >> build_command.txt
done

Checkstyle

Currently, Checkstyle is automatically checked on all projects in the workspace. For this process, an ant task stored in checkstyle_workflow.xml is used. The default task which should be called is run. It first automatically checks if the given output directory (Parameter checkstyle_output_dir) for the result files exists. If not, the directory is created. The run task then iterates over all directories in the provided target directory (Parameter checkstyle_target) and checks them using a checkstyle task. The results are stored in a file named like the folder name in the output directory. Additionally to the two parameters described above, three more parameters are needed:

  • checkstyle_jar: Path to the checkstyle binary jar
  • checkstyle_config: Path to a checkstyle config file
  • ant_contrib_jar: Path to the ant-contrib binary jar file (used for the for statement)

Further information on how to use Checkstyle in your project are documented in the Palladio Coding Conventions.

ANT Script:

<?xml version="1.0" encoding="UTF-8"?>
<project name="PCM Core Checkstyle" default="run" basedir=".">
	<taskdef resource="checkstyletask.properties" classpath="${checkstyle_jar}"/>

<!--	Performs a checkstyle check on all *.java files in subdirectories of the given checkstyle_target and places the output in the given output directory.
	
	Parameters:
	checkstyle_target: Path to the directory on which checkstyle should be executed (for each subdir)
	checkstyle_config: Path to a checkstyle config file
	checkstyle_jar: Path to the checkstyle binary jar file
	checkstyle_output_dir: Path to the output location
	ant_contrib_jar: Path to the ant-contrib binary jar file
-->

<!-- Main target. Iterates over all directories (excluding folders starting with .) in ${checkstyle_target} and calls modify_dir_name with the directory path -->
<target name="run" depends="createCheckstyleDir">
		<taskdef resource="net/sf/antcontrib/antlib.xml" classpath="${ant_contrib_jar}"></taskdef>
	<for param="current" keepgoing="true">
		<path>
		<dirset dir="${checkstyle_target}">
				<include name="*"/>
				<exclude name=".*"/>
		</dirset>
		</path>
		<sequential>
				<antcall target="modify_dir_name">
						<param name="currentDir" value="@{current}"/>
				</antcall>
		</sequential>
	</for>
	</target>

<!-- Helper target to extract a directory name from a whole path. Calls checkstyle target with the directory name-->
	<target name="modify_dir_name">
		<basename property="dir.name" file="${currentDir}"/>
		<antcall target="checkstyle">
			<param name="currentDir" value="${dir.name}"/>
		</antcall>
	</target>

<!-- Performs a checkstyle check on the directory ${currentDir} in the base directory ${checkstyle_target}.
	this method is called for every directory in ${checkstyle_target} by run->modify_dir_name -->
	<target name="checkstyle">
		<echo>Starting checkstyle for project ${currentDir}</echo>
	<checkstyle config="${checkstyle_config}" failOnViolation="false">
			<fileset dir="${checkstyle_target}/${currentDir}" includes="**/*.java"/>
			<formatter type="xml" toFile="${checkstyle_output_dir}/${currentDir}.xml"/>
		</checkstyle>
		<echo>Checkstyle for project ${currentDir} finished</echo>
	</target>

<!-- Calls dir.check to check if the output directory exists. If not, creates it -->
<target name="createCheckstyleDir" depends="dir.check" unless="dir.exists">
	<echo>Create checkstyle output dir...</echo>
	<mkdir dir="${checkstyle_output_dir}"/>
</target>

<!-- Checks if the output directory exists-->
	<target name="dir.check">
		<condition property="dir.exists">
			<available file="${checkstyle_output_dir}" type="dir"/>
		</condition>
	</target>
</project>

SVN Check

de.uka.ipd.sdq.hudson.svncheck is a Jenkins plugin which allows the user to check which projects under a number of given svn urls haven't been checked out through the svn function of the job.

It is activated in the a job's configuration in the post-build actions through the item "Publish SVN repository report". The plugin lets you choose a number of svn urls which shall be considered (e.g. https://svnserver.informatik.kit.edu/i43/svn/code/Palladio/Core/trunk/Build/) and a number of project names/patterns which shall be excluded (e.g. https://svnserver.informatik.kit.edu/i43/svn/code/Palladio/Core/trunk/Examples/SubsystemTest). Also, project names with a wildcard-* at the end can be used (e.g. https://svnserver.informatik.kit.edu/i43/svn/code/Palladio/Core/trunk/Examples/de.uka.ipd.*). After each build, the projects which are not being checked out from the given urls (and which are not excluded) will be published on the build overview page.

The project itself is a maven project. It can be imported into Eclipse through File->Import->Maven->Existing maven project. However, both the build and the error check won't work. The plugin is build through the console by navigating to the project folder and doing a "mvn package". The plugin file is then in the target folder (de.uka.ipd.sdq.hudson.svnchek.hpi). This file can then be installed in Jenkins through the "Manage plugins" function. Further information on the jenkins plugin structure and how to build jenkins plugins is found under https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial

Job Clean Up - Pre-Build Step

Before the build itself it executed, the reporting directories (e.g. for checkstyle and emma) must be cleaned up to produce fresh results. This is done by executing a shell command before running buckminster. The command to be used should be:

if [ -d "output/checkstyle" ]; then
  rm -f output/checkstyle/*
fi
if [ -d "output/junit" ]; then
  rm -f output/junit/*
fi
if [ -d "output/emma" ]; then
  rm -f output/emma/*
fi

Update Site Server

Update sites are generated by buckminster and published to via ssh to our webserver located at the base uri: http://sdqweb.ipd.kit.edu/eclipse/

Each project gets its own update site space below this base uri. For example, palladio update sites are published at http://sdqweb.ipd.kit.edu/eclipse/palladio.
We use a common update site structure for each project:

  • nightly - the nightly update site based on the projects trunk
  • release - explicitly published and persisted update sites
    • current - the latest published update site
    • 1.0 - the update site of version 1.0
    • 1.1 - the update site of version 1.1
    • ...

Running a Release Build

Running a release build, follow the following steps:

  1. Click on the 'Release' action on the left-hand action bar of the build job. This is only visible if your Jenkins user has the necessary privileges for this action.
  2. If you want to re-release a plugin under the same version, select the last successful release build ("Letzter erfolgreicher Release Build").
    1. Click on the button "Diesen Build nur befristet aufbewahren".
    2. Delete the build.
    3. Start the release build.

Template-Based Build

  1. For new projects, you should use the template-build, analogously to other projects like ProtoCom
  2. For enabling xTend/xtText-base builds, configure an XtendCompiler.launch and add a dependency to org.eclipse.xtend.standalone (same as for ProtoCom; also see http://www.lorenzobettini.it/2013/09/using-the-xtend-compiler-in-buckminster-builds/)

See also