Building with oi-userland
Using OpenIndiana's unified build system
OpenIndiana Hipster's primary build framework is oi-userland. It's tied into OpenIndiana continuous integration platform.
When an update is committed to the oi-userland git repository:
- an automated build is kicked off,
- then automatically the binary package will be published to the /hipster repository,
- finally, the status of the build will be reported by oibot to the #oi-dev freenode IRC channel.
Overview of oi-userland
Originally oi-userland is a fork of Oracle's userland-gate which evolved in an independent way. The layout is very similar.
Inside oi-userland is a directory called "components", under which directories for software package category groups live. Inside each of these package group directories are typically directories for all software packages belonging to the category. Some have even more sub-directories before reaching the leaf directories containing the software packages. Note that some software packages don't have a package category group yet. Inside each of the software package directories there is a main Makefile and other files necessary to build one or more packages. The complete set of instructions to build one or more related packages is called "a component".
A component usually consists of several files:
Makefile: the recipe to build the software and install it locally (usually to the
patches/: directory containing patches applied before the configuration
files/: directory containing additional files distributed with packages
*.p5m: manifests used to generate the IPS package
$(COMPONENT_NAME).license: file containing the licenses applicable to the software
To build a component you simply cd into the directory of the software, and type "gmake TARGET" (here we use gmake to call GNU make), where TARGET can be one of:
||cleans up the component directory completely, including deleting source|
||fetches the source archive and verify its
||extract and apply patches|
||configure and build|
||install software into the prototype directory|
||generate a sample IPS manifest based on the files installed to the prototype directory|
||publish the package to the local repository|
||run all pre-publication checks (does actually what publish does, just without sending package to local repository)|
||guess and generate build dependencies for the packages, manual edit might be needed|
||check build environment for missing packages|
||install missing build dependencies (requires elevated privileges)|
Before considering adding a new package to oi-userland, please check first whether someone else is working on the package by checking the issue tracker, mailing email@example.com or asking on the IRC (#oi-dev at irc.freenode.net)
- If you don't find anyone already working on a port, please register your effort by opening an issue.
- If you wish to update an existing port, look at the log for the component Makefile ("git log Makefile") and make sure you either contact the person who last updated the Makefile or include them on notifications for the issue by ticking their name.
This will ensure efforts aren't duplicated and help to ensure sanity and comity amongst project members.
Setting up your build environment
We strongly recommend building packages inside a fresh local zone set up exclusively for building. See Quick zone setup example for simple instructions.
Further we assume that you are logged into the build zone if you set up the build environment in a zone the directory were oi-userland is cloned can otherwise be anywhere you like.
Adding RBAC profile to the build user
Installing software requires privileges, so your build user must have at minimum the 'Software Installation' profile:
$ profiles Software Installation ZFS File System Management Console User Suspend To RAM Suspend To Disk Brightness CPU Power Management Network Autoconf User Desktop Removable Media User Basic Solaris User All
If it is not the case add this profile to your build user:
pfexec su - usermod -P'Software Installation' <username>
This is not necessary if your user has already the 'Primary Administrator' profile.
Start by forking oi-userland repository on Github and then check out the repository (subdirectory oi-userland must not pre-exist):
cd ~ git clone https://github.com/mylogin/oi-userland cd oi-userland
Add https://github.com/OpenIndiana/oi-userland/ as upstream to your repository to resync your repository with oi-userland.
git remote add upstream https://github.com/OpenIndiana/oi-userland/
Run the setup stage which will prepare some tools and create an IPS pkg5 repository for first use under the i386 directory:
cd $HOME/oi-userland gmake setup
Adding the local repository to your publisher list
pfexec pkg set-publisher -g file://$HOME/oi-userland/i386/repo userland pfexec pkg set-publisher --non-sticky openindiana.org
Setting the local repository as primary publisher
pfexec pkg set-publisher --search-first userland
Optional: Running a local pkg server for installation on other zones/hosts
If you would like to use your oi-userland repository on other zones or hosts, you can run a pkg server:
$ pfexec svccfg -s pkg/server svc:/application/pkg/server> add oi-userland svc:/application/pkg/server> select oi-userland svc:/application/pkg/server:oi-userland> addpg pkg application svc:/application/pkg/server:oi-userland> setprop pkg/inst_root=astring:"/export/home/username/oi-userland/i386/repo" svc:/application/pkg/server:oi-userland> setprop pkg/port=count:"10000" svc:/application/pkg/server:oi-userland> setprop pkg/readonly=boolean:"true" svc:/application/pkg/server:oi-userland> refresh svc:/application/pkg/server:oi-userland> exit $ pfexec svcadm enable oi-userland
On other hosts you can then specify http://hostname:10000 instead of the file:// address above. If you only intend to install and test packages locally this is not necessary as on-disk repository access suffices.
Building and installing your first packages
Enter to oi-userland/components/PATH/TO/COMPONENT directory and run:
cd $HOME/oi-userland/components/SOFTWARE gmake env-prep gmake publish pfexec pkg refresh pfexec pkg install pkg://userland/PACKAGE_FMRI
Here PACKAGE_FMRI is a full name (FMRI) of package which you want to install. The FMRIs of published packages will be printed in the end of
Note, that running
gmake env-prep is strictly not required, if you are sure that all build requirements are satisfied.
To speed up the compilation you can pass an optional argument to gmake setting COMPONENT_BUILD_ARGS variable in your environment, for instance with
to use 4 jobs for builds.
Creating your first component
The easiest way to create new component is to take one which is similar to yours and modify it as needed. Also you can look at Makefile templates delivered with oi-userland.
A component Makefile usually contains variables describing how a component should be built, installed and packaged.
On the top of the Makefile it includes
../../../ is the relative path from the component directory to the make-rules directory.
The file shared-macros.mk contains global constants used by other makefiles. Sometimes some global variables that alter these constants are declared before this include.
The first section of the Makefile contains definitions of the component name, version, an url where the software should be fetched from, a short description embedded in the package metadata and so on.
Look, for example, at
COMPONENT_NAME= libjpeg6-ijg COMPONENT_VERSION= 6.0.2 LIBJPEG_API_VERSION= 6b COMPONENT_FMRI= image/library/libjpeg6-ijg COMPONENT_CLASSIFICATION=System/Multimedia Libraries COMPONENT_PROJECT_URL= http://www.ijg.org/ COMPONENT_SUMMARY= libjpeg - Independent JPEG Group library version 6b COMPONENT_SRC= jpeg-$(LIBJPEG_API_VERSION) COMPONENT_ARCHIVE= $(COMPONENT_NAME)-$(COMPONENT_VERSION).tar.gz COMPONENT_ARCHIVE_HASH= \ sha256:75c3ec241e9996504fe02a9ed4d12f16b74ade713972f3db9e65ce95cd27e35d COMPONENT_ARCHIVE_URL= http://www.ijg.org/files/jpegsrc.v$(LIBJPEG_API_VERSION).tar.gz COMPONENT_LICENSE= IJG,GPLv2.0 COMPONENT_LICENSE_FILE= $(COMPONENT_NAME).license
|COMPONENT_NAME||libjpeg6-ijg||The name of the component, usually it's a well-known software name|
|COMPONENT_VERSION||6.0.2||The software version. If the version contains letters, the IPS_COMPONENT_VERSION variable should define a numerical version used for the package, as the IPS version string doesn't allow for letters|
|LIBJPEG_API_VERSION||6b||In this example this is a local variable declared in the Makefile.|
|COMPONENT_FMRI||image/library/libjpeg6-ijg||This variable can be used in an IPS manifest to specify the FMRI (a name) of the package. It should folllow the conventions for package FMRIs.|
|COMPONENT_CLASSIFICATION||System/Multimedia Libraries||This entry should be in the OpenSolaris IPS Classification 2008|
|COMPONENT_PROJECT_URL||http://www.ijg.org/||Upstream project website|
|COMPONENT_SUMMARY||libjpeg - Independent JPEG Group library version 6b||A short description (one-liner)|
||The name of source after unpacking the archive|
||The software archive|
|COMPONENT_ARCHIVE_HASH||sha256:75c3ec241e9996504fe02a9ed4d12f16b74ade713972f3db9e65ce95cd27e35d||The SHA256 checksum of software archive|
||The URI to get the COMPONENT_ARCHIVE|
|COMPONENT_LICENSE||IJG,GPLv2.0||A comma separated list of licenses|
||The file with license text|
Components are usually based on one of the following Makefiles depending on build system used by packaged software (look in the
make-rules directory for more makefiles):
||AT&T package tools|
.mk file to see which variables you can modify, in general you can find variables such as:
For example, you may add this line for an Autotools-based component:
After creating the component Makefile you can run
Now you can create necessary patches for the component and put them in the patches directory.
When the component is built and installed correctly (via
gmake build and
gmake install), look if you can run the test suite if one comes with the software.
It's advised to put the expected test ouput in test/results-BITS.master (where BITS are either 32 or 64) and to ensure
gmake test target generates reproducible results.
You can use the COMPONENT_TEST_TRANSFORMS variable to set a list of sed directives to transform the test output and
make it reproducible.
Creating p5m files
install target passes you can run:
to generate a manifest from the list of installed files.
Copy the file
$(COMPONENT_NAME).p5m and edit it:
- Add your name as a contributor
- Remove unused entries from the manifest:
- static libs:
- libtool files:
- Python *.pyc:
For some components, specific rules need to be applied: they can be implemented with transforms.
Some Makefile targets defined in
make-rules/ips.mk apply transforms from files in the
transforms directory at the root of oi-userland.
These transforms can be used as examples if you need custom transforms for your component.
After creating a p5m file run
gmake REQUIRED_PACKAGES to automatically generate a list of run-time dependencies of the package (REQUIRED_PACKAGES section of the Makefile).
Add necessary build time dependencies on the top of the generated section.
gmake publish. If the manifest is valid your package is published to the local repository.
To be able to search for the new packages in the local repository you need to rebuild search indexes:
pkgrepo refresh -s /path/to/my_repo
You can even rebuild the entire metadata:
pkgrepo rebuild -s /path/to/my_repo
Installing the package
After you've published the package to your local repository and rebuilt the repository index or metadata you can install the package and perform whatever testing is appropriate.
If the package you built and published to the local userland repository is not already part of hipster it should be straightforward to install it:
pfexec pkg install your/package/name
If, however, the package you built is an updated version of an existing package then you may have to take additional steps before it can be updated.
pkg refuses to install the package from your local repository it may be because the userland-incorporation is preventing updates to the version of the package:
$ pfexec pkg update image/library/libjpeg6-ijg No updates available for this image. $ pfexec pkg update pkg://firstname.lastname@example.org:20180211T125627Z pkg update: No matching version of image/library/libjpeg6-ijg can be installed: Reject: pkg://email@example.com Reason: This version is excluded by installed incorporation firstname.lastname@example.org
If you will install many test versions of packages on your development system you may find it easiest to uninstall the userland-incorporation. Alternately, if you want to test a package on a system while keeping userland-incorporation, you can use
pkg change-facet to relax the version constraint for just that package:
pfexec pkg change-facet facet.version-lock.your/package/name/here=false
After you have performed one of these steps to remove the version constraint there is one more issue you may encounter. Because the installed version of the package came from the openindiana.org publisher but the updated version you want to install and test is associated with the userland publisher,
pkg will by default not allow the package update to switch which publisher provides the package.
One option to work around this is to make the openindiana.org publisher non-sticky:
pfexec pkg set-publisher --non-sticky openindiana.org
You only need to perform that operation on your development system once.
Alternately, you can force
pkg to apply an update from a different publisher by specifying the full FMRI for the package, including the publisher:
pfexec pkg update pkg://email@example.com:20180211T125627Z
Contributing changes back to oi-userland
Every time you add or modify a component, create a new branch:
git checkout -b my_feature
Keep this branch synchronized with upstream/oi/hipster:
git pull --rebase upstream oi/hipster
Your local branch is forwarded to the last commit of oi/hipster and your additional commits are kept on top of the stack.
When you think you are ready with changes you need to commit them locally and push those changes back to your Github repository.
cd ~/oi-userland/components/SOFTWARE git add Makefile *.p5m <etc.> git commit
Commit messages should be simple and describe what you did, e.g "Added XYZ", "XYZ: updated to SOME_VERSION", "XYZ: fixed SOMETHING" or "BUG_ID BUG_SUMMARY", where BUG_ID is issue number from issue tracker and BUG_SUMMARY is the issue name.
If you've created several commits while working on your component it's necessary to squash all your commits into one.
To do it, first check how many commits are to be considered:
git rebase -i HEAD~N
with N the number of commits to be squashed and follow the instructions: the letter 's' should be put in place of 'pick' for the N - 1 commits before the last.
If you made a mistake with the commit message or author, use:
git commit --amend
with the relevant option.
Now push your changes to your repository to GitHub.
git push my_name my_feature
git push -f my_name my_feature
if the branch you just rebased had already been pushed: since the history is rewritten you need to force the push, be careful.
Asking for change integration
This is as simple as creating a Pull Request into the main oi-userland repository and asking developers to review your changeset. We should beware of possibly breaking packages as it adds additional work and can be unpleasant for other contributors (imagine a situation where gcc, perl or anything else needed for building packages is broken).
Changes can be reverted quite easily but once the package is built and published additional steps are needed. So try taking per-package testing and asking for wider testing into consideration.
If you contribute a package which is known to work but its functionality might be broken because of some issues, consider disabling it until the issue is removed.
Checking Jenkins instance
Once the changes are merged into the main oi-userland repository, the Jenkins instance will pick up those bits and build them. If the build was successful, the built packages will be pushed into http://pkg.openindiana.org/hipster repository. If the package build was unsuccessful check the build logs and please try to come up with a solution and fix the problem so you can have the package published into the repository.