Note

This project is under active development.

YAML2RPM Documentation

Welcome to the documentation for YAML2RPM.

We developed YAML2RPM to manage the multiple versions, dependencies, and other details of a resilient software deployment. YAML-formatted specification files are used to describe how to build an application, encode dependencies, and where to install. Through programmatic translation via a custom python program, the YAML input generates the ingredients to build a RedHat-compatible RPM using the distribution’s native rpmbuild tool. The full process creates human-recognizable package names, supports multiple installed versions, and easily encodes dependencies for repeatable and robust application stacks. In a full-stack recompilation, over 2000 individual RPMs are created.

The documentation will describe the process, point out many advantages and describe many examples in detail. Our approach makes routine packaging of applications very straightforward and reduces difficult builds to manageable ones. YAML2RPM and our recipes for building 100s of applications are open source.

yaml2rpm has its documentation hosted on Read The Docs.

What is YAML2RPM?

At UCI we run a campus-accessible, shared-use, high-performance computing cluster with hundreds of domain-specific applications. The applications need to be compiled and installed, often with updated toolchains (prerequisites and dependencies). Users regularly require multiple versions to be installed. We developed YAML2RPM to manage the multiple versions, dependencies, and other details of a resilient software deployment. YAML-formatted specification files are used to describe how to build an application, encode dependencies, and where to install. Through programmatic translation via a custom python program, the YAML input generates the ingredients to build a RedHat-compatible RPM using the distribution’s native rpmbuild tool. The full process creates human-recognizable package names, supports multiple installed versions, and easily encodes dependencies for repeatable and robust application stacks. In a full-stack recompilation, over 2000 individual RPMs are created.

YAML2RPM is a Generic Methodology for building RPMS This software uses the underlying Rocks methodology for automatically creating RPM spec files to create packages. Where it differs from Rocks is that a YAML-based specification of a package is used to define the specific details of a component, instead of a subdirectory structure for each package (see the rocksclusters github rolls for many examples of the subdirectory structure)

Most (open-source) software starts with a source tarball, extracts the tarball, configures for the local environment, executes make and then executes make install In the above, make might be cmake or some other software-specific tool. The last step, to turn the result of the steps, community-standard, process into an installable package is often deemed too time-consuming or difficult. The advantages of creating a package include:

File system conflicts resulting from different software builds are flagged prior-to-install
Software dependency resolution staves off many errors when attempting to remove a key software package that other packages depend upon
Binary packages can be “compiled once” and reused in testing, staging, production environments
System tools (e.g., yum) can be used a common tool to interrogate the file system for integrity, ownership of specific files and other items.

This software relies on only one Rocks-created software package, but otherwise is completely compatible with Generic CentOS and RedHat.

Motivation

The approach used here is one where programmatic translations are used to progressively create a subdirectory structure that mirrors the way Rocks builds RPMS (an example of building a qrencode RPM). In that structure, an RPM spec file is automatically created and files are put in appropriate places in which rpmbuild can success fully build a package. The generated spec file must define a source in a %source as well as %build, %install, %file and other RPM-specific directives. In particular, the %source is a tarball of this directory in gith ub (e.g. the base/src/qrencode directory). However, prior to creating the tarball, the upstream tarball (e.g. qrencode-3.4.0.tar.bz2) must be placed in base/src/qrencode directory. The automatically generated spec fil e, the %build directive invokes the build target of the Makefile provided here. In this example the section looks like:

build:
        bunzip2 -c $(NAME)-$(VERSION).$(TARBALL_POSTFIX) | $(TAR) -xf -
        (                                                    \
             cd $(NAME)-$(VERSION);                          \
             ./configure --prefix=$(PKGROOT);                \
             $(MAKE);                                        \
        )

This build target looks very similar to what you would do if your are building software without placing it into a package. And that is intentional. In building literally hundreds of packages, the basic approach works quite well, but it comes that price of excessive code copying. The goal of the YAML-based generation of a package is to remove as much gratuitous “copy-and-paste” structure as possible.

While the above is very simple (and actually a quite common build motif), there are many variations on how something is built, what it is dependent upon, and the like. In real use on, for example, academic computing clusters groupings of software have common dependencies. A routine example is software that depends upon a newer version of gcc. The new version of gcc must be installed alongside the system version. The gcc-admix repository uses YAML2RPM to build an updated version of gcc, a module file, and some compatible libraries. Those packages can then be installed and used to build other software.

Yaml2rpm set out to solve part of the problem of building and packaging relatively complex software. At its core, it creates packages in the RPM format, but without the pain of manually building and maintaining spec files. A developer who wants to build yaml2rpm-generated packages must still have some familiarity with RPM concepts and yum.

Quickstart

If you want to use prebuilt RPMS for testing on a standard CentOS machine, you can follow what is below. The following was tested on the official CentOS 7 Amazon machine image.

If you want to build yaml2rpm RPMS and install them from source repo, see Building section.

Prerequisites

Python 3. Required python modules: argparse, socket, datetime. There are 4 python modules that will be automatically built and installed during the “Building” step:
- future: for compatibility of python 2/3 code
- ruamel-yaml & ruamle-ycml-clib: used by the main script gen-definitions.py
- setuptools: for building python dependent packages.
If you are using a very stripped-down CentOS image (similar to the official CentOS 7 image in Amazon, you will want to make certain you have the following packages and package groups installed
```
yum groupinstall "Development Tools" "Console Internet Tools"
yum install redhat-lsb wget zlib-devel environment-modules
. /etc/profile.d/modules.sh
```
Install the development RPMS. Go to the Development RPMS repository for the latest pre-built RPMs and instructions. After following those instructions, you can build your first RPM from source.

Building

You may want to build the yaml2rpm rpms and install them from the source git repository. Do the following in the top-level directory You will need to set DISPLAY prior to doing this so that firefox can ask for your permission to read public data

./first-build.sh

After this step is complete the followign RPMs are built and installed:

python-future
python-setuptools
python-ruamel-yaml
python-ruamel-yaml-clib
rcic-module-support
rcic-module-path
yaml2rpm

The python- RPMs provide 4 needed python modules for your default system python install. The rcic-module-support, rcic-module-path, and yaml2rpm provide all the building structure and support files for the packages builds. They include a couple of profiles files that are added to the /etc/profile.d.

In order to proceed with next steps simply execute them (for future logins they will be automatically sourced by the login shell):

. /etc/profile.d/rcic-modules.sh
. /etc/profile.d/yaml2rpm.sh

A simple test build

For a very simple build of an RPM, create a working directory workdir in this simple example. And then download the source tarball into the workdir/sources directory. Then create the cmake RPM, it will be placed in workdir/RPMS/x86_64

mkdir -p workdir/yamlspecs
cd workdir/yamlspecs; cp /opt/rocks/yaml2rpm/samples/* .
make download PKG=cmake
make download PKG=scons
make

At the end of the process, you should have 4 RPMs in workdir/RPMS/x86_64/. You could install them on the local machine and have an updated version of cmake and scons, and corresponding environment modules. For example. the module for cmake can be loaded in order to use cmake:

module load cmake
which cmake

The version of cmake is defined in the versions.yaml file, if you wanted to update the version, you could edit that file, download the new source tarball directly from the source website and then rebuild a new package via

make download PKG=cmake
make cmake.pkg
make cmake-module.pkg

Architecture

Python scripts

There is one main driver scipt called gen-definitions.py This is the definitions parser that reads info defined in yaml files and creates necessary output files in order to build RPMS.

Templates

There are a few Makefils that are used as templates when crating packages

In additon, there are a few templates writtien in yaml that provide basic configuratio and settings. These templates are included in the specific software yaml description files

Requirements

Quickstart

If you want to use prebuilt RPMS for testing on a standard CentOS machine, you can follow what is below. The following was tested on the official CentOS 7 Amazon machine image.

If you want to build yaml2rpm RPMS and install them from source repo, see Building

Prerequisites

Python 2 or 3. Required python modules: argparse, socket, datetime. There are 4 python modules that will be automatically built and installed during the “Building” step:
- future: for compatibility of python 2/3 code
- ruamel-yaml & ruamle-ycml-clib: used by the main script gen-definitions.py
- setuptools: for build python dependent packages.
If you are using a very stripped-down CentOS image (similar to the official CentOS 7 image in Amazon, you will want to make certain you have the following packages and package groups installed
```
yum groupinstall "Development Tools" "Console Internet Tools"
yum install redhat-lsb wget zlib-devel environment-modules
. /etc/profile.d/modules.sh
```
Install the development RPMS

Go to the Development RPMS repository for the latest pre-built RPMs and instructions. After following those instructions, you can build your first RPM from source.

Usage

Installation

To use Pkgname, first install it using pip:

(.venv) $ pip install pkgname

This is an example of using just the double colon:

ls -l
pwd

This is an example of code block with language:

ls -l
pwd
conda config --add channels defaults
conda config --add channels conda-forge

Creating recipes

This is coming from the original lumache.py file

To retrieve a list of random ingredients, you can use the lumache.get_random_ingredients() function:

lumache.get_random_ingredients(kind=None)

Return a list of random ingredients as strings.

Parameters:: kind (list[str] or None) – Optional “kind” of ingredients.
Raises:: lumache.InvalidKindError – If the kind is invalid.
Returns:: The ingredients list.
Return type:: list[str]

The kind parameter should be either "meat", "fish", or "veggies". Otherwise, lumache.get_random_ingredients() will raise an exception.

exception lumache.InvalidKindError: Raised if the kind is invalid.

For example:

>>> import pkgname
>>> pkgname.get_random_ingredients()
['shells', 'gorgonzola', 'parsley']

add python source file as an example

#!/usr/bin/env python3

a = 2
b = 3

c = a + b
print ("c = %d" % c)

add bash source file as an example

#!/bin/bash

# Purpose: extract lines with "module-hpc"  from /var/log/messages* files
# Arguments: list of files names to process. 
# If empty, assume messages* files in the current directory

if [ $# -eq 0 ];  then
    echo "No arguments supplied. Assuming files in current directory `pwd`"
    files=`ls messages*`
else
    files=$@
fi

prog=`basename $0`

help () {

    printf "\nNAME\n       $prog - eXtract MODule Lines from log files\n"
    printf "SYNOPSIS\n       $prog [OPTION]... [FILE]...\n"
    printf "DESCRIPTION\n"
    printf "       Extract lines with 'module-hpc' from the /var/log/messages-DATE files.\n"
    printf "       For each processed messages-DATE file generate an output file modules-DATE.\n"
    printf "       If no  arguments are given on the command line assume messages-DATE files\n"
    printf "       in the  current directory.\n\n"
    printf "       -h, --h, -help, --help, help\n"
    printf "              print usage info\n\n"

    exit 0
}

case "$1" in
  -*h*|*help)
	help
	;;
  *)
        ;;
esac

for File in $files; do
    if [ ! -f $File ]; then
        echo "Skip $File: not a file"
        continue
    fi
    if [[ $File =~ ".gz" ]];  then
        gunzip $File
        raw=${File%.gz}
        Compress=True
    else
        raw=$File
    fi
    modfile=${raw//messages/modules}
    grep module-hpc $raw > $modfile
    echo  "Wrote " $modfile
    if [ $Compress ]; then
        gzip $raw
        compress=False
    fi
done

!include rcic-package.yaml
---
   package: iperf version 3 network tester
   name: iperf
   version: "3.6"
   extension: tar.gz
   description: >
       iperf3 version {{ version }}. iperf is a tool for active measurements 
       of the maximum achievable bandwidth on IP networks. It supports tuning
       of various parameters related to timing, protocols, and buffers. For
       For each test it reports the measured throughput/bitrate, loss,
       and other parameters.
   vendor_source: https://downloads.es.net/pub/iperf/iperf-{{ version }}.tar.gz
   root: "{{ pkg_defaults.app_path }}/{{ name }}/{{ version }}"
   build:
       modules:
       target:
   install:
       installextra: $(INSTALL) -m 644  README* LICENSE $(ROOT){{ root }}

List

This is a bulleted list.
It has two items, the second item uses two lines.

This is a numbered list.
It has two items too.
This is a numbered list.
It has two items too.

Nested list

this is
a list
- with a nested list
- and some subitems
and here the parent list continues

Table

Header row, column 1 (header rows optional)	Header 2	Header 3	Header 4
body row 1, column 1	column 2	column 3	column 4
body row 2	…	…

Links

Use Link text for inline web links. If the link text should be the web address, you don’t need special markup at all, the parser finds links and mail addresses in ordinary text.

You can also separate the link and the target definition (ref), like this:

This is a paragraph that contains a link.

This example is fro mthe top tutorial of read the docs The tutorial is aimed at people interested in learning how to use Read the Docs to host their documentation projects. You will fork a fictional software library similar to the one developed in the official Sphinx tutorial. No prior experience with Sphinx is required and you can follow this tutorial without having done the Sphinx one.

Trying the popup text

Read the Docs will try to build the documentation of your project right after you create it. To see the build logs, click on the Your documentation is building link on the project home, or alternatively navigate to the Builds page,

Trying a note:

Tip

We strongly recommend pinning all the versions required to build the documentation to avoid unexpected build errors.

Attention

We strongly recommend reading the pinning all the versions

Note

This is a simple note

Button

trying from tutorial text snippet here you will see a green Use this template button. Click it to open a new page that will ask you for some details:

TEXT

This is the repository you will import on Read the Docs, and it contains the following files:

README.rst: Basic description of the repository, you will leave it untouched.
pyproject.toml: Python project metadata that makes it installable. Useful for automatic documentation generation from sources.
pkgname.py: Source code of the fictional Python library.
docs/: Directory holding all the Sphinx documentation sources, including the Sphinx configuration docs/source/conf.py and the root document docs/source/index.rst written in reStructuredText.

Figures

This is made with figure

This is made with image

API

lumache

Lumache - Python library for cooks and food lovers.

usage

Usage - collect usage info

Add empty line

GCC admix

A deeper example

GCC (the GNU compiler collection) is relatively complex build. It is often useful to have an updated version of gcc on your system without destroying the system-supplied gcc. The GCC build has to be done in a certain way, packages need to be named to be non-conflicting and other items. If you have completed the Quickstart above you can build an updated version of gcc and a set a packages. WARNING! This process will install RPMS as it builds. You should do this on a ‘disposable’ build system. It takes hours to compile a gcc.

Here is the full process for building gcc using the gcc-admix repo

This is block-code console

git clone https://github.com/RCIC-UCI-Public/gcc-admix.git
cd gcc-admix
make download
cd yamlspecs
(make bootstrap &> /tmp/bootstrap-gcc.log; make &> /tmp/build-gcc.log) &

This is code-block bash

git clone https://github.com/RCIC-UCI-Public/gcc-admix.git
cd gcc-admix
make download
cd yamlspecs
(make bootstrap &> /tmp/bootstrap-gcc.log; make &> /tmp/build-gcc.log) &

An optional last step would be to unbootstrap the system, which will remove any locally-built and installed RPMS that result from the bootstrap process.

make unbootstrap

At this point, you should have a compatible set of RPMS for gcc and some key supporting libraries.

The YAML Definition file

The basic notion of the defintion file to record on the bare minimum needed to define a package. Since it is usual for groupings of packages to share some common definitions, the generator supports a defaults file. Here is a very simple yaml file taken from the samples directory of the yaml2rpm package.

filename.yaml

package: iperf version 3 network tester
name: iperf
version: "3.6"
extension: tar.gz
description: >
    iperf3 version {{ version }}. iperf is a tool for active measurements
    of the maximum achievable bandwidth on IP networks. It supports tuning
    of various parameters related to timing, protocols, and buffers. For
    For each test it reports the measured throughput/bitrate, loss,
    and other parameters.
vendor_source: https://downloads.es.net/pub/iperf/iperf-{{ version }}.tar.gz
root: "{{ pkg_defaults.app_path }}/{{ name }}/{{ version }}"
build:
    modules:
    target:
install:
    installextra: $(INSTALL) -m 644  README* LICENSE $(ROOT){{ root }}

Glossary

This page includes a number of terms that we use in our documentation.

project home: https://npw-rtd-tutorial.readthedocs.io

Changelog

Release x.y.z

Dependencies

Incompatible changes

Deprecated

Features added

Bugs fixed

Testing

Release x.y.z.rc1

Dependencies

Require Docutils 0.18.1 or greater.

Incompatible changes

removal of some internal comment
Remove support from …
Remove deprecated code in x.py

Deprecated

Features added

process modules