Discussion:
[gentoo-alt] 'Continuous Integration' for Gentoo Prefix?
Sam Pfeiffer
2018-11-26 04:02:26 UTC
Permalink
Hello,

First, let me apologise if I have any wrong preexisting ideas/guesses about
Gentoo Prefix and how it is developed. Secondly, sorry if this email is a
bit too long.

I'll present myself, explain my use-case and my troubles and tricks, and
then ask for feedback.

Hello, I'm Sammy Pfeiffer. I am a PhD student at University of Technology
Sydney. I'm a (software) robotics engineer pursuing my PhD in a lab with a
bunch of different robots and machine learning setups. I'm using Gentoo
Prefix for deploying a big bunch of software into robots where the OS is
old and frozen (and also to deploy in computation farms experiments... once
again with no option to touch the OS). (The bunch of software is ROS, the
'Robotics Operating System' which has a ton of dependencies, and there is
an existing overlay which I'm helping on maintaining and improving (
https://github.com/ros/ros-overlay)).

I'd like to share the trick I found to overcome a few annoying bits of my
platforms:
* I have no root access in the machines I need to deploy.
* The machines don't have Docker installed (too old kernel anyways).
* I have 32 bit and 64 bit machines (the OS running in them is).
* I have a different username (and home folder) on each machine.
* Each machine has a different disk(s)/partition(s) structure (main issue
here is that I may need to use a different disk to store my data, cause of
size constrains).

The trick to be able to use Gentoo Prefix with all these constraints
bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.
Then you can deploy the full bootstrapped system in any folder/disk and
just do a softlink to /tmp/gentoo. This works nicely (I was scared of the
softlink breaking stuff somewhere, but it was alright).
The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run in
any 32b or 64b box (it's not ideal but simplifies my deployment currently).

Also, having all these machines use the same Gentoo Prefix in the same
place, with the same 32bit compilation, I can setup a binary package
server, which all the deployments can point to and just get all the
packages skipping the long compilation times (specially in very old and low
powered machines).

The final trick I'm experimenting with is to use a set of Docker images
(and soon in a continuous integration environment) to bootstrap all the
system. With that I can save snapshot of successfully built systems &
packages to serve as an easy deployment (and easy installation of extra
packages thru the binary package server).

Given my particular annoyance of the 32bit system (and noting that my hosts
are 64bits, as is the standard nowadays) I found that I can use either:
* https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
* https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
Docker image

And then execute the build step of Docker with:

setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .

Which will trick any program trying to do uname -m to assume 32bit machine.
(Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping,
but I found some problems, which didn't appear with this method).

Once I have all this setup working nicely, I'd like to trigger rebuilds
every X time, and on changes on main players (or all dependencies actually)
of my setup, like the bootstrapping of Gentoo Prefix (Ideally, on every
change of a part of the system just trigger a rebuild from that point on,
Docker layers make this possible).

With Azure announcing unlimited minutes on CI/CD for open source projects:
https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/

Even bootstrapping Gentoo prefix, with pieces of software like gcc taking
very long to compile, is possible.

The point is: I have been trying to build Gentoo Prefix during the last
days after a few months of break since the last time I touched the system.
And it's failing. I haven't managed yet to bootstrap it completely. I feel
there is no CI/CD setup to catch these issues and be able to offer a
working version of Gentoo Prefix at any time.

I was going to build it for myself (cause I need it), but instead, I'd like
to offer my help to build it for the community. At least offer as an option
ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker
image, just copy the full folder structure, do the softlink, and you are
ready to play with Gentoo. If you mess up anything, just re-deploy.

To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to
get blocked on little issues that take a while to google or debug
(specially with the long compilation times of the bootstrap and some big
packages).

I've posted a short issue in the Docker repo of Gentoo images about this (
https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
maintainers probably don't usually work with Gentoo Prefix.

Thank you very much for your time, and for your open source efforts.
--
*Sammy Pfeiffer*
PhD Candidate at The Magic Lab within UTS.
Fabian Groffen
2018-11-26 08:11:33 UTC
Permalink
Hi Sam,
Post by Sam Pfeiffer
Hello,
First, let me apologise if I have any wrong preexisting ideas/guesses about
Gentoo Prefix and how it is developed. Secondly, sorry if this email is a bit
too long.
I'll present myself, explain my use-case and my troubles and tricks, and then
ask for feedback.
Hello, I'm Sammy Pfeiffer. I am a PhD student at University of Technology
Sydney. I'm a (software) robotics engineer pursuing my PhD in a lab with a bunch
of different robots and machine learning setups. I'm using Gentoo Prefix for
deploying a big bunch of software into robots where the OS is old and frozen
(and also to deploy in computation farms experiments... once again with no
option to touch the OS). (The bunch of software is ROS, the 'Robotics Operating
System' which has a ton of dependencies, and there is an existing overlay which
I'm helping on maintaining and improving
([1]https://github.com/ros/ros-overlay)).
I'd like to share the trick I found to overcome a few annoying bits of my
* I have no root access in the machines I need to deploy.
* The machines don't have Docker installed (too old kernel anyways).
* I have 32 bit and 64 bit machines (the OS running in them is).
* I have a different username (and home folder) on each machine.
* Each machine has a different disk(s)/partition(s) structure (main issue here
is that I may need to use a different disk to store my data, cause of size
constrains).
The trick to be able to use Gentoo Prefix with all these constraints
bootstrapping it only once is to set the EPREFIX to /tmp/gentoo.
That will work best, I presume.
Post by Sam Pfeiffer
Then you can deploy the full bootstrapped system in any folder/disk and just do
a softlink to /tmp/gentoo. This works nicely (I was scared of the softlink
breaking stuff somewhere, but it was alright).
The other trick is to bootstrap a 32bit Gentoo Prefix, which can be run in any
32b or 64b box (it's not ideal but simplifies my deployment currently).
Also, having all these machines use the same Gentoo Prefix in the same place,
with the same 32bit compilation, I can setup a binary package server, which all
the deployments can point to and just get all the packages skipping the long
compilation times (specially in very old and low powered machines).
Prefix branch knows a trick which can "relocate" packages. The
constrains is that it can only "shorten" a prefix. Thus, if you'd build
for /a/very/very/very/long/prefix you could install those binary
packages to /a/shorter/prefix. I think this will also addresses your
userid problems (if any).
Post by Sam Pfeiffer
The final trick I'm experimenting with is to use a set of Docker images (and
soon in a continuous integration environment) to bootstrap all the system. With
that I can save snapshot of successfully built systems & packages to serve as an
easy deployment (and easy installation of extra packages thru the binary package
server).
Neat!
Post by Sam Pfeiffer
Given my particular annoyance of the 32bit system (and noting that my hosts are
* [2]https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
* [3]https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
Docker image
setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
Which will trick any program trying to do uname -m to assume 32bit machine.
(Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I
found some problems, which didn't appear with this method).
Hmmm.
Post by Sam Pfeiffer
Once I have all this setup working nicely, I'd like to trigger rebuilds every X
time, and on changes on main players (or all dependencies actually) of my setup,
like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of
the system just trigger a rebuild from that point on, Docker layers make this
possible).
This may be a bit too much, but you could start from every day
(LATEST_TREE_YES), or we can see to building a tar more frequently from
rsync0.
Post by Sam Pfeiffer
With Azure announcing unlimited minutes on CI/CD for open source projects: 
[4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
long to compile, is possible.
I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap. I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world. Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.
Post by Sam Pfeiffer
The point is: I have been trying to build Gentoo Prefix during the last days
after a few months of break since the last time I touched the system. And it's
failing. I haven't managed yet to bootstrap it completely. I feel there is no
CI/CD setup to catch these issues and be able to offer a working version of
Gentoo Prefix at any time.
Sorry about that. I need to make another pass over it.
Post by Sam Pfeiffer
I was going to build it for myself (cause I need it), but instead, I'd like to
offer my help to build it for the community. At least offer as an option
ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
just copy the full folder structure, do the softlink, and you are ready to play
with Gentoo. If you mess up anything, just re-deploy.
To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get
blocked on little issues that take a while to google or debug (specially with
the long compilation times of the bootstrap and some big packages).
I've posted a short issue in the Docker repo of Gentoo images about this
([5]https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
maintainers probably don't usually work with Gentoo Prefix.
I'm not very familiar myself with Docker (mostly hiding under a rock
here), but I suspect the gentoo-docker-images project responds better to
email to their list (as in Contributing from their README). I may be
wrong though.
Post by Sam Pfeiffer
Thank you very much for your time, and for your open source efforts.
Thanks for your efforts sofar. I hope we can pull something together.
At least in the coming period I hope to be able to fix the bootstrapping
issues for Solaris/OpenIndiana/Darwin.

Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
M. J. Everitt
2018-11-26 08:27:41 UTC
Permalink
Post by Fabian Groffen
Hi Sam,
Post by Sam Pfeiffer
With Azure announcing unlimited minutes on CI/CD for open source projects: 
[4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
long to compile, is possible.
I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap. I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world. Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.
From what I have observed, breakage seems to be undesirably frequent, and
there are regularly users in #gentoo-prefix on IRC complaining that
bootstrap-prefix.sh fails.

I'm sure that if native Gentoo hardware isn't available, it should be
possible to approach OSL or other providers (I, myself, am using
IntegriCloud for powerpc development) to sponsor a host, and/or put
together a funding bid to the Trustees (there is capital available).

TL;DR I don't think "ENOINFRA" is a particularly good excuse in this
day-and-age of cloud-based solutions.

Let us help you, and perhaps we can get the very popular Gentoo Prefix onto
a slightly firmer footing! :)

Best Regards,

Michael "veremitz" Everitt.
Fabian Groffen
2018-11-26 08:32:13 UTC
Permalink
Post by M. J. Everitt
Post by Fabian Groffen
Hi Sam,
Post by Sam Pfeiffer
With Azure announcing unlimited minutes on CI/CD for open source projects: 
[4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
long to compile, is possible.
I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap. I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world. Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.
From what I have observed, breakage seems to be undesirably frequent, and
there are regularly users in #gentoo-prefix on IRC complaining that
bootstrap-prefix.sh fails.
I'm sure that if native Gentoo hardware isn't available, it should be
possible to approach OSL or other providers (I, myself, am using
IntegriCloud for powerpc development) to sponsor a host, and/or put
together a funding bid to the Trustees (there is capital available).
TL;DR I don't think "ENOINFRA" is a particularly good excuse in this
day-and-age of cloud-based solutions.
Sorry. with Infra, I didn't mean the team or the hardware, I meant the
scripting/software to keep track of the builds and visualise their
results. I can fire off a bootstrap right now, but there is nothing
picking up on their results.
Post by M. J. Everitt
Let us help you, and perhaps we can get the very popular Gentoo Prefix onto
a slightly firmer footing! :)
At this point I think he project at the very least needs a lot of
helping hands. I totally agree, automation is the way forward, but it
always will need caretakers.

Thanks,
Fabian
--
Fabian Groffen
Gentoo on a different level
M. J. Everitt
2018-11-26 08:38:08 UTC
Permalink
Post by Fabian Groffen
Post by M. J. Everitt
Post by Fabian Groffen
Hi Sam,
Post by Sam Pfeiffer
With Azure announcing unlimited minutes on CI/CD for open source projects: 
[4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
long to compile, is possible.
I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap. I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world. Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.
From what I have observed, breakage seems to be undesirably frequent, and
there are regularly users in #gentoo-prefix on IRC complaining that
bootstrap-prefix.sh fails.
I'm sure that if native Gentoo hardware isn't available, it should be
possible to approach OSL or other providers (I, myself, am using
IntegriCloud for powerpc development) to sponsor a host, and/or put
together a funding bid to the Trustees (there is capital available).
TL;DR I don't think "ENOINFRA" is a particularly good excuse in this
day-and-age of cloud-based solutions.
Sorry. with Infra, I didn't mean the team or the hardware, I meant the
scripting/software to keep track of the builds and visualise their
results. I can fire off a bootstrap right now, but there is nothing
picking up on their results.
Post by M. J. Everitt
Let us help you, and perhaps we can get the very popular Gentoo Prefix onto
a slightly firmer footing! :)
At this point I think he project at the very least needs a lot of
helping hands. I totally agree, automation is the way forward, but it
always will need caretakers.
Thanks,
Fabian
I'll throw my 'oar' in here too - I've been helping out extensively with
the kernelCI project @alicef spawned. We're presently using buildbot to do
automated emerges and a 'make defconfig' on the kernel sources, just to
make sure patching works, and compiles. We've actually reached a point too,
where we QEMU-boot said kernel image into a pre-built stage4.

I'm happy to share some of the know-how to get some of this spun up for
prefix if it's desirable. I've had to replicate the 'production' setup
locally in order to develop and test changes, so its a process I'm becoming
pretty familiar with.

Buildbot is python, so it integrates pretty well with portage, and is
accessible for devs. Let me know if this is a direction that would be
favourable.

https://github.com/gentoo/Gentoo_kernelCI/

Best regards,

veremitz/Michael.
Michael Haubenwallner
2018-11-26 09:53:11 UTC
Permalink
Hi Sam,

On 11/26/2018 09:11 AM, Fabian Groffen wrote:
<snip intro>
Post by Fabian Groffen
Post by Sam Pfeiffer
Given my particular annoyance of the 32bit system (and noting that my hosts are
* [2]https://github.com/docker-32bit/ubuntu a 32bit Ubuntu Docker image
* [3]https://github.com/gentoo/gentoo-docker-images stage3-x86 32bit Gentoo
Docker image
setarch i686 docker build -t my_bootstrapping_gentoo_prefix_32b_image .
Which will trick any program trying to do uname -m to assume 32bit machine.
(Previously I used the variable CHOST=i686-pc-linux-gnu for bootstrapping, but I
found some problems, which didn't appear with this method).
Hmmm.
On 64bit machines, one can use 'linux32' to trick the autodetection into 32bit:
$ uname -m
x86_64
$ linux32 uname -m
i686
Post by Fabian Groffen
Post by Sam Pfeiffer
Once I have all this setup working nicely, I'd like to trigger rebuilds every X
time, and on changes on main players (or all dependencies actually) of my setup,
like the bootstrapping of Gentoo Prefix (Ideally, on every change of a part of
the system just trigger a rebuild from that point on, Docker layers make this
possible).
This may be a bit too much, but you could start from every day
(LATEST_TREE_YES), or we can see to building a tar more frequently from
rsync0.
Post by Sam Pfeiffer
With Azure announcing unlimited minutes on CI/CD for open source projects: 
[4]https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
Nice, wasn't aware of that one!
Post by Fabian Groffen
Post by Sam Pfeiffer
Even bootstrapping Gentoo prefix, with pieces of software like gcc taking very
long to compile, is possible.
I'm still missing the infra to administer this, but I have a wrapper
script to bootstrap-prefix.sh (I'm sure others (haubi?) have too) to do
an unattended bootstrap. I want this to run automatically on some boxes
I have, but some of them will take several days to complete.
Another thing is for a bootstrapped prefix to periodically emerge --sync
and emerge -Dua world. Add on top of that detecting which packages have
keywords for said prefix and installing them, and we should be able to
be notice breakage rather sooner than later.
Yes, I do have a buildbot setup within the company I'm working at,
but unfortunately I don't have the time to inspect the results and
fix upcoming problems on a regular base.
Beyond that, I cannot open the buildbot-master to the public, but
I believe I could share build results if we had some public place
to store them.

Anyway - the Prefix buildbot slaves here are doing a clean bootstrap
(with LATEST_TREE_YES=1, noninteractive) once per week, and an emerge
update/deep/world once per day. The list of "KEYWORD (hosts)" here is:
ppc-aix (AIX 6.1/64bit, AIX 7.1/64bit)
x86_64-cygwin (Windows Server 2012)
sparc-solaris (Solaris 10/64bit)
x86-linux (SLES11.1/64bit, SLES12.0/64bit, RHEL7.1/64bit, RHEL6.2/64bit)
amd64-linux (Gentoo/64bit)

Also, I do have need for Prefix Guest only, no RAP here - but there should be
hardware resources left to perform both builds.
Post by Fabian Groffen
Post by Sam Pfeiffer
The point is: I have been trying to build Gentoo Prefix during the last days
after a few months of break since the last time I touched the system. And it's
failing. I haven't managed yet to bootstrap it completely. I feel there is no
CI/CD setup to catch these issues and be able to offer a working version of
Gentoo Prefix at any time.
Sorry about that. I need to make another pass over it.
Indeed - only the 64bit (Cygwin & Linux) ones were succeeding recently,
all the 32bit ones failed.
Post by Fabian Groffen
Post by Sam Pfeiffer
I was going to build it for myself (cause I need it), but instead, I'd like to
offer my help to build it for the community. At least offer as an option
ready-to-use /tmp/gentoo EPREFIX'ed built Gentoo Prefix from a Docker image,
just copy the full folder structure, do the softlink, and you are ready to play
with Gentoo. If you mess up anything, just re-deploy.
To do this I'd need a bit of help as I'm quite new to Gentoo and I tend to get
blocked on little issues that take a while to google or debug (specially with
the long compilation times of the bootstrap and some big packages).
I've posted a short issue in the Docker repo of Gentoo images about this
([5]https://github.com/gentoo/gentoo-docker-images/issues/62) but I think the
maintainers probably don't usually work with Gentoo Prefix.
I'm not very familiar myself with Docker (mostly hiding under a rock
here), but I suspect the gentoo-docker-images project responds better to
email to their list (as in Contributing from their README). I may be
wrong though.
Post by Sam Pfeiffer
Thank you very much for your time, and for your open source efforts.
Thanks for your efforts sofar. I hope we can pull something together.
At least in the coming period I hope to be able to fix the bootstrapping
issues for Solaris/OpenIndiana/Darwin.
I would love to see my buildbot slave's results somewhere in the public,
thank you for triggering this!

/haubi/
Benda Xu
2018-11-26 13:00:54 UTC
Permalink
Hi Sam,
Post by Sam Pfeiffer
https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source/
Even bootstrapping Gentoo prefix, with pieces of software like gcc
taking very long to compile, is possible.
The point is: I have been trying to build Gentoo Prefix during the
last days after a few months of break since the last time I touched
the system. And it's failing. I haven't managed yet to bootstrap it
completely. I feel there is no CI/CD setup to catch these issues and
be able to offer a working version of Gentoo Prefix at any time.
I completely agree with you. I hope you can carry on this project to
setup proper CI for Gentoo Prefix. I am all in for help, portage/ebuild
mentoring and coorperation.

A CI for Gentoo Prefix has been on my list for ages. Thank you for
triggering this.

Yours,
Benda

Loading...