Adopting CoreOS Ignition into Fedora+derivatives ecosystem

[Colin Walters]

We're working on merging some of the CoreOS <https://coreos.com/> technologies with
the Fedora/Atomic technologies. One thing that came up is they have
a new take on system bootstrap called Ignition:
https://github.com/coreos/ignition

We're investigating making this work in Fedora as well:
https://pagure.io/atomic-wg/issue/450

There's a fair amount of overlap with the role of Anaconda in
terms of system provisioning. Some of this is actually similar
to the overlap between kickstart + cloud-init I posted previously:

https://www.redhat.com/archives/anaconda-devel-list/2014-December/msg00008.html

We're currently thinking Ignition should take over from cloud-init (though the details
there are still TBD). Its ability to do partitioning makes it a lot
closer to kickstart[1].

One thing I'd floated in some of our discussions was that conceptually
one could write a kickstart-to-ignition transpiler just like the
"clc-to-ignition" transpiler:
https://coreos.com/os/docs/latest/overview-of-ct.html
I'm not sure if this would be valuable versus writing ct since most
people are going to be starting "fresh" I suspect, but the fact
that we could helps understand the concepts here I think.

Probably where the biggest overlap is around baremetal provisioning;
CoreOS supports a "PXE-to-Live" model for servers:
https://coreos.com/os/docs/latest/booting-with-ipxe.html
as well as a script to install to disk persistently:
https://coreos.com/os/docs/latest/installing-to-disk.html

The latter path is like the existing Anaconda+kickstart
path; so where things intersect most strongly here is what we do
with those two paths. I'd suggest that for PXE-to-Live which we
don't currently productize that we focus on Ignition.

For persistent installations - I think we'd clearly need to support
kickstarts, but it's probably useful to look at a path where
Anaconda includes support for Ignition; perhaps it something like
a %ignition config stanza in kickstart?

[1] They even support partitioning the root filesystem
but I think that only works for the existing CoreOS with the dual-partition
/usr A/B approach; what I call "classic" Fedora with yum as well as
the existing Atomic with rpm-ostree both have the operating system
content in /, not in separate partitions.

[Martin Kolman]

Post by Colin Walters
We're working on merging some of the CoreOS <https://coreos.com/> technologies with
the Fedora/Atomic technologies. One thing that came up is they have
https://github.com/coreos/ignition
https://pagure.io/atomic-wg/issue/450
There's a fair amount of overlap with the role of Anaconda in
terms of system provisioning. Some of this is actually similar
https://www.redhat.com/archives/anaconda-devel-list/2014-December/msg00008.html
We're currently thinking Ignition should take over from cloud-init (though the details
there are still TBD). Its ability to do partitioning makes it a lot
closer to kickstart[1].

Yeah, Ignittion also seems to me like more like an enhanced Cloudinit with some
installer bits bolted on & it indeed seems like it could work like a good cloudinit replacement.

Post by Colin Walters
One thing I'd floated in some of our discussions was that conceptually
one could write a kickstart-to-ignition transpiler just like the
https://coreos.com/os/docs/latest/overview-of-ct.html
I'm not sure if this would be valuable versus writing ct since most
people are going to be starting "fresh" I suspect, but the fact
that we could helps understand the concepts here I think.
Probably where the biggest overlap is around baremetal provisioning;
https://coreos.com/os/docs/latest/booting-with-ipxe.html
https://coreos.com/os/docs/latest/installing-to-disk.html
The latter path is like the existing Anaconda+kickstart
path; so where things intersect most strongly here is what we do
with those two paths. I'd suggest that for PXE-to-Live which we
don't currently productize that we focus on Ignition.

Yeah, PXE-to-Live is pretty different to how Anaconda does,
we generally differentiate the installation environment and
the target system quite clearly and in this case they are basically
combined together. So if Ignition can do it it makes sense to
me to continu using it for this usecase, at least for the time being.

Post by Colin Walters
For persistent installations - I think we'd clearly need to support
kickstarts, but it's probably useful to look at a path where
Anaconda includes support for Ignition; perhaps it something like
a %ignition config stanza in kickstart?

I think that would work - would Anaconda be expected to parse the content
of the section or would just writing the content to a file on the target system
where Ignition will pick it up be enough ?
We could also make sure the ignition package is installed on the target system
when when the %ignition section shows up in kickstart (we already do that for various
kickstart commands such as realm, firewall, etc.).

Also alternatively if you wanted some more advanced configuration done during the
installation or wanted to provide a configuration GUI/TUI, Anaconda can be extended via
addons:
https://rhinstaller.github.io/anaconda-addon-development-guide/

Addons can have their own kickstart section and can (but don't have to) provide
TUI & GUI screens.

Post by Colin Walters
[1] They even support partitioning the root filesystem
but I think that only works for the existing CoreOS with the dual-partition
/usr A/B approach; what I call "classic" Fedora with yum as well as
the existing Atomic with rpm-ostree both have the operating system
content in /, not in separate partitions.

This might be a bit unrelated to current topic, but what are the plans with supporting the CoreOS A/B approach ?

The git-like aproach rpm-ostree provides seems to me as a much more advanced & flexible than just using the
relatively primitive and not very flexibile "you have two and exactly two copies of everything" approach.

It seems to me that compared to A/B ostree makes it possibly to have an arbitrary number of system states
available and should also be more space efficient (unless CoreOS deduplicates the A/B copies somehow).

Post by Colin Walters
_______________________________________________
Anaconda-devel-list mailing list
https://www.redhat.com/mailman/listinfo/anaconda-devel-list

[Colin Walters]

A basic thing to keep in mind here is that the existing CoreOS didn't have kickstart
and *only* has the PXE-to-Live model for bare metal (with optional persistent install), so there's
a lot of overlap going on here. But some of the technology makes sense and
is obviously in use by the existing CoreOS community. We're hoping to have
a "best of both" message.

We don't need to make any hard decisions here right away; the existing
Fedora/Atomic and CoreOS will be maintained for a while, but we do
hope to unify.

Post by Martin Kolman
Yeah, Ignittion also seems to me like more like an enhanced Cloudinit with some
installer bits bolted on & it indeed seems like it could work like a
good cloudinit replacement.

One thing that only became a bit clearer to me recently is we actually
still need:
https://github.com/coreos/coreos-metadata
which handles the "processed every boot" aspect that one needs in
many cloud environments; however the ignition ~= kickstart still remains,
as well as the ssh-key/useradd type overlap between both of them
and cloud-init/coreos-metadat.

Post by Martin Kolman
Yeah, PXE-to-Live is pretty different to how Anaconda does,
we generally differentiate the installation environment and
the target system quite clearly and in this case they are basically
combined together. So if Ignition can do it it makes sense to
me to continu using it for this usecase, at least for the time being.

Yeah, given that we don't have tooling that processes a kickstart
at boot time.

Post by Martin Kolman
I think that would work - would Anaconda be expected to parse the content
of the section or would just writing the content to a file on the target system
where Ignition will pick it up be enough ?

This is a key question. One thing that we're still discussing in the background
is whether or not we *really* need ignition to be run in the initramfs or
not for our primary use cases. If we don't, a whole lot of things get simpler,
it's much more like a better cloud-init that would still run earlier in the boot
process (before /var is mounted ideally so you could dynamically configure
local OS state, or potentially even replace /).

I think for a lot of use cases just writing the file to the target system will be
enough, so we don't need Anaconda integration immediately. That said
I haven't actually tried this yet.

Post by Martin Kolman
The git-like aproach rpm-ostree provides seems to me as a much more
advanced & flexible than just using the
relatively primitive and not very flexibile "you have two and exactly
two copies of everything" approach.

Yeah; this is still being discussed because it has much higher level
ramifications up the stack - the existing CoreOS updater has some sophisticated
bits around the https://github.com/google/omaha protocol for example.
But we're definitely aiming for "best of both" again.

[Martin Kolman]

----- Original Message -----

Sent: Friday, April 20, 2018 4:40:16 PM
Subject: Re: Adopting CoreOS Ignition into Fedora+derivatives ecosystem
A basic thing to keep in mind here is that the existing CoreOS didn't have kickstart
and *only* has the PXE-to-Live model for bare metal (with optional persistent
install), so there's
a lot of overlap going on here. But some of the technology makes sense and
is obviously in use by the existing CoreOS community. We're hoping to have
a "best of both" message.

Yep, that definitely sounds like a good plan. :)

We don't need to make any hard decisions here right away; the existing
Fedora/Atomic and CoreOS will be maintained for a while, but we do
hope to unify.

Post by Martin Kolman
Yeah, Ignittion also seems to me like more like an enhanced Cloudinit with some
installer bits bolted on & it indeed seems like it could work like a
good cloudinit replacement.

One thing that only became a bit clearer to me recently is we actually
https://github.com/coreos/coreos-metadata
which handles the "processed every boot" aspect that one needs in
many cloud environments; however the ignition ~= kickstart still remains,
as well as the ssh-key/useradd type overlap between both of them
and cloud-init/coreos-metadat.

Post by Martin Kolman
Yeah, PXE-to-Live is pretty different to how Anaconda does,
we generally differentiate the installation environment and
the target system quite clearly and in this case they are basically
combined together. So if Ignition can do it it makes sense to
me to continu using it for this usecase, at least for the time being.

Yeah, given that we don't have tooling that processes a kickstart
at boot time.

Technically speaking, Initial Setup[0] does that - it reads the kickstart Anaconda generated
during installation and proceeds accordingly.

There are a couple expectations though:
- IS is run on the first boot of a system after installation (or on arm from pre-prepared image)
- the run is generally one-off and IS is disabled once it has done it's thing
- IS runs on the system after switchroot (eq. not in initrd)

I don't think there was ever a use-case considered where IS would run on every boot and act as a
system configuration agent.

[0] https://github.com/rhinstaller/initial-setup

Post by Martin Kolman
I think that would work - would Anaconda be expected to parse the content
of the section or would just writing the content to a file on the target system
where Ignition will pick it up be enough ?

This is a key question. One thing that we're still discussing in the background
is whether or not we *really* need ignition to be run in the initramfs or
not for our primary use cases. If we don't, a whole lot of things get simpler,
it's much more like a better cloud-init that would still run earlier in the boot
process (before /var is mounted ideally so you could dynamically configure
local OS state, or potentially even replace /).
I think for a lot of use cases just writing the file to the target system will be
enough, so we don't need Anaconda integration immediately. That said
I haven't actually tried this yet.

BTW, we do regenerate initrd during installation so if you had Ignition as
a Dracut module, it could pick up the configuration file from the rootfs
when the initrd is regenerated.
(That's why we regenerate the initrd during installation - so that all packages
and any configuration changes done by Anaconda are in place when Dracut generates
the initrd image.)

Post by Martin Kolman
The git-like aproach rpm-ostree provides seems to me as a much more
advanced & flexible than just using the
relatively primitive and not very flexibile "you have two and exactly
two copies of everything" approach.

Yeah; this is still being discussed because it has much higher level
ramifications up the stack - the existing CoreOS updater has some sophisticated
bits around the https://github.com/google/omaha protocol for example.

Interesting! That's indeed more sophisticated than just a plain A/B system.

But we're definitely aiming for "best of both" again.

Yet again sounds like a good plan. :)

_______________________________________________
Anaconda-devel-list mailing list
https://www.redhat.com/mailman/listinfo/anaconda-devel-list