Anaconda and cloud-init

[Colin Walters]

Fedora right now has three products, and I'm going to ignore Workstation for this post.

Server: Download boot.iso, configure via kickstart
Cloud: Download qcow2 image, configure via cloud-init

Anaconda is used in generating the Cloud image (see my previous post about ImageFactory), with the resulting issues there.

But note the kickstart functionality is not accessible to the Cloud user. Instead, cloud-init is the pervasive way to configure cloud images.

But if you look at this, cloud-init actually has a lot of great features that *also* make sense on bare metal. A good example is ssh key provisioning. Look at the "ssh-authorized-keys" entry in the YAML here:

http://cloudinit.readthedocs.org/en/latest/topics/examples.html#including-users-and-groups

Kickstart has no declarative syntax to update the ssh keys for a user. And this lack of support is actually reflected in the Anaconda UI/framework logic - it won't let you proceed unless you have a root password or an administrative user (with a password) But ssh keys are *more* secure than passwords, and we should strongly encourage their use.

In the RHEL6 era, people of course did it in %post, but there was a common trap around SELinux labeling:
https://www.redhat.com/archives/rhelv6-beta-list/2010-November/msg00005.html

Basically I'd like to discuss pulling in cloud-init into the installation environment. Ideally, I could chain kickstart -> cloud-config.

Say a new toplevel verb like this in %kickstart:

%cloud-config
users:
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end

The simplest way to implement this would be to simply dump the file into /var/tmp on the target system, and then have the cloud-init program execute it on boot.

But to fix the issue of anaconda being unaware of ssh keys as a valid way to log in would require parsing it in the install environment and looking at the users stanza.

At a high level, I think this is a good architectural match; both programs are Python 2, etc. (cloud-init is GPLv3 though...hmm)
cloud-init also has a lot of popularity due to its cross-distribution nature.

Finally, the reason I bring this up is because I work on Project Atomic, and as part of that one of the features we're investigating is a PXE-to-Live scenario. For that, cloud-init is a perfect match, because it's designed to configure an already existing system (versus kickstart which is designed to be processed for an installation). Laying the foundation with regular bare metal support for cloud-init would help this as well.

[David Shea]

Post by Colin Walters
%cloud-config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end

Looking at this, it looks awfully similar to:

%addon org_fedora_cloud_config
users:
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end


and that's something that anaconda already provides the framework for

[Colin Walters]

Post by David Shea

Post by Colin Walters
%cloud-config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end

%addon org_fedora_cloud_config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end
and that's something that anaconda already provides the framework for

Interesting, might investigate that. But...personally I would really love if this was integrated into the core. For example, if there was some button in the UI where I could just type in a kickstart (and cloud-config) instead of clicking in the GUI.

In addition, a UI way to just enter in a URL for a cloud-config.

(Yes, you can do this before you boot via TAB on the commandline and inst.ks, but that's hidden and unfriendly, if you typo the URL you have to reboot and try again, etc.)

[Martin Kolman]

Post by Colin Walters

Post by David Shea

Post by Colin Walters
%cloud-config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end

%addon org_fedora_cloud_config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end
and that's something that anaconda already provides the framework for

Interesting, might investigate that. But...personally I would really love if this was integrated into the core. For example, if there was some button in the UI where I could just type in a kickstart (and cloud-config) instead of clicking in the GUI.
In addition, a UI way to just enter in a URL for a cloud-config.
(Yes, you can do this before you boot via TAB on the commandline and inst.ks, but that's hidden and unfriendly, if you typo the URL you have to reboot and try again, etc.)

Anaconda dddons can also provide a GUI in the form of a new spoke and
the the OSCAP[0] or KDUMP[1] addons already do that. :)

So your "cloud config" addon might can do the same thing - provide
options accessible from its %addon section in kickstart, but it can also
have a GUI screen for graphical configuration. :) See the Anaconda addon
development guide[2] for more information about Anaconda addons.

[0] https://fedorahosted.org/oscap-anaconda-addon/
[1] https://github.com/daveyoung/kdump-anaconda-addon
[2] https://vpodzime.fedorapeople.org/anaconda-addon-development-guide/

[Colin Walters]

Post by Martin Kolman
Anaconda dddons can also provide a GUI in the form of a new spoke and
the the OSCAP[0] or KDUMP[1] addons already do that. :)

This needs designer input, but I was more thinking that ssh keys would be part of the existing authentication spoke.

[Brian C. Lane]

Post by Colin Walters
The simplest way to implement this would be to simply dump the file into /var/tmp on the target system, and then have the cloud-init program execute it on boot.

Can cloud-init be run from %post? It is chrooted into the target
filesystem.

Post by Colin Walters
But to fix the issue of anaconda being unaware of ssh keys as a valid way to log in would require parsing it in the install environment and looking at the users stanza.

I've been meaning to add some way for ssh keys to be included in the
kickstart. Yes, that's just one part of your point, but it will probably
show up in F22.

[Colin Walters]

Post by Brian C. Lane

Post by Colin Walters
The simplest way to implement this would be to simply dump the file into /var/tmp on the target system, and then have the cloud-init program execute it on boot.

Can cloud-init be run from %post? It is chrooted into the target
filesystem.

I looked at the code, and it doesn't today. It looks like this would be a lot of work.

I'm not sure there are many advantages to that either versus having it just run on boot. And actually doing so would break features like the "phone-home" where the entire point is that they happen on boot.

Post by Brian C. Lane
I've been meaning to add some way for ssh keys to be included in the
kickstart. Yes, that's just one part of your point, but it will probably
show up in F22.

Ok, that would make sense, and it would help lay the groundwork for solving bugs like the "must have password" issue.

I scanned the cloud-config examples again and I guess there isn't a lot more that's really compelling. A more minor one is cloud-config has explicit and flexible sudo support, whereas with kickstart you only have user --name=foo --groups=wheel. (Though of course it's not hard to write into /etc/sudoers.d from %post).


That said, even skipping the ssh key example, there'd be a lot of value in making it convenient to use from kickstart regardless - admins could take their existing cloud-init configurations and migrate to bare metal more easily. And it's fundamentally necessary for PXE-to-Live, because Anaconda/kickstart isn't involved on the user side there. (We're planning to use LMC on the server side of course).

[Chris Lumens]

Post by Colin Walters
Ok, that would make sense, and it would help lay the groundwork for
solving bugs like the "must have password" issue.

Both rootpw and user support a --lock option, which allows you to skip
having a password. It will, of course, also lock the account so you
cannot login. Perhaps this helps?

- Chris

[Leslie S Satenstein]

Has anyone else experienced problems with RC5.  I get a hard lockup with trying to reset value into hostname field

https://dl.fedoraproject.org/pub/alt/stage/21_RC5/

Bug 1170803
Tested network install version and retested with workspace version. Both lockup
 Is this the version going live on the 9th?

[Adam Williamson]

Has anyone else experienced problems with RC5. I get a hard lockup with trying to reset value into hostname field
https://dl.fedoraproject.org/pub/alt/stage/21_RC5/
Bug 1170803
Tested network install version and retested with workspace version. Both lockup
Is this the version going live on the 9th?

Yes. I just tested setting hostname in the network spoke in a VM, it
worked fine.

[Martin Kolman]

On Fri, 2014-12-05 at 16:36 -0500, Colin Walters wrote:
<snip>

Post by Colin Walters
Basically I'd like to discuss pulling in cloud-init into the installation environment. Ideally, I could chain kickstart -> cloud-config.
%cloud-config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end
The simplest way to implement this would be to simply dump the file into /var/tmp on the target system, and then have the cloud-init program execute it on boot.
But to fix the issue of anaconda being unaware of ssh keys as a valid way to log in would require parsing it in the install environment and looking at the users stanza.
At a high level, I think this is a good architectural match; both programs are Python 2, etc.

Do they have a Python 3 version/support as well ? We are in the process
of dropping Python 2 only dependencies (pyblock, urlrgrabber, etc.) so
that Anaconda can run with Python 3, as a needed by the Fedora 22
"Python 3 as Default" system-wide change[0]. So adding new Python 2 only
dependencies would not make much sense.

Post by Colin Walters
(cloud-init is GPLv3 though...hmm)
cloud-init also has a lot of popularity due to its cross-distribution nature.
Finally, the reason I bring this up is because I work on Project Atomic, and as part of that one of the features we're investigating is a PXE-to-Live scenario. For that, cloud-init is a perfect match, because it's designed to configure an already existing system (versus kickstart which is designed to be processed for an installation). Laying the foundation with regular bare metal support for cloud-init would help this as well.

[0] http://fedoraproject.org/wiki/Changes/Python_3_as_Default

[Colin Walters]

Post by Martin Kolman
Do they have a Python 3 version/support as well ? We are in the process
of dropping Python 2 only dependencies (pyblock, urlrgrabber, etc.) so
that Anaconda can run with Python 3, as a needed by the Fedora 22
"Python 3 as Default" system-wide change[0].

I'm concerned about the additional difficulty that would entail in backporting features to RHEL.

Post by Martin Kolman
So adding new Python 2 only
dependencies would not make much sense.

Well, cloud-init would add new visible end user features. Python 3 is a much more indirect end user benefit.

I suspect at a higher level, a Python 3 effort would likely consume most of Anaconda's resources for the Fedora 22 timeframe, and leave comparatively much less time for major features.

[Matthew Miller]

Post by Martin Kolman

Post by Colin Walters
At a high level, I think this is a good architectural match; both
programs are Python 2, etc.

Do they have a Python 3 version/support as well ? We are in the process
of dropping Python 2 only dependencies (pyblock, urlrgrabber, etc.) so
that Anaconda can run with Python 3, as a needed by the Fedora 22
"Python 3 as Default" system-wide change[0]. So adding new Python 2 only
dependencies would not make much sense.

Yes — cloud-init is part of the python3 change, and there's a 2→3 patch
at https://code.launchpad.net/~harlowja/cloud-init/py2-3

[Colin Walters]

Post by Matthew Miller

Post by Martin Kolman

Post by Colin Walters
At a high level, I think this is a good architectural match; both
programs are Python 2, etc.

Do they have a Python 3 version/support as well ? We are in the process
of dropping Python 2 only dependencies (pyblock, urlrgrabber, etc.) so
that Anaconda can run with Python 3, as a needed by the Fedora 22
"Python 3 as Default" system-wide change[0]. So adding new Python 2 only
dependencies would not make much sense.

Yes — cloud-init is part of the python3 change, and there's a 2→3 patch
at https://code.launchpad.net/~harlowja/cloud-init/py2-3

We're getting off into 2 → 3 architecture, but at least for Atomic, I am
committed to shipping Python 2 on the host system for "a long time"[1]
due to Ansible ( http://docs.ansible.com/faq.html#how-do-i-handle-python-pathing-not-having-a-python-2-x-in-usr-bin-python-on-a-remote-machine )

It injects Python (2) programs from remote machines onto the target.
In that case, Python 3 would basically mean I have to carry both.

That plus non-packaged custom scripts makes it look like to me the only realistic outcome of
this push is going to be both on the majority of non-minimal/embedded systems. Which doesn't
mean it's a fundamentally bad idea. Maybe just change the term "switching" on the change page
to "add".

[1] No formal commitment on timeframe, it was a verbal agreement with a major consumer of the OS, but if I had to handwave this, "long time" = "RHEL major version".

[Martin Kolman]

Post by Colin Walters

Post by Matthew Miller

Post by Martin Kolman

Post by Colin Walters
At a high level, I think this is a good architectural match; both
programs are Python 2, etc.

Do they have a Python 3 version/support as well ? We are in the process
of dropping Python 2 only dependencies (pyblock, urlrgrabber, etc.) so
that Anaconda can run with Python 3, as a needed by the Fedora 22
"Python 3 as Default" system-wide change[0]. So adding new Python 2 only
dependencies would not make much sense.

Yes — cloud-init is part of the python3 change, and there's a 2→3 patch
at https://code.launchpad.net/~harlowja/cloud-init/py2-3

We're getting off into 2 → 3 architecture, but at least for Atomic, I am
committed to shipping Python 2 on the host system for "a long time"[1]
due to Ansible ( http://docs.ansible.com/faq.html#how-do-i-handle-python-pathing-not-having-a-python-2-x-in-usr-bin-python-on-a-remote-machine )
It injects Python (2) programs from remote machines onto the target.
In that case, Python 3 would basically mean I have to carry both.
That plus non-packaged custom scripts makes it look like to me the only realistic outcome of
this push is going to be both on the majority of non-minimal/embedded systems. Which doesn't
mean it's a fundamentally bad idea. Maybe just change the term "switching" on the change page
to "add".

Well, I think that the current plan is to switch Anaconda to Python 3
once all dependencies can run with Python 3, not making it 2 & 3
compatible, due to all the additional development & testing overhead.

On the other hand I think Blivet & Pykickstart will use the 2 & 3
compatibility route, as they are used on various (older) systems and not
just on Fedora.

Post by Colin Walters
[1] No formal commitment on timeframe, it was a verbal agreement with a major consumer of the OS, but if I had to handwave this, "long time" = "RHEL major version".
_______________________________________________
Anaconda-devel-list mailing list
https://www.redhat.com/mailman/listinfo/anaconda-devel-list

[Radek Vykydal]

To me it seems there are several possible levels of cloud-init
integration here

1) first step would be to allow configuration of cloud-config source and
its application after reboot - by means of ks or gui, by specifying URL
or supplying the configuration values. Anaconda would just add
appropriate boot option (eg ds="nocloud-net;seedfrom=<URL>" for
cloud-init --url=<URL> in kickstart) and perhaps drop the cloud-config
to the target filesystem for the configuration created in installer. No
processing of cloud-init would happen in installer. So for users/keys
case we'd then probably need a way to configure users in ks saying that
they will be configured by other means so incomplete configuration is
valid. Users/keys in kickstart and cloud-config need to be kept in
accord by the user.

2) Another step may be modifying installation by cloud-config (ie users
configuration in installer derived from cloud-config values).

3) For both approaches we may want to generate/modify cloud-config via
GUI (addon?).

I have doubts about making cloud-init embedded in kickstart as
additional installer configuration using another format (2) above).
Keeping the domains of the tools strictly separated (ks for
installation, cloud-init for post-reboot configuration 1) above) seems
like less pain and cleaner way to me. But maybe the interference between
the two in installer is manageable easier than I imagine.

Post by Colin Walters
Fedora right now has three products, and I'm going to ignore Workstation for this post.
Server: Download boot.iso, configure via kickstart
Cloud: Download qcow2 image, configure via cloud-init
Anaconda is used in generating the Cloud image (see my previous post about ImageFactory), with the resulting issues there.
But note the kickstart functionality is not accessible to the Cloud user. Instead, cloud-init is the pervasive way to configure cloud images.
http://cloudinit.readthedocs.org/en/latest/topics/examples.html#including-users-and-groups
Kickstart has no declarative syntax to update the ssh keys for a user. And this lack of support is actually reflected in the Anaconda UI/framework logic - it won't let you proceed unless you have a root password or an administrative user (with a password) But ssh keys are *more* secure than passwords, and we should strongly encourage their use.
https://www.redhat.com/archives/rhelv6-beta-list/2010-November/msg00005.html
Basically I'd like to discuss pulling in cloud-init into the installation environment. Ideally, I could chain kickstart -> cloud-config.
%cloud-config
- name: walters
ssh-authorized-keys: ssh-rsa ...
%end
The simplest way to implement this would be to simply dump the file into /var/tmp on the target system, and then have the cloud-init program execute it on boot.
But to fix the issue of anaconda being unaware of ssh keys as a valid way to log in would require parsing it in the install environment and looking at the users stanza.
At a high level, I think this is a good architectural match; both programs are Python 2, etc. (cloud-init is GPLv3 though...hmm)
cloud-init also has a lot of popularity due to its cross-distribution nature.
Finally, the reason I bring this up is because I work on Project Atomic, and as part of that one of the features we're investigating is a PXE-to-Live scenario. For that, cloud-init is a perfect match, because it's designed to configure an already existing system (versus kickstart which is designed to be processed for an installation). Laying the foundation with regular bare metal support for cloud-init would help this as well.
_______________________________________________
Anaconda-devel-list mailing list
https://www.redhat.com/mailman/listinfo/anaconda-devel-list

[Colin Walters]

Post by Radek Vykydal
1) first step would be to allow configuration of cloud-config source and
its application after reboot - by means of ks or gui, by specifying URL
or supplying the configuration values.

Below you mention doubting the "embed cloud-init in kickstart" approach,
so how would a user specify a cloudconfig?

I could imagine an
inst.cloudconfig=http://example.com/blah.yml
kernel commandline option?

This seems like it could work quite well if cloud-init covers everything
you need. Conversely, looking at the feature overlap of kickstart
and cloud-init, the latter is lacking sophisticated storage management.

In other words, you could only use inst.cloudconfig if you were happy
with autopart. Even that strikes me as a bit too magical.

This sort of thing is why I was thinking it was most realistic to combine
the strengths of the two and support embedding cloud-init in kickstart.

Then it's only for PXE-to-Live that you can skip kickstart and use
cloud-init alone.

Post by Radek Vykydal
Anaconda would just add
appropriate boot option (eg ds="nocloud-net;seedfrom=<URL>" for
cloud-init --url=<URL> in kickstart) and perhaps drop the cloud-config
to the target filesystem for the configuration created in installer. No
processing of cloud-init would happen in installer. So for users/keys
case we'd then probably need a way to configure users in ks saying that
they will be configured by other means so incomplete configuration is
valid. Users/keys in kickstart and cloud-config need to be kept in
accord by the user.

Yeah. I think we want a way to declare this in kickstart regardless of the
cloud-init effort.

Post by Radek Vykydal
2) Another step may be modifying installation by cloud-config (ie users
configuration in installer derived from cloud-config values).

This is an interesting topic on its own; what would be the value of
doing useradd at install time versus on first boot?

Post by Radek Vykydal
3) For both approaches we may want to generate/modify cloud-config via
GUI (addon?).

Also for kickstart. I'm sure this has come up before, but I'd love
a button that let me interactively input a URL. And another that
launched gedit.

[Chris Lumens]

Post by Colin Walters
Also for kickstart. I'm sure this has come up before, but I'd love
a button that let me interactively input a URL. And another that
launched gedit.

I have a patch that fires up a VTE with ksshell in it (see pykickstart
package), which is pretty close to what you are asking for here. The
problem is that getting anaconda to notice totally different kickstart
data is not trivial. It might be easier (though pretty jarring) to just
restart anaconda once that process is done.

- Chris

[Radek Vykydal]

Post by Colin Walters

Post by Radek Vykydal
1) first step would be to allow configuration of cloud-config source and
its application after reboot - by means of ks or gui, by specifying URL
or supplying the configuration values.

Below you mention doubting the "embed cloud-init in kickstart" approach,
so how would a user specify a cloudconfig?

My concern here was not specifying cloudconfig in kickstart (be it
poniting URL
or by embedding the yaml) which is OK, but using the content
of cloudconfig "as additional installer configuration" (vs using
post-reboot only)
by which I mean parsing it in installer and changing installation data
(eg those concerning user settings) based on it.

Post by Colin Walters
I could imagine an
inst.cloudconfig=http://example.com/blah.yml
kernel commandline option?
This seems like it could work quite well if cloud-init covers everything
you need. Conversely, looking at the feature overlap of kickstart
and cloud-init, the latter is lacking sophisticated storage management.
In other words, you could only use inst.cloudconfig if you were happy
with autopart. Even that strikes me as a bit too magical.
This sort of thing is why I was thinking it was most realistic to combine
the strengths of the two and support embedding cloud-init in kickstart.

Yes, definitely using both. But again, kickstart for installation,
cloud-config for first boot configuration. I am not sure if
benefits of using cloud-config as another installation configuration
source are worth it. I mean are there enough reasons we should
make anaconda care about content of cloudconfig (read and
use it during installation)? Do we have any other examples
then users / ssh keys case?

Post by Colin Walters
Then it's only for PXE-to-Live that you can skip kickstart and use
cloud-init alone.

Post by Radek Vykydal
Anaconda would just add
appropriate boot option (eg ds="nocloud-net;seedfrom=<URL>" for
cloud-init --url=<URL> in kickstart) and perhaps drop the cloud-config
to the target filesystem for the configuration created in installer. No
processing of cloud-init would happen in installer. So for users/keys
case we'd then probably need a way to configure users in ks saying that
they will be configured by other means so incomplete configuration is
valid. Users/keys in kickstart and cloud-config need to be kept in
accord by the user.

Yeah. I think we want a way to declare this in kickstart regardless of the
cloud-init effort.

Post by Radek Vykydal
2) Another step may be modifying installation by cloud-config (ie users
configuration in installer derived from cloud-config values).

This is an interesting topic on its own; what would be the value of
doing useradd at install time versus on first boot?

I think my example was misleading, creating users seems more like a job
for cloudinit. Anaconda shouldn't just require creating of users in case
cloudinit takes care of it. The difference between 1) an 2) level of
cloud-init
support here would be that, having users configuration in cloudinit,
in 1) user would need to specify something like users --otherconfig in
kickstart, while
in 2) anaconda would automagically, based on reading the cloudconfig, not
require a user created in installer.

Radek

[Colin Walters]

Post by Radek Vykydal
My concern here was not specifying cloudconfig in kickstart (be it
poniting URL
or by embedding the yaml) which is OK, but using the content
of cloudconfig "as additional installer configuration" (vs using
post-reboot only)
by which I mean parsing it in installer and changing installation data
(eg those concerning user settings) based on it.

Right, having Anaconda parse it would be the major difficult step.

Post by Radek Vykydal
Yes, definitely using both. But again, kickstart for installation,
cloud-config for first boot configuration. I am not sure if
benefits of using cloud-config as another installation configuration
source are worth it. I mean are there enough reasons we should
make anaconda care about content of cloudconfig (read and
use it during installation)? Do we have any other examples
then users / ssh keys case?

I posted about this in another mail but from what I can see it
is mainly just the ssh key handling.

So if for Fedora 22 kickstart gained ssh functionality that would
close most of the gap. I'd be OK with that.

Though questions remain how to expose it in
the UI if at all, and whether some of the details of how cloud-init
does it are applied (for example, also disabling ssh password auth).

[Colin Walters]

Performing some thread necromancy here:

http://www.projectatomic.io/blog/2015/06/creating-a-simple-bare-metal-atomic-host-cluster/

is a recent blog entry which shows how to link together
kickstart + cloud-init.

This helps solve several problems beyond just the SSH key and
other minor kickstart vs cloud-init differences.

For example, you can `docker pull` in cloud-init,
whereas one can't in kickstart as the installer environment
isn't set up to run Docker.

[Radek Vykydal]

Yes, that's an option (one can even embed the cloud config data right in
kickstart instead of fetching it with curl). One question is
installation-time cloud-init configuration (eg for bunch of machines) vs
boot-time cloud-init configuration (which could differ for specific
machines from the bunch). But perhaps two cloud-init sources - 1)
/var/cloud-init/ created at install time and 2) nocloud-net (ds= boot
option) set up when booting a machine - can be used at the same time
seamlessly?

Post by Colin Walters
http://www.projectatomic.io/blog/2015/06/creating-a-simple-bare-metal-atomic-host-cluster/
is a recent blog entry which shows how to link together
kickstart + cloud-init.
This helps solve several problems beyond just the SSH key and
other minor kickstart vs cloud-init differences.
For example, you can `docker pull` in cloud-init,
whereas one can't in kickstart as the installer environment
isn't set up to run Docker.
_______________________________________________
Anaconda-devel-list mailing list
https://www.redhat.com/mailman/listinfo/anaconda-devel-list