#ubuntu-ec2 2010-06-16
<flaccid> i'm researching ec2 kernels and mods - what to use for debian particularly
<flaccid> confused on what to use basically
<flaccid> ubuntu lucid i386 linux image 2 6 32 302 ec2 v 2 6 32 302 6 kernel i think will work ok with squeeze
<erichammond> flaccid: The supported Ubuntu kernel images (AKIs) would be the ones which are used by the active public Ubuntu AMIs.
<erichammond> Anything else is potentially subject to removal, which could ruin your day.
<flaccid> well in terms of debian however?
<flaccid> squeeze seems to run well on the lucid aki but needs the linux modules (which i guess i could install from the lucid repos)
<flaccid> erichammond what you think?
<flaccid> http://thecloudmarket.com/image/ami-bb3a6bfe--debian-6-0-squeeze-server#/definition uses older fedora kernel/ramdisk images
<erichammond> I abandoned support for Debian squeeze because of kernel incompatibilities.  I would avoid any Alestic squeeze AMIs.
<flaccid> well i'm finally getting around to the debian support thing
<flaccid> did lenny play nicely enough with the amazon kernels?
<erichammond> I'd recommend building your AMIs from scratch so that you can reliably reproduce the process as time goes on.
<flaccid> erichammond: yep thats what i'm doing now
<flaccid> i have created a servertemplate
<flaccid> erichammond: i've updated http://support.rightscale.com/12-Guides/RightLink/04-Creating_RightScale-enabled_Images_with_RightLink/RightLink_Installer_for_Ubuntu to work with lucid
<flaccid> you can add that to your vmbuilder process which will give the latest rightlink v5 package
<flaccid> so now i'm going back to cover debian
<flaccid> wb
<flaccid> i basically would like to know the link between the kernel and it loading kernel mods from /lib/modules on boot
<flaccid> it appears to be a hardcoded relationship
<flaccid> i'm trying to make the ami independent of the kernel used
<erichammond1> The kernel modules must match the kernel version exactly.
<erichammond> Whenever you run a different kernel, you must have those kernel modules available.
<erichammond> Ubuntu was looking at downloading the kernel modules at run time or adding them to the ARI instead of the AMI.
<erichammond> but I don't think that went anywhere.
<flaccid> yeah its cool to have a switchable kernel but if the mods are not installed then no good
<flaccid> so i guess the ami can only support 'a range of kernels'
<erichammond> Yes, this is the general practice.
<flaccid> so where would i get the kernel modules for http://thecloudmarket.com/image/aki-f70657b2.html?page=6 (aki-f70657b2) ?
<flaccid> this is the default kernel to Debian Squeeze Alestic, root@
<erichammond> Most often it is a single recommended kernel and new AMIs are released when new kernels come out.
<erichammond> flaccid: What region?
<flaccid> us-west atm
<flaccid> i just want to bundle vol with the same kernel and ramdisk as you are and see how it boots
<erichammond> If you have an AMI that uses that kernel, then the kernel modules should be installed on that AMI.
<flaccid> erichammond: thats what i am trying to do
<erichammond> No, I'm saying you could copy the kernel modules from the Alestic AMI.
<flaccid> erichammond: where did they come from originally? i'm debootstrapping
<erichammond> The Alestic Debian AMIs were built with ec2ubuntu-build-ami which would have pointers to the downloadable kernel modules.
<flaccid> must of missed it in http://ec2ubuntu.googlecode.com/svn/trunk/bin/ec2ubuntu-build-ami
<flaccid> http://alestic-downloads.s3.amazonaws.com/ec2-kernel-modules-2.6.21.7-2.fc8xen-ec2-x86_64-20090930.tar.gz
<erichammond> looks right.
<flaccid> doing a re-run now
<erichammond> flaccid: The point, however, was that that old Amazon kernel will no longer work with squeeze.
<flaccid> does it work with lenny?
<flaccid> i'm also creating the bundle on /tmp/image-bundles/foo of the deboostrap root. does not seem like an ebs volume is required for s3 only building
<flaccid> bundled created:
<flaccid> Kernel image	Amazon vmlinuz 2.6.21 2.fc8xen ec2 v1.0.i386.aki
<flaccid> Ramdisk image	Amazon initrd 2.6.21.7 2.fc8xen ec2 v1.0.1.i386.ari
<flaccid> boot instance
<flaccid> doesn't seem to like lenny
<erichammond> flaccid: The old lenny AMI I built months ago still runs.  It's possible lenny upgrades no longer work with that old kernel.
<flaccid> erichammond: i got squeeze going with an ubuntu lucid kernel
<erichammond> flaccid: Congrats on getting squeeze to run.  The ec2debian Google group might be interested in your findings.
<smoser> erichammond, "new 32-bit Karmic ebs server based on Alestic's ami-6743ae0e"
<smoser> when will you stop taking credit for Ubuntu's work ;-)
<erichammond> Where are you reading that?
<smoser> i'm amazed at the namespace you've carved out (and i completely realize that you give Ubuntu credit and correct those statements)
<smoser> ec2ubuntu mail
<erichammond> It's amazed me, too.
<erichammond> I did originally recommend that you put "canonical" in the bucket names to make the source clear to folks.
<erichammond> oh, wait.  That AMI *is* one I published.
<erichammond> I use the Ubuntu image built by Canonical, but then, pretty much every Linux image anybody publishes uses files created by somebody else.
<erichammond> :)
<erichammond> The key is that if I die and Amazon cancels my account, that AMI will disappear :-/
<smoser> gah! it is, you're right.
<smoser> your naming is the same as ours too :)
<smoser> so i was confused.
<erichammond> er, your naming is the same as min :)
<erichammond> e
<smoser> anyway
<smoser> sorry
<erichammond> but your original point is true with other examples.
<smoser> thats what you get for doing a helluva good job
<smoser> i'll respond to that thread with: "I would help you, but i have no idea what that crazy 'alestic.com' does with their images"
<smoser> :)
<erichammond> :)
<erichammond> I think you already know this, but the Karmic EBS boot AMIs I published are exact file system image copies from your published Ubuntu images.
<smoser> yeah i know
<erichammond> smoser: Thanks for all the great support you've been providing to the ec2ubuntu group.  If you'd like to take over approving new posters and filtering spam, let me know :)
<flaccid> erichammond: and smoser yeah guys still not very happy with kernels for debian* atm. still pondering on what to use that will work with at least lenny and squeeze without even the smallest issue
#ubuntu-ec2 2010-06-17
<smoser> flaccid, what are your issues with the ubuntu kernels ?
#ubuntu-ec2 2010-06-18
<flaccid> smoser: http://dpaste.org/Se9H/ only a couple of things - Couldn't get a file descriptor referring to the console; error: unexpectedly disconnected from boot status daemon; FATAL: Could not load /lib/modules/2.6.32-305-ec2/modules.dep: No such file or directory (need to work out best way to install these on squeeze/debian);
<flaccid> and Waiting for /dev to be fully populated...udevd-work[258]: kernel-provided name 'evtchn' and NAME= 'xen/evtchn' disagree, please use SYMLINK+= or change the kernel to provide the proper name
<flaccid> and Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key
<flaccid> most of the above is kernel related. smoser i just need to address/fix the above and all should be good. if you could also advise what the latest actual kernel is that would be great
<smoser> "actual kernel"
<smoser> all the ubuntu published items conform to https://wiki.ubuntu.com/UEC/Images/NamingConvention
<flaccid> yes
<smoser> (or close to it)
<smoser> for released kernels, they'll have ubuntu-images/<...> in their name (per describe-images)
<smoser> er.. for kernels it will be
<flaccid> yeah so what is the latest kernel id ?
<smoser> well, for us-east-1, and lucid (released) that would be:
<smoser> 099720109477/ubuntu-kernels/ubuntu-lucid-amd64-linux-image-2.6.32-305-ec2-v-2.6.32-305.9-kernel
<smoser> aki-0b4aa462
<flaccid> is there someone that lists these ?
<smoser> i have considered putting kernels in the data at http://uec-images.ubuntu.com/query/
<smoser> but largely, we don't consider kernels aki to be a product as much as a side effect of the image
<smoser> that said, http://uec-images.ubuntu.com/query/lucid/server/released.current.txt
<smoser> will have a list of the latest ami
<smoser> and their assicoated aki
<flaccid> thanks that helps a lot
<smoser> there is 'daily.current.txt' there also
<flaccid> now back to the errors if you don't mind helping?
<smoser> and maverick ...
<smoser> so lots to chose from
<smoser> :)
<smoser> i'vve seen the "Couldn't get a file descriptor" error before, but dont really recall what that was about.
<smoser> fwiw, jjohansen is the kernel team member for this ec2
<smoser> i'm looking at some logs
<smoser> we see the "Could'nt get a file descriptor" on karmic currently
<flaccid> cool thanks
<smoser> and on lucid prior to alpha2
<smoser> err.. prior to alpha3
<flaccid> and would also like your advise on installing/enabling the kernel mods from the ubuntu package in debian and if this is an ok practice etc.
<smoser> where did you get the ramdisk ?
<smoser> or am i mistaken:
<smoser> Begin: Loading essential drivers... ...
<smoser> Done.
<smoser> Begin: Running /scripts/init-premount ...
<flaccid> its ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100114 1
<flaccid> ari-3f2d7c7a
<smoser> yeah, so we dont use ramdisk anymore
<flaccid> right so set to default
<smoser> set to default ?
<flaccid> i'm assuming default=none now
<flaccid> i'll relaunch with that
<smoser> yeah, it probably wont hurt or affect it.
<smoser> that has been my experience.  the ramdisk doesn't hurt anything. just doesn't help.
<smoser> regarding installing ubuntu packages, are you asking from licensing perspective ?
<smoser> those modules come from the packages... our ec2 kernels are normal kernle builds and have normal packages.
<smoser> https://launchpad.net/ubuntu/+source/linux-ec2
<smoser> i dont know how well those would install onto a debian
<flaccid> more from an operational point of view. this is to address FATAL: Could not load /lib/modules/2.6.32-305-ec2/modules.dep: No such file or directory
<flaccid> so yes the question is regarding installing them on debian
<flaccid> see because the amazon kernels are not working with debian, i am having to use the ubuntu kernels
<smoser> yeah. so i dont know whether or not the packages there will "just work" (ie install) on debian or not.
<smoser> but almost certainly a dpkg -x linux-image-*.deb would get you the /lib/modules/ tha tyou need
<flaccid> well i would either have to use the lucid repos briefly and install via apt or manually install the package and its deps via dpkg
<flaccid> well the linux-image-* pkgs have deps..
<flaccid> ok well no ramdisk fixed that
<flaccid> zcat: invalid magic - another small one to address
<flaccid> i mean no ramdisk fixed the console descriptor errors
<flaccid> well it still defaulted to ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0
<flaccid> that must be tied to the kernel
<flaccid> wb
<flaccid> smoser: if i do a reboot and i have installed these modules, this should remove the FATAL: Could not load /lib/modules/2.6.32-305-ec2/modules.dep: No such file or directory message right?
<smoser> well maybe
<smoser> you may be getting that from the ramdisk
<smoser> at least i'm fairly certain that ramdisks do 'modprobe' which will result in a message like that.
<flaccid> but if the file is there, it should not return No such file or directory right?
<smoser> are you using a ramdisk ?
<flaccid> this kernel appears to default to ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0
<smoser> if you are using a ramdisk,and it was built with a differnet kernel then it won't have that /lib/modules/<kernel-ver> and it will try to modprobe
<flaccid> aki-296f3e6c
<smoser> kernels dont default to a ramdisk
<smoser> this is something that sucks.
<flaccid> oh but an image defaults to a ramdisk?
<smoser> if you registered an image with a ramdisk and kernel, then you are not able (that I know of) to boot without a ramdisk or kernel
<smoser> and, actually, it goes back step, since you can give ramdisk and kernel to the bundle command
<flaccid> ok so ramdisk is bogus, no need to build image with a ramdisk value?
<smoser> if you've done that, then there is no way to specify "None'
<smoser> right
<smoser> then, you can (if you wanted) specify --ramdisk <ari-abcdefg>
<smoser> but there is no --ramdisk <none>
<flaccid> sweet then i'll omit ramdisk
<smoser> right. thats a superset of the functionality, but then, if you ever wanted a ramdisk, the user has to specify one, which just isn't very friendly
<flaccid> if ec2-bundle-vol rcvs --kernel --ramdisk --generate-fstab will it simply ignore the no param values
<smoser> just do not pass --ramdisk
<smoser> or --kernel
<smoser> on bundle or on register
<smoser> fwiw, there are scripts 'uec-publish-image' that you could use that, in my opinion are more friendly than ec2-bundle, ec2-upload, ec2-register
<flaccid> well i still want a default kernel
<smoser> http://manpages.ubuntu.com/manpages/lucid/man1/uec-publish-image.1.html
<smoser> right.
<smoser> i *think* (you'd have to test this)
<smoser> that you can bundle without anything
<smoser> and then add --kernel when you register the image
<smoser> which, in this case, would allow you to avoid re-bundling
<flaccid> thats cool i'll test that
<flaccid> hmm it doesn't specify params http://docs.amazonwebservices.com/AmazonEC2/dg/2007-01-03/CLTRG-register.html
<smoser> see the date in that url
<smoser> http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-RegisterImage.html
<smoser> is what you want
<flaccid> oh yeah thanks, eek on google!
<flaccid> thanks muchly :)
<flaccid> standby, i'll have some results soon
<flaccid> 1 is building now
<flaccid> the good thing i've noticed so far is that these AMIs i'm creating work fine with the lucid kernel and boot up really fast
<flaccid> hmm that doc doesn't mention -K and -C options yet i am using them currently..
<flaccid> Required option '-K, --private-key KEY' missing (-h for usage)
<flaccid> heh its not in their doc
<flaccid> ah must be in http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-RegisterImage.html
<flaccid> err http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?CLTRG-common-args-api.html
<flaccid> Unable to read instance meta-data for ramdisk-id
<flaccid> Unable to read instance meta-data for product-codes
<flaccid> ^^ from ec2-bundle-vol. what should i do about this ?
<uvirtbot> flaccid: Error: "^" is not a valid command.
<flaccid> smoser: built and registered with no ramdisk, a launch with default assigns ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0
<flaccid> the default kernel being assigned is ubuntu-kernels-testing/ubuntu-lucid-i386-linux-image-2.6.32-306-ec2-v-2.6.32-306.11-kernel
<smoser> those errors don't surprise me.
<flaccid> i guess we'll see in console output if it is wanting 306 or 301 modules that will determine if the ramdisk is asking for it or not
<flaccid> yes but i want to get rid of those errors :)
<smoser> no, the "unable to read instance metadat" errors from bundle-vol
<smoser> they dont surprise me
<flaccid> yeah i want to fix those..
<smoser> bundle-vol is a yucky hack. you shouldn't rely on it.
<flaccid> well thats what aws publishes..
<smoser> bundle-vol basically has to "un-boot" a system.
<smoser> which is just simply going to require a bunch of distribution/OS specific HACKs
<smoser> its much cleaner to use debootstrap and bundle-img
<flaccid> well FATAL: Could not load /lib/modules/2.6.32-306-ec2/modules.dep: No such file or directory. so its the kernel that wants it. i'll extra the mods in this build
<flaccid> where is bundle-img ?
<flaccid> i am using debootstrap
<smoser> ec2-ami-tools
<flaccid> oh you mean ec2-bundle-img
<smoser> yeah
<smoser> can you give me output of describe-istances for the image you booted ?
<smoser> i think you booted with a ramdisk.
<smoser> or, you dont 'have those modules in the root filesystem
<smoser> ec2-describe-instances i-abcdefg
<flaccid> for some reason that returns Client.InvalidInstanceID.NotFound: The instance ID 'i-ba897dfe' does not exist
<flaccid> yet its running
<flaccid> i didn't include the modules in that build however
<flaccid> rightscale dashboard is indicating use of ubuntu lucid i386 linux image 2 6 32 301 ec2 v 2 6 32 301 4 ramdisk 20100108 smoser0
<smoser> it maybe that rightscale is buggy
<smoser> if you did not bundle with '--ramdisk'
<smoser> and did not register with '--ramdisk'
<smoser> then you should not have a kernel
<smoser> err... you should not have a ramdisk
<smoser> it is possible that rightscale is being nice, and trying to use that one, in which case, that really sucks.
<smoser> but describe-instances certinaly should be able to give you output
<smoser> what about describe-images of the ami
<flaccid> that returns does not exist as well. no idea why
<flaccid> hmm are these region specific?
<flaccid> i think its only returning us-east when i need us-west
<smoser> yes. all of it is region specific
<smoser> each region is its own "cloud". the only data they share is your account id
<flaccid> nps
<flaccid> this is the image
<flaccid> IMAGE	ami-b5b8e9f0	041819229125/RightImage-Debian-squeeze-20100618033019-i386-v5.4.6	041819229125	available	private		i386	machine	aki-3197c674			instance-store
<flaccid> this is the instance
<flaccid> INSTANCE	i-ba897dfe	ami-e7b6e7a2	ec2-204-236-179-7.us-west-1.compute.amazonaws.com	ip-10-162-143-68.us-west-1.compute.internal	running	CJF: AWS US-West (testing)	0		m1.small	2010-06-18T03:39:43+0000	us-west-1b	aki-c18edf84	ari-8f2978ca		monitoring-disabled	204.236.179.7	10.162.143.68			instance-store	
<flaccid> so i guess either rightscale is allocating ari-8f2978ca or ec2 (likely rightscale)
<flaccid> can you help confirm that there is no such thing as a 'default' ramdisk when registering or launching?
<flaccid> we seem to be using 'default' with a reserved ari then..
<smoser> yep
<smoser> its rightscale is being "nice"
<smoser> well, when you register or bundle and pass '--ramdisk' that gives you a "default ramdisk"
<smoser> and if you do that, you can't ever run an instance without a ramdisk.
<smoser> but, the describe-images info above shows that the image has no ramdisk associated with it.
<smoser> which means either
<smoser> a.) the user must provide one in run-instances
<smoser> b.) there will not be one used
<flaccid> that would of came up in the describe-images but right? i took this out of the build process
<flaccid> yep we are on same level
<smoser> right. it would be shown in describe-images output
<flaccid> i will raise a bug for the rightscale dashboard issue
<smoser> look at any other one, you'll see an ari
<smoser> fwiw, that 'smoser0' is a debug ramdisk :)
<smoser> (with some interesting characteristics, it can process user data)
<flaccid> hehe, looks like we chose your ramdisk image for the default... :)
<smoser> our choice to not have ramdisks has thrown errors much furrther than i thought it would have
<flaccid> there is issues with no ramdisk?
<flaccid> is there any way to remove a debootstrap target dir? i.e. you can't remove /proc and /dev
<smoser> you have to unmount proc and dev first
<smoser> then remove
<smoser> there are no issues with no ramdisk other than tools that expected a ramdisk
<smoser> UEC had fits (Eucalyptus)
<smoser> righscale apparently had fits
<smoser> the expectation of an ari was just hard coded many places
<smoser> oh yeah, other places.
<smoser> landscape
<smoser> and the canonical "image store" (part of UEC)
<flaccid> i see
<flaccid> ok now launching with a new image
<flaccid> smoser: extracting from the .deb for those modules doesn't give the desired files
<smoser> no?
<smoser> you're right.sorry.
<flaccid> i'll cp them across from the worker which is lucid
<smoser> just the wrong package
<smoser> the linux-ec2 is a metapackage
<smoser> that depends on the right linux-image-2.6.32-305-ec2
<flaccid> i'm extracting linux-image-2.6.32-305-ec2_2.6.32-305.9_i386.deb
<flaccid> i tell ya what but it boots so fast
<smoser> http://packages.ubuntu.com/hu/lucid/linux-image-2.6.32-305-ec2
<smoser> yeah, thats what you want
<flaccid> hopefully cp -R /lib/modules/2.6.32-305-ec2 "$DEBOOTSTRAP_TARGET_DIR"/lib/modules/ is good enough..
<smoser> i'd think so
<flaccid> i'm getting 1.2min startups
<flaccid> 14.5mins to create and build an AMI
<flaccid> smoser: only a few things left to address now, http://dpaste.org/5fAp/
<smoser> start up is all over the map
<smoser> what do you think is left ?
<smoser> it looks pretty good
<smoser> what do you mjean by "1.2min startups" ?
<smoser> total time to login/ssh from what
<smoser> flaccid,
<flaccid> i think thats the boot time, not sure how its configured
<flaccid> need to address the ' WARNING: Currently emulating unsupported memory accesses'
<flaccid> there is no /lib/tls on that instance
<flaccid> also Waiting for /dev to be fully populated...udevd-work[86]: kernel-provided name 'evtchn' and NAME= 'xen/evtchn' disagree, please use SYMLINK+= or change the kernel to provide the proper name
<flaccid> and the few under cannot unset powersave mode
<flaccid> and Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key
<flaccid> and maybe /build/buildd/linux-ec2-2.6.32/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
<flaccid> also Setting kernel variables ...error: "net.ipv6.bindv6only" is an unknown key
<flaccid> i'll build a lenny now
<flaccid> lenny works but not with rightlink
<smoser> i'm going to bed. good luck.
<flaccid> thanks; cya later
