Register Discussions Communities Projects Download Source Browser

July 29, 2010

links for 2010-07-29

  • "Intel's rebates amounted to 38 per cent of Dell's operating profit in the fiscal year 2006, and rose to 76 per cent (or $720m) in one quarter alone, Q1 2007. While almost all of the Intel funds were incorporated into Dell's component costs, Dell did not disclose the existence, much less the magnitude, of the Intel exclusivity payments." — Entirely jaw-dropping disclosure shows Dell was not the profitable behemoth it appeared. I wonder just how much of this Intel has done over the years?
    (tags: Fraud Dell Intel)
  • "Do you know where your data is in the cloud?" Fascinating map to explore that summarises the privacy legislation in each country around the world.
  • O'Grady has come to the same conclusion that I did at OSCON, namely that the "open source bubble" may be over – the period where it was assumed open source would be directly monetised – and it's time for a resurgence of the traditional approach of many parties with their core business elsewhere synchronising fragments of their interests with fragments of the interests of others. The announcement of OpenStack was iconic, but there were plenty of other signs once I'd started looking for them.

There's naturally plenty of scope for variety, but I am certain that businesses whose leverage of open source extends only to the licence and not the community will find it increasingly hard to survive. Those whose revenue is derived from software must have models that at worst cope with and at best leverage the presence of many interests in the communities upon which they depend.

  • Oracle shut down its three PostgreSQL build farm servers without warning…"If they had given us, say, three months warning, I'd have been less peeved," Dunstan told iTnews. "It can't have been costing them much – the thing pretty much runs itself, and they can't be short on hardware."

  • ✍ Jailbreaking Decision Is Temporary Relief

    While the decision by the US Library of Congress to create exceptions to the Digital Millennium Copyright Act for unlocking cellphones and jailbreaking iPhones (among other things) in the USA are very welcome, the reaction has been just a touch too euphoric. Not by everyone, mind you. Dan Gilmore begins to explain why this isn’t a solution, and Wendy Seltzer nudges close to the problem as well. But plenty of people think they’ve been granted more than they really have.

    There are two big reasons I’m only vaguely impressed, and neither of them are down to the fact that I live in a different jurisdiction where the trend is opposite. One concerns market power and its potential abuse, the other concerns global trade. Both lead to a toxicity which is harmful to digital liberty in general.

    Market Power

    First, market power. This action only removes one weapon (admittedly a nasty one) from the arsenal that is available to Apple and other behemoth corporations to control the market in which they operate. Unlike printer manufacturers, they can no longer file suit against you under the DMCA when you want to operate outside the patterns they have deemed acceptable for their customers and partners. They should probably never have been able to – copyright already has plenty of exceptions concerning fair use and reverse engineering that should have been respected when the DMCA was created. But Apple won’t have a problem enforcing their will without it. The contractual terms they are able to impose on both their partners and their customers do the trick perfectly well.

    Yes, participation is optional, but to avoid getting burned you have to stay out of their kitchen completely. As a customer it’s too late to discover your device is incompatible with something you want to do after you’ve invested in it for a few months, and as a partner it’s too late to discover your business is too close to something Apple would rather control and own once you’ve submitted the app to the store. They can’t have you thrown in jail so easily any more, but they can just as easily prevent you participating and impose sanctions if you fight back.. They will no doubt continue to do so as capriciously as they have already.

    Having just got a great Android phone, I’m less gloomy about this than I was; Android is a powerful Foundation to Apple’s Empire (as long as there’s no Mule – check Wikipedia if you’re behind on your Asimov).  It’s possible that the removal of the DMCA as a blunt instrument may be enough to balance the market forces and promote true competition, facilitated by open source.

    Global Trade

    Second, global trade. US legal norms for technology businesses for patents and copyrights may still be forming (for patents they are still “only” the result of case law), but that hasn’t stopped the US Trade Representative (USTR) and US trade missions globally from treating them as if they were handed down on stone tablets. They have been using conformance with “US norms” as a trading card in their rough games of political poker with various world governments. You know the sort of thing. “Nice export industry you have there for your agricultural produce. It would be a shame if anything happened to it. You can make sure it doesn’t if you legislate to prevent your citizens harming our noble media industries.” Kipling wrote about it eloquently, but people are still paying the USTR-geld.

    Which is probably the intent of the copyright- and patent-dependent companies sponsoring the action anonymously through their trade associations. If they can get foreign governments to make hard rules where they can only persuade their own governments to make soft rules, the battle is all but won for them. They can use “international harmonisation” as the justification to get the draconian rules reinstated. That seems to be the reason ACTA has been given so much endorsement by the USA, as well as why they have been so keen on veiling its proceedings in secrecy. It’s not just USTR either – the equivalent functions in the European Commission seem to be working just as hard against their citizens’ interests.

    Muted Applause

    My positive reaction to the decision by the Library of Congress is thus more out of a sense of gratitude toward the Electronic Frontier Foundation (EFF) and their excellent and insightful team than any long-term relief. It was largely their work that produced this advance and it will likely be their work that holds back the process I’ve outlined. No wonder the media industry puppets hate this sort of organisation. But no-one should believe for a moment that this development makes it OK to jailbreak your iPhone. Doing that is asking for trouble. Its masters have merely been pushed back a layer in their defences, and temporarily at that.


    July 27, 2010

    How to use ISC DHCP for the OpenSolaris Automated Installer

    DHCP, DHCP, DHCP everywhere!

    Most everyone is used to using DHCP. It's used at coffee shops and wireless networks to acquaint traveling laptops with their DNS and router settings as well as, of course, to provide the machines an IP address too. However, corporate and enterprise use of DHCP is often reasonable too. One can use dynamic DNS updates to handle having a static reference for a machine traveling on various networks. When network migrations are necessary (i.e. say your Fortune 500 gets bought by another) and you need to move thousands of machines, it's much easier to simply tell your DHCP server to migrate the machines than have to log on to each and every machine and change network settings.

    How does DHCP apply to the OpenSolaris Automated Installer?

    The OpenSolaris Auto Installer uses DHCP to allow administrators to perform hands-off installations. The Auto Installer client (machine to be installed) receives its IP address, subnet mask, router, DNS server and boot image all though DHCP. The installadm(1M) tool which one uses to configure an Auto Installer server provides the commands for a Solaris DHCP server but below are the steps for an ISC DHCP server as is common on Linux and even Solaris shops which are simply more comfortable with ISC's DHCP implementation.

    Where to get ISC DHCP for Solaris?

    Software from ISC is usually very stable and well maintained to be easy to compile. However, Solaris seems to have changed a bit from the expectations of ISC DHCP. In my testing, on build 132, I found that ISC DHCP 4.0.0 from Sun Freeware, the ISC DHCP 4.1.0 from pkg.opensolaris.org/contrib and the ISC DHCP 4.1.1 off ISC's download page would all not respond to DHCPDISCOVER's on the wire (but it would report a DHCPOFFER in the logs still just to confuse things). I suspect a compilation issue I saw while compiling 4.1.1 (but I have no actual knowledge why responses are not getting on the wire):

    ld: warning: symbol `MD5_version' has differing sizes:
            (file ../dst/libdst.a(md5_dgst.o) value=0x4; file /lib/libcrypto.so valu
    e=0x76);
            ../dst/libdst.a(md5_dgst.o) definition taken
    

    However, ISC's 4.2.0a1 worked flawlessly! You can get their 4.2.0 alpha from their download page and easily compile it with pfexec pkg install SUNWhea SUNWgcc; ./configure; make; make install.

    Great, I have ISC DHCP compiled and installed, but how do I Configure this thing?

    Normally the issue is not installation and compilation but configuration. For the OpenSolaris Auto Installer there are a number of things to think about:

    • IP addresses
    • Subnet mask
    • Router address
    • DNS server and search domain
    • Boot server
    • Boot file
    • The Auto Installer client's architecture

    Networking primitives (IP address, subnet, router, DNS)

    Without some vital information, even fully functional networks seem useless. The use of an IP address, the subnet mask for the network and a router to get off the network all fit this bill. For most intents and purposes DNS is also in this same boat -- though some administrators may not find DNS as necessary. To get ISC DHCP to serve these pieces of information require certain directives.

    A basic network

    In one's dhcpd.conf a basic network is very easy to define, it looks like the following, here defining for the 192.168.0.0/24 network to serve out IP addresses between 2-100 and with a router of 192.168.0.1:

    subnet 192.168.0.0 netmask 255.255.255.0 {
      range 192.168.0.2 192.168.0.100;
      option routers 192.168.0.1;
    }
    

    DNS

    To add DNS information to one's dhcpd.conf is similarly easy. If one wants each subnet served to get different DNS info one may put the lines in the subnet block or else the directives can go at the beginning of the file and apply to all subnets served:
    option domain-name "example.com";
    option domain-name-servers 192.168.0.1;
    

    Boot server and boot file

    Now, for the Auto Installer specific pieces: to boot a machine, one needs a machine to download a boot file from and the name of such a boot file. These pieces of information will be given by the installadm create-service [...] command when setting up a Auto Installer service. For example:

    Boot server IP (BootSrvA) : 192.168.0.1
    Boot file      (BootFile) : install_test_ai_x86
    GRUB Menu      (GrubMenu) : menu.lst.install_test_ai_x86
    
    The boot server in ISC DHCP terms is the next-sever directive and the boot file is the filename directive. To make it simple, just add these to your subnet group:
    subnet 192.168.0.0 netmask 255.255.255.0 {
      range 192.168.0.2 192.168.0.100;
      option routers 192.168.0.1;
      filename "install_test_ai_x86";
      next-server 192.168.0.1;
    }
    

    If you're using a SPARC Auto Installer service, you can use the same directives for the BootFile object. What about the GrubMenu entry you ask? Well, that's unnecessary and will be removed (see bug 7481) so do not worry about it; similarly, SPARC does not need a next-server (BootSvrA) directive.

    What if you have both SPARC and X86 architectures?

    If you have both SPARC and X86 machines in your Auto Installer environment then there are are some simple ways to define classes to provide your SPARC machines their correct boot-file and X86 machines their boot-file and boot-server. This allows one to have a default service for each architecture on the network.

    A PXE boot class?

    If you want a specific way to separate out your SPARC and X86 clients, then one uses ISC DHCP's class directive and applies all boot specific information there. To create a class for X86 hardware booting, then you can use the following class definition:

    class "PXEBoot" {
      option dhcp-class-identifier "PXEClient";
      filename "install_test_ai_x86";
      next-server 192.168.0.1;
    }
    

    SPARC class

    To define a class to match SPARC clients, one uses ISC DHCP's class directive and applies all boot specific information there. Note, you do not need a next-server directive for SPARC:

    class "SPARC" {
      match if ( substring (option vendor-class-identifier, 0, 5) = "SUNW." ) and not
               ( option vendor-class-identifier = "SUNW.i86pc" );
      filename "http://192.168.0.1:5555/cgi-bin/wanboot-cgi";
    }
    

    Now, SPARC clients will request a lease and get SPARC specific information, while X86 clients will request and get information specific to an X86.

    The entire dhcpd.conf looks like:

    # option definitions common to all supported networks...
    option domain-name "example.com";
    option domain-name-servers 192.168.0.1;
    
    default-lease-time 600;
    max-lease-time 7200;
    
    # If this DHCP server is the official DHCP server for the local
    # network, the authoritative directive should be uncommented.
    authoritative;
    
    # Use this to send dhcp log messages to a different log file (you also
    # have to hack syslog.conf to complete the redirection).
    log-facility local7;
    
    # This is an easy way to discriminate on SPARC clients
    class "SPARC" {
      match if ( substring (option vendor-class-identifier, 0, 5) = "SUNW." ) and not
               ( option vendor-class-identifier = "SUNW.i86pc" );
      filename "http://192.168.0.1:5555/cgi-bin/wanboot-cgi";
    }
    
    # This is a class to discriminate on PXE booting X86 clients
    class "PXEBoot" {
      option dhcp-class-identifier "PXEClient";
      filename "install_test_ai_x86";
      next-server 192.168.0.1;
    }
            
    # This is a very basic subnet declaration
    subnet 192.168.0.0 netmask 255.255.255.0 {
      range 192.168.0.2 192.168.0.100;
      option routers 192.168.0.1;
    }
    

    ☞ Public Service


    July 26, 2010

    ☞ What Will You Do?

    • Worthwhile reading for a Monday from Clayton Christensen.
    • Bryan Cantrill says his farewells but rehosts his blog promisingly at dtrace.org and makes this great observation: “One of Sun’s greatest strengths was that we technologists were never discouraged from interacting directly and candidly with our customers and users, and many of our most important innovations came from these relationships.”

    Alan

    I finally managed to get the SRSS 4.2 server software running on my setup. Note that SRSS 4.2 is the server version that is included with the Sun Ray 5.0 download.

    First up I needed to install the software. It comes packaged as a zip file (srss_4.2_solaris.zip), extracting into srss_4.2.

    It looks like if you want the web configuration interface you need to install tomcat from this distribution too. You can find this in srss_4.2/Supplemental/Apache_Tomcat/apache-tomcat-5.5.20.tar.gz. I installed it like this.

    $ gzcat apache-tomcat-5.5.20.tar.gz | (cd /opt ; pfexec tar xf -)
    $ cd /opt
    $ ln -s apache-tomcat-5.5.20 apache-tomcat
    

    You can then do the srss install, answering the questions asked (not included here as I forgot to record it). The symlink above means that you can sensibly accept the default location for Tomcat.

    $ cd srss_4.2
    $ pfexec ./utinstall
    

    This is unfortunately not enough to get us going. When I went to run /opt/SUNWut/sbin/utadm -L it immediately complained about not being able to find the DHCP server stuff. I have no interest whatsoever in installing the DHCP stuff. After some poking around I found that all I really needed to do was to edit /etc/opt/SUNWut/auth.props and make the setting

    allowLANConnections = true
    

    But that’s not all, …

    I was still getting the 26B on the Sun Ray DTU on attempting to utswitch to the new machine.

    This error code apparently means “Waiting for X”. Fortunately there was a wonderful wiki entry on getting SRSS 4.1 working with OpenSolaris 2009.06. I’ll re-document it as there were some things that I did not need to do (as they had been addressed in SRSS 4.2) and others which I thought less than clear.

    You need the motif libraries. Sorry, no way around it, it’s linked against them. Fortunately they are available in the opensolaris.org repository.

    $ pfexec pkg install SUNWmfrun SUNWtltk SUNWdtbas
    

    I saw the following error in /var/log/gdb/:11.log (11 being the session number it allocated me).

    ld.so.1: Xnewt: fatal: libXfont.so.1: open failed: No such file or directory
    

    Ahhh this was because some libraries moved. To fix it you need to do the following

    # cd /usr/lib
    # ln -s xorg/libXfont.so .
    # ln -s xorg/libXfont.so.1 .
    # ln -s xorg/libfontenc.so .
    # ln -s xorg/libfontenc.so.1 .
    # cd sparcv9
    # ln -s ../xorg/sparcv9/libXfont.so .
    # ln -s ../xorg/sparcv9/libXfont.so.1 .
    # ln -s ../xorg/sparcv9/libfontenc.so .
    # ln -s ../xorg/sparcv9/libfontenc.so.1 .
    

    I also created /etc/opt/SUNWut/X11/fontpath with the following contents

    /usr/share/fonts/X11/100dpi
    /usr/share/fonts/X11/100dpi-ISO8859-1
    /usr/share/fonts/X11/100dpi-ISO8859-15
    /usr/share/fonts/X11/75dpi
    /usr/share/fonts/X11/75dpi-ISO8859-1
    /usr/share/fonts/X11/75dpi-ISO8859-15
    /usr/share/fonts/X11/encodings
    /usr/share/fonts/X11/isas
    /usr/share/fonts/X11/misc
    /usr/share/fonts/X11/misc-ISO8859-1
    /usr/share/fonts/X11/misc-ISO8859-15
    /usr/share/fonts/X11/Type1
    

    Finally I had to edit /opt/SUNWut/lib/xmgr/gdm/remove-dpy since gdmdynamic arguments appear to have changed. Replacing

    gdmglue="; gdmdynamic -b -d "'$UT_DPY'
    

    with

    gdmglue="; gdmdynamic -d "'$UT_DPY'
    

    Just to make sure everything had “taken”, I rebooted the box.

    It looks like this did the trick. I am able to /opt/SUNWut/bin/utswitch -h sb2000-b and login correctly and actually get a usable desktop.

    As I’ve done a bit of work on vesvi since I synced the filesystems onto sb2000-b I’ll need to sync those over again before I start trying to use this as my desktop for a bit before physically moving the disk over to vesvi (and all of the associated network reconfiguration and probably creating another certificate for imapd).

    I can see the light at the end of the tunnel and it is not an oncoming train.

    It should go without saying (but I’ll say it anyway). Getting Sun Ray running like this OpenSolaris completely voids any warranty and support. Don’t do it if you don’t know what you are doing.


    Revelations: 145

    Phoronix published a sensationalist article last week claiming that my regular e-mail updates of our biweekly builds somehow signified some out of the ordinary newsworthy event, without bothering to do even the most basic of fact checking. While I pointed this out in their forums within hours of publication, I'm still seeing it cited by other web magazines that don't bother to fact check, as well as in various e-mails and blogs, so am publishing a more complete explanation here of why it really is a non-event.

    The article claimed:

    As the first email of its kind in months, Alan Coopersmith who is a known X.Org contributor and longtime Sun Microsystems employee now working for Oracle, has written a new email entitled "IPS distro-import changes needed for X packages for nv_145." Alan immediately began this public email by saying, "Just when you thought you'd never see another one of these biweekly mails..."

    Sadly, all they needed to do to disprove the claim that it was the “first of its kind in months” was simply follow the links from the e-mail archive page they linked to, to see that I had sent a similar message two weeks earlier for the previous biweekly build nv_144. In fact, if they checked the archives for previous months, they would have found that, except for missing build 143 (a mistake on my part), I've sent these approximately every two weeks for every biweekly build for a very long time.

    Perhaps I'd confused the article's author with the offhand comment he seems to have misinterpreted, but explaining that requires a bit of background explaining what these e-mails are and why I send them in the first place.

    As many OpenSolaris users know, over the course of the last couple of years, we've been transitioning from the old SVR4 package system used in Solaris 2.0 through Solaris 10 to the new Image Packaging System (also sometimes known as “IPS” or “pkg(5)”) being developed by a team of Solaris engineers and community members. Initially, we maintained two parallel distros, Solaris Express: Community Edition (SXCE), which was built using the SVR4 packages and the old Solaris installer that worked with them, and OpenSolaris (originally codenamed “Project Indiana”), which was built using the IPS packages and the new Caiman install software designed to work with them. The teams providing the package contents continued to deliver SVR4 packages, and the OpenSolaris distro team converted those to IPS.

    One of the goals of the OpenSolaris distro was to provide a set of ISO install images and package repository that was completely, freely redistributable, so that it could be easily mirrored, copied, and downloaded without having to deal with the various encumbrances required by some of the third-party licenses in the traditional Solaris and SXCE packages. Unfortunately, at that time, we had not yet finished separating the encumbered code from the open source code in our X packages so that they could be included, since when Sun made its proprietary fork of X11R6 in the early 90's, the engineers never figured we'd be open sourcing Solaris a decade later and need to easily separate out the encumbered bits they were merging into the main code base.

    The initial Developer Preview releases of the OpenSolaris distro thus included a set of X packages that Moinak Ghosh built from the Fully Open X (FOX) project work we'd done to rebuild our source trees from the ground up using the open source code from the X11R7 modular releases in order to ensure everything was either open source or cleanly separated out. Over the first few months, we migrated from those to the packages our team delivered to the OS as we integrated the FOX project work into our main source tree. Because these changes weren't always obvious to the external observer, I started sending notes for each biweekly build to let the team maintaining the SVR4 to IPS conversion tables know which parts of our packages they could now include, as well as any other changes they needed to know about, such as version number changes. (Our SVR4 X packages all used the same version number, a holdover from the monolithic X source days, but we've migrated to using the upstream version numbers as much as possible in the new X IPS packages.) I did this not only to try to help the distro building team, but also to help myself keep on top of the changes they made in converting our packages, so that we could understand issues users hit and know how to help them, and to learn better what we'd need to do when it came time for us to start building the IPS packages ourselves.

    Last year, Sun announced that the time had finally come to start converting the builds to generate IPS packages directly, taking the next step in transitioning off SVR4 and ending the production of the SXCE distro, since the SVR4 packages used to build it would no longer be made. The ON consolidation went first, converting the packages containing the kernel, drivers and core utilities in build snv_136. The Install consolidation went next, in build snv_143, and X followed in build snv_144.

    So, when I wrote “Just when you thought you'd never see another one of these biweekly mails...”, I simply meant that after build 144, all the X packages are already delivered in IPS format, so there are no SVR4 to IPS conversion files to update for them any more, so I won't need to send those - except in cases like we had in 145, where I relocated one of the files that was listed as a dependency in one of the other packages still being converted by the distro builders, so they needed to update the dependency statement for it to list the new path to the file.

    Despite what Phoronix seems to have assumed, I was not in any way referring to the limbo state the OpenSolaris distro is currently in (and unfortunately, as much as I'd like to explain that, I can't), nor stating anything about build 145 that is fundamentally different than the previous builds. It should come as no surprise to anyone that while build 134 was the last build to be publicly released, we have continued work on the Nevada builds after that - after all as we've said since 2005, Nevada is the code name for the development branch in which we're working towards the next full release of Solaris (i.e. not another Solaris 10 update release, but the one we may someday call Solaris 11) - while that's been released under various forms, as the OpenSolaris open source code, or via the binary releases of the original Solaris Express, Solaris Express: Community Edition, Solaris Express: Developer Edition, or the OpenSolaris 2008.05, 2008.11, and 2009.06 distros, we've always kept driving towards the same goal, with biweekly builds assembled to test the current progress.

    My mail is hardly the only externally visible sign of this - you can see changelogs for the major consolidations (ON, SFW, X, and Desktop/JDS) for build 144 (the last fully finished build, as 145 is just finishing pre-integration testing now, with a delivery deadline of Monday for packages to be included, and the distro build process starting after that. Of course, the sources are also available, and there's plenty of activity on the various commit notification and discussion mailing lists showing that we're continuing to work away on Nevada.

    So unfortunately, Phoronix succeeded in making a mountain out of a molehill, confusing their readers and fellow webzine authors, but likely meeting their goal of driving more traffic to their site to generate page views for their ad revenue as people passed the link around twitter, IRC & email or linked to it from their blogs & articles. As others have pointed out, checking the facts or contacting the developers to find out the story is less juicy than it seems doesn't play well with that business model (and that's not just true for Phoronix - look at any number of the columnists for other web-based "trade publications" that generate traffic via controversial posts, and the more outraged the community gets over them and angrily passes them around to denounce, the better their numbers are - you can just imagine how many of their articles are designed to bait Groklaw or Slashdot readers).

    Good-bye, Sun

    In Februrary 1996, I came out to Sun Microsystems to interview for a job knowing only two things: that I wanted to do operating systems kernel development -- and that I didn't particularly want to work for Sun. I was right on the first count, but knew I was wrong on the second just moments into my first conversation with Jeff. He was emphatic that I should join him in forging the future, sharing both my enthusiasm for what was possible and my disdain for the broken, busted and boogered-up. Fourteen years later, I don't for a moment regret my decision to join Jeff and Sun: we fostered an environment where the OS was viewed not as a regrettable drag on progress, but rather as a nexus of innovation -- incubating technologies that today make a real difference in people's lives.

    In 2006, itching to try something new, Mike and I talked the company into taking the risk of allowing several of us to start Fishworks. That Sun supported our endeavor so enthusiastically was the company at its finest: empowering engineers to tackle hard problems, and inspiring them to bring innovative solutions to market. And with the budding success of the 7000 Series, I would like to believe that we made good on the company's faith in us -- and more generally on its belief in innovation as differentiator.

    Now the time has come for me to venture again into something new -- but this time it is to be beyond the company's walls. This is obviously with mixed emotion; while I am excited about the future, it is very difficult for me personally to leave a company in which I have had such close relationships with so many. One of Sun's greatest strengths was that we technologists were never discouraged from interacting directly and candidly with our customers and users, and many of our most important innovations came from these relationships. This symbiosis was critically important at several junctures of my own career, and I owe many of you a profound debt of gratitude -- both for your counsel over the years, and for your willingness to bet your own business and livelihood on the technologies that I helped develop. You, like us, are innovators who love nothing more than great technology, and your steadfast faith in us means more to me than I can express; thank you.

    As for my virtual address, it too is changing. This post will be my last at blogs.sun.com; in the future, you can find my blog at its new (permanent) home: http://dtrace.org/blogs/bmc (where comments on this entry will be open). As for e-mail, you can find me at the first letter of my first name concatenated with my last name at acm.org.

    Thank you again for everything; take care -- and stay in touch!

    July 25, 2010

    Finding ON Flag Day Announcements

    ON flag day emails are archived automatically to an internal server and then mirrored externally. Unfortunately, folks sometimes give the internal URL when referring to a particular flag day message, which doesn't do you much good if you don't have access to the Oracle/Sun internal network.

    While it would be preferable to have the external URL to start with, the mapping from the internal URL to external URL is pretty simple. Replace "onnv.sfbay.sun.com" with "static.opensolaris.org/on". So, for example,

    http://onnv.sfbay.sun.com/flagdays/pages/20100303235050.html

    becomes

    http://static.opensolaris.org/on/flagdays/pages/20100303235050.html

    July 23, 2010

    ☞ Early Standardisation

    • In the debate about open standards at the Cloud Summit on Tuesday, one speaker argued powerfully that standards follow innovation rather than delivering it, an observation that those of us with involvement in the world of standards have long understood. Early standardisation indicates there are vendors attempting to lock a market, and that’s exactly what lay behind the formation of WS-I in 2002. Now that web services are generally marginalised in favour of REST, WS-I has packed up shop and archived itself at OASIS. Good move.

    ☞ Logical Core


    July 22, 2010

    Fun With vpnc

    I recently got a new laptop at work and I decided to put OpenSolaris on it. This meant I had to setup vpnc in order to access the server networks and wireless here. I installed my vpnc package, copied the profile from my Ubuntu workstation, and started it up. It connected, but no packets flowed. I didn’t have time to investigate, so I decided to work on it some more at home.

    The strange thing is that it connected from home with the very same profile and everything worked fine. I immediately suspected something was wrong with the routing tables, like maybe some of the routes installed by vpnc-script were conflicting with the routes necessary to talk to the VPN concentrator. I endlessly compared the routing tables between work and home and my working Ubuntu workstation, removing routes, adding routes, and manually constructing the routing table until I was positive it could not be that.

    Everything I pinged worked. I could ping the concentrator. I could ping the gateway. I could ping the tunnel device. I could ping the physical interface—or so I thought.

    As I was preparing to write a message to the vpnc-devel mailing list requesting help, I did some pings to post the output in the email. I ran

    $ ping <concentrator ip>
    <concentrator ip> is alive
    

    which looked good, but I wanted the full ping output, so I ran

    $ ping -s <concentrator ip>
    PING <concentrator ip>: 56 data bytes
    ^C
    ----<concentrator ip> PING Statistics----
    4 packets transmitted, 1 packets received, 75% packet loss
    round-trip (ms)  min/avg/max/stddev = 9223372036854776.000/0.000/0.000/-NaN
    

    For some reason, only the first ping was getting through. The rest were getting hung up somewhere. The really strange thing was that I saw the same behavior on the local physical interface:

    $ ifconfig bge0
    bge0: flags=1004843 mtu 1500 index 3
            inet 161.253.143.151 netmask ffffff00 broadcast 161.253.143.255
    $ ping -s 161.253.143.151
    PING 161.253.143.151: 56 data bytes
    ^C
    ----161.253.143.151 PING Statistics----
    5 packets transmitted, 1 packets received, 80% packet loss
    round-trip (ms)  min/avg/max/stddev = 9223372036854776.000/0.000/0.000/-NaN
    

    I have never seen a situation where you couldn’t even ping a local physical interface! I checked and double checked that IPFilter wasn’t running. Finally I started a packet capture of the physical interface to see what was happening to my pings:

    # snoop -d bge0 icmp
    Using device bge0 (promiscuous mode)
    161.253.143.151 -> <concentrator ip> ICMP Destination unreachable (Bad protocol 50)
    161.253.143.151 -> <concentrator ip> ICMP Destination unreachable (Bad protocol 50)
    161.253.143.151 -> <concentrator ip> ICMP Destination unreachable (Bad protocol 50)
    ^C
    

    That’s when by chance I saw messages being sent to the VPN concentrator saying “bad protocol 50.” IP protocol 50 represents “ESP”, commonly used for IPsec. Apparently Solaris eats these packets. Haven’t figured out why.

    I remembered seeing something in the vpnc manpage about ESP packets:

    --natt-mode <natt/none/force-natt/cisco-udp>

          Which NAT-Traversal Method to use:
          o    natt -- NAT-T as defined in RFC3947
          o    none -- disable use of any NAT-T method
          o    force-natt -- always use NAT-T encapsulation  even
               without presence of a NAT device (useful if the OS
               captures all ESP traffic)
          o    cisco-udp -- Cisco proprietary UDP  encapsulation,
               commonly over Port 10000

    I enabled force-natt mode, which encapsulates the ESP packet in a UDP packet, normally to get past NAT, and it started working! In retrospect, I should have been able to figure that out much easier. First, it pretty much says it on the vpnc homepage: “Solaris (7 works, 9 only with –natt-mode forced).” I didn’t even notice that. Second, I should have realized that I was behind a NAT at home and not at work, so they would be using a different NAT-traversal mode by default. Oh well, it was a good diagnostic exercise, hence the post to share the experience.

    In other vpnc related news, I’ve ported Kazuyoshi’s patch to the open_tun and solaris_close_tun functions of OpenVPN to the tun_open and tun_close functions of vpnc. His sets up the tunnel interface a little bit differently and adds TAP support. It solves the random problems vpnc had with bringing up the tunnel interface such as:

    # ifconfig tun0
    tun0: flags=10010008d0<POINTOPOINT,RUNNING,NOARP,MULTICAST,IPv4,FIXEDMTU> mtu 1412 index 8
            inet 128.164.xxx.yy --> 128.164.xxx.yy netmask ffffffff
            ether f:ea:1:ff:ff:ff
    # ifconfig tun0 up
    ifconfig: setifflags: SIOCSLIFFLAGS: tun0: no such interface
    # dmesg | grep tun0
    Jul 23 14:56:05 swan ip: [ID 728316 kern.error] tun0: DL_BIND_REQ failed: DL_OUTSTATE
    

    The changes are in the latest vpnc package available from my package repository.

    Diversion: Ode to Lego Technic

    Nova, my first daughter, is now 6 and Glenn, my first son, is now 5. As a GeekDad I ensure to bathe them in geeky goodness. I've been thankful that Glenn is obsessed with Lego. The kool thing about it is that of course, I get to help him, so its just a great time. Here was last nights project:

    Teaching him has gotten me thinking back to my own youth. I had a box of Lego's but not a lot of sets. The one that I did get was in 1988, when my parents got me perhaps my favorite (but forgotten until recently) toy of youth: the Lego Technic 8865 "Test Car".

    That set was amazing. I proudly displayed it on my shelf in my room, both because of my pride in building it as well as just how outright kool it is.

    Since that time Technic has grown up as much as I have. Take a look at the Technic Lego 8421 Mobile Crane:

    So tempted to buy that. I already have the Ferrari F1 set, which Tamarah bought for my birthday several years ago.

    But most fun of all... this week Glenn is in a one-week Lego Pre-Engineering class. For 3 hours a day they geek out and build all manner of fun stuff.

    One thing I'll throw out there for Dad's... Lego has an Education dept: Lego Education. Of particular interest to Tamarah and I, is that they have a complete Homeschool Curriculum and various kits, including robotics kits, for education. A really amazing resource for parents.

    Devops in Practice: How They Do It

    Damon Edwards (DTO Solutions) & John Willis (Opscode) are the two guys really pumping out the "good news" of devops. They started a new podcast, Devops Cafe several weeks ago. Already on episode 8, having featured guests such as John Allspaw, R.I. Pienaar, Andrew Shafer, and more. Highly recommended.

    Whats interesting is that John & Damon really aware of an outcry from the community, that is: "How do all these devops shops do it!!" We want to emulate them, know what tools they have, how they use them, what works, what doesn't, etc. So to facilitate just that, they started a videocast sub-series called: Open Mic.

    Open Mic 1: DevOps Metrics and Dashboards at Shopzilla from dev2ops.org on Vimeo.

    In the first episode, they take us into Shopzilla, where Juan Paul Ramirez shows us their tools, metrics, and talks extensively about how they got to where they are. Excellent content!

    If you haven't already seen, perhaps the most popular talk this year at Velocity, "A Day in the Life of Facebook", in which the Facebook Ops team introduces us to their tools and organization.

    Whats really great here is that we're not share deeper information about how we're doing things, such that we can be a community of organizations. In the past, only a handful would really share and they were always far removed from useful pratice. I really hope this trend continues.

    Big thanks to John and Damon for helping fuel that fire!

    What is RAID-Z?

    The mission of ZFS was to simplify storage and to construct an enterprise level of quality from volume components by building smarter software — indeed that notion is at the heart of the 7000 series. An important piece of that puzzle was eliminating the expensive RAID card used in traditional storage and replacing it with high performance, software RAID. To that end, Jeff invented RAID-Z; it's key innovation over other software RAID techniques was to close the "RAID-5 write hole" by using variable width stripes. RAID-Z, however, is definitely not RAID-5 despite that being the most common comparison.

    RAID levels

    Last year I wrote about the need for triple-parity RAID, and in that article I summarized the various RAID levels as enumerated by Gibson, Katz, and Patterson, along with Peter Chen, Edward Lee, and myself:

    • RAID-0 Data is striped across devices for maximal write performance. It is an outlier among the other RAID levels as it provides no actual data protection.
    • RAID-1 Disks are organized into mirrored pairs and data is duplicated on both halves of the mirror. This is typically the highest-performing RAID level, but at the expense of lower usable capacity.
    • RAID-2 Data is protected by memory-style ECC (error correcting codes). The number of parity disks required is proportional to the log of the number of data disks.
    • RAID-3 Protection is provided against the failure of any disk in a group of N+1 by carving up blocks and spreading them across the disks — bitwise parity. Parity resides on a single disk.
    • RAID-4 A group of N+1 disks is maintained such that the loss of any one disk would not result in data loss. A single disks is designated as the dedicated parity disk. Not all disks participate in reads (the dedicated parity disk is not read except in the case of a failure). Typically parity is computed simply as the bitwise XOR of the other blocks in the row.
    • RAID-5 N+1 redundancy as with RAID-4, but with distributed parity so that all disks participate equally in reads.
    • RAID-6 This is like RAID-5, but employs two parity blocks, P and Q, for each logical row of N+2 disk blocks.
    • RAID-7 Generalized M+N RAID with M data disks protected by N parity disks (without specifications regarding layout, parity distribution, etc).

    RAID-Z: RAID-5 or RAID-3?

    Initially, ZFS supported just one parity disk (raidz1), and later added two (raidz2) and then three (raidz3) parity disks. But raidz1 is not RAID-5, and raidz2 is not RAID-6. RAID-Z avoids the RAID-5 write hole by distributing logical blocks among disks whereas RAID-5 aggregates unrelated blocks into fixed-width stripes protected by a parity block. This actually means that RAID-Z is far more similar to RAID-3 where blocks are carved up and distributed among the disks; whereas RAID-5 puts a single block on a single disk, RAID-Z and RAID-3 must access all disks to read a single block thus reducing the effective IOPS.

    RAID-Z takes a significant step forward by enabling software RAID, but at the cost of backtracking on the evolutionary hierarchy of RAID. Now with advances like flash pools and the Hybrid Storage Pool, the IOPS from a single disk may be of less importance. But a RAID variant that shuns specialized hardware like RAID-Z and yet is economical with disk IOPS like RAID-5 would be a significant advancement for ZFS.