Blocking blog spam with Lighttpd.

I've switched to "Lighty" (lighttpd) for my website, and a good part of the reason is because I like the configuration file syntax. However, I didn't quite realize at the time just how much spam was being blocked by my "no referer means no POST" rule in Apache.

Fortunately, I've figured out how to do the same in Lighty, thanks to nested conditionals, which are new in Lighttpd 1.4 - included in Sun's Webstack:

$HTTP["host"] == "blugh.bofh.ca" {

        $HTTP["request-method"] == "POST" {

                $HTTP["referer"] == "" {

                        url.access-deny =  ( "" )

                }

        }        

}

IF the host is "blugh.bofh.ca" AND IF "request-method" is "POST" then you'd better have a referer. That'll do the job for now... I'm sure some enterprising spammer might adapt his code to supply a referer of any sort, but then he'll just get stopped by my own moderation... they can feel free to waste their time for no benefit. I encourage it!

Posted at 03:18PM Jun 17, 2009 by Brandon Hume in Solaris  |  Comments[0]

Solaris 10 U7!

Den upgraded, now with ZFS boot/root! This should hopefully make my life easier... I've very much gotten used to Live Upgrade (aka: beadm) on my OpenSolaris box, and Den's UFS layout wasn't built for it. But partitioning doesn't really EXIST under ZFS. So... win!

Posted at 04:08PM May 28, 2009 by Brandon Hume in Solaris  |  Comments[0]

Jumpstarting a SPARC over the WAN - Part 2, the Jump.

In the last segment, we got a remote machine booting off the WAN. But altogether that isn't terribly useful. The goal here is to get that machine installed off the WAN... probably, so that we can get other machines installed off that first box.

The main problem we're working around is that the Solaris Jumpstart scripts will, at least on the SPARC platform, assume that your install media is located on the same type of media as your boot filesystem. Booting off the WAN, using wanboot, is meant to work around the lack of any control of the network your machine is located on. Installing this way is meant to work around the assumption that boot media == install media. In this case, the idea that since the boot media is HTTP, that your install media is located via HTTP as well (a FLAR).

To get our Jumpstart Jumpstarting, the very next step we'll need to do is configure our system_conf file, as was specified in the wanboot.conf configuration file. In my example earlier, I used sys-conf.s10-sparc as the name of this file. So, the file we'll need to edit is /etc/netboot/sys-conf.s10-sparc, which is a plain text file. In mine, I placed the following two lines:

SsysidCF=http://192.168.1.10/wanboot/client-sysidcfg

SjumpsCF=http://192.168.1.10/wanboot/js-rules

Note how similar the keywords are to how you'd often configure your DHCP server for Jumpstart?

"client-sysidcfg" should be a directory. If you've Jumpstarted Solaris machines on your LAN, then you know this trick... Solaris always looks for a file named "sysidcfg", so you create a separate directory for your differing client machines. So, in keeping with my earlier examples, this directory would be /var/apache2/htdocs/wanboot/client-sysidcfg on my webserver.

In this directory, place your sysidcfg file. Keep in mind that Solaris processes this file just like it would if it was fetching it via NFS... if you make any mistakes in it, the Jumpstart won't be non-interactive. In this case that's probably fine. You'll likely be babysitting a weird install like this.

Now you'll need your Jumpstart files, particularly your "rules.ok" and install profile. These files are made in exactly the same way as you would for a normal, NFS-based install. You'll need to edit your "rules" file, create a jumpstart profile, and then process them using the "check" script to generate rules.ok. However, there is one more step.

The Jumpstart "begin" script isn't something used often... most of us don't need it. So we leave the field in rules - the field between the rule conditions and the Jumpstart profile - as the plain dash to indicate we don't need it. However, in this case we do.

After Solaris WAN-boots, it'll be confused about the install media. If you were installing via a local server, or with a disc, the install media would be mounted on /cdrom (regardless of the actual media type). This won't happen in this case, so Jumpstart will drive on blindly and eventually explode, complaining that it can't find the install content. However, this error will happen *after* the begin script has run.

So, to fool Jumpstart, we slip in and mount the install media ourselves. The script itself is trivial. Assuming you've shared the install media via NFS, with anon=0 and ro to the world or (better) just to the client machine, you mount it just as you would via the command-line. Assuming your NFS server has the IP address of 192.168.2.10, and your Jumpstart base directory (the directory with the .volume.inf* files in it) is mounted at /js/sol10, your begin script would look like:

#!/bin/sh

echo 'Mounting WAN install media'

mount -F nfs -o ro 192.168.2.10:/js/sol10 /cdrom

Nothing to it. I should note that the install media can be located *anywhere*. It might be on the same machine as you have serving the wanboot media. It could be a general campus Jumpstart server. It doesn't matter, so long as the machine you're installing is allowed to mount it. The only caveat I would give is that I'd avoid using a Linux machine. I know Linux does a poor job booting SPARC clients with an NFS root... those problems might extend into installing as well, although I haven't tested. Mounting in this way actually gives more wiggle room, as you can provide mount options to NFS, without letting Solaris figure it all out... so Linux might actually work in this case.

I saved this begin script as "wan_start.sh". So, my rules file actually looked like:

hostname client     wan_start.sh    client.js      -

A quick run of "check", and I have my rules.ok file.

Create the directory /var/apache2/htdocs/wanboot/js-rules. Dump rules, rules.ok, wan_start.sh, and client.js into it. Now, your client machine can fetch each of them as it needs.

At this point, you should be able to reset your machine, do a "boot net - install" (network-boot-arguments is persistent across reboots), and Jumpstart should roll, doing a complete install.

The Jumpstart will fail at the end, because some post-setup scripts can't be run... in my testing, these failures were not fatal to the Jumpstart. I believe it was just various crap that Solaris sets up to run the installers for extra discs and the like, not particularly needed during a Jumpstart. Still, this is a hack, so problems might arise later. As I've said, this is mostly just to get a foothold when installing a number of remote machines.

I've left out a number of little tweaks to this process. For instance, wanboot-cgi can be made far more secure... it has the capability of hashes that the client can present to the wanboot server to verify its identity and the like. That would be part of a more formal wanboot infrastructure, and hopefully any serious setup like that would have the capability of generating a proper FLAR, making this whole WAN-NFS thing a bit of quaint nonsense.

I encourage people to read the actual scripts that deal with the guts of the Jumpstart process. The majority can be found right on your install media, under Solaris_10/Tools/Boot/sbin, named install-*. Keep in mind that you can activate debugging of many of the scripts at the ok prompt, using "boot net - install install_debug". It can be interesting to see some of the extra noise generated, as you follow the Jumpstart through some (not all) of the Jumpstart process.

Posted at 09:58PM Sep 02, 2008 by Brandon Hume in Solaris  |  Comments[0]

Jumpstarting a SPARC over the WAN

A post was recently made to comp.unix.solaris, where someone asked if he could Jumpstart several T2000s that he had hosted in a remote facility. He had no administrative access to the remote network, and apparently the other admins (being Windows monkeys) could not be relied upon to set up DHCP and the traditional TFTP/NFS infrastructure.

To make things worse, he had ONLY the remote T2000s. He had no matching-architecture local machines to build a FLAR on.

A number of responders told him (in essence) that he was screwed, that he'd have to take a laptop or something into the colo facility and Jumpstart at least one.

But I read that and thought, "That can't be right." Certainly, taking a machine in (or, more likely, just an install DVD) would be the EASIEST solution. But was it the ONLY solution?

Most SPARC administrators are familiar, at least passingly, with Jumpstart. You set up your infrastructure - namely, a DHCP or RARP server, and an NFS server - hop on the PROM, and and one "boot net - install" and interminable wait later, you've got a fully installed and functioning machine. The feature is also available for x86, since a huge majority of machines come with PXE capability.

However, lesser-known is that modern OpenBOOT PROMs are actually capable of being told their networking information, and not relying completely on DHCP or RARP. This is done with the network-boot-arguments OpenBOOT variable. This is documented in the eeprom(1M) manpage. With this variable, you can supply OpenBOOT with pretty much everything it needs to start up a working, routable IP connection, which can be used to fetch the next-stage bootloader, wanboot, over HTTP.

Traditionally, Jumpstarting in this manner causes the system to try and build via an HTTP-fetched FLAR. Certainly, if you can build a FLAR even vaguely matching the box you're trying to build, this would be the easiest solution. Even if it didn't entirely match your needs, it's enough of a starting point to get a useable Jumpstart platform for the other boxes in the rack.

But, in this person's case, a FLAR wasn't feasible. He was able to export the install media via NFS, and NFS works fine, if slowly, over a WAN. But how to get the installer started in the first place?

(As an aside, Sun - or the community - should really try to figure out a means of building a FLAR for one architecture on another. But, then, with the new packaging/install system coming, perhaps there's hope that FLAR and Jumpstart itself might become obsolete.)

The scripts which start the Jumpstart process make a number of assumptions... some reasonable, some less-so. One of the most annoying is that it assumes your boot media is the same as your install media. It's more annoying, for the SPARC administrator, because Jumpstarting an x86 box does not make this particular assumption... you can supply "-B install_media=" on the GRUB command-line.

There's hope for the future, as supposedly SPARC will be moving to a GRUB, or GRUB-alike booting architecture for the sake of ZFS boot. But how does that help our poor admin, who doesn't want to drive X kilometres to spend hours in a cold colo facility, surrounded by confused, barbaric Windows admins?

First step, in the next entry, is setting up the wanboot process.

Posted at 09:57PM Sep 02, 2008 by Brandon Hume in Solaris  |  Comments[0]

Sexy new T5140s

So pretty...

[Read More]

Posted at 12:00PM Jun 06, 2008 by Brandon Hume in Solaris  |  Comments[0]

SMF, segfaults, and contracts on Solaris 10.

Something that isn't immediately obvious when composing an SMF manifest on Solaris is that the process contract doesn't just include the processes launched by SMF, but also those processes launched by those processes.

Most of the time it doesn't matter. But it can catch you off-guard when you neglect to enter a proper ignore_error propval in the manifest. Then, SMF suddenly cares if any of the sub-processes core while running. Core dumps are obviously problems... they shouldn't be happening if your application is well-behaved. But you don't necessarily want a high-profile service like a webserver being shut down by SMF just because some piddly CGI version of "Hello World" chased a NULL.

It can also be frustrating, because these little sub-processes are also not typically blatantly associated with the main service. SMF will typically log a somewhat-helpful error, "pid exited with signal 11" or some such... and, of course, that's where you need process accounting, to find out what "pid" was, if it wasn't kind enough to leave the core in the obvious place.

ptree -c pid will show the contract ID associated with a tree of processes. You can expand on the nature of the process contract with ctstat -vi contract-id and find out what faults SMF will let ride.

Also, something handy I can't believe I didn't know before: svcs -p FMRI will show the currently-running processes associated with an SMF-managed service. Handy.

Posted at 02:55PM Dec 05, 2007 by Brandon Hume in Solaris  |  Comments[0]

Some pictures of new!Den.

Some pictures of the new Blade 2000 that is Den (and this website).

[Read More]

Posted at 03:29PM Nov 02, 2007 by Brandon Hume in Solaris  |  Comments[0]

Playing with iSCSI on Solaris.

I've been hearing bits and pieces about iSCSI for a while now, but not much of how to use it. So I decided to grab my two toy Solaris machines and give it a shot.

It still seems to be somewhat underdocumented; although there's a docs.sun.com page for iSCSI administration, it mostly centres around the initiator (read: client), not so much from the target (read: server). The iscsitadm man page doesn't even exist on my machines, although I notice that it's present in the OpenSolaris source tree, so it's at least in the pipe for the future.

To set up a very simple shared "LUN", I used my PC running OpenSolaris as the target/server, and my SPARC running Solaris 10 06/06 as the initiator/client.

On my PC, the first thing I did was create a scratch-area to hold the backing store. Easy enough, with ZFS:

PC# zfs create zp01/iscsi-play

The next part is to create the "LUN". This can be a raw disk, but instead I went with a plain file as a backing store, which seems to be the default. Also unstated is that the target is created as a "disk" device... there are other options, including "tape", and "raw".

PC# iscsitadm create target -b /zp01/iscsi-play/test-volume -z 10g test-volume

And then you can see the following:

PC# ls -l

total 551255

-rw-------   1 root     sys      10737418240 Feb 22 16:47 test-volume

If you think it created such a big file awfully fast, you're right. For the moment, test-volume is a sparse file. Keep that in mind the first time you write to the LUN/file. It'll be mind-bogglingly slow as the OS fills out the file.

I can see the target  I've created with the following:

PC# iscsitadm list target -v
Target: test-volume

    iSCSI Name: iqn.1986-03.com.sun:02:241dafd0-6d3b-ef2f-e6c9-8b62f63ece19.test-volume

    Connections: 0

    ACL list:

    TPGT list:

    LUN information:

        LUN: 0

            GUID: 0100000c7694924000002a0045de016a

            VID: SUN

            PID: SOLARIS

            Type: disk

            Size:   10G

            Backing store: /zp01/iscsi-play/test-volume

            Status: offline

Note the "Status: offline" bit? We can fix that next. Also, I need to make note of that long "iSCSI Name" string... I'll need that later.

First, I want some kind of authentication. So I set a CHAP username and password with the following commands:

PC# iscsitadm modify admin -H shumira

PC# iscsitadm modify admin -C

Enter secret:<gimmesomedisk>

Re-enter secret:<gimmesomedisk>

So now I have a CHAP username of 'shumira' and a password/secret of 'gimmesomedisk'.

Next, I needed to create a Target  Port Group Tag (TPGT). I do so with the following:

PC# iscsitadm create tpgt 1

PC# iscsitadm modify tpgt -i <my PC's IP address> 1

This creates a TPGT and associates my IP address with it. I'm not sure if it was strictly necessary in my case -- although if I had multiple interfaces, and I can see a machine using iSCSI having a dedicated high-speed NIC to the purpose -- I'd probably want to.

Now, since I'm not using iSNS or some "dynamic" discovery method, I enable static discovery...

PC# iscsitadm modify admin -S enable

Now, for the client!

On my initiator box, I need to set up discovery and create a static configuration so it can find the target. I begin with the following, using iscsiadm:

PC# iscsiadm add discovery-address <my PC's IP>

That lets iSCSI know which machine to discover LUNs on. Next, I need to provide the SPARC with the pre-set CHAP authentication credentials:

SPARC# iscsiadm modify initiator-node --authentication CHAP

SPARC# iscsiadm modify initiator-node --CHAP-name shumira

SPARC# iscsiadm modify initiator-node --CHAP-secret

Enter secret:<gimmesomedisk>

Re-enter secret:<gimmesomedisk>

Now I have to add a static configuration. Remember the "iSCSI Name" from way up above? I need that now.

SPARC# iscsiadm add static-config iqn.1986-03.com.sun:02:241dafd0-6d3b-ef2f-e6c9-8b62f63ece19.test-volume,<my PC's IP address>,1

The ",1" at the end is the TPG tag I created above.  So the string is basically <iSCSI Name>,<IP>,<TPGT>

Now, I have to enable static discovery on the client's side, as well:

SPARC# iscsiadm modify discovery --static enable

And that should be that. Now I used devfsadm to discover the new "disk", and format to check that it's there.

SPARC# devfsadm -i iscsi

SPARC# format

Searching for disks...done

c1t0100000C7694924000002A0045DE016Ad0: configured with capacity of 10.00GB

AVAILABLE DISK SELECTIONS:

       0. c0t0d0 <QUANTUM FIREBALLP AS60.0 cyl 28739 alt 2 hd 16 sec 255>

          /pci@1f,0/pci@1,1/ide@3/dad@0,0

       1. c1t0100000C7694924000002A0045DE016Ad0 <SUN-SOLARIS-1 cyl 32766 alt 2 hd 4 sec 160>

          /scsi_vhci/ssd@g0100000c7694924000002a0045de016a

Woo-hah. There it is.  From there, you can label and slice it just like you would any other disk in Solaris. Keep in mind that since the backing store is a sparse file, you can expect some godawful performance at first. At one point I'd thought that newfs had hung.

And there you have it. An iSCSI-shared LUN. Surprisingly easy, actually. Keep in mind that this is a very basic setup, and I am definitely no expert. I would be surprised if I *haven't* done something wrong, which might make it insecure or inefficient. But it was enough to get the idea, which I've written down here, mostly for myself.

How do I undo all this?

Well, first, if I've newfs'd/mounted the filesystem, unmount it. Then I get rid of the static mapping on the SPARC side with the following:

SPARC# iscsiadm list static-config

Static Configuration Target: iqn.1986-03.com.sun:02:241dafd0-6d3b-ef2f-e6c9-8b62f63ece19.test-volume,<my PC's IP address>:3260,1

SPARC# iscsiadm remove static-config iqn.1986-03.com.sun:02:241dafd0-6d3b-ef2f-e6c9-8b62f63ece19.test-volume,<my PC's IP address>:3260,1

  Update the iSCSI driver, and then clean up lost device nodes under /dev/dsk and /dev/rdsk...

SPARC# devfsadm -i iscsi

SPARC# devfsadm -C

On the PC side, I clean up with the following commands, where 0 is the LUN and 1 is the TPGT...

PC# iscsitadm delete target -u 0 test-volume

PC# iscsitadm delete tpgt -A 1

You'll notice that when you delete the target, the backing store files are deleted as well! Don't expect to easily recover from doing this.

One last thing... it happened silently, so someone might not notice this, but when you start mucking about with iscsitadm, it automatically starts the "iscsitgt" service under SMF. You can show that it's running with the following:

# svcs -a | grep iscsi

disabled       Feb_07   svc:/network/iscsi_initiator:default

online         15:06:09 svc:/system/iscsitgt:default

If you're just mucking about with a toy, like I was, you might want to turn it back off when you're done. I don't know if it presents any kind of security risk -- I suspect not -- but there's not much reason to leave it going. So:

PC# svcadm disable iscsitgt

... and you're done. Or maybe just starting?  :)

Posted at 05:28PM Feb 22, 2007 by Brandon Hume in Solaris  |  Comments[0]

Solaris MPxIO and the stone-age SAN

Here's some things I've learned from my trials with trying to get MPxIO working with our SAN.  A lot of this stuff doesn't appear to be obvious to me, being new to MPxIO and SANs in the bargain.

• In order for Solaris MPxIO to work with "a random SAN", your SAN must be a symmetric
  SAN, often called an "active/active" SAN.

• If your SAN is asymmetric, or "active/passive", then your SAN must be supported. There's no guesswork at this stuff.  On Solaris 10 11/06 release or beyond, there should be an "mpathadm" command available.  The command:
  

mpathadm show mpath-support libmpscsi_vhci.so

will show a list of SANs supported.  If your SAN isn't in there, and your SAN vendor isn't quick with a suggestion (that is, if they say something along the lines of "Let us check into it") then it's very likely you're out of luck.

• A compatible SAN will support the T10 standard, or "Target Port Group", sometimes called ALUA (Asymmetric Logical Unit Addressing).

After a ridiculous amount of work, we've determined that the IBM FASTt500 SAN is not compatible with Solaris MPxIO, at least on the T2000 platform.  It's very easy to fool yourself into thinking you're making progress.

Many people online will advise you to add an entry to /kernel/drv/scsi_vhci.conf, tricking Solaris into activating MPxIO on your devices.  It will seem to work... Solaris will remove the devices associated with the multiple paths to a LUN, and merge them together into the logical devices you want.  Sometimes, those devices will actually work. But it's a random thing. What isn't obvious to the newcomer is that simply by putting that line in the scsi_vhci.conf, you're declaring that your SAN is symmetric, even if it isn't. (Some people - who aren't me, apparently - will pick up on the symmetric-option bit in the conf file and realize what the file is actually telling MPxIO...)

luxadm display <dev> will show the improper thing... namely, that Solaris thinks both paths are PRIMARY and ONLINE. That's not what you want with an asymmetric SAN, obviously. One of the paths should be SECONDARY and STANDBY.

If you try to use the logical device and it works, chances are you're just getting lucky, and you've got a single controller, or your fabric running through a single switch. Deversify your fabric across switches and across controllers, and check your preferred paths - which is presumably what you'd want to do for a high-availability setup - and you might start finding that your LUNs respond to inquiry commands, but any attempt to label or otherwise talk to the disks starts generating SCSI errors.  (On an IBM SAN, you'll get 0x94 SCSI error codes... which stands for "Not preferred path"...)

Anyway, that was our experience.  Hopefully this will save someone some pain in the future.  As always, if in doubt, harrass the SAN vendor and Sun.  And, as always, "We're not sure" is probably just another way of saying, "Hah!  Fat chance!"

Posted at 02:44PM Feb 01, 2007 by Brandon Hume in Solaris  |  Comments[0]