The corrosion of Aaron Stone

Writing C Unit Tests in Ruby

2012-04-26T00:00:00-07:00

Let’s say you usually code in Ruby, and your company and its build systems are built around Rakefiles and the like. Today you’ve written some C code, and you want to add unit tests. In this blog post, I present a method of writing those C unit tests in Ruby using FFI and RSpec.

/**
 * This is a very silly function that clearly requires some unit tests.
 */
int foo_count_letters(const char *source, size_t *count)
{
  if (!source || !count) return 0;

  for (*count = 0; *source; (*count)++, source++)
    ;

  return 1;
}

First, compile your C code as position independent and symbols exported. This allows you to dlopen() the executable:

$ gcc -pie -rdynamic -o foo foo.c

Next, add the FFI gem to your Gemset, in Gemfile:

source :rubygems

gem 'ffi'

Then write your rspec tests spec/foo_spec.rb:

#!/usr/bin/env ruby
require 'ffi'

# This module is your bridge from Ruby to C and back
module FOO
  extend FFI::Library

  # Use an absolute path to the executable under test, otherwise ffi will search LD_LIBRARY_PATH.
  ffi_lib File.absolute_path(File.join(File.dirname(__FILE__), "..", "foo"))

  # Function signatures for each function to be tested
  attach_function :foo_count_letters, [:string, :pointer], :int
end

describe "unit tests for foo.c" do
  before(:each) do
  end

  it "should really foo" do
    # This function takes a pointer-to-uint32 out-param
    out = FFI::MemoryPointer.new :uint32

    res = FOO.foo_count_letters("Hello", out)

    # Read back the pointers to Ruby data types, then use rspec's verification functions
    out.get_uint32(0).should == 5
    res.should == 1
  end
end

I’m excited about this approach because the tests run under rspec along with the rest of your spec tests.

Moving my Blog to GitHub

2012-04-25T00:00:00-07:00

Today I moved my blog from my own server to GitHub. I exported my blog from Movable Type to Jekyll source files, checked it a GitHub repo, and pointed my domain’s DNS at GitHub Pages.

Introducing a Memcached Module for Varnish 3

2012-01-28T13:01:59-08:00

Varnish makes your websites go faster. Memcached, makes your websites go faster. Blog posts abound on using Varnish’s excellent embedded C functionality to link libmemcached into your VCLs, and Poul himself has written about his surprise in learning that many VCLs were actually thin wrappers around a C library where all of the business logic of a VCL would live.

Starting in Varnish 3, the balance can now shift back towards the VCL itself as the driver of business logic, with first-class “vmod” modules providing new VCL functions. (If you’ve been around a while, you might recognize the same progression that PHP took in the late 90’s and early 2000’s, back when PHP looked like a promising future.)

Without further ado, I present libvmod-memcached, a Varnish Memcached Module. Here’s a quick example script that flushed the cache for a page every 20 hits.

import memcached;

sub vcl_init {
	memcached.servers("localhost");
}

sub vcl_hit {
	if( memcached.incr(req.url, 1) == 0 ) {
		memcached.set(req.url, 0, 86400, 0);
	}
	if( memcached.get(req.url) == "20" {
		purge_url(req.url);
	}
}

Note that VCL does not allow arbitrary variables, so I’m doing two memcached queries here. Fixing that will be another blog post!

Things Dilburn learned at Burning Man

2011-09-26T14:58:19-07:00

The Dilburn campers jotted some useful notes on a little whiteboard at our camp. Eventually I wanted to erase the whiteboard, so here’s what it said!

Burning Man is what you make it.
RVs are really nice.
Art cars are fun!
Hexayurts are the way of the future.
Tents are dusty.
White carports don't reflect enough sun.
Bring extra rebar and sledgehammers. Not just extra. Bring too much.
Coffee.
Couches.
NO PACKAGING!
Repack everything into bins, sorted by type-of-thing.
Bring power tools. Charged.
Try camping on the 7:30 side. It's crowded, and different than the 4:30 side.
Create wind and sun breaks.
Designate a shit table.
Give foot baths.
Drinks are communal. Expect your beer cooler to be raided, so just have everyone pay in upfront.
There's no such thing as a quick trip at Burning Man - never say, "I'll be right back!"

Upgrading an HP dv2000

2011-05-25T18:17:40-07:00

My gal has a laptop that she really likes, an HP dv2000 - dv2225nr to be exact - even though it’s slooow. I’d gotten pretty near to talking her into replacing it when the new job provided a spiffy ThinkPad that she can bring home now and then. Needing only a personal machine to sync her iPhone and surf the web, I said, “What if we just put in a hundred bucks to make it less slow and do a Windows Vista detox?”

Ok, $185 bucks later, and I have some recommendations to make to the lazywebs about this laptop model.

Things I learned:

The AMD Turion 64 x2 with GeForce Go 6150 chipset in this system will happily accept sticks of 4GB DDR2 RAM, probably for a maximum capacity of 8GB. But I stopped at mixing a 4GB stick with an existing 1GB stick, for a total of 5GB.

Check in the BIOS how much RAM is set aside for that GeForce Go 6150. The default is probably 64MB from the factory (with the default 1GB memory configuration, this makes sense I suppose), but you can go to 128MB. Now that 128MB is basically a rounding error on the total memory size, use it.

I tried to clone the original hard drive, but taking it out of the machine caused it to lose its magic smoke. That meant my Windows 7 Upgrade edition was no longer valid for a clean install on a new 7200RPM drive. Follow these directions to get around that:

Finish the installation with the activate-later option.
Once you’re up and running, run regedit as Administrator
Change this key from 1 to 0:
HKEY_LOCAL_MACHINE/Software/Microsoft/Windows/CurrentVersion/Setup/OOBE/MediaBootInstall
Close regedit
Run cmd as Administrator
Run slmgr /rearm at the command line
Close cmd
Activate by going to System -> Change product key

Windows 7 64-bit runs great, uses all of the RAM and all that jazz. [Insert pointless argument about pointer sizes and 64-bit bloat and how I’d be better off with 3.5GB usable RAM at 32-bits. I don’t care; everything else I touch these days is 64-bit. Consistency FTW.]

Finally, the 1.6GHz Turion TL-50 CPU sits in an S1G1 socket, so $20 got me a 2GHz TL-60 with double the L1 cache and a 31W TDP, down from 35W. Runs faster, cooler, and I pretend that the battery lasts a little more. That’s what I told my girlfriend, anyhow! Winning!

DilBURN: Get Fired!

2011-04-16T15:01:09-07:00

Dilburn: Get Fired! a Burning Man 2011 theme camp.

Short description:

Escape the corporate ratrace - come to Dilburn to get “fired.” Pick up your dream job title in the lobby. Come join our office cubicle mixer after hours!

Long description:

Burners can wander our office cubicle maze in search of more meaningful paths. Your first job in cubitraz that may not have met your dreams. Take back your dream! Our wide ties, blue blazers, and black rimmed glasses will teleport you to an alternate reality where you get to be the executive outer space cab driver, senior couch potato, regional manager of coitus, horse and buggy driver, sunbath coordinator, daydream distributor, professional shusher – shhh, we’re trying to work here!

Relax to our muzak and let our inspirational posters boost your productivity by the minute. Be sure to stop by for the office mixer! This is not your daddy’s cube farm!

Strawman layout:

Layout description:

The office portion of the camp will open to the road. This is the theme camp / interactive space; a cubicle farm and maze under a 25’ to 30’ shade structure. Behind the cubicles, vehicles will be used as a perimeter and personal tents arranged inside. Approximately 10 personal tents and smaller shades are expected. Shared spaces will include a 20’ diameter shade structure, 10’x20’ carport cooking and eating space, evaporation device, shower bag structure, and various comfy camping chairs. We will attempt to use solar power, and may supplement with a small Honda-type generator.

Domains for Sale

2011-04-16T14:56:10-07:00

I have a few domains for sale. Would you like to buy one? Contact me and let’s discuss!

Whole Wheat Oatmeal Teff Chocolate Chip Cookies

2010-02-26T10:41:14-08:00

The title is quite a mouthful, and so are these cookies! Crunchy, soft, crumbly, textured, tasty, and as it happens, vegan and parve! I made this recipe up last weekend while it was raining in San Francisco, and had a hankering for cookies. Ate a bunch, and brought the rest in to work to share and meet my coworkers. We all knew it was a rainy weekend, because the company message board was full of “extra cookies in the break room!” messages :)

2/3 cup whole wheat flour
2/3 c. oatmeal
1/3 c. teff flour
1/2 c. sugar (white, demerara, brown, etc.)
1/3 c. olive oil
1/3 c. dark chocolate chips
2 tbsp. vanilla
1 tsp. baking soda
1 tsp. baking powder

Now you’re probably looking through that list and thinking, “Ok, all that looks familiar, except for that teff stuff. What’s that!?” Teff is a grain grown in Ethiopia. It’s one of the most protein-dense grains, and also the smallest whole grain. Ethiopians use it to make their native style of bread, called injera, which I can best describe as a sourdough crêpe. Bob’s Red Mill sells teff flour, available at most natural foods stores and Whole Foods.

Next time I make this recipe, I might also throw in some frozen concentrated orange juice to add a little bit more binding to the finished batter. Eggs would do the trick, too, but I didn’t have any in the fridge when I whipped this up.

Ikea Hacking: Gorm Wine Rack

2010-01-04T13:32:11-08:00

Ikea has at least half a dozen options for wine storage, but its Gorm shelf system takes the cake for versatility. The Gorm is made of solid, unfinished pine, allowing you to hack it more easily than Ikea products made of cardboard, ~~spit shine~~ veneer, and plastic cams. Gorms can stand alone, be chained together, mixed-and-matched with any part in the series, and, because of its simple, solid wood construction, sawed, drilled, screwed, painted, or stained to your heart’s content.

For this hack, I started with a Gorm 30”w x 13”d x 68”h kit, using three of the four shelves, and mixed in three more Gorm wine rack shelves. Before I left Ikea, I went over to the parts department and got eight more screws to match the ones that hold the backstays onto the Gorm.

I picked up two chrome wine glass racks and a dozen wine glasses from Bed, Bath & Beyond, screwed the racks to the underside of the top Gorm shelf (with those extra screws from Ikea), and voilà, my very own wine bar!

Gorm assembly hint: get a 10mm socket for your drill-driver.

Python Meta Decorators

2009-12-08T12:08:16-08:00

Recently, I was working on a project in Python framework that was built around decorators. Each of my functions needed something like six decorators, each with its own arguments, but mostly all the same. I thought, “This is a classic case of cut-and-paste coding; there has to be a better way!” I toiled over decorators for a while, and finally came up with this pattern:

def monkey(arg):
  def outer(f):
    def inner(*args, **kwargs):
      print "Monkey: %s" % arg
      f(*args, **kwargs)
    return inner
  return outer

def zombie(arg):
  def outer(f):
    def inner(*args, **kwargs):
      print "Zombie: %s" % arg
      f(*args, **kwargs)
    return inner
  return outer

def wrapper(m, z):
  def outer(f):
    @monkey(m)
    @zombie(z)
    def inner(*args, **kwargs):
      f(*args, **kwargs)
    return inner
  return outer

@wrapper("coconut", "brains")
def mytesta(foo):
  print foo

@wrapper("bacon", "chainsaw")
def mytestb(foo):
  print foo

mytesta("Hello World")
mytestb("Hello World")

What I’m doing here is first setting up two standard argument-taking decorators, @monkey and @zombie. Then I’m setting up a decorator that returns a decorated function, whose arguments are built at decoration time. This lets me template out a complex decoration pattern where only a few arguments need to change for each function, and pass just those arguments into my meta-decorator.

This is the output:

Monkey: coconut
Zombie: brains
Hello World

Monkey: bacon
Zombie: chainsaw
Hello World

The Globe

2009-10-20T18:54:31-07:00

Since it’s my birthday, I’ll share my new favorite drink:

The Globe from the SFist.

3 oz. 209 Gin
1 oz. St. Germain
1 oz. lime juice
1/2 oz. agave nectar

Shake and serve up with a mint garnish.

Convert a Postfix queue to mbox format

2009-10-12T20:48:05-07:00

Today I wanted to convert the contents of my postfix ‘deferred’ queue, which was all spam stuck in my MTA, to mbox format so that I could feed it into various spam-learning systems.

First I converted my Postfix queue to human-readable format with postcat:

cd /var/spool/postfix/deferred
mkdir spam
for i in {0..9} {A..F}; do mkdir spam/$i; done
for i in `ls */*`; do postcat $i > spam/$i; done

This made a copy of each file in postcat format, but that’s only halfway there – postcat has its own output format that isn’t anything like an mbox or a maildir. Inspecting the output, and brushing up on my sed, I came up with this:

A final pass over all of the messages, and I had my mbox file to train SpamAssassin with:

for i in `ls */*`; do sh mkmbox.sh < $i >> spam.mbox; done
sa-learn --spam --mbox spam.mbox

Recipe for Seitan Stew

2009-10-10T17:13:49-07:00

Found in a backcopy of Vegetarian Journal, March 1996:

1 cup of water plus 1/2 cup water
1 ounce dried wild mushrooms
1 Tablespoon oil
1 large onion, chopped
2 carrots, diced
3 small turnips, peeled and cut in quarters
4-5 small potatoes, cut in half
1/2 pound mushrooms, halved
3 dried tomatoes, made into powder
8 ounces seitan, cut in small chunks
1 teaspoon dried rosemary
1 teaspoon dried thyme
1 teaspoon dried sage
1 Tablespoon miso
1 Tablespoon arrowroot
2 Tablespoons fresh chopped parsley
Black pepper to taste

Boil one cup of the water and soak the dried mushrooms (if they are morels or shiitake) for 30 minutes. Save soaking water. If using porcini add when recommended.

Heat oil in pan over medium heat. Add onion, carrot, turnips, and potatoes. Sauté for 3 to 5 minutes until onion begins to soften. Add fresh mushrooms, tomato powder, and 1/4 cup water. Cook for 5 more minutes. Then add seitan chunks, dried herbs, and rehydrated mushrooms that have been cut in pieces. Cook for 5 more minutes.

Add soaking water drained of any debris and porcini, if using them. Add the miso and stir. Cook for about 10 more minutes until vegetables are almost tender.

Combine the remaining 1/4 cup water with the arrowroot and add to the pan over medium heat, stirring until thickened. If too thick add water 1 tablespoon at a time. If too thin add arrowroot 1 teaspoon at a time. Season with black pepper. Add parsley just before serving.

Health care reform story spotlights selfish stupidity

2009-10-08T15:56:30-07:00

A recent AP story about health care spotlights the sort of close-minded, numb stupidity that fuels anti-reformers:

Andrew Newcomb, 28, who works in sales and lives near Destin, Fla., said he doesn't think taxpayers should have to take on the costs of covering the uninsured. "I don't want my tax money to pay for some pill-popper to fake some injury and go to the hospital when I don't ever go to the hospital," said Newcomb, adding he can afford to go to the doctor and pay $60 for a checkup.

Andrew, since you never go to the hospital, I’m sure it won’t ever be a problem. But if you do get sick, be sure to die quickly. Otherwise you might cost us taxpayers a lot of money at a taxpayer-funded emergency room.

Updating Soekris firmware using Kermit

2009-09-23T18:59:44-07:00

I’ve had trouble getting minicom’s xmodem to fully upload a Soekris firmware image over USB serial adapters, and I’ve never been a big fan of minicom in the first place. I like good old Kermit. Recently I discovered how to get Kermit to speak xmodem, and used it to update the firmware on my Soekris Net4801.

First, install kermit and lrzsz. Next, connect your Soekris box. Now fire up kermit and go!

$ kermit

set protocol xmodem-crc download {} {lsx %s} {} {lrx %s} {}
set line /dev/ttyUSB0
set speed 19200
set carrier-watch off
set flow none
connect

> download -

send /binary firmware-file-name
connect

> flashupdate

Python's Lame Lexical Closures

2009-08-12T07:14:08-07:00

Here’s a simple generator function in Python that you can use in a test harness to find out how many times the function was called. Call the function once with an argument, and then each time you call the function again it will return that argument and increment an internal counter. Call the function with the named argument ‘got_called’ and it will reset and return its current call count.

def GotCalled(ret):
  """Stub that counts how many times it was called."""
  count = [0]

  def _GotCalled(*unused_args, **kwargs):
    if kwargs.get('got_called'):
      lame = count[0]
      count[0] = 0
      return lame
    else:
      count[0] += 1
      return ret

  return _GotCalled

But note how count is a list. And I’m incrementing and returning the first member of the list. Guess what – you can’t use a variable directly in this situation! It’ll error out on the ‘count += 1’ because the act of assigning to the variable in the inner function creates a new instance of that variable, one that hasn’t been used before and so fails the + part of the +=. LAME.

Perl Hash Slices

2009-07-17T15:56:58-07:00

One of my favorite things to do with Perl hashes is slice them with this incredibly concise syntax. The first time I saw this, I Just Got It (TM). Lifting a chunk of text from the excellent (though dated) online book [Picking Up Perl](http://www.ebb.org/PickingUpPerl/), here’s how it works:

Slices

It turns out you can slice hashes just like you were able to slice arrays. This can be useful if you need to extract a certain set of values out of a hash into a list.

use strict;
my %table = qw/schmoe joe smith john simpson bart/;
my @friends = @table{'schmoe', 'smith'};
# @friends has qw/joe john/

Note the use of the @ in front of the hash name. This shows that we are indeed producing a normal list, and you can use this construct in any list context you would like.

Context Considerations

We have now discussed all the different ways you can use variables in list and scalar context. At this point, it might be helpful to review all the ways we have used variables in different contexts. The table that follows identifies many of the ways variables are used in Perl.

Expression	Context	Variable	Evaluates to
`$scalar`	scalar	`$scalar`, a scalar	the value held in `$scalar`
`@array`	list	`@array`, an array	the list of values (in order) held in `@array`
`@array`	scalar	`@array`, an array	the total number of elements in `@array` (same as `$\#array + 1`)
`$array\[$x\]`	scalar	`@array`, an array	the `($x+1)`th element of `@array`
`$\#array`	scalar	`@array`, an array	the subscript of the last element in `@array` (same as `@array -1`)
`@array\[$x, $y\]`	list	`@array`, an array	a slice, listing two elements from `@array` (same as `($array[$x], $array[$y])`)
`"$scalar"`	scalar $interpolated$	`$scalar`, a scalar	a string containing the contents of `$scalar`
`"@array"`	scalar $interpolated$	`@array`, an array	a string containing the elements of `@array`, separated by spaces
`%hash`	list	`%hash`, a hash	a list of alternating keys and values from `%hash`
`$hash\{$x\}`	scalar	`%hash`, a hash	the element from `%hash` with the key of `$x`
`@hash\{$x, $y\}`	list	`%hash`, a hash	a slice, listing two elements from `%hash` $same as `($hash\{$x\}, $hash\{$y\}$`

On Republicans and Obama's healthcare plans

2009-06-15T22:56:47-07:00

From this AP story on Obama pressing doctors to back health care overhaul:

GOP Rep. Tom Price of Georgia – a former orthopedic surgeon – reacted preemptively to Obama’s speech by accusing him of seeking a “government takeover” of health care. Speaking to reporters on a conference call organized by the Republican National Committee, Price said a committee that Obama’s administration has established to study the effectiveness of various medical treatments would turn into a “rationing board” to overrule doctors and deny patients care. Sen. Jon Kyl, R-Ariz., and other Republicans introduced legislation to ban the rationing of care on such a basis.

Hello, pot? Yeah, this is kettle. You're black!

If these guys are going to draft legislation that prevents all insurance companies from denying coverage, awesome. If they allow private insurers to cut coverage but not a future public insurance, that’s just one more scummy way that Republicans write laws specifically to doom government agencies to fail, then rail against those same agencies saying they shouldn’t exist.

Slicehost as secondary DNS and backup MX

2009-06-02T11:18:50-07:00

This blog runs on my own server, on my own DSL line, with my own domains and DNS, running my own email service using some of my own email code.

Until those days when the lines are down and the tubes are tied up and the power is out and the magic smoke goes poof. Those days, I want nothing more than a normal hosted email service like everyone else.

But let’s get real. No I don’t! I like running my own stuff! Of course it’s embarrassing to bounce email when my server is offline, so what’s really important is to follow the best-practices for running a proper site, including secondary DNS and backup MX.

The first thing to do when figuring out secondary DNS is to figure out how many domains you have. I only run email for 3 domains, but I have a raft of neat domains that I own. Most secondary DNS services charge on a per-domain basis, so getting secondary service for 25 domains will cost a bundle, and doesn’t even begin to solve the problem of a backup MX.

Looking around some more, I found that some hosting services provide unlimited DNS service along with a subscription for a hosted virtual machine. The best of those services is Slicehost, starting at $20/month for a basic virtual machine instance with 256 MB of RAM and 10 GB of disk. I highly recommend this service and the basic configuration for your backup MX.

Ok, so we’ve got a virtual machine instance, now what? Gotta install something on it! I recommend whatever is the most recent Ubuntu LTS. You don’t need or want any new hotness for your backup MX. You need and want stupidly simple and someone-else-is-supporting-this reliable. Ubuntu LTS. Do it.

Back in the code days

2009-04-14T14:13:32-07:00

Why, when I was a kid, we had to write code while walking 20 miles to the computer building, in 12 feet of snow in the middle of winter. And it was uphill both ways! Course we couldn’t wear gloves, because it was too hard to line up the hole punch on the punched card. They didn’t have knapsacks in those days, so we just had to keep our card stack on a string tied to our belt. Now, a hole punch cost a nickel, and in those days nickels had pictures of bumblebees on ‘em. “Give me five bees for a quarter,” you’d say. Now where were we? Oh yeah, the important thing was I had a stack of punch cards on my belt, was the style at the time. They didn’t have standard 5081 cards in stock, because of the war. The only thing you could get was graph papyrus, and you had to draw all the tables by hand.

Best. Slashdot. Comment. Evar.

Yemenite and Rhodes Haroset

2009-04-09T23:56:29-07:00

From The World of Jewish Cooking, by Gil Marks:

Yemenite

15 dried figs, chopped
2 to 3 tablespoons sesame seeds, lightly toasted
1 teaspoon ground cinnamon
1 teaspoon ground ginger
Dash of ground coriander or cardamom
1 small chili or pinch of cayenne
Dry red wine

Finely chop the figs, dates, sesame seeds, cinnamon, ginger, coriander or cardamom, and chili or cayenne. Stir in enough wine to make a paste. Storein the refrigerator. Serve at room temperature.

Rhodes

1/2 cup pitted dates, finely chopped
1/2 cup raisins
1 orange, peeled, seeded, and chopped
1/4 cup honey
1/2 cup almonds, finely chopped
cinnamon
about 1/4 cup sweet red wine

Cook dates, raisins, orange, and honey, stirring, until thick (about 20 minutes). Remove from heat and add remaining ingredients.

Register.com teaches SSL customers how to phish

2009-03-05T23:58:40-08:00

I just bought an SSL certificate for this domain from Register.com. The price was great ($38 for 3 years! Just Google for “discount ssl certificate” to get a Register.com AdWord banner discount), the process went really smoothly, the certificate installed easily into Apache, and is working well for me so far.

In the middle of the process, while I was downloading the certs and reading their directions, I came across this doozy of a page. What Register.com is giving you are static pictures of locks to make users feel safe. In other words, teaching people to trust the telltale signs of phishing tactics! And I’m downright shocked at the favicon of a lock. That’s a straight-up bald-faced attempt at making the site look like it’s being seen by the browser as secured. Not cool!

Multitouching more of New York

2009-01-22T10:45:15-08:00

This new Google New York project at nycgo.com, using Google Maps and Earth with lots of local data is neat. But where it gets really incredible is the touchscreen interface that reminds me of Minority Report. Specifically, I’m thinking about the large immersive gesture system that Tom Cruise uses to gather information about his situation.

Which is, of course, already something of a reality! Jeff Han demonstrated a large multitouch screen with correspondingly incredible user interface at TED in 2006. Now Art Lebedev of Optimus Maximus keyboard fame is working on the Optimus Tactus, a concept keyboard where the keys are replaced with a touch screen, and the “keyboard” itself becomes reconfigurable to suit the application – keys for typing, mixer controls for audio, video jogs and patches, and so on.

Next document

2009-01-16T15:55:36-08:00

draft-ietf-sieve-include!

mod_perlite on O'Reilly Broadcast

2009-01-16T15:46:47-08:00

w00t –

http://broadcast.oreilly.com/2009/01/cgi-is-dead-mod-perlite-is-ali.html

Back in December, Byrne and I had a telephone interview with chromatic talking about mod_perlite, one of the five features Perl 5 needs now.

Firefox needs to chillax

2008-11-21T10:58:55-08:00

Top causes for wakeups:

8% ( 68.5) firefox-bin : futex_wait (hrtimer_wakeup)
3% ( 27.0) firefox-bin : schedule_timeout (process_timeout)

Let my CPU get to C3 once in a while, plz!

Sieve Reject

2008-11-17T11:13:03-08:00

At long last, I posted the next draft

Neat!

2008-10-23T11:36:12-07:00

MTMultiBlog FTW! At some point I’d like to make my scrappy sidebar blog itself look more like my own twitter-alike.

Open Letter for Bicycles on Public Transit

2008-10-14T13:34:41-07:00

Thoughts on living in San Francisco, and on getting around this place…

San Francisco is an extremely car-congested city. Even if I owned a car, I’d be fighting every day for parking and space on the roads. Why would I do that to myself? So, I bike to work, or I take the bus and walk a few blocks from the bus stop. When I used to live on the Peninsula, I took Caltrain to work. Before that, I lived in Oakland and took BART to work.

So let’s focus on problem areas: First, the roads are in terrible condition, with potholes big enough to eat bicycle tires whole - sometimes even smack in the middle of a dedicated bicycle lane! Second, our bridges do not have consistent bicycle paths. Third, our public transit is outright hostile to bicycles.

Let me dwell on that third point. Each and every public transit agency has different policies about when, how many, and what kind of bicycles are allowed on-board a bus, light-rail (SF Muni), electrified-rail (BART) or heavy-rail train (Caltrain).

It should be national policy to encourage and subsidize the expansion of bicycle commuting capacity in all public transportation systems. The cost of adding one rack-seat combo to a bus or train is a heavy burden upon a municipal transit agency, but in terms of societal cost vs. cars – that require roads, traffic signals, gas stations, parking lots, garages – that dollar spent subsidizing the municipal transit agency’s bike-on-board rack is suddenly a fraction of what we’d otherwise spend to enable that same person to commute by car.

Bicycles on board public transit. It just makes sense.

Hacking

2008-09-26T11:40:04-07:00

Nothing this week :-(

Work

2008-09-26T11:27:59-07:00

Very busy.

Running nmbd on OpenBSD

2008-09-12T12:45:52-07:00

Some days I need to share files across subnets at home, and always end up frustrated that some part of the infrastructure is there to give me a headache. Like, say, that Windows file shares still cannot readily browse across subnets! (Neither did Rendezvous/Bonjour on the Mac, originally, but it does now.) Consequently I set up just nmbd on my OpenBSD firewall, which routes between my various wireless and wired networks, so provide the NetBIOS name resolution across subnets needed to allow for cross-subnet Windows network browsing. Note, crucially, that I’m not running smbd. No files are being shared here, no users are being authenticated, I am strictly using nmbd as a name server.Take the smb.conf from part 1 and add these lines to the [global] section:

   domain master = yes    preferred master = yes     local master = yes    wins support = yes    wins proxy = yes

os level = 65

http://files.dns-sd.org/draft-cheshire-dnsext-dns-sd.txt

http://www.ietf.org/rfc/rfc2136.txt

http://www.apple.com/support/downloads/bonjourforwindows.html

http://www.chaoticsoftware.com/ProductPages/NetworkBeacon.html

http://developer.apple.com/networking/bonjour/faq.html

Coding for a Reentrant Flex

2008-09-12T12:45:52-07:00

Bottle Return

2008-07-29T03:31:55-07:00

When I was in the Czech Republic in March 2007, I went to see a movie called Vratné lahve, which in Czech means “Bottle Return” – although they titled the movie “Empties” in English. There’s a unique feature of Czech supermarkets, a bottle return department, where you return glass bottles of juice, water, and especially beer, and they are returned to the bottling company to be washed a reused without going through an American-style crushing and remanufacturing process. The movie centers around a fellow who loses his job as a teacher of literature and finds work as the man in the bottle return booth at a local supermarket. In the opening sequence of the film, the man is reading a poem to his class, Za trochu lásky... by Jaroslav Vrchlický.

Za trochu lásky šel bych svĕta kraj,
šel s hlavou odkrytou a šel bych bosý,
šel v ledu - ale v duši vĕčný máj,
šel vichřicí - však slyšel zpívat kosy,
šel pouští - a mĕl v srdci perly rosy.
Za trochu lásky šel bych svĕta kraj,
jak ten, kdo zpívá u dveří a prosí.

I’ve been looking for an English translation of this poem since I saw the movie, and thanks to the help of the linguaphiles community of LJ, and my Czech friend Dana, I now have one!

For a bit of love, I would go to the end of the world,
I would go bareheaded and I would go barefoot,
I would go through the ice - but with eternal May in my soul,
I would go through the storm - but would hear blackbirds singing,
I would go into the wilderness - and would have of pearls or dew in the heart
For a bit of love, I would go to the end of the world,
as one who sings begging at the door.

Na zdraví!

Plugin Strategies in Open Source, Part 2

2008-06-04T16:28:50-07:00

In any project, feature-creep can be a problem. In an open source project, this can be particularly acute when there are few developers. Worse yet, someone becomes super-active just until their major new feature lands and then they disappear! And then there’s the problem of noisy people who won’t stop whining about getting some peculiar feature, or some major new functionality, either or both of which are just not interesting to anybody else.

An effective plugin strategy actually embraces these people and their pet features, providing a mechanism for decoupling code from the code project, yet allowing key features to be injected into the running system from a separate and contained plugins area. Note the words separate and contained. Both are very important.

The plugins area must be separate because you want to protect the clear messaging of your application’s features. You also want to keep plugins separate because there can be a tendency to make everything a plugin. Don’t. Just don’t. Really. Core features are core. They are not plugins. Nobody wants a framework that does nothing until you’ve loaded a jillion plugins into it in order to create a working application. So separate is a double-edged sword: keep the non-essential stuff away from the core applications, and keep all the essential stuff inside the core application. Sometimes it means swallowing a plugin into the app. If it makes sense, do it.

The plugins must be contained. And, ideally, also self-contained. That is, you have a directory like ‘/your/app/plugins/some_crappy_plugin’ for each plugin. The plugins are contained within ‘/your/app/plugins’ and each one is further self-contained another directory level below that. Then, provide an API that allows the plugins to act at a distance. Yes, act at a distance. Normally this is something that you don’t want because it is hard to figure out. But in the case of plugins, it is just right. The plugin does not need to patch into the main app code, but rather register itself with the main app and declare which of its functions should be called from which parts of the main app. It’s runtime integration, runtime configurability, and runtime enable/disable. Sure it can be slower. But it’s so much better than having people distributing patches that implement their functionality by hacking up your beautiful code.

Handling exit within an eval

2008-03-15T17:16:24-07:00

So I’ve run into a problem with mod_perlite, and it’s that I cannot just override Perl’s exit function with a straight perl_destruct call, a longjmp back to the Apache handler, or anything else at all.

Here’s my favorite…

bar.pl

exit;

Foo.pm

package Foo;

sub new {
  return bless { };
}

sub DESTROY {
  print "Foo is destroyed\n";
}

1;

foo.pl

use Foo;

my $foo = Foo-&gt;new;

print "I have a foo!\n";

BEGIN { *CORE::GLOBAL::exit = sub { goto EXIT; }; }

eval {
  bar("bar.pl");
}; if ($@) {
  print "$@ happens\n";
}

print "And I'm still here\n";

sub bar {
  eval require $_[0];
  EXIT:
  print "Exiting!\n";
  goto REALLY_EXIT;
}

REALLY_EXIT:

Yep, that actually works!

I have a foo!  
Exiting!  
Foo is destroyed.

Plugin Strategies in Open Source, Part 1

2008-03-15T17:11:01-07:00

Once upon a time, I worked * on an open source groupware application called TWIG, The Web Information Gateway. I started posting fixes, wrote some code to scratch my itches, and pretty soon I got CVS commit access. I wrote a new module for scheduling meetings, and the guys on the development team loved it, so I committed it, and life was good. Pretty soon, I was using this module to schedule meetings in rooms that got to be conflicting and overlapped. I also had heard that back at my high school, there was a particular teacher who thought she was the Goddess-of-all-Writing and therefore could simply walk up to the whiteboard in the main computer lab and erase other teachers from the schedule in order to accommodate her clearly superior, holy and blessed classes.

So I wrote a reservations module that integrated into this groupware platform, and it worked, and I got positive feedback from the folks back at my old high school, folks on the twig users mailing list, and from the development team, so I committed it.

The maintainer flipped out. “NO MORE MODULES!” he said, “REMOVE THIS OR I WILL REVOKE YOUR CVS ACCESS!”

Other developers and users came to my defense and said, “But this module is useful, it’s excellent, it complements the scheduling and meetings features. It’s killer!”

“DIE MODULES, DIE!” the maintainer said.

The project died.

Actually, I still use TWIG personally, and I still hack on it when I have time, and every now and then there are new users who post to the mailing list, and I generally respond to help them out within a few hours. But the project is effectively dead.

libSieve Hacking

2008-02-22T13:57:39-08:00

This month a number of Sieve extensions became published RFCs, along with an update to the Sieve base spec itself:

[RFC 5228] Sieve: An Email Filtering Language.
[RFC 5229] Sieve Email Filtering: Variables Extension.
[RFC 5230] Sieve Email Filtering: Vacation Extension.
[RFC 5231] Sieve Email Filtering: Relational Extension.
[RFC 5232] Sieve Email Filtering: Imap4flags Extension.
[RFC 5233] Sieve Email Filtering: Subaddress Extension.
[RFC 5235] Sieve Email Filtering: Spamtest and Virustest Extensions.

Time for teh libSieve hacken!

TWIG Hacking

2008-01-05T20:06:50-08:00

For the first time in a long time, I spent a weekend hacking on TWIG! :-D Tons of things work now in TWIG 4, lots of code flow improvements, CSS and Javascript improvements for Web 2.0 (more on that in a minute), database cleanliness and efficiency improvements, Reservations feature checked back in for the first time in a very long time, HelpDesk works, and the guts of a totally new Mail backend based on the IlohaMail IMAP library.

Back when the world was about separation content from presentation, TWIG 2 was all about mixing them together in the early tradition weekend-hack PHP apps, despite TWIG being a very large and otherwise well organized framework. Now that we’re in Web 2.0 days, the point at which content mixes with presentation to form an output web page has moved from being entirely on the server to being partially on the server and partially on the client. This is the Ajax way.

When I first started working on TWIG 4, I thought I would need a structural mechanism to allow data to be mixed into the source HTML or to be retrieved via Ajax. But it’s been on the table for so long now that Web 2.0 caught up with us and every reason for not relying on Javascript – old browsers, mobile devices, screen readers – now support at least basic Ajax functionality. With this now nearly universal support for running Javascript and requesting data via Ajax, I’ve now worked it into TWIG 4 as a requirement and a basic part of the architecture.

Next time, writing your own mashups using the TWIG query API and native Wiki feature!

Another MySQL headache

2007-11-27T16:55:11-08:00

In http://www.dbmail.org/mantis/view.php?id=655, Mr. Maenaka writes:

MySQL's automatic reconnection is unsafe because of the following reason.First, SET NAMES query is a required arbitration between server and client if both's character encoding is different. This should be done at the time of connection open. (Of course you can issue SET NAMES between every query with the huge overhead though.) By the way, MySQL's automatic reconnection is made transparent to the client. So the client never know that SET NAMES should be issued again. Therefore, if the connection is restored this way, the character encoding mismatch may occur (and some or all data is garbled or lost).

No, really, seriously, you auto-reconnect without setting my connection-specific settings in the new connection? What kind of crack are you smoking at MySQL AB!?

Two request monty for persistent services on non-persistent hosts

2007-11-07T12:15:46-08:00

A problem I’ve been thinking about for a long time was how to build a backend service for a stateless web frontend. All of the solutions I have seen involve building a daemon that lives on the server and holds the persistent data. Or, in the case of PHP, weird ass service side session cookie things built right into the language.

The solution I want is a hack that is as full featured as building a daemon and does not require modifying the operating environment. And today it hit me: the two request monty.

Here’s the setup: you have a script that’s running in the web server, it has an execution time limit, you’re allowed to create and listen on sockets. Yes, I’m thinking of PHP safe mode here. To create the service, your web app makes a request back to the host web server, spinning up the service script. That script then creates a socket listening to some local port, tries to increase its max execution time, sets a timer for just under whatever that time limit is, writes this information to some common location (typically the database behind your web app) then begins listening on the socket for requests for whatever kind of persistent data you need. When the timer fires, the script makes a request back to the host web server, spinning up a new service script. The new service spins up on some other port, connects to the first service and grabs all of its data, then writes its information to the database and begins serving persistent data to your application.

Total hack beyond hack, but satisfies the requirements: does not require a new daemon, does not require new modules on the server, runs in safe mode, runs with time limits.

Perl in Apache with mod_perlite

2007-11-06T14:10:12-08:00

Lately at work a few folks have been batting around possible solutions to the perceived problem of how hard it is to run Perl code from Apache. Of course there are very good solutions, mod_cgi, mod_fastcgi, and best of all mod_perl. But there are very good reasons why web hosts shy away from CGI, and why they're deathly afraid of FastCGI and mod_perl. In a nutshell, it all comes down to persistence. PHP is incredibly popular because there is no persistence!

Byrne Reese had the idea of taking mod_php and ripping out the calls to the PHP interpreter, replacing them with a Perl interpreter. "I don't think it's quite that simple..." I said. But Byrne was 80% correct. The other 20% is grabbing the Apache - PerlIO layer from mod_perl and using that to shovel data from STDIN and STDOUT in and out of the Perl script. The resulting code is pretty simple!

Fundamentally, mod_perlite tries not to solve all your problems. It is specifically targeted at being 80% good at the 80% problem. With luck, we'll be able to get it onto 64% of web servers like PHP and pals.

To Do:

Thrash at the Apache - PerlIO interface some more.
Develop a script caching model (ala Zend Accelerator or APC).
Add a script run-timer to kill long-running scripts (ala PHP's max_execution_time limit).

SSL Key Management App

2007-11-02T14:17:14-07:00

The crux of my SSL management headache is that I have a lot of domains, but only four public IP addresses. I bet that most home-operated sites are like that. I also have a number of different applications running – HTTPS, SMTPS, and IMAPS in particular. For each HTTPS domain, I need to have an SSL key that is bound to that domain. But since most of the domains are virtual hosts on a single IP, I don’t know the domain until after the SSL negotiation. TLS is supposed to solve some of this, I think, but there’s enough SSL out there that I need to deal with it. A tool that can tell me that I have { X } number of HTTPS domains but only { X, n > 0 : X-n } IPs, and allow me to pick which ones get keys, then generate the keys for me, would be grand! Furthermore, I often create convenience sub-domains for particular applications to facilitate portable DNS inside and outside of my home firewall. smtp.serendipity.cx, for example, resolves differently inside and outside my firewall. Outside the firewall, it resolves the same as serendipity.cx. Inside the firewall, it does not, since I don’t NAT my servers from the inside. So I need separate SSL keys for these two domains. But they’re also different apps, so there’s no IP conflict. Am I making sense? Yeah, so I need an open source SSL key management app. Or I need to write one ;-)

Rolling in Internets

2007-10-26T18:18:41-07:00

Well, it finally happened. I caved. I bought Internets everywhere with a Sprint plan and the Novatel USB727 doohickey. Bonus points to Novatel for packing a GPS and a EVDO rev A modem into such a small package and providing working Linux instructions involving only standard kernel serial drivers!

My critique of the instructions is also a strength: they’re looong, repeated several times for various distributions, but also totally idiot-proofed. At the very end, the steps to quickly get the modem dialing from the command line are presented.

Instruction for using the Sprint U727 in Linux, (and very likely the Verizon USB727, too). Here’s what works for me:

First, add this line to /etc/modprobe.conf:options usbserial vendor=0x1410 product=0x4100
Create /etc/wvdial.conf with these contents: [Dialer Defaults] Modem = /dev/ttyUSB0 Baud = 460800 Init1 = ATZ Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0 ISDN = 0 Modem Type = USB Modem Phone = #777 Username = '' Password = '' Carrier Check = no Stupid Mode = yes
Then, each time you want to use the modem:

Insert the USB modem.
Wait for the device to be recognized as a cdrom.
eject /dev/cdrom(since I don't have any other cdrom - you might need to eject /dev/cdrom1 or something)
Wait a beat.
wvdial
route add default ppp0

Things that make MySQL dumb

2007-10-19T17:56:18-07:00

http://forums.mysql.com/read.php?92,133026,133102

Hi. Since the updates on a memory table should be very fast, the locks are taken for a very short time only. I won’t expect that they become a bottleneck. But I must admit that I don’t have any experiences with applications like this. If they do indeed become a bottleneck, you could try to use a transactional engine that uses row level locks, like InnoDB. But I guess that the overhead will outweigh the improvements in locking.

Regards Ingo

Ingo Strüwing, Senior Software Developer - Storage Engines MySQL AB, www.mysql.com

Ouch, man. That’s my cerebellum you’re destroying.

Coloring in the planet

2007-10-19T17:55:27-07:00

First of October ToDo list

2007-10-01T11:57:23-07:00

Here we go…

Done! Edit draft-ietf-ipv6-ra-flags-option-02 and draft-weiler-dnssec-dlv-04
Done! Submit draft-ietf-sieve-refuse-reject-05
Done! Release libSieve 2.2.6
- Update docs
- New callbacks for address and header parser errors

Saw Raw & Uncut Festival and Balkan Beat Box on Saturday evening. Hurrah for fun!

libSieve Todo List

2007-09-06T18:22:37-07:00

A few things to do before I get to libSieve 2.2.6…

More checkins.
Bracketed comments.
Stress tests for header and address parsers.
Resolve extant memory leaks (switch to a memory pool system? methinks yes).
Benchmark parsing a monster script. Compare with previous versions.
Stress test the stack with deeply nested if blocks.

Whenever it takes me a while to close a bug…

2007-08-06T12:06:39-07:00

Opened: 2006-03-01 00:32 Gist of fix required: accept “{” bytecount “+}” in addition to “{” bytecount ”}” for RFC compliance. Closed: 2007-08-06 19:29

Thanks nonetheless to the KDE folks for closing this bug; fewer complaints from my DBMail users! :-)

Romanesca Cauliflower

2007-08-05T18:25:52-07:00

This week’s Two Small Farms CSA basket came with this weird thing: Well, what to do? Cut it up and saute it in olive oil with some garlic! Yum :-)

libSieve hacking

2007-07-28T20:00:45-07:00

Today I spent some time hacking on libSieve for the first time in way too long. I was telling a friend about the inefficiency of putting all of your keywords into a lexer because it has to go through tons of state shifts to get to the terminals. A better way to do it is to match a generic identifier with a regular expression, then look up the matches in a hash table of keywords. Well, enough was enough, time for me to do that! The old CMU sieve-lex.l, with 56 keywords each as its own lexer state, yielded this flex -v output:

/bin/sh ../ylwrap sieve-lex.l lex.yy.c sieve-lex.c -- flex -v -s -olex.yy.c
flex version 2.5.33 usage statistics:
scanner options: -svB8 -Cem -olex.yy.c -Plibsieve_sieve
584/2000 NFA states
340/1000 DFA states (737 words)
72 rules
Compressed tables always back-up
Beginning-of-line patterns used
3/40 start conditions
179 epsilon states, 98 double epsilon states
8/100 character classes needed 22/500 words of storage, 0 reused
1235 state/nextstate pairs created
414/821 unique/duplicate transitions
347/1000 base-def entries created
448/2000 (peak 399) nxt-chk entries created
28/2500 (peak 266) template nxt-chk entries created
0 empty table entries
8 protos created
7 templates created, 23 uses
38/256 equivalence classes created
4/256 meta-equivalence classes created
0 (15 saved) hash collisions, 81 DFAs equal
0 sets of reallocations needed
1884 total table entries needed

After moving the keywords out of sieve-lex.l and into sieve-keywords.gperf, replacing them all with a generic state,

ident           :?[a-zA-Z_][a-zA-Z_0-9]*
&lt;INITIAL&gt;{ident}        { int k = libsieve_keyword(yytext, yyleng); ... error handling ... }

the flex -v output reads like this:

flex version 2.5.33 usage statistics:
scanner options: -svB8 -Cem -olex.yy.c -Plibsieve_sieve
146/2000 NFA states
72/1000 DFA states (304 words)
19 rules
Compressed tables always back-up
Beginning-of-line patterns used
3/40 start conditions
79 epsilon states, 50 double epsilon states
11/100 character classes needed 165/500 words of storage, 1 reused
830 state/nextstate pairs created
144/686 unique/duplicate transitions
83/1000 base-def entries created
204/2000 (peak 392) nxt-chk entries created
88/2500 (peak 308) template nxt-chk entries created
0 empty table entries
13 protos created
11 templates created, 35 uses
28/256 equivalence classes created
8/256 meta-equivalence classes created
3 (3 saved) hash collisions, 79 DFAs equal
0 sets of reallocations needed
858 total table entries needed

Ok, that’s better. But what about that one ident with the :? up front? What if it were this:

ident           [a-zA-Z_][a-zA-Z_0-9]*
&lt;INITIAL&gt;{ident}        { int k = libsieve_keyword(yytext, yyleng); ... error handling ... }

&lt;INITIAL&gt;:{ident}        { int k = libsieve_keyword(yytext, yyleng); ... error handling ... }

And flex -v says:

flex version 2.5.33 usage statistics:
scanner options: -svB8 -Cem -olex.yy.c -Plibsieve_sieve
150/2000 NFA states
73/1000 DFA states (306 words)
20 rules
Compressed tables always back-up
Beginning-of-line patterns used
3/40 start conditions
81 epsilon states, 52 double epsilon states
11/100 character classes needed 165/500 words of storage, 3 reused
844 state/nextstate pairs created
145/699 unique/duplicate transitions
85/1000 base-def entries created
207/2000 (peak 420) nxt-chk entries created
96/2500 (peak 336) template nxt-chk entries created
0 empty table entries
14 protos created
12 templates created, 36 uses
28/256 equivalence classes created
8/256 meta-equivalence classes created
0 (4 saved) hash collisions, 79 DFAs equal
0 sets of reallocations needed
868 total table entries needed

Calendaring is still a pain

2007-07-27T13:59:19-07:00

Why is calendaring still such a pain in the ass? There’s been nothing but a proliferation of propriety solutions, web apps that store their data locally and do not share it, servers that speak antiquated versions of dodgy protocols, and clients that are continually being rebuilt from the ground up instead of evolved to be more useful over time.

Killing Exchange is still the killer application. We can’t think past that because, much as we’d like to, we still have to work with our sales, marketing, executive, and other non-technical staffs that Just Won’t Use Anything Else (TM). Or will they? Is the big problem here that everyone is so accustomed to Outlook that they cannot switch, or that organizations think this way and therefore attempt to deploy Exchange-compatible services to cater to a perceived need for Outlook clients?

Is anybody actually running Mozilla Sunbird in a real office environment?

Off in my project world, I was just thinking to myself, "Hmm, maybe I should build a WebDAV server into DBMail. The storage model is all key-value, so I could fake the whole thing as email headers." Then I found this DAV-IMAP proxy. I’m not actually sure if it speaks DAV or just the subset required for Mozilla Sunbird. But whatever - the point is, it’s a calendar in a mailbox. On a non-calendar-aware client, you could just read through your Calendar mailbox to see an ugly-but-workable view of your calendar events. On a calendar-aware client, you get the full awesomeness of a calendar. And shared calendars? That might just come for free if your mail server implements shared folders, as DBMail indeed does.

Crosspost test

2007-07-17T12:11:05-07:00

This is a crosspost. Let’s see if it works… This is more stuff. It should be an lj-cut that links back to my blog. Wow cool if it works!

Working at Six Apart

2007-04-29T11:44:45-07:00

I am setting sail for San Francisco, to a web development firm. My new daily grind awaits! Oh, heh, and hopefully they won’t mind that this blog runs WordPress ;-)

Private: Simplifi Digital - Home Theater Audio

2007-03-07T15:44:05-08:00

A while back I got a Simplifi Simplifi 5075D USB amplifier to build up my home theater system. But there was a catch: Simplifi Digital did not support Linux or Mac OS X, and I was not going near Windows MCE in my living room. What to do? Ask Simplifi Digital for a loaner box to test out the cross-platform support!

I had been planning out my digital home theater system for a while, but always hit one stumbling block in designing the system: I needed a remote control to turn the amplifier on and off, and to change the volume, or I needed an IR blaster to do it from the computer. But an IR blaster doesn’t know if its turning the device on or off and can’t bring the amp to a known volume level at startup, say for use as an alarm clock or scheduling it to turn on your favorite show when you get home from work. Yes, I know some amplifiers have distinct remote control commands for ‘on’ and ‘off’, but that sort of esoteric minutiae is nearly impossible to figure out before buying the device.

What I needed was an amplifier that could be completely controlled by my home theater system, without any buttons, switches, remote controls, or strange serial protocols requiring one-off, hacked up cabling. Enter Simplifi Digital.

In a Nutshell

The Simplifi Digital 5075D USB amplifier works great under Linux, with many distributions supporting its 5.1 audio right out of the box. For those that don’t, you configure it just like a SoundBlaster 24-bit USB audio card (details below). The 5075D does not work under Mac OS X in anything but 2.0 channel mode. Instead, I recommend the Simplifi Digital 5075 Analog 5.1 amplifier and a Griffin FireWave 1394 sound card; this combination works in 2.1 right out of the box, and in 5.1 with Griffin’s drivers (details below).

Linux in Detail

foo.

Mac OS X in Detail

foo.

Powerset

2007-02-26T10:45:04-08:00

I had a pretty good job offer this past month from a natural language search firm looking for computational linguists – a dream job, right up my alley – but had to turn it down, for no other reason but the horrendously lopsided employment contract. Bottom ine: this management team does not trust the people it hires. So, for the benefit of those out there who have a habit of not reading what you sign, pay heed to some of the onerous provisions you may be committing yourself to.

They did invite me to launch party, though, and between meeting famous linguists whose papers I studied in school, and watching the party crashing going on, the entire scene almost had me break out in uncontrollable lulz.

Working at Decru

2006-12-09T00:12:37-08:00

In November, I took a job at Decru, a Network Appliance company, as a QA Engineer. I’m in the Partner Relations group. Our job is to establish relationships with third party QA labs to run formal interoperability certifications between our products and their products. There’s two other people in the group, and they’re both really great to work with.

Decru’s products are a set of hardware encryption appliances that sit on Ethernet, Fibre Channel or SCSI links between users, servers, disk arrays and tape backup systems, encrypting the data as it goes to the storage device and decrypting the data it goes towards the users. Decru itself is a fun little company, very laid back and friendly people who are really keen on security, good code, and sane release schedules. This is a company that sells to Fortune 500’s and the government, so there’s no hot-dogging of features that don’t work, don’t make sense or won’t be reliable. But this is still Silicon Valley, so while the products need to be really staid devices, the company itself is still a dynamic place with lots of neat ideas floating around.

Company trivia: the co-founders knew that they wanted to start a company, and knew the crew that they wanted to start it with, and knew the wanted to do something in either storage or security. They combined the two, called up ‘da crew’ (think bad Israeli accents) and thus Decru was formed.

Job Hunting

2006-10-20T12:33:52-07:00

I’ve been interviewing all week with small and large firms in the San Jose / Silicon Valley area for work as a Software Engineer. Interviewing is a lot of fun – dress just slightly better than the firm you’re walking into, but not too much better, else they’ll think you’re a business person and not an engineer – get tons of adrenaline flowing but stay calm and try not to forget everything you learned in school (which you’ll inevitably be asked to draw out on the nearest whiteboard) – and, most importantly, don’t forget to interview them! The company wants someone who wants to be there, so you have to ask a lot of questions about the company to see if you do want to be there. What’s their management style? How well funded are they? Are they turning a profit on current products? How many people work there? Is there a program in place to reward extra effort that you put into projects? Do they order in dinner on the late work nights before a major product launch? And when it’s over, don’t tarry, just thank them, give an honest impression if you liked the work place, and be sure to follow up in 2-3 days.

Summer of Code!

2006-05-27T19:05:22-07:00

I got in! Joomla! will be my project, with Lee Cher Yeong as my mentor. To celebrate, I’m reading two books and building a barbeque pit.

Done with Midterms and SoC apps

2006-05-09T09:23:24-07:00

I wrote four applications for the Summer of Code. The first one took like 12 hours to research and write, and the other three took 1-3 hours each. Once I got the hang of how I wanted to present stuff, it was just a matter of serializing it out of my head. In descending order of time put into the application (which happens to be inversely proportional to the amount of time I’ve already spent on the project): Joomla, Network Monitoring, libSieve, TWIG.

Amazingly I got through my midterms alive, got back an A on an essay, and am not going crazy! I am totally sick with a yucky head cold, though. Damn you, not enough sleep demon! And on an unrelated note, I installed WorkRave to keep me from typing too much at a time. It would really benefit from having information from XIDLE to keep track of how long I’m actually at the computer rather than just blindly counting down all the time.

Summer of Code 2006

2006-04-28T13:00:57-07:00

I’m planning on putting out a few proposals to get into the SoC 2006. I had hoped to get TWIG in as a mentor, but that didn’t go through. So I’ll take a tangent, and definitely see if there’s other good PHP work that can benefit everyone. My ideas are:

Portland State University: Open to “projects that might fall through the cracks” – I’m going to pitch XIDLE/XCB, TWIG and the network monitoring integration.

X.Org: See if they’ll go for my XIDLE request to get that working again. Otherwise, check out their XCB stuff. I’d like to learn it!

Joomla: wants to do a phpGACL layer, a DB abstraction layer and an AJAX layer. I’m good for all of those. I’m not sure if I want to be reporting to this project, although the work looks good.

Horde LDAP Browser / Manager: Develop a Horde LDAP application capable of both browsing and managing an LDAP directory. Should also support importing and exporting LDIF files. Now this would just be funny. Basically I’d port/rewrite my TWIG ldap browser into Horde. Or maybe funny-sad :-\

PostgreSQL: PL/PHP Build Improvments: PL/PHP is a PHP based procedural language for PostgreSQL. This project would clean up memory usage and the plphp_proc_desc struct, be built without Apache’s SAPI module, allow large resultsets to be processed through transparent use of prefetching on a cursor when spi_exec is called, and support IN/OUT and named parameters.

Proposal guidelines and example.

Asynchronous TWIG

2006-04-17T09:20:14-07:00

Last week I got an itch to learn something about AJAX, and procrastinate a little bit on some work. I started out by trying to find some useful springboard to get into AJAXland, but discovered that a) most of the PHP toolkits for AJAX are really big and foreboding, and b) JSON notation is the new preferred X format in AJAX. So we’ll call it Ajax and Jason, I guess.

After a while, I scoped out a few bare-bones tutorials and hacked together my own script in their fashion. Then I found a Jason converter for PHP that follows the semantics of what will probably be a pair of PHP built-ins, json_encode and json_decode. Simple as that. Pass in an array and it spits out the array in Jason, ready for consumption by your JavaScript. Well, about six hours of hacking later, I now have TWIG 4’s data tables slurping data asynchronously from Jason modules. Model-View separation suddenly became incredibly easy! In fact, when I copy-pasted code from the Wiki feature, the first to get its Jason upgrade, into the Mail feature, and tied it together with a simple public domain tree system, it pulled up my mailboxes but showed me the contents of the Wiki system! Wait, that’s good! It took just five keystrokes to switch the request over to grab data from Mail :-)

Looking for a new ThinkPad.

2006-03-28T10:55:53-08:00

Ok, here’s the new wishlist item: ThinkPad X60s

Cash in hand: $400. More to raise: $1400.

I wish I could go for something cheaper, but the battery life on this thing, especially with the extra battery pack, is just amazing. Two models ago, when new, I got an honest 8 hours. After a year and a half, it has died down to 5 hours. This new model has the same size batteries, and lower power consumption on the hard drive and CPU. The reviews have been awesome, just as they were with my poor old laptop :-|

Stop, Thief!

2006-03-20T15:47:19-08:00

My laptop was stolen at about 4am-ish this past Friday. I’d been pulling all-nighters at the dining room table for weeks, and I’d just turned in at about 2am. The thief let himself in one of the doors, headed for a housemate’s room, scoping it out with a flashlight and woke her up. At first she thought it was a housemate needing help or something, but then the thief bolted out of her room, grabbed whatever he could from the dining room, stuffed it in a backpack stolen from the living room, and left out the side door, taking a bike as a get-away vehicle. He also got two wallets and two sets of keys, but chose to steal a bike to get away. Major bummer! I’m starting a collection for the buy-a-new-laptop fund.

Sieve in DBMail: Check!

2006-03-08T02:27:31-08:00

DBMail now has Sieve script support, starting with DBMail 2.1.4 and libSieve 2.1.10. A marathon two weeks of integration, three releases of libSieve, and two releases of the DBMail 2.1 development series, and it’s together and working. And it was only on my todo list for two years. Probably the most interesting part of the experience was working on both sides of a library: maintaining the library and the downstream application consuming it. Even when it was tempting to Just Fix It (TM) when I came across a bug, most of the time these bugs were crashes caused by misunderstandings or garbage data flowing between the programs. Within a single program, you can just document that a function doesn’t like garbage, refactor the function and its consumer, whatever. But when you’ve got a library intended for consumption in its own right – and an almost frozen API – Garbage In Garbage Out (TM) is a less than ideal situation. The better approach is to handle the garbage elegantly and return useful error messages, like this: Error: you gave me garbage but all you’re going to get back is this lousy error message. And a t-shirt, I suppose ;-)

Job Hunt

2006-03-08T02:13:42-08:00

As of November 2006, I will be available for full-time employment. I am primarily interested in communications and email software development, focusing on backend network services and database storage. But anything fun and challenging will do the trick.

In the interim, I have taken on several small businesses and individuals as clients for network operations and office computer maintenance services on a monthly basis. I have found this small-scale work to fun and rewarding, building positive references and networking with potential dream-job employers.

Funding my open source habit

2005-12-05T09:30:00-08:00

I’ve been thinking that it might be time to start “development bounties” for all of my projects. I’ve gotten a few gift certificates, a hard drive, and some offers of cash, and I know that it really changes my prioritization when someone says, “Hey, I know you’re busy. Can I entice you to make time with a few bucks or some stuff?” And you know, it sure does change my priorities. I get right up off my butt and change my schedule. It’s a paying gig, and I think the more of them you do the more offers you’ll get.

If I could fund most of my computing and open source habit with bounties, I’d be pretty happy. A little back of the envelope shows that it could even be part of an ad-hoc career: a $50,000 a year salary is $1,000 a week. How many sub-projects are there out there that would take me about a week each, and be worth $1,000 to someone? Tons! How many could I land? Maybe not that many. Hmm.

DBMail progress

2005-11-30T15:21:56-08:00

This week I got a bunch of DBMail stuff done:

Loadable database modules. Then Paul reorganized the files into a single, consistent dbmail/modules directory, and that inspired me to to authentication and sort modularization. Still a few tweaks left to do on auth, and the guts of sort are totally not figured out yet.
Cleared the way for Sieve reintegration, handling username+mailbox@domain addressing, and better stderr debugging.
Lots of updates on the DBMail Wiki so that people can find out what’s cooking.

Back onto DBMail

2005-11-26T06:55:25-08:00

Been a while since I was last hacking actively on DBMail, but I’m back! Thanks much to a development bounty that’s motivating me, and thanks to generally clearing my schedule of some useless junk . I also rather ought to use this blog more. The two things I’ve been working on now for DBMail are:

Loadable modules for database drivers, because I'm sick of recompiling between MySQL and PostgreSQL.
Sieve support and integration, because it's been on the back burner for almost two years, and a fellow on the DBMail list offered me $1000 to finish it!

Saved by Foremost

2005-05-28T17:50:32-07:00

Many of the indirect blocks on the filesystem were hosed, so a big, big chunk of data cannot be organized back into a coherent chunk. sigh Making things worse is that there’s a filesystem-in-a-filesystem, because the backup image has an older backup image in it. So even with indirect block searches, there’s no way to know which filesystem’s block offsets you’re looking at. Foremost is the tool that scanned the data and recovered a goodly many of my pictures.

For the future, everyone should be using XFS, and regularly running xfs_fsr which comes from the xfsdump package. It reorganizes a file’s blocks so that they are contiguous. Apparently it’s a daily cron job on IRIX. Make it part of your life on Linux.

Salvaging a backup

2005-05-13T21:34:48-07:00

So I have this backup of a hard drive I sent in for RMA, and on that was a backup from my old laptop hard drive. Fortunately for me, the filesystem of the outer backup is corrupted to shit. I’ve been trying various combinations of e2retrieve, e2salvage, debugfs, and have had no luck. Another fellow on the e2salvage-devel mailing list referred me to this set of tools, e2extract. Wish me luck.

Crappity crap, homework crap

2005-05-10T12:10:48-07:00

Homework all shot to hell. Crap. Crap crap crap. Crap.

Finger’s all better, so I’ll make a Wishlist

2005-05-08T12:46:08-07:00

It took nearly 3 weeks, but my finger’s 100% back up to speed now, there’s just a bunch of redness around the wound site. I’m putting up a wishlist on my main page. I realized that there’s a bunch of tech stuff I’d like and keep re-researching everytime I think about it. Very silly. I’m making a list.

Nope, no books after all

2005-04-21T23:59:37-07:00

I didn’t end up reading any books. Instead I’ve been sleeping much more than usual, my finger is driving me up a wall, I’ve had non-stop meetings every night and days now, too, and that’s taking quite a toll. The hurry-up-and-wait of a 1 hour meeting is like 3 hours out of my day to prep for, participate in, and recover from each meeting.

I took the bandage off my finger for the evening to let it air out a bit and get some feeling back in my fingertip. It’s going to get bandaged up again before bedtime because it’s still basically an open wound. I suppose if I weren’t so tired, I’d be more interested in all of the layers of skin I get to see now. Bah.

Finger mauling

2005-04-16T11:44:31-07:00

I was going to go camping this weekend, but while packing up the trunk of the car discovered a wonderful razor edge on the decorative chrome trim of the 2004 Jetta. My right ring finger has a giant gash along the topside. Programming is probably not going to be happening this week at all, due to lack of curly-brace-button-pushing-fingers. Grumble. I may actually have to relax and read a book!

TWIG NG testing on the web

2005-04-15T10:51:14-07:00

My work on TWIG NG is now up on the TWIG website. The URL is secret for the time being, just in case my new security model breaks or I left any crufty code that would sink it. But it’s exciting to have it up, even if it is just for a few people right now!

On the DBMail side, looks like there’s a critical mass of bugs now. Time to get cracking on them and heading to another release…

Building a TWIG Roadmap

2005-04-06T16:47:38-07:00

And so from here we embark on a journey to the future of TWIG. I’m working on a roadmap that incorporates the various TWIG 3 proposals and renaming my own work at TWIG NG. All reasonable goals will become part of TWIG NG, and anything that can be done in a reasonable time frame will be moved into the TWIG 3 list. And so begins the who-wants-what shootout…

What the heck is this DocBook error?

2005-04-01T10:52:36-08:00

I DocBookified the TWIG manual, but I can’t compile it due to some strange errors that don’t appear to be coming from the DTD…

jade -t tex -V tex-backend 
        -d print.dsl 
        /usr/share/sgml/xml.dcl AdminManual.db
jade:AdminManual.db:3:57:E: URL not supported by this version
jade:print.dsl:1:0:E: end of document in prolog
jade:E: specification document does not have the DSSSL architecture as a base architecture
make: *** [AdminManual.tex] Error 1

Considering what will be TWIG 3

2005-03-23T10:47:48-08:00

So now there’s an alternate proposal for TWIG 3. It’s much less ambitious, and completely based on improvements from the current codebase rather than my work so far. Sort of annoying. But there’s an upside to this, and it’s that I may be able to convince everyone else of the dire need for good UI’s in TWIG for first-time install, setup, ongoing configuration, and first-time user logins. Frankly, if we can tackle all of that then I’ll be happy calling it TWIG 3.

Finals over, DocBook is on

2005-03-17T14:00:56-08:00

Finals this quarter weren’t horrible so much as because the tests were hard, but I just wasn’t in the state of mind for them. Total bummer. I’m wishing myself luck in passing classes. Otherwise. No otherwise.

On a more exciting note, I’m in the process of rewriting the TWIG Admin Manual into DocBook. Apparently it was originally written with MS FrontPage, then lots of editing by hand, then let to rot for too many versions. DocBook is a semantic markup format that can easily produce HTML, PDF and other formats with pretty looking layout. I really don’t want to spend more than a long afternoon on this; I’m not going to rewrite the actual manual right now, just convert the data over to DocBook, generate something good looking but still horridly out of date, and make it much easier to be updated in the future.

Whew, cleared those bugs up

2005-03-13T10:18:07-08:00

And thenceforth a fortnight later the bugs were done or passed along. Big relief. Can’t wait for the next releases! I need to kick some butt on my computer science homework this weekend. It’s almost mind numbing, because the instructor wants things done in just such a way, and worse yet, on tests, won’t accept any form of written explanation. Code or die. On paper. This must make sense to people who come to these classes with their own experience and knowledge in square one, but mine’s not. I hate coding real C on paper. Where they ding you for all sorts of stupid syntactic and even sylistic conventions. Dude, it’s paper. Not paper tape, not punchcards, but pencil and paper. I code in vim.

Lagging on my bugs

2005-02-18T05:51:38-08:00

I have two bugs on my plate for over a week, and I’m really lagging on them. Stupid other things to do. DBMail 2.0.3 needs a bug fix for users who are over quota. Right now it bounces the message with a temporary failure. This leads to the MTA trying to deliver again. Temporary failure. Again. Temporary failure. Eventually the queue gets pretty blocked up. Not so good. And TWIG 2.8.1 needs a fix for the FindHTML function that gets called to process the To, From, CC, BCC headers for possible HTML in email addresses. Better handled by the imap_rfc822_parse_address function provided by the imap c-client. On a 40,000 address test corpus I ran, imap_rfc822 took 1.6 seconds to parse those 40k addresses. FindHTML took an hour before I killed the process. Not so good.

So instead I’m working on some homework and distracting myself with the TWIG website. Two years ago I had the idea to write the TWIG website in TWIG. So I hacked up a version to do that. And today I had an epiphany for how I can do it a lot smarter from within the TWIG 3 framework I’ve began 8 months ago. Can’t wait to have time to get this really cooking!

Looking at WiClear again

2005-02-14T04:30:06-08:00

Wow, my first comment! Thanks to David, the author of WiClear for noticing me! Well, and that I was ripping on WiClear… or more specifically, the WikiRenderer engine used by WiClear.

It’s true that I didn’t do any sort of big comparison between DokuWiki and WiClear, but I noted that the syntax and engine of WiClear didn’t seem like much. David says otherwise, and that it’s all pretty customizable anyways. So, definitely worth a second look. The one toughie is that my lack of French makes it fairly difficult to dig into the websites for WiClear (mostly translated, though) or WikiRenderer (not translated much at all). When I have more time next week I’ll look into the WikiRenderer code and see if it’s worth jumping back to WiClear instead of DokuWiki. If it actually does have enough of the syntax needed to support the basics of my site, I can write whatever little bits I need to make the details work out.

CSS in TWIG

2005-02-11T14:50:40-08:00

So the raging debate on the TWIG mailing list seems to come down to this: Should CSS become an intrinsic part of our menu generation, or should there be “static HTML” menus and CSS menus and the user gets to choose. I say scrap the crufty full-HTML menus. They’re dinosaurs from HTML 3. We can describe them in lightweight, structural HTML and use CSS to make it pretty, and that’s that. The user should not be choosing to use an HTML 3 menu or a CSS menu, and then their choice of stylesheets that may or may not affect the HTML 3 menu and that will make the CSS menu look good. I believe it will lead to balkanized, useless styles that nobody wants to write because of the horrible interaction testing that would be needed. (quoting myself) I have no problem defining 7 types of menus:

Style Defined Primary Style Defined Secondary Style Defined Tabs Style Defined Vertical Style Defined Dropdown List Style Defined Icons with Text Style Defined Icons without Text

And throw the rest away.

DokuWiki and WiClear

2005-02-09T07:25:53-08:00

I’ve been searching out the perfect wiki engine lately, and while I have been lucky enough to have settled on DokuWiki because it’s simple, pretty and works… except that it’s really not all that pretty once you try putting anything other than technical documentation into it. I’m actually trying to run a whole website from within DokuWiki. It’s working very well, has reduced my webmaster duties a lot because I can hand out user accounts and don’t have to fiddle with repetitive HTML at all anymore… but it’s really pissing off the form-before-function people. So I’ve found WiClear which is really good looking but has a pretty crappy wiki engine inside.

And so off to add templating to DokuWiki… Bug #104.

Projects day

2005-02-06T11:53:13-08:00

Sunday is projects day! So to spurn the upsettedness over TWIG, I committed some more work on TWIG3 – it’s been almost 8 months since my last commit. Shoot. One day TWIG3 will happen and it’ll be awesome. Over on the DBMail front, I fixed a bug. Er, used someone else’s fix to yet another person’s well-researched bug. So basically I just typed a few things into CVS. But that’s my responsibility just as much as fixing bugs myself. Good work without too much hassle. A job well done :-) And on the co-op-front, I need to finish up some attic wiring before the garage-dwellers string me up by my ankles.

Back from the woodwork

2005-02-05T19:07:32-08:00

This week a former maintainer of TWIG came out of the woodwork to complain about some of the changes in TWIG 2.8.1. It was entirely out of the blue, and although he posted well to the mailing list and seemed to ask for input from the current maintainer group, he also dug up his old CVS account and started throwing things into CVS according to his preferences. I am more than a little bit dismayed. One simply does not “look up my old password” three years later and start changing things just because, three years ago, he used to run the project. Most of his code has been good, and had he at least had the courtesy to ask if anyone would mind him going straight to CVS I wouldn’t be so peeved. But he didn’t, and thus I am quite peeved.

Better now.

2005-01-25T17:20:40-08:00

Ok, it’s a little less ugly and more inline with my #069 color scheme. Still need to work on this grey text, though. It’s fairly annoying.

First Post

2005-01-25T16:51:27-08:00

I decided that it would be a good idea to start keeping a blog. First thing I notice, though, is that the default wordpress template is UGLY. So I’ll have to fix that.