Decoded Node

Plain Old Data

2012-01-26T21:00:00.001-05:00

I’m coming to the conclusion that there’s actually no such thing as “plain data;” it always has some metadata attached. If it doesn’t, it might be displayed incorrectly, and then a human needs to interfere to determine the correct metadata to apply to fix the problem. (Example: View → Character Encoding in Firefox.) Pushed to the extreme, even “just numbers” have metadata: they can be encoded as text, a binary integer/float (IEEE 754 or otherwise) of some size/endianness, or an ASN.1 encoding.

Another conclusion I’m reaching is that HTTP conflates all kinds of metadata. Coupled with the lack of self-contained metadata in file formats and filesystems, things start to accumulate hacks.

In my ramblings about REST and RPC, I mentioned that the combination of ETag/Last-Modified (a point-in-time) with a URL (an identity) provides a specific state of that identity. If an HTTP client wants to save that state to disk, how does it proceed? Typically, it saves the entity-body and discards the headers. Yet at least some of those headers specified important data like the ETag, Cache-Control, Content-Location, and optionally for text/* types, character encoding. (Other headers are specific to the transfer/message itself, such as Connection and the curiously named TE. These are concerned with layer 6 rather than layer 7.)

HTML, being in use world wide on the WWW, rapidly hit problems with locale, so it has long had the capability to use <meta http-equiv> tags for specifying the charset. (Amusingly, HTML 4 seems to define it as something the HTTP server is supposed to parse, and add to the actual response headers. If you need it after the transfer is done, it’s clearly not the transfer’s metadata, but the content’s.) A lot of formats have gone Unicode only, like Java source code, or had Unicode hacked into them at a later date to solve the depressingly familiar issues of character set.

The funny thing about specifying the character set in HTML is that you have to be able to interpret the characters in the first place in order to find the meta tag. Thus, more hacking has ensued: if you have an ASCII-compatible encoding, your only requirement is to put the meta tag in the first 1K of the document; if it’s Unicode, you can start it with a BOM. Otherwise, it must be specified through the HTTP header, or else it’s up to the user-agent to guess/use latin-1/do something else.

Meanwhile on Unix, everything is “just” a stream of bytes, including things that aren’t, and things that are more than that. On one hand, this made Unix programs much more adaptable to handling multiple, and new, encodings; on the other, it was a massive effort to make every program handle new encodings, since there was never much support for it (especially for variable-length encodings) baked into C, and for compatibility, it had to be opt-in. Also, when working on a remote system, the whole stack had to agree on a character encoding.

By pushing encoding issues up to the programs, Unix keeps knowledge of character encoding out of the kernel. Yet it doesn’t provide much to allow applications to keep the encoding of text—or any other metadata, really—attached to the bytes. Nobody relies on extended attributes (or file forks, or alternate streams), because they might be turned off, or a file might be transferred to a filesystem that doesn’t support them, such as the infamous FAT family. To my knowledge, there’s no “metadata channel” attached to IPC, either. In-band communication is the only reliable form.

This might be fairly sensible, though. Some formats can have multiple kinds of data that they can store. In particular, certain formats of ID3 tags come out on my pocket MP3 player as sequences of Hanzi and squares. It seems that the encoding of the tags has changed between versions, and the poor old thing gets horribly confused when it doesn’t have its matching revision of tags.

Still, most of the time when I’m dealing with “data” in a programming language, I’m keeping track of both the data and the format. Python still has some warts, like the encoding of sys.stdout goes missing when stdout is a pipe instead of a terminal, but it still seems to be the right approach, and is thus doomed to failure at the hands of dirtier alternatives. If you try to double-encode in Perl or PHP, that string is going to get double encoded, and you’ll have junk like Â€œ everywhere. Python will throw an exception (“bytes object has no encode method” in 3.x, and “could not decode character” when a non-ASCII byte was present in the string in 2.x). On the flip side, this makes dealing with damaged data trickier, because this “don’t compound common problems” attitude protects you from a simple double-decoding.

The other interesting thing about PHP not understanding character sets is that addslashes/stripslashes can mangle Big5 and Shift-JIS text. At one point, someone realized they could make magic_quotes cause a SQL injection, by using a double-byte character ending in 0x27. Adding the backslash changes the character, and moves the 0x27 out to become a real single-quote.

On the other hand, since PHP is so completely unaware of encodings, when you print a string and it comes out weird, you need hardly more than those bytes to find your problem. Perl’s backwards-compatible idea that the world is single-byte unless specified means that when you try to print Unicode data without configuring the proper IO encoding on a handle, it tries to convert to its legacy encoding and print that. If code points are out of range, then it prints all of them in UTF-8, with the lovely “wide character in print” warning. So it may appear to work, in spite of being completely broken, if you happened to choose data that won’t break it and have warnings off.

Perl’s functions work the same way; if you try to pack a UTF-8 string (with the utf8 flag on) into hex codes, hoping to see what the bytes are before they get to the filehandle, it doesn’t work—they go through the same encoding process. To have a fully Unicode pipeline in Perl or Python, you need to set up the appropriate encodings for every segment of the process before you can actually see the data that you have, so you can understand whether it is correct in the first place.

</tangent>

The point is, there’s no such thing as “just text,” or “just data.” MP3 may contain different text encodings in ID3 tags, the same is true of JPEG/EXIF comments, and MIME’s whole purpose in life is to stitch together randomly encoded things, including any of the above, into a parse-able whole. “Text” based, of course.

Looking at HTTP once more, there’s a whole stack of data to deal with: data about the message itself, where a 202 Accepted status implies that there will be no response body; about management of the ephemeral connection, with Connection and TE; about the response body when present, as in Content-Type and Last-Modified; and about the server, the common examples being Server and X-Powered-By. Some modern standards groups are rushing to add security and privacy as well, through the DNT (do not track) header and the various “do stuff across origins” specifications.

And in the end, the response body might carry something that has its own metadata, like a MIME message with its parts, or HTML document with HEAD tags. Formats which embraced “In-band communication is the only reliable form”.

Formats which don’t necessarily think all data has a self-evident format.

Irrelevant tl;dr: stalk me via twitter, or ye olde atom feed.

What If: Weak Memory Pages

2012-01-25T13:02:00.002-05:00

Raymond Chen wrote about the "what if everybody did this?" problem of applications written to consume up to some threshold of memory and free some of it under pressure: if multiple applications have different thresholds that they're trying to maintain, then the one with the smallest-free threshold wins. Of course, the extreme of this is a normal application that doesn't try to do anything fancy, which acts like it has a negative-infinity threshold. If it never adjusts its allocations in response to free memory, then it always wins.

Some of the solutions batted around in the comment thread involve using mmap() or other tricks to try to get the OS to manage the cache, but this brings up its own problems.

The obvious problem with mmap is that, under pressure, the OS will still treat the mapped data as precious, and write it back to the file. This is almost indistinguishable from swapping normal memory, but since we're using mmap(), this is a separate file from the actual swap partition. That means it may live on top of all sorts of other layers: the filesystem itself, a volume manager, software RAID, a transparently encrypted and/or compressed block device, and/or a block device pointing to network-attached storage. The final one is also a roundabout way of having a networked file system, while hiding the traditional signs of NFS from applications. In any case, the network may be invoked by accident. On traditional hard disks, separating the application cache's swap space from the actual swap partition may cost more in disk seeks when swapping actually happens. On a solid-state drive, the user may not actually want the cache swapped to it at all, to prolong the drive's longevity.

So that's a lot of points against mmap, which are all consequences of its swapping, since the application doesn't have a way to tell the OS that "this data isn't that valuable—you can discard dirty pages." What would happen if we added a system call for that, and extended the malloc family to request it?

Well, the first question is, "How do you know when the OS discarded the page?" If you just try to read the memory, you'll most likely generate a segfault, which you'll have to distinguish somehow as a weak-pointer... and then do something about it because the machine can't just keep executing. That's why it generated the fault in the first place! Java can get away with an exception, because the language intrinsically provides them, and the necessary stack unwinding control. C, on the other hand, tries to avoid having a stack visible in the language.

Alternatively, if reading a discarded page always returns 0, then you can't store data there that's legitimately zero. Otherwise, you can't distinguish a discarded page from your legitimate data. Even a struct: if you read individual members into the CPU, then those individual members cannot be zero, or you're opening yourself to weird time-of-check/time-of-use scenarios where the struct validated, but you got invalid data back by the time you tried to read your "real" target.

Not to mention, I'm unsure exactly how that zero read would be implemented--either "mapped-but-zero" has to be supported in the hardware's memory management, or the OS' fault handler has to go disassemble the instruction from the user IP and write a 0 into the appropriate place. Trying to do anything else, like deliver a signal, seems to run into the exact same signal-handling problems: how does the application set itself up to abort when the signal handler returns? Does it have to add checks before every write using data that was read from a weak page? How does anyone keep track of all that?

This also adds concerns to the system's memory manager. It has a new type of page to handle everywhere, along with a discarding policy. If pages can be discarded out of the middle of the discardable space, then discard-fragmentation can happen. Depending on how the application re-allocates weak space, the system may never be able to fill it again.

I feel like I'm writing Have You Ever Legalized Marijuana? but shorter and focused on only one example.

If all of the above are solved, there's still a pathological case where an application that discovers it lost some weak pages, re-populates them (as weak pages), and then tries to use them—only to discover they're gone already because of continuing memory pressure. This same problem occurs with ordinary weak data structures in Java, when memory is low. The application needs to know what data is precious enough to keep in ordinary memory, and what can be moved to a weak area. Moving data is easier in Java, because there aren't unrestricted pointers, so moving an object need not invoke copy constructors or invalidate any pointers.

All this, to drag down the entire performance of every application on the system, in order to provide support for a few apps to not-swap? It's probably not worth the effort.

I briefly considered "what if the OS could send low-memory notifications to applications?" but that still divides apps into two categories: ones that react, and ones that ignore the information. The latter ones will then win all the "excess" memory from the ones who play nicely.

Perl and Unicode in Brief

2012-01-18T10:12:00.000-05:00

If you want to handle Unicode and avoid The Unicode Bug, in which your strings sometimes act like they aren't actually Unicode: in perl 5.12+, use feature 'unicode_strings';. For older perl, see Unicode::Semantics, or use utf8::upgrade by hand.
If you want strings in your source text with non-ASCII: save it as a utf-8 encoded file and use utf8;. Or you can encode Unicode code points with hex-escapes, \xae → ®, or \x{30ab} → カ. There are technically other options, which have additional drawbacks (utf-16 breaks the #! line; latin-1 is restricted to latin-1 unless you decode it yourself.)
If you want to print to a UTF-8 aware environment like your terminal emulator or CGI STDOUT after issuing a Content-Type: text/html; charset=utf-8 header: setting UTF-8 on the filehandle with binmode(STDOUT, ':utf8') is the minimum, but :encoding(utf-8) instead of :utf8 makes stricter guarantees that real code points are coming out.
If you want to read a UTF-16 encoded document into a Unicode string: open(FH, '< :encoding(utf-16)', $name).
If you want to convert a Unicode string to a specific set of bytes for some encoding-unaware module to throw on the wire, use the encode function from the Encode module: use Encode; $message->attr('content-type.charset', 'utf-16'); $message->data(encode("UTF-16", $body));
If you want to read a file encoded with charset X, into a string encoded with charset Y, I've found no instant way to do this. It's probably best to pass the input-encoding along as the output-encoding if at all possible. But you might find from_to() or string-IO as in IO::File->new(\$out, '>:') or maybe a whole PerlIO filter as in PerlIO::code helpful if you can't.
If you see "Wide character in ..." warnings, then you passed a string with code points >=0x100 to something that expected a byte string of some sort: either really latin-1, or an encoded string.
If you see longer strings of gibberish where you expected sensible non-ASCII characters, then you have probably double-encoded, either literally, or by printing an encoded string to a filehandle which does encoding.
If you see the Unicode replacement character in a stream that should be UTF-8, you haven't encoded at all, such as printing a byte string on a raw filehandle in an environment expecting UTF-8. Most likely, the filehandle should have an encoding set on it, per point #3 above, though that may cause #8 on other strings you've printed.

The most difficult thing to come to terms with for me was, Perl doesn't have any notion of "the string's encoding" despite being Unicode-aware. It may be a Unicode string, which you could reasonably call decoded. Or, it's an encoded string, sometimes called a byte string--which happens to mean "a string of Unicode code points with values <= 0xFF." This gets especially complicated by the UTF-8 flag, which you are not supposed to care about, and shouldn't have to care about if you understand all of the above.

I would try to explain, but I've basically given up on trying to understand the UTF-8 flag for now.

A nice vim highlighting hack

2012-01-13T15:18:00.001-05:00

I wanted to highlight places where control flow could be redirected in my perl code, so I hacked up my personal colorscheme file to highlight Exceptions specifically:

hi Exception ctermfg=white ctermbg=blue

Now, I just needed to define the things I wanted highlighted as Exception*. Thus, the newly added ~/.vim/after/syntax/perl.vim:

" flow control highlighting
syn keyword perlStatementCtlExit return die croak confess last next redo
syn keyword perlStatementWarn    warn carp cluck
hi link perlStatementCtlExit Exception
hi link perlStatementWarn    Statement

" and i'm tired of everything being yellow
hi link perlStatementStorage Define

The last line isn't related to the above, but it recolors my/local/our in Preprocessor Blue instead of Statement Yellow. They do, after all, affect the state of the compiler at parse time.

* This means that I'm going to open a non-Perl file sometime and weird things will have Exception highlighting. Nobody notices the subtle differences when it's all Statement colored by default.

Layer 7 Routing: HTTP Ate the Internet

2012-01-11T22:58:00.000-05:00

In the beginning was TCP/IP, and the predominant model was that servers would listen for clients using a pre-established port number. Then came Sun RPC, in which RPC servers were established dynamically, and listened on semi-random ports (still, one port per service provided); the problem was solved by baking the port mapper into the protocol. The mapper listens on a pre-established port, and the client first connects there to inquire, "On what port shall I find service X?"

Then came HTTP, the layer 6 protocol masquerading as layer 7.

Doing One Thing

Shortly after I started this post, I ran across The Lost Art of Telnet, which explains how the Internet used to be a bunch of stuff, listening on different ports, each running their own simple text protocol at layer 7. gopher, telnet, MUDs and their friends, and IRC: all text based.

That was the way HTTP started, as well. To this day, you can often make a successful request with nothing more than the request line and an appropriate Host header.

Layer Six

The funny thing is that in all my classes that ever touched on the OSI Model, there was never any example for anything at layer 6. Of course, the world was IP by then, so we weren't staring at the implementation the model was designed for, but I think TLS (nee SSL) is the obvious candidate here.

And today, I'd extend HTTP down there as well, so that it would be spanning layers 6-7.

Wikipedia tells us that layer 6 included "capabilities such as converting an EBCDIC-coded text file to an ASCII-coded file...." This would be irrelevant, except that HTTP explicitly permits servers to translate texts, which I learned while looking for the correct JavaScript MIME type. In fact, Wikipedia lists MIME itself as a layer 6 protocol in the TCP/IP table.

HTTP's various encoding options, such as chunked and/or compressed encodings, also clearly pertain to layer 6.

Layer 7 is still clearly involved, though, since that's where entity bodies fall, as well as some of the metadata relating to them like Last-Modified.

I'm actually not sure what layer HTTP redirection would match up with. If the application allows transparent redirection, then it would appear as part of level 6, but most libraries allow the application to be informed and specify further action when a redirect code is received.

Routing and REST

Something happened to HTTP over time. The request-line itself provides almost no semantics, which means you can add your own. Either through verbs, the WebDAV/DeltaV way, or by tunneling your application through some other verb, Siri ACE style. (Or POST with _method=put for all you web browsers out there.) Although the spec says some operations are idempotent (have the same result if they are repeated multiple times), you don't have to respect that. You can make state-changing GET requests, for instance those /cgi-bin/counter.cgi images that were all the rage a decade ago.

I think I first noticed the extension of semantics to URLs in the days of Rails 1.1, when URLs were mapped into method calls, such as POST /bread/slice/3 which was handled by BreadController#slice. What happened was that the URL space was fully virtual, not just some filesystem view, and it could have application-level semantics associated with it.

User Space Networking

HTTP had become, essentially, user-space networking; in contrast to TCP/IP and the data link layers, which were all in the kernel and had comparatively limited configuration available. You just can't slap Perl into the TCP processing in the way you can put a PerlAuthenHandler into Apache with mod_perl.

The ability to perform any action on any URL and have this translated into some real effect was actually envisioned by the designers of HTTP itself. Many of the response codes, such as 202 Accepted, were created for this "HTTP as an interface to other systems" vision.

Firewall Evasion

There's another factor involved in the rise of HTTP as a near-universal transport, which is the predominance of the old paradigm of stateless firewalls filtering by TCP port. You could block ports 6667 and 119 and nobody would be able to waste time. Well, that was the theory. In those times, processing power and available memory didn't allow for much more advanced filtering than port numbers anyway.

In any case, the web turned out to be so important that it was usually unfiltered by these firewalls, so HTTP worked "everywhere" from the perspective of a client. Thus, RTMP, SOAP, and probably many other protocols have defined standards for communicating over HTTP.

The obvious result of allowing only "secure" communication in old times has been to make "insecure" communications over "secure" channels. Therefore, it's now required to run deep packet inspection tools to sort out whether a given HTTP message appears to be genuine HTTP or some tunneled protocol.

You could carry nearly any layer inside HTTP if you wanted. As a thought experiment, I just designed SOCKS over HTTP, which is amusingly absurd, but not much worse than an L2TP VPN. And if you wanted to cheat, there's always the Upgrade functionality.

Dumb Pipes are Still Dumb on a Smart Network

One last interesting bit of HTTP is the caching architecture. If you obey the standards for idempotent verbs and provide caching headers, then intermediaries can perform their own processing on your requests. The network is no longer a truly dumb pipe, shuttling data mindlessly between origin server and client.

However, the pipes between the nodes remain somewhat dumb. All they're dealing with are packets.

OSI Goes Fractal

Wrapping up, HTTP bears characteristics of layers 5 (through cookies), 6, and 7, and virtually any other layer except 1 could, in theory, be tunneled over HTTP. There are a lot of tunnels defined already for various levels, even IP-in-IP, and the obvious cases of VPNs.

Therefore HTTP is not "just" layer 7, and anything being carried at layer 7 may not be "just" application data: HTTP or SMTP may be acting as layer 6 for a SOAP message. You could technically be tunneling your Ethernet (and the TCP/IP it's carrying) inside those SOAP messages. There's really no end to the amount of insanity you can come up with by wrapping tunnels inside each other.

The disadvantage to such tunneling is that the ultimate application is affected by the performance characteristics of the tunnel, such as overhead. Or in the case of TCP, tunneling it over a reliable transport interferes with its heuristics and yields unreliable performance.

PHP: retrospective

2011-12-26T16:00:00.000-05:00

Once in a while, someone on Reddit asks for justification why everyone there hates PHP. I never reply, because there's too much to list in a comment, but maybe I can write a definitive post here.

I've put them in an ordered list more for your convenience in referring to the individual points, not because they're actually ordered. I have also limited this list to things that have annoyed me in particular, so you won't find a simple list of commonly derided PHP warts here. I knew about them early enough to dodge them.

Things that have made the language more clumsy to use:

E_NOTICE: the language defines a default value (null) for missing array keys, object properties, and variable names, then complains at you if you rely on this fact. Adding injury to insult, some versions of the engine fetch and format the entire string before passing it down to the layer that checks error_reporting to see if it needs to do anything.
E_ALL: that is not actually "all error levels". After adding this insanity to 5.0.0, I think E_ALL actually includes almost everything again, somewhere >= 5.3.0.
Write context: isset(some_func()) yields "Can't use function return value in write context" because, to suppress the E_NOTICE if it isn't set, isset generates opcodes to fetch the value for writing, so it has to be an lvalue. This is also why it's a special operator: so it can use the write context to incidentally find out whether the value was already set.
Callbacks: the callback pseudo-type accepts any of: a string (name of a function); an array of 2 strings (static call to class and method); an array of 1 object and 1 string (method call on the object instance); or an actual closure in PHP 5.3+. The first and last can be called using $callback("arg", "list"); but the OOP calls cannot. You have to pass them through call_user_func instead.
There's call_user_func_array because there's no *args style operator.
No keyword arguments: they have to be simulated with associative arrays. (It doesn't make sense to use the Builder pattern, because it's not statically checked anyway.)
Language operators disguised as functions: isset, list, and so on. I think you'd put func_get_args() here too even though some limitations have been lifted in recent PHP versions.
Iterators: many, many functions expect and only operate on arrays. You can't pass anything that implements Traversable to http_build_query. It fails on my machine (5.3.2-1ubuntu4.11 in Lucid) with "Corrupt member variable name", though in the past I do believe it would warn, then blithely do nothing.
array_key_exists: since isset() returns false if a value is defined-but-null, there's an extra function for finding if an array key itself exists, which doesn't work on objects implementing ArrayAccess because that only handles the special language operators.
Magic quotes: the first thing anyone does is writes an "input normalizer" that disenchants everything if magic_quotes is on. Even then, PHP isn't aware of character encoding, so addslashes() (that magic_quotes uses) once had a fix issued for causing the SQL injection it was meant to block in some character encodings.
Since short_tags and asp_tags are optional, environment-independent code always needs <?php echo ?> wrapped around variables, if it wants to pretend PHP is a "template" language.
htmlspecialchars: is just too long and full of options. me.inc.php therefore has function HE($s) { return htmlspecialchars($s, ENT_QUOTES, 'UTF-8'); }. And, it's not the right thing for encoding URI components. This is a Web language that provides no help in producing HTML.
You barely have support for HTTP. There's no shortcut for specifying "Let this cache for five minutes" nor "This expires on $DATE" nor offering a file download nor setting charset without having to specify the rest of the content type header.
1. There's no obvious way to specify 100 Continue headers. Your SAPI might require header("HTTP/1.0 410 Gone") even when you're handling an HTTP/1.1 request.
2. There's no way to handle Expect: 100-continue since mod_php doesn't have a way to invoke code between receiving the headers and receiving the request-body.
3. Likewise, the core team had to build upload progress monitoring themselves before you could do it portably.
The date function defines meaning for individual alphabetical letters, so a format like "at 13:14 on Dec 12, '11" has to be specified with '\a\t' and '\o\n'. In single quotes, so that "\t" and "\n" aren't tab and newline.
Class constants: can't be named dynamically.
Namespaces.
1. Using backslash instead of ::, or if that was really so impossible, :::.
2. The random restrictions. You can't actually import functions or variables, only namespaces and classes.
3. The resolution rules. You can't just throw new Exception anymore, unless you did use \Exception.
4. Constant names cannot be defined dynamically. You can define('MyNS\Foo') but you can only get it back using constant(), because those constants are an entirely separate thing.
PDO.
1. It caused a lot of segfaults in 5.1.x.
2. It doesn't have any conveniences like DBI's selectcol_arrayref or selectall_hashref($sql, 'id').
3. Using certain versions of PHP+libmysql, prepares could fail and return null, causing a fatal error when trying to call execute on it, and maybe segfaulting.
4. mysqli's bound parameter interface is even worse.
Tokenizer extension: gives you either array(T_TOKEN_TYPE, $literal_text) or a 1-character literal text string.
There's no parser/AST extension, opcode dumper, or debugger bundled with the language. Though we do have xdebug, at least.
The PCRE extension names its functions beginning preg_.
I'm always getting passthru/fpassthru/readfile, and shell_exec/exec/system mixed up. Usually, I need readfile or proc_open respectively, and the latter is still a pain to work with. (Especially if you want to launch something async, without invoking the shell, and also know if it actually died instead of launching OK.)

Philosophical issues:

Case-insensitivity: for function, class, and method names. But not for variables, properties, or constants. The Turkish problem was baked right in. For some versions of the engine, case wasn't preserved when reporting errors or returning class names from get_class(). Once it was, everyone's PHP4-compatible-instanceof checks needed peppered with strtolower everywhere.
Constants: define() runs at runtime, and takes its first argument as a string. PHP steals the semantic of "equals the constant name if undefined" in Perl, which seems to be related to why you can get "unexpected token T_STRING" out of foo("data " CONSTANT . " blah");. And why strings are T_ENCAPSED_STRING.
The "impossible" ability to parse $klass::foo(), also used as a justification against allowing "::" as the namespace separator, and which forced everyone to use call_user_func(array($klass, 'foo')), was implemented in PHP 5.3.
addslashes and stripslashes: they're backslashes, not slashes.
Parentheses: optional on 0-argument constructors, and nowhere else.
Dynamic class names to the new operator: from new $klass(...) to new parent;.
Constructor return values: I'm ashamed to admit I wrote something that had a use for $foo = parent::__construct(). (Of course, in the standard use of $f = new Foo(); the return value is discarded, because new returns the constructed instance.)
Functions that do things functions can't do: compact and extract. (Yes, these can be called through call_user_func.)
Variables that do things variables can't do: superglobals.
Until PHP 5.1.3, true/false/null were looked up at runtime using the standard constant mechanism. Though they are case-insensitive in spite of this.
It's almost shared-nothing, except for the file upload handlers and session extension, that usually comes configured out-of-the-box to produce the longest ID possible.

In other news, the things that PHP does exceptionally well:

SimpleXML.
The procedural PCRE interface. It's not Perl or Ruby, but at least it's not Python.
There is no three.

I think we're done here.

Loved this? Hated it? Blog about it and drop the link to @sapphirepaw_org on Twitter.

War Story: The Stock List

2011-12-16T23:04:00.002-05:00

A day like any other: In order to test that all the categories of products are behaving correctly on the website, I spend an hour writing a page to display a table of in-stock (further subdivided) and out-of-stock items.

About 6 business days after finishing, while waiting for review: instead of reading the entire history of every single Planet MySQL blog, I spend another half hour fancying up the CSS of my page. My boss catches me, asks what the page is about, rejects the hypothesis that testing is important, and lectures me. We are not making enough money to pay you your pathetic rate; do not do extra work.

Several business days later: the system is finally approved and live. Nobody in the office is trained on it when an order comes in. The order is for an out-of-stock item. The Big Boss is rather angry, and demands to know whether there is some way to find out "what the site thinks it has in stock." My boss answers "No." I am silent. I'm already looking for a new job.

Business Day 88 (about four months into the 90-day evaluation period): after 2 days and 2 emails, I finally get a meeting with the Big Boss to announce that I'm going to terminate my at-will employment after Day 89 to start my next job, 45 miles closer to home, at $pay * 1.38 + $benefits * 1.25. (I ultimately decide to tell him the exact offered salary, though I can't tell if he's BS'ing me on whether it's an acceptable/common question to ask, because I figure he won't match it. He doesn't even try to come up with a counteroffer.) He threatens that I might need to stay 2 weeks because he doesn't know if I can leave. The last project was finished somewhere around Day 76, and has been waiting for review. Every time I pinged my boss on a review, ever, including this final task, the answer was: "later today."

Day 89 was thankfully uneventful.

Observations on SPF: Sender Policy Framework

2011-12-13T19:36:00.001-05:00

Recently at work, I updated our SPF policy to something accurate. Along the way, to understand the policy I was deploying and what the previous version actually meant, I had to understand the various rules and types involved.

What's the difference between Sender ID and SPF?

As far as I can tell, Sender ID (Microsoft; Wikipedia) supports more ways to identify the sender: not just the SMTP MAIL command (what I am calling the "From address" below), but also message headers. They refer to the address that they decide to use as the Purported Responsible Address, PRA. Other than that, the overall design seems practically equivalent to SPF.

What are the differences among 'a', 'ptr', and 'mx' in the SPF record?

These are all SPF "mechanisms", also called "types", to avoid confusion with the DNS records they're named after.

A bare "a" indicates that a forward (DNS A record) lookup for the hostname in the From address must be done. If any resulting IP matches the IP of the sending machine, then the mail is permitted. The form of "a:mail.example.com" means the same thing, but the A record to be looked up is "mail.example.com" instead, not the From address.

The "mx" type is similar, except that the mail exchanger (MX) record of the From address is looked up, with the resulting domain name used. That is, a plain "mx" is equivalent to "a:mx.example.com" if the MX lookup returned mx.example.com. The form "mx:example.net" is similar, but instead of using the From address domain, it is the MX record of example.net that is looked up in the first step. I don't think the mx type has that much use in the real world; SPF is concerned with sending outbound email, while the MX records are for receiving inbound email.

The "ptr" type indicates that the sender's IP address should be looked up with a reverse (PTR) lookup. (To protect against spoofing, the PTR hostname is then looked up forward (A), and if the original IP doesn't come back out, the "ptr" mechanism is ignored.) Then, if the PTR hostname has the same suffix as the From address hostname, the mail is permitted. Again, the From address hostname can be overridden by giving a hostname to the ptr record, as in "ptr:example.com". This lets any machine with a reverse-address such as mail4.b.stl.example.com send mail (as *.example.com matches example.com).

Technically, when I said the mail is permitted, I mean "the mechanism matched"; since I didn't supply an action like "~a", the default is "+a", so the action taken because the mechanism matched is acceptance, more often called SPF Pass. (This post just reflects how I think about it inside my head.)

You said "A record," which is IPv4. This is the future—what about IPv6?

It turns out that SPF is defined intelligently; if the sender's connection is over IPv6, then all "A lookups" I mentioned above become AAAA lookups, and of course the PTR lookup also operates on the IPv6 specific domain.

An SPF policy can declare ANY unrelated domain to be a permitted sender?

Yes. The designation happens in the SPF record, which is initially looked up through the From address. Therefore, there's nothing that example.com can (legally) do to be specified as responsible for example.org's mail without cooperation, since they can't alter example.org's SPF record directly.

Can't spammers use SPF too? They already do fancy DNS tricks for their websites.

Yes, and I've seen it in the wild (a spam message with SPF permitting it.)

Logically, then, SPF's role must be to authenticate the sender of a message. Once the sender can be trusted (SPF accepts the message), the domain name can be used to look up its reputation. In older times, a spammer could send mail claiming to be From example.com, trading on example.com's reputation, and there was no way to uncover the deception. SPF should solve that last problem, but authentication alone doesn't guarantee non-spam email.

So what does your SPF record look like, sapphirepaw?

At work, we send out mail from one local server; from our hosted Exchange; and soon, from Amazon SES. The policy is therefore a simple chain of rules:

v=spf1 a:notifier.example.com ptr:xchg.example.net include:amazonses.com -all

The "a" type specifies our outbound server; the ptr is specified by our Exchange hosting company; the include is specified by Amazon SES; and we forbid the rest. For the road warrior, native Exchange, smtp+imap access, and OWA will all send via xchg.example.net, so we didn't need to make any special provisions there.

Names have been changed, except for Amazon SES, which is a public service with publicly available documentation. There didn't seem to be much point in hiding that.

At home, sapphirepaw.org has just one server, so it's dead simple:

v=spf1 a ~all

I haven't gotten around to thoroughly testing it yet, so it still soft-fails, but that's how it is so far. Both work and home records are published as TXT since the respective DNS managers don't support SPF types yet.

You can find more posts like this under the tip tag; or check explanation for longer, in-depth technical articles. You can also follow me on twitter.

War Story: power.sh

2011-12-08T20:54:00.001-05:00

When I attended community college, the computers in the labs were running Windows 95, pretty much in a state of constant hilarity. I'll get to that some other time, though; today's wacky hijinks are about The Server: the most secure machine on all of the campus, since it was the master authentication source.

The Server was running SunOS, if I do remember correctly. Though that makes it a tad more obsolete than the Windows 95 machines.

In any case, when logged into the Sun via good olde Telnet, we puny students were running in the shell's restricted mode. One of the goals of this process was to prevent us from having the Full Power of ~~UNIX~~ SunOS at our fingertips. I have long since forgotten the allowed commands, but there were only a handful of them, such that they used less than two lines when listed out. The only one of these that was technically needed in my time at that college was passwd.

Though it prevented writes to $PATH so that I couldn't add "allowed" commands, rbash didn't prevent reading from it. I noticed that ~/bin was part of the list, and of course we had write access to our home directories through some GUI on the Windows machines. (Whatever provided that access also allowed for setting permission bits.) I can't really remember how that all worked, but I quickly studied the Internet and put together a shell script in Notepad that would simply run an unrestricted shell. This was the ever-so-subtly named power.sh.

Then I discovered that Notepad writes "CRLF" line-endings, and the extra "CR" characters were preventing the script from running. I searched the Internet again; clearly, I needed dos2unix. But the code I found was implemented as another shell script that just called tr -d '\r'. Being rather ignorant, I went home and rewrote power.sh on my Linux machine there. I tested it at home, but it was still torture to wait for the next day in the lab to try it out.

It worked. I felt like the biggest genius on the whole campus, having clearly outsmarted the Guys Who Were Paid to Secure the Server.

But other than that, it was actually kind of anticlimactic. Being so ignorant, I didn't really have a use for the Full Power of ~~UNIX~~ SunOS myself. I probably would have explored it harder if I hadn't had Linux at home, though.

I also didn't bother to tell the administrators. According to the policy, merely attempting to circumvent any restriction for any purpose, including but not limited to reporting the vulnerability, was grounds for having your access revoked. And as a computer science major, that was just a senseless risk to take.

Need more pointless rambling? Follow me on twitter.

The Chariot

2011-12-02T21:49:00.000-05:00

I read "Linking in JSON" the other day. I knew someone had already gotten started on JSON Schema. (A quick search shows JSON namespace ideas floating around.) JSON as the lightweight alternative to XML being turned into XML? This is beginning to sound familiar.

With lightweight formats, we tend to get a proliferation of variants for different uses. (Not just images, naturally, and CPAN manages to use more than one.) Heavier formats tend to have a problem I'm going to call "Accessories Not Included": they get sufficiently large and complex that not all readers support all format options. If the growth is arrested early enough, you end up with a handful of profiles; if it gets out of hand, you have over ten of them.

I never expected XML to be so widely used for so much stuff, or spawn so many related specs. After all, it was verbose! And you could make up just any tag name you wanted! But it turns out to scale well from a not-quite-simple tree data structure, with annotated nodes, all the way up to unfashionable Enterprisey uses. But scalability bothers people, because who knows what wacky thing someone else on the team is going to foist on you, so more restricted alternatives rise to popularity. If this is true, the rise of Java should correlate to (non-game) companies getting burned enough times on C++. (If you think about Safety, it makes sense. He wants you to use Java, because he can't hack reversing the polarity of the template stack flow.)

This sort of thing is practically destined to keep happening. More features generally cost more memory and processing time, or some other inconvenience like a compilation step, which is against the religion of some developers. Thus, lightweight versions of things spring up in opposition to whatever is perceived to be too heavy. Sometimes compilation is considered the lightweight alternative, since it's not done on every request.

Though sometimes many similar projects proliferate because they just aren't that hard. It's easier to write a web framework than to learn one, so there are a lot of them.

Made it out of the link forest and need something more to kill the time? Maybe you want to subscribe to my feed, or follow me on twitter.

Resolution Dependence

2011-11-28T22:17:00.000-05:00

Why are device pixels so meaningful that we get stuck designing around pixels, even though we "know" we should design for device-independent units?

The main characteristic of a pixel is that it is crisp. When rendered on a display with 50% more PPI, a 1px line will be either thinner (physical size reduced) or antialiased (blurred). On the other hand, doubling the PPI lets those 1px lines render exactly as crisply, on precisely 2 physical pixels. (More crispness is possible, but Apple's version doesn't alter any art.)

If a user wants to zoom so that features are physically 50% larger, then the same problems of rendering 1-pixel features on 1.5px areas occur, but this time we know we can't tweak physical size. Antialiasing happens instead, resulting in a zoomed but blurry UI. Worse, subpixel rendering adds noise when not rendering precisely onto the intended subpixels, but the font rendering is done by the time the zooming layer gets to see it on Linux.

Unless everything is lovingly hinted and/or provided at multiple PPI steps, there's basically no solution to the problem. I'm willing to bet that people will skip properly handling multiple PPI settings if it's any more complicated than supporting power-of-2 sizes. As long as pixels matter, which they will up to 600 PPI or more, people are going to design for pixels.

REST and RPC: Not Actually Antonyms

2011-11-23T21:47:00.000-05:00

Last month found me writing a rant about REST and the shortcomings of interpreting it as "REST = HTTP+HATEOAS". I submerged myself into some writings of Fielding, and took some time for reflection, and I've found one of the sources of my problems with "REST". (Another.)

This problem is that there's too much writing on the web that attacks "RPC systems" as the logical opposite of REST, and I took this assumption unquestioned.

State Transfer

The very foundation of REST, the part that puts "ST" in "REST" and relegates the "RE" to a mere Latinate adjective, is state transfer. This is, as far as I can tell, the basic property that makes REST distinct from other styles, and also interesting. The whole thing about state management in REST is: do not keep hidden state; transfer it all.

You know, "shared nothing." Like NFSv2 or "modern" scalable architecture.

RPC Isn't About State

Wikipedia focuses on the definition of "RPC" as a means of invoking a service on some server in a manner indistinguishable at point-of-use from some local function call. It just happens to be a special function (the RPC stub) that packs up its arguments and sends them over the network.

Since the server doesn't have access to the process-local memory, the RPC call must also be a pure function: all of the state it needs to perform its action is carried in its arguments, and your program is not affected beyond receiving the return value. This is state transfer. Can it be called REST? It probably can, if the function is duck-typed—when any representation can conform to the interface and be accepted.

What about HATEOAS with our pure-functional RPC? As I understand it, the rationale for HATEOAS is that the server should be in control of its namespace. The systems that I am familiar with and labeled RPC are available on the network at a single URL, just like a "proper" RESTful server. Depending on the exact protocol, there may be a separate file specifying what methods, parameters, and return types are available. Providing a definition file allows the server to retain control of its namespace, as long as clients don't cache a stale definition for too long.

Of course, nothing prevents an RPC system from implementing its own definition method, or proxying individual methods to other servers on the backend.

But the Internet Says a "Method" Parameter is Definitely Not RESTful?

I know, but I don't think that RPC prevents a RESTful design from occurring. If the server can specify what methods it has, and also use that specification in results (such as listCustomers() being able to indicate "customer id 1, details at getCustomer(id=1)") to form a hyperlink, then it can be RESTful.

The implementation probably doesn't work out very well with RPC stubs in languages like C, where the function names to be called have to be fully known at compile time, well before talking to the server. But a language that allows for running code when a function name isn't found is perfectly capable of the trick.

However, most RPC systems probably don't have the hyperlink aspect to them, in which case they would indeed not be RESTful. This is what makes for the above rule: a "method" parameter generally indicates RPC, which is generally not RESTfully designed, therefore the method parameter is strongly correlated with non-REST architecture. That correlation is simply not a guarantee.

Batch Operations

Most of the RESTful API design advice I can find seems to be centered around basic CRUD, using the familiar GET, POST, PUT, and DELETE verbs to act on a single object in the data store at a time: DELETE /users/1 If-Match: SomeETagValue and so forth.

Actually, I lied. Most of the design advice doesn't even cover If-Match. However, they do ignore searching, as well as anything that you would approach with a transaction in the average non-RESTful system. I ran across a comment (#22 at this page) by Fielding on the batch operation matter:

People perceive a need for batch operations because they don’t understand the scope of resources.

Resources are not storage items (or, at least, they aren’t always equivalent to some storage item on the back-end). The same resource state can be overlayed by multiple resources, just as an XML document can be represented as a sequence of bytes or a tree of individually addressable nodes. Likewise, a single resource can be the equivalent of a database stored procedure, with the power to abstract state changes over any number of storage items.

If you find yourself in need of a batch operation, then most likely you just haven’t defined enough resources.

Emphasis mine.

It seems that being equal to a stored procedure is simply another implication of state transfer: all the necessary state to perform the operation is sent in a single request. HTTP support like If-Match offers the ability to use the server in a lock-free style. Request some bits, do work, and send updated bits with some sort of version indication so that the server can tell whether they're out of date. If they are, the server returns 409/Conflict and the client tries again.

URLs as Identities

If you're not familiar with Clojure: identities are a series of states (actual stored data) over time. These are very similar to variables, but Clojure offers identities as specific kinds of variables, each with unique semantics for updating them. A value is simply the state of an identity at some point in time. Conceptually, it's a copy, so that once you have the value, it doesn't change, even if the state of the identity is updated.

If we access /users/3 in our RESTful API, we expect to get the most recent version of user 3. Since the data at this URL can change over time, it's clearly not a value, so the URL must be an identity instead. The URL+ETag together form a state, and associated data is the value.

The lock-free style must identify a specific state, so that the Conflict error can be generated if the state operated on doesn't match the current state. That's what atomic compare-and-swap instructions offer in client-side lock-free programming, and what If-Match provides when we PUT /users/3; but how does it carry over to batch operations, where the resource may need to identify several entities' states?

Apparently, the URLs need to include the ETag to indicate the state rather than the identity.

Refining REST

Now that I defined state as the value of the entity at a URL at some point in time, we can use that definition to get a deeper understanding of state transfer. In its basic form, when we PUT /users/3, we have both an identity (the request URL) and value; the request itself is asking to associate the two, as of the current time. Once time is involved, we can say there's a state: the PUT request is defining a new state of the identity.

Therefore, "state transfer" means that we're sending a complete state along the wire, either literally or by reference through URIs to other states. This is how we can have idempotent requests and caching: if a request is conditional upon a state, has no effect when the condition does not hold, and changes the condition when successful, then duplicate requests will fail if any others succeed. An RPC-based system may or may not make this guarantee. Systems implemented over HTTP must follow the guarantees that HTTP defines for its methods, or else use POST for everything, which is an anti-pattern in its own right.

SOAP and WSDL Don't Guarantee RPC or a Non-RESTful Architecture

I want to wrap this up with an example. QuickBooks offers an SDK, which allows for scripting the QuickBooks GUI. (Literally so: you cannot interact with company X if the GUI is logged into company Y.) They also provide a QuickBooks Web Connector (QBWC), which connects to a server from the client machine and makes requests. The server (eventually) responds with chunks of qbXML that represent QuickBooks operations, which the web connector passes to QuickBooks on the server's behalf. Finally, QBWC gets another qbXML document from QuickBooks in response, which it returns to the server.

This server is actually a web service, available over SOAP, using the WSDL that Intuit provides with the SDK. However, HTTP and SOAP are simply building a tunnel for the exchange of qbXML, and the design of those files is RESTful.

To build an audit log and/or prevent conflicting changes in multi-user mode, QuickBooks can only allow the most-recent state to be updated. Therefore, every object in QuickBooks has both a unique identity and state: ListID and EditSequence. Writes MUST provide an EditSequence, and the EditSequence value MUST match; otherwise, a conflict has occurred, and the write is denied. This rule applies to all calls, user-initiated and SDK-generated, and therefore naturally applies to QBWC as well.

Since the server's only point of contact with QuickBooks is qbXML, and it doesn't support constructing a change over the course of multiple calls, all of the data to complete a request must also be included in a single qbXML request. Once again, this is a signature of REST.

As far as I can tell, the stack involves SOAP and WSDL to provide a guide to the interface between the client and server. (Even so, the WSDL indicates certain methods are optional, but QBWC will not proceed if they are missing. I think there's a grumpy comment in our source about that, regarding clientVersion and serverVersion.) In other words, the stack tunnels over SOAP and WSDL, and uses XML as its resource format, because things like WSDL-to-code generators and XSD schema validators exist.

Using SOAP and WSDL means that Intuit didn't have to create any sort of WSDL equivalent for services exposed directly on HTTP, nor do people implementing their SOAP service have to worry about URL routing. Implementors can put their service at any URL, and that's all that the tunnel needs to work. Using this to build the layer that RESTful qbXML travels on butters the bread on both sides: Intuit gets the benefit of WS-* and XML tooling for setting up the requests, and also the benefits of REST when processing them.

Closing Remarks

Fielding has also remarked that it's not possible to determine whether an application is using an interface in the REST style by scanning its network traffic. If the server provides URLs, but the client ignores them and requests hard-coded URLs that happened to be in the response, then the network traffic is identical, but the decision hasn't been made by actually chasing the hypertext pointers.

It's important to remember that REST is a style. It's not about integrating with HTTP. It's not about "not being RPC" or "not being SOAP". If a service were implemented as EXI sent over SCTP just to be exotic, the result could still be RESTful.

I am now of the opinion that similar things befell the terms "object oriented" and "REST", at least if we take Smalltalk as 'defining' OO: some important, useful, and somewhat mind-bending concept went mainstream in the programming world, and the difficult parts were lost in translation. This makes it rather annoying to come along later and learn about it, because it seems silly or half-baked instead of deeply insightful, and I begin to wonder why everyone is so enamored with it. At least until I find the True Meaning.™

Now get out there and rock.

Smart TV and Split Attention

2011-11-18T22:58:00.000-05:00

I don't want to waste much effort on trivialities, but regarding Gruber's semi-recent post:

Imagine watching a baseball game on a TV where ESPN is a smart app, not a dumb channel. When you’re watching a game, you could tell the TV to show you the career statistics for the current batter. You could ask the HBO app which other movies this actress has been in. Point is: it’d be better for both viewers and the networks if a TV “channel” were an interactive app rather than a mere single stream of video.

This is not actually a universal good for viewers. They'll probably like it and want it, but if there's one thing I have learned about myself, it is simply:

Splitting my attention between things means I don't remember either thing.

Worse, the things that a smart channel offers me in Gruber's vision—the things actually related to the show I'm watching—are useless trivialities. If I had smart TV and lacked the discipline to avoid these side quests, then I wouldn't gain anything out of my screen time. I'd forget the answers to the fleeting distractions, and also not be able to remember what I was watching in the first place.

I can say all this because I already know what the price of distraction is. I refuse to pick up my iPod while watching things, no matter how interesting it seems at the time, because I'd rather focus on the show or movie.

What makes me happy? It's not the Internet; it's not TV; it's not apps; it won't be all three of them rolled together into smart TV. However, a smart TV done well will still be a success in the market. We'll find out sooner or later whether Apple did it well. It's almost certain that they'll try.

Programming Languages to Learn

2011-11-16T18:25:00.001-05:00

Many languages these days are fairly Lispy, except for being homoiconic and thus having a full-strength macro system instead of C's token pasting or many other languages' nothing.

But which ones are absolutely vital to learn, and which ones are "just different languages"?

Ruby

Ruby is an incredibly thoughtful language, and using it, you can't help but think about the decisions Matz made, and the ones you're making. Coming to Ruby after having seen Perl, the inspiration is clear, but Ruby has been distilled. You can probably make Perl do everything Ruby does, but in Perl you have to work for it. In Ruby, it's frictionless.

Ruby also has regular expression syntax in the language. This power will corrupt you, and you'll wish for it every time you need to run regular expressions in any other language (except Perl, which also has it). But it's also worth it.

Clojure

It's just as thoughtful of a language as Ruby, but it's also on the JVM, and its Java integration is so good that it's a better Java than the Java language itself.

It's also a Lisp that isn't Common Lisp or Scheme. This frees it from a lot of historical decisions, which makes the language much more focused on being a modern functional language. It keeps all the benefits of Lisp, but also brings in immutability and leverages that for its concurrency models. (Because you don't always want the same kind of concurrency.)

In short, Clojure should expand your mind just as much as any other Lisp, and then some, all without the archaic names and relatively poor suitability for modern work. You don't need a well-stocked asdf to have a basic, portable environment set up. Since Clojure is, at the moment, just one canonical implementation, you don't even have to worry that hard about portability, unless you want to write for ClojureScript. (But for learning Clojure itself, you probably don't.)

C

I debated putting this one on the list, but I think it's still vital simply for the fact that it offers raw, unfettered pointers. Once you grasp pointers vs. values, you have the foundation to understand pointers in any other language. Though they won't be called pointers, since those got a bad rap for being "hard" because C crashes when you make a mistake. Which you do a lot, when you're learning.

C style syntax also forms the basis of several other languages, and it's still a sort of lingua franca, though that role has been moving to Java. (Which tells you something: Java is the least mind-bending language around, if it's believed to be easiest to understand for the most people.)

Beyond the Gate

There are probably other interesting languages to learn, but I would have a hard time calling any of them out as vital. Clojure and Ruby pack a lot of attention to design in their languages, and that shows through in everyday programming. At the same time, though, I feel like there's something deep that I'm missing about the Forth family.

Therefore, I don't think this post can be the definitive list for all time; it's incomplete because I haven't learned every language, and it's not a static list because languages and the practice of programming itself will change over time. But for now, today, this is my list.

Go forth and program.

Private Streaming with CloudFront: A Guide

2011-11-15T22:00:00.000-05:00

I'll just assume you're aware of the IaaS offering known as Amazon Web Services, AWS. CloudFront is a CDN in the AWS micropayments-as-you-go style, which offers the ability to serve non-public content stored in S3. This is a compendium of the things I learned setting up a private streaming distribution for use with PHP.

This is going to be fairly low-level, since I like to drink deeply of the systems I'm working with. I don't think AWS works smoothly enough yet that you can put the API on the "it's magic" side of the line.

Don't touch that yet: Making S3 CloudFront friendly

CloudFront will request content from S3 using the virtual-hosting style, your-bucket-name.s3.amazonaws.com, so you can only create a CloudFront distribution for a bucket that has a "DNS safe" name. For me, that meant going back and renaming the bucket since I didn't want to upload another 1G of files.

If you're going to serve your content with TLS (nee SSL) enabled, over https, you need an ssl-safe name: one without dots. Amazon has a wildcard certificate for *.s3.amazonaws.com which will not match bucket.example.com.s3.amazonaws.com, since the * in SSL only applies to one level of name. Replacing the dots with dashes, bucket-example-com would work fine.

Once you have a suitable bucket name ready for CloudFront, you can proceed with creating your distribution.

Creating or Configuring a Distribution

The CloudFront web console (the AWS management console's CloudFront tab) is particularly lacking in support for private distributions at the moment. If I were doing things again, I'd probably skip creating the distribution in the console, and hack up a script to do everything that way. The lack of support means that you end up having to call the API somehow, regardless.

At any rate, whether you're creating the distribution from whole cloth in a script or re-configuring one created via the console, there are three important settings to engage, along with one dependency. That dependency is to create an origin access identity (OAI). Then, you can associate the OAI to the account and to the origin, and set yourself as trusted signer. At this point, the distribution is fully private, and you can grant access to the OAI to your S3 objects, and move on to signing your URLs.

I'm working in PHP as usual, so I grabbed the AWS SDK for PHP via Amazon's pear channel: pear channel-discover pear.amazonwebservices.com ; pear install pear.amazonwebservices.com/sdk

This lets you include('AWSSDKforPHP/sdk.php'); in your PHP script, and start using calls as documented here. The first thing we need to call is create_oai which will be one of our dependencies. Save yourself some time, and print every response you get. In particular, you'll need the Id and S3CanonicalUser from the create_oai response later. Otherwise, you'll have to call get_oai_list (which returns array rather than CFResponse) and get_oai to find what you missed.

Creating a Distribution from Scratch (Recommended)

If you have created a configuration through generate_config_xml, or a distribution through create_distribution which internally calls generate_config_xml, then S3Origin will properly receive the OriginAccessIdentity child, and you can skip the following section. All you need to do is pass 'OriginAccessIdentity' => $oai_id, 'TrustedSigners' => array('Self'), 'Streaming' => true in the options, and everything will work.

Updating the Config of an Existing Distribution

If you've created the distribution already, you can get its ID by looking at its properties in the console, or through list_distributions with the Streaming option set. Next, call get_distribution_config with the Streaming option to fetch the distribution's configuration. You can pass this response straight to update_config_xml; again, with the Streaming option set, but also 'OriginAccessIdentity' => $oai_id, 'TrustedSigners' => array('Self'). Now here's the part that tripped me up for almost three days: the XML that comes back does not include an OriginAccessIdentity element as a child of S3Origin. This seems to apply to 1.4.7 as well as the 1.4.6 version of the PHP SDK I was using. The distribution is private (will require signed URLs) because the OriginAccessIdentity is present in the distribution config, but you need to add the OriginAccessIdentity under S3Origin by hand.

For me, it was something like this:

$sxml = simplexml_load_string($new_cfg);
$sxml->registerXPathNamespace("cf",
    "http://cloudfront.amazonaws.com/doc/2010-11-01/");
foreach ($sxml->xpath('//cf:S3Origin') as $node) {
    $node->addChild('OriginAccessIdentity',
        "origin-access-identity/cloudfront/$oai_id");
}
$new_cfg = $sxml->asXML();

The call to registerXPathNamespace tells SimpleXML that we want to access items in Amazon's namespace with a prefix of "cf", then we use that in the XPath expression to find the S3Origin node, to which we add our OAI. Without this, CloudFront won't try to actually use the OAI to access S3 content, and will therefore be limited to public files.

At this point, you have something suitable for set_distribution_config, with the Streaming option, of course.

Letting CloudFront Access S3

I didn't want to set every video as accessible to the CloudFront OAI individually, so I set up a bucket policy instead. Bucket policies let you grant/revoke permissions for all objects in the bucket, overriding the individual object ACLs.

However, they are not terribly easy to write. I took the example from the S3 documentation Access Control → Using Bucket Policies → Example Cases.... The CanonicalUser to use can be found via get_oai's S3CanonicalUser element, if you didn't record it after calling create_oai.

After substituting your S3CanonicalUser and bucket name in the appropriate spots in the policy, it needs to be set as the actual bucket policy. To do this at the console: click the bucket, choose Actions → Properties from the button above, and look for the "Edit bucket policy" button on the Permissions tab. Paste and save your policy, and you should be set.

Signing URLs

CloudFront does request authentication differently than other AWS services, so you need to create an RSA keypair for it in your AWS account. From the console, click your name way up in the upper right, above the tabs, and choose Security Credentials. Log in again, and scroll down a bit, and you'll see a tab for Key Pairs. This is where you generate your signing key. You'll need to save the private key and keep it on your server for creating the signatures.

If you search the web hard enough, you can find some URL signing code from RyanP@AWS. I played with this for a while, but for streaming distributions it gives you a certainly-wrong URL, and I never did test whether it actually worked once I understood how to sign URLs. (I suspect it doesn't.) The URL signer that was actually helpful was the one that is included in the CloudFront Developer's Guide under "Using a Signed URL.... → Signature Code, Examples, and Tools → Create ... using PHP".

Whatever you're using to sign your URLs with, test it with the private key and example policies from the developer's guide to make sure you get the same output. I had to go back and undo some of my "improvements" to my signing script. For one, the canned policy is not necessarily the JSON which would be output by a json encoder: slashes in the resource URL are not backslash-escaped. Secondly, I assembled some parameters through http_build_query, which turned the '~' characters in the Signature field into %7E, and CloudFront didn't care for that either.

Since we're streaming, using RTMP, there are two URLs involved in the process. There's a streamer at syourhostname.cloudfront.net/cfx/st which is the "streamer" in jwPlayer parlance, or "netConnectionURL" in flowplayer. This is where Flash connects to access the stream; it is like a server in its own right, that takes another "file" or "URL" (again, in jwPlayer and flowplayer terms, respectively) to determine what content should be streamed over that connection.

The streamer does not participate in the URL signing process at all. It is only the content to be streamed that is signed. Therefore, the URL that actually gets signed is the full path of your file, with the extension, within your S3 bucket. If you have video-example-com as your bucket, then video-example-com.s3.amazonaws.com/usa/managing-payments.flv will use usa/managing-payments.flv as the resource to sign.

The resulting signature then gets attached to the stream name as Flash prefers it, flv:usa/managing-payments?Expires=.... jwPlayer seems to work with the URL as-is, when invoking it through its jwplayer JavaScript function, but flowplayer needs the signed URL to be URL-encoded, except for the colon. That is, my final URL-building for flowplayer takes the shape: $file_url = 'flv:' . urlencode("folder/file?$sign_params"); I did not need any more encoding of slashes or spaces in the folder/file names.

I can't really say more about jwPlayer here, because once I got the OriginAccessIdentity associated to the S3Origin and could access my files in either player, I went back to using flowplayer.

Troubleshooting

If you've just made changes, wait a bit and try again. S3 ACLs don't seem to apply instantly, so bucket policies probably don't either, and Amazon notes somewhere that it can take up to 15 minutes for a CloudFront distribution to be updated after an edit is made.

If you're still having problems after waiting it out, turn on access logging for both CloudFront and the S3 bucket. That's what ultimately directed me toward my problem, that CloudFront was not using its OAI to access the files in S3.

Access logging of an S3 bucket uses the "prefix" parameter as-is: if you set it to 's3-videos-' you'll get files in the root of the access logging bucket, with names beginning 's3-videos-'. The CloudFront console's Edit Distribution screen seems to suggest the same, but it actually uses it as a folder name. I gave it a prefix of 'cf-' and all my CloudFront logs ended up in 'cf-/'.

The access log files aren't available instantly, but in time, many very short files will show up. This is where a tool like s3sync or Cloudberry Explorer comes in handy, to fetch them so you can see them without a three-click download process between each one. CloudFront's logs are gzip-compressed, but S3's are just text.

Incidentally, you can edit a distribution in CloudFront without affecting its private status. The console only changes the properties that it supports.

I can't really endorse any particular app for accessing or managing CloudFront distributions, because of the ones that understood streaming distributions, they all seemed blissfully unaware of my misconfiguration.

Conclusion

To compress all of the above into an "If I were doing this again from scratch" set of steps:

Create a DNS- and possibly SSL-safe S3 bucket.
Upload your videos into it.
Create a CloudFront Origin Access Identity, through the API. Record the OAI's Id and S3CanonicalUser values.
Create a CloudFront Distribution through the API, using the OAI Id from the previous step, TrustedSigners=[Self], and Streaming=true to make it a private streaming distribution.
Create a bucket policy on the S3 bucket, using the S3CanonicalUser from the OAI, granting it GetObject on all items in the bucket.
Create an RSA keypair in the Security Credentials meta-console. (For lack of a better term.)
Save the private key to your server.
Fetch the example code from the CloudFront Developer's Guide to sign your URLs.
Use the signing code to sign URLs for "path/file.flv".
Use "flv:path/file?Expires=...&Signature=...&Key-Pair-Id=..." as the file URL in your player.
Use the http or https://syourhostname.cloudfront.net/cfx/st URL as the streamer URL in your player.

At this point, everything should be working. If not, Amazon allows themselves 15 minutes for changes to CloudFront to propagate, so take a little break and try again.

PHP's hash(): how tiger192,3 and tiger192,4 differ

2011-11-05T18:55:00.001-04:00

PHP lists a handful of hash algorithms for Tiger:

tiger128,3 tiger160,3 tiger192,3
tiger128,4 tiger160,4 tiger192,4

What's the difference? Which one is standard? Is one harder to break? Why don't any of the outputs match either of Wikipedia's examples?

The first number—128, 160, or 192—is easy. It's the number of bits of output, so the 128/160/192 variants will return 16/20/24 bytes of output. By default, it's in ASCII format, so the actual strings are 32/40/48 characters. The shorter versions are simply the first bytes of the longer output, so any of the tiger___,3 functions will output a hash starting "24f0130c63ac93321616" when hashing the empty string.

I had to dip into the source code to understand the second number. It is the number of passes that PHP performs when computing the hash. The compression function (the compress macro) is set up to perform either 3 or 4 passes, with a key scheduling between each. The code is slightly tricky because they store the distinction between 3 and 4 passes in 1 bit of the hash context, as the number of additional passes to perform.

The algorithms ending in ",3" are the standard Tiger algorithm; their output for the empty string matches the empty string's hash in the testtiger code from the Tiger website. But the output is not in MSB-first order; see the note about Wikipedia below. (The bug has been filed with PHP.)

The algorithms ending in ",4" are non-standard, and should perform 33% slower because there's 33% more work being done. As I understand it, this extends Tiger to 32 rounds (from 24), which should make it significantly harder to break. Disclaimer: I'm not a crypto expert, so you should consult with experts if you want accurate, up-to-date information on the security of your specific application. Ideally, you'd be using a well-vetted library that already made a secure choice.

For the last question, Wikipedia prints the output MSB-first, but testtiger and PHP don't. The testtiger code prints each 64-bit chunk in byte-swapped order: so 3293...f024 in the output is printed as 24f0...9332. Therefore, although PHP's 3-pass algorithm is standard Tiger, its hashes won't match any software that chose the MSB-first representation instead. (For instance, the Digest::Tiger NOTE for the Perl implementation explains that it returns hashes MSB-first.) In that case, one of them needs to be reordered before the match is attempted.

Renaming buckets on S3

2011-11-01T21:36:00.000-04:00

A technical note, since the search engines of the internet don't seem to have noticed: Amazon's S3 management console lets you cut and paste files now (including whole folders). So the process to "rename" an S3 bucket is simply:

Create the new bucket with the desired name.
Go to the old bucket and select all files: click the first and then shift+click the last.
Above the file listing, in the button row, is one marked "Actions", which opens a menu that includes "Cut" and "Copy". Pick one.
Go to the new bucket, click Actions, and Paste your files.

Done. No 3rd-party software required.

Why would anyone want to rename a bucket? In our case, we created a StudlyCapsStyle bucket, which can't be used with CloudFront's dns-compatible-style.

In double-checking this post for accuracy, I noticed that Cut/Copy are available on the right-click menu for a single selection, but not the multi-select. Weird.

Character Sets: Get PHP, Perl, MySQL, and Unicode to Play Together

2011-10-25T20:59:00.000-04:00

This is an extended remix of my recent post on the subject, only less of a rambling story and more focused. Again, I'll start with some background definitions.

I'll also assume that you're going to make everything UTF-8, because as a US-centric American who has the luxury of using English, that's what makes the most sense for my systems. However, if you understand everything I wrote, it should not be difficult to make everything UTF-16 or any other encoding you desire.

Terminology and History

First, character sets. A character set defines a set of characters and numbers for representing them. For instance, ASCII defines 0-127, including the Latin alphabet necessary for US English. So, "A" is 65 and "a" is 97 in ASCII. After this was defined, the 8-bit byte was standardized and computers went overseas. ASCII wasn't suitable, so various locales took advantage of the 8th bit to add 127 more characters for their languages. The most common of these in the Western world are the ISO-8859 family, which gave us latin-1, latin-7, and so forth. These sorts of character sets are known as "8-bit" because they define 8 bits worth of space.

Meanwhile in Asia, they had character encodings like Big5 that let them use far more than 256 codes for their scripts that had far more than 256 symbols, and also embed ASCII for quoting English text. However, Big5 did not fully specify what the embedded character set was.

Looking at this mess, Unicode was born, to define a single character set, once and for all. Ideally, everything would be Unicode everywhere, and any user could see any text and be relieved of figuring out whether the document was (for example) CP1251 or KOI8-R. The first encoding of Unicode, UCS-2, used 16 bits but was otherwise largely identical to other systems in that the character codes (what Unicode called code points) and output bytes were identical. UCS-2 proved to be too small, yet was considered bloated by US and Western European users who had mostly-ASCII text, since UCS-2 took twice the space for most of their characters.

UTF-8 solved this problem by dividing bytes into segments that contained information about the byte stream (such as "Part 1 of 3" or "Subsequent part") and "character data". With a bit of cleverness I'll skip here, ASCII remains as-is, and higher characters are represented with a multi-byte sequence. Thus, 0xceb1 is the specific UTF-8 encoding of the character 0x03b1 (greek small letter alpha).

In the US, we're lazy and still call UTF-8 a character set. This is just a historical artifact of us not using Big5 or some other encoding that can have different character sets embedded in it, and it doesn't end up being too confusing since UTF-8 can contain only one character set, Unicode.

Symptoms of Character Encoding Problems

There are two obvious symptoms when trying to output UTF-8. First is the double encoding problem, where you see things like "â€™" where there should be only one special character. This typically indicates that UTF-8 encoded data was interpreted as if it were an 8-bit encoding, and converted to UTF-8. Each UTF-8 byte has become a whole UTF-8 character.

The opposite symptom is a text littered with �, the "replacement character", instead of your special characters. This usually means some 8-bit encoding is actually being interpreted as if it were UTF-8, and the bytes not following the rules of the latter encoding are replaced with the Dreaded Diamond Question Mark (or possibly Dreaded Box for some Windows machines).

A more subtle symptom is when everything appears to work, but the labels are not entirely accurate: a Web page declares its character set to be UTF-8, but the database is labeled as "latin1". Yet if the script that serves the Web page tells the database to send results in UTF-8 mode, then suddenly the Web page appears double-encoded (our first symptom from above). In this case, the database actually contains UTF-8 data, but the connection and column (or table or database/schema) are both labeled as latin1; thus no conversion is happening and sending the result in a UTF-8 page allows the client to display it correctly.

Another symptom I have seen in the wild is that strings in UTF-8 containing only some particular 8-bit encoding's characters (such as Latin-1) get littered with �, while other strings that happen to contain any characters outside that range get printed in correct UTF-8—including any characters that break in other strings! This one is Perl with a non-UTF-8 output encoding. If you have warnings on, and check the appropriate error log, you may find "Wide char in print" warnings generated for the strings that seemed to print correctly.

Fixing Things

As mentioned in my previous post, to fix the database mislabeling with MySQL, I could dump the data from mysql and restore it. I created two dumps: one with --no-data that I piped through sed -e s/latin1/utf8/g and one with --no-structure --no-create-db --default-character-set=latin1 that I changed the SET NAMES line to be utf8 in my favorite editor. This gave me a database structure labeled UTF-8, which I then populated with UTF-8 data (as it was) also labeled as UTF-8 (because of the new SET NAMES).

I also took a moment to edit /etc/my.cnf to add a [client] section with default-character-set=utf8. I also set the server's variables to utf8: character_set_X, where X is client, connection, database, results, and server.

The PHP website was already issuing the appropriate header for UTF-8 output, so it displayed replacement characters until I changed the connection character set. With PHP 5.3.6 and up with PDO, adding charset=utf8 to the DSN works; otherwise, I simply issued SET NAMES utf8 after connecting. If I were using mysqli, I would use set_charset after connecting, so that the library would also know about the new encoding in use.

Perl also displayed replacement characters after I added mysql_enable_utf8 => 1 to the DBI connection options. To avoid breaking older programs, Perl considers everything to be in a "machine" encoding unless it's told otherwise. I'm not entirely sure where it comes from; it's certainly not locale, because that was UTF-8. However, when faced with a UTF-8 string being printed on some filehandle that is not UTF-8, Perl tries to convert the string to the filehandle encoding. If it succeeds, it writes that 8-bit encoding out. If it fails, it writes out the UTF-8 bytes and generates the "Wide char in print" warning.

The fix for Perl was to set the filehandle to UTF-8 output with binmode(STDOUT, ':encoding(UTF-8)'); before writing non-ASCII data to it. If I had any in my source code, I would also need use utf8; to prevent Perl from double-encoding UTF-8 sequences in the program source (particularly, in strings). And if I were using Perl 5.14, I'd probably also use feature 'unicode_strings'; as well, as mentioned in The "Unicode Bug".

I've Earned It, Now

I need to buy one of the I � Unicode items from this shop. And if you want more perspective on this topic, Jeff Atwood collected a few links in his I {entity} Unicode post.

The Trouble with REST

2011-10-21T20:52:00.001-04:00

REST is easy to describe. It goes a little something like this: "You have some representation, and you send (or receive) the whole thing to read it or make changes." People coming from Clojure would understand it as REST sends values. I can GET an object, receive the value, manipulate it, and PUT the new value. It's so easy because it just uses HTTP!

Right? Maybe not. If REST is so easy, why is there HATEOAS*? Shouldn't that have been obvious? Why do we have arguments about versioning and parameters and formats and headers on Reddit?

REST is not easy. It's quick to explain in a limited manner, but the basic explanation leaves out important details, like how one actually goes about designing a proper REST API, and what makes an API RESTful to begin with.

Let's design our first API. With the CRUD operations mapped to HTTP verbs (POST, GET, PUT, DELETE), we start with a URL space for users. /users/21 for a specific user, or I can POST to /users to create a new user. All well and good, but now I want to search for a user, which doesn't have an obvious, direct mapping onto HTTP verbs.

I can define a special URL, /search/user?name=bob or /users/search/bob. I can define a new verb to SEARCH /users?name=bob. I can (though this heads straight out of REST territory) define a parameter acting as a sub-method: /users?search=bob. But then what if I want to find a particular Bob who is younger than 26 and has a GMail address?

The "simple" explanation just blew up in my face because there's a gaping hole in the map. Nobody even labels it "Here be Dragons" because we're all optimists.

The post on Reddit today was focused on versioning. There are those who think a "REST API" should never contain a version embedded in the URL, and apparently I just have to hope that nobody anywhere is running an old, incompatible client when that shiny new version is rolled out on the same URL. If I changed the URL, that's like putting a version in the URL, which is forbidden.

This discussion also went off into whether it was better to version by asking for "Accept: application/vnd.my-api.user+v3" or something like that, instead of text/xml.

Speaking of formats, when an API provides both XML and JSON (and probably does one badly), that invokes more argument around format specification. Accept header? format=json query parameter? And of course there's the (old?) Rails way of /users/get/31.json. And hey, that "get" looks like a verb in the URL, which isn't RESTful.

And then there's the "use http" ethos. If GET /users/superman is invoked with authorization (HTTP auth or session cookie (speaking of which... that's state and that's bad, mkay?)) and the user is not allowed to see that particular user, what code do you return... and how is it differentiated from "your session has timed out"? Do you start adding random custom headers like X-Session: timed-out?

Let me not forget conflicting edits. Most systems solve this by returning some sort of version identifier for an item, then requesting that identifier to be returned on save. If another client has sent in an edit, then the identifier doesn't match what's stored on the server, and the edit is refused. Isn't RPC roundly criticized for relying on server state? Isn't the version tag inherently shared state? Sure, REST may call for sending the entire state around in every request/response, necessary or not, but I really don't see a difference between sending an EditSequence to QBWC over SOAP, which is as RPC as you can get (until the world decided "let's use all document-literals all the time"), and sending a version tag back to bugzilla so it can detect mid-air collisions. Well, at least we have HTTP 409!

If this is all supposed to be so simple, why do we have all these arguments? And if Roy Fielding knows everything about REST, maybe he should drop by reddit and enlighten us once and for all.

Actually, for five minutes. This is the Internet.

My answer to the question I boldly asked is that this stuff isn't easy nor clear-cut. Programming is about wrestling the dragon on the path of implementation, and all of the paths have a dragon of some sort. Pretty much anything can be "made to work," which is why most seasoned programmers hate everyone else's code—working code is not necessarily easy to understand, and nothing interesting is universally easy (since easy is subjective).

I think I must be reading the wrong part of the Internet again, because I'm wishing for less talk of "what makes something RESTful", and more focus on real-world API design considerations. Especially the disadvantages of each approach. Nobody wants to talk weakness; it gets left to be rediscovered over and over by those that walk the same road.

I think, in essence, the trouble with X is that nobody talks about the disadvantages of X. Today, REST just happens to be filling in for X.**

* BTW, when did Service Oriented Architecture start meaning WS-* and J2EE? I guess I'd better start calling it API Oriented.

** There's no conclusion here; this is a rant.

Notes on using mysqlbinlog for copying updates

2011-10-19T15:36:00.000-04:00

I commented on this post, but for posterity:

It seems by sheer luck that I stumbled over a way to take care of everything. I save a copy of the interpreted binlog as it files through the pipe:

mysqlbinlog ... | tee binlog-play.sql | mysql ...

Then if I get an error message, mysql will tell me e.g. "Error ... at line 42100". Running "vim +42100 binlog-play.sql" lets me inspect the stream to see what went wrong in detail.

Inside binlog-play.sql, the "#at 112294949" comments can be used in e.g. "--start-position=112294949" to the next mysqlbinlog command, to retry the statement after I fix the problem. (Alternatively end_pos seems to tell the position of the next command, if I need to skip the one which failed, e.g. I was testing out CREATE FUNCTION and it was logged as "CREATE DEFINER=... FUNCTION" which RDS refuses.)

The final piece of the puzzle is that executing "FLUSH LOGS;" or "mysqladmin flush-logs" will push mysqld on to the next binlog file, so you can safely play out the one you want. Once you've finished processing a file through mysqlbinlog, you can just remember the file boundary, and flush mysql's logs if you want to process the one it's presently writing to.

This is in regards to piping mysqlbinlog output from one mysql server into the mysql client to execute on another; the post I linked above discusses doing so for switching to Amazon RDS. The basic strategy is to minimize downtime by loading a database dump from the source on the destination, then use mysqlbinlog on the source and the mysql client to feed updates from the source to the destination. The updates can be faster to load than a new dump; and when it's time to switch servers, it's a matter of stopping database clients, turning off the source mysqld, sending the final binlog updates, pointing the clients to the destination server, and turning the clients back on. As opposed to waiting for a whole dump to load while the clients are off.

Character Sets, Encodings, MySQL, and your data

2011-10-18T19:38:00.000-04:00

I'm currently moving data from a (relatively old now) MySQL 5.0 server into Amazon RDS. I've been here before, when I was moving data from MySQL 4.x into 5.0 and mangling character sets. This time, I want to make 100% sure everything comes across with maximum fidelity, and also get the character encoding as stored to be labeled correctly in MySQL.

First, a quick definition or two:

Character Set: a specific table to translate between characters and numbers. Example: ASCII defines characters for numbers 0-127; "A" is 65. This can also be described as "a set of characters, and their corresponding representation inside the computer."
Character Encoding: a means of "packing" numbers from the character set into a container. Example: UTF-8. The Unicode character 0x2013 becomes 0xE2,80,99. The "E" signifies "Part 1 of 3", and part of the remaining bytes simply indicate "Continued"; the 0x2013 is then divided up to fit in the parts of the bytes that aren't indicating their "Part 1" or "Continued" status. In the specific case of UTF-8, the encoding is designed so that the ASCII range 0-127 (0x00-7F) is encoded without change: a leading 0-7 means "Part 1 of 1".
8-bit character encoding: In older, simpler days, character sets defined only as many characters as could fit in 8 bits, and defined the encoding as simply the numbers. Character number 181 would encode as a byte (8 bits) with value 181.
A character encoding implies the associated character set, because the encoding defines how numbers in its character set become individual bytes. How characters in other sets would be encoded is left undefined and basically impossible.

This last point is why MySQL lets you set "character sets" to UTF-8, though the latter is an encoding.

Now, I'm staring at a database where the actual bits that make up the data are encoded in UTF-8, but MySQL thinks they are in Latin-1. Whenever anything connects to the database, the character set is always Latin-1, so MySQL does no translation. Nor does the code serving our web page, preferring instead to send it as-is to the user. In the end, the HTML page is labeled as UTF-8, so the untranslated UTF-8 bytes display correctly in the browser.

The problem is, every database backup that I have taken might be secretly broken: mysqldump is Unicode-aware, and left to its own devices (no --default-character-set given), it sets the connection into UTF-8 mode and reads out the data. This means the bits that end up in the file are double-encoded: when coming across a left-curly-quote, the server says, "Hmm, the client is asking for UTF-8 and I have Latin-1. I'll need to translate these 0xE2,80,9C bytes here into UTF-8 and send it that way."

What comes out on disk? I checked a hex dump, and it's 0xC3,A2,E2,82,AC,C5,93. Viewed as UTF-8, that byte stream is A-circumflex, Euro, and lowercase OE ligature. This is clearly impossible, since Latin-1 predated the creation of the Euro, and it's Latin-15 that contains it; it's double impossible because the Euro character in Latin-15 is 0xA4 which has nothing to do with any of the bytes we were trying to encode.

However, in Windows-1252, the 0x80 byte is the Euro symbol, which can only mean: when MySQL says "latin1", it is actually interpreting the data as Windows-1252. But it's not so bad, because the entire transformation is completely reversible (as long as latin1 continues to mean the same thing)... but what about the right-curly-quote, which ends in 0x9D, whose value is not defined in Windows-1252? The "9D" value is just assumed to map to Unicode character 9D, so 0xC5,93 is replaced by 0xC2,9D.

What character is that, anyway? It's OSC, the Operating System Command, part of the C1 control set and a control sequence introducer. Headdesk!

I think this still works okay for me, because the exact reverse happens going back into Windows-1252. At least, on the same server version and whatever Unicode libraries it's using. Nobody tries to interpret the OSC anyway, but it still looks odd to find a "?" showing where there actually is a valid UTF-8 sequence in the source.

Now that I knew all this, I could get a database dump out in an unconverted format, by including --default-character-set=latin1 on the mysqldump command line. The only problem was, this would give me a file which was exactly like the database: it contained UTF-8 data labeled as Latin-1. Loading it into a MySQL server would recreate the same situation as before.

My solution, found nowhere on the Internet, and which would probably make a lot of purists scream at me, was to take two dumps: one with --no-data that I piped through sed -e s/latin1/utf8/g to change the labeling, and one with --no-create-info --no-create-db in which I edited the SET NAMES latin1 by hand (to utf8, of course), to avoid editing the "latin1" in actual data when I was talking about the problem in the bug database.

Then I did some additional sleuthing and fixed the non-binary non-UTF-8 data which was lying around in the dump, and it all imported into RDS fine.

As a possible point of interest, it took about 10 minutes and 11 seconds to load the 351 MB data-only dump into a small DB instance. (Work has a symmetrical 8 Mbps connection, so it wasn't necessarily network-bound.) It did take a noticeable amount of time to load the 1 MB of schema, but unfortunately I didn't expect that to be difficult, so I don't have timing for it.

iPad vs. Tablet PC

2011-10-11T09:30:00.001-04:00

One of them succumbed to death by risk-aversion.

One of them couldn't let go of the tether and fly.

I think Linus said the same of svn: paraphrased, "If you're trying to make 'a better CVS' then you have already lost, because CVS is too broken to fix."

Hey, sapphirepaw: make sure what you do is good on its own, not "an X only different".

Steve Jobs

2011-10-08T10:12:00.002-04:00

I'm getting old: if I were to pass on at the same age Jobs did, my life would be more than half over already.

What separates me from Jobs? There's the matter of leverage, where he could take his vision and coordinate the prototyping and development of it, into the iPod, the iPhone, the Macbook Air, the iPad. There's also the matter of having vision.

In 2006 or so, I beheld my first iPod in real life, an old (FireWire based) model with a physical click-wheel. In 2008 I picked up a different, small MP3 player and for the first time, immediately noticed the limitations of digital control. Without having handled the iPod and getting a feel for the analog response of the wheel, I probably wouldn't have given the buttons a second thought. Do you want to scroll on the generic? Click-click-click-click. Or click-and-hold, guess at how long you need to go (since the screen is slow enough to be unreadable at this scrolling speed, and they don't slow updates to compensate), and release.

The point here is, Jobs saw humans as inherently analog, and adapted all of his machines to analog control. It's a simple thing, but Jobs was apparently devoted to HCI. The "vision" simply falls out of that.

It's not like the limitations of digital control weren't apparent in the 1980s. Compare Rad Racer to a real car's steering wheel. Anyone focused on "how it feels" could have been Jobs back then, inventing 2010 in the 16-bit era instead of carrying 8-bit paradigms through the 1990s.

In contrast, I seem to lack vision because I'm busy implementing arbitrarily complex business rules at work, and staying away from the bleeding edge of gadgetry. I'm not in the consumer space; I'm not taking any research toward the consumer space; and I'm not thinking about what's next for it, either (at least, not beyond what turns out to actually be the next thing*.) But, I'm also having little impact on the wider world, writing code that never leaves the house. It's important, but after I am gone, will these be the best years of my life? Will I think college was the best time of my life, forever?

I think it's time to put my free time to better use and do something instead of watching the world slowly develop towards Jobs' vision on its own.

* I have a dead draft which discusses the crazy idea of "having a set-top box inside the remote" in 2006 or so. It then points out that h.264-over-wifi ought to handle the bandwidth to do exactly that from your iPhone now. It starts fleshing out what would be necessary to make it happen, then abruptly ends with a note: "Two days after I started writing this, Apple announced AirPlay."

Setting Everything on Fire

2011-10-06T12:39:00.000-04:00

I created a new user, gave them wheel group, and in case I needed another admin user, added %wheel to sudoers through visudo. Then, I was trying to do more stuff, and...

[sudo] password for ec2-user: _

Wait. What? Not only does ec2-user have no password, but I didn't change its NOPASSWD line in sudoers.

It turns out that ec2-user is also in group wheel, and when confronted with the two permission sets, sudo did what I didn't mean: applied the %wheel rule and started requiring passwords for ec2-user. Of course su was no help either: root likewise has no password set, because you have sudo as ec2-user....

Thus began the adventure. I hacked around a little bit, but setgroups(2) is a privileged call, and thus can't be used to drop supplemental groups if you're not root. My hope was that I could use it to shed the wheel group to make sudo skip the problematic %wheel rule, but the reality was not to be.

I did some reading on how to recover this the hard way, and worked my way through it. Apparently by a massive amount of concentrated luck, I attached the volume with the busted sudoers after booting the new instance (a second one, in the matching availability zone: the volume in 1b didn't offer the choice of attaching to the instance in 1a), and fixed sudoers.

Next, I stopped the new instance and reattached the EBS volume to my old one.

Then the old instance wouldn't boot. The AWS management console claimed it was running, but it was dead to the world. The system log was empty (a handful of spaces). So I detached its volume, put it back on the new instance, started it up, and things got really confusing: the system log from the old volume had events from the new system on it. Eventually, I sorted this out: someone (AWS or the kernel) sees a disk labeled "/" and boots off of it, or mounts root there; since "LABEL=/" is listed as the root device in whatever /etc/fstab is in use, and both volumes have that label, it's possible that the kernel just picks one to resolve the conflict.

So apparently the old instance wouldn't boot, yet the new instance would boot with the old instance's disk at sdf1 mounted as root, though its boot device was sda1; even after some hackery, when sda1's fstab pointed specifically at sda1, the new instance ended up with the old volume visible as the root filesystem. Confusing as this was, it was at least after I had fixed sudoers, so I could poke around the system as root, since the old volume's sudoers file was in effect by this point.

I never did solve the mystery of why the old instance wouldn't boot. The new instance booted with its new volume detached and the old volume mounted at sda1, so I ended up deleting the new volume and old instance and calling it good enough.

War Story: The Training Jump

2011-09-09T20:47:00.001-04:00

One of the first things I did at my current job was to rewrite a Perl/CGI (the module, and the actual cgi-script execution model) site into PHP. Part of this site implemented a single-signon (SSO) system for a partner site that hosts our training videos. Clicking the link led to the innocuously-named "training_jump.pl" CGI script.

The goal in life of the training_jump is to redirect a user to the partner site, passing along the username and email address. The partner site creates the user if necessary, starts a session for them on its server, and ultimately displays the actual training content.

Inside training_jump is an innocuous-looking "use OAuth::Lite;" line. I didn't know what OAuth was at the time, so of course I went and looked it up: OAuth is designed to let a site like ExampleMashup authenticate someone as "twitter user chris" without needing to ask chris directly for their twitter password. Of course, this makes no sense, because in our case, we possess the account, not our partner. Likewise, once the login is complete, the user should end at our partner's site rather than our own. We have nothing to use the oauth token for, because we don't perform any operation at the partner site aside from the login.

Yet here inside training_jump was OAuth. The user hit training_jump; we redirected to the partner by IP address (!) with the OAuth request token, all the necessary user data, a callback URL (training_jump again); they duly redirected; we collected the response token and redirected the user back to the partner with that token as the parameter. The end result is still kind of fragile, in that AFAIK, it only works in the first browser you sign up with. If you log in with Firefox, then try it in Chrome, the latter gives you an error somewhere along the line instead of videos.

IIRC, research at the time indicated that there was no good PHP OAuth library, and/or the suitable libraries didn't implement the exact flavor of OAuth API that was being used by the Perl code. I'm absolutely certain I considered replacing the Perl entirely, but I don't remember why I rejected PHP OAuth as a solution.

I couldn't simply continue using training_jump as-is, because the CGI module and PHP store their session data in different locations, in different formats. The username in the PHP session wouldn't be accessible to pass through the authentication dance, and it was clearly inadvisable to modify training_jump to accept a username as a URL parameter.

Nowadays, training_jump has been succeeded by the cleverly named training_jump2, which actually reads request variables on stdin and produces an answer on stdout. (The format of this text is much like LiveJournal's ancient API, from back when I had a LiveJournal client. There was no convenient interchange format, as the Perl code didn't have JSON installed at the time and PHP didn't have XML. "Lightweight" eats you again.) The PHP training_jump manages the connection between server environment and training_jump2, and training_jump2 simply had its server environment replaced with communicating over pipes.

We're in the negotiation phase of moving to the provider's newer platform, which has a proper, encrypted SSO system. training_jump2 is slated to become irrelevant, eventually. In the meantime, it's the only bit of Perl CGI that never made the jump to mod_php.

sapphirepaw's Introduction to Pointers, version 2

2011-09-04T10:35:00.000-04:00

I programmed in assembly for some time, using pointers without understanding what they were, or that they were called pointers. When I finally got to learning C, the pointer syntax was downright inscrutable, but when I got it, suddenly all of C and all of assembler laid clear before me, all at once. It was a beautiful thing.

I was reminded about this while reading this post from HN. It inspired me to try explaining pointers from the opposite direction. Instead of trying to teach pointers via C syntax, let me try to start with pointers outside of programming, then discuss them in relation to C and PHP.

Pointers IRL: use and mention

A distinction can be made between using a word and mentioning it. Consider the sentence, "The password is incorrect." If it is saying that the wrong password has been given, then 'incorrect' is being used; if it is saying that the word 'incorrect' is the password, then incorrect is being mentioned. When mentioned, the word is not used for its meaning, but for the word itself.

In effect, a word is a pointer to its definition. Normally, a word is used, so there is no written convention to signify use. We just write the words. When mentioned, the word is typically quoted or italicized, to indicate the mention and separate it from a mundane use. The example sentence would then become, "The password is 'incorrect'." In speech we don't have quotation marks, except for the much-maligned air-quotes, so inflection and timing are used instead: "The password is... incorrect."

Steve Yegge has a talk somewhere on branding, and how brands are also pointers. They're like any other word, except that they have to be distinctive names, and the company associates this name to their specific product. Thus, when anyone says 'Dr. Pepper,' there is a single product that comes to mind. Yet there's nothing specific to branding that makes a brand into a pointer, without applying to other words. The only difference is that a brand's word—its pointer—is invented so that the owner can have exclusive control over the definition, instead of the multiple definitions that ordinary words commonly have.

Variables in Memory

Our next stop on this trip is to examine the structure of the average C program. Everything the program is working on has to be held in memory. This consists of the actual program instructions, an area called the heap for large or long-lived data that the program needs to handle, and a dual-purpose area known as the stack. The stack's primary purpose is for holding return addresses, so that when a function is called, the place to return to is stored on the stack, and is read when the function returns in order to access the next instruction in the caller. The nature of the stack also makes it possible for individual functions to store small, temporary data there.

In order to catch runaway programs, the stack often has a limited size. It may only be possible to store 8 MB there, while the heap may hold 1,000 MB or more.

To use the value stored in memory, a variable has an address. At some level, every variable is a pointer, pointing to its associated storage. Normal use of a variable, as in "c = a+b", reads or modifies the contents of memory at the variable's address, like normal uses of words rely on their definitions.

Function Calls

What happens when a variable is passed to a function? It's used in the call, so its contents are retrieved, then given to the function, which reserves its own storage for its parameters, then places the value it received into its respective new address. Any use of the variable within the function uses this new address, and the caller's copy is unaffected because it's stored at a different address.

If the stack is a small pile of index cards, which can be taken, written on in pencil, erased, and then returned in order, then calling a function takes an index card for each parameter, copies the value from the given parameter onto the card, and gives the card to the function.

This works fairly well for small data, like numbers, but what if we loaded a copy of War and Peace into the heap, and wanted to find the length of it? It's long enough to take a lot of time to copy onto a series of index cards, and we may not even have enough cards available to do so. After all, the stack is limited in size. It would be ideal if we could simply tell the function where to find the text, instead of trying to send it a copy of the text itself.

If we could attach a cord to the text, and the other end to an index card, then the function could follow that cord to get to the text, without us having to copy it anywhere. And, it would only use one of our cards, instead of consuming a huge amount of stack space.

This is essentially what a pointer is. If we pass the address of the start of the text, then the called function (the callee) can go directly to that address and start reading the text to find its length. In this case, we are now mentioning where the text is located, instead of using the whole text itself.

char *: Your First Pointer

This is exactly how it works in C. In fact, C is famous for not having a specific "string" type, because strings are treated as arrays of characters, with an array being a continuous series of addresses all holding the same type. A string begins at some starting address, then continues until some address whose content is 0, also known as the '\0' character, which is simply pre-defined to represent the end of a string.

C's string is represented to the programmer as an array of characters, but pointers and arrays are practically equivalent in C. Thus, char[] and char * more or less represent the same type: a variable which holds the address of a character. When code operates on a string, it assumes that more characters follow, according to the convention above.

Therefore, when we call strlen(war_and_peace), we give it the address of war_and_peace rather than the text itself (because as a string, war_and_peace already is a pointer to the text, the address of the start of the text); and for its part, strlen expects this, and dereferences the pointer it receives to work with the data. It actively converts the mention into a use.

Yet, there's no reason that pointers must point to primitive values, like "char" in "char *". A pointer's contents can be another pointer. If you have a pointer-to-char, you have an array of char, which is a string; it follows that pointer-to-pointer-to-char is pointer-to-string, which is an array of string. Notably, this is how command-line arguments appear to a program. (Although it's still easier to reason about as char*[] than char** for me, even though the effect is equivalent.)

Use and Mention in C

There are two pointer-related functions of the * operator in C. In a declaration, it tells how many levels of pointers are needed to reach the final variable contents. Outside of a declaration, which I'll refer to as "as an instruction", it accesses the content of a pointer, yielding the pointed-to type. Both of these may be stacked, as we saw with char** for declarations. For instructions, a function with a signature of int strlen(char *s) would access the character at s with *s; the first letter of the first string in char **argv could be accessed with **argv. Likewise, *argv would yield the entire first argument as a string (since it would move from char** to char* as a type), for instance if you wanted to find strlen(*argv).

There is one more pointer-related operator, &. Given a variable, instead of using its value, we can mention it (retrieving its address) with the ampersand. Consider a function is defined as int get_config_int(char* name, int* out) which returns a success or error code as its return value, and on success writes the value associated with name into *out. If the caller has some variable defined as int mem_limit which it would like to use, it can use the & operator to get the address of mem_limit to pass to get_config_int, as in: ok = get_config_int("max_mem", &mem_limit);

Unlike *, & can't be repeated. A normal variable has an address; but the address does not have its own place to be stored. It can be stored into some other variable of the appropriate type, and that variable can have its address taken, but this is simply returning the address of the second variable, not the address of an address.

I've been using the terms already, but to keep things as clear as possible, in a declaration, char **argv reads literally as "char-pointer-pointer", though I usually swap the order to get "pointer-to-pointer-to-char". In either case, this can be translated to "string array" as well, but I think of that as a translation rather than how it appears in the code. In an instruction, *out might be read as "value-at-out", though I'm used to C enough to just read it as "star-out" and know what it means. Finally, &mem_limit is conveniently said as "address-of-mem_limit".

Thinking in Pictures

It helps, especially when building larger data structures, to draw out the relationship between variables and pointers in a diagram. The usual way is a box-and-arrow diagram, in which the boxes represent storage space. Individual boxes may be named, if they are a variable. The boxes contain either a primitive value or a pointer, the latter of which is represented by an arrow leading to the box it points to.

Here's an example with a hypothetical char **argv, for a program named "hello" run with one argument of "there". I've additionally chosen to list types beneath the boxes to help demonstrate how a star is consumed when a pointer is followed.

Note that the two actual strings, and the intermediate array, don't actually have names: they're accessed using the argv variable and some combination of dereferencing (another technical name for following the pointer, converting its mention to a use). Simply using the * operator to follow the pointer doesn't let us reach "sideways" into the arrays, though—for that, the natural way is to use e.g. argv[1] to access the string "there".

Not Explaining C

There are a number of details I've forced myself to leave out, since I'm trying to make this more about pointers and less about C. This includes pointer arithmetic, a long discussion of equivalence between pointer arithmetic and array syntax, when and why you use pointers and pointers-to-pointers, and so forth. These things are simply beyond the scope of this article.

Pointers Meet PHP

In PHP, pointers don't exist as a first-class concept. However, a few things are built on the same ideas, which tends to make the manual complicated when it doesn't want to use the word "pointer" for any of them.

Let us first consider the case of a humble variable. A PHP variable name is a string pointing to a value. In contrast to C, where a variable has a type, it is the value which carries the type in PHP. (This is what lets PHP be dynamically typed: a name may point to different values over time, and each value may be of any type.) Another difference to C is that PHP always uses values through a name, so it doesn't provide a way to access the address of a value. If you want to use it, you need one of its names.

References in PHP are simply pointing a second name to the same value. When you write $y =& $x, then you are doing a pointer copy, similar to y = x; in C when both are of type int*.

Variable-variables, which let you do $x=42; $y="x"; echo $$y; to print 42, are just a chain of ordinary variable lookups. $y is not directly pointing to the value of $x, but only to a string value containing the name "x", which is then immediately used to look for the value that $x is pointing to.

PHP's manual also has a long explanation regarding unset(), which becomes fairly simple to describe with pointers: unset() removes the name and its link to its value, leaving the value unaffected (unless this was the last name for the value, at which point the value is garbage collected). $y =& $x; unset($y); does not result in observable change to $x. The equivalent C is y = x; y = NULL; and this also does not affect x.

The other major caution regarding unset() is that the global keyword creates a reference, so that global $x; unset($x); has no effect, as with any other reference: you delete the current $x, which points to the same value as the global $x, without affecting the pointed-to value.

PHP variables also act like they're copied by value when used, except for object instances as of PHP 5.0. This works by secretly including another pointer: an object-type value holds not the actual object, but an identifier for the instance, also known as an "object handle". When you use such a variable, PHP (under the hood) notices that it's an object, and fetches the real object based on the identifier in the value. This way, the value can be copied like any other, yet still references "the same object" which is nearly always what you want.

You can generate a new instance by using clone on the value, which makes PHP actually create a new instance, with a new identifier, and return a new value which contains that new identifier. I think I used this precisely once in my career, back when I was less skilled as a system designer.

Actually, this value-is-a-pointer pattern is not new with OO: resources work the same way. When you get a file pointer with fopen(), assigning it does not open the file again.

But what about assembler?

It's pretty much irrelevant by now. But just for fun, here's how things looked in AssemPro on our Amiga, running a 68000:

move.l #42, d0 ; load a constant, aka immediate value, into register d0
lea intbase, a0 ; load &intbase into a0 (PC-relative)
move.l #intbase, a6 ; load &intbase as a constant into a6 (not PC-relative)
move.l intbase, a6 ; load *(&intbase) into a6
move.l 0(a6), d0 ; load *(a6) into d0: a6 must hold some pointer aka address

The third example loads the value of &intbase at the time of assembly. This may not be the actual address of the intbase label once the code is loaded to run, which is why relocation was invented. (Though I am entirely clueless about how AmigaOS actually handled them.)

I can't remember now if the fourth example is PC-relative or not. I didn't understand yet why you would write such code, when I was doing this stuff in the 1990s, and it was "more restrictive" so I usually didn't bother. PC-relative code used only relative offsets to the current instruction as addresses, so it could be loaded at any position in memory without having to be relocated, nor having to use an offset table.

Feedback Still Encouraged

This post has been largely rewritten for quality, but suggestions and questions are still welcome in the comments. In spite of my efforts to pare down the excess, there remain many digressions and hints of deeper layers throughout. I have just too much knowledge to know what's useful and what's extraneous for the topic of pointers specifically.

What Would "Better" Be? PDF Reader Edition

2011-08-08T19:46:00.002-04:00

I opened up a PDF in the default Gnome PDF reader last night, and it was once again a terrible experience. It opened with the zoom set to "fit page width", and the scrolling set to continuous. There's no concept of a persistent user preference, or user preferences that override the document preferences.

Then I got to considering the underlying reasons why I didn't like the default display.

I read a wide range of PDFs from a variety of sources. Some have columns, some don't, and my zoom preference varies based on which it is. If there are columns, I typically make the viewer as tall as possible, set the zoom to "Fit Page", and put up with relatively tiny fonts because then scrolling doesn't keep switching direction to go back up the page when switching columns, then down to the next page (which never aligns with a clean PageDown keystroke).

Without columns, Fit Page Width and continuous scrolling are almost what I want. I really wouldn't mind fitting the content width instead, like OpenOffice's "optimal" zoom, but the page width is typically close enough in spite of losing 10% to the margins. With optimal zoom, I could have the same text size in a smaller window.

But underneath it all, it seems like there's another problem with the interface, carried forward from the days of yore: all scrolling is vertical. On the keyboard, there are no PageLeft or PageRight keys. The mouse wheel offers the vertical axis as a nice wheel, but the horizontal one is reduced to tilting that glorious wheel, which thus becomes as ineffective as the left and right arrow keys for large scrolling.

The vertical bias to scrolling means there's no support for horizontal scrolling. I have a beautiful widescreen display, but I can't use the width except to make a single page extremely wide.

Ultimately, it would be nice to have a PDF renderer that was aware of structure, so that it could destructure the PDF and present it in a screen-friendly way—that is, without columns—on the screen. Presumably, since I can select from a single column at a time, the text is stored linearly inside the PDF (although it also carries glyphs and kerning info, to get that perfectly exact rendition on any interpreter). I'm also guessing that since the text is selectable, there's not a dumb bitmap coming back from libpoppler. And in the event where a PDF turns to mush when trying to linearize it, the faithful representation would be at hand as a fallback, albeit with reduced user happiness.

The Facets of Net Neutrality

2011-07-14T12:29:00.004-04:00

In its original conception, "Network Neutrality" as I understood it was about a lack of privilege amongst competing traffic sources: that Google, Viacom, the atheism reddit, the Anglican Council, and the Time Cube site would all be subject to equal traffic slowdowns in the face of congestion. A bit of thought would suggest that treating individual packets equally was not, in fact, desirable: you probably don't want your VOIP call and each individual P2P connection to be subject to the same rules, really. You'd rather the call got through even at the expense of delaying a few packets of your (or your neighbor's) download.

Certain large ISPs have been trying to twist it to mean they can charge on both sides, for content providers to be allowed to send data to "their" customers, though the customers are already paying (quite profitably for the companies) for their own access. They would be charging everyone for access, so it's "neutral," right? This is an anti-neutrality stance trying to co-opt the word so that it sounds like a good thing.

Pro-neutrality forces (in the first sense) argue that requiring content providers to pay for carriage, or for "premium" speeds, would completely destroy the internet as we know it. Also, many of them believe they are preserving existing neutrality, but this turns out to be incorrect. A content delivery network (CDN) essentially is an implementation of pay-for-speed, because the content provider pays for their content to be stored closer to end-users, which reduces load time for those users. Although the end-user's ISP doesn't receive payment directly, the content provider's payment to the CDN also funds the overall system by paying for the CDN's own connectivity at the ends, and infrastructure in the middle.

I think the value of the Internet is in two things: uniformity of access for end-users, and fair division of capacity. Uniformity of access is simply that any connection should be able to carry packets from any content provider, so that the view of "the Internet" from any one ISP is the same view as from any other. Otherwise, "the Internet" would cease to have meaning, as it reverted to the days of online services like CompuServe, Prodigy, and AOL.

Fair division of capacity is exactly what it says on the tin, that speeds and latencies should be balanced among customers of an ISP. I shouldn't be able to start a download and prevent Netflix from delivering video to my neighbor, and a bunch of people on 6Mbps connections shouldn't be able to deny service to 1.5Mbps subscribers.

The real emotional punch that gets brought into neutrality discussions seems to come from the leonine terms the ISPs would like to apply: around one-tenth of the current (often secret) usage limits, for as low as six-tenths of the price, as in Time-Warner's experiment last year. Though the current arrangement is apparently profitable and growing more so over time: the cost of carriage is falling faster than inflation is diluting revenues. The fear is that ISPs will establish these terms "in order to build out next-generation networks" and then not follow through on that investment, artificially limiting their service and allowing inflated payments that do nothing but lift the artificial restriction—in order to offer what is on the market today.

Promises, after all, are cheap.

This fear is only exacerbated by the incumbent ISPs' wars against municipal broadband. City-owned networks are being opposed in many states as 'unfair' competition. In at least one case, the city in question embarked on its network building course because the ISP claimed they would never offer higher speed. Yet as soon as the city decided to offer higher speed itself if nobody else was going to, the ISP frantically began upgrading their infrastructure, hurrying to complete it before the city's project was finished, so they could argue that the city network was 'unnecessary' due to the ISP offering its (new) high-speed service.

This fear is further exacerbated by the regular broadband reports showing that countries with more competition amongst ISPs, regardless of urbanization, have the fastest speeds and highest limits on data transferred, where applicable. If larger companies truly did have more efficiency and more benefit to the customer as they claim, then the average US broadband connection should meet—or exceed—the average connection in Japan. Instead, large companies' performance suggests they are the major impediment to improved service.

For the Internet to continue its course of innovation and convenience for the American consumer, protection of uniformity of access and fair division of capacity are sorely needed. Placing these responsibilities into the hands of existing large ISPs who have been actively demonstrating their complete lack of commitment to the principles, or their customers, except when threatened en masse with an alternative network, is clearly the wrong course of action to ensure the result. It is putting the fox with feathers stuck in its teeth in charge of the hen house.

TCP: Conflicting Goals

2011-07-11T18:47:00.001-04:00

David Singleton writes in "Why mobile apps suck when you're mobile (TCP over 3G)":

TCP assumes that the connection has a more or less constant RTT and assumes delays are losses due to congestion somewhere on the path from A to B.

This struck a special chord with me, because I had just recently read about TCP algorithms that had been designed to combat "buffer bloat": instead of scaling strictly based on packet loss, assume increases in latency are due to buffering on the path. Then, back off to avoid both packet loss and longer latency, which is measured by RTT.

Since 3G attempts to implement reliable delivery itself, TCP-in-3G bears performance characteristics similar to TCP-in-TCP that is explained in Avery Penwarr's sshuttle README. (sshuttle takes care to extract data from the one TCP connection and copy it to a technically distinct connection, instead of wrapping it, in order to avoid the problem.) And actually, I see that Singleton linked to another source going into more detail, which I skipped reading the first time around.

So not only is 3G a bad transport for that reason, but the variable RTT its delivery mechanism introduces also sinks TCP algorithms which try to use increased RTT to avoid queueing in buffers. The buffer-avoidance aspect can't distinguish between "bad" buffers like those in a cheap home router that take huge chunks of data off the Ethernet at 100 Mbps, then dribble it out at 0.6 Mbps to the Internet at large; and "good" buffers like those in the 3G system that are unclogging the spectrum rather than crowding other users of the tubes.

Singleton proposes some mitigations for app developers; I'd rather try to "fix" TCP so that it gracefully handles variable RTT. It may violate the perfect conceptual segregation of the OSI Seven Layer Model, but simply having the phone's TCP stack aware of the wireless interface itself would go a long way toward mitigating the problem. Perhaps if the 3G hardware could indicate "link restored" and "backlog cleared", TCP could skip using the RTT of packets received between those events in its congestion avoidance.

It seems like WiFi would need some mitigations as well. It is particularly prone to periods of "solid" packet loss, occasionally even destroying the beacon signal and thus kicking everyone off, and periods of fairly reliable reception. However, when you do get reception back, the data pours in without significant degradation in speed, so the underlying issue is a bit different. However, the connection always seems to be particularly slow if it has the bad luck of being started during a period of loss.

In the end, the problems seem to come from allowing endpoints to specify receive-windows, but not the network. TCP views the network as a dumb thing that it can draw conclusions about based on end-to-end behavior. Yet the increasing prevalence of wireless, and of sending TCP over wireless links, seems to indicate that "the network" should be able to add metadata to the packets (probably at the IP level, since the network is conceptually unable to peek inside of IP data) to indicate that the delivery of the packet was delayed for reliability. Unfortunately, rogue devices could set that bit for their buffer-bloated packets, so it's about as practical as the Evil Bit.

Python's sum()

2011-06-05T08:22:00.000-04:00

In Python, the sum() builtin gives you the ability to take a list, say [1, 2, 10] and find the sum of it as if you had written out 1 + 2 + 10.

The + operator is also defined for lists, where if you write out [1] + [2] + [10] you'll get a list back: [1, 2, 10]

What happens if we put these two observations together? Can we sum() a list of lists to get one flattened list?

Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print sum([[1],[2],[10]])
Traceback (most recent call last):
  File "<stdin>", line 1, in 
TypeError: unsupported operand type(s) for +: 'int' and 'list'
>>>

Nope. sum() internally starts with "0 + (first element of sequence)" so you can only pass things that can be added to integers.

The First Step of a Long Journey

2011-06-03T21:02:00.001-04:00

Over the past couple of weeks, I have assembled a reader in PHP, such that it understands code of the form (print (== (+ 4 4 6) (- 30 15 1))) and will be able to create PHP source that ultimately prints out "1". It's kind of brokenly stupid in other ways, but it's the bare-bones skeleton of a working compiler. Something I have never been able to build prior to this attempt, largely because I wanted to tokenize something superficially like PHP, and I always got bored of defining all the stupid tokens. Going with s-expressions made for only a handful of token types so that I could get on with the interesting bits instead of grinding out pages of /&&|\|\|/ crud. Because almost anything can go in an identifier, I can treat everything as identifiers for now.

There are a few obvious things it needs next: string types. Variables. defun. defmacro. Separate namespaces for functions and variables, defined by context, so you can say (array_map htmlspecialchars row) and it will know that the first argument passed is a callable and the second is a expression, so that they can compile to 'htmlspecialchars' and $row, respectively. And to serve its original purpose as an "enhanced PHP"-to-PHP compiler, it needs to read that source language rather than s-expressions. Of course, with a non-sexp-based language, macros might not work out so well, but I do want to be able to run code to rewrite the AST (or the whole tokenizer: aka reader macros) at compile-time.

There's a bunch of features I want to add, too. Proper named arguments. Multiple-value return. Ubiquitous lexical scope, so obviously let and its function equivalent (flet perhaps?). Something else that I'm forgetting at the moment.

In the long run, I also want to do some optimizations; ideally, I could turn $efoo = array_map('htmlspecialchars', $foo); into $efoo=array(); foreach ($foo as $k=>$v) $efoo[$k]=htmlspecialchars($v); as well as doing simple optimizations like i++; to ++i;. I'd also love to be able to compile some 5.3 code like $foo::bar("baz"), ?:, and "nowdoc" syntax into 5.2-compatible renditions (answer to the first: call_user_func(array($foo, 'bar'), "baz") though my accumulated wisdom now considers such things to be a code smell).

The weird thing about this is that if I succeed, I'll be doing what Rasmus did to create PHP—riffing on an existing system in the domain to come up with something a little better.

Accidental Lisp

2011-05-19T22:31:00.001-04:00

It began with a simple bit of laziness: I wanted a preprocessor so that I could write as if PHP had multiple return values. I'd write "return $x, $y;" in the callee, and "$a, $b = fn();" in the caller, and the preprocessor would rewrite it to valid PHP (throwing array() and list() around the appropriate expressions).

But I'm even too lazy for that. To do this right, I'd need to fully parse the PHP, so I could understand more complicated return expressions like method calls. So instead of that, I slapped together a lexer for s-expressions. They're a lot less hairy, and this is just some twisted experiment.

I was halfway through putting together a parser this evening for the lexer output, when I realized: a few years ago, I ported the metacircular evaluator from the SICP lectures into Ruby... then discovered I would need to write an s-expression parser, which you get for free with Lisp. (That project then died.) But if I finish an s-expression parser... I can port the metacircular evaluator to it and have the world's stupidest Lisp-1 implementation, i.e. it'll be done in PHP.*

Alternatively, I can define a package in SBCL that emits PHP, and have the reader and macros for free. Then my head exploded.

* Because this tool was intended for PHP shops, the compiler would have to be written in and emit PHP so there's no Scary Foreign Language involved, other than the compiler's input. And originally, that input language was going to be almost PHP.

Quickie: Diffie-Hellman Groups

2011-05-10T22:31:00.000-04:00

Relying on others' suggested magic numbers for crypto is probably a Bad Idea, so recently I studied Diffie-Hellman a while to understand what the "DH Group" parameter was in my IPSEC setup, and my PuTTY settings.

DH turns out to be a lot like RSA, so bit lengths are comparable between the two and neither is directly comparable to symmetric ciphers like AES. A specific Diffie-Hellman exchange happens using some parameters: a generator for the base, and a prime to use as modulus. (An exponent remains secret.) DH Groups refer to specific, pre-chosen prime-and-generator pairs so that, for example, SSH can negotiate "group 14" instead of transferring the complete parameters themselves.

These groups have been standardized in RFC 2409, with additional groups defined in RFC 3526. The latter RFC defines the bit lengths of the groups explicitly, stating that group 5 is 1536 bits, group 14 is 2048, and group 16 is 4096 bits. As far as I can tell, groups 1 and 2 defined in the earlier RFC are only 768 and 1024 bits, respectively.

Note well: I believe this means DH groups 1 and 2 are dangerously short and should not be used to set up an IPSEC VPN today. Likewise, PuTTY should really be configured out-of-the-box to warn about the use of anything less than DH group 14. However, before I take my own advice, I need to do some experiments to determine whether the IPSEC client in iOS actually handles DH groups other than 2.

Quickie: The Necessity of Whimsical Names

2011-05-04T20:07:00.000-04:00

Rackspace recently announced that they'd like to discontinue Slicehost at some point, migrate everyone to the EC2-like Rackspace Cloud, and make people worry per GB about the bandwidth they're consuming. So I'm preparing a move to Linode for more of everything*, and in the planning, I've come across a new argument in favor of whimsical names for servers.

If I give each server a whimsical name, like alice.example.com and bob.example.com, I can always refer to the old and new IP addresses as "alice" and "bob", while the change of IP of "www" propagates through the DNS. Between the time where the new address is set and the old one is expired (and note that there's no way to force an ISP's resolver to honor the TTL if they choose to assume "no TTLs will be shorter than an hour") the name being transitioned points to a more-or-less random server.

Basically, the whimsical name is like a server ID, and the service-based names are just conveniences. Though a program is three lines long, someday it must be maintained; though a server hosts one service, someday it will have to be replaced. When an organization gets big enough that it can't generate whimsy as fast as it needs servers, then it should go with something more regular for the server name, but each server should still have a unique, non-service-based name.

* Except bandwidth, but the 11% difference is smaller than my current monthly consumption, so it turns out not to matter much. Even if it did matter, that much transfer on The Cloud (insert angelic chord here) would be expensive, so Linode still wins.

Perception

2011-04-21T22:45:00.002-04:00

They say you don't get a second chance to make a first impression, but that depends on who you are. Apple seems to have managed a couple of major architecture transitions and their own Vista without too much ill will, yet Microsoft was practically crucified for Vista with no architecture transitions.

Fair warning: many links in this post lead to tvtropes.

To be sure, Apple had the advantage on their architecture transitions that they were moving between chips that had significantly more performance, which greatly reduced the penalty of emulating the old chips; a 100 MHz PowerPC (that, at the time, Amiga Computing* boasted would run like a 200 MHz Pentium, or something like that) was rather adept at pretending to be a 40 MHz 68040.

Vista, on the other hand, was widely panned in the tech press, and subject to the same problems regarding driver availability as Mac OS X 10.0, if anyone remembers that. "zOMG there's no drivers!" And it was so bloated. Well, on the machines of the day. I don't really fault Microsoft for believing that PCs would continue getting faster and everyone would keep buying new ones for Vista, considering the market had been working that way for something like 15 years. Multicore just threw them off their old, proven, and profitable game.

Anyway, in 2009, I got a new machine provisioned for me at my new job, the first one that wasn't cobbled together in-house out of spare parts, or handed down from the previous developer. Unfortunately my boss forgot to check the "Vista Home Premium" upgrade, so it came with Home Basic and was thus ineligible for the free Windows 7 upgrade program. So I've been using Vista for a year and a half, and it sucks. Totally, it's... uh... er, yeah. There's actually nothing wrong with it other than the fact that it doesn't have full Aero so I don't get semi-transparent blurring titlebars, which actually means the title text is readable. Yep.

Compare to my days of being a Tech Press Believer back in 2007, when I was worried (near to the point of tears, on one occasion) about building a computer for my then-fiancée 'soon enough' that I could still get XP for it, because Vista was so indescribably awful.

I didn't know then that Vista was actually a truly modernized OS, with a sane security model, and all the horrors of UAC were because 90% of Windows software was completely broken, and Windows of yore just didn't enforce proper coding at all.

Yet the tech press buried Vista under a negative wave of publicity. It became Microsoft's New Coke. The refrain of Windows 7, like Coke Classic, was essentially the message of "We listened." Now in the se7en era, the tech-o-sphere passes around the Official Belief that "7 is what Vista should have been, and I can prove it because it's even actually called 6.1", somewhat oblivious to the fact that 7 is largely what Vista is. Had 7 been released without Vista in between, how would it have been received?

Probably like KDE 4, Gnome 2, or Vista. "There's no drivers and they changed everything! Whyyyyyyyyy!" Because without the intervening release of Vista, many drivers at the launch of seven would have been in a similar situation as the launch of Vista: companies don't want to move on things until it's proven that they must, because preparing for futures that don't arrive is almost completely wasted effort.

An interesting point of divergence here between Apple's and Microsoft's approaches to OS X and Vista, respectively, is that OS X included an emulator for Mac OS Classic. There was much more of a break between the environments, with the new not making any pretense of compatibility with the old beyond a sandbox.

With Vista, however, Microsoft faced a bunch of bad choices: either they could continue in XP's footsteps, releasing NT 5.(x+1), and continue taking heat for security**; they could try an OS X or NT 3.x-like break with history, and get flamed for lack of compatibility or performance of an XP Mode if they tried emulating it; or they could launch Vista as we saw it, breaking compatibility with drivers and bad (but widespread) coding practices to put necessary pressure on just about everyone to fix their broken stuff.

No matter what they did, someone would complain. "XP mode" would inevitably be slower than running XP on the bare metal, and if Microsoft put in a lot of effort to paravirtualize it, doing so would make it possible for others to mimic that layer under it and run it on unauthorized systems. On the security front, they could continue the "anyone can break the system completely" model of security inherited from their single-user days, they could keep silently failing like limited accounts on XP tend to, or they could add elevation prompts and face complaints about the necessary inconvenience with legacy programs.

People forgave Apple for OS X*** 10.0, but they didn't cut Microsoft any slack for Vista. At least, the tech press and its devoted fans didn't. The Mojave Experiment brings out another interesting similarity to New Coke: supposedly, when people were using a rebranded version of Vista, they rated it much more highly than Vista itself. Likewise, Coke only went through with New Coke because testing showed that people liked it better. But take the blinds off, and New Coke? Eww!

In the end, 'success' (when defined as widespread usage of a product) is a result of perception of that product. Substantially similar products may end up at very different outcomes simply because of a difference in perception between the two that gives one an advantage relative to the other. Much-maligned Vista, a major and necessary improvement to Windows, underperformed because of widespread perception that it was Bad, among people who decided what to buy for their Fortune 1000 Company; whereas Windows 7 adoption is moving full steam ahead, such that it overtook cumulative Vista share quite a while ago now.

It makes me wonder how the scenario would have been played out if the tech press and their followers had not worked so hard to squelch Vista. Would Microsoft have deployed some of the efficiency improvements found in Windows 7 as a service pack or "R2" to Vista, bringing the memory efficiency improvements into their older OS? I doubt they would have overhauled the Taskbar so drastically, but as it was, the need to escape the "Vista = Awful" conception drove them to roll out a new brand for the OS as soon as they could.

* For a while after the end of Commodore, the Amiga IP staggered around, zombie-fashion, among companies that didn't really take much advantage of it. One of the possible, wonderful futures of those days called for an Amiga resurrection around the PowerPC platform; formerly, it also ran on the 68000 series, like the Mac, so it's not as irrelevant or crazy as it sounds. Then again, if they wanted to do anything like switch from bitplane to pixel-based graphics, it would have required seriously hardcore emulation compared to the Macs. AFAIK.

** In reality, as the biggest and most valuable target, Microsoft needs a much greater level of security in order to result in similar exploit counts, which is how people like to simplify 'security.' OSS proponents used to whine when someone totaled MS Advisories vs. Red Hat advisories for a year and concluded that Linux was less secure because RH issued more bulletins, since they issued them on a much wider range of software. However, even if you counted only "equivalent" vulnerabilities, you would find that bulletins-per-installation is higher for Linux because Microsoft has such a huge divisor on that metric.

*** I pronounce this "Oh Ess Ex," by the way, because "OS Ten Ten Point Five" sounds completely dumb, and nobody ever writes "OS X.5". Though I'd call it "Oh Ess Ex Five" if they did, because nobody uses Roman Decimals either. Maybe I should make a set of Perl modules so we can `use V::X::III` instead of 5.010_03 or 5.10.3, but then, Damian Conway probably beat me to it.

The Authority of the User

2011-03-04T07:00:00.001-05:00

I used to believe that my computer was mine, and no program had any authority to do anything without my consent. (This can probably be traced back to my days on Slashdot, a decade ago; if I didn't get the opinion from there, they certainly reinforced it.) I believed I was sufficiently smart to manage my own software, without everyone's updater constantly nagging me to do so. I especially didn't want the updater to do it on its own; this often lead to problems, especially when Firefox got updated behind the scenes while I was using it. However, I liked automatic security updates on Linux, so I got rather used to restarting Firefox when links mysteriously failed to be followed, or menus and tabs couldn't be opened—these being the days before the "Firefox has been updated and needs to be restarted" notification.

Then, everything changed.

It started with Chrome. Following a shift to pragmatism over idealism, my work PC running Windows started to collect useful, if bloated, software like Safari, Flash, Java, Adobe Reader, and the like. Then I had a bunch of update reminders popping up at random, because they all live in their own little universe and can't be coordinated. (And many of these use their update notification to try to sneak in the non-Google-search-toolbar-du-jour or worse, iTunes, if you're not paying attention.) Many of these updates also needed a restart of the system afterwards.

Chrome, on the other hand, has neither nag screen, nor reboot requirement, nor sneaky-ware. Which, as a user, is immensely convenient, in comparison to manual updates or the alternative "automatic" update systems that I have seen. Once again, it turns out that users don't know what they actually want.

However, I've heard on the internet (and therefore it must be true) that Chrome doesn't offer any control to an overworked IT administrator in a large company. It doesn't integrate with Group Policy, it frequently writes to the user profile and stores a lot of data there, and it doesn't offer any control over the update process. All of which combine to make it quite a thorn in the side of someone responsible for (say) 1,000 desktops.

Taking these points into consideration, the authority of The User over Their Machine is not an absolute thing. For one, the machine may not be technically "theirs," and for another, they may not be truly qualified or involved enough to make the decision themselves. These days, I would prefer "don't annoy me" over that feeling of "don't do anything without my permission" that I so fervently believed in before.

The nature of the software being updated also comes into play. If Chrome breaks, there's typically IE, Safari, or Firefox co-installed. Failure of Chrome is not as important as a failure in Windows, Photoshop, or a line-of-business application, all of which may be vital to a worker in a company. When Chrome fails, it's extremely likely that there's an alternative already installed on the machine which can be used with no downtime—and Linux distributions relying on Chromium (Chrome's parent project) as a primary browser do not take updates as they are released, but use a version from the channel that has been further vetted for a week or two.

Another consideration with the software is that the updates to Chrome have been relatively minor, from an interface perspective. It's not comparable to the interface jump between major versions of Firefox, for instance. I don't think many users would take it well if they opened up Photoshop one day and it looked entirely different than the day before, without any warning nor any way to get the comfortable UI back. In comparison, even if you jumped from Chrome 1.0 to 9.0, about the only thing to contend with is a unified menubar, and if you installed any, the Apps area of the New Tab page.

In comparison to my old whiny-egotistical-child view of "IT'S MINE," my current perception of the amount of authority a user should have over the computer is, "It depends." (That's the ultimate cop-out answer to everything, it seems.) The amount of warning and veto power a user needs depends on their skill, the software and its rate of change, what they're using the software for, and what environment the computer is in (work vs. home vs. mobile broadband connection).

Quick tip: extending the man search path without $MANPATH

2011-03-02T10:35:00.001-05:00

If you've ever tried to add a directory to man's search path, you've undoubtedly noticed that the MANPATH environment variable replaces rather than extends man's built-in search path. Today, I rediscovered a clever little setup on a machine at work.

Copy /etc/man.config to somewhere in your home dir. Mine seems to be at ~/.config/man/man.config for optimal redundant redundancy. (I will say that keeping the "man.config" name of the file makes vim highlight it without additional fuss.)
Add your desired MANPATH lines to this file at whatever position you wish. Don't forget to curse the lack of an include mechanism at this point, which prevents you from automatically getting changes to /etc/man.config. Cheer up, because there probably won't be any.
Add an alias to your shell. For bash, you would put something like alias man='man -C ~/.config/man/man.config' (which obviously includes the name of the file chosen in step 1) into ~/.bashrc. Remember to source ~/.bashrc to make it take effect in the current session.

That's all! Now when you run man, your personal manpages will be searched as well.

The documentation for man on the system in question claims that it will use $PATH to guess at additional man page locations, but this does not actually work for me. Having a command in ~/.install/bin does not allow man to find the manpage in ~/.install/share/man.

Variable scope, require, and use

2011-02-28T22:29:00.002-05:00

I ran into some interesting problems in Perl, which invoked more learning around the require/use mechanisms and how constants are interpreted. In this post, I'll lay out some general terms about variable scoping, such as lexical scope, dynamic scope, the differences between them, and how they all interact in Perl. And then I'll cover require and use with that foundation in place.

If you've been wondering about lexicals or closures, this is your post. I've tried to lay things out more or less from basic principles, despite the verbosity of the approach, because this has taken me years to understand. I started programming with Perl in 2000 and still learned a bit more about it today. Yes, it's 2011 now. Hopefully, you can read this and get it in less time.

Lexical Scope

With lexical scope, the variables visible to a certain block of code depend on the physical layout of the source code. A lexical variable can be used by code in the same block scope; typically in Perl, this is a package, a subroutine, an if/else structure, or a loop, and "a package" is often synonymous with an entire file.

An example is helpful here: note that lexical variables (variables to which lexical scoping rules are applied) are defined in Perl by the my keyword. The standard use of lexicals in perl is for variables local to a subroutine:

$x = 10;
sub foo { my $x = 13; print "In foo, it's $x\n"; }
print "At first, x is $x.\n";
foo();
print "Back outside, it's $x.\n";

This will inform you that x is 10, 13, and 10. foo's definition of my $x restricts the use of that variable to the scope of foo's declaration. Instead, suppose the file declares $x with my, and the sub does not use my:

my $x = 10;
sub foo { $x = 13; print "Of course, foo sees $x.\n"; }
print "At first, $x is still $x.\n";
foo();
print "Afterward, it is $x.\n";

In this case, you'll see that foo was able to change the variable declared outside of it. How? The sub foo is defined within the scope where $x was declared lexical, in this case the file. If you move the sub foo into a separate file called foo.pl:

sub foo { $x = 13; print "Of course, foo sees $x.\n"; }
1;

And then change the original to require foo.pl:

my $x = 10;
require 'foo.pl'; # THIS IS BAD CODE
foo();
print "With require, x is $x after calling foo.\n";

Now you'll see final value as 10 again. The require establishes a new lexical scope, so foo can no longer see the my $x declaration in the main script. Note the bad code comment; this is not a good way to use require, and I'll explain why later in this post. I'm using it here simply to demonstrate the point.

Since foo.pl isn't modifying the lexical, what variable gets changed? It's actually the global $x, since variables in Perl are global by default. If you were to print the value of $main::x before and after the require line, the values would be undef and 13, respectively.

Where does 'closure' come into lexical scope, you ask? This is your program:

sub series {
my $x = 0;
return sub { return ++$x; };
}
my $seq = series();
for my $i (20..24) { print $seq->(), "\n"; }

Well, this uses a lot more of Perl than the previous examples, but the output is pretty simple: 1 through 5. What's going on? The call to series() returns an anonymous sub, which we store in $seq as a code ref; then, we can call it with $seq->() where the parentheses are a standard argument list, which is empty in this case. The print is just returning the value of that call, which is the next value of $x. The loop variable runs from 20 to 24 just so you can see that the loop variable is not involved in the printing at all.

The anonymous sub is allowed to see the my $x that series defined, and it's always allowed to see it—even when the sub is passed out to code that cannot see it. This is what lexical scope is all about: the variables visible to you, based on your location in the source, remain visible to you, and nobody from outside the scope can affect them. (Unless your variable holds a reference to some data that they can modify.)

To differentiate variables like the last example's $x which are used in, but not defined within, a specific scope, the inner scope (the anonymous sub stored in $seq) is said to have "closed over $x", and $seq itself "is a closure" because it has closed over something. The notion of closure separates purely local variables which are re-initialized on each call (as in the first example) from variables that are defined outside of a function, which can be consistent from invocation to invocation. The variable doesn't live within the function's definition, so it is neither destroyed nor reset when the function returns or is called again.

There are two last Perl-specific things I want to point out before continuing. For one, even when a local variable is defined, the global is still accessible under the fully-qualified name:

$x = 10;
sub foo { my $x = 13; print "my x is $x but yours is $main::x\n"; }
foo();

This will show the values 13 and 10.

Also, Perl offers the our keyword to "undo" the effects of a my:

$x = 10;
{
my $x = 13;
print "my x is $x\n";
{
our $x;
print "our x is $x\n";
$x = 42;
}
print "x is $x, I swear\n";
}
print "x is really $x\n";

This example will show values of 13, 10, 13, and 42. I've used unnamed blocks here because they're more convenient for this example than trying to nest subs for no reason. our establishes that, for its lexical container, the variable should reference the global variable and value.

Perl 5.10 added a couple of features: one is the ability to use new features added to Perl, and another is the specific inclusion of the 'state' feature. (See perldoc feature for more in-depth information on features and their usage.) state variables are similar to my variables that were declared in a separate scope just outside the function. Only the current function can access it, and its value doesn't get reset when the function is called again. Our example from above with series and seq could be written as:

use 5.010; # activate all 5.10 features
sub serial {
state $x = 0;
return ++$x;
}
for my $i (20..24) { say serial(); }

There are important differences. In this example, there can only be one counter: you wouldn't be able to establish multiple independent sequences with multiple calls into the sub series that the other example featured. Another important difference is that state variables can be initialized later than their traditional lexical counterparts.

Finally, use 5.010; is not a typo. It's just the old version number format with 3 digits for the minor version (what perl calls "version" when reporting revision/version/subversion.)

Dynamic scope

In contrast to lexical scope, dynamic scope cares nothing for source code layout. Instead, it's affected by the run time. Perl's "global" scope is technically dynamic, and actual dynamic scoping is created by the local keyword. How does it work? Let's look at an example:

$x = 10;
sub foo { local $x = 42; bar(); }
sub bar { print "x seems to be $x\n"; ++$x; }
bar();
bar();
foo();
bar();

This shows the values 10, 11, 42, and... 12! Each time it's called, bar increments the "global" variable. However, foo establishes a new value for the variable, for itself and any function it calls, and this variable disappears again when foo returns (technically, the block containing the local exits). So bar updated it to 43, but then it and foo returned, and the original variable (whose value was then 12) came back into effect.

foo would not have been able to affect any variables declared inside bar as lexicals with my, but using local to change the dynamic scope, foo can affect bar's view of the global variables. This would happen regardless of whether bar is in a different file. It's also possible to affect another package, if bar explicitly references the global in foo's package--or if the latter calls local on a punctuation variable like $/ which are always forced into a canonical package (main if I'm not mistaken).

Dynamic scope is "dynamic" because it can change from call to call, depending on whether any callers in the current call chain have used local or not. There's nothing stopping a variable from being localized several times in the chain, either. Under dynamic scope, the visible variables can be redefined by action outside your location in the source. When bar() updates $x, it is not guaranteed to be updating the $x defined at the top of the file.

Perl's man pages note (or used to note) that if you're unsure, use my to define a local variable for your subroutine, not local. This is a helpfully interpreted instruction to use lexical scope. Dynamic scope turns out to be bad for larger systems, as your functions not only communicate through globals, but the "real" variable can be hidden and updated with a "fake" value. Alternatively, forgetting to set (or not knowing that you need to set) a particular "fake" value before calling some subroutine can lead to surprisingly different results from a function call that appears to be the same: the argument lists may be the same, but all of the state the function relies on is not.

Thus, you most often see local used inside Perl to redefine one of Perl's special global variables, like $/, for the duration of a block in order to set it to a known value without disturbing the view of that value from the rest of the program. Such code rarely calls down into other functions, since it needs to have the variable set for its own work. Also, Perl only allows for lexicals that are alphanumeric, so local is the only way to apply a temporary value to one of the special variables.

The last thing to note about globals is that they are actually package global variables. Take the following code:

$x = 12;
package Foo;
$x = 16;
sub foo { print "Foo's global x is $x\n"; }
package main;
sub print_x { print "main's x is $x\n"; }
Foo::foo();
print_x();

Note that for the sake of illustration, I've crammed everything into one file. You wouldn't do this in real code. An interesting side effect I discovered while testing this example is that the effect of our actually lasts to the end of file (or the next declaration of the same variable, as discussed below) on my Perl, so that if I define our $x = 16; inside package Foo, then the print_x subroutine uses it instead of $main::x!

Perl scope resolution

All of this has been implied already, but I want to bring it together here. When an unqualified $x is used, how does Perl decide which variable that actually means? It simply travels up the lexical scope chain, looking for a my or our declaration that applies for that variable. If it is a lexical (declared with my or state), then it's done—Perl just uses that variable.

If there is another lexical of the same name earlier in the current scope, or at a more outer scope, then that earlier/outer variable becomes inaccessible from the later declaration forward. This is known as the later variable shadowing the former.

If the variable is found and was declared as global with our, or not found in any lexical scope, then it is considered a global, which is taken from the dynamic scope.

Using local does not affect the resolution of a variable. It must be global where the local is invoked, or else you'll see an error at compile time, "Can't localize lexical variable $x".

In other words, my and our lexically control whether a variable is lexical or global, respectively; local provides a mechanism to shadow global variables.

Why (and how) "use strict" complains so much

One goal of use strict (and the only goal of use strict 'vars') is to prevent you from unintentionally using a global variable when you meant to access a lexical one. When strict is in effect, traversing the entire lexical scope chain without finding a declaration of the variable with my or our triggers the error instead of falling back to global scope. Fully-qualified variables aren't affected, because the full qualification implicitly means they are globals. Lexical variables do not have a fully-qualified name.

Doing it wrong: "require $target" and "require 'file.pl'"

In the days before the package and module systems were invented for Perl 5, the way to load a library was through giving the path to the filename as a string to require. This still works today, but is widely considered wrong, because of several subtle problems that it causes. These problems are probably what led to its replacement with the modern module system.

The first problem is one of scope. Using require this way will effectively import everything in the required file into your own scope, whether you wanted it or not—but only for global variables! Lexicals remain local to their respective files. It's easy to miss the distinction, or worse, have multiple files including the code, some of which have a global variable defined as lexical.

The second problem appears to be one of scope as well, but isn't. Constants defined inside a file that has been loaded with require may not be visible to the file that performed the require. Perl's constants are really constant, because they're determined at compile time. If a constant wasn't defined at compile time, then it becomes a bareword, which may end up being the "constant" name. And to make this distinction important, require is not normally executed until run time. By then, the file doing the require has already been compiled without the constants set, and does not get recompiled after the require completes. One quick patch is to use BEGIN { require "foo.pl"; } which will force the require to occur at compile time, which will define the constants in time for them to actually be useful.

A third problem is one of paths. To follow the real-world example, if a Web server is set up to serve *.pl as CGI scripts, and this Web server changes directory to the running script, then the scripts loaded with code like require '../lib/site.pl'; cannot readily find the path to require more code from the lib directory themselves.

A fourth problem is one of Perl's magic: require includes an entry in %INC after loading the file, which means that the same file required twice will not be loaded the second time. If you do try to load the same file twice using the same path name, this is almost certainly not what you want to happen. Likewise, if you happen to load the same file twice by using different path names, when you expected to be able to load it only once, this can also result in undesirable effects, depending on how the loaded file is written.

There are probably more problems, but these are the ones I know of.

Doing it properly: "require Module" and "use Module"

The use SomeModule form imposes some restrictions on your code: the name you give it must be a bareword (an unquoted identifier, more or less); it must be available in the include path, @INC, after converting double-colons to path separators; it must be named with a .pm extension; and it must define the package SomeModule with an import subroutine for use to work. For require, the import sub isn't entirely necessary, but you'd be crazy not to include it.

All these restrictions combine to make the code-inclusion mechanism more robust. Modules have a canonical name, so they can't be loaded twice under different paths. This name resolves to the same path in the filesystem, independently of where the inclusion is initiated, which allows included files to include more files without worrying about the current directory or the path to the files being included. The definition of a package name also creates a unique entry in the global namespace for the code, so that it doesn't need to be loaded more than once.

The import subroutine allows for a controlled inclusion of symbols into the caller's namespace, instead of dumping everything non-lexical (including all subroutines) into it. use has the added benefit of implicitly wrapping BEGIN around the inclusion, so that constants defined in the included module are available for use in the caller without them needing to remember to write a BEGIN of their own.

Incidentally, you now know why constants are defined with use constant: the constant pragma wouldn't be able to do its job if it wasn't running at compile time.

Managing the Conversion from "require 'file.pl'" to "use SomeModule"

However, extolling the virtues of this system doesn't help you much if you have a codebase that relies on require 'foo.pl'; and its implicit export. But, it's possible to write a file which can be easily converted from wrong to right. Start by organizing the code into a package (copying the file to a '.pm' if necessary), then add some bridge code at the beginning to 'export' subs that simply call into the package:

sub frobnicate { &SomeModule::frobnicate; }
package SomeModule;
sub frobnicate {
die "zomg, you didn't code this function";
}

Now, loading this code via require "../path/to/SomeModule.pm"; will create a frobnicate function that simply dies. The &Function; syntax with the ampersand and without the parentheses (both details are important!) will call function and alias its view of @_ to the current @_; this is usually not recommended, but we're doing dastardly things here, and this is just a shim that's not running any other code. If SomeModule::frobnicate messes up @_, it doesn't affect the unqualified frobnicate, because it never uses the value itself. It just returns the result back out, using Perl's implicit return feature.

When it comes time to make SomeModule available for use, the unqualified frobnicate gets deleted, and appropriate Exporter code added into the package. Doing it in two phases like this lets you test that the simple reorganization into a package didn't break any of the callers, and that you have accounted for all the symbols that will need to be exported.

Then again, all this section may just be pointless, since you could go straight to using use and Exporter. It's not like they're all that different from this hack.

This must be why Steve Yegge hates everything

Perl has a rich history starting with being a throwaway language with convenient features for the problems Larry Wall was working on at the time. Since the Perl team has put a lot of work into backwards compatibility and convenience, it has created a lot of subtlety and minor traps. You have to learn a lot about Perl before everything makes sense, and you're not just sprinkling my in front of every variable and mumbling, "WTF, why don't I just turn off use strict? This doesn't seem helpful."

Likewise, the other languages he's shared his hate for have evolved quite a bit over time into their current chimerical forms: Javascript and Python both started with global variables and hacked lexical scope in later. PHP probably gets the same complaint, but they added lexical scoping so recently that he hasn't had either time or inclination to blog about them yet, and we know the story anyway. Lastly, C++ was an experiment in grafting a specific view of OOP into a language that was never meant to have it. (For other views, consider Rees Re: OO.)

Lexical scope in other languages

Variables in Python began as members of one of three scopes: local to the current function, global to the current module, or built-in to Python. The module global scope could be accessed using the global keyword, which is similar in spirit to Perl's our keyword. Python added something like lexical variables as the nested_scopes feature, which allowed for a function nested within some other function to see variables of the outer function that weren't in global scope. However, all variable writing was assumed to be local, so those outer variables were effectively read-only. Some hacks arose around this, like pointing a variable in the outer function to a writable object, allowing for communication via object updates, but in Python 3, the nonlocal keyword has been introduced to allow for declaring that you want to use a name from the outer (but still not global) scope instead of local scope.

PHP began with two scope levels. Variables were either global or local to a function, with globals accessible from a function using the global keyword, or at some point, the $GLOBALS superglobal was introduced. (Superglobals main difference from globals is that they're pre-defined to be always global, so a function can use them without having to declare them with global first.) This all changed with PHP 5.3.0, which introduced anonymous functions and the ability to use (close over) variables outside the function.

Unlike many other languages, however, PHP went its own way on two points: only anonymous functions may close over variables, and the variables to be closed over must also be included in the function declaration. For example, $onUpdate = function ($x) use (&$y) { ... }; to allow the function to read and write the value of $y from the outer scope. (The changes are probably justified: regular functions which need state should be objects, since PHP offers class-based OOP, and explicit naming of variables fits the philosophy of "it shouldn't be possible to have bugs because something was in an unexpected scope" that drove PHP's original scope design.)

In Conclusion

Lexical variables are visible based on their position in the source code; in Perl, this is from their declaration forward to the end of their block or file. Lexical variables can be skipped over in favor of a global (package) variable, using our, which has the same area of effect as defining a lexical with my.

When a subroutine references a lexical variable in an outer scope, the subroutine is said to be "a closure", and it "closes over the variable". This bears no relation to closing doors, emotional closure, or set theory (where, for instance, integers are closed under addition: add any two integers, and the result is still an integer. This is not true of division.)

As of Perl 5.10, state variables are also possible, and are also private and lexically scoped; however, their value is saved between calls into the scope. my variables would be re-initialized when their declaration occurs.

The value of dynamically scoped variables are controlled by the call stack; in Perl, this is accomplished by temporarily reassigning a global variable with local. If you're not using local on Perl's special interpreter-control variables, then you are (or the code you're trying to affect is) most likely doing it wrong.

use strict prevents the fallback from lexicals to globals. If the desired scope isn't selected lexically with my, our, or state, nor fully qualified by a package name (which implies a global variable), then the error is generated.

require $target or require "file.pl" are legacy features that have some unexpected interactions with modern Perl, and some limitations in creating truly reusable modules. Instead, it is much preferred to create a package that can be imported with use; besides giving the caller control over the imported symbols, this also makes for a more robust module system that doesn't depend on the current working directory to find the modules. (Unless you have use lib '.', but that seems inadvisable at best.)

Python and PHP have similar global keywords which are similar to our, and as of Python 3, the nonlocal keyword allows for a lexical variable to be written to (instead of creating a local variable of the same name). PHP gained closures in version 5.3, but only for anonymous functions, through the optional use (...) clause on the function declaration.

Changing a Tablet's Active Area in Ubuntu Lucid

2011-02-19T19:01:00.000-05:00

The following information applies to Ubuntu 10.04 LTS, Lucid Lynx, with xserver-xorg-input-wacom installed to provide xsetwacom. This is about fine-tuning your tablet; if your tablet isn't working at all, you probably need bug #568064.

There used to be a wacomcpl program to graphically configure a Wacom tablet; this quit working with changes to the upstream project and/or the Tcl dependency, so it hasn't been working for me for some time. Before it quit working, I set up a script to call the xsetwacom command-line program with the desired results, so the loss didn't affect me. Mainly, I had adjusted the active area so that tracing a circle on the tablet would result in a circular shape on the monitor.

With a new monitor came a new need to reconfigure the tablet, without using wacomcpl this time. I ultimately created a couple of formulas to make a strip of the tablet inactive. Without further ado, these are the formulas:

 $x_offset = $w - ($h * $aspect) # narrower monitor

 $y_offset = $h - ($w / $aspect) # wider monitor

$aspect is the aspect ratio of the monitor, obtained by dividing where you write the colon. For example, 16:10 = 16/10 = 1.6. Alternatively, you can divide the width in pixels by the height, so a 2560x1600 display has an aspect of 2560/1600 = 1.6. (If you have square pixels, which practically everyone does because they're so convenient.) The monitor being narrower or wider refers to whether the monitor's aspect is lower or higher than the tablet's, respectively. You can calculate the tablet's aspect by dividing $w by $h; obtaining them is the subject of the next section.

$w and $h come from the actual tablet, which you can find easily enough. In these commands, $T represents your tablet's name, which you can get from `xsetwacom list dev`. In my case, there's a tool name attached, so it prints "Wacom Bamboo 4x5 Pen STYLUS" (among other things) but only the "Wacom Bamboo 4x5 Pen" portion is the actual device name. The first command simply resets the coordinates to cover the full tablet, just in case they have been changed.

 xsetwacom set "$T" xyDefault 1

 xsetwacom get "$T" BottomX

 xsetwacom get "$T" TopX

 xsetwacom get "$T" BottomY

xsetwacom get "$T" TopY

$w is BottomX-TopX, and $h is BottomY-TopY.

Armed with this information, you should now choose the correct formula from above, and substitute all the numbers. In my case, the top coordinates are both 0, so BottomX=$w=14720, and BottomY=$h=9200.

My old monitor was much narrower (at 1280/1024=1.25) than the tablet (at 14720/9200=1.6), so I used the first formula, thus:

 $x_offset = 14720 - (9400*1.25) = 3220

And to set that value:

xsetwacom set "$T" TopX 3220

My new monitor runs at 1920x1080, which yields 1.7778 for aspect. The monitor is wider than the tablet, so now I need the second formula:

 $y_offset = 9200 - (14720/1.7778) = 920

Now that the offset is known, it's a simple matter to set up. I just add it to the original TopY value (zero for me, so no different) and set that as the new TopY:

xsetwacom set "$T" TopY 920

Altering TopX or TopY means that the inactive portion of the tablet runs down the left or across the top. I don't really care where the dead zone ends up, so I chose the method that results in the fewest calculations needed. You could just as easily set BottomX to BottomX-$x_offset to move the dead zone to the right side of the tablet, or adjust both TopX and BottomX by half of the $x_offset to keep the active area centered.

Customizing Everything I Touch: The Lucid Login Screen

2010-12-01T19:59:00.002-05:00

This post is purposefully dense, due to my lack of time/typing budget. If you need clarification, please leave a comment.

Directions (shortcut):
you$ sudo -u gdm dbus-launch gnome-appearance-properties -p background

Directions (original):
you$ sudo -u gdm dbus-launch bash

  gdm$ cp wallpaper.png /var/lib/gdm

  gdm$ gconf-editor

  (set /desktop/gnome/background/picture_filename to /var/lib/gdm/wallpaper.png)

  gdm$ exit

The changes will take effect once all users log out (or possibly, the user on :0.)

Commentary:
The directions above copy the wallpaper to /var/lib/gdm (which is the home directory of the gdm user) to protect it against corruption. If you don't care, you can run the single command `sudo -u gdm dbus-launch gconf-editor` and set picture_filename to any file on the system.

In either case, dbus-launch is necessary so that there's a session bus running as the gdm user, which gconf needs to successfully run. Otherwise, AFAICT, it tries to connect to your own session bus, which the gdm user isn't allowed to do.

This edits the keys in /var/lib/gdm/.gconf, a config source set by /var/lib/gdm/gconf.path, which is wedged into the regular paths by inclusion from /etc/gconf/2/path.

Validity:
Tested on Ubuntu 10.04 LTS, Lucid Lynx; most likely, this applies to 9.10/Karmic and later Gnome 2.x-based Ubuntu releases. (IIRC, Karmic is when the new, non-themable GDM landed in Ubuntu.) The general philosophy probably applies to other contemporary Gnome environments, but the details may differ as there's a lot of Debian-ness in the GConf paths.

Autostart in Gnome: the missing docs [Updated 10/10]

2010-10-07T21:01:00.008-04:00

I've been trying to understand the autostart mechanism in Gnome 2 for a small program I'm working on. This may continue to be a supported system in gnome-3, since it seems to be standardized by freedesktop.org and not the Gnome "Let's just rewrite around our bugs and drop features" team.

Without further ado or bitterness, here's a brief but technical dive into the modern autostart system on Gnome 2, as observed on a Lucid Lynx system (originally installed as Intrepid Ibex, I believe.) [Updated: autostart itself is also a freedesktop standard.]

The autostart system is built on the aforementioned XDG Base Directory specification. Inside the config dirs are directories named "autostart", and these directories are filled with .desktop files. By default, a desktop file in one of these directories represents an application to be auto-started. The file may override this by including the line, "X-GNOME-Autostart-enabled=false". [10/10: Actually, according to the autostart spec, it should use a line "Hidden=false" to accomplish this. Furthermore, an application will not be autostarted by Gnome if the desktop entry contains an OnlyShowIn key without GNOME in the value, or a NotShowIn key which includes GNOME. This lets KDE add autostarts for its services without disturbing Gnome, because OnlyShowIn=KDE is set.]

So, there are places like /usr/share/gnome/autostart, /usr/share/autostart, and /etc/xdg/autostart that contain these apps for the system. Then, ~/.config/autostart directory (where ~/.config is the default value for $XDG_CONFIG_HOME) contains any modifications to the system files made in the Startup Applications preferences window (on the System > Preferences menu).

I suspect that if your only modification in the preference pane is to turn off autostart for a command, then the system adds a .desktop file for the command to your home dir that disables the autostart. If you then turn the command back on, the system must be deleting your personal .desktop file to revert to the system-wide one. This accounts for reports of "~/.config/autostart isn't right, it only contains things that don't autostart." I'm pretty sure I looked into this system in the past, briefly, and thought the same thing.

If you add a custom item like freshwall to the Startup Applications preferences, then it creates a .desktop file in ~/.config/autostart with the configured command, but no disable-autostart line. So this autostart directory must be the correct one. I see that Dropbox also leaves a .desktop file in there. (And if you sign up through that link, you get an extra quarter gig of space.)

Obviously, the autostart directories are searched in a specific order, and the first matching filename wins, with the user-specific directories scanned before the system-wide ones. I don't know if all the autostart services run in any defined order, or in parallel.

Lastly, these autostart entries are hooked into the Gnome session manager. The session manager sets the DESKTOP_AUTOSTART_ID environment variable when launching the desktop, and a session-aware program should use this ID as the client ID when connecting to the session manager. Note that this is not XSMP, but a Gnome-specific service exported via D-Bus on the session bus under the name org.gnome.SessionManager. (There is another post planned on this session manager interface; but if you want to dig into it yourself, I recommend D-Feet, which is packaged for ubuntu. [10/10: Ubuntu also documents this interface at /usr/share/doc/gnome-session/dbus/gnome-session.html, but I can't find it anywhere on the Web to provide a link.])

To summarize: autostart relies on the XDG Base Directory specification; items in ~/.config/autostart override items in the system directories, in accordance with the spec; and a .desktop file containing the X-GNOME-Autostart-enabled=false line represents a command that will not be autostarted by Gnome. An app that's being autostarted by the Gnome session manager, which wants to connect to it, should use the client ID stored in $DESKTOP_AUTOSTART_ID.

References for this post:

Groupthink

2010-10-03T20:59:00.005-04:00

I don't fit in; I never have. Some of it is by my choice, but in the tech communities that I've found, I still don't fit in, and I've often wondered why. It seems the short answer is right there in my blog description: "Frequently contrarian and occasionally cynical."

I tend to find communities once they get large enough to have formed opinions on things, and then I come in to disagree. On XML and Data Formats was essentially written to disagree with Reddit's opinion that JSON is Super Awesome. Which isn't too bad, in and of itself, because such an extreme opinion on Reddit's part probably is wrong, even if I have all the wrong reasons as to why. But over time, especially if I want to post frequently, I run the risk of disagreeing just to appear different and unique, even if by chance Reddit or HN happened to be right this time. (These communities tend to produce pressure in that direction anyway, since they hate "Me too" posts.)

I've also wondered about the tendency for people to disagree with me, when it's a provably correct fact. Someone will ask a question, and get three people answering with (also provably correct) method A. So I come in and mention that they might prefer method B. Finally, someone replies to me that instead of that, they could use method A. Well, obviously; it was mentioned three times already.

But, I think I've figured out that pattern, too. Where the group consensus is on one thing, then any alternatives (which is me, since I speak up when I have something Different and Awesome to say) threaten the group cohesion, and the outsider must be either driven off, or accept the group's opinion.

I noticed this happening one day to someone else when I dropped by a prominent Subaru Forester forum; except for a Subaru salesman who said that people who drove it loved the CVT, the whole forum piled on to argue that anyone buying a CVT is a complete and utter moron. Of course, the regular forum members are enthusiasts. To them, anything but a manual is evil, because they mess up your starting, drifting, and anything else you're doing out there in the mud; meanwhile, everyday drivers are snapping up CVTs off the sales lot. It's not that one point of view is invalid, or that the forum members don't realize they're talking about different scenarios—it's the existing group establishing solidarity against non-enthusiast intruders.

Experience is funny like that. When it's not personal, it makes the situation so much clearer. And once again, as I gain more experience, I come to appreciate its value more.

In order to avoid wasting time with them, and to pursue Happiness at the expense of Strife, I have blocked the tech news sites I frequent on my computer. And when I blog in the future, I want it to be something actually interesting and perhaps in-depth, like that post on the future toolkit at my old blog. Not anti-reddit stuff that they're not going to see, read, or agree with anyway.

Post Schedule Reduction

2010-09-18T20:52:00.000-04:00

I have decided to post high-quality work when it is ready. Writing weekly is making me rush out junk, which conflicts with my primary goal of improving my writing.

Dealing with Ambiguity

2010-09-14T19:31:00.000-04:00

Apple and their fans tend to view their products as the top of the market, with price and attitude to match (and this is helped by their competitors trying to undercut them with unrefined but cheap ripoffs). Yet they clearly market heavily, which suggests according to the Advertising Curve that there's room above Apple for a better product, for even more money.

Which naturally leads to the question: considering an Apple product like the iPod Touch, what would be better, and in particular, enough better that you could actually sell them?

My Ideal Player

The ideal music player would be able to connect wirelessly to access the music store, in order to buy or retrieve new music. It would support backing up and restoring music. To provide the ultimate choice in music, it would allow me to download a plugin to allow access to any music store, like Amazon or Magnatune. It would be able to be charged by induction. Finally, if it had Bluetooth to support any of these features, it would also support playing audio over Bluetooth.

In other words, my ideal player hardware is... an iPod Touch with an induction power input. It has all the wireless capabilities, and the rest is software.

The Problem with Software

It's always terrible. Today, Exaile is showing me two separate "Unknown" albums in the album list—one which sorts in with U in the normal way, and one which sorts as a separate letter after Z. Also, there has never been a way to roll all the DDR-related albums ("DDR 5th Mix", "DDR PSX Exclusives", etc.) into a single, virtual DDR album.

But that's complicated stuff. Before settling on Exaile on Linux and Foobar2000 on Windows, I have run into plenty of players that couldn't understand multi-disc collections, either sorting by disc number alone (tracks randomized), or track number alone (alternating discs). Again with media players, the ability to stop playing at the end of the current track is a rare feature. With RSS readers, I quit reading feeds entirely when I quit using KDE, because nothing else offered a stable, convenient reader with a "Next Unread" key.

Even the hallowed iTunes from the Temple of Apple falls short of letting a user do what they need to do. If I make a podcast, but don't author it so that iTunes knows it's supposed to be a podcast, there's no way to convince it to show as a podcast in iTunes or on the Nano. Don't worry; we are smarter than you.

The Problem with Customers

People love cheap.

Even if the iPod Touch is my ideal hardware, I'm still uncertain about the price tag. I could settle for a non-Apple player for that kind of money, and have enough left over for a Kindle. It's sufficiently hard to estimate the value of these things, without access to any of them, that it makes it quite difficult to decide whether I would, in fact, be satisfied with owning one for that price. This estimation of value is not helped by the apparent inability to find what apps are even available in the app store without already owning the device. It may have an app for what I want (direct purchase from Amazon), but this information is hidden.

Nor is the value enhanced by the fact that I wouldn't normally be able to run iTunes on my Linux machine, and Apple has probably changed the interface again to break compatibility with Linux iPod software. Assuming, of course, that any free software even supported the previous iPod Touch models.

The Sum of the Vectors

Since people don't like to pay for software, and stable, functioning-as-designed software is expensive enough as it is, it's rare to find anyone putting more effort into their software. When faced with the messy vagueness of the Real World, it's so much easier to file it under Unknown than to work at all on designing or implementing a better solution.

Bias

2010-09-05T20:57:00.002-04:00

The topic of alternative keyboard layouts inevitably comes up from time to time on programmers' forums: is Dvorak really better than Qwerty? Is it objectively proven? If nobody has proven it, why would anyone switch? Also if it's unproven, why aren't there any satisfactory studies?

It is this latter point that fascinates me. We have studies by August Dvorak, inventor of the Dvorak keyboard layout, which purportedly show that it's faster than Qwerty. We also have studies by Strong, commissioned by the GSA in 1956, which claim to show the opposite; supporters of the Dvorak keyboard claim that Strong was biased in favor of Qwerty, and most accept that Dvorak may have been biased in favor of his own creation.

How does one design and execute an unbiased study in a world where only the biased even care about the outcome? If such a study were performed, would the 'losing' side accept the results? Or would the study get ignored (or contested) for all time? If an alternative keyboard were, in fact, proven to be better than the existing choice, how many people would actually switch?

Other things to consider besides raw metrics

Right now, Qwerty enjoys an enormous network effect. It's easiest to be proficient on only one keyboard layout at a given time, and you can effortlessly type on everyone else's computer that has that particular layout set. (Qwerty on computers enjoys the additional advantage that if you don't touch-type, you can still type in the letters you want, because they match the letters printed on the keycaps.) The vast majority of "other keyboards" are Qwerty: not only are most people not typing at the computer frequently enough to switch, but the majority of the ones who are were trained in Qwerty, and continue to use it.

Also, the majority of other people can only handle Qwerty keyboards. If I loan my computer to someone, I have to remember to set it up as Qwerty for them, or they'll be really annoyed with me.

To back up these wild assertions, I can think of only three other people IRL who actually use Dvorak, and one who tried it until it killed his Qwerty skill, which he needed to repair everyone else's computers. I would consider them all techies, although one of them is more aligned in art and cooking than machines. This is why I think it's safe to say that a minority-of-a-minority use Dvorak. (And it's the popular alternative: I'm the only one I know IRL who has even heard of Maltron, QGMLWB, or Colemak. If you don't count me telling Eric about Colemak.)

Another interesting thing a modern study of Dvorak could consider would be coding: in Dvorak, compared to Qwerty or Colemak, the punctuation is rearranged, and programming text relies a lot more heavily on it than English prose. I find Dvorak's choices a lot more suitable for typing arrows in PHP and Perl, since they alternate hands, and don't require a same-hand jump from the number row to the bottom row. Although that means other frequently-used punctuation like brackets and braces move up into the number row, they don't seem to run into the same slowdowns as the arrows did.

Finally, the actual keyboard in use may affect the results. My wife has difficulty typing on my MS Natural 4000. (I had difficulty typing on it with the Ergonomist Approved Reverse Slope board installed, which also made my wrists hurt worse than they ever had before. Quite possibly, I needed a drafting chair to keep it from pushing my wrists into the sky.) Any modern test of keyboarding prowess is going to have to let people use their own keyboards, or give them time to train exclusively on the standardized keyboard hardware used in the test.

The market argument

Another reason Dvorak vs. Qwerty seems to come up a lot is that it is the primary example of a technology that is considered technically superior, that has failed in the marketplace. It's the quintessential example of the first-mover advantage leading to the entrenchment of Qwerty, and of the fact that technical superiority doesn't matter.

Unfortunately, neither of these theories are laws of the universe. There are plenty of examples of the first mover failing. Just ask Sega. Or perhaps Apple: the company that brought WIMP environments to the masses, only to be out-executed by the Amiga for a while, and then the PC. (Apple got the last laugh vs. the Amiga, though: they successfully switched to PowerPC, to OS X, and once again to Intel. They also are managing/have managed the 64-bit transition quite smoothly, at least from my vantage point outside the system.) For all their prowess, though, the only Apple product leading the market years after introduction is the iPod—it's too soon to tell for the iPhone, although their marketshare is uncertain in the face of Android. The iPod was in the unique position of being both desirable and DRM-locked, so again we see other factors at play besides "first" or "best".

A prediction

I doubt that any sound study of Dvorak is going to show such incredible gains in speed as Dvorak's studies did, but on the other hand, I don't think that they're going to show incredible losses, either. Such a study would also need to take self-reported comfort into account, in case Dvorak typists are self-limiting in speed in order to self-limit pain. Life is rarely perfect and simple.

Pagelib Retrospective

2010-08-29T20:59:00.001-04:00

I have always been in love with magic: one of the first languages I taught myself once I was old enough to understand what I was doing was Perl, and it fit me like a skin-tight sci-fi spacesuit that we thought we were all going to wear in the future, back in the 1980's. Steve Yegge's rant about Perl was entertaining, but I can't say as I identified with it much, since references weren't hard for me, nor do I have his problem of forgetting everything when I don't use the language for a while. Maybe I don't drink enough.

Still, being in love with magic and DWIM doesn't mean that there aren't any hazards. It's rather like fairy-tale magic, which can result in nasty, surprising, unexpected consequences if used without the utmost care. This is a story about me exercising a complete and utter lack of care, utmost or otherwise.

Pagelib was a templating system I devised, which used straight PHP in the templates, as well as being embedded in a larger PHP system. One of pagelib's design goals was to give the template programmer (also myself, and sometimes Brian) nigh ultimate power! to create without limits. Because as everyone knows, limitations are absolutely horrible.

The principle of operation of the system was simple: the caller set up some variables, and maybe a callback if the template being invoked needed one, and then launched the template. Of course, this meant that the caller had to know everything the template could possibly want, as well as whether it would ask for another template to be embedded. If it did, then the callback would need to launch another template with another set of variables. Theoretically, it could also set up another callback to continue to darker depths, but I'm not sure this capability was actually used.

Meanwhile, I didn't really understand proper template design, so instead of passing in a domain object and letting the template pull data from it, I extracted and computed all the property values that the template would need, and assigned them all to individual variables. Neither "$user->first $user->last" nor "$user->getFullname()" would sully my code! "$user_fullname" all the way! An unfortunate consequence of this decision was that the code naturally wanted to be "$data['user_fullname']" because the variables were given to the page as an associative array in $data. But I didn't like typing $data[''] all the time, so the templates all called extract($data) at the beginning.

Pagelib was also designed as a strictly one-pass system, since I hadn't yet convinced myself of the need for multiple passes to allow fully self-contained modules. As a result, the uppermost caller needed to set up any CSS or script inclusions that any template would need, totally breaking the self-contained modularity that the lower templates were supposed to provide. Which is also (partly) why all the CSS for individual pages ended up in one giant CSS file.

Finally, I'm pretty sure that pagelib took some measures to work around the fact that calling include() in PHP runs in the context of the code calling include, which means it will have access to $this or self:: if called from inside a class method. This is why it only offered nigh ultimate power, rather than the real thing.

Everything about pagelib makes me cringe today.

Ur Doin it Wrong

The variables being passed to a pagelib template were effectively hidden: only the template to be called next was being constructed at any time, so variables to nested templates were completely invisible. Even from the point of view of a template, only the variables passed by the given execution path were visible. Of course there was no documentation, because we could always grep the code... right?

The power exposed by the doCallback() system also turned out to be basically useless. In almost every case, the callback was no more complicated than a simple "include page-specific template" instruction, but since this happened in the Ultimately Powerful Callback, the code was essentially six lines of boilerplate everywhere. The template could even pass arguments into doCallback, which would be passed along to the actual callback function, but that turned out to be even more useless. The upper level page knew everything anyway, since it had to set up the callback, so why bother passing a message from it to itself via the template? Yo dawg....

I'm not entirely convinced that PHP was the best choice of template language, either. Walking that path either requires short tags to be on, or litters the template with <?php echo ?> every time a variable needs printed. And even with short tags, the bracket-question combinations are somewhat difficult to type so frequently.

Pagelib's Legacy

Ultimately, Pagelib and its larger system became a victim of the software Peter principle. Quite scorched, and with newfound humility, I designed a much simpler system (unimaginatively named "Output") for my next and final project with that employer. Output was still a single-pass system, so the caller of a template still needed to know that it had a slot for including another template.

But this time, the association was done by name, not implicitly in the control flow. Also, Output's variable system was designed that each template had its own set of local variables, and inherited its parents' as well. So when the top-level code assigned variables to a template, it was clear where they would be visible, and that they were being made available.

Output's simplicity and clarity was the seed of my acceptance of the Zen of Python.

Quickie: The Cross Platform Quandry

2010-08-22T20:56:00.044-04:00

When an app runs on multiple platforms, it seems like there are two basic designs that they can choose to follow. Either they try to integrate with each platform, or they try to appear the same on each platform.

It's not clear to me that either of these ways are correct. Obviously, in the case where the differences between the platforms are hidden to the greatest degree possible, the app doesn't match the look and feel of platform-native applications. This is the situation with old Mozilla builds that drew virtually all the UI themselves: the chosen GTK+ theme had no effect on Linux. An app that looks "the same everywhere" can look like it belongs with at most one platform.

Meanwhile, since Mac users in particular are vocal about their dislike for using non-Mac-like apps, most modern cross-platform apps have switched to trying to integrate with each platform. Firefox has a different theme and adjusts its button order on the various platforms; likewise, GIMP for Windows switches its button order to match the Windows custom of putting Cancel in the corner instead of OK.

As a habitual user of Firefox, Gimp, and Pidgin on both Linux and Windows, this second approach turns out to be less-than-ideal for people who frequently switch platforms. In this case, the "same" app behaves differently on the different platforms, and I have to remember which platform I'm on so that I can hit the correct buttons. Of course, this doesn't always happen, so frequently in Windows Gimp, I spend a couple of minutes selecting the perfect color, only to hit Cancel by mistake. Another frequent source of mistakes is the way that Pidgin's tray icon toggles the buddy list with a double-click on Windows, but a single-click on Linux. I'm frequently opening-then-closing Linux Pidgin with a double-click.

One last problem with writing a cross-platform app for Linux/BSD specifically is that these are not all-in-one platforms. An app like Firefox that's written in GTK and follows the Gnome HIG is still going to fail to integrate with the KDE desktop on the points where their respective conventions differ.

Given these difficulties, it's not surprising that most apps aren't cross-platform. The ones which are face the choice of whether they should integrate with platforms and appear different in the details to people who work on multiple platforms, or whether they should look the same everywhere and look alien or second-class on each platform.

Perfectly Flexible

2010-08-15T20:58:00.001-04:00

Sometimes, a stakeholder doesn't like the idea of requiring a Programmer to make lengthy changes to a project's code if the requirements were to change at some future date. The proposed solution is always simple: The System just needs to be more flexible.

Having been the unsuspecting Programmer on a couple of these projects, I can report that the simple solution does not work.
The typical pattern was for the interested stakeholder to think up a few possible directions that they would like to take their business/software in the next version or two. These would be delivered as kind of soft 'keep this in mind' requirements. Doing so necessitated additional flexibility above what the base system would normally require, and the complexity added by this flexibility increased the amount of code to design and write. In effect, a down payment was made on a future version or two, but only the initial version was actually delivered.

This situation would have been fine, except that when the future arrives, it has the unpleasant habit of taking a completely unanticipated path. Frequently, this would be one that would violate some key assumption that was never supposed to be possible in the original system. Which, of course, meant that the assumption was coded right into the system with no thought for flexibility at all. The down payments on the other features went to waste, and no time was ultimately saved.

Of course, the stakeholder seemed to see this as a failure to be sufficiently flexible. It made me think, though: how much flexibility is required in order to avoid future surprises from causing a major re-engineering effort? What is the ultimate limit that flexibility can be pushed to?

Lisp again

It seemed like the best way to make changing the system as easy as possible would be to split the application into layers. In a shopping-cart system, one layer would provide things like invoices, line-items, shipping cost estimation, and so forth; and the next layer up would link these facilities together into a cohesive whole. What would that upper layer consist of?

Why, that would almost be like source code to the base-level interpreter, and all those lovingly-crafted objects like the invoice would appear as something akin to host objects in Javascript: pre-populated values that provided access to an otherwise unreachable set of functionality. As a speed hack, both layers could hypothetically be in the same language, but then it would take some discipline to keep them cleanly separated. Well, it would've taken my 24-year-old self some discipline, anyway.

Thinking of these layers as engine and script, I realized I had actually seen this before. These were domain-specific languages, and in a shocking parallel realization, UnrealScript was also a domain-specific language. That old, oft-repeated advantage about creating mini-languages in Lisp suddenly began to take on real meaning, once I could see that these languages were so useful that they were implemented outside of Lisp, too.

Worse, the Java+XML world that I was always making fun of, being way too smart to bother with such obviously silly nonsense, began to make sense. In the Kingdom of Nouns, Java classes implemented domain-specific semantics in order to execute an XML script. Java's XML facilities provided a standard, runtime-controllable reader for languages written in XML syntax, just like Lisp provided runtime control for its reader of S-expressions. Stupid Java, not being dumb after all.

After those revelations, it seemed that the ultimate level of flexibility is also a domain-specific language. As long as the necessary primitives are in place, anything computable can be computed in the language. (Although this is sometimes a uselessly academic distinction, if the operations are too hard or cumbersome to be usable.)

I didn't get to try out my newly-discovered ideas at the time, and I haven't since; back then, I preferred the devil I knew, and in most of the places I have worked, the management has not been nearly so interested in technical awesomeness as simple, maintainable code. I wonder how things would go if I tried to create a DSL today.

On XML and data formats

2010-08-08T20:55:00.010-04:00

In many discussions of XML, there seems to be a faction of programmers who are completely dead-set against XML. They'll insist on JSON, or YAML, or any other cool technology that isn't supported in that 5-year-old version of whatever language your company is still running. The usual complaints leveled against XML-based formats are verbosity and the complexity of the DOM. (Sometimes, leading or trailing whitespace on element contents in pretty-printed XML will bite a project, but this never seems to come up in internet flamewars.)

The really young, or maybe just incurably naive, programmers will even chime in that anything that can be done in XML can be done, or even done better, in JSON. I even thought this once, until I tried to write a generic XML-to-JSON converter, which showed me how wrong I was.
Ultimately, I learned that XML—based on SGML, perhaps with some lessons learned from HTML—is at its heart a document format, and it makes the most sense when used to mark up document text. XML tags and attributes contain all sorts of useful metadata, and the angle brackets isolate it from the core text so well that something intelligible may still come out if you strip all the tags and examine nothing but the element content.

Corollary: if something useful doesn't result after the XML tags are stripped, then XML was not the optimal choice of format. Things like XML-RPC come to mind for this. In those cases, XML was chosen for familiarity more than anything else.

When I was attempting to write my converter, I started simple. I figured out how to handle <tag attribute="value">text content</tag> without collision. In the upper hash table, "tag" would point to another hash table with keys for each attribute, plus a special non-colliding one for the element's content. Then, I asked myself, "What happens if I need to encode '<strong>some <em>great</em> markup</strong>'?"

Answer: My JSON structure would be a reimplementation of the DOM tree. To faithfully represent the XML, I would have to know how many children an element had, what those child types were, and what order they belonged in. To extract a given node's text content, I'd have to visit all the children of that node and join their text content together, in order, just like XML. (If the XML API has an innerText call, it's just a convenient way to ask the API to do the exact same task.)

I was forced to conclude that, if there was no way to convert XML to JSON other than to create a structure that describes every nuance of the XML source, then XML must be strictly more expressive than JSON. The fact that you can represent XML in JSON is only interesting in an academic manner, similar to how you can write programs with a few fundamental operations—but a high-level language with a broad standard library is generally more productive.

Once the difference between XML and JSON is understood, then it becomes much simpler to assess a problem and determine the correct approach. Are you sending data structures, such as function names and parameter lists, back and forth? JSON-RPC is the more natural choice. Do you need to mark portions of a text with links, formatting, attribution, or other information that spans arbitrary ranges of the text? XML will probably be easier to handle.

Occasionally, the best solution may be passing a JSON data structure, with one or more values containing XML as string data, to be parsed separately from the JSON at the destination. Sometimes the cry for "Only one format!" for simplicity's sake actually makes the single-format solution more complex overall. Maybe there's a cost to loading both XML and JSON libraries, but in general, costs should be cut only when they have been proven to be excessive; otherwise, it is too easy to be bit wise and word foolish.

Author's Note: The term "DOM tree" in this post refers specifically to the DOM as a conceptual representation of an XML structure, not the Java-oriented API of the same name. The DOM API is a terrible way to work with the DOM tree in any other language, and the author's mild fondness for the tree should not, in any circumstance, be mistaken for condoning the API.

Functional vs. Object Oriented

2010-08-01T20:59:00.006-04:00

By the time I reached my first programming job, I had learned a number of languages well enough to program in, and thought of myself as a pretty good programmer. Then I met Brian, who frequently talked about Lisp, but had trouble articulating the advantages himself. (This turns out to be a common problem.) He introduced me to the SICP videos, and I found Slava Akhmechet's "The Nature of Lisp" essay on defmacro.org at about the same time. Certain things about Perl and Javascript suddenly made sense, filling me with newfound wonder.

What I thought I knew about OOP as taught in college—Encapsulation, Polymorphism, and Inheritance—seemed like a sham.

I looked harder at Javascript and came across Douglas Crockford's module pattern, which essentially uses lexical scoping to hide data, instead of an object system. Inheritance, too, turns all weird in Javascript, since it's based on prototypes instead of classes, to say nothing of Perl's build-your-own system with the conventional approach of using a hash to store all the fields. Polymorphism seemed to be nothing different than passing first-class functions.

Perhaps I should go into more detail about that. With polymorphism, the code actually called when a particular object method is invoked depends on the type of object. Different objects that implement the same method signature are interchangeable, although in statically checked languages, they may need to share a base class or interface. But, what is different about this as compared to passing a function directly? Both cases store different code-to-be-called into the same variable, and both cases have to agree on the call's signature. Objects can have internal state, but passed-in functions can achieve the same by closing over lexical variables.

The only thing objects really add above a function is the ability to call several different methods on the object. Even so, this feature is easily replicated by passing a hash table instead of a function, where the keys are strings forming method names, and the values are the corresponding function. Again, some languages provide more guarantees about the structure of an object than the structure of data pretending to be an object, but in others, the two are indistinguishable. You wouldn't be warned until runtime that an object method could not be found.

Functional and OO programming seemed to be equivalent. I demonstrated this by producing a primitive object system using only Perl's functional programming constructs, along with a class whose objects reasonably represented closures in PHP 5. (This was long before PHP 5.3, where they added actual closures, and also a __call method to allow objects to pretend to be functions.)

I eventually discovered Jonathan Rees' thoughts on the meaning of object orientation, which justified my appraisal of OOP-as-it-was-taught-to-me as the Emperor's new clothes. It agreed that that kind of OOP was not the only possible manifestation, nor was it necessarily the One True Way.

FP clearly had a different cadence to it than the OOP it was equivalent to, but that was more of a natural consequence of reducing friction between naming and using functions. Things like map can be written in OO style, but it's cumbersome enough there that nobody does so unless backed into a corner.

Recently, I've been reading about game coding, and how there is very little in the way of OOP, because of the need to keep caches well-utilized. Iterating through an array of x-values, where each index represents a different game object, ends up being faster than iterating through an array of objects, since there are many object properties that need skipped over. (Also, if objects are of differing length, the accesses are harder for hardware to predict.) Unused properties in a loop represent memory that was fetched and paid for, but wasted.

Thus, game coding really strikes me as data-oriented programming. There may be an "object design", but memory is laid out for maximum speed and parallelism of data access.

Where does "procedural programming" fall? I don't think it's a useful distinction anymore, because I don't think there are any purely procedural languages left in wide use. (Niche use, perhaps, like legacy COBOL systems, or physicists writing FORTRAN. But not for anything in my slice of the programming world, certainly.) Even lowly C offers function pointers, which allow for both OOP and FP idioms, as the case may warrant. Pretty much any GTK+ program uses both styles—FP for event callbacks, and OOP for the widgets (and possibly data). C can't, AFAIK, pass pointers to lexical variables outside of the lexical container, so GTK+ has a userdata parameter threaded through every event call to simulate closure, but other than that, the illusion is fairly complete.