Cooking up Dishes
by Jonathan C. on Aug.08, 2010, under Aspect, Web programming
I’ve just begun building the messaging model for Aspect, and I’m developing it as a separate, reusable Rack-based framework. It’s called Dishes, which is an extraordinarily lame pun derived from “asynchronous” sounding like it has a “sink” in it. So far things are looking good, but there’s a lot of work still ahead!
As it happens, I read up a little on BOSH, and it uses exactly the same two-connection model I described in my last post. It’s interesting reading and I’m sure to take a lot of inspiration from it. I don’t really want to adopt BOSH in its entirety because I think it’s way more than I need; Dishes is probably going to be fairly small.
Something cool I’m trying in Dishes is putting each request in its own Fiber. Since Aspect is going to be extremely I/O bound, it makes sense to re-invert the evented model using fibers to make handlers feel more synchronous. That will make it easier to use Dishes.
Anyways, that’s enough late-night rambling for tonight. When I have a simple working Dishes example, I’ll post again and explain how everything works. Meanwhile, you can always take a look at the current code. I even have a basic goal application to aim for.
Comments always welcome!
~Jonathan
Aspect’s messaging model
by Jonathan C. on Jul.28, 2010, under Aspect, Computers, Programming
(Before I begin, I should warn you that I haven’t had much time to actively work on this yet.)
I found an article about Starbuck’s method of handling coffee orders on Hacker News today. I was surprised, because I had explained this exact concept to some friends already, but using Pat & Oscar’s instead! This kind of query-response architecture strikes me as the perfect model for Aspect, because everything you do constitutes a request, and everything Aspect tells you is some form of response.
The Pat & Oscar’s analogy is a little different from the Starbucks one, though, and it highlights a few key points. If you’ve never been to a Pat & Oscar’s, it works like this: You go to the counter, make your order, receive a number, sit down at a table, and place the number card in the little holder on the table so they can find you. When the order’s ready, they come to you with the food. Importantly, you can make multiple orders and receive multiple numbers, and the orders will be served whenever they’re ready.
Now, how do you apply this to web communication? The classic HTTP protocol has a strictly blocking request/response format, meaning that every request must wait until the response is sent before you can reuse the connection. Most browsers have a cap on how many active connections you can have, and the bare minimum is IE’s two connections. So we need to make this work by using only two connections.
The solution is to use one connection for making requests, and the other for receiving results! You keep the results connection open constantly, and use the other whenever you make a request. The request connection must be closed as quickly as possible, so you just return an “order number”, a job ID. When the job is done, it’s pushed down the response connection along with its job ID. Then you re-connect the response connection so it’s constantly open. The approach the response connection is using is called long polling. Instead of polling periodically, the server hangs on to the connection until it has something to send.
I believe that this is a powerful approach to Comet-like communication. Unlike pure long polling – where the server simply holds the connection instead of responding immediately, and the client doesn’t need to care – it does require some infrastructure on both sides of the network gap. But if it can be abstracted properly, it should really be very easy to use. It’s just a layer you can work on top of.
I’ll be working on this as time permits, and of course I’ll open-source my work on this (though probably not Aspect itself) after a certain point. I’m a firm believer in open-sourcing platforms so everyone can benefit. If anyone else is interested in helping though, I’d be glad to make it public sooner! I think this is a useful model that definitely has applications beyond Aspect.
~Jonathan Castello
Lots of projects, not enough time
by Jonathan C. on Jul.14, 2010, under Programming
I’ve got a lot of interesting projects I’m dealing with right now, but I just don’t have enough time in the day to give each one the attention it needs. And naturally, some projects take more priority than others. Some of them are pretty cool, though, so I figured I’d list them here.
Aspect – The big one. I’ve been tinkering a lot with the MUSHclient source in my spare time instead, because it doesn’t take as much focus as building a whole new client, so at least I’m getting more experience with MUD clients. But this guy deserves a lot more attention from me.
Unnamed Rails project – This one takes priority, because I’m building it for my father and I actually get paid. It’s getting closer to completion, but I’m no designer, so getting the CSS just right takes a lot of my time. Inexperience rules the day here (but I’m not a total kludge!).
MUSHclient plugins and libraries – I’ve had plenty of ideas here, and a good amount have actually been completed. My newest project is a highlighting library which makes it much simpler to “paint” parts of lines different colors and styles without manually creating a trigger for each change, or worse, gagging and re-echoing the line with your changes. It should be simple to implement if I can just sit down and get down to business.
Misc. jobs – I do a lot of odds and ends for my father, like setting up Webalizer for a website. I’m very new to the whole Linux scene, so I (get|have) to learn something new with almost every job. (With Webalizer it was cron jobs.)
And then I do things in Real Life™ and in online communities like the MUSHclient forums. Of course, most of this is self-inflicted, and I enjoy everything I do; I’m definitely not complaining. I just come up with too many projects for my own good. 
…As a rather funny side-note, I remember learning C++ and having no idea what I should do for my next project, and just kind of muddling around. Now I find myself with an excess of them. Life is bittersweet.
A little update on Aspect
by Jonathan C. on Jul.02, 2010, under Aspect
So it’s been a while since I’ve posted about Aspect. I was talking with a friend who had read up on Aspect here, and it seems I had forgotten to mention a rather key change I made: switching to Ruby. So I’ll take some time to explain exactly what’s up with Aspect right now.
Previously, I had explained that I was using Python, using the Tornado server. Unfortunately – and sorry, Python-lovers – I can’t stand Python. Development got to a point where I couldn’t make any progress because I was fighting the language. I had originally planned on using Ruby, in fact, but at the time I couldn’t find any Ruby libraries that did what I wanted.
Apparently I just didn’t look in the right place, because once I started looking again, I immediately found EventMachine, a Reactor-based library. EventMachine is pretty awesome, and suits my needs perfectly. But I still needed an HTTP server, and preferably one that could handle lots of concurrent conncetions. Thin fits the bill nicely. And lastly I needed a framework that could deal with holding onto connections until I have content ready. So far, Cramp handles that perfectly well.
Recently I’ve tossed Nginx into the mix, since I’m hoping I can have multiple Thin workers behind a front-end Nginx. Nginx also uses the event-based model, and it doesn’t give a thread to each connection. That would bring my server an early death, I think.
So that’s my network pipeline. From start to finish, it’s built to handle multiple concurrent, persistent connections, and it should be scalable too. Next time I’ll post about the messaging layer I’m building between the user and Aspect.
First Thoughts on Nginx
by Jonathan C. on Jun.26, 2010, under Aspect
I’ve been experimenting a bit with Nginx lately. I haven’t had a chance to actually benchmark it – nor, in fact, do I have any idea how I would actually do that – but I’m pretty happy with the learning curve. My only problem was with getting Passenger working, since I use RVM to manage my rubies. It turned out to be as simple as “rvm ruby –passenger” and setting the passenger_ruby configuration option.
Nginx seems to be a better choice to run Aspect behind than Apache, because Nginx is built to manage simultaneous connections. Just like Thin, it’s event-driven instead of thread-based, and I’m going to have plenty of long-running (and recurring) connections.
Now that I have a local server running on my computer, too, I think I’ll have an easier time juggling my projects. I recently discovered /etc/hosts, too, and I just can’t describe how much nicer it is to use “aspect.local” instead of “127.0.0.1″ all the time.
In other news, the 9 key on my laptop broke. You may also know this key as the “left parenthesis” key. Now… If you’ve done any programming at all, you should know just how important this key is. Gah!!
Aspect (plugin infrastructure)
by Jonathan C. on Mar.23, 2010, under Aspect
This is still entirely conceptual at the moment, so don’t get too excited. 
Also, if you’re not too into technical stuff, you might want to read only the first paragraph.
I’m planning on making Aspect extensible to a certain extent. Users will be able to create plugins written in Lua, run on the server in a highly sandboxed environment, with a specialized API so the plugins can do useful stuff. One such API will probably be a widget interface, to create and manage HTML widgets on the browser side. This means you could create plugins that manage a visual set of stat gauges, or show a visual map, etc etc. in a similar vein to MUSHclient and Mudlet.
Now, plugins will be run on the server. This is because (a) I don’t think you can run Javascript sandboxed very easily, (b) it makes it much harder to communicate between the plugins and Aspect and the MUD, and (c) Lua is just so much nicer to script this kind of stuff with. For speed and manageability, I’ve also decided that all plugins will share the same Lua interpreter space. You have to be careful with security in all cases here, because third-party scripts will be running on the Aspect server. You don’t want arbitrary plugins to share player info, or clash with eachother in other ways.
Obviously I have to only give scripts access to utilities I want them to have access to. But what if the plugin goes into an infinite loop, or just takes too much time to execute? In the best case, it will lag everyone else connected to Aspect. That’s clearly not a good thing!
There is functionality in Lua to set a “hook” which can be called automatically every so often. Theoretically, I can use this to halt execution, go do other stuff, then come back to the plugins. But we also want some way to keep plugins completely separate, and we don’t want any one plugin to hog all the time so no other plugin gets a chance. It’s a bit of a sticky situation.
First of all, we can isolate each plugin by starting each one in a coroutine, and setting the coroutine functions environment to its own special table. Coroutines are like threads, but they’re cooperative rather than preemptive. That is to say, a coroutine has to say “Okay, I’m going to take a break now” or “Hey, you, do some work” explicitly in order for another coroutine to begin. And changing the environment keeps the plugins from sharing any state.
Now we’re down to one more problem (or at least the last major one for now). I mentioned before that I don’t want any one plugin to hog all the time. I also mentioned hook functions. It just so happens that you can set a hook function separately for each coroutine. If we could yield from that hook, it would be almost like preemptive threads. And while you can’t yield from a hook set with debug.sethook (in Lua), you can from a hook set with lua_sethook (in C). So I just have to figure out how to write C-based hook functions to yield every so often.
So that’s a rather verbose explanation of how I want to implement plugins. Well, it’s 2am and I needed to talk. 
Tl;dr summary: every plugin will be a completely isolated Lua coroutine that shares execution time with other plugins. Plugins will be able to do a lot of interesting things, like create and manage visual widgets on the user’s browser, like status bars and maps and stuff, and could even augment/replace the default output window and input bar. It’ll be extremely extensible.
Now if only I could figure out how lua_yield works.
~Jonathan
Project Aspect
by Jonathan C. on Mar.18, 2010, under MUDs
I’ve been working on a new personal project lately, which I call Aspect. It’ll a web-based MUD client, but without Java or Flash. (A bit like PHudBase, actually, but built differently.) I haven’t done much with it yet, but I’ve got a local webserver running using the Python-based Tornado server. Tornado is great because it’s built specifically for handling AJAX long polling, which is what I use to communicate between Aspect and the browser. I’ve never used Python before, so it’s an interesting experience.
I’m planning on making the client extremely customizable by way of plugins, drawing from my ongoing experiences with MUSHclient. I’m also taking philosophical inspiration from Mibbit, which seems to work similarly (though the focus is completely different). With some MUDs, though, there’s a bit of an issue. Many MUDs disallow multiplaying, which is often tracked through the user’s IP address. But all users playing through Aspect will appear to connect from the same IP address; namely, the IP of Aspect itself. This is something I’ll have to work out eventually, but I have some ideas.
More to come eventually. I’d like to keep most of the details under wraps for now, and there are a lot of details.
~Jonathan
MUSHclient, cont.
by Jonathan C. on Mar.07, 2010, under Uncategorized
So… Lately, I’ve been playing around a lot with MUSHclient’s source code. It’s pretty interesting to see how a program I’ve used for a long time actually functions (even though he uses the Whitesmiths indentation style, which I cannot abide. I’ve always been a fan of the Allman style, myself). I’ve actually made a few relatively minor contributions, which I’m pretty happy about. 
I also contributed (and I suppose maintain?) a pair of Visual Studio 2005 solution and project files, and helped ease the compilation process under it. Nick uses VC6, which is really old these days, but it’s what he used when he first started, and he’s not exactly willing to shell out hundreds of dollars to update. (I can’t honestly blame him.)
At any rate, there was a discussion about a new protocol Nick had scratched out as a hidden-data-transfer medium for MUDs, similar to ATCP and ZMP. It came about that JSON was a perfect fit for the protocol’s semantics, so I modified MUSHclient to add a scripted interface to a previously-created JSON library (called json-c). I’m really happy with how it turned out. I mean, I only wrote the glue between Lua and the library, but it was pretty interesting designing the glue nonetheless. The interface hasn’t been officially added to MUSHclient (yet?), but it’s in my GitHub fork of MUSHclient. (You can find my additions here, if you’re into that sort of thing)
Here’s an example of the interface in action:
chunk = json.encode{1, 2, 3, {foo = "bar", baz = {10, 20, 30}}}
print(chunk.to_json())
Output: [ 1, 2, 3, { "foo": "bar", "baz": [ 10, 20, 30 ] } ]
Admittedly, that’s pretty boring. The glue code is more interesting, but the end result is still useful. In my (admittedly very brief and minor) tests, json.decode():to_lua() was faster by 2ms than loadstring()(), which was a little surprising. JSON also has the advantage of being rather well supported by many languages, at least indirectly. The protocol specification Nick originally proposed used Lua itself as the data format, which worked but brought along some odd issues like defining or calling functions within the data. I rewrote a Lua protocol snippet as JSON and compared the two, and they were nearly identical, so there’s really not much to lose by using JSON instead.
With that pretty much done, now I’m working on converting MUSHclient’s help files from .hlp to the .chm format. Is it a lot of work? You bet. But at some point, somebody has to do it… But I’m using an sqlite3 database, Lua, and ltp to autogenerate the files, so at least it’s entertaining.
~Jonathan
MUSHclient
by Jonathan C. on Feb.14, 2010, under MUSHclient
As you may know, I often play an online text-based game called Achaea, which is one of the deepest MUDs I’ve ever personally played. And unlike most graphical MMOs, there are a variety of clients you can use to connect to any given MUD. Some of the most popular clients include zMUD or Mudlet – and you could even connect through plain telnet – but I prefer MUSHclient by far.
MUSHclient, authored by Nick Gammon, has been in existence for over ten years, and is still constantly developed and supported to this day. One of the main reasons I enjoy it so much is because it exposes an incredible amount of functionality through a scripting interface. You can use any of a number of languages, including Lua, VBscript, and Jscript, so long as they can be installed into the Windows Script Host. (Lua is an exception; support is baked directly into MUSHclient). Plugins, consisting of XML files containing triggers, aliases, script sections, and other items, can be created and distributed, which helps players of a specific MUD to create useful extensions specific to their MUD and pass them around.
MUSHclient is also open-source. You can download the source, compile it, tinker with it, and even (if you so desired) distribute your own version. Nick is great about suggestions, though, and more often than not, sensible suggestions will be incorporated into the next version. And since MUSHclient’s source is hosted on GitHub, you can easily fork the source and create a patch yourself, saving Nick the work if he decides to add the idea. (If he doesn’t, you’ve got your own custom MUSHclient build anyways)
But on to the main point: MUSHclient is also my primary development environment. Sound strange? Well, maybe. But I’ve done more concrete programming/scripting since I began using MUSHclient than ever before, meaning I’m getting a load of good experience even while I have fun with it. I’ve written a simple state machine for telnet sequences. I’ve written a serialization module to pass data between plugins (which might even be written in different scripting languages). I’m even working, albeit slowly, on a GUI framework that takes full advantage of MUSHclient’s “miniwindows” functionality, which allows you to draw graphical elements to the screen. I can honestly say I’d never have done anything like this if I hadn’t heard about MUSHclient.
I’ll frequently be posting about my escapades with MUSHclient, so if that’s your kind of thing, watch this space.
Stagnation
by Jonathan C. on Feb.10, 2010, under Uncategorized
It’s painfully obvious to anyone who’s passed by here that my blog has been neglected for a long, long time. I’ve been caught in the trap of second-guessing what my audience cares about. Most often, I assume that what’s on my mind just isn’t interesting enough to post about. I do a lot of programming and imagining (the latter being a tremendous asset to the former), so I also assume that nobody who reads my blog will care about my technical adventures.
Well, I don’t know if I’m right, and I probably never will. But I’ve decided to start writing about all that stuff I never got around to. After all, posting something is better than posting nothing at all, and just because Jonathan.com gets 30k hits a month without even trying doesn’t mean I have to please every one of them. That ends up pleasing nobody, anyways.
Now lets just see if I can stick to the plan… and maybe get a new blog template.
~Jonathan
EDIT: New template! Not too shabby, hmm?
