Simple Worker Queue with Procs in MagLev

At the last PDX.rb meeting, Jesse Cooke presented “A very simple implementation of a background queue/worker” in MagLev. His program uses ruby Proc objects as the unit of work. Since procs are just objects, MagLev can save them to disk and run them in any MagLev VM connected to the repository. Instant, native distributed worker queue!

Indexing Collections in MagLev

MagLev brings built-in, scalable object persistence to Ruby. You can store large numbers of objects persistently on disk and retrieve them from any VM connected to the repository, all with ACID transactions. Large collections of data often need to be searched. This post introduces MagLev’s indexing and collection querying features.

Multiple Repositories in MagLev

MagLev stores all of its state, including application classes and code, in a repository. By default, MagLev has one repository named “maglev”. But there are times you may want to have multiple repositories. If you have multiple applications, you may want to store each in its own repository. Or, you may want one repository per developer, or different repositories for development, test, staging and deployment. MagLev provides some management tools that allow you to create, destroy, start, stop and perform other tasks on repositories.

Persistence by Reachability

MagLev stores ruby objects in a persistent repository using “persistence by reachability”: given a well known, persistent object, the “root object”, all objects that are reachable from the root will also be persistent (saved in the repository). The most common form of reachability is for one object to refer to another in an instance variable. ObjectA is reachable from ObjectB if ObjectB has ObjectA as the value of one of its instance variables. There are a some other cases as well. An object’s class is reachable from the object (as are any mixed in modules). Constants of a persistent class or module are also reachable from that class or module.

GemStone internals videos

I’m working on a few new posts, but in the meantime, you may be interested in some videos James Foster (one of the GemStone Smalltalk engineers) has posted on the GemStone VM. MagLev is based on the same VM, so much of what James has to say is applicable to MagLev.

Here are the videos so far:

• What is GemStone?
• Object Representation
• Special Objects and Object Headers
• Large Objects
• Repository, Extents, Pages and the Object Table
• GemStone/S Components
• Shared Page Cache
• New Object
• Reading Existing Objects
• Modifying and Dereferencing an Object (new)
• Commit Records (new)
• Avoiding a Commit Record Backlog (new)
• Commit Processing (new)
• Concurrency (new)
• Lock Granularity (new)
• Reduced Conflict Classes (new)
• The Stone Process (new)
• Live and Dead Objects (new)
• Nine Steps in Repository-Wide Garbage Collection (new)
• And more to come…

KD-Trees and MagLev

MagLev KD-Tree Example

A few weeks ago, I came across this blog post by Adam Doppelt from Urbanspoon. In the post, Adam describes how Urbanspoon solved a problem they had managing their data to enable nearest neighbor searches. They need to find the nearest N items to a given location, e.g., find the 10 nearest coffee shops to my current location. It’s an interesting post and I encourage you to read it to see the various solutions they tried and rejected. In the end, they solved their problem by writing a KD-Tree C-extension, which they have published as a RubyGem.

When I read the post, it seemed that the problem was a natural fit for MagLev. So, I implemented a pure ruby version of a KD-Tree and compared it to the kdtree gem and to the pure ruby solution running on MRI 1.8.6 and 1.9.2 ¹.

VM(s) + Repository = MagLev

MagLev is a new Ruby VM with built-in object persistence. GemStone has recently released a public alpha of MagLev.  I’m going to give a quick overview of how the VM and the object Repository work to serve up Ruby objects.