The last few days its been dawning on me how Escape Analysis changes the rules on performance critical code.

Its a big deal. Short lived objects become nearly free to use. Convenience rules. If you need to tuple-ize 2 Ints, then stick them in a List, to pass through an API, only to immediately unpack back to the Ints on the other side: go ahead. Dont worry. Those objects probably will never be created; they’re conceptually there.

Get back to what you ought to be doing: Abstracting and Composing.

Software Transactional Memory relies on optimistic locking to avoid deadlock, but as a result suffers at least one serious problem.

An optimistic update transaction will succeed when the data being updated remains stable for the period of the transaction.

If n threads are trying to update a shared variable at m Hz, on average there will be an attempt to change the variables value about every (n m)-1 secs.

As that interval falls towards the average time it takes to perform an update transaction, the success rate for transactions will fall precipitously to almost zero, and the system will grind to a starved halted state that’s as bad as deadlock.

We must make some progress even under heavy contention

Pessimistic locking is one approach, but it suffers from deadlock problems.

The software equivalent of “Changeset Merging” is another.

In a version controlled system, we do not Revert Changes any time someone else has edited the same file. That seems wasteful, but that’s essentially what basic STM does. Rather, someone or something examines the two conflicting changesets, and merges them together. This might involve discarding part or all of one change, or some blend.

Changeset merges in STM

A transaction could declare a merge routine to be run after a conflict. It is given the current system state, and the transaction that lost out on the update race, and can either transform them into one merged state that is saved, or fallback to the default retry if  it cannot do so.

Of course, it’s likely there are other (3rd, 4th ..etc) transactions in progress while that merge is taking place. But this can simply be resolved by the running the merge routine again, which now blends the previous merge with the next incoming transaction, as the new system state.

I must credit [http://lists.basho.com/pipermail/riak-users_lists.basho.com/2009-September/000029.html] and [http://debasishg.blogspot.com/] for leading me to this idea.

This is a sight I havent seen for 3 months – my game’s level editor working. Its a relief to have it back (partially).

Heavy refactoring of game code, technical debt and experimentation have conspired to make it broken. Its taken much of the past month to fix.

Ive recently been using XSL to do schema migration on XML data; ie small, incremental modifications to XML where most of the existing data is preserved. The application was to massage some Heroes of Arcadia level data, serialized with XStream, to keep in step with code changes I’d made.

First I tried using Scala’s scala.xml.transform.RewriteRule, but hit some bugs (#2124 and #2125) that convinced me to let it stabilize further before applied use.

So instead I used the new XSL support in Eclipse 3.5  (in WTP 3.1). Seems to work well.

I’d only used XSL back in 2001 and never fully understood it. I found most entry level tutorials (eg ZVON and W3 Schools) unhelpful for this kind of application, because they don’t explain copy existing nodes well.

My breakthrough came when I discovered Jesper Tverskov’s article on the Identity Template. This explains how to recursively process XML by copying with modification.

One case Jesper doesnt cover is copying where segments of the subtree are omitted and others are retained but modified.

I solved that in my template by using a <for-each> to pickout the subtree part I wanted to preserve, wrapping the <xsl:copy><xsl:apply-templates select=”@*|node()”/></xsl:copy> structure thats fundamental to recursive processing.

<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>

<xsl:template match="//vector[default]">
<size><xsl:value-of select="default/elementCount"/></size>
<xsl:for-each select="default/elementData" >
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:for-each>
</xsl:template>

Sample input and output:

<__features id="50" serialization="custom">
<vector>
<default>
<capacityIncrement>4</capacityIncrement>
<elementCount>1</elementCount>
<elementData id="51">
<arcadia.domain.Unit reference="46"/>
<null/>
<null/>
<null/>
<null/>
</elementData>
</default>
</vector>
</__features>

<__features id="50" serialization="custom">
<size>1</size>
<elementData id="51">
<arcadia.domain.Unit reference="46"/>
<null/>
<null/>
<null/>
<null/>
</elementData>
</__features>

!Warning! This is one of my most abstract and part-formed blog posts, a rough sketch of some ideas that are still forming in my mind. You might want to skip onto to something more tangible.

I beleive the ideas expressed in Rich Hickey’s monograph “On State and Identity” (and my own independent grasping toward these ideas) are true.

The past few days I have been contemplating what large scale software look like in this model. The answer seems quite exciting. Traditional immutable versus mutable state conflicts are finally cleared away – instead, each has its defined place.

Values

A value is something time-invariant. All observations of a value should be equivalent. Computationally, a value may be passed around in unresolved or part-resolved form, and further computation is required when parts of the value are accessed.

Entities

An entity is a time-varying stream of values associated with a particular identity. Note that the history of an entity’s value, as well its its current value, is available, which is why a Stream is returned (the history may need to be truncated to fit within real-world computational and storage limit).

def entity[T](id: ID): Stream[T, Timestamp]

Entities change their value transactionally. All changes to the value are recorded permanently. Changes are timestamped in system-time.

def update[T](id: ID, T newValue) //in transaction

Logically, the number of entities is infinite and they are never created or destroyed. If one asks for an entity which has not previously been set to a value, then it returns an empty stream.

Note: There is no way to ask what entities exist. One must find entities via references from other already known entities, and not via a system-level “browse-style” method.

IDs

An opaque immutable value representing an identity. IDs can be used to obtain the values of the entity they reference. IDs are values like any other, can be stored as/in entities, passed to functions, etc.

There is one special identity, system-time. This refers to an entity that holds the current system time, that updates in some discretized approximation of a smoothly changing variable.

Programs

Programs are the things that happen, the things that update entity values and respond to updates in entity state. Most operations are functional, but imperative operations may be required when communicating with external systems, ie for IO.

Functions with Bindings

Functional Programs consist of two things:

• a (purely functional) Function from N inputs to M outputs. It can and should contain other functions nested to any depth.
• a Binding of the function’s inputs and outputs to either
  • the value of an entity. In the case of inputs, this means the function is evaluated passing the entity’s value as a parameter. For outputs, this means the entity’s value is assigned that value returned by the function.
  • the event of an entity changing value, for inputs only. This means that a function fires when a entity has changed value. (cf FRP)

The ability to launch programs when entities change state is a key to the operation of the paradigm.

Imperative Operations

Programs can also run imperatively, in order to sequence IO operations for communicating without outside systems.  They can bind inputs to entity values or entity change events, exactly as functional programs do, but do not return any outputs that are assigned back onto entities.

This explanation is rather terse. I need to give some examples, some diagrams, and some explanations. Perahps some outline of what my goals are. To do anything practical with this paradigm requires alot of infrastructure built on top of this base. For now, Im mainly writing this down as a record of my own thought process, for later re-editing

Im coming late to the party: in February Errki Lindpere posed an worthwhile question on his blog that want to talk about:

“Anyway, please comment, if you are also a programmer, or a geek with a different specialization, who has a lot of hobby projects, how do you manage not to get swamped with them or get the feeling that you are always working on cool stuff but never releasing?”

Myself, I have been developing a fantasy mobile game Heroes of Arcadia since early 2005. While so far Ive got some funding, formed a company and released 2 beta versions, a finished game remains /at least/ 12 months off. [full story:  http://bit.ly/e3UKE]

I would never have dreamed in early 2005 that I’d still be unfinished in 2009. I would never have imagined the marathon effort it  would require of me to create a polished, fun, playable and stable game.

This experience has taught me an important life lesson: Pick your creative battles!

Our lives are finite. There are actually a limited number of things one person can attempt and complete in one lifetime.

Therefore, I feel it’s important to try to understand what you truly aspire to, what you are really good at, what you’re going to be happy doing. Find the sweet spot and really try to focus on it.

My strategy increasingly is to ensure that everything I am involved with fits with my overall aspirations, reinforces each other, and reuses what Ive learned and done already in life. In a word, Synergy.

By trying to create something large, Ive gained a sense of how much creative ‘mana’ Ive got in me. The blunt truth is that it feels somewhat finite, and less than I had idealistically imagined in my youthful daydreams.

So I want to ensure that I spend this mana in the most fruitful way I can.

Anything I attempt to do, I want to

• yield some outcome
• create something beautiful
• be proud of what Im doing
• enjoy the journey as the well as the destination
• learn and grow from

Since I wrote this post, I have become aware of Rich Hickey’s discussion of State and Identity on Clojure.org. I feel he expresses the same truths in a generally clearer, sharper way than I have managed to here…

++++++++++++++++++

This post arose out of a Twitter discussion with Daniel Spiewak on twitter. I dont know whether Newton could have encoded his laws of Mechanics into 140 characters, but I need more space to share my thoughts on the (hopefully) simpler matter of: What is appropriate use of Mutable Data in Functional software systems?

Entities and Identities

I feel the answer is inter-connected with the idea of Identity represented by the data in a software system. As Eric Evans explained so well, data that has a strong identity is called an Entity.

A classic example, present in almost every business software system, is some “User Entity”, modeling a person’s account in that system. In such a system, when we speak about a particular User, conceptually we really want there to be only one such entity in existence.

The identity of an entity is normally governed by a unique & immutable primary key. Other than the primary key, everything attribute of the entity may change. A User, for example, might change their favorites list, address, name or possibly even sex.

Functional Interaction with Entities

Conceptually there’s only one entity. But actually, there’s only one “offical version”, and other possibly divergent versions, which may be promoted to official status.

Entities should be accessed via transactions (in broadest sense, including STM or concurrent data structures). A functional program makes an immutable snapshot of the entity at some time. It may derive an updated version of the entity from that, and then request that its copy be promoted as the “master” copy. Typically, this promotion would be done using optimistic concurrency, so that if another process changed the entity first, the update must retry. But other models (eg pessimistic concurr, change merging) might be used instead.

Note that

• An executing program has no guarantee that it operates on the “latest” version of an entity.
• After the fact, it is possible to say “at time X, the official state of the entity was thus…”, by time-stamping updates. This is important, for example, in a legal case if we wanted to know who owned some property at a particular point in time in a dispute.
• This is either Shared State and Message Passing concurrency. We can view the “official entity version” as shared state, or as an agent to whom we send read-entity and write-entity messages.

Pretty much as Daniel put it (borrowed from Clojure, apparently) “Everything is immutable except for concurrency-safe entities“

The meaning of Entity Update

Something special happens when an entity’s master state is updated. This is the synchronization/convergence point between separate threads and components. It is the boundary between the functional code, which transforms some input state into some new entity states, and the imperative world of sequentially updating mutable state. It is the end of a Unit of Work, a time when a functional program says “this is not a means to an end, this is the end”. The transient becomes persisted, allowing the state in the current moment to affect and interact with other points in time.

Granularity, Scalability and Consistency

So how many entities do we need? Coarse or fine grained? A few larger, internally complex entities, or many simpler and smaller entities?

Its a matter of chosing a point on a spectrum whose extremes are

• A single StateOfTheWorld entity whose attributes encompass all data in the system. As in a purely functional system; the program sees only its own private version of state, without any interference.
• Every tiny piece of data is its own entity. This would be like programming only with atomic global variables. Everyone sees and shares all state without privacy.

I don’t think there’s one “right” answer. All I can offer is an outline of some design forces acting in either direction:

Fewer Entities each containing more data

• For: data consistency within an entity. A coarse grained entity starts in a consistent state and is never externally updated. A program is free to compute changes to the entity’s state independently without risk of being disrupted by other program’s updates. The benefits of the purely functional model accrue.
• Against: Contention if multiple processes try to simultaneously update the same entity. For an extreme example, imagine if two functional programs simulatneously computed a new StateOfTheWorld entity – they would operate in a perpetual state of fatal contention.
• Against: Copying Overhead. Need to copy minimum of log(N)  (where N is the “size” of the entity), and often much more, of an entity’s data to update it.

More Entities each containing less data

• For: Less copying overhead. We still need to copy minimum of log(N)  of an entity’s data to update it, but N is smaller.
• For: Less contention when multiple processes try to simultaneously update entities, because the updates are spread across more fine-grained locks.
• Against: data inconsistency between inter-related entities. We either have to tolerate inconsistency between entities (eg Person entity ‘Ben’ has a son ‘Otto’, but Person entity ‘Otto’ doesnt have a father ‘Ben’), or introduce an extra layer of locking over the entities (as most databases do). Note that if we inroduce locking above the entity level, our contention benefits go away, and we may be creating de-facto coarse-grain entities.

Its interesting to note that the same trade-off between consistency and scalability shows up many times elsewhere – eBay being a nice example.

Non-Global Identity

Above, I presented Entities as having a globally unique identity. Much of the time, thats a good way to think about them, but be aware it is a simplification. In fact, identity is inherently defined relative to some scope. That scope needn’t be global.

Here are 3 different scopes for the entity The number of people on earth at midnight on Dec 31, 1999.

1. The globally true answer that an omnipotent god might know
2. My national government’s official figure
3. My personal estimate

Scopes may nest. Im still contemplating exactly how this affects the design of systems with mutable data.

Right now, if you want to understand the state of the art in computer programming, those are your choices as I see them.

Sorry to be so blunt … please don’t shoot the messenger.

My awakening started when I began searching for a better Java, and found Scala. Scala had these weird (in my ignorance) functional features and learned Scala people often talked about Haskell. A uni friend Bernie Pope raved about it too.

I took a look at Haskell. It wasn’t a pleasurable experience. Much of the code was almost incomprehensible. Half the concepts I’d never heard of before. It made me feel stupid, but actually what I was was ignorant.

I say ignorant, rather than innocent, because Haskell has been around for over a decade, and FP much longer.

I still struggle to read most Haskell, and I certainly can’t use it to build anything. But I am starting to get a sense of just how sophisticated it is, and a map of it’s concepts in my mind. I began by going down a rabbit-hole, and expected to find a burrow, a community of Haskelliers programming in their own unique way. Instead, bit by bit Ive realised Ive ended up in a vast cavern absolutely full of stuff I barely even knew existed.

• Monads & friends
• Typeclasses (see “Typeclassopedia”, a must-read for advanced programmers, if only to appreciate what you don’t yet know )
• Composability (Need a good succinct web ref here, please suggest)
• Laziness
• Higher order functions
• Functional Reactive Programming
• Software Transactional memory
• Point-free style
• Zippers, Finger Trees and other Immutable Data Structures
• Real World Haskell and Hudak’s book

The bit that stirs me up is that this “stuff” isn’t, repeat is not, Haskell-specific. Its rooted in the fabric of our reality, in our mathematics and our problem domains, and bits poke up like the tips of icebergs into mainstream OO languages, their true structure part-revealed. But rarely in the OO-world have I found such carefully abstracted and subtle techniques of programming in daily use.

Ideas from Haskell & Functional Programming will continue to flow into the mainstream over the next couple of decades. Innovations will be trumpeted, trends identified, features debated, technologies evangelized.

But personally, Im too curious (and too lazy a typist) to wait that long.,

In 2001/2002 I worked for 18 months at the now defunct UK Javaspace vendor Intamission, building their Javaspace implementation Autevo, which went on to  … er… “inspire” the open source Blitz product , released by Intamission co-founder & lead programmer Dan Creswell after he left the company.

One of the less satisfying aspects of that project was that, as I worked with Javaspace/Tuplespace -based systems, I came to believe in them less and less. I now believe that while Javaspaces have some great qualities, they also have some serious flaws. Here’s a brief outline of what I think these flaws are.

Some background for the layman

Javaspaces pride themselves on having a simple API based on 4 methods

• write puts an object into the shared distributed space
• read returns a copy of one object matching some provided query criteria. There are blocking and polling variants.
• take returns a copy of one object matching some provided query criteria and removes it from the space. There are blocking and polling variants.
• notify registers interest in being notified should an object enter the space matching the provided criteria.

Javaspaces use a Query by Example mechanism to query the state of the space. One presents an template Entry to the space, with zero or more public, non-primitive, serializable fields, any of which may be null, and the space matches any Entry in the space whose fields are equals() to the template’s field, while treating null fields of the Entry as wildcards that match anything. Thinking of Entry as being a tuple is a good analogy.

I subscribe to the design aesthetic as simple as possible, but no simpler. Unfortunately,  the Javaspace query mechanism is too simplistic, and not sufficiently powerful to solve practical distributed systems problems with:

No Range Queries

The query API allows one to specify either an exact value, or any value, but nothing in between. You can say “a Person of Age == 25″, or “a Person of any Age”, but not “a Person of Age 25- 34″. This is a result of Javaspace’s fixation with a flawed query-by-example model. As you might imagine, this limitation quickly becomes problematic when you try to build real applications.

Use of Null as Wildcard

Another nasty side-effect of the query-by-example model is that null becomes overloaded as a wildcard value. Accordingly, Entry fields cannot be primitive, and may not be null for any legitimate application purpose.

Lack of a readAll Operation

readAll would be the Javaspace analog of a SQL Select statement; ie find me all Entries in the space matching the provided criteria. Sorry, no can do. A Javaspace can give you one of them, but not all of them.

So for example, if we want to ask our space-based application, “How many tasks are in the queue?” , it can only reply “All I can tell you is that there is at least one.”

The Future

Someone should combine the best ideas from Javaspaces – a focus on high-performance, short-term storage of self-contained objects/messages, leases, and event notification, with a decent, extensible query mechanism. Now that would fly. (Maybe someone already has? … I havent been following this field closely in recent years)

Deleting Objects from a Domain Model in a safe and scalable way with minimal adverse effects, is a thorny problem.

Eric Evans Domain Driven Design book has the best discussion Ive read under the section on Aggregates, but its not comprehensive.

Im struggling to find a really nice solution in Heroes of Arcadia. My best shot so far involves a mixed approach – cleaning out as many root refs as I can, plus stamping the deleted object as inactive. An example:

Units can be removed from a Zone.

There are two primary references to a Unit, via its Faction and its Location. These are bi-directional. When a unit is removed, it notifies its Faction and Location that it is going and they remove their ref to it. As these are the primary indices, cleaning them out will prevent the creation of new refs to the removed unit.

Units may also be referenced from other domain entities uni directionally (meaning they are unaware they’re referenced), for example, by being the target of a spell effect. These refs will remain after the unit is “removed”. The unit must stamp itself as being removed/inactive, and all domain operations need to tolerate this state. Gradually, the removed unit should “drain out” of the system as refs expire.