Attached are the slides from my April 2011 Melbourne Scala user group presentation, on “Heroscape 2.0:  A case study in using Scala for complex HTML document generation”

Here are the slides from a “State of Scala” talk I gave at OSDC in November 2010, summarizing where Scala is currently at.

http://2010.osdc.com.au/

In simple terms, it is a lament about the shortcomings off Test- (aka Behavior) Driven Development (TDD or BDD), a testing practice that is widely considered current best practice for software engineering, or least my experience of doing it.

In this method, software is constructed by first building executable tests, or specifications, that function somewhat like moulds in the casting of metals or plastics; they determine form of the software by specifying very precisely how it must behave. This takes more of the time and effort; once the specifications are constructed, it is usually less effort to construct the software to satisfy them.

Its been discovered, by myself empirically, but also by many others, that using many fine-grained, non-overlapping, unit tests yields software and tests that are easier/cheaper to build and maintain. However, an uncanny thing happens to unit tests as they become fine-grained: they start to look eerily similar to, or perhaps mirror images of, the code they specify.

So, the mirrors halves of photo in the previous post represents the two sides of a system (in this case, the text on the sign): the code (on the left), and the tests (on the right). What I was seeking to highlight is how the information content of the system has not been increased by adding the tests. We have said the same thing twice: once, as code, and again, as tests over that code.

What we have added is redundancy, which serves to detect errors, just as redundancy does in error correcting memory or RAID disks.

So, to me, the essence of the TDD practices is actually to ensure redundancy in a system’s specification. When the 2 copies of the specification, Code and Tests, do not agree, we have a test failure that is easily detected.

My problems with present day TDD are that

1. This redundancy is currently achieved by enormous manual effort. In the most efficient cases Ive seen where system behavior is 100% specified by tests, test code is typically 200% of the bulk of the code it specifies, and it can be far worse, 300-400% is not uncommon. In my experience, that consumes a great deal of effort/time/money.
2. It seems very difficult to measure or control the level of redundancy in the system introduced by the TDD practices. There seems to be a tendency to prescribe a “one-size-fits-all” approach of 100% unit test redundancy (ie every line, every branch, every data case in code covered by a unit test) as being appropriate for every project.
   In contrast, when we use redundancy to control errors for information storage or transmission, the amount of redundancy is a parameter that we can freely vary, to achieve a trade-off between bulky robustness and lightweight delicateness. We need to find a similar variable dial for software testing practices.
3. When we develop software in the TDD style, we are writing down the same pieces of information twice in 2 ledgers as we work: Write a test. Write the matching code. Execute and verify matching. Rinse and Repeat.
   For me, that feels tedious and unnatural. My instinct is to write it once, and see it execute, to engage in a flowing conversation with the compiler and the runtime.

Ive been thinking alot lately about what could replace TDD, to give us reliable software at more lightweight redundancy levels, in the 10-50% of code size range, rather than the 100%+ redundancy of textbook TDD. Some early ideas:

• Static Typing
• Dependent Types
• Design by Contract (which seems partly isomorphic to the very powerful Property-Based Testing approaches)
• Automatic type-driven generation of function inputs, so that function can be executed without explicitly writing tests (The other half of Property-Based Testing).
• Inline executable Examples, expressed as annotations, that describe valid sample function inputs and corresponding outputs, much as tests do, but much closer to the code.

A common use case. A cache data structure that:

• has a bounded maximum size and thus memory consumption (which in turn implies a cache eviction policy like LRU)
• is safe and performant for concurrent use

AFAICT, there’s nothing in the JDK or the concurrency JSRs that hits both these requirements. I googled and found the open source Concurrent Linked Hashmap library, which does.

[An incorrect comment about the project missing tests has been removed]

Unbounded Garbage Collectable Cache

If instead, you want a cache that will grow with use, but can be reclaimed if memory is short, then the use of a ConcurrentReferenceHashMap, configured with SoftReferences, is a good solution.

Its a big deal. Short lived objects become nearly free to use. Convenience rules. If you need to tuple-ize 2 Ints, then stick them in a List, to pass through an API, only to immediately unpack back to the Ints on the other side: go ahead. Dont worry. Those objects probably will never be created; they’re conceptually there.

Get back to what you ought to be doing: Abstracting and Composing.

Heavy refactoring of game code, technical debt and experimentation have conspired to make it broken. Its taken much of the past month to fix.

First I tried using Scala’s scala.xml.transform.RewriteRule, but hit some bugs (#2124 and #2125) that convinced me to let it stabilize further before applied use.

So instead I used the new XSL support in Eclipse 3.5  (in WTP 3.1). Seems to work well.

I’d only used XSL back in 2001 and never fully understood it. I found most entry level tutorials (eg ZVON and W3 Schools) unhelpful for this kind of application, because they don’t explain copy existing nodes well.

My breakthrough came when I discovered Jesper Tverskov’s article on the Identity Template. This explains how to recursively process XML by copying with modification.

One case Jesper doesnt cover is copying where segments of the subtree are omitted and others are retained but modified.

I solved that in my template by using a <for-each> to pickout the subtree part I wanted to preserve, wrapping the <xsl:copy><xsl:apply-templates select=”@*|node()”/></xsl:copy> structure thats fundamental to recursive processing.

<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>

<xsl:template match="//vector[default]">
<size><xsl:value-of select="default/elementCount"/></size>
<xsl:for-each select="default/elementData" >
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:for-each>
</xsl:template>

Sample input and output:

<__features id="50" serialization="custom">
<vector>
<default>
<capacityIncrement>4</capacityIncrement>
<elementCount>1</elementCount>
<elementData id="51">
<arcadia.domain.Unit reference="46"/>
<null/>
<null/>
<null/>
<null/>
</elementData>
</default>
</vector>
</__features>

<__features id="50" serialization="custom">
<size>1</size>
<elementData id="51">
<arcadia.domain.Unit reference="46"/>
<null/>
<null/>
<null/>
<null/>
</elementData>
</__features>

++++++++++++++++++

This post arose out of a Twitter discussion with Daniel Spiewak on twitter. I dont know whether Newton could have encoded his laws of Mechanics into 140 characters, but I need more space to share my thoughts on the (hopefully) simpler matter of: What is appropriate use of Mutable Data in Functional software systems?

Entities and Identities

I feel the answer is inter-connected with the idea of Identity represented by the data in a software system. As Eric Evans explained so well, data that has a strong identity is called an Entity.

A classic example, present in almost every business software system, is some “User Entity”, modeling a person’s account in that system. In such a system, when we speak about a particular User, conceptually we really want there to be only one such entity in existence.

The identity of an entity is normally governed by a unique & immutable primary key. Other than the primary key, everything attribute of the entity may change. A User, for example, might change their favorites list, address, name or possibly even sex.

Functional Interaction with Entities

Conceptually there’s only one entity. But actually, there’s only one “offical version”, and other possibly divergent versions, which may be promoted to official status.

Entities should be accessed via transactions (in broadest sense, including STM or concurrent data structures). A functional program makes an immutable snapshot of the entity at some time. It may derive an updated version of the entity from that, and then request that its copy be promoted as the “master” copy. Typically, this promotion would be done using optimistic concurrency, so that if another process changed the entity first, the update must retry. But other models (eg pessimistic concurr, change merging) might be used instead.

Note that

• An executing program has no guarantee that it operates on the “latest” version of an entity.
• After the fact, it is possible to say “at time X, the official state of the entity was thus…”, by time-stamping updates. This is important, for example, in a legal case if we wanted to know who owned some property at a particular point in time in a dispute.
• This is either Shared State and Message Passing concurrency. We can view the “official entity version” as shared state, or as an agent to whom we send read-entity and write-entity messages.

Pretty much as Daniel put it (borrowed from Clojure, apparently) “Everything is immutable except for concurrency-safe entities“

The meaning of Entity Update

Something special happens when an entity’s master state is updated. This is the synchronization/convergence point between separate threads and components. It is the boundary between the functional code, which transforms some input state into some new entity states, and the imperative world of sequentially updating mutable state. It is the end of a Unit of Work, a time when a functional program says “this is not a means to an end, this is the end”. The transient becomes persisted, allowing the state in the current moment to affect and interact with other points in time.

Granularity, Scalability and Consistency

So how many entities do we need? Coarse or fine grained? A few larger, internally complex entities, or many simpler and smaller entities?

Its a matter of chosing a point on a spectrum whose extremes are

• A single StateOfTheWorld entity whose attributes encompass all data in the system. As in a purely functional system; the program sees only its own private version of state, without any interference.
• Every tiny piece of data is its own entity. This would be like programming only with atomic global variables. Everyone sees and shares all state without privacy.

I don’t think there’s one “right” answer. All I can offer is an outline of some design forces acting in either direction:

Fewer Entities each containing more data

• For: data consistency within an entity. A coarse grained entity starts in a consistent state and is never externally updated. A program is free to compute changes to the entity’s state independently without risk of being disrupted by other program’s updates. The benefits of the purely functional model accrue.
• Against: Contention if multiple processes try to simultaneously update the same entity. For an extreme example, imagine if two functional programs simulatneously computed a new StateOfTheWorld entity – they would operate in a perpetual state of fatal contention.
• Against: Copying Overhead. Need to copy minimum of log(N)  (where N is the “size” of the entity), and often much more, of an entity’s data to update it.

More Entities each containing less data

• For: Less copying overhead. We still need to copy minimum of log(N)  of an entity’s data to update it, but N is smaller.
• For: Less contention when multiple processes try to simultaneously update entities, because the updates are spread across more fine-grained locks.
• Against: data inconsistency between inter-related entities. We either have to tolerate inconsistency between entities (eg Person entity ‘Ben’ has a son ‘Otto’, but Person entity ‘Otto’ doesnt have a father ‘Ben’), or introduce an extra layer of locking over the entities (as most databases do). Note that if we inroduce locking above the entity level, our contention benefits go away, and we may be creating de-facto coarse-grain entities.

Its interesting to note that the same trade-off between consistency and scalability shows up many times elsewhere – eBay being a nice example.

Non-Global Identity

Above, I presented Entities as having a globally unique identity. Much of the time, thats a good way to think about them, but be aware it is a simplification. In fact, identity is inherently defined relative to some scope. That scope needn’t be global.

Here are 3 different scopes for the entity The number of people on earth at midnight on Dec 31, 1999.

1. The globally true answer that an omnipotent god might know
2. My national government’s official figure
3. My personal estimate

Scopes may nest. Im still contemplating exactly how this affects the design of systems with mutable data.

Sorry to be so blunt … please don’t shoot the messenger.

My awakening started when I began searching for a better Java, and found Scala. Scala had these weird (in my ignorance) functional features and learned Scala people often talked about Haskell. A uni friend Bernie Pope raved about it too.

I took a look at Haskell. It wasn’t a pleasurable experience. Much of the code was almost incomprehensible. Half the concepts I’d never heard of before. It made me feel stupid, but actually what I was was ignorant.

I say ignorant, rather than innocent, because Haskell has been around for over a decade, and FP much longer.

I still struggle to read most Haskell, and I certainly can’t use it to build anything. But I am starting to get a sense of just how sophisticated it is, and a map of it’s concepts in my mind. I began by going down a rabbit-hole, and expected to find a burrow, a community of Haskelliers programming in their own unique way. Instead, bit by bit Ive realised Ive ended up in a vast cavern absolutely full of stuff I barely even knew existed.

• Monads & friends
• Typeclasses (see “Typeclassopedia”, a must-read for advanced programmers, if only to appreciate what you don’t yet know )
• Composability (Need a good succinct web ref here, please suggest)
• Laziness
• Higher order functions
• Functional Reactive Programming
• Software Transactional memory
• Point-free style
• Zippers, Finger Trees and other Immutable Data Structures
• Real World Haskell and Hudak’s book

The bit that stirs me up is that this “stuff” isn’t, repeat is not, Haskell-specific. Its rooted in the fabric of our reality, in our mathematics and our problem domains, and bits poke up like the tips of icebergs into mainstream OO languages, their true structure part-revealed. But rarely in the OO-world have I found such carefully abstracted and subtle techniques of programming in daily use.

Ideas from Haskell & Functional Programming will continue to flow into the mainstream over the next couple of decades. Innovations will be trumpeted, trends identified, features debated, technologies evangelized.

But personally, Im too curious (and too lazy a typist) to wait that long.,