Sazerac

I want to sit at a bar with Chris McMillian behind it!

Background tasks

I don't like waiting on the completion of a task that should be run in the background. For example, when I bookmark a page using a Delicious bookmarklet it presents a modal dialog box that has me wait for the bookmark to be made on Delicious' servers. The saving might only take a short time but the length of this short time varies greatly. The unknown wait is a serious obstacle to maintaining flow.

What I would like to see is for web browsers to have a background tasks API and user interface. A task is a JavaScript function. A web page or application (webapp) can add a task to the list of background tasks. The execution of the background task has the same constraints as any other task running in a give webapp from a specific web site.

The task is run by the browser when it has the chance. There are no guarantee of timeliness; just a promise of best effort. For simplicity, all the tasks in the list are run in the order added by the adding webapp. That is, if pages A and B both add 5 tasks to the background list then the tasks will be run in the order A1, B1, A2, B2, A3, B3, A4, B4, A5, B5. All tasks "succeed" in that any errors that occur must be conveyed to the originating webapp and its existing enqueued tasks. For example, a status of the task is kept in a cookie and the webapp & its tasks look at this cookie to determine the right course of action. The only indication to the user of tasks status will be a message in the task history. The user must not be expected to "respond" to the status message. The webapp is allowed to enumerate its tasks and to cancel them.

The "Narrator" service

I have an itch. I want to create a podcast. Not of myself talking on & on about something of importance to me. (I do this in my day job and in the mirror.) What I want is a podcast of the long articles or papers of interest that I discover. I want to be able to access these in the car or when I am otherwise not able to read them online or in print. A podcast of readings of these longer texts subscribed to in iTunes would be very helpful. And so I am now thinking about implementing a service to achieve this.

The "Narrator" service is a very simple pipeline. At this stage it looks like this:

1. URL of article to read is deposited with Narrator.
2. Narrator deposits the URL to Instapaper.
3. Narrator confirms Instapaper's understanding of the text with the depositor.
4. Narrator starts a HIT with Amazon.com Mechanical Turk.
5. The HIT's reader uses Narrator's online audio capture to record the reading.
6. The depositor is notified of the reader's audio availability.
7. Narrator adds the article's link and the reader's audio to the depositor's podcast feed.

I will keep this posting updated as I find out more about implementing this service. If these already exists, then please tell me.

Update: The Umano app and service is getting close to what I want.

Spring, archetypes, conditional sections, and properties.

Comment on Project Configuration with Spring

We added a further twist to the properties and added "conditional sections" to the contexts.
For properties, we have the normal "common" properties that define the default values for all properties for all deployments, deployment specific properties that override the defaults, and archetype specific properties that (mostly) enable conditional sections. Examples of archetypes are "full text search", "full text indexing", "analytics processing", "upload processing", etc.

The conditional sections allow for the inclusion of beans based on the value of a property. We have many properties that have names of the form "xxxx.enabled". When the value is "true" then the conditional section's beans are defined. (A similar effect can be had by using different "application" context files but we found that conditional sections worked better for us.)

The startup procedure is to

1. load the common properties to "prime" the environment,
2. load the deployment specific properties where are defined the archetypes used by the deployment,
3. load each of the archetype properties, and then, finally,
4. reload the deployment specific properties again (to reset deployment specific property values).

Together these two features -- conditionals and archetypes -- allows us to deploy a specific configuration with very little customization. And it is all done within Spring with just two custom tags.

Another amazing animation by Piotr Kaczmarek!

Another amazing animation by Piotr Kaczmarek!

Pay for service and tools you rely on

I am happy to pay for good service and tools. I have decided that Google does not give either. Perhaps it is because I am both a customer and product to them. Perhaps not. The upshot is that I have discontinued Google Drive. I am back to Safari. I have no intention of trying Google Keep. Instead, I am paying DropBox and Evernote for their services and tools. And so far I am very happy.

Crazy world example #1

Crazy world example #1: Staples wants $23.40 to print 40 pages in color and duplex in their Copy & Print center. They can also sell me an HP Deskjet 2512 All-in-One Printer for $39.99. So, for an extra $16.59 I can buy a printer for this one use and a few more pages afterwards and then throw it away.

OS X and system-wide default paths

If you want to change the system-wide set of paths for OS X edit the file /etc/paths. (It took me too long to find this out.)

Chirp

For some applications and tools there can be no output for a long time. When these are tailed seeing nothing is disquieting. The following Perl script will output a "chrip" every few seconds, intermixed with the log file's content. For example,

tail -F foo.log | grep bar | chirp

Save the following to /usr/local/bin/chrip

#!/usr/bin/perl -w

use strict;
use POSIX qw(strftime);

$::timeout = 5;
$::format = "%F %T";

while ( @ARGV ) {
        my $arg = shift @ARGV;
        if ( $arg eq "-t" ) {
                $::timeout = shift @ARGV;
        } 
        elsif ( $arg eq "-f" ) {
                $::format = shift @ARGV;
        }
        elsif ( $arg =~ /^-/ ) {
                print "usage: chirp [ -t seconds ] [ -f strftime-pattern ]\n";
                exit 1;
        }
}

local $SIG{ALRM} = sub { 
        print strftime( $::format, localtime ), " chirp\n";
        alarm $::timeout;
};

alarm $::timeout;

while ( <> ) {
        print strftime( $::format, localtime ), " ", $_;
}

# END

Installing mt-daapd on OS X

The following story is not pretty. I am hoping there is an easier way in the future. Until then, here is how I installed a music server on a Mac.

We have a Mac Mini that is used to hold the family's music collection. However, since this machine is also used by the kids for homework, iTunes is not always running. I finally got around to installing a DAAP server on the machine so I could expect to find the music available when I wanted it. I used brew to build and install the mt-daapd server. Assuming you have brew installed you need only run

brew install mt-daapd

However, for some reason the brew recipe did not complete the installation. This turned out to be fortuitous as I continue to want use iTunes to manage the music located in my home directory. And so I configured mt-daapd to run from within my home directory. First I created the /Users/ajg/Library/Application Support/mt-daapd directory for mt-daapd data. Then created a configuration file at /Users/ajg/Library/Application Support/mt-daapd/mt-daapd.conf containing

[general] 
web_root /usr/local/share/mt-daapd/admin-root 
port 3688 
admin_pw password 
mp3_dir /Users/ajg/Music/iTunes/iTunes Media/Music
db_dir /Users/ajg/Library/Application Support/mt-daapd
servername Minimac Fulltime
runas nobody 
extensions .mp3,.m4a,.m4p,.wav,.wma,.aiff,.ogg 
logfile /Users/ajg/Library/Application Support/mt-daapd/mt-daapd.log 
rescan_interval 600 
process_m3u 1 
scan_type 0 
compress 1 

Your configuration should be the same except, perhaps, for port (the default is 3689 which conflicts with iTunes), password, mp3_dir, and servername. To test the installation run the following and then open iTunes so see the "Minimac Fulltime" server.

/usr/local/sbin/mt-daapd -y -c '/Users/ajg/Library/Application Support/mt-daapd/mt-daapd.conf' -f -d 1

Once it is working, the next step is to keep it running using OS X's launchd. Create the file /Users/ajg/Library/Application Support/mt-daapd/mt-daapd.plist containing

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>mt-daapd</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/sbin/mt-daapd</string>
    <string>-y</string>
    <string>-c</string>
    <string>/Users/ajg/Library/Application Support/mt-daapd/mt-daapd.conf</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
</dict>
</plist>

Install it and start the server running

launchctl load "/Users/ajg/Library/Application Support/mt-daapd/mt-daapd.plist"
launchctl start mt-daapd

I can now listen to music on any of my devices around the house; which includes a first generation iPod Touch using Simple DAAP Client. And if I need to check the server's status I can open http://localhost:3688/.

Update: A link to the FireFly Media Server at the Internet Archive.

View Evernote note via an Everynote Link

Everynote is a good "everything bucket" tool. I have begun to use it more and more for notes, tasks, etc. A feature of Evernote is that you can create Evernote Links. These links are URIs that reference a note, but only locally. However, it occurred to me that the data in the link could be used to create a URL to the same Everynote note as displayed on the web. The following is a bookmarklet that prompts for the link, converts it to a URL, and then directs the browser to view the page at the URL.

(function(){
var link_pattern = new RegExp( 
  "^evernote:\/"+
  ".*"+
  "\/"+
  "([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})"+
  "\/"+
  "([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})"+
  "\/"+
  "$"); 
var link = "";
for(;;) {
  link = window.prompt("Evernote link?",link);
  if ( link ) {
    var m = link_pattern.exec(link);
    if ( m ) {
      var url = "https://www.evernote.com/view/notebook/"+m[1]+"?&n="+m[2];
      window.location.assign(url);
      break;
    }
  }
  else {
    break;
  }
}
}
)();

Show in Everynote.

Building an Atlassian Confluence plugin without Atlas, et al

Note: This posting has been updated so as to work with Confluence 4.x. (The previous version of the posting targeted Confluence 2.x.)

I spent several semi-productive hours this weekend writing a plugin for Atlassian's Confluence wiki. The plugin enables the running of JavaScript on the server and placing the result into the page displayed. I have found that using a wiki to intermix static content and dynamic content makes for a great reporting and situation-awareness tool kit (see JSPWiki). I wanted to do the same again within the confines of Confluence 4.x.

Confluence has a really good plugin manager webapp. You can search a repository for existing plugins for immediate inclusion. And you can also upload the plugin from your desktop. I gave the plugin manager a work out getting my plugin to work and can attest to its stability.

And here is the rub, Atlassian has made including and using plugins simple but has made creating a simple plugin almost impossible. My plugin's actual logic is very simple

private static final ScriptEngineManager factory = new ScriptEngineManager();
...
ScriptEngine engine = factory.getEngineByName("JavaScript");
Object result = engine.eval(script);
return result.toString();

To make this happen, however, Atlassian wants me to use Atlas. Atlas, as a far as I can tell, is much like Ruby on Rails and Spring Roo where the tool lays out a directory structure with files that together are the parts of and tools for building an "Hello World" application/plugin. In addition to Atlas, Maven and (perhaps) Eclipse are also needed. If my occupation was building whole applications on top of the Atlassian products and their APIs I could understand the logic of this tool chain. But I have a 4 line (!) plugin.

Confluence has been around for a long time and I guessed that the pre-Atlas way of building plugins was documented and with code examples. As far as I can tell, and this is after much searching, this is not the case. I was not able to find a single example of a basic plugin built with, for example, Ant. Since, in the end, a plugin is nothing more than a jar containing code and configuration I was shocked that this was missing from the mass of other documentation Atlassian provides. A whole population of, mostly in-house, programmers are being ignored. These are the programmers that are going to build plugins, i.e. small extensions to big tools that aid the better match between Confluence and the users needs.

To this end, here is how I build a basic macro plugin. (Note that the plugin documented here is not what I finally created. In the process of using the JDK's implementation of JavaScript I discovered that it is a old version of Mozilla's Rhino that does not support E4x, the XML language extensions. E4X makes XML a first class data type within JavaScript. Even the JavaScript syntax has been extends to allow for XML constants, for example x = <a/>. And so the final plugin uses Rhino 1.7R3 which does support E4X and JavaScript 1.8.)
The plugin's jar needs a minimum of two files. The Java class and the atlassian-plugin.xml configuration file. The development directory tree is

.
./build.xml
./src/atlassian-plugin.xml
./src/com/andrewgilmartin/confluence/plugins/script/JavaScriptPlugin.java

The JavaScriptPlugin class extends BasicMacro and must override the methods isInline(), hasBody(), getBodyRenderMode() and execute(). The isInline method specifies if the output of the plugin is suitable for an HTML span or block. The hasBody method specifies if the plugin has content, for example, as does the {code} macro. The getBodyRenderMode() specifies how Confluence is to handle the macro's output. Returning RenderMode.COMPATIBILITY_MODE specifies that the output is wiki text to be rendered as HTML. And, finally, execute does the work of the plugin.

package com.andrewgilmartin.confluence.plugins.script;

import java.util.Map;
import com.atlassian.renderer.RenderContext;
import com.atlassian.renderer.v2.macro.BaseMacro;
import com.atlassian.renderer.v2.macro.MacroException;
import com.atlassian.renderer.v2.RenderMode;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;

public class JavaScriptPlugin extends BaseMacro {

    private static ScriptEngineManager factory = new ScriptEngineManager();

    @Override
    public boolean isInline() {
        return true;
    }

    @Override
    public boolean hasBody() {
        return true;
    }

    @Override
    public RenderMode getBodyRenderMode() {
        return RenderMode.COMPATIBILITY_MODE;
    }

    @Override
    public String execute(Map params, String body, RenderContext renderContext) throws MacroException {
        try {
            ScriptEngine engine = factory.getEngineByName("JavaScript");
            engine.put("params", params);
            Object evalResult = engine.eval(body);
            String result = evalResult.toString();
            return result;
        }
        catch (ScriptException e) {
            throw new MacroException(e);
        }
    }
}

// END

The atlassian-plugin.xml is an XML declaration of the plugin. It must be in the root of the plugin's jar file. The file contains two main sections. The first is plugin-info which declares the plugin as a whole. The second is the (repeatable) macro which declares the specific plugin. The atlassian-plugin.xml has several further refinements of these sections. I was not able to find an XML schema for this file type.

What I have discovered about it is from reviewing Atlassian's own plugins. The atlassian-plugin element's key attribute seems to be a unique identifier but does not have a prescribed structure: I am here just using the JavaScritMacro class's package name. The macro element's name attribute is the name of the macro as used by the user. For example {javascript}1+2+3{javascript}

<atlassian-plugin
 name="JavaScript Macro Plugin"
 key="com.andrewgilmartin.confluence.plugins.script">

    <plugin-info>
        <description>A JavaScript macro plugin</description>
        <vendor name="Andrew Gilmartin" url="http://www.andrewgilmartin.com" />
        <version>1.0</version>
    </plugin-info>

    <macro
 name="javascript"
 class="com.andrewgilmartin.confluence.plugins.script.JavaScriptPlugin"
 key="com.andrewgilmartin.confluence.plugins.script.javascript">
        <description>A JavaScript macro plugin. Place the script to execute within the body of the macro.</description>
    </macro>

</atlassian-plugin>

The build script is

<project name="com.andrewgilmartin.confluence.plugins.script" default="dist">

    <property environment="env" />

    <path id="build.classpath">
        <fileset dir="${env.HOME}/lib/atlassian-confluence-4.3.3/confluence/WEB-INF/lib">
            <include name="**/*.jar"/>
        </fileset>
        <pathelement location="${basedir}"/>
    </path>

    <target name="dist">
        <javac
            classpathref="build.classpath"
            srcdir="${basedir}/src"
            destdir="${basedir}/src"
            source="1.5"
            target="1.5"/>
        <jar
            destfile="${basedir}/confluence-plugins-script.jar"
            basedir="${basedir}/src"/>
    </target>

    <target name="clean">
        <delete>
            <fileset dir="${basedir}" includes="**/*.class"/>
            <fileset dir="${basedir}" includes="**/*.jar"/>
        </delete>
    </target>

</project>

Replace ${env.HOME}/src/confluence-2.10.4-std/confluence/WEB-INF/lib/ with the location of your Confluence jar files.
For more information about building out your plugin do read the documentation and review the code for Atlassian's own plugins, for example Confluence Basic Macros, Confluence Advanced Macros, Confluence Information Macros and Chart Macro.

Atlassian uses the Spring toolkit in their development. Since Spring performs dependency-injection of objects matching the results of using Java reflection to find "bean" names, a lot of Atlassian's code looks like magic is happening. That is, there is no visible configuration or other assignment of object to values and yet the assignments have to made for the code to work. Spring is the magician.
And that is it. The built jar file is a valid Confluence 4.x plugin. Happy wiki scripting
Download an archive of the development tree (Thanks to David Wilkinson for the tool to create this data URI.)

Coda: I was asked why I created my own scripting plugin when two already exist in Atlassian's plugin repository. The initial reason was that our MySql 5.0 installation has too small a max_allowed_packet size and so Confluence was not able to install the existing script plugins into the database. The ultimate reason was that I knew what I wanted from the plugin and said to myself "how hard can it be?"

Tools & notes

What I am focusing on in this photo is that he is using a computer and a notebook. I do the same. It suddenly occurred to me that every crafts person, since the beginning of time, uses this work arrangement. Tool to one side and notes to the other side. This is a great pattern. Why do we not perpetuate it today? Why put both on one machine? Some places have. When I worked at Lotus we always had a "development" machine and an "administration" machine. Perhaps some places do.

I should be able to open my laptop for use on one side of the desk and then slip out the tablet to place on the other side. Just say'n.

Ubuntu

I am because we are.

Horizon charts

I have been slowly working on a metrics collection and monitoring service. There are many others, but I wanted something very simple to feed and to integrate with a wiki to monitor/display. During the development of the service I discovered Horizon charts by way of (the brilliant) Cubism.js. The goal of an horizon chart is to use the a minimum of vertical height without loss of precision. Horizon charts look like bar charts. However, horizon charts use both the top and bottom edge as axises. The top edge is used to show values below a threshold with the bars going downward and the bottom edge is used to show values above the threshold with the bars going upward. Further, to extend the range of a value beyond the height of the chart the values are "folded" and the folds layered on the chart. Each fold is drawn as a bar on top of the previous fold's bar. The illustration at Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations shows this very well: I was a little intimidated by Cubism.js and D3.js; they are sophisticated toolkits that will take more time for me to understand than I wanted to commit just now. Plus I really wanted to learn more about using HTML's canvas element, and so I set about my own implementation. The code at https://gist.github.com/4264347 is my first cut at the implementation. It works for a fixed, two fold horizon chart. The example chart below plot the values between +/-200 with folds at +/-100.
 

Update: Updated the working example to be more general. Removed the code from this posting in preference to the Gist. 

FYI: To run the code just download the gist and open it in a browser. To play with it copy the gist into the HTML panel of https://jsfiddle.net/ and run.

Blackout Friday!

Looking forward to Blackout Friday! I am so surprised that the national brand stores and the electrical grid operators have come together to turn off the power on Friday so we all can stop and view, with utter revulsion, the rapacity that is now "Christmas."

Permatext

THIS IS A WORKING DOCUMENT
Jeff Atwood recently proposed that Markdown users come together and create a unified specification. While I do think a unified specification would be useful I do not think Markdown is the right syntax. I have used it and many of the other structured text markup alternatives. For programmer types and anyone else used to machine readable documents, structured text markup is a great way to reduce the effort needed to get content on the page. However, HTML is a great way to reduce the effort needed too. The problem with both is that Markdown is too weak a syntax and HTML is too strong of a one. We need a middle ground.

Here is a proposal for a middle ground. I am going to call this syntax Permatext:

• A valid HTML document is a valid Permatext document.
• HTML class and style information is discarded by a Permatext processor.
• Any HTML end tags can be dropped if it is clear from the context that the end tag would be used. So, for example, an opening P tag automatically closes the previous P tag. An opening A tag need not use the closing tag if the link's title is a single word. Etc.
• A short closing tag, </>, closes the most recient open tag.
• Text within a code or pre tag need not be XML escaped. However, escaped XML will be honored. That is, if a valid named or numbered entity is found then it is honored. [tricky]
• TODO

For example, the document contained in Atwood's posting would be marked up as follows:

<h1>Lightweight Markup Languages</>

<p>According to <i>Wikipedia</>:

   <blockquote>
   A <a href="http://is.gd/gns">lightweight markup language</>
   is a markup language with a simple syntax, designed 
   to be easy for a human to enter with a simple text 
   editor, and easy to read in its raw form. 

<p>Some examples are:

   <li>Markdown
   <li>Textile
   <li>BBCode
   <li>Wikipedia

Markup should also extend to <i>code</>: 

<pre>
    10 PRINT "I ROCK AT BASIC!"
    20 GOTO 10

[NOTE: The A tag is hard to read. Perhaps assume href.]

  For all Permatext details see Permatext.info.

A simple monitor with SMS messaging

Now and then you need to watch a directory for changes. For me, this is mostly to ensure a data build is continuing. When the directory size remains constant then it is likely that the data build has failed. A simple monitor is to regularly check the directory's size and send an email message should it not change. If the email is to a email-to-SMS service then you will know sooner about the problem.
To regularly check use cron. While you can have a program that loops over the check and then sleeps for an interval this is very likely to fail at some time, that is, the program dies without warning. Cron will not fail in this way.

Since you are interested in change you need to record the previous value to compare to the new value. Just use a file. Anything else will likely fail.

So here is your crontab line (place all lines on one line) to run the command several times per hour

*/5 * * * * $HOME/bin/watch-directory-and-mail
                -t 1234567890@txt.att.net
                -o $HOME/var/data-2012-09-18.watch 
                -w $HOME/var/data-2012-09-18/

This says to watch the directory "$HOME/var/data-2012-09-18/", keep the record of previous directory size in "$HOME/var/data-2012-09-18.watch", and to raise an alarm by sending email to "1234567890@txt.att.net" (AT&T's email-to-SMS service). The "$HOME/bin/watch-directory-and-mail" script is

#!/bin/bash

function send-message() {
  echo "$3" | mail -s "$2" "$1" >/dev/null 2>&1
}

TO="$USER"
WATCH="$(pwd)"
OUTPUT="$WATCH.watch"
SUBJECT="$(basename $WATCH)"

while getopts "t:s:w:o:h" opt
do
 case $opt in
  t) TO=$OPTARG ;;
  w) WATCH=$OPTARG ;;
  o) OUTPUT=$OPTARG ;;
  s) SUBJECT=$OPTARG ;;
  *) echo "usage: $(basename $0) \
        -t email \
        -s subject \
        -w directory/file \
        -o status-file" ;;
 esac
done

V1=$(du -s $WATCH|cut -f 1)
if [ -r "$OUTPUT" ]
then
  V0=$(cat "$OUTPUT")
  if [ $V0 -eq $V1 ]
  then
    send-message $TO $SUBJECT "$(basename $WATCH) unchanged size at $V1"
  fi
else 
  send-message $TO $SUBJECT "$(basename $WATCH) initial size is $V1"
fi

echo $V1 >$OUTPUT

# END

Update: The script works just as well with a file as with a directory. If the rate of growth of the directory/file is slow then use "du -sb" to get byte counts. (OS X's du does not have the -b option.)

Update: If you have multi watching going on then the message sent is not very helpful. Have updated the code to enable to use to specify the message.

Update: If you don't want the script txting you during the night then change the crontab schedule to 9 AM to 5 PM, eg "*/5 8-17 * * * $HOME/bin/watch-directory-and-mail ...".

Using a simple command line script and Marked to keep and display notes

I was reading this and it sparked an idea for the combined use of Marked and a command line script. I often would like to simply collect notes as I work at the command line. The notes can all go into the same file and I will sort them out afterwards (GTD "inbox" style). So the simple command line

function n() { echo "$(date): $*" >>~/Inbox/Note.txt; }

defines a "n" command that works well enough. Usage is simply to type "n this is my note" at the command line and the script will put the timestamp and the "this is my note" text in to the file ~/Inbox/Note.txt.

The spark, however, was that I could use Marked to display an automatically updated view of the note file's contents. Further, I could add a little markdown notation to the script to help visually separate the notes. The final script is

function n() { printf "\n### %s\n%s\n" "$(date)" "$*" >>~/Inbox/Note.txt; }

Here the date is formatted as a header and the note's text as a paragraph.

The end result is a window like this