Multi-Terra Helio Cell Vehicle

My children would have loved this Multi-Terra Helio Cell Vehicle 407 cockpit control. I especially like the addition of the fans!

Streamlit and a trivial cities explorer

I spent a little time this morning learning the basics of Streamlit. Earlier this month I had found a US cities dataset¹ and decided to use Streamlit to explore city populations. 

This dashboard shows each state's population along with a state selector to dynamically show a state's city's population along with a map showing its location and relative population size. 

While it took me much of the morning to figure out how to use the Streamlit and Pandas SDKs, along with a side-quest on how to debug a Streamlit application in Visual Studio Code, the final application is hearteningly trivial at barely a dozen lines of Python.

1. Unfortunately, the dataset of not very good based on what I know about RI and NH cities and towns.

Board game UX

I don't know if the game Star Wars: Outer Rim is enjoyable, but the game's physical design is amazing. Which reminds me I have What the Tech World Can Learn from Video Game UX lined up to watch.

Update: The video was about using gamification to enhance learning to use a handheld ultrasound scanner. Nothing new.

Ad hoc systems for managing work

I love seeing people's systems for managing their work. Even those of fictional people. This short from The Bear on managing the restaurant's guests and their orders is great. 

A list of physical infrastructure monitoring tools

A small list of found physical infrastructure monitoring tools

Flights into Providence
https://flightaware.com/live/airport/KPVD

Shipping in Narragansett Bay
https://www.marinetraffic.com/en/ais/home/centerx:-71.5/centery:41.4/zoom:10

RI DOT Traffic Camera
https://www.mapquest.com/?center=41.809012,-71.40841&zoom=12&layer=traffic

Gritters in Scotland
http://scotgov.maps.arcgis.com/apps/webappviewer/index.html?id=2de764a9303848ffb9a4cac0bd0b1aab

NYC roads snowplowed

https://plownyc.cityofnewyork.us/plownyc/

Furthermore on dashboards for maintenance developers

What would a dashboard for visualizing and managing the flow of messages between actors and their dependents look like? My skills run in the direction of information architecture and not visual design, unfortunately. I can tell you what should be on the dashboard, the relationships between the contents  controls, the visual and interactive priorities of these, and how all of this changes over time. On a Friday night, this is not something I am prepared to do; which is really an excuse as I am nearing the end of my interest in this thought experiment. Nevertheless, here are a few ideas.

The transitions of a message between the actor and the dependent are

- Actor sends the message to Dependent, or fails to send it.

- Dependent receives the message, or fails to receive it.

- Dependent responds with a response-message to the received-message, or fails to respond.

- Actor receives the response-message, or fails to receive it.

We can symbolize these statuses as

A display for showing details of actors, messages, and listeners (dependents) might be

Each message is given a row and the delivery and response status is detailed for each listener. My bare tabular example would have much more detail on it in practice. This is not a overview display and so concerns with overwhelming the user with too much detail is not of relevance. If you only have to show a few listeners and perhaps a brief time window of messages this display will quickly show specific problem with delivery and response. 

Inverting and aggregating the above example leads to the tighter display

We are, again, showing each message, but the columns now indicate how many listeners are in each of the delivery or response states.

Lastly, as everyone loves a set of small multiples graphs here is 40 minutes of activity and status where the green line is the number of listeners, the black lines show positive trends and the red negative ones. 

Man, that is an ugly chart (and is playing loose with small multiples).

More on dashboards for maintenance developers

Yesterday I wrote about providing tools to visualize & manage the internal workings of applications for maintenance developers. That note was long on advocacy and sort on detail. This note somewhere between those two.

We are all familiar with run-time debuggers. We can get our job done with gdb [1], but for day in and day out work a visual debugger is a must have. A debugger is a point-in-time dashboard of the execution of the programming language used to implement the application. It is not a dashboard of the application, it does not directly represent the application's data or processing models. Nor does it provide a history of where the application was or a prediction of where it will be heading too next. Nevertheless, it is a useful analogous image of the application dashboard to keep in your mind.

If, for example, your application relies on the observer pattern [2] then you will need to visualize & manage the actors (observed), their dependents (observers), and the messages. The effort to implement this dashboard can be significant, which is the exact opposite of implementing the pattern itself. The pattern is often implemented as a list of callbacks. This implementation is so insignificant that your high performing developers probably choose not to use a framework at all, but to re-implement for each use. And there is the root of the trouble ahead for the maintenance developers, there is no one place to watch this running application. Callbacks and messages are being created and dispatched all over the place.

Suppose there had been a reusable implementation. The management of the listeners was done in a single place. The dispatching of messages to the listeners was done in a single place. Each reuse required that the developer explain the purpose, configuration, and expectations of this reuse. For example, "Status of the nightly recalculations of base prices: the messages are life cycle events; messages are ticketed and numbered for correlation; the listeners can be local or remote; the listeners are contacted asynchronously; no receipt acknowledgement is wanted, but when given will be logged; expect a dozen messages over a 2 hour period in every 24 hour period to be sent to a few hundred listeners." When this pattern is visualized on the dashboard not only is the context clear, but timelines of planned to actual and switches to toggle synchronicity will automatically be included. And what if you could dynamically toggle receipt acknowledgement? How many of your application's ailments are the result of lost messages sent to remote listeners?

The rise of DevOps has brought with it both tools and new (renewed?) appreciation for visualizing processes and their artifacts. Lets keep on pushing these ever deeper into the application itself.

[1] gdb, https://www.gnu.org/software/gdb/
[2] Observer pattern, https://en.wikipedia.org/wiki/Observer_pattern

We all want visibility into a running application

We all want visibility into a running application. The need becomes acute when the application is ailing and that visible schedule, map, and cartogram of the data and processing will swiftly guide you to the failing part.

What is less recognized is that these same visuals are needed in maintenance. Your original, high performing developers are unlikely to be around after the application has been in production for a few years. They will have moved on to new green pastures. If you are lucky and they are still with the company then they will be somewhat available to the maintenance developers. If a little cantankerous about that.

The maintenance developer usually has little broad knowledge of the application’s operation. Her first task at resolving an application’s ailment will be to find suitable entrance and exit points. This is not an easy task for our applications now run continuously, across multiple processes and hosts, and are highly asynchronous. Hopefully, those high performing developers adhered to standards, used a common framework throughout, and kept to best-of-class external libraries and supporting tools. If they did then she might have a straightforward resolution path. In all likelihood, however, those high performing developers probably invented too much and documented too little. [I speak from the experience of my younger self.]

What the maintain crew needs is not logs alone. They need meta tools that connect to the running application and visually show its operation. Once connected the tool must be able to slow down the application to a human pace. Restrict it to one active user at a time. It might even need to switch it to run synchronously.

These are not easy features to implement, but the cost of not doing it is not just the ever lengthening backlog of ailments to fix, but the erosion of customer confidence and, in the end, loss of customers.

All modern applications have three users. The first is the customer. The second is the customer support. The third is the maintenance crew. It is way past time that the maintenance developer is provided the meta tools needed to look inside the running application and keep it fit for market.

BitBar shout out

I use the MacOS menu bar tool BitBar for a few uses. I have the time in UTC and the status of a few servers. Behind these presentations are very simple bash or perl scripts. I was reading today of a small utility to show how many days remained until a set of events. The BitBar equivalent is I will leave removing past events from the output to the reader.

Incident Response Slack App

As we learn how to better use Slack we are experimenting with different ways of managing our response to incidents, aka emergencies. One experiment is that when an incident is discovered the existing #incident-response channel is used to send an alert to those on-call. We then immediately create a new channel for only the new incident's staffing and communications. While we rarely have overlapping incidents, having a dedicated channel does prevent the interleaving of messages about other incidents, too many tangents, and only those working the incident are disturbed by @channel or @here messages. When the incident is resolved the channel's messages can be copied into the beginnings of the post-mortem document, and then archived.

During the response, tasks emerge that need to be assigned and tracked. Slack itself is not good at this alone. There are many applications for task management that can be made accessible via Slack slash-commands. For incident response tasks, however, these general purpose applications were too focused on the user and not enough on the channel. When listing tasks we only want to see those for this incident. Their extra features also had a cognitive weight that I brisselled at. Overall, their fit for purpose was poor.

What was needed was a task manager with a scope limited to one channel. The task manager would be installed in the workspace, and so accessible to everyone, everywhere without configuration, but when in use was channel focused. The task manager needed to support these simple the use cases

• Adding tasks with a description, assignments, and a status.
• Updating a task’s description, assignments, or status. Only notify the channel when the changes are pertinent to all.
• Listing the tasks with optional criteria.

These use cases turned into the /ir slash-command

/ir description [ user … ] [ status ]
/ir task-id [ description [ user … ] [ status ]
/ir [ all | finished ] [ user … ] [ status … ]

If you are interested in the implementation then see github.com/andrewgilmartin/com.andrewgilmartin.incidentresponse

Two kinds of noise

I started to read Twitter again. The user experience sucks. Twitter is mostly for information delivery. Let's look at a page of contents as presented and as is actually present:



Three tweets in 100 square inches of screen space. (Large editorial photos and illustrations do not count as content.)

Grumble about messaging conversation indistinguishability

Why, why! oh mighty God, do Slack, Skype, Messenger, iMessage, and every other messaging application not make any visual distinction between two or more conversations? Squint at the screen and you will see that every conversation looks identical -- even across messaging applications! How many times have you and I and, most likely, the developers of these applications sent a message to the wrong conversation? Lots. Please, oh pretty please, allow me to at least change the background color of conversations.

Meeting room geek lights

I have been giving more thought to the meeting room occupancy display. Right now I am considering what would be a minimal presentation. There are 2 reasons for this. The first is that the problem is not about showing details of who has reserved the room, but, instead, is the room free for squatting and, if so, for how long? The second problem is a technical one and that is the display device should run for weeks on a single battery charge. (I am assuming no mains electricity.)

So here is the new display.

⬤⬤⬤⬤⬤⬤⬤⬤⬤⬤⬤⬤

Each light, red and green, represents 10 minutes of time. The display shows a total of 2 hours of time. The above display indicates that the room is reserved for another 40 minutes, free for the following 60 minutes. and then occupied for at least 20 minutes -- it could be more, but the display can't indicate for how much more.

⬤⬤⬤⬤⬤⬤⬤⬤⬤⬤⬤⬤

Here the room is free for another 20 minutes, then reserved for 30 minutes, and the free again for at last 70 minutes.

The more I look at this the more I think that this is the solution.

Update 2025-03-17: The Japanese stationery manufacturer King Jim has created a similar product, the Visual Time Bar (VBT10). When I wrote the original post I had not considered the cultural situation of reading the display from left-to-right or right-to-left. The VBT10 is read right-to-left.

How are branches being used? A visualization

I really like the terminal for visualizations. There is something in its restrictions that gives it the right balance of detail for rough visualizations. The fixed width font, the ASCII character set, the small overall grid, etc, all help. I wanted to see how our developers are using branches over the last 180 days. This script gets the branches log and outputs a chart for days that had checkins by each developer. Note that a day without checkins by anyone will have no column in the chart.

The code is

#!/bin/bash

date1=$(date -v -180d +'%Y-%m-%d')
date2=$(date -v -1d +'%Y-%m-%d')
svn='svn+ssh://.../branches'

svn log --revision {$date1}:{$date2} $svn | perl -e '
  my %D = (); # dates
  my %U = (); # users
  while( <> ) {
    if ( /\| (.*?) \| (\d\d\d\d-\d\d-\d\d) / ) {
      my $u = $1;
      my $d = $2;
      $D{$d} ||= {};
      $D{$d}->{$u} += 1;
      $U{$u} = 1;
    }
  }
  my @d = sort keys %D;
  my ( $l ) = reverse sort map { length($_); } @d; # max length
  for my $u ( sort keys %U ) {
    printf( "%*s %s | ", -$l, $u, $d[0] );
    for my $d ( @d ) {
      print $D{$d}->{$u} ? "*" : "-";
    }
    printf( " | %s\n", $d[$#d] );
  }
'

# END

Update: A failing of this script is that it only shows days with activity. Days without activity by any developer will be missing from the timeline. I replaced this script with this one

#!/usr/bin/perl -w

use strict;
use DateTime;

my $svn_url='svn+ssh://.../branches';

my $stop = DateTime->today();
my $start = $stop->clone()->subtract( days => shift @ARGV || 90 );

my %W = (); # week days
my %D = (); # dates
my %U = (); # users

# initialize the dates and the weekdays hashs with days between the start and stop dates
for ( my $next = $start->clone(); $next->compare($stop) <= 0; $next->add( days => 1) ) {
 my $F = $next->strftime('%Y-%m-%d');
 $D{$F} = {};
 $W{$F} = $next->day_of_week();
}

# collect the svn usage data
if ( open( IN, "svn log --revision \{$start\}:\{$stop\} $svn_url |" ) ) { 
 while( <IN> ) {
  if ( /\| (.*?) \| (\d\d\d\d-\d\d-\d\d) / ) {
   my $u = $1;
   my $d = $2;
   if ( defined $D{$d} ) {
    $D{$d}->{$u} += 1;
    $U{$u} = 1;
   }
  }
 }
 close(IN);
}

my @d = sort keys %D;
my ( $l ) = reverse sort { $a <=> $b } map { length($_); } keys %U; # max user length

# output the chart
for my $u ( sort keys %U ) {
 printf( "%*s %s | ", -$l, $u, $d[0] );
 for my $d ( @d ) {
  print $D{$d}->{$u} ? "*" : "-";
 }
 printf( " | %s\n", $d[$#d] );
}

# output the line of mondays
printf( "%*s %*s | ", -$l, "", length($d[0]), "" );
for my $d ( @d ) {
 print $W{$d} == 1 ? "M" : " ";
}
printf( " |\n" );

# END

A gnuplot example

If you searched for Gnuplot in this blog you know I am a fan of the tool. It is easy to install, somewhat easy to use, and very likely someone has already had and solved the exact same problem you are having now and wrote a blog entry about it. This is my contribution to the globally distributed Gnuplot cookbook.
I want on a single chart the lines graphs of count of all queries, count of matching queries, and mean duration of all queries. This chart gives me the temperature of the previous day's query load which is a useful qualitative factor for capacity planning. The data retrieval results in records representing a 10 minute interval with counts of all queries, counts of matching queries, and mean durations. This is  then plotted to give a chart like the following:

The file of records contains values that looks like

2014-07-07T00:10:00     10449   3850    934
2014-07-07T00:20:00     18445   5676    407
2014-07-07T00:30:00     18497   6507    535
2014-07-07T00:40:00     11419   4463    734
2014-07-07T00:50:00     11634   4454    724
2014-07-07T01:00:00     11960   5077    808
...

The Gnuplot script is

set terminal png size 800, 400 small interlace
set output "/tmp/chart.png"
set size ratio 0.5

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"

set xlabel "Hour"
set format x "%H:%M"

set ylabel "Count"
set ytics nomirror
set yrange [0:40000]

set y2label "Milliseconds"
set y2tics
set y2range [0:4000]

set title "MetaData matching search counts and average durations (10 min intervals) for 2014-07-07"

plot \
  "/tmp/chart.data" using 1:2 title "Query count" with lines, \
  "/tmp/chart.data" using 1:3 title "Matching count" with lines, \
  "/tmp/chart.data" using 1:4 title "Request duration" axes x1y2 with lines

The chart is produced daily and it is useful to visually align several days of charts to look for patterns. For comparable charts this requires that the charts all have the same y-axes ranges. To do this Gnuplot's range setting is used. If you do not want a fixed range then remove the set yrange ... and set y2range ... lines.

I hope this helps someone else with their Gnuplot use.

Too little skeuomorphism!

I find iOS 7's UI boring rather than utilitarian and/or minimalistic. I am, frankly, surprised at my reaction to this utilitarianism as it is something I have wanted for some time. How I have it, I think Apple went too far in removing the skeuomorphism.

Horizon charts

I have been slowly working on a metrics collection and monitoring service. There are many others, but I wanted something very simple to feed and to integrate with a wiki to monitor/display. During the development of the service I discovered Horizon charts by way of (the brilliant) Cubism.js. The goal of an horizon chart is to use the a minimum of vertical height without loss of precision. Horizon charts look like bar charts. However, horizon charts use both the top and bottom edge as axises. The top edge is used to show values below a threshold with the bars going downward and the bottom edge is used to show values above the threshold with the bars going upward. Further, to extend the range of a value beyond the height of the chart the values are "folded" and the folds layered on the chart. Each fold is drawn as a bar on top of the previous fold's bar. The illustration at Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations shows this very well: I was a little intimidated by Cubism.js and D3.js; they are sophisticated toolkits that will take more time for me to understand than I wanted to commit just now. Plus I really wanted to learn more about using HTML's canvas element, and so I set about my own implementation. The code at https://gist.github.com/4264347 is my first cut at the implementation. It works for a fixed, two fold horizon chart. The example chart below plot the values between +/-200 with folds at +/-100.
 

Update: Updated the working example to be more general. Removed the code from this posting in preference to the Gist. 

FYI: To run the code just download the gist and open it in a browser. To play with it copy the gist into the HTML panel of https://jsfiddle.net/ and run.

I want more "geek lights"

Geek-lights are the indicator lights embedded on the surface of your electrical equipment. The green "On" light. The blinking light on your ethernet port or USB thumb-drive telling you data is being transferred. Most of them tell you redundant or otherwise obvious facts. I don't need an "on" light for the monitor I am using. It is useful information when in power saving mode, however. Apple seems to have learned this.


However, geek-lights are incredibly useful for other kinds of information. Information that is different for you and me. For example, one to indicate if traffic is too heavy to venture out now. Another to tell me that I have email. Another to tell others that I am in a conference call and not just talking to myself again.

I would like to have a box of these geek-lights that I can use each for a specific purpose. The geek-light would be nothing more than a shallow and wide cylinder with the light on the top, temporary adhesive on the bottom, and the electronics sandwiched in between. Let's call them eyelets as they let you see something ordinarily invisible. Each eyelet is individually addressable by a short range wireless signal. An eyelet receives a signal that indicates whether it is illuminated or not. Eyelets don't send signals back. I can imagine the hardware being something akin to Logitech's Unifying receiver -- that is, unobtrusive.

The question now is, should eyelets have a battery charge indicator light?

Need multiple-view calendar

The calendar displays that we have had forever just don't help me anymore. Too much information and/or too much interaction need. I want my calendars to show me what I want without interaction. I want to distinguish what is common and what is unique. Common, for example, are events such as every workday at 9 AM I have a standup meeting; Thursdays I deploy new releases; Every second Monday of the month during a school year I have a PTA meeting. Unique ones are the dentist appointment and the odd professional event at night. I don't really want one-view calendar anymore. I need multiple-view calendar like he one illustrated here.

The top chart contains my unique events over the last 2 and next 8 days. The second chart contains my common events over the last 2 and next 8 days. The bottom left chart is a list of common events for the next several weeks. And the bottom right chart is a list of unique events for the next several weeks.

Since we have separated events that actually occupy the same timeline we need to rejoin them somehow. My initial thought was to add a ghost to each chart that signifies the presents of an event in another chart, but I am not wedded to this.

Does anyone make a calendar like this, either visually or purposefully?

In praise of gnuplot's dumb terminal support

I have to say, again, I find gnuplot's dumb terminal support is so useful when you are at the command line and need to see a plot of some data. The plot is very rough but this is usually enough to give you enough insight into the data as to whether or not to continue to exploring it. The script I am using now is

#!/bin/bash

function show_usage {
 echo \
  usage: $(basename $0) \
  [-x label] \
  [-y label] \
  [-t title] \
  [-s width:height] \
  [-f time-format-strftime] \
  timeseries-input ...
}

function swap {
 echo $2 $1
}

function parse_size {
 W=$(expr $1 : "\\([0-9]*\\):[0-9]*")
 H=$(expr $1 : "[0-9]*:\\([0-9]*\\)")
 echo "$W $H"
}

TIMEFMT="%Y-%m-%dT%H:%M:%S"
XLABEL="Time"
YLABEL="Units"
TITLE="Timeseries"
SIZE=$(swap $(stty size))

while getopts "x:y:t:f:s:h" opt
do
 case $opt in
  x) XLABEL=$OPTARG ;;
  y) YLABEL=$OPTARG ;;
  t) TITLE=$OPTARG ;;
  f) TIMEFMT=$OPTARG ;;
  s) SIZE=$(parse_size $OPTARG) ;;
  h) show_usage ; exit 0 ;;
  *) show_usage ; exit 1 ;;
 esac
done
shift $(expr $OPTIND - 1)

for INPUT in $*
do
 if [ "$INPUT" = "-" ]
 then
  INPUT=$(mktemp /tmp/timeplot.XXXXXXXXXX)
  cat > $INPUT
 fi

 gnuplot <<EOH
  set terminal dumb $SIZE
  set autoscale
  set xdata time
  set timefmt "$TIMEFMT"
  set xlabel "$XLABEL"
  set ylabel "$YLABEL"
  set title "$TITLE"
  plot "$INPUT" using 1:2 with lines
EOH
done

# END

For example, if the data in /tmp/data is

2011 34
2012 34
2013 56
2014 24

then this can be quickly plotted using

timeplot -f %Y /tmp/data

and get