Creating a Maven project for the common packages

This posting continues the series on moving from an Ant to a Maven build.

When you have replaced all your external libraries with Maven dependencies you can move on to the next stage in moving from Ant to Maven. This stage is to separate out one or more of your Java packages into their own Maven projects. There are a number of good reasons to separate out packages, but the practical one is that Maven is best used to build one deliverable.

Every organization has a "common" package that contains classes adding general purpose functionality. There are likely several "utility" packages that augment the standard Java libraries for IO, networking, text, collections, functions, streams, etc. There are packages for data structures and algorithms that were sufficiently independent of the primary application to be have been separated out from its packages. These common packages are also likely used by supplementary tools to your primary application. If you already build separate jars for your common packages this step is not that much work. If you don't then you are likely to find some unexpected and unwanted entanglements with application code that needs to be worked out.

A note about the version control. Separating out packages does not mean you also have to move from a monorepo if that is what you are currently using. Moving away from a monorepo does simplify assuring that your common packages are isolated from your application code, however. If you have been building separate jars for your common packages then you have already established build mechanisms to maintain the isolation. If you haven't then this is something you will need to do in your monorepo. This tutorial, however, assumes you are not using a monorepo.

The common packages will be your first Maven project. Create an empty Maven project using archetype:generate

mvn archetype:generate \
  -DgroupId=com.andrewgilmartin \
  -DartifactId=system-common \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DarchetypeVersion=1.4 \
  -DinteractiveMode=false

Change the "groupId" and "artifactId" appropriately. The command will create the tree

.
└── system-common
    ├── pom.xml
    └── src
        ├── main
        │   └── java
        │       └── com
        │           └── andrewgilmartin
        │               └── App.java
        └── test
            └── java
                └── com
                    └── andrewgilmartin
                        └── AppTest.java

Move your common packages into the src/main/java tree. Move any tests you have for these packages into the src/test/java tree. (Delete AppTest.java from both trees at some point.) Your IDE will likely be very helpful in moving files and even version histories.

A build now will very likely fail due to Java language level and missing dependencies. Edit "pom.xml" and update the "maven.compiler.source" and "maven.compiler.target" properties appropriately, eg for Java 8 use

<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.9</maven.compiler.target>

Try a compile just to see what happens!

mvn package

Lots of missing dependencies! Using the information you gathered before add each dependency under the pom.xml file's <dependencies/> element. For example, if the common package is dependent on Apache's HTTP Client 4 then add

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.10</version>
</dependency>

The org.apache.httpcomponents are located at Central and this is most likely already listed in your Maven installation's $HOME/.m2/settings.xml file. If it is not or you would rather not depend on the installation's settings (a good practice) then add the <repository/> element under the <repositories/> element. If the <repositories/> is missing from pom.xml then add it above the <project/> end tag.

<project>
  ...
  <repositories>
    <repository>
      <id>central</id>
      <name>Central Repository</name>
      <url>https://repo1.maven.org/maven2/</url>
    </repository>
  </repositories>
</project>

Continue adding dependencies and building and testing until you get a clean result.

You now have your common packages in their own jar

target/system-common-1.0-SNAPSHOT.jar

After installing this in you local Maven cache, $HOME/.m2/repository/, you can use it in your Ant build.xml. To install it use

mvn install

And you should see the logged output similar to

[INFO] Installing /home/ajg/src/system-common/target/system-common-1.0-SNAPSHOT.jar to /home/ajg/.m2/repository/com/andrewgilmartin/system-common/1.0-SNAPSHOT/system-common-1.0-SNAPSHOT.jar
[INFO] Installing /home/ajg/src/system-common/pom.xml to /home/ajg/.m2/repository/com/andrewgilmartin/system-common/1.0-SNAPSHOT/system-common-1.0-SNAPSHOT.pom

To use the installed jar in your Ant build you need only to add the new dependency, eg

<mvn-dependencies pathId="runtime-dependencies.classpath" ... >
  ...
  <dependency groupId="com.andrewgilmartin" artifactid="system-common" version="1.0-SNAPSHOT"/>
  ...
</mvn-dependencies>

Remove from the system's source tree all the common packages now compiled into system-common-1.0-SNAPSHOT.jar. Also remove all the <dependency/> elements that were only used by the common packages as Maven already knows the jar's dependencies (from its pom.xml file that was also installed into $HOME/.m2/repository).

If you have other common packages, eg enhancements to Swing or extensions to J2EE, then repeat the above procedure for each of them.

A series of posting about moving an ancient Ant build to Maven

This is a table of contents page for a series of posting about moving an ancient Ant build to Maven.

  1. Moving to Maven from an ancient Ant build
  2. Running a Maven repository manager
  3. Running a Maven repository manager using Apache WebDav
  4. Creating a Maven project for the common packages
  5. Creating Maven projects for the core packages and command line tools
  6. Creating a Maven project for your web application

The code for examples is at

https://github.com/andrewgilmartin/maven_examples

Running a Maven repository manager using Apache WebDav

This posting continues the series on moving from an Ant to a Maven build.

Implementing a Maven repository manager using Apache's HTTPd with WebDav is straightforward. For this example, create the directory tree

.
├── apache2
│   ├── httpd.conf
│   └── logs
└── www
    └── repository

For the purposes of this example the directory tree is in /var/maven/

The httpd.conf file contains

LoadModule mpm_prefork_module libexec/apache2/mod_mpm_prefork.so
LoadModule unixd_module libexec/apache2/mod_unixd.so 
LoadModule authz_core_module libexec/apache2/mod_authz_core.so  
LoadModule access_compat_module libexec/apache2/mod_access_compat.so   
LoadModule mime_module libexec/apache2/mod_mime.so 

LoadModule dav_module libexec/apache2/mod_dav.so
LoadModule dav_fs_module libexec/apache2/mod_dav_fs.so

ServerRoot ${BASEDIR}/apache2
LogLevel warn
PidFile logs/httpd.pid

Listen 8080
ServerName localhost:8080
ServerAdmin andrew@andrewgilmartin.com

AddType application/octet-stream .sha1
AddType application/octet-stream .md5
AddType text/xml .xml
AddType text/xml .pom
AddType application/octet-stream .jar
AddType application/octet-stream .war

DocumentRoot ${BASEDIR}/www

DavLockDB DavLock
<Directory />
    DAV On
</Directory>

Start the HTTP server (in the foreground) using

BASEDIR=/var/maven/ httpd -f /var/maven/apache2/httpd.conf -X

For Maven deploy to this repository manager you need to make two changes to your pom.xml. The first is to include the extension in the <build/> section

<extensions>
  <extension>
    <groupId>org.apache.maven.wagon</groupId>
    <artifactId>wagon-webdav-jackrabbit</artifactId>
    <version>3.3.4</version>
  </extension>
</extensions>

The second is to use the "dav:" prefix to your <distributionManagement/> element URLs

<distributionManagement>
  <snapshotRepository>
    <id>neighborhood-snapshots</id>
    <name>Neighborhood Snapshots</name>
    <url>dav:http://localhost:8080/repository/</url>
  </snapshotRepository>
  <repository>
    <id>neighborhood-releases</id>
    <name>Neighborhood Releases</name>
    <url>dav:http://localhost:8080/repository/</url>
  </repository>
</distributionManagement>

To deploy your project use

mvn clean package deploy

This posting is part of the series about moving to Maven from an ancient Ant build.

Running a Maven repository manager

This posting continues the series on moving from an Ant to a Maven build.

The availability of most Java libraries in Maven's repositories is the most valuable of its features. It is difficult to say if Maven's build and deploy tools were the catalyst or that its appearance simply coincided with a time when it was obvious to all that centralized repositories with a common access protocol was needed. The why does not matter much now as we have Maven and it is well established in the Java ecosystem.

A Maven repository (aka a site) is a well defined file system accessible via one of several protocols. If you are using HTTP then the repository server needs to implement the GET and PUT methods for files. For example, if your dependency is (pom.xml)

<dependency>
  <groupId>com.andrewgilmartin</groupId>
  <artifactId>example1</artifactId>
  <version>1.0</version>
</dependency>

and the repository is

<repository>
  <id>neighborhood</id>
  <url>http://localhost:8080/repository/</url>
</repository>

then when you use Maven to build, eg

mvn -U clean package

Maven will initially GET the dependency's files

/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.sha1

and then based on the detail in maven-metadata.xml it get will GET the files

/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar.sha1
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom.sha1

When you deploy, eg

mvn deploy

using the distribution management (pom.xml)

<distributionManagement>
  <snapshotRepository>
    <id>neighborhood-snapshots</id>
    <name>Neighborhood Snapshots</name>
    <url>http://localhost:8080/repository/</url>
  </snapshotRepository>
  <repository>
    <id>neighborhood-release</id>
    <name>Neighborhood Releases</name>
    <url>http://localhost:8080/repository/</url>
  </repository>
</distributionManagement>

Maven will first GET the files

/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.sha1
/repository/com/andrewgilmartin/example1/maven-metadata.xml
/repository/com/andrewgilmartin/example1/maven-metadata.xml.sha1

and then PUT the files

/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.md5
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.sha1
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar.md5
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar.sha1
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom.md5
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom.sha1
/repository/com/andrewgilmartin/example1/maven-metadata.xml
/repository/com/andrewgilmartin/example1/maven-metadata.xml.md5
/repository/com/andrewgilmartin/example1/maven-metadata.xml.sha1

The Maven client does all the work to ensure that the repository has all the necessary metadata about the deployed package. That is, the HTTP server needs no Maven intelligence. A dedicated repository manager, like Apache Archiva, Sonatype Nexus, and JFrog Artifactory, does far more than just manage files, but for small collections these features are not needed.

For the next stage in migrating your Ant project to Maven you need only configure an HTTP server for GET and PUT. If the HTTP server uses the file system directly, then it will need to be able to create the intermediate directories for any PUT files. During the Ant to Maven transition, however, you can use my simple HTTP server at https://github.com/andrewgilmartin/filehttpserver/. For production you will likely want to use, for example, Apache2 and WebDav.

Moving to Maven from an ancient Ant build

This posting initiates the series on moving from an Ant to a Maven build.

You use Ant to build your system of one or more deliverables. You have one build.xml file. You have toyed with separating out the deliverables into individual build.xml files, but there seemed not to be enough value in doing so. Over time your build.xml has actually had less unique things to do, eg RMI no longer needs a compile step, you moved database initialization and configuration into another tool, and the newer React web front end is built elsewhere. Effectively the build.xml just complies the source, runs the tests, and assembles one or more jars or wars.

The Ant build while now simpler has a number of downsides. You still manually manage the large set of external libraries. Perhaps those libraries have not been updated in years as it is too cumbersome to track down their origin and incorporate the most recent, viable version. Your IDE does not integrate well with your bespoke build environment. The IDE's tools for creating test classes, running tests, showing library JavaDoc, navigating into library sources, and code completion is ineffective or unusable.

It would be better all-round if you moved to Maven. This series of postings is an approach to accomplishing that.

If you have not yet installed Maven then do so from

https://maven.apache.org/install.html

The first stage to moving to a Maven build is to replace the external libraries with Maven dependencies. You don't need to replace Ant to do this thanks to the Maven Ant Tasks

https://maven.apache.org/ant-tasks/

Download the most recent jar and place it in your build libraries. Now add the following Ant task definition to your build.xml (near the top of the file)

<taskdef
  name="mvn-dependencies"
  classname="org.apache.maven.artifact.ant.DependenciesTask"
  classpath="${basedir}/buildlib/maven-ant-tasks-2.1.3.jar"
  onerror="report"/>

Update the "classpath" attribute as necessary. Do a clean build to ensure that the "mvn-dependencies" task is accessible.

The Ant build likely has different and/or overlapping external libraries for building, testing, and running. You will be replacing these with equivalent dependencies declared in <mvn-dependencies/> elements. Add the following elements to your build.xml (after the <taskdef/>)

<mvn-dependencies
  pathId="build-dependencies.classpath"
  sourcesFilesetId="build-dependencies-sources.classpath"
  javadocFilesetId="build-dependencies-javadoc.classpath"
  settingsfile="${basedir}/maven-settings.xml" >
  <!--
  <dependency groupId="" artifactid="" version=""/>
  -->
</mvn-dependencies>

<mvn-dependencies
  pathId="test-dependencies.classpath"
  sourcesFilesetId="test-dependencies-sources.classpath"
  javadocFilesetId="test-dependencies-javadoc.classpath"
  settingsfile="${basedir}/maven-settings.xml" >
  <!--
  <dependency groupId="" artifactid="" version=""/>
  -->
</mvn-dependencies>

<mvn-dependencies
  pathId="runtime-dependencies.classpath"
  sourcesFilesetId="runtime-dependencies-sources.classpath"
  javadocFilesetId="runtime-dependencies-javadoc.classpath"
  settingsfile="${basedir}/maven-settings.xml" >
  <!--
  <dependency groupId="" artifactid="" version=""/>
  -->
</mvn-dependencies>

The "pathId" is going to be used in conjunction with your existing <path/> elements for constructing the classpaths used in building, testing, and running. Eg,

<path id="build.classpath">
  <path refid="runtime-dependencies.classpath"/>
  <path refid="build-dependencies.classpath"/>
  <!-- ... -->
</path>

<path id="test.classpath">
  <path refid="runtime-dependencies.classpath"/>
  <path refid="build-dependencies.classpath"/>
  <path refid="test-dependencies.classpath"/>
  <!-- ... -->
</path>

<path id="runtime.classpath">
  <path refid="runtime-dependencies.classpath"/>
  <!-- ... -->
</path>

The <mvn-dependencies/> "settingsfile" attribute references a Maven configuration. This file is used to specify the Maven repository managers, ie where dependences can be found. You should place this file under version control alongside the build.xml. The base file is

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <profiles>
    <profile>
      <id>default</id>
      <repositories>
        <!--
        <repository>
          <id></id>
          <url></url>
        </repository>
        -->
      </repositories>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>default</activeProfile>
  </activeProfiles>
</settings>

You are now ready to use Maven dependencies. For each external library you will need to know its name, version, and, if possible, origin URL. To find these search the Maven repository index at

https://mvnrepository.com/

Eg, if you are using Apache Codec at runtime then search for "apache codec" and navigate to the dependency version page

https://mvnrepository.com/artifact/commons-codec/commons-codec/1.8

You will need to record the group id, artifact id, version, and repository URL. Get the group id, artifact id, and version from the Maven tab, ie

<dependency>
  <groupId>commons-codec</groupId>
  <artifactId>commons-codec</artifactId>
  <version>1.8</version>
</dependency>

Add this to the runtime <mvn-dependencies/> element in build.xml, eg

<dependency groupId="commons-codec" artifactid="commons-codec" version="1.8"/>

Get the repository URL from the Repositories field, ie following the "Central" link reveals that the URL is

https://repo1.maven.org/maven2/

If the repository URL is unique then update maven-settings.xml, eg under <repositories/> add

<repository>
  <id>Central</id>
  <url>https://repo1.maven.org/maven2/</url>
</repository>

Now delete the external library's jar. Do a clean build to test that you have correctly replaced the dependency.

Continue replacing external libraries with dependencies until you are done. It has been my experience that this will progress far quicker than you initially worried it would.

You may find a few external libraries that were themselves never built with Maven or were never deposited to a repository. For these you will need to continue to use the manually managed jar. Your best option moving forward is to replace the library with a functional equivalent. That task is for another day, however.

If you have now emptied your directories of external libraries then remove their reference from your <path/> elements (or <javac/>, etc).

A word of warning. It is very tempting to replace the external library with a newer dependency. Don't do it. Your task right now is only to replace external libraries with equivalent dependencies. Simply make note that a newer version exists.

Completing tasks stepwise

After waiting on an improbable response from the HealthSource RI health benefits web app I lamented on Twitter
Every time I have to wait for the interactive response of some application I wish the Envoy Framework was common. December 20, 2019
So what was so great about the Envoy Framework, a framework that was designed 28 years ago for a pre-Internet environment? You should read the whole article, but in short there were two user interaction aspects that I wanted

1. A user is able to disconnect from their session and return at a another time and continue where they left off.

2. A user is notified when requested work has been completed or can continue using their immediately appropriate notification mechanism.

In the HealthSource RI situation I was signing up for 2020 benefits. There is a short window of time to make these benefit decisions and there are many people making them. These circumstances put a strain on an implementation not designed to scale. Even an implemented designed for the cloud will be handicapped when there are choke points. That is, one or more resources that every request needs to interactive with. There are a number of technical solutions to these, but there are also user interaction solutions also. The easiest solution is to ask the user to come back later to complete the work.

On the surface interrupting the user seems the wrong action to take. The person is here and now and so the system should use this opportunity to complete the task. The rub comes when the system slows down and the task is taking longer than the person has time to give here and now. It is better to let the user leave and notify them when the system is ready to continue. When the user returns the system reestablishes the user's context, ie it brings them back up to speed, so as to not unduly prolong the time to completion.

If I were designing HealthSource RI I would make interactions interruptible. Long tasks are broken down into short tasks, each of which can be completed with limited (near zero) interaction with the system. Each time the user returns to work further on the long task they are presented with a plain summary of what has been accomplished and what is left to accomplish. This summary can be written or visual. When a long task is interrupted the user will be contacted, repeatedly if necessary, to return and complete the task. For time critical tasks the contact mechanisms may need to escalate; a person might need to actually phone the user and compete the task over the phone.

People are used to completing tasks stepwise in the physical world. Why not make enabling that a norm in the virtual world too?

Good advice on how to resolve the four failings of operations


The first 17 mins are the setup for subsequent good advice on how to resolve the the four failings of ops, ie trust, toil, silos, and queues. In fact, I think these four failings are true for most project management.

Direct observation and indirect reportage

In an episode of House one of his  diagnosticians said, something to the effect of,
"If only we had looked at the blood under a microscope we would have immediately seen the problem."
The team had relied on tests that produced results that were reported on paper. They had moved away from direct observation to indirect reportage. They never looked at the thing in front of them. I used to work with a brilliant software engineer who, at the end of his day, would check the online weather sites for what it was like outside. He sat next to a window. I could not sway him to trust his direct observation.

Over the years I see more and more examples of reliance on indirect reportage or blind trust in numbers. People who do not stop to judge the accuracy of a calculation, but instead simply copy and paste it, and move on. They don't even check that the value's magnitude seems right. I don't understand how one can accept a value without needing to know if it fits with your expectations. Is the gist of it right?

I was listening to an interview with Maura O'Connor on Science for the People podcast  on the occasion of her new book Wayfinding: The Science and Mystery of How Humans Navigate the World. In the interview she talks about how direct observation and experience strengthens memory and the role the hippocampus plays in this. For example, relying on turn-by-turn directions from your navigation device will not only remove you from the engagement with & enjoyment of your journey, but will prevent you from actually remembering it too. I think this book has moved to the top of my reading list.

Lastly, the MIT Technology Review has a good article about spoofing. Ships in the port of Shanghai seem to move around without regard to physics let alone basic seamanship. Well, on the automatic identification system (AIS) displays that is. Like Gregory House's diagnostician team found so did the captain in the story
"Eventually, mystified, the captain picked up his binoculars and scanned the dockside. The other ship had been stationary at the dock the entire time."

The end is nigh for NetBeans and Java GUIs

I recently updated my Mac to Mojave -- yea, just days before Catalina was released! I did this for a few reasons, but mostly I was unsure if JDK 11 would work on the macOS I was running and I needed JDK 11 for some consulting work. Since updating to Mojave I have found both NetBeans 8.2 (using JDK 8) and Apache NetBeans 11.1 (using OpenJDK 11) to be unstable. They don't freeze, but parts of the interface just stop working after a few hours of use. Mouse clicking fails in some places. Some keyboard commands fail. Typed input fails. The only solution is to restart NetBeans. This is not a good nor long term development environment tactic.

I have been worried for some time that Java on the Mac is at an end. Apple abandoned Java back in 2017. I didn't expect Oracle or anyone else to really put much effort into keeping the Java GUI toolkits current with macOS changes. I don't even expect Java GUIs to remain for much longer, either. JavaFX was dropped in JDK 11. I am sure Swing would have gone too if it were not for some very important customers that are still dependent on it, for example, the US military and especially the Navy.

I have similar observations about NetBeans. NetBeans always seemed like Oracle's unwanted stepchild from the Sun acquisition. Updates were infrequent and with little innovation. Oracle has now passed it on to Apache, but how much attention is it going to get there? There are commercial applications that are built on top of the NetBeans platform and so some support will come from them, but it would be foolish for these commercial applications to remain desktop bound for too much longer. Even Eclipse, the quintessential Java tool for building Java tools, has moved its focus from being a desktop application to being a web application.

It is finally time to move off of NetBeans and to move off of any Java GUI application. I say this with a heavy heart as I have enjoyed and benefited from both over very many years. RIP.

Aside: If you are interested in large scale, pluggable, user interface design then do read everything you can about the NetBeans platform. It is remarkably well designed.

healthyrhode.ri.gov and the worst online and telephone process

That Rhode Islanders have Health Source RI is great. Unfortunately, healthsourceri.com leads you to healthyrhode.ri.gov and this is by far the worst online and telephone process I have ever had to go through to get any kind of service. That the end result is a family's healthcare coverage means the stakes are really high for every customer interacting with the service. It better be rock solid and crystal clear to both the customer and the support representatives. As long as governments continue to select service designers the same way they select pickup trucks or any other interchangeable commodity we will continue to get crap.

Update: Coincidentally there was a useful Twitter thread about this same topic.

Andrew Gilmartin Consulting

Do you have a collection of data in one or more sources that need to be brought together, harmonized, and made accessible with an online search, or reports delivered daily via email or a wiki?

Do you have a back-office workflow that has become complicated due to multiple pathways to completion and referenced data scattered across multiple sources?

Do you want to bring transparency to your handling of customer requests, but need to integrate this with a legacy system?

Do your online business processes generate voluminous detailed logs within which you are sure you can get useful historical perspectives, current awareness, and even some predictions?

Andrew Gilmartin Consulting is a solo software engineering consultancy. There is much one person can do these days. The availability of inexpensive online infrastructure and best-in-class open-source tooling has greatly reduced the cost and effort required to provide a solution. I have decades of experience designing and building software and user interfaces. I have built everything from small gadgets to web-scale applications. It is your need to have it solved that interests me, and less so the kind of problem and my compensation.

So, please do call me or send me an email if you would like to chat about the problem and how I can help.

Andrew Gilmartin
andrew@andrewgilmartin.com
401-441-2062

Adding a Table of TODOs to a Google Document

I have been working on a document that details the changes needed to migrate a system from Java 8 to OpenJDK 11. While writing documents I add notes to myself in-situ, eg "TODO: Buy bread and milk before the next nor'easter." As the document was getting longer and the TODOs were mounting an itch was developing. I wanted a "Table of TODOs."

I use Google Docs and so the itch was scratched using Google's App Script. The script collects the text of all the TODO paragraphs and organizes them under a Table of TODOs heading. The TODO is a normal-styled paragraph that starts with "TODO: ". The Table of TODOs is a heading-styled paragraph that contains only "Table of TODOs". The script organizes the TODO texts in to numbered list items under the heading. The script also adds a Bookmark to each TODO paragraph so that it can be referenced from the numbered list item. To see the example you will need a Google account and to make a copy before you can view the script (under Tools / Script editor). [Update: Placed the script in a gist.]

The script was far more difficult to write than I expected. Google's scripting documentation is sparse and the IDE would randomly stop working. There is noApp Script language documentation (a mixed set of JavaScript versions) and the documentation for the Document API is a bare description of each class. Searching for help was hindered by the generality of any kind of questions I had, eg "google app script api bookmark" was more likely to get results about using URI bookmarks in Chrome. I am sure there are good resources and communities out there, but I have not found them yet. I will keep looking as the Google Suite can be combined into sophisticated applications and I would like to know how to do that.

Gists, Terraform, horizon charts, and PDF thumbnails, but not all at the same time

I was working my way through the Terraform Two-Tier example and after finding omissions I wrote down the missing parts and posted them to a GitHub gist. I used a gist as, I think, search engines will likely rank this higher than a posting here in my blog and so is better for folks looking for help.

I don't use GitHub gists much, but, apparently, I have been using them them for a long time -- 11 years! I had forgotten about some of the work I had done and dropped into gists. The horizon chart gist was part of my Horizon Chart blog post. I don't seemed to have posted anything about PDF Thumbnailer, however.

I originally created PDF Thumbnailer as part of a larger project to enable public commentary on South Kingstown, RI's budget. (SK is my home town.) When this project started in 2007 there were no available online tools for multiple users commenting on a PDF. So I needed to build one. The web interface consisted of just two principle pages, an overview page containing thumbnails of the whole budget with comment indicators laid over the thumbnails, and a details page of the whole budget and commentary interleaved.

To interleave the comments I needed to place them between the text, charts, and tables of the budget document. Parsing and rebuilding the PDF or converting it to HTML and rending it in the browsers of the time was more work than I could commit too. Instead, I compromised on the spacial proximity of a comment and its context. I converted the PDF pages to images and sliced each image into 1" tall strips. I could then interleave strips and comments on the details page as needed. While a comment was not attached to the context it was close enough to establish the connection.

In these sketches the pages are bounded in red, the strips in blue, and the comments in purple.



The project was never completed, but it did leave behind the thumbnail creator. I rebuilt it this morning and here are a few example pages from Rescue.org's recent report Missing Persons: Refugees Left Out and Left Behind in the SDGs

"The Rise and Fall of CORBA"

I am currently working on a large command and control system written many years ago when CORBA was still a hot technology. At the time it really was the only suitable realtime middleware with C++ and Java interoperability available and amenable to the US Navy. CORBA has many failings, but it can be successfully used if you constrained your use of it. (Of course, this is a truism for all technologies and appetites.) Nevertheless, CORBA is now an antique and should not be used for new work. In 2006 Michi Henning wrote "The Rise and Fall of CORBA" [1,2] which is still an excellent enumeration of the breakdown of a technology and a cautionary tale about committees and consortia.

[1] https://queue.acm.org/detail.cfm?id=1142044
[2] https://web.archive.org/web/20081229231146/https://queue.acm.org/detail.cfm?id=1142044
Born at the ebb tide in a shallow rock pool. Hours I lay in the warming water. Alas, the tide turned.

Little need for fully sentient beings

In the Winter of 2001 the US Navy began the re-development of the DD-21 destroyer class warship after Congress slashed the original budget. The new ship class DD(X) and named after Admiral Zumwalt was to be the Navy’s first fully integrated command and control (C2) vessel featuring a small crew, multipurpose vertical launch tubes, large flight deck, and with enough electric generation capacity to power small cities or the anticipated kinetic and laser weapons.

What was not divulged at the time was that the Zumwalt’s C2 left little need for fully sentient beings to operate the ship. It has been learned that a black budget division within the Navy’s Marine Mammal Program has been replacing the crew of the DDG-1000, currently at sea in the San Diego, CA area, with mollusks and cephalopods. Mollusks, due to their limited intelligence, (newly hardened) protective shells, and stationary nature, have been integrated into the ship-wide computing networks and are now responsible for intermediating with all sensors and actuators — from missile launch tubes to freshwater valves.

How far along the chain of command has been replaced with coleoidea is unclear. There is strong circumstantial evidence that the ship's captain is a human-octopus hybrid. The recent waterproofing polyethylene application to the inside walls of the ship’s bridge and mission control centers suggest that the hybridization is more octopus than human.

More news to come.

Furthermore on dashboards for maintenance developers

What would a dashboard for visualizing and managing the flow of messages between actors and their dependents look like? My skills run in the direction of information architecture and not visual design, unfortunately. I can tell you what should be on the dashboard, the relationships between the contents  controls, the visual and interactive priorities of these, and how all of this changes over time. On a Friday night, this is not something I am prepared to do; which is really an excuse as I am nearing the end of my interest in this thought experiment. Nevertheless, here are a few ideas.

The transitions of a message between the actor and the dependent are

- Actor sends the message to Dependent, or fails to send it.
- Dependent receives the message, or fails to receive it.
- Dependent responds with a response-message to the received-message, or fails to respond.
- Actor receives the response-message, or fails to receive it.

We can symbolize these statuses as


A display for showing details of actors, messages, and listeners (dependents) might be


Each message is given a row and the delivery and response status is detailed for each listener. My bare tabular example would have much more detail on it in practice. This is not a overview display and so concerns with overwhelming the user with too much detail is not of relevance. If you only have to show a few listeners and perhaps a brief time window of messages this display will quickly show specific problem with delivery and response. 

Inverting and aggregating the above example leads to the tighter display


We are, again, showing each message, but the columns now indicate how many listeners are in each of the delivery or response states.

Lastly, as everyone loves a set of small multiples graphs here is 40 minutes of activity and status where the green line is the number of listeners, the black lines show positive trends and the red negative ones. 


Man, that is an ugly chart (and is playing loose with small multiples).

More on dashboards for maintenance developers

Yesterday I wrote about providing tools to visualize & manage the internal workings of applications for maintenance developers. That note was long on advocacy and sort on detail. This note somewhere between those two.

We are all familiar with run-time debuggers. We can get our job done with gdb [1], but for day in and day out work a visual debugger is a must have. A debugger is a point-in-time dashboard of the execution of the programming language used to implement the application. It is not a dashboard of the application, it does not directly represent the application's data or processing models. Nor does it provide a history of where the application was or a prediction of where it will be heading too next. Nevertheless, it is a useful analogous image of the application dashboard to keep in your mind.

If, for example, your application relies on the observer pattern [2] then you will need to visualize & manage the actors (observed), their dependents (observers), and the messages. The effort to implement this dashboard can be significant, which is the exact opposite of implementing the pattern itself. The pattern is often implemented as a list of callbacks. This implementation is so insignificant that your high performing developers probably choose not to use a framework at all, but to re-implement for each use. And there is the root of the trouble ahead for the maintenance developers, there is no one place to watch this running application. Callbacks and messages are being created and dispatched all over the place.

Suppose there had been a reusable implementation. The management of the listeners was done in a single place. The dispatching of messages to the listeners was done in a single place. Each reuse required that the developer explain the purpose, configuration, and expectations of this reuse. For example, "Status of the nightly recalculations of base prices: the messages are life cycle events; messages are ticketed and numbered for correlation; the listeners can be local or remote; the listeners are contacted asynchronously; no receipt acknowledgement is wanted, but when given will be logged; expect a dozen messages over a 2 hour period in every 24 hour period to be sent to a few hundred listeners." When this pattern is visualized on the dashboard not only is the context clear, but timelines of planned to actual and switches to toggle synchronicity will automatically be included. And what if you could dynamically toggle receipt acknowledgement? How many of your application's ailments are the result of lost messages sent to remote listeners?

The rise of DevOps has brought with it both tools and new (renewed?) appreciation for visualizing processes and their artifacts. Lets keep on pushing these ever deeper into the application itself.

[1] gdb, https://www.gnu.org/software/gdb/
[2] Observer pattern, https://en.wikipedia.org/wiki/Observer_pattern

We all want visibility into a running application

We all want visibility into a running application. The need becomes acute when the application is ailing and that visible schedule, map, and cartogram of the data and processing will swiftly guide you to the failing part.

What is less recognized is that these same visuals are needed in maintenance. Your original, high performing developers are unlikely to be around after the application has been in production for a few years. They will have moved on to new green pastures. If you are lucky and they are still with the company then they will be somewhat available to the maintenance developers. If a little cantankerous about that.

The maintenance developer usually has little broad knowledge of the application’s operation. Her first task at resolving an application’s ailment will be to find suitable entrance and exit points. This is not an easy task for our applications now run continuously, across multiple processes and hosts, and are highly asynchronous. Hopefully, those high performing developers adhered to standards, used a common framework throughout, and kept to best-of-class external libraries and supporting tools. If they did then she might have a straightforward resolution path. In all likelihood, however, those high performing developers probably invented too much and documented too little. [I speak from the experience of my younger self.]

What the maintain crew needs is not logs alone. They need meta tools that connect to the running application and visually show its operation. Once connected the tool must be able to slow down the application to a human pace. Restrict it to one active user at a time. It might even need to switch it to run synchronously.

These are not easy features to implement, but the cost of not doing it is not just the ever lengthening backlog of ailments to fix, but the erosion of customer confidence and, in the end, loss of customers.

All modern applications have three users. The first is the customer. The second is the customer support. The third is the maintenance crew. It is way past time that the maintenance developer is provided the meta tools needed to look inside the running application and keep it fit for market.

Warhammer 40k: I don't really like the game

I have enjoyed my time with Warhammer 40k. The Games Workshop plastic models are very well produced and a joy to assemble and paint. The background lore is deep and some of its retelling is masterfully done in the Black Library publications. I particularly like the Horus Heresy stories even if they are set 10,000 years before the world I am playing in. Unfortunately, I don't really like the game for the simple reason that there are far too many stats. There are stats from the individual weapons, to figures, to units, and so on all the way up to the whole army. This makes for a great variety of play as every game truly is a unique combinatorial experience. But, I can't play a game like that. I don't have the inclination or, frankly, the time to dedicate to learning not only my faction's specifics, but also those of my opponent's army. An opponent whose army can be made up from many dozens of races.

I have decided to keep the figures. I had intended on selling them, but, the figures are very nice and perhaps I will play the One Page Grimdark Future Rules one day.

A top software development position in a startup wants ...

Having just taken a job with Raytheon and lamenting my lack of success getting a top software development position in a startup a friend asked "What is needed for new startups?"

A top software development position in a startup wants several years of experience with AWS infrastructure and a comprehensive knowledge of JavaScript use (and packaging) in the backend (node.js) and the frontend (Angular, React, and sometimes Vue). The startup still want the other skills of scalable architecture design (processing and data), excellent written and spoken communication, and mentoring, but having these is not enough. I was also surprised that even startups that will hire remote staff still want that staff close to home base.

As to the Java language and its ecology, this has little interest for them. In fact, many of the infrastructure foundations that are today implemented in Java -- ActiveMQ, Kafka, Zookeeper, Tomcat, etc -- are being replaced with "lighter" Go and Rust implementations. And with the accelerating move to managed infrastructure startups care little about how these foundations are implemented. They would rather rent, eg, a queuing service from AWS or GCP, and let the provider worry about implementing turnkey, scalable performance.

Even Citizens Bank, after having recently become independent of the Royal Bank of Scotland, is replacing their Java J2EE implementation with an AWS hosted, node.js microserviced, and React front ends. I have even heard that some of their services are from a "bank in a box" supplier.

I don't disagree with these changes except for the unfounded confidence in JavaScript -- the one language to rule them all. I would not want to run my own data center and infrastructure anymore, either. Assuming, that is, I had the very sizable budget for it.

Unwelcome advertisement for throat lozenges

The content on YouTube is really tremendous. I have used it to watch tutorials for fixing a food mixer, replacing my minivan's factory radio, painting miniatures vikings and ultramarines, learning boardgames, and refreshing some math theory. I have used it to watch documentaries and fan-made movies. I have listened to audio books. Sometimes, I even listen to music and other forms of audio & video performances. Unfortunately, the interstitial advertisements are beginning to kill YouTube.

I accept the need for YouTube to generate revenue with advertising, and, as crazy as it sounds, if the ad is short, I let it play through rather than take action to skip it. What I can't accept is the jarring interruption of the interstitial ads. You are listening to a 7 minute recording of a live performance. The performance beautifully builds for the first 3 or 4 minutes and then, just when it reaches a crescendo, an ad for throat lozenges breaks into the middle and ruins the whole thing.

I will continue to use YouTube as it has so much content useful in my daily life. I am going to skip most everything else until they replace the interstitial ads.

Phoenix Checklist

[Copied from Boing Boing.]

The "Phoenix Checklist" is a set of questions developed by the CIA to define and think about a problem, and how to develop a solution.

The Problem
  • Why is it necessary to solve the problem?
  • What benefits will you receive by solving the problem?
  • What is the unknown?
  • What is it you don’t yet understand?
  • What is the information you have?
  • What isn’t the problem?
  • Is the information sufficient? Or is it insufficient? Or redundant? Or contradictory?
  • Should you draw a diagram of the problem? A figure?
  • Where are the boundaries of the problem?
  • Can you separate the various parts of the problem? Can you write them down? What are the relationships of the parts of the problem? What are the constants of the problem?
  • Have you seen this problem before?
  • Have you seen this problem in a slightly different form? Do you know a related problem?
  • Try to think of a familiar problem having the same or a similar unknown
  • Suppose you find a problem related to yours that has already been solved. Can you use it? Can you use its method?
  • Can you restate your problem? How many different ways can you restate it? More general? More specific? Can the rules be changed?
  • What are the best, worst and most probable cases you can imagine?
The Plan
  • Can you solve the whole problem? Part of the problem?
  • What would you like the resolution to be? Can you picture it?
  • How much of the unknown can you determine?
  • Can you derive something useful from the information you have?
  • Have you used all the information?
  • Have you taken into account all essential notions in the problem?
  • Can you separate the steps in the problem-solving process? Can you determine the correctness of each step?
  • What creative thinking techniques can you use to generate ideas? How many different techniques?
  • Can you see the result? How many different kinds of results can you see?
  • How many different ways have you tried to solve the problem?
  • What have others done?
  • Can you intuit the solution? Can you check the result?
  • What should be done? How should it be done?
  • Where should it be done?
  • When should it be done?
  • Who should do it?
  • What do you need to do at this time?
  • Who will be responsible for what?
  • Can you use this problem to solve some other problem?
  • What is the unique set of qualities that makes this problem what it is and none other?
  • What milestones can best mark your progress?
  • How will you know when you are successful?
From the book, Simply Brilliant: Powerful Techniques to Unlock Your Creativity and Spark New Ideas, by Bernhard Schroeder

The splendor and complexity of networked service at scale

The splendor and complexity of networked service at scale. Josh Evans' talk Mastering Chaos - A Netflix Guide to Microservices at InfoQ.

"Contempt isn't cool"

I really like Benno Rice's talk "The Tragedy of systemd" about accepting change. I had resisted systemd only because I didn't need anything different than inet.d for what I was doing. This was a mistake and one I wish I had rectified sooner. I was missing out on taking advantage of all the other great services systemd offered. Had I switched earlier I would have enjoyed the slow, methodical accrual of a deeper knowledge and practical experience.

The link above is to a point near the end of the talk to the "Contempt isn't cool" slide. Accepting change requires to accept ones own limitations, biases, and blindspots. Do this respectfully as others have done it with you. Contempt is a destructiveness that you never want to bring into your team.

Waste Wars

I was reading the New York Times last year and came across an article about the hazardous working conditions of private waste management companies. The work is hard, the pay too little, and the time pressures too great. I don't mean to make little of this horrible situation, but, later, my mind wandered, as it often does, to Terry Gilliam’s movie Brazil and the gorilla plumber. I have always been intrigued by the plumber’s statement that plumbing was the last radical trade in the near distant future. What if garbage collection was too? Could its work be made into a game?
"The year 2017 saw the rise of the litteral cut-throat waste management companies. Their goal was to grow their business by any means possible. By 2027 waste management had become paramilitarised." Waste Wars
I never completed the game development. What I have developed is more, I think, a source of ideas for an actual game developer or for repurposing an existing game.

Gluing Warhammer 40k Space Marines

Started to glue together my Warhammer 40,000 Space Marine models. I am not fond of the aesthetic of their uniforms, but the modelmakers have made good use of it whereby each successive layer can be used used to hide the mistakes made underneath.

I don't know if other hobbyists do this, but after assembly I give all the surfaces where I scraped off the mold lines a light wash of plastic glue, aka acetone. For these models this seems to further help hide the mold lines. Will know more once they are primed.

Update: Is it just mine or are all 40k space marines right handed and turning towards the left?

Update: I should not have put them on bases and attached the guns before painting them!

Update: Awaiting the enemy ...

Fiskars Easy Change Fabric Knife

The Cool Tools blog posted a review of several alternatives to the ubiquitous, badly design, barrel handle X-Acto knives. The Fiskars Easy Change Knife is the favorite and so I bought two via the posting's associates link. They both have NO grip on the blade and are dangerous. Do not buy that version. There seem to be 2 versions of the knife in the world.

I contacted Fiskars and they sent me two replacements that are awesome! The replacements are Fiskars Easy Change Fabric Knife (3 blades) 164010-1001. The only visible difference between the two is that the Fabric Knife's has a gray tinted, translucent cap (rather than untinted). I hesitate to provide a product link as none that I found show the packaging. If you do find a package image it should look like the image in this posting.

Thank you Fiskars for great customer service.