Creating a Maven project for the common packages

This posting continues the series on moving from an Ant to a Maven build.

When you have replaced all your external libraries with Maven dependencies you can move on to the next stage in moving from Ant to Maven. This stage is to separate out one or more of your Java packages into their own Maven projects. There are a number of good reasons to separate out packages, but the practical one is that Maven is best used to build one deliverable.

Every organization has a "common" package that contains classes adding general purpose functionality. There are likely several "utility" packages that augment the standard Java libraries for IO, networking, text, collections, functions, streams, etc. There are packages for data structures and algorithms that were sufficiently independent of the primary application to be have been separated out from its packages. These common packages are also likely used by supplementary tools to your primary application. If you already build separate jars for your common packages this step is not that much work. If you don't then you are likely to find some unexpected and unwanted entanglements with application code that needs to be worked out.

A note about the version control. Separating out packages does not mean you also have to move from a monorepo if that is what you are currently using. Moving away from a monorepo does simplify assuring that your common packages are isolated from your application code, however. If you have been building separate jars for your common packages then you have already established build mechanisms to maintain the isolation. If you haven't then this is something you will need to do in your monorepo. This tutorial, however, assumes you are not using a monorepo.

The common packages will be your first Maven project. Create an empty Maven project using archetype:generate

mvn archetype:generate \
  -DgroupId=com.andrewgilmartin \
  -DartifactId=system-common \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DarchetypeVersion=1.4 \
  -DinteractiveMode=false

Change the "groupId" and "artifactId" appropriately. The command will create the tree

.
└── system-common
    ├── pom.xml
    └── src
        ├── main
        │   └── java
        │       └── com
        │           └── andrewgilmartin
        │               └── App.java
        └── test
            └── java
                └── com
                    └── andrewgilmartin
                        └── AppTest.java

Move your common packages into the src/main/java tree. Move any tests you have for these packages into the src/test/java tree. (Delete AppTest.java from both trees at some point.) Your IDE will likely be very helpful in moving files and even version histories.

A build now will very likely fail due to Java language level and missing dependencies. Edit "pom.xml" and update the "maven.compiler.source" and "maven.compiler.target" properties appropriately, eg for Java 8 use

<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.9</maven.compiler.target>

Try a compile just to see what happens!

mvn package

Lots of missing dependencies! Using the information you gathered before add each dependency under the pom.xml file's <dependencies/> element. For example, if the common package is dependent on Apache's HTTP Client 4 then add

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.10</version>
</dependency>

The org.apache.httpcomponents are located at Central and this is most likely already listed in your Maven installation's $HOME/.m2/settings.xml file. If it is not or you would rather not depend on the installation's settings (a good practice) then add the <repository/> element under the <repositories/> element. If the <repositories/> is missing from pom.xml then add it above the <project/> end tag.

<project>
  ...
  <repositories>
    <repository>
      <id>central</id>
      <name>Central Repository</name>
      <url>https://repo1.maven.org/maven2/</url>
    </repository>
  </repositories>
</project>

Continue adding dependencies and building and testing until you get a clean result.

You now have your common packages in their own jar

target/system-common-1.0-SNAPSHOT.jar

After installing this in you local Maven cache, $HOME/.m2/repository/, you can use it in your Ant build.xml. To install it use

mvn install

And you should see the logged output similar to

[INFO] Installing /home/ajg/src/system-common/target/system-common-1.0-SNAPSHOT.jar to /home/ajg/.m2/repository/com/andrewgilmartin/system-common/1.0-SNAPSHOT/system-common-1.0-SNAPSHOT.jar
[INFO] Installing /home/ajg/src/system-common/pom.xml to /home/ajg/.m2/repository/com/andrewgilmartin/system-common/1.0-SNAPSHOT/system-common-1.0-SNAPSHOT.pom

To use the installed jar in your Ant build you need only to add the new dependency, eg

<mvn-dependencies pathId="runtime-dependencies.classpath" ... >
  ...
  <dependency groupId="com.andrewgilmartin" artifactid="system-common" version="1.0-SNAPSHOT"/>
  ...
</mvn-dependencies>

Remove from the system's source tree all the common packages now compiled into system-common-1.0-SNAPSHOT.jar. Also remove all the <dependency/> elements that were only used by the common packages as Maven already knows the jar's dependencies (from its pom.xml file that was also installed into $HOME/.m2/repository).

If you have other common packages, eg enhancements to Swing or extensions to J2EE, then repeat the above procedure for each of them.

A series of posting about moving an ancient Ant build to Maven

This is a table of contents page for a series of posting about moving an ancient Ant build to Maven.

  1. Moving to Maven from an ancient Ant build
  2. Running a Maven repository manager
  3. Running a Maven repository manager using Apache WebDav
  4. Creating a Maven project for the common packages
  5. Creating Maven projects for the core packages and command line tools
  6. Creating a Maven project for your web application

The code for examples is at

https://github.com/andrewgilmartin/maven_examples

Running a Maven repository manager using Apache WebDav

This posting continues the series on moving from an Ant to a Maven build.

Implementing a Maven repository manager using Apache's HTTPd with WebDav is straightforward. For this example, create the directory tree

.
├── apache2
│   ├── httpd.conf
│   └── logs
└── www
    └── repository

For the purposes of this example the directory tree is in /var/maven/

The httpd.conf file contains

LoadModule mpm_prefork_module libexec/apache2/mod_mpm_prefork.so
LoadModule unixd_module libexec/apache2/mod_unixd.so 
LoadModule authz_core_module libexec/apache2/mod_authz_core.so  
LoadModule access_compat_module libexec/apache2/mod_access_compat.so   
LoadModule mime_module libexec/apache2/mod_mime.so 

LoadModule dav_module libexec/apache2/mod_dav.so
LoadModule dav_fs_module libexec/apache2/mod_dav_fs.so

ServerRoot ${BASEDIR}/apache2
LogLevel warn
PidFile logs/httpd.pid

Listen 8080
ServerName localhost:8080
ServerAdmin andrew@andrewgilmartin.com

AddType application/octet-stream .sha1
AddType application/octet-stream .md5
AddType text/xml .xml
AddType text/xml .pom
AddType application/octet-stream .jar
AddType application/octet-stream .war

DocumentRoot ${BASEDIR}/www

DavLockDB DavLock
<Directory />
    DAV On
</Directory>

Start the HTTP server (in the foreground) using

BASEDIR=/var/maven/ httpd -f /var/maven/apache2/httpd.conf -X

For Maven deploy to this repository manager you need to make two changes to your pom.xml. The first is to include the extension in the <build/> section

<extensions>
  <extension>
    <groupId>org.apache.maven.wagon</groupId>
    <artifactId>wagon-webdav-jackrabbit</artifactId>
    <version>3.3.4</version>
  </extension>
</extensions>

The second is to use the "dav:" prefix to your <distributionManagement/> element URLs

<distributionManagement>
  <snapshotRepository>
    <id>neighborhood-snapshots</id>
    <name>Neighborhood Snapshots</name>
    <url>dav:http://localhost:8080/repository/</url>
  </snapshotRepository>
  <repository>
    <id>neighborhood-releases</id>
    <name>Neighborhood Releases</name>
    <url>dav:http://localhost:8080/repository/</url>
  </repository>
</distributionManagement>

To deploy your project use

mvn clean package deploy

This posting is part of the series about moving to Maven from an ancient Ant build.

Running a Maven repository manager

This posting continues the series on moving from an Ant to a Maven build.

The availability of most Java libraries in Maven's repositories is the most valuable of its features. It is difficult to say if Maven's build and deploy tools were the catalyst or that its appearance simply coincided with a time when it was obvious to all that centralized repositories with a common access protocol was needed. The why does not matter much now as we have Maven and it is well established in the Java ecosystem.

A Maven repository (aka a site) is a well defined file system accessible via one of several protocols. If you are using HTTP then the repository server needs to implement the GET and PUT methods for files. For example, if your dependency is (pom.xml)

<dependency>
  <groupId>com.andrewgilmartin</groupId>
  <artifactId>example1</artifactId>
  <version>1.0</version>
</dependency>

and the repository is

<repository>
  <id>neighborhood</id>
  <url>http://localhost:8080/repository/</url>
</repository>

then when you use Maven to build, eg

mvn -U clean package

Maven will initially GET the dependency's files

/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.sha1

and then based on the detail in maven-metadata.xml it get will GET the files

/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar.sha1
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom.sha1

When you deploy, eg

mvn deploy

using the distribution management (pom.xml)

<distributionManagement>
  <snapshotRepository>
    <id>neighborhood-snapshots</id>
    <name>Neighborhood Snapshots</name>
    <url>http://localhost:8080/repository/</url>
  </snapshotRepository>
  <repository>
    <id>neighborhood-release</id>
    <name>Neighborhood Releases</name>
    <url>http://localhost:8080/repository/</url>
  </repository>
</distributionManagement>

Maven will first GET the files

/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.sha1
/repository/com/andrewgilmartin/example1/maven-metadata.xml
/repository/com/andrewgilmartin/example1/maven-metadata.xml.sha1

and then PUT the files

/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.md5
/repository/com/andrewgilmartin/example1/1.0/maven-metadata.xml.sha1
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar.md5
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.jar.sha1
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom.md5
/repository/com/andrewgilmartin/example1/1.0/example1-1.0.pom.sha1
/repository/com/andrewgilmartin/example1/maven-metadata.xml
/repository/com/andrewgilmartin/example1/maven-metadata.xml.md5
/repository/com/andrewgilmartin/example1/maven-metadata.xml.sha1

The Maven client does all the work to ensure that the repository has all the necessary metadata about the deployed package. That is, the HTTP server needs no Maven intelligence. A dedicated repository manager, like Apache Archiva, Sonatype Nexus, and JFrog Artifactory, does far more than just manage files, but for small collections these features are not needed.

For the next stage in migrating your Ant project to Maven you need only configure an HTTP server for GET and PUT. If the HTTP server uses the file system directly, then it will need to be able to create the intermediate directories for any PUT files. During the Ant to Maven transition, however, you can use my simple HTTP server at https://github.com/andrewgilmartin/filehttpserver/. For production you will likely want to use, for example, Apache2 and WebDav.

Moving to Maven from an ancient Ant build

This posting initiates the series on moving from an Ant to a Maven build.

You use Ant to build your system of one or more deliverables. You have one build.xml file. You have toyed with separating out the deliverables into individual build.xml files, but there seemed not to be enough value in doing so. Over time your build.xml has actually had less unique things to do, eg RMI no longer needs a compile step, you moved database initialization and configuration into another tool, and the newer React web front end is built elsewhere. Effectively the build.xml just complies the source, runs the tests, and assembles one or more jars or wars.

The Ant build while now simpler has a number of downsides. You still manually manage the large set of external libraries. Perhaps those libraries have not been updated in years as it is too cumbersome to track down their origin and incorporate the most recent, viable version. Your IDE does not integrate well with your bespoke build environment. The IDE's tools for creating test classes, running tests, showing library JavaDoc, navigating into library sources, and code completion is ineffective or unusable.

It would be better all-round if you moved to Maven. This series of postings is an approach to accomplishing that.

If you have not yet installed Maven then do so from

https://maven.apache.org/install.html

The first stage to moving to a Maven build is to replace the external libraries with Maven dependencies. You don't need to replace Ant to do this thanks to the Maven Ant Tasks

https://maven.apache.org/ant-tasks/

Download the most recent jar and place it in your build libraries. Now add the following Ant task definition to your build.xml (near the top of the file)

<taskdef
  name="mvn-dependencies"
  classname="org.apache.maven.artifact.ant.DependenciesTask"
  classpath="${basedir}/buildlib/maven-ant-tasks-2.1.3.jar"
  onerror="report"/>

Update the "classpath" attribute as necessary. Do a clean build to ensure that the "mvn-dependencies" task is accessible.

The Ant build likely has different and/or overlapping external libraries for building, testing, and running. You will be replacing these with equivalent dependencies declared in <mvn-dependencies/> elements. Add the following elements to your build.xml (after the <taskdef/>)

<mvn-dependencies
  pathId="build-dependencies.classpath"
  sourcesFilesetId="build-dependencies-sources.classpath"
  javadocFilesetId="build-dependencies-javadoc.classpath"
  settingsfile="${basedir}/maven-settings.xml" >
  <!--
  <dependency groupId="" artifactid="" version=""/>
  -->
</mvn-dependencies>

<mvn-dependencies
  pathId="test-dependencies.classpath"
  sourcesFilesetId="test-dependencies-sources.classpath"
  javadocFilesetId="test-dependencies-javadoc.classpath"
  settingsfile="${basedir}/maven-settings.xml" >
  <!--
  <dependency groupId="" artifactid="" version=""/>
  -->
</mvn-dependencies>

<mvn-dependencies
  pathId="runtime-dependencies.classpath"
  sourcesFilesetId="runtime-dependencies-sources.classpath"
  javadocFilesetId="runtime-dependencies-javadoc.classpath"
  settingsfile="${basedir}/maven-settings.xml" >
  <!--
  <dependency groupId="" artifactid="" version=""/>
  -->
</mvn-dependencies>

The "pathId" is going to be used in conjunction with your existing <path/> elements for constructing the classpaths used in building, testing, and running. Eg,

<path id="build.classpath">
  <path refid="runtime-dependencies.classpath"/>
  <path refid="build-dependencies.classpath"/>
  <!-- ... -->
</path>

<path id="test.classpath">
  <path refid="runtime-dependencies.classpath"/>
  <path refid="build-dependencies.classpath"/>
  <path refid="test-dependencies.classpath"/>
  <!-- ... -->
</path>

<path id="runtime.classpath">
  <path refid="runtime-dependencies.classpath"/>
  <!-- ... -->
</path>

The <mvn-dependencies/> "settingsfile" attribute references a Maven configuration. This file is used to specify the Maven repository managers, ie where dependences can be found. You should place this file under version control alongside the build.xml. The base file is

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <profiles>
    <profile>
      <id>default</id>
      <repositories>
        <!--
        <repository>
          <id></id>
          <url></url>
        </repository>
        -->
      </repositories>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>default</activeProfile>
  </activeProfiles>
</settings>

You are now ready to use Maven dependencies. For each external library you will need to know its name, version, and, if possible, origin URL. To find these search the Maven repository index at

https://mvnrepository.com/

Eg, if you are using Apache Codec at runtime then search for "apache codec" and navigate to the dependency version page

https://mvnrepository.com/artifact/commons-codec/commons-codec/1.8

You will need to record the group id, artifact id, version, and repository URL. Get the group id, artifact id, and version from the Maven tab, ie

<dependency>
  <groupId>commons-codec</groupId>
  <artifactId>commons-codec</artifactId>
  <version>1.8</version>
</dependency>

Add this to the runtime <mvn-dependencies/> element in build.xml, eg

<dependency groupId="commons-codec" artifactid="commons-codec" version="1.8"/>

Get the repository URL from the Repositories field, ie following the "Central" link reveals that the URL is

https://repo1.maven.org/maven2/

If the repository URL is unique then update maven-settings.xml, eg under <repositories/> add

<repository>
  <id>Central</id>
  <url>https://repo1.maven.org/maven2/</url>
</repository>

Now delete the external library's jar. Do a clean build to test that you have correctly replaced the dependency.

Continue replacing external libraries with dependencies until you are done. It has been my experience that this will progress far quicker than you initially worried it would.

You may find a few external libraries that were themselves never built with Maven or were never deposited to a repository. For these you will need to continue to use the manually managed jar. Your best option moving forward is to replace the library with a functional equivalent. That task is for another day, however.

If you have now emptied your directories of external libraries then remove their reference from your <path/> elements (or <javac/>, etc).

A word of warning. It is very tempting to replace the external library with a newer dependency. Don't do it. Your task right now is only to replace external libraries with equivalent dependencies. Simply make note that a newer version exists.

Completing tasks stepwise

After waiting on an improbable response from the HealthSource RI health benefits web app I lamented on Twitter
Every time I have to wait for the interactive response of some application I wish the Envoy Framework was common. December 20, 2019
So what was so great about the Envoy Framework, a framework that was designed 28 years ago for a pre-Internet environment? You should read the whole article, but in short there were two user interaction aspects that I wanted

1. A user is able to disconnect from their session and return at a another time and continue where they left off.

2. A user is notified when requested work has been completed or can continue using their immediately appropriate notification mechanism.

In the HealthSource RI situation I was signing up for 2020 benefits. There is a short window of time to make these benefit decisions and there are many people making them. These circumstances put a strain on an implementation not designed to scale. Even an implemented designed for the cloud will be handicapped when there are choke points. That is, one or more resources that every request needs to interactive with. There are a number of technical solutions to these, but there are also user interaction solutions also. The easiest solution is to ask the user to come back later to complete the work.

On the surface interrupting the user seems the wrong action to take. The person is here and now and so the system should use this opportunity to complete the task. The rub comes when the system slows down and the task is taking longer than the person has time to give here and now. It is better to let the user leave and notify them when the system is ready to continue. When the user returns the system reestablishes the user's context, ie it brings them back up to speed, so as to not unduly prolong the time to completion.

If I were designing HealthSource RI I would make interactions interruptible. Long tasks are broken down into short tasks, each of which can be completed with limited (near zero) interaction with the system. Each time the user returns to work further on the long task they are presented with a plain summary of what has been accomplished and what is left to accomplish. This summary can be written or visual. When a long task is interrupted the user will be contacted, repeatedly if necessary, to return and complete the task. For time critical tasks the contact mechanisms may need to escalate; a person might need to actually phone the user and compete the task over the phone.

People are used to completing tasks stepwise in the physical world. Why not make enabling that a norm in the virtual world too?

Good advice on how to resolve the four failings of operations


The first 17 mins are the setup for subsequent good advice on how to resolve the the four failings of ops, ie trust, toil, silos, and queues. In fact, I think these four failings are true for most project management.