I don't know if the game Star Wars: Outer Rim is enjoyable, but the game's physical design is amazing. Which reminds me I have What the Tech World Can Learn from Video Game UX lined up to watch.
Inter-company event sourcing
I was in a client's meeting today and they were discussing how one of their batch processes has a dependency on data generated by a supplier's batch process. The problem is that the client does not know when the supplier's process is complete so they can start their process. (The supplier's data is accessed via an API rather than a bulk file delivery.) The discussion went along the lines of
We know their process starts around 10 PM and takes about an hour to complete. So let's give it a couple of hours to finish, just in case it's slow that day, and we will start our process at, say, midnight.
I scream into the void. This kind inter-company data processing coordination is quite common. Common enough that I am still surprised there is no standard, or industry-specific, solution we use.
South Kingstown & Narragansett restaurants
If you are looking for good restaurants in South Kingstown or Narragansett try Purslane, Duck Press, Tsunama, and Agave Social Cocina Mexicana.
Recent history of education funding in Rhode Island
“For decades, Rhode Island’s policymakers have operated under the myth that cutting taxes for businesses and high earners would spur economic growth. Yet the data is clear..."
SEEDS UNPLANTED - The Recent History of Education Funding in Rhode Island
A pair of formal diagram explanations. One for software and one for buildings.
The C4 set of diagrams are directly relevant to my work as a software engineer. Seeing the architecture profession's sets of diagrams is a useful reminder -- and an immediately obvious one for anyone who has lived in a house! -- that different diagrams are needed for different purposes (ie, contexts).
The C4 Model – Misconceptions, Misuses & Mistakes • Simon Brown • GOTO 2024
What's in my set of architectural documents? Sharing everything: drawings, schedules, + specs.
Why I don't want color coded logs
Most developers laugh at me when I say I don't want color coded logs. They rarely ask why. Logs with color coded structures provide no actionable information. They actually obscure actionable information by forcing a distinction without a difference. What I am looking for are clues in the logs. Those clues are often easily overlooked small tokens. I use color or inversion to distinguish them so that their occurrence immediately stands out from the background noise. The highlight script is a simple tool for this.
Unnecessary frustration and toil
I spent a good part of yesterday tracking down a problem with the staging deployment of a feature I first started back in October of last year. (That it has taken this long to get it to staging has everything to do with how this organization manages work.) When you have such a extended period between implementation and deployment you rarely retain the feature's context and even rarer an environment within which to debug problems. It took me some time to regain that context and environment. (I should have left better notes for my future self.) Once I had that it became obvious that the feature worked and that the problem lay in the deployment.
The deployment is a small Kubernetes cluster. Each service has 2 or 3 container in several pods. I figured out the pod ids and container names (why are they all different!) and opened a terminal window for each and streamed the logs. I used a small script to highlight text that I was expecting to find. I then used the feature and discovered the problem was due to a corrupted private key stored in the deployment's database.
The organization uses Honeybadger.io to record exceptions and Elasticsearch to aggregate logs. These tools are intended to improve access to the details needed to debug issues. Each tool has its own user interface and mechanisms for accessing and searching. To use them you obviously need to understand these mechanisms and, more significantly, you need to know how the organization has configured them. That is, no two organizations use the same data model for how it records exceptions and log details.
The developer needs documentation about the configuration and there was none. Well, that is not quite true. This organization has thousands of incomplete, unmaintained, and contradictory Confluence pages. The "information" available to the developer is actually worse than none at all as they will waste time trying to piece together some semblance of a coherent (partial) picture. What I eventually concluded was that it could not be done and my best path forward was to look at the raw container logs.
I understand that at this organization I am a contractor and so just developer meat. But what I have seen is that this global, financial, highly profitable organization does not do any better for their developer employees. Perhaps all industries are like this. I have only experienced the software development industry and here they are mostly the same. It makes me sad and mad to see and experience such unnecessary frustration and toil.
Transactions and some concurrency problems
A group of us are reading Kleppmann's Designing Data-Intensive Applications. Chapter 7 is on transactions and especially the different approaches used to address concurrency problems, ie the Isolation in ACID. What becomes clear is that transaction isolation levels can only mitigate some problems. It is your application's data model design and use that are mostly responsible for avoiding them. Here are the concurrency problems raised in this chapter:
Lost Updates. These occur when a process reads some records, modifies them, and writes them back. If the updated records had been modified by another process after the read then those updates would be lost.
Read Skew. This is a variation of Lost Updates due to the delays between steps in a multiple step operation. Processes A and B are interacting with the same records. Process A reads records X and Y (two steps). Process B updates records X and Y (two steps). Due to the delay between A's and B's steps, process A has the original X value but the updated Y value.
Write Skew. This occurs when process A reads some records to make a decision and then updates other records appropriate to the decision. While the decision is being made process B changes some records that would alter process A's decision. Process A is unaware of process B's changes and continues to make its updates which invalidates the data model.
Phantoms. This is a variation of Write Skew. Process A queries for records to make a decision. Process B inserts records that would have been included in process A's query results. Unaware of these inserts, process A makes its updates which invalidates the data model. The "phantoms" are the records not included in process A's query result.
Costs of helpful data flexibility.
I'm having a discussion currently with a young developer who has only ever worked in Ruby and JavaScript. I noticed that the developer had chained "symbolize_keys" to the end of a method call. In Ruby this converts a hash's keys from strings to symbols, ie { "a" => 123 } becomes { :a => 123 }. Their reason for this was to offer flexibility to the called as to how it returned the result. They thought this provided flexility and robustness. I countered that it did the exact opposite.
When a function can be given parameters and return results in multiple formats then robustness is only had when the function handles all formats equally. To do that the function needs to be tested with all formats. This can be done, but in practice, and I've seen across many organizations, it is not. Not only the function needs to be tested with the multiple formats, but the callers and the called need to be tested too. It's a combinatorial explosion of testing.
The other detriment to this flexibility is that since no function is sure of the format every function converts the data to its preferred format even if the data is already in the preferred format. This conversion adds to the function's code size and has a runtime cost (CPU and memory) on every invocation. The cost of a single use might be small, but our applications work in a world with thousands of concurrent sessions each with deep call chains, and expect microsecond responses. Those single uses add up.
My recommendation to the developer was to require one format as part of its contract and add validation that runs at least during testing. (I'd like to just tell them to use a typed language where this wouldn't even be an issue!)
I mentioned that the developer's experience is in Ruby and JavaScript. I have found that is common for such developers to not expect data to be in a specific format or type. I assume some of this comes from never being trained to always validate and convert data coming from the outside before using it inside. (Eg, directly passing around an INPUT element's value or a database's column value.) Once inside, you can be assured of its correctness. Instead, data is passed around without any function knowing a priori that it is correct.
I am unsure if I will convince this developer to not use "symbolize_keys". I am rowing against the tide.
Update: Not only did I not convince the developer, but the system's architect rejected it also.
Subscribe to:
Posts (Atom)