Establishing a new leader and informing the lead

I have a distributed system design problem. The problem brief is

We have many clients to a service. The service needs to be fault-tolerant and so it will have many replicas. When a client can no longer access the service it will switch to a replica and inform all the other clients will switch to the same replica. (It is not acceptable to load-balance across the replicas as the replicas' data values are not exactly the same but all the clients must return the same data values at all times.)

My current design is to have the client, on failing to reach the service, ask for a new service leader. When a new service leader is established it will then notify all the clients to use it.

As with many distributed coordination designs I need a distributed group manager. I am considering the use of JGroups and/or Apache Zookeeper in a solution.

Is there an existing recipe or recipes that I should be looking at to solve this problem?

Update: We finally decided to have a master make a new copy of the repository, copy it to all service hosts, and notify the service hosts to switch to the new repository. The master tracks who has switched and raises an alarm if there is missing switchers after a fixed interval.

"I like Obamacare."

About a week ago now I was tired of NPR using the term "Obamacare" instead of the laws real name, The Affordable Care Act. And so I sent email and Facebook messages to the local and national NPR offices:

Dear NPR, please call the law by its rightful name, The Affordable Care Act, and not the prejudicial name its opponents want it called.

Both offices wrote back. The local office, RIPR, said that they would use the real name during its initial reference in a story and Obomacare afterwards. Further, "[...] because (like it or not) that's what many people know it by." I was peeved by this sloppiness and the lack of an historical grasp of the significance of names and so sent a rather strident response (for which I did apologize later). The message I receive from NPR's ombudsman was remarkable:

Thank you for your inquiry. We asked Ron Elving about the term and here is his reponse:

Initially, the term was coined and used by opponents of the ACA. It had a sneering kind of tone to it, implying that the president was trying to imitate or piggyback on the popularity of Medicare. (As indeed a lot of commercial products have done since 1965.) The White House initially resisted the term for this reason, preferring Patient Protection and Affordable Care Act.

In the headline wars of the cable tv news world, of course, PPACA never had a chance. Obamacare became increasingly common.

So some while ago, the White House did a turnaround and embraced the term. I am attaching a copy of the memo David Axelrod wrote about the term and why it was okay to use it as far as he was concerned. Axelrod was still in the White House as the top political advisor at the time and is now in Chicago co-directing the re-elect campaign.

From: David Axelrod
Date: Fri, 23 Mar 2012 17:34
To: [Supporters]
Subject: Hell yeah, I like Obamacare

Friend --

I like Obamacare.

I'm proud of it -- and you should be, too.

Here's why: Because it works.

So if you're with me, say it: "I like Obamacare."

Obamacare means never having to worry about getting sick and running up against a lifetime cap on insurance coverage. It gives parents the comfort of knowing their kids can stay on their insurance until they're 26, and that a "pre-existing condition" like an ear infection will never compromise their child's coverage.

It's about ending the practice of letting insurance companies charge women 50 percent more -- just because they're women.

And Obamacare can save seniors hundreds of dollars a year on prescription drugs -- and gives them access to preventive care that is saving their lives.

President Obama never lost sight of the fact that this reform is about people. People like his own mother, who spent the last years of her life fighting cancer -- and fighting with insurance companies, too.

That shouldn't happen. And because of Obamacare, it can't.

So next time you hear someone railing against Obamacare, remember what they're actually saying they want to take away.

And, today, stand with me in saying, "Hell yeah, I'm for Obamacare":



P.S. -- Side note: Can you imagine if the opposition called Social Security "Roosevelt Security"? Or if Medicare was "LBJ-Care"? Seriously, have these guys ever heard of the long view?

If President Obama wants to call it Obamacare then I am dropping my opposition and joining those who are reclaiming it as a term for a positive accomplishment of his first term.

Open a new terminal window from the command line with helper script on OSX

With the help of a little script called term you can quickly tail the logs of remote machines each in its own window using

for n in 1 2 3 4 ; do \
term -t ds$n ssh ds$n tail -F ~qs/var1/logs/\?s-common.log ; \

The four machines I am watching are called ds1, ds2, ds3, and ds4 and the two log files I am interested in tailing -- qs-common.log and ds-common.log -- are in the same location. Term uses the -t option to set the window's title (if it is not given then the first argument is used.) Don't forget to use ssh-id-copy to configured the remote hosts to use a private key so no password is needed

The term script is

if [ "$1" == "-t" ]
shift 2
osascript <<EOH
tell app "Terminal"
set currentTab to do script "printf \\"\\\\e]0;$TITLE\\\\a\\""
do script "$@" in currentTab
end tell

A kanban for one

I know that a visual workplace works to improve productivity. What I did not think about was that it can work for the individual just as well as for the team. During the weekend's browsing I came across this posting about Kanban for One. And, better, the inspiring photograph of Nomad8's office with a kanban board at most desks.

Thus inspired, I created my own Kanban board and used it right from the start of the day. And it worked! It was much easier to manage the set of activities over the day. It was delightful seeing the steady accumulation of "done" tasks. It was also surprising how quickly tasks got stuck. (And it is the stuck tasks that tend to get forgotten.) I plan to keep using the system and see how it goes.

The board is a 1'×2' piece of foam core board from a discarded marketing sign, the markings are with white pencil, and the tasks are written on 2"×1.5" Post It notes. It really couldn't be simpler to construct.

Note that I am using the kanban board in conjunction with an existing bug tracking system. The backlog is managed by the bug tracker. The queue needs to be on the kanban board, however.

Less opportunity to see how much hard work is necessary

Comment on Ian Schreiber's blog post Dreamers:

There seems to be less and less opportunity to see how much hard work is necessary to achieve proficiency and (sometimes) success. The actions of a craftsman in wood, for example, can be seen by sitting at the side of a bench and being attentive. What works and what does not is equally visible. And if the craftsman has a the habit of talking to the work some of his interior worries and choices are available.

My children have no visibility into my work. I am a software programmer. I have been at this trade for 30 years but what can be seen of my efforts and the growth? There is nothing.

From my children's perspective there is no history there is only now and the future. No wonder they and the kids you write of think they can be famous. It requires so little effort.