MySql load balancing with HAProxy

We use pair of HAProxy servers at work to route and balance all of our traffic. It is a highly reliable tool with little time or performance overhead. Yesterday I need to balance a load of MySql connections across several slave servers and so used HAProxy to do that. Since I did not find the exact recipe I need on the web here is my test configuration contribution to the MySql global distributed cookbook:

defaults
    mode http
    stats uri /haproxy-stats
    stats enable
    log global
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

listen admin
    bind *:8000
    mode http

listen mysql
    bind *:3306
    balance  leastconn
    mode     tcp
    option   tcpka
    option   srvtcpka
    option   mysql-check user haproxy
    server mysql1 192.168.1.101:3306 weight 1 maxconn 200 check
    server mysql2 192.168.1.102:3306 weight 1 maxconn 200 check
    server mysql3 192.168.1.103:3306 weight 1 maxconn 200 check

# END

Each MySql server used must have the haproxy user created using

create user 'haproxy'@'%';
flush privileges;

The wildcard can be replaced with the IP address of the HAProxy server.
I want to thank the folks at Homebrew for the haproxy formula.

Low tech needed

And how is this better than writing the inception and expiration dates on the container with a grease pencil?

Given the pace of ...

Where is the call from?

I wish my phone told me where the person was calling from and not where there phone number was registered. I would have been much more prepared to take a call from Matunuck, RI than from Long Beach, CA.

When the solution hides the problem.

A comment on How to Create Your Own 'Dynamic' Bean Definitions in Spring:

The solution hides the problem. I enjoyed reading about the details of the insides of Spring, but the solution was too complicated for the problem. A real value of Spring is that generally it does not get in the way of straightforward solutions. The problem here, picking a datasource based on key could have been handled with a simple DataSource delegate that can pick the actual datasource from an injected mapping of key to datasource. The management of the mapping would be done elsewhere -- JMX, webapp, etc.

For me, the red flag would have been when I thought the solution required a weekend of prototyping to understand the internals of Spring.

This is off topic, but related to picking solutions. You also need to think about the person maintaining your solution. A complex solution will require that the maintainer be at least as talented as you and that can be an unnecessary burden on the organization.

Cloudbeard

With all the Viking history and fiction I have been reading recently, I would that you now address me as Cloudbeard.

Problem solving sub-101

Is a whole book needed to tell you to solve problems by enumerating the subproblems, stop enumerating when you find one with few dependencies, visualize the dependencies, fix the subproblem, revert the fix when necessary, rinse and repeat. Oh, and give the method a catchy name like The Mikado Method. I don't think so.

Update: This comment was a bit too snarky. I sometimes forget that junior developers do need to be reminded of the basics. And being able to name this is important as the Gang of Four confirmed for us with Design Patterns: Elements of Reusable Object-Oriented Software.

I no longer have patience for certain things

"I no longer have patience for certain things, not because I've become arrogant, but simply because I reached a point in my life where I do not want to waste more time with what displeases me or hurts me. I have no patience for cynicism, excessive criticism and demands of any nature. I lost the will to please those who do not like me, to love those who do not love me and to smile at those who do not want to smile at me. I no longer spend a single minute on those who lie or want to manipulate. I decided not to coexist anymore with pretense, hypocrisy, dishonesty and cheap praise. I do not tolerate selective erudition nor academic arrogance. I do not adjust either to popular gossiping. I hate conflict and comparisons. I believe in a world of opposites and that's why I avoid people with rigid and inflexible personalities. In friendship I dislike the lack of loyalty and betrayal. I do not get along with those who do not know how to give a compliment or a word of encouragement. Exaggerations bore me and I have difficulty accepting those who do not like animals. And on top of everything I have no patience for anyone who does not deserve my patience." -- Meryl Streep

Re: Using exceptions when designing an API

A comment on Using exceptions when designing an API:

When designing an API I use the "get vs find rule." The API's user needs to know how to handle the absence of a result. A get is expected to always return a result while a find is not. And so when coding "getX()" it returns the result or throws an exception -- such as IllegalStateException -- if there is not one. Where as, when coding "findX()" it returns a result if available or a null, or an empty collection, if there not one.

Quick guide to making PDFs with text outlines from HTML

Create the HTML document.

<html>
<body style="margin: 0; padding: 0">
<div style="width: 2in; height: 2in; border: 1pt solid blue;”>
<span style="font-size: 0.5in">Hel</span>lo
</div>
</body>
</html>

Use the OS X command line tool wkhtmltopdf to convert the document to pdf

wkhtmltopdf -B 0 -T 0 -L 0 -R 0 --page-width 2in --page-height 2in box.html box.pdf

The fonts in this are embedded (or likely to be). To replace the fonts with text outlines use

gs -sDEVICE=ps2write -dNOCACHE -sOutputFile=- -q -dbatch -dNOPAUSE -dQUIET box.pdf -c quit | ps2pdf - box-nofont.pdf

Done.

I'm going to miss that man.

Response to Tom Hoffman's "Peter Greene on Tenure"

Response to "The threat of firing is the great 'Do this or else...'" in Tom Hoffman's Peter Greene on Tenure:

On the whole I find that teachers take on too much accountability and the administration too little. To some degree this is self imposed. I don't know of any other profession that has such a shallow managerial hierarchy. The whole point of a manager is to make this kind of crap go away -- if not in actuality then at least from visibility. Tenure is not the solution and, understandably, sets up teachers for further criticism as being detached from appropriate accountability and economics of their employment. Teaching needs to add management and remove tenure.

Inner curmudgeon awoken

My inner curmudgeon was awoken this morning when I read of yet another OS X application for note taking with cloud storage and a corresponding web application. Doesn't anyone know how to use a text editor in conjunction with a remote file service like Google Drive, et al? And, for the vast majority of users, there is no purchase price or recurring fees.

I want a Markdown "during-view"

I like that Markdown and its kin have been adopted by more than just the geek community. The simple inline directives do not take away the focus of writing the text and the formatting available covers most of needs. However, the trend to provide an immediate, fully formatted view of the text is misguided. These "previews" take away from the goal of maintaining focus on the text and not it's presentation.

However, I too like the comfort of a correctness that comes with seeing a full formatted document and especially so with regards to sections and lists. What Markdown editors need is not preview but a kind of WYSIWYG presentation that keeps the primacy on the text and secondarily on the structure of the text. A means of doing this is to present the inline directives with formatting and the directives too. Thus, when I type *emphasize me* it is shown in the editor as *emphasize me*. If I use a heading directive

A short work
============

it presented for editing as

A short work
============

I did once see this kind of editor in a Java wiki application, but I have long since lost my reference too it. Perhaps it will resurface again as the Markdown tools authors rediscover this style of editing.

I am not going to create another account just to use your feature

Dear Marco,

I really like the audio tools included in Overcast, but I am not going to create another account just to use them. I read the FAQ and while I understand that in general saving battery life is great, but, really, how much battery is used to check a few feeds once a day? So, good luck and keep me informed of the non-account version of Overcast.

What would one use programming for if programming were easy?

I stopped by Slashdot today and found Normal Humans Effectively Excluded From Developing Software. I read it and many of the comments and started to think about this problem again. In my mind the problem is if programming were easy then to what purpose would someone put programming too? I am going to assume that what is meant by programming is of a general purpose kind so that, for example, using R is not programming.

Update: People collect and organize things and while they do that they share the experience, the lessons learned, and the things found. Would you program that?

To be continued...

Logging and the utility of message patterns

I wrote a logging layer over log4j sometime ago. The layer was to encourage our developers to log at any point in the code where they thought some extra context would help resolve runtime errors. And so the layer had to have a familiar API, be effortless to code, and cheap to call. This posting is mostly about the cost to call the logger.

When a message is not going to be used you don't want to incur the cost of creating it. How often have we seen logging code that looks like this

logger.debug("short explanation: v1="+v1+"; v2="+v2+";");

Here the logged message is composed before the call to logger.debug() and it is composed even if the logging level is above DEBUG. If v1 or v2 have a complex toString() implementation, then you add this compositional time to the calling cost too.

The simple solution is to use java.text.MessageFormat with the debug() method, for example

logger.debug("short explanation: v1={0}; v2={1};", v1, v2 );

In this way, the time cost to calling debug() is the time taken to push three values on to the stack, call the method, and pop the three values off the stack. No composition time (or memory) costs are incurred at all. (I handle exception messages the same way, that is, all exception constructors accept a message pattern and message parameters and the message string is not composed until it is actually needed.)

I have since replaced Java's MessageFormat with a (mostly) compatible local implementation. This local implementation does simple composition tasks like formatting numbers without commas and Unicode escaping all non-ASCII, and non-visual characters. However, the primary reason to implement a replacement was to expand the available format types and format styles.

The first new format type added was json so that now when a parameter is so typed its value is transformed to a JSON representation before adding to the message. For example,

Map v1 = ...;
logger.debug("short explanation: v1={0,json}", v1 );

will convert v1 to a JSON string only if the log level is DEBUG

short explanation: v1={ "k1": [ "e1", "e2", ... ], "k2": [ ... ] }

(In hindsight it should have been a format style.) 

Today I added the string format type and two format styles. One format style specifies a substring of the value using a start index & length and the other a start & an end index. The start and end indexes can be given as numbers or as searches. For example, the style /foo/:10 specifies a 5 character substring starting where "foo" is first found, and the style /foo/../bar/ specifiers a substring between (and including) the two searches. The using search is particularly useful if you know that somewhere in a very long string is an identifier that would be useful to have in the log.

Basic human concern for the welfare of others

I had a surreal moment this morning while walking Milo at our usual spot. A car was found in the nearby pond. (I later found out that two cars were stolen last night and this was one of them.) I happened to encounter a small group of acquaintances running with their dogs. I told them what I knew and their immediate response was 1) having a car in the pond is not healthy for the pond and 2) why were the police leaving their cars idling. WTF! Not one of them expressed interest or concern for a person or persons that might be in the car. I was shocked at their lack of the basic human concern for the welfare of others. I will resist extrapolating this one data point to the causes of America's problems.

A gnuplot example

If you searched for Gnuplot in this blog you know I am a fan of the tool. It is easy to install, somewhat easy to use, and very likely someone has already had and solved the exact same problem you are having now and wrote a blog entry about it. This is my contribution to the globally distributed Gnuplot cookbook.
I want on a single chart the lines graphs of count of all queries, count of matching queries, and mean duration of all queries. This chart gives me the temperature of the previous day's query load which is a useful qualitative factor for capacity planning. The data retrieval results in records representing a 10 minute interval with counts of all queries, counts of matching queries, and mean durations. This is  then plotted to give a chart like the following:

The file of records contains values that looks like

2014-07-07T00:10:00     10449   3850    934
2014-07-07T00:20:00     18445   5676    407
2014-07-07T00:30:00     18497   6507    535
2014-07-07T00:40:00     11419   4463    734
2014-07-07T00:50:00     11634   4454    724
2014-07-07T01:00:00     11960   5077    808
...

The Gnuplot script is

set terminal png size 800, 400 small interlace
set output "/tmp/chart.png"
set size ratio 0.5

set xdata time
set timefmt "%Y-%m-%dT%H:%M:%S"

set xlabel "Hour"
set format x "%H:%M"

set ylabel "Count"
set ytics nomirror
set yrange [0:40000]

set y2label "Milliseconds"
set y2tics
set y2range [0:4000]

set title "MetaData matching search counts and average durations (10 min intervals) for 2014-07-07"

plot \
  "/tmp/chart.data" using 1:2 title "Query count" with lines, \
  "/tmp/chart.data" using 1:3 title "Matching count" with lines, \
  "/tmp/chart.data" using 1:4 title "Request duration" axes x1y2 with lines

The chart is produced daily and it is useful to visually align several days of charts to look for patterns. For comparable charts this requires that the charts all have the same y-axes ranges. To do this Gnuplot's range setting is used. If you do not want a fixed range then remove the set yrange ... and set y2range ... lines.

I hope this helps someone else with their Gnuplot use.

Atlassian has forgotten that Jira is a communication tool

Atlassian has forgotten that Jira is a communication tool. Comments need numbers so the team can communicate with specificity everywhere. See Skype and development teams as it applies to Jira also.