On security

My VPS recently got banned for spam which surprised me since none of my soft there sending email. So my first thoughts were that this is a mistake (e.g. due to IP address spoofing) or some vulnerability in that VPS is exploited. Since I could not verify or affect first and second one is more dangerous I started investigation.
So now I assume that someone broke into VPS. It could be 'traditional' breach with gaining login or transient one without leaving traces on disk.
Inspected my services: top; netstat; lsof -i; df; docker ps -a
See logs: /var/log/syslog*, /var/log/auth*
And there are continuous attempts to brute-force ssh passwords - someone who already got in would not to do that. Miss.
Checked for extra users or installed email servers - none.
Syslog contained multiple messages like

TCP: TCP: Possible SYN flooding on port 8080. Sending cookies.  Check SNMP counters.
nf_conntrack: table full, dropping packet
net_ratelimit: 20 callbacks suppressed

then

ziproxy invoked oom-killer

So now I get a clue. Now I get something that looks like DOS but no 8080 port should be visible outside... It turns out that syslog shows port of docker container not host's.
One of slightly surprising things of docker is that it uses LXC wchich are "chroot on steroids". So any process running "in docker" is actually normal process running on host's kernel and there's no "container" process. That said, ziproxy that runs in Docker was accessed from outside what caused it to send emails. Now ziproxy is a web proxy. Was there some overflow that injected outside code or proxy was used as is?
I found no known vulnerabilities for ziproxy, so it's probably normal operation. Since complaints were for SMTP it should have been CONNECT method. Looking at ziproxy sources I found out that CONNECT is supported by ziproxy. To check this I tried:

$ telnet localhost 8888
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
CONNECT my-vps-domain.net:80 HTTP/1.1
HTTP/1.0 200 Connection established
GET /favicon.ico
301 Moved Permanently
nginx/1.2.1Connection closed by foreign host.

(lines in bold is what I typed)

Inspecting syslog I found that first "Possible SYN flooding" was exactly at the moment of spam message sent. And there are no similar events before that.

Now most likely situation was that I accidentally exposed ziproxy (it was intended to be visible from localhost only) and someone used CONNECT method to relay email sending. Far simpler than I was afraid of.
To verify this I started ziproxy again and there's "Possible SYN flooding" again.

The actual problem with my setup was that I misunderstood defaults of "-P" and "-p" Docker switches. "-P"  exposes all ports even without parameters. "-p", given ports only exposes ports on all interfaces while I needed only "loopback".

I used "nmap" to find opened ports onl VPS and that helped to fix Docker settings.
And I also installed fail2boot. It's default settings are reasonable to throttle DOS and brute-forces.

Receipe of the day: gradle with eclipse config.

I use Eclipse STS (Spring Tools Suite) with Spring support. To help editing gradle.build file two plugins should be installed:

• Groovy eclipse plugin http://groovy.codehaus.org/Eclipse+Plugin
• SpringSource Tool Suite Gradle Integration (is included into STS)

To generate eclipse project file from build coniguration add following to build.gradle:

apply plugin: 'eclipse' 
eclipse {    
  project {
    natures 'com.springsource.sts.gradle.core.nature',
            'org.eclipse.jdt.groovy.core.groovyNature',
            'com.springsource.sts.gradle.core.nature'
  }            
}

I remove about 40 lines of code a day

Two days of work:

422  ++++++++---------
248  +++++-----
84   ++--
141  ++----
85   ++--
36   +-
480  ++++----------------
9    +-
58   ---
453  +++++++++---------
126  +++--
71   ++-
44   +-
72   +--
79   ++--
109  +++--
111  ++---
12   +-
16   -
20   +-
30   +-
32   --
16   +-
40   +-
21   +-
1    -
15   +-
16   +-
16   -
12   +-
39   +-
3    -
11   -
52   ---

AND several bugs fixed in code that was changed.

It is a destiny

Every project that I take from previous developers I have to answer the same question: "Was developer who wrote this brain-damaged, just sloppy or this horrifying spaghetti mess reflects some business logic that is actually needed".

The worst part is that there is never easy answer to this. So it always ends up with tedious ongoing refactoring and meticulous analysis.

Parentheses in Lisp: let's count them.

Parentheses are the first thing any newcomer notices about Lisp program. This is something that put off many who does not want to learn the language just by looking at the syntax. However, If they think that there are too many parentheses, do they actually count them? Let's look at a fragment of java code:

public static void closeSocket(final Socket socket) {
        try {
            socket.close();
        } catch (IOException e) {
            LOG.error("Error closing socket " + socket, e);
        }
}

Total: 14 braces and parentheses.

Now, below is how equivalent Lisp code could look like (let it be in Clojure as an example)

(defn closeSocket [^:Socket socket]
  (try 
    (.close socket)

      (catch IOException e
        (LOG/error (str "Error closing socket " socket) e)))))

Total: 14 braces and parentheses.

Now I would say that those who reject a language judging by it's syntax do not even look at syntax closely.

More on concise code

"How many lines of code have you deleted in one day?"
My answer is "~2150 of perfectly working code and not a single feature broke".

And one more essay by Michael Feathers: The Carrying-Cost of Code: Taking Lean Seriously

Why programming language design is hard

It is harder to write reusable code because it is used in more than one context. In particular it is the case with unit-testable code, because you have to write code for use in tests and in production. But programming language is probably an extreme: sooner or later every feature you have will be used in every possible combination.

Quote from Neal Gafter

There's more to my previous post. The quote is actually not about unnecessary code but about unnecessary features but still pertinent http://www.infoq.com/articles/neal-gafter-on-java :

And when you add something that someone doesn't benefit from, it's actually a negative for them. Even though they don't have to use it, or look at it, or care about it, it makes the system more complicated for them.

Concise code

This subject has been bothering me for a long time. So I decided to collect my arguments in single place, here.
I insist on that good code is concise code (but not necessarily otherwise, to be precise). Objections to this that I heard most often so far are
1. The code is being read more times than it is written. Denser code is hard to read.
2. The code is for human beings and that is why it should be clear. Denser code is harder to read.
3. Copy-past is not bad since in future requirements may change and then copied parts would naturally diverge. Extra code in the project is just extra code - it sits there and costs nothing.

And all of them ARE WRONG.
Yes code is being read more often than it is written. And that is why short code matters. There are numerous observations that human brain can manipulate by limited number of symbols at a time. If you present whole program in a way that brain can grasp at once you get a bonus understanding what is going on here. This is instead of working as meat-based decompiler trying to understand pieces of system, then trying to put pieces together, then trying to understand how this system works as a whole. By writing program that fits in your brain you turn computer into your friend that has interface compatible with you. This is surprising how may program systems where not even one developer understands. Developers working on them just get used to navigating symbol browsers in their IDE without even trying to understand parts of code that are not directly related to task at hand.
Yes code is being read more often than it is written. To an extent. New code almost always needs refactoring as requirements and more appropriate designs become more apparent. The less code is there the more chances rewriting it would be feasible the better chances design would be sound in future. In that very future rewrite is often impossible. So keeping code compact is one of ways to better software. I have seen developers who were reluctant to rewrite code that was first written yesterday even though every one understand that rewrite would significantly decrease complexity. You see? Even yesterday's code might be already too old for rewrite.
Every middle or large size project I participated in contained many hundreds or even thousands lines of unnecessary code. Worse then it's redundancy is often not apparent since only some parts of it have textual similarity. You need time to understand that this large chunk of code is completely unnecessary.
And anyone who will dig this code will have to expend an effort to understand these parts of code. One have to understand code before changing it, right? Instead of immediately seeing that this and that parts of system are essentially the same thing with minor tweak in the middle that is passed as parameter, I have to work as overpaid diff program that scans two large chunks of code for that single dot that is in different place. And after finally finding it still wondering if it is bug introduced by sloppy maintenance or a deliberate change in functionality. This is part of increased support cost due to unnecessary code. Unnecessary  code is not jut sitting there on a hard drive and bothers no one. Instead it constantly sucks team's time and energy to support it.

They say that when requirements change those copy-pasted pieces can be changed separately. The problem is that you never know how those requirement will change. And there is big chance at least, say 50%, that requirements will change in a way that would require both of those pieces. And the large project the higher chances that required changes will be cross-cutting. You ain't gonna need it.

Yes, in less denser code it is easier to understand one isolated line of code. But is there a point in understanding line "i++;" or "}"? Is there a point in understanding separate lines of code if  you still do not understand what is going on? You need to understand what the program is doing if not as whole but at least a substantial piece of it. Yes in denser code you probably need some times more mental effort to understand it but is it a problem since you need in proportion less lines of code that does the same thing? And remember that you will have better chances to understand and maintain the code.

I have one more observation related to good design. To write concise code you have to understand deeply what your program should do. And this is only way to do it. Often people who unconsciously equate boated code to more approachable one imagine some big system written as 1 megabyte regexp as example of brief code. No you cannot do that. At most you could cram what you have into a few percent less lines of code indeed impairing readability. To have better code one should understand it.

Several years I have been working with Java. And there is one thing that is related to it. Some tools (programming languages)  just do not have adequate means for code structuring. Often trying to implement in Java approaches that reduce code significantly in other languages lead to more bloated code. Java just resist being concise. So it is not just about design.

One last piece is somehow related to Java is use of design patterns. As any good idea that gets into popular culture (popular programming culture in this case) it becomes a parody on itself. I find this gem by Sarah A. Sheard to be most memorable text on the subject. They forget that every pattern has not only benefits but also an area of applicability and drawbacks. So one have to ponder if code would be better if  a pattern is applied. They think instead that the more patterns used the better. They forget that patterns is not something devised by demigods and given to us mortals to be unquestioningly used. Patterns are extracted from real systems. And ones system might need some pattern that is not in GoF bible. And the best way to extract relevant patterns is refactoring. Seeing how often patterns are overused is upsetting since most of them increase amount of code. How often have you seen factory is used where a plain constructor would suffice? Sigh.

Logging and bug reports

A useful bug report has all three parts: how to locate or reproduce it, what is actually there and what should there be.
I would extend this to the all error reporting in software. Consider, for example, logging. I have many times seen and written code where log messages are written in ad-hoc way just to help with code debugging. Then such system goes production and something terrible happens. And then you discover that error message was written but there's tiny missing piece of information, say identifier of troubling object, that would help you to diagnose the problem. Developer would not notice this before because he could just start debugger with a breakpoint, and look into actual state to find all interesting values. But in production one usually can not afford this.
So here is a rule I have now: if you write error log message put yourself in a position of someone who needs to diagnose or resolve this problem in a production environment. This normally means that error message is a brief bug report:

• You say where it happened so developer might find code in question and sysadmin might find configuration problems. And both may find data that caused problem if there is such thing.
• Error message says what's wrong
• Error message says what is expected if that is not obvious from above. For example, if some value is inappropriate, say also what are allowed values. Not just "Salary value out of range", but "New salary value 1000 is out of range. Allowed range is [1000000, 5000000).".

PS This classic text is somehow on topic.

Do not do this, please: this instanceof

Since I do like refactoring I decided that I would document code patterns that I stumble upon.
Consider following code (Java). First snippet shows
It is uneasy for me to explain why this is not a good solution because it is so obvious to me. It's like explaining why anecdote is funny to someone who does not get it. But since I met such code in more than one project written by different people there must be something worth explaining.
First, look at better version of this code:
This rewrite eliminates runtime check, removes dependency of base class on it's subclasses and removes temptation of junior developers who will modify this code after you to insert more insanceof checks in same method. And after all, in Java subclassing is language's idiomatic way do do such things.
I have written what wanted on this case but it is hard to me to stop here, so I continue.
If I would do refactoring I would stop after this rewrite. But looking at the methods I would scent that something else needs improvement: call to "bark" suggests that it belongs not to absrtact animal but to a dog. So if indeed we have only two subclasses Cat and Dog, I would further rewrite the code as:
I would advocate using composition over subclassing if everything else is equal. So if after these changes there's no real code left in Animal I would convert it to interface:
They tell me "ship it" and here I stop.

Index optimization: the code.

Below is an implementation of index optimization algorithm implemented in Clojure. Perhaps it is not an example of good Clojure style but it leaves less ambiguities than plain words description from my previous post. The GitHub's gist is below, I hope that it is self-explanatory:

Optimizing set of SQL indexes.

In my current project we generating lots of code including DDL for MySQL database. To make generator code cleaner and DB happier I have added optimization procedure that takes set of indexes that cover all necessary query cases and outputs only those indexes that are really needed. For example there is no need  to add two indexes with same specification, if you have unique and non unique index with same fields then unique index is sufficient and so on. One more observation is that  if you have index on fields [a, b, c, d] then you do not need separate indexes [a, b, c],  [a, b] and [a]. There are lso 3 kinds of indexes that are important here

1. Plain, normal index
2. Unique index (is also a plain index)
3. Primary key index (is also a unique index)

So the algorithm that optimizes indexes uses

1. Some partial order on index specifications, that can be implemented as predicate that says "this index is covered by other"/"this index adds new query optimizations compared to other"
2. Procedure that finds "supremum" of given index set

There is one further optimization that I have not implemeneted because improvements from it would be minimal for our project. This optimizaation would re-order fields to try to reduce number of indexes. For example if there is laready indexes [a, b, c, d] and [a, d] then one may leave [a, d, b, c] instead.

Note that if you you use indexes not only for query/filtering but also for ordering queries then reordering filelds in indexes can be not appropriate. In this case one should also consider "ascending"/"descending" property of indexes (and do not mix them).

Showing dependency cycle in Eclipse.

Here is tiny plug-in that gathers complete build path dependency tree of a Java project in Eclipse. Eclipse tells only that there is some build path cycle but does not tell which projects are involved. If your workspace is crowded this could be a problem. The plugin allows to automatically find out cycles in project dependencies.

JDK7 - little better

My gripe about closing streams is addressed in latest (not released yet) JDK7. This version allows to write following (see http://mail.openjdk.java.net/pipermail/coin-dev/2009-February/000011.html):

try (BufferedReader br = new BufferedReader(new FileReader(path)) {
           br.readLine();
}

And it actually compiles :) My complain is still intact however: instead of allowing me to write abstractions that I need Java requires me to for next release version and pray that it will have programming features I need. Why I still complain? Because JDK7 still does not have closures (I hope - yet).

XML query with Clojure

I tried to write some script that would allow me to look for particular values in set of xml files. I found out that Clojure supports XML parsing, query and editing of XML out of box (thanks to clojure-contib). The script follows (it's for Clojure 1.1):

(ns main.xquery
  (:require
    clojure.xml       
    [clojure.zip :as zz]
    [clojure.contrib.zip-filter :as zf])
  (:use
    clojure.contrib.zip-filter.xml))
 
(defn dir [dirname]
  (seq (map #(.getCanonicalFile %) (.listFiles (java.io.File. dirname)))))
 
(defn xml-file? [file]
  (and (.. file (getName) (endsWith ".xml")) (.isFile file)))

(defn query [input-file query]
  (let [xml (zz/xml-zip (clojure.xml/parse input-file))]
    (apply xml-> xml query)))

(let [dirname "/home/petr/workspace/CloTry/"]
  (filter #(not (empty? %))
    (map #(query % [zf/children :item (attr= :type "green") (attr :name)])
      (filter xml-file? (dir dirname)))))

This script scans all XMLs in given directory and returns list of matched "name" attribute values. Interesting work is in "query" function that is called with XML file and XPath-like query expression:

[zf/children [:item (attr= :type "green")] (attr :name)]

The query expression can be read as: take children of root element, retain ones whose attribute "type" has value "green" and take value of "name" attribute of these elements. Note usage of clojure.zip functions that are generic way for traversing of editing hierarchical data structures, XML being one of them. xml-zip function takes parsed representation of XML file and returns it's representation that can be used with generic functions. There was one problem: function "children" is defined in both clojure.zip and clojure.contrib.zip-filter namespaces. Both functions have same semantics but former gives NPE for some reason so I explicitly referred to one from clojure.contrib.zip-filter namespace. Given that a XML file has this content:

<doc>
 <section name="everything"/>
 <item name="first" type="black"/>
 <item name="second" type="green"/>
 <item name="third" type="yellow"/>
 <item name="fourth" type="green"/>
</doc>

The script outputs:

(("second" "fourth"))

Update: If you are interested to know file which produced result the above solution is OK. If you need to know just elements that matched then this would be neater

(let [dirname "/home/petr/workspace/CloTry/"]
  (apply concat
    (map #(query % [zf/children :item (attr= :type "green") (attr :name)])
      (filter xml-file? (dir dirname)))))

Update 2: Or even shorter :)

(let [dirname "/home/petr/workspace/CloTry/"]
    (mapcat #(query % [zf/children :item (attr= :type "green") (attr :name)])
      (filter xml-file? (dir dirname))))

Stored procedures. I love to hate them.

Why should it be so damn involved?

declare done int default 0;
declare continue handler for not found 
    set done = 1;            
open carrots;
repeat
    fetch carrots into weight, location;
    do_something(weight, location);
until done end repeat;
close carrots;

Instead of something like:

for weight, location in carrots do
    do_something(weight, location)

SQL match functional approach so nicely but storead procedures are horrible procedural mess. And it feels like programming in fortran or assembly.

Java sub lists

This feaute is there for a while but I did not noticed it. It turns out that java.util.List#subList allows not only read but also manipulate original list just like list slice manipulation in Python but with clumsy syntax:

java.util.List l = new java.util.LinkedList();
 l.add("a");
 l.add("b");
 l.add("c");
java.util.List s = l.subList(1, 2);
 s.add("*");
System.out.println(l); // prints [a, b, *, c]
 s.remove(0);
System.out.println(l); // prints [a, *, c]
java.util.List r = l.subList(2,3);
 r.addAll(s); 
System.out.println(l); // prints [a, *, c, *]

Again about jobs and workspace resources

There is one more caveat about resources manipulation in Eclipse's workspace. Scheduling a job with a resource scheduling rule helps to avoid unwanted intersections between different tasks. Say one job wants to modify some resource but other wants ro read some consistent state or modify same resoruce. This has two problems. First, there is no robust way to ensure jobs order, and second there is chance of lockups when using jobs for resource manipulation.

It turned out that this does not play well if your job is scheduled as part of bigger transaction. Say, Eclipse started some rename refactoring, your hob invoked as part of "save resources" procedure and wants to modify things so resources are consistent with saved editor state. And then you get into deadlock. Refactoring locked your resource for modifications, your job want to lock it for modifications too and there's no way to tell that your job is "nested" transaction. Oops.

Scheduling one job after other is tricky too. Even Eclipse's code waits for build completion before launch by polling job manager to ensure that there are no active jobs that tagged as "build job".

I found that in most cases (if your requirements are simple) better approach to modifying resources in Eclipse's workspace is to consistently use IResource's (IFile, IFolder, IProject) methods throughout your code instead of using plain java.io package. This would ensure that Eclipse is aware of your changes and you do not need to refresh resources after every modification to make Eclipse aware of it.

I want hyper continuations.

A situation: I debug a problem in program and paused it at some breakpoint. At this moment by examining state of program I find out that this problem can be fixed by other developer and want to re assign the bug to him. Now: how do I describe the state of program? Describe reproduction steps? Add stack trace from logs? Call him to come at my desk so he can debug further?
That the real options. Unreal but wanted one is: I serialize state of whole program, attach it to the bug report, that other developer loads state from program and can examine everything he needs.
That's what I would call a hyper continuation - you send it to other part of the world and do not care about details.