Wednesday, 22 March 2017

The three stages of a dev wiki

First They Ignore You
A well-meaning dev sets up the wiki on a spare server. It contains some important information about the network that nobody else understands, a very terse guide to application architecture, a page about local bars and restaurants that is, somehow, already out of date.

Then They Like You
Very gradually, through dint of a handful of well-meaning devs continually suggesting people add things to it, it grows. Useful information about how to set up environments is added, as well as how to debug more complex problems.

At some point it reaches a tipping point of usefulness and becomes the defacto place for documentation. Reams of notes from meeting, observations from research spikes, and ideas from brainstorming sessions are added, large parts duplicated and contradictory. It is chaotic but a goldmine. The page on local restaurants and bars is never updated. Only one of the bars is still in business.

Then They Ignore You Again
Eventually the wiki comes to the attention of upper management. Maybe Support. Maybe Sales. Maybe Ops. Somebody sees it, sees that it is good, sees that it is a democratic nirvana to which everybody can contribute and nobody has overall control, and decides it should be theirs.

They institute a Wiki Transformation Project, which despite being a glorified database migration somehow takes six months to complete. When finally complete it is introduced with a company-wide powerpoint presentation (attendance mandatory).

The new wiki is on some more intimidating platform: Confluence maybe, or Sharepoint if you are really unlucky. Half the users do not have permission to edit pages they wrote themselves. All of the formatting is screwed up and several important diagrams and flow charts have been lost completely. The page about local bars and restaurants has been quietly deleted along with, it becomes apparent only later, a number of other pages people relied upon. Only a handful of well-meaning devs ever add anything new to it again.

Thursday, 9 March 2017

Scrum is dumb

Agile, as I never tire of telling people, is a broad set of pretty uncontroversial principles while Scrum is something invented by management consultants to make money. Nowhere in the agile manifesto does it mention stand-ups, story points, or sprints. You can be agile without doing any of those things.

In seventeen years I've never been a part of a team building a thing for a customer who was remotely interested in getting updates every fortnight. About 90% of the time I've been in teams where we all worked on different parts of the same massive mature application (or family of applications). Most <air quotes>projects</air quotes> were coded by only a single developer, very few indeed by more than three. Most could only be released with a great deal of difficulty (sometimes having to navigate the audit processes of customers, and the limited resources therein) and were rarely done so more than once a quarter at the absolute most.

The one time I can recall building a new piece of software as a collaborative team, the customer was represented in the sprint reviews by our own support analyst and, when it came time to hand over some money, declined having never once seen what we produced. Far more often than not there simply was no customer beyond a products guy with an idea they had long since lost interest in.

Scrum, then, has always felt like a solution in search of a problem.

Of the three aspects mentioned above, daily stand-ups I am generally in favour of. They are short so even if unproductive not very harmful, and they do provide a degree of team coherence, when everybody is -as usual- working on different things.

Anecdote: Once upon a time, long before I'd ever heard of a sprint, teams would send out weekly progress emails detailing what each member had achieved last week and was planning for the new week. One poor sod in a different team became a running joke for having the same 'finish N-Patch' status for well over six months. In the end he left the company without ever finishing it. At the time I thought he was just an idiot but now I tend to view the whole episode as a failure of pastoral care. He wasn't getting on with the people he was living with and obviously wasn't coping with his work. I suspect he was probably a bit depressed. He needed support and he didn't get enough of it. If the team had done daily stand-ups then the work issues would have been obvious and, hopefully, they would have stepped in (where his manager did not) long before everything got so bad. At the very least it would have made the problem much harder to ignore.

Until recently I was kind-of okay with story-points and burn-downs and velocity and all the rest. Accept that estimations are wrong, I thought, instead measure your actual progress toward a finished product and do not get snared by the beguiling false certainty of man-hours. But too many interminable estimation meetings have beaten it out of me, too many questions of the form: "but how much is a story point really?"

In any case I don't think I've ever had a burn-down that actually burnt down, never had a velocity that didn't hover about zero for sprint after sprint after sprint. We couldn't book the points till the story has passed QA and QA were always trying to catch up with something more urgent. By the time our work became urgent nobody cared anymore. It was only urgent because it was being released anyway.

Stories that fit in a sprint are rare beasts too. As a user I want to do X we would write, and then realise this was a months work at least and our velocity was going to be zero-zero-zero-one hundred! Gamely we would try and split it up but rarely ended up with anything better than As a user I want an X button, and As a user I want X to happen when I press the X button. And sure enough our velocity would transform into half-zero-zero-one hundred!

In the end we started writing stories from the point of view of the components. As a thingabob widget I want the doohicky service to support Z in order that I know what to prepare in order to begin X. And the whole thing became meaningless.

I guess it comes down to why do sprints at all? To deliver frequently make sense, but who are you delivering to? If it's QA then who cares? Certainly they don't. If it's a metaphor, delivering kind-of-atomic chunks of work to a golden master version, then adopting git-flow will give you all this without four hours of mandatory meetings every fortnight. If it's to check you're going in the right direction with a product owner, why only every two weeks. I've never worked anywhere where I wouldn't frequently wander over to the product owner, be she architect or saleswoman, and ask her to clarify what she wanted.

I've sometimes said that I've drunk the TDD Kool-Aid. It works. I've seen it work. I'm convinced. My attitude to Scrum has always been more circumspect. I bought the principles, was optimistic, but had never seen it really work. Now I'm embroiled in a second attempt to "Go Agile," at a second company attempting it, and I see all the same failings all over again, I'm forced to confront the possibility that the whole thing is snake-oil.

Tuesday, 1 July 2014

Fun with standards

This, as I understand it, is completely legitimate c99:

>$ cat main.c
#include <stdio.h>

inline void foo()
{
        printf("inline\n");
}

int main()
{
        foo();
        return 0;
}


>$ cat outline.c
#include <stdio.h>

void foo()
{
        printf("out-of-line\n");
}

>$ clang -O0 -c main.c -o main.o && clang -O0 -c outline.c -o outline.o && clang -o inline-test-O0 main.o outline.o
>$ clang -O2 -c main.c -o main.o && clang -O2 -c outline.c -o outline.o && clang -o inline-test-O2 main.o outline.o
>$ ./inline-test-O0
out-of-line
>$ ./inline-test-O2
inline

Wednesday, 18 June 2014

Nice Languages Finish Last

I am writing things in Lua. Lua is a nice language, everybody keeps telling me so.

A very compelling use case for Lua is to use it as a scriptable configuration file. You can imagine how wonderful that would be:

Client wants to get configuration elements from the environment. They can already do that.

Client wants to get configuration from a different file with a weird proprietary format. They can already do that.

Client wants to use the most recent different file from a whole directory of them. They can already... Wait. They can't do that! They want to perhaps sleep for a bit and then check again. They can't do that either. You see Lua is a nice language and doesn't have a massive bloated API that can do everything. Python, Perl, or PHP could have done those things, because they are not nice.

They are useful.

Friday, 12 April 2013

Lessons From Working With Timezones.

I have been reviewing, and then when the reviewing turned up a load of pre-existing bugs actually fixing the damn things myself, a lot of code to do with handling timestamps. Here are some of the lessons learned.

Use boost date_time.
It's one of the boost bits that comes in a library rather than just a header file but don't let that put you off, it makes easy some things that were hard, and makes possible some things that were impossible. It is a good thing.

The rest of these rules will focus on the more prevalent posix types and functions.

Never put anything other than posix time in a time_t.
By posix time I mean time since midnight 1st Jan 1970 UTC, aka epoch time, or unix time, or timestamp. You will have a hard time persuading anything to treat your time_t value as anything else so just don't do it.

Never put anything other than the local time in a struct tm.
Effectively this means never calling gmtime. There is only one reason to ever call gmtime, and that is if you need to get a UTC time string, in which case don't keep the value around for any longer than it takes to create the string. Doing so is asking for trouble.

Anybody writing code that relies on being run in a particular timezone shall be taken outside and shot.
This should be obvious and strikes me as lenient, personally I would beat them with sticks until dead. Shooting's too good for them.

Anybody writing tests that rely on being run in a particular timezone shall be taken outside and beaten with sticks until they are sorry.
It may look like an easy and harmless shortcut but No, just No!

Never go near the TZ environment variable.
Oh TZ environment variable, how I hate thee? Let me count the ways:
- You are not thread safe
- You can be changed invisibly by other parts of the codebase.
- You are hard to clean up correctly.
- You do not even bloody work properly.
Yes, it turns out that Windows is, as you'd expect, expertly engineered to work in every location you can imagine except if you change the TZ environment variable, after which it will get the timezones right (but of course), but likely get the dates of change from standard to daylight saving and back again wrong. it will use the American dates, even when it is not in America.

Alter your timezone and check it again.
I have come to the belief that writing time-handling code in a timezone that coincides with UTC for half the year is an actual disadvantage allowing bugs to go unspotted. Now, whenever I go near it, I tell my dev box it is in Road Town.

anegada sunrise
The weather's better too.

Tuesday, 9 April 2013

Different Ways Of Thinking

An engineer's mind is focussed on objective reality, it needs to be, it is the medium in which he or she works. When an engineer tells you they will do something it means they intend to do it, and believe they will be able to do it.

A manager's mind is focussed on subjective reality, it needs to be, it is the medium in which she or he works. When a manager tells you they intend to do something it means they agree it would be good if it were done.

When a salesman tells you they intend to do something, they are just making noises they think will make you happy.

Monday, 18 March 2013

How Code Reviews Lead To Crappy Code

I have finally figured out what it is about code reviews I do not like.

There is a lot to like about code reviews, they extend knowledge, they allow developers with little experience in certain areas to work in those areas with the confidence that any dumb mistakes they make will be caught, and most importantly, they genuinely catch bugs - often very nasty subtle bugs at that.

At other shops I have done reviews of the 'cast your eyes over this before I press commit would ya' form, and these were low impact and high value. Here we are having a real quality drive in an (I think successful) attempt to pull a legacy product back from the brink of becoming a bug-ridden mess, and are doing long, involved, line-by-line dissections of every commit.

Here is what happens, like any department anywhere, some people are better at the job than others. Always fond of a massive generalisation I shall divide this entire gamut of human ability into two sorts of people which I shall refer to as senior devs and peons. Quickly, I find, the department falls into the state where peons do most of the work and the senior devs review it. Senior devs would love to do the work themselves, and would do it faster and better, and would use things like Patterns, and Modern Language Features, and all the wealth of experience and knowledge that has made them senior; but they are too busy with their reviews.

Ahha, you might say, but peons code slower, and so produce less work to review, so it should even out, right?

Would this were so, but peon work is often poor, or more often sort of okay but with subtle errors and missing some corner cases. It gets sent back, with constructive criticism carefully worded not to offend, and then returns, with a few different errors, and cycles backwards and forwards, and becomes a mix of intents, of the parts where the reviewer has put her foot down and explicitly dictated a design (sometimes I think I should just copy and paste a diff into the review comments) and, and here's the rub, parts where she has not had the heart, or the patience, to criticise any further, and has allowed the peon to muddle through in their own way.

I didn't bother picking up on the stinky bits in the first review you see, because at that point it didn't work at all and a bit of a pong was the least of its problems, and I kind of assumed they'd go when the thing was rewritten to, you know, work. But they did not, and now, on the third cycle of the whole depressing business, it seems cruel to pick up on them, and with heavy heart I am forced to let them through because, stinky though they may be, I'm just happy that they work.

And so you end up with code which, while it contains no known issues, is just not very good; it is fragile, and has bad encapsulation, and the tests are close to useless, and frankly, it sucks. Meanwhile it has actually taken longer than it would have taken the senior developer to do the damn thing themselves, and they would have done it better, and they are wondering if the department wouldn't be more productive without the peon in the first place. 

Friday, 21 September 2012

One Definition to Rule them all

Generally I am a little bit suspicious of people who can quote the C++ standard, knowing the exact definitions and all the obscure intricacies should only be necessary for compiler writers. The rest of us should get by with a broad understanding of how things work and the compiler should catch us when we stray.

In fact, and as an aside, I would actually consider too much knowledge of operator precedence to be positively harmful. People who know things can't help themselves but to use that knowledge and end up leaving out unnecessary braces in compound statements because it's obvious to them that they are unnecessary. Some idiot who hasn't memorised the rules (like me) then inherits the code, makes an incorrect assumption about what it's trying to do, and screws it up.

Bytes are cheap, just use brackets.

At least that was what I thought until I got bitten firmly in the arse by this little beauty:
http://stackoverflow.com/questions/6379422/c-multiple-classes-with-same-name
Which if I had been more familiar with standard I would have spotted immediately.

In my case I had a little interface I needed to mock for a couple of tests.

class ThingResponder
{
public: 
   virtual void respond(int) = 0;
};

In the first test I didn't care about it.

test1.cpp:

class MyThingResponder : public ThingResponder
{
public: 
   virtual void respond(int) {}
};

void test1::someTest()
{
    MyThingResponder responder;
    foo(responder);
    ...

But in the second test I wanted to verify it was called, so did something a little different

test2.cpp:

class MyThingResponder : public ThingResponder{
public: 
    std::vector<int> responces;
    virtual void respond(int responce)
    {
        responces.push.back(responce);
    }
};

void test2::someTest()
{
    MyThingResponder responder;
    foo(responder);
    ...

Can you guess what happened next.

I had already broken the one definition rule, this did not result in any build errors or warnings, instead when the calling frame wanted to allocate an object of MyThingResponder it used the local definition, but when it called the constructor it used the one from test2.cpp. This meant after I wrote test2, test1 started crashing in weird ways because the constructor for MyThingResponder was initialising a std::vector that wasn't there and screwing up a load of other local variables.

Had it been the other way around the responces vector would have never been initialised and it would have been even harder to track down the error.

These are the things that scared me.

  1. As we move to using more functors and the like, small locally defined classes with generic names will become more common.
  2. There is no way to guard against this. Anyone else's new class could break your code.
And these are the things I realised:
  1. I am going to put everything in a namespace from now on. Everything!
  2. I suppose I do kind-of have to know the obscure intricacies of the C++ standard after all.

Tuesday, 18 September 2012

Another pointless C vs C++ musing

I really wanted to agree with this blog post, I largely agreed with part1 and having once tried to integrate ZeroMQ with an application that was tied to an old version of gcc I really wished he'd written it in C too.

Ultimately I think it's a bit wrong-headed though. He seems to want to break encapsulation for a performance benefit (which is fine, so long you appreciate the trade-off and genuinely do need the speed) and for some reason thinks that's fine in C but not in C++. Personally I think ugly hard-to-maintain code is ugly and hard to maintain whatever language you write it in.

I have a pet theory that C will out-live C++. Managed languages will slowly intrude on the application space because the pain of developing in them is so much less, and the performance penalty will continue to get smaller until the one outweighs the other for all but a tiny subset of problems. Even the resource scarce embedded space, where you'd have thought C++ would have trounced, say, Java every time, seems to have been largely ceded, Android being prepared to take the hit of a JVM in return for stability and safety (how stable and safe is arguable, but certainly more stable and safe than running native code from unknown third parties).

Meanwhile C++, with horrors such as exceptions and (yuck!) template meta-programming, is never going to make much inroad into the system space. Operating systems, while they may employ more and more C++ components, are for the foreseeable future going to be written mostly in C. In fact it's clear from noises coming out of Microsoft that they'd like to do as much as possible in C# and are only stymied by performance issues; performance issues that will surely be resolved sooner or later, if only by faster hardware.

So to come full circle, one of the pain points of C++ that really irritates me, and does not seem to get talked about much, is people who insist on writing APIs in C++. In theory it's all nice, you simply compile everything up and off you go, and if they've used some later code feature then you update your compiler. In the real world it doesn't work that way. If I need to update my compiler I need to recompile about a dozen other open-source C++ libraries and go and find updated version of the two or three third party proprietary ones. All that functionality now needs to be re-tested, most of it by human beings, all at great cost.

For these reasons we rarely update compilers, and so the chances of a given C++ API not compiling on one of the (several) compilers we use is greatly increased. Greatly increasing the pain of using it.

On top of all that I can no longer dynamically load it at run time (or at least, not without jumping through more hoops than is worthwhile) which means it becomes an absolute dependency whether or not the user needs the functionality, and I either have to link it statically bloating the binary or leave myself vulnerable to version mismatches.

All of this makes me sad. I love C++; I like the philosophy behind it, believe that smart pointers trump any other form of garbage collection, that templates are great, and think that it is generally just unforgiving enough to force people into actually thinking about what their code is doing rather than muddle through by trial and error. But I think the lack of an ABI (is it even possible? I don't know) is becoming such a large pain point that it will eventually (though hopefully not in my professional lifetime) kill the language off.

Tuesday, 29 May 2012

The Unit Testing Catch 22

I am unit testing. Unit testing falls closer to the second of the two types of tests I enumerated here and is, I think everyone agrees, a 'good thing.'

The problem with unit testing is that, like everything short of full functional testing, it relies on domain knowledge. In the days when everything was specified in documents, about which architects had meetings and then bestowed like blessings upon the rest of us, this was not a big deal; the document described exactly what stimulus the unit would receive and if it was wrong it was not your problem. These days we are all agile and stuff, and there are no documents, and we are forced to rely on our own understanding of the wider system to know what stimulus our unit will receive, which in a large system is almost certainly incomplete.

The catch 22 here is that the majority of bugs are due to incomplete domain knowledge. I simply didn't realise that the system would, in some rare occasions, provide me with two notifications before actually doing anything. I assumed that when I received the second notification the first operation was complete. No amount of unit testing would have caught this bug because I would no more have tested it than I would have tested how it coped receiving references to deleted objects.

This is why I am suspicious of mocking frameworks. Mocking is useful, obviously, but it forces you to rely even more on domain knowledge, which is a point of weakness.

Monday, 9 January 2012

Solaris Memory Allocators

Is your multi-threaded application running like a dog on Solaris?

Mine was, and after weeks of head scratching I was reduced to suggesting that the client ran it on Linux instead, which was a mite embarrassing to say the least.

Fast forward a year or so and I discover the wonderful Libumem, or, because we have to support Sol8 the almost as wonderful mtmalloc (incidentally, Oracle need to up their SEO game, it took me ages to dig up that link), two multithread optimised memory allocators that speed things up a lot! Brilliantly all you need to do is set LD_PRELOAD and you're off, you don't even need to recompile anything.

Except that I am not working on that application any more. Now I am trying to make something else go faster. And this something else is old, and sits on the legacy comms layer.

Way back in the dim and distant past somebody who isn't here any more wrote a buffered interface for the comms layer. Now if you or I were implementing a data buffer we'd probably put it in a unit tested class, and be all agile about it, and probably use templates and a design pattern and stuff. We'd definitely keep at least two counters, one to track the size of the buffer, and one to track the size of the data in it.

Things weren't like that back then. Classes were just considered uppity structures, unit-testing was viewed as a dubious eccentricity, and templates were for Microsoft Word and Microsoft Word only. Our long forgotten developer didn't need two counters. He just made it so the buffer was always the size of the data in it and whenever that changed he realloc'd it.

No messin'!

The writers of mtmalloc and libumem were not like that, they were like you and me, and they did not worry about realloc because everyone these days uses c++y things like new and delete and there was no c++y way to do realloc. So when they implimented realloc they simply did a malloc, a memcopy, and a free.

Easy!

Except you know what happens next. I set my LD_PRELOAD and everything runs swimmingly, for a while, until it hits a certain data-rate where, suddenly, the comms buffering is being exercised and the application slows back down to a crawl. Even worse than it was before, and I have to open up the labyrinthine antediluvian comms code and take out all those reallocs.

Which was no fun.

Monday, 5 December 2011

If the test writer has to forsee the bug then it will not be caught.

I am stuck automating software tests. Not unit tests you understand, but functional tests. It is a very dull necessity.

We have a nice little scriptable home-brewed test-bed where we can enact user stimulus and run scripts, change files, etc. It's really quite nifty but, I think, ultimately of very limited use.

Why? Because there are two sorts of automatic tests. The first feeds in stimulus X and checks for result Y, the second feeds in stimulus X and checks that result Y was the only thing that happened. The problem with the first type is that the test writer is required to foresee the bug, which nobody in their right mind could expect him to do. The problem with the second is it is hard to automate at a functional level.

Needless to say...

Tuesday, 25 October 2011

Aladdin Sane's Cat And Other Tools

I call the above diagram Aladdin Sane's Cat and for a while this summer it was the bane of my life. What it depicts is per thread cpu usage, the orange line represents the main process thread and the other lines represent my worker threads. Everything starts off okay, the main thread starts feeding the workers, they are rather heavily contended but we expect them to be contended at startup and are prepared to live with it. Then something terrible happens, the main thread turns its attention to consuming the worker thread results and the workers run dry - it doesn't look so awful in the above case because after startup the application calms down but in other test cases it started to beat; main thread, worker threads, main threads, worker threads, etc. (those cases didn't look like animals so I didn't take a screenshot).

The point of all this is not how to fix this one rather obscure problem (it took ages!) or even to go Hey, I drew a cat! (though don't think for a moment that I'm beyond writing a post just for that) but to demonstrate how incredibly useful something as simple as a chart of per-thread cpu usage can be. Without that chart I not only wouldn't have been able to fix the problem; I wouldn't even have known there was a problem.

In the above case the chart was drawn by our own software, although Excel would have been as good, and the numbers were generated from a script I wrote (and have sadly lost) that simply polled /proc/<pid>/task/<tid>/stat.

Looking back, this was a so simple to do, and so incredibly useful*, that I'm amazed I went so long before I thought of doing it. With that in mind I thought I'd produce a quick rundown of other tools I found useful developing multithreaded applications:

  1. Per thread CPU usage. On linux you can get it from the proc directory or using 'top -H' (although top is hard to script). On solaris from 'prstat -L' (which is easier to script). On Windows I have no idea but there must be ways. This is invaluable, as I hope I've demonstrated, especially once you chart it over time.
  2. Talking of prstat, on solaris you have the 'prstat -m' option for microstate accounting which is very funky indeed. This will give you a column, LCK, for the time spent waiting for userspace locks. You have to be aware that this includes time waiting in pthread_cond_wait but it is still useful.
  3. Valgrind. Both helgrind (or drd if you prefer) can be invaluable, particularly picking up lock ordering violations. The problem with both of these is that they run like dogs so you can't test the program under normal load and it takes forever.
  4. Sun Studio collect and er_print tools are really quite nifty once you get the hang of them. They will give you lock contention time by function and optionally filtered by thread.
  5. VTune, if you can persuade your boss to buy it (like I did, woo!) is surely the bee's knees among profilers and is just chock full of bells and whistles. For instance it is a doddle to identify contention on a per lock, per thread, per call-stack basis, which - when your wizz-bang new multi-threaded program is not scaling like it should, and your employer is breathing down your neck asking why - is a bloody godsend.
There are other tools. dtrace plockstat sounds fantastic but I could never get it to work. AMD CodeAnalyst is supposed to be great and is free but is only really useful on AMD hardware, upon which I was not. No doubt there are plenty more.


* It is a testament to both the usefulness of the script and the laziness of its author that even after I accidentally deleted it I left a copy running for nearly two months and changed the name of every process I wanted to examine in order that it would pick them up. Sadly after two months someone rebooted the server. It was a good script, and I miss it still, though so far not enough to rewrite it.

Wednesday, 28 September 2011

Sparc's Overlapping Register Windows

One of the nifty* features in Sparc chips is that the register file is in fact just a window onto a larger circular buffer. In Sparc you have 32 registers, eight of these are globals and do not interest us, of the remaining 24 eight are shared with the calling stack frame, and another eight will be shared with any called stack frame. There is a good explanation here: http://www.sics.se/~psm/sparcstack.html

All this should in theory make calling functions blindingly fast (not that functions calls are especially slow) so long as you don't pass more than 8 registers worth of arguments (which is plenty for most people). However, as the above link points out, whenever another thread wants the core you have to flush the whole lot to memory, whether it was in use or not (I suppose you could track and only flush the used registers but it probably wouldn't win you anything).

It occurred to me, while working on a piece of code some 40 odd stack frames below the main run loop, that function calls would get more expensive once you ran out of buffer, so I wrote a little benchmark to test it.

The benchmark is fairly simple, it calls the following function a few thousand times and you can set the initial first argument to whatever you want.

int recursive(int nLevel, int val1, int val2)
{
        if (0 == nLevel)
                return val1 + val2;
        else
                return recursive(nLevel-1, val2, (int)&val1);
}

(The casting of the pointer of val1 was the easiest way I found to stop the optimiser inlining the whole damn thing, bit clever that optimiser!)

I ran it with a selection of recursion levels, optimised and unoptimised, on Solaris x86 and Sparc boxes.




As you can see the cost of function calls after about 7 suddenly leaps up on the Sparc box but stays constant on the x86 box (I adjusted the number of iterations on the two boxes to normalise the figures so they aren't really directly comparable, my x86 box was a lot faster than my Sparc box). I had initially wondered if I would see a jump every 7 or 8 frames as the whole file was dumped to disk but in retrospect that would have been a stupid way to implement it and obviously only one frame is flushed at a time.

Incidentally the graph stays the same no matter how many initial frames you are down when you start the benchmark, proving that frames aren't loaded back in until they are needed. This is fairly obvious when you think about it but I checked anyway.

The other interesting thing to note is how much faster the first seven frames are on the optimised Sparc calls, after which it's about the same cost as the unoptimised version. The only optimisation to the function call was to replacing a tortuous series or dependent ORs with a single MOV:

Vanilla:
0x00010fbc: recursive+0x002c:   add      %l2, -1, %o0
0x00010fc0: recursive+0x0030:   add      %fp, 72, %o2
0x00010fc4: recursive+0x0034:   call     recursive      ! 0x10f90
0x00010fc8: recursive+0x0038:   or       %i2, %g0, %o1
0x00010fcc: recursive+0x003c:   or       %o0, %g0, %l7
0x00010fd0: recursive+0x0040:   or       %l7, %g0, %i0
0x00010fd4: recursive+0x0044:   ret      

Optimised:

0x00010f58: recursive+0x001c:   add      %i0, -1, %o0
0x00010f5c: recursive+0x0020:   add      %fp, 72, %o2
0x00010f60: recursive+0x0024:   call     recursive      ! 0x10f3c
0x00010f64: recursive+0x0028:   mov      %i2, %o1
0x00010f68: recursive+0x002c:   ret  


(there were other changes before the conditional jump that will also account for some speedup too)

The fact that this improvement is more or less invisible once we start flushing the buffer goes to show just how much the flush costs.

I have no idea why the x86 lines wobble so much.

The conclusion then, is to try and avoid regularly bouncing above seven stack frames high if you are targeting Sparc machines. Unfortunately with my 40+ frames there was little I could do about that.

* Well I think it's nifty. I've read criticisms that the feature makes Sparc chips hard to lay out and is the main reason they have never reached the clock speeds of other architectures.**


** That said, in my experience hardware engineers are always moaning about some feature making layout difficult and if they ever moan enough to make the architect compromise they just seamlessly move on to moaning about something else, so I still think it's nifty.

Friday, 16 September 2011

'Soft Deadlock'

Came across a funny situation with a standard worker pool pattern. When the workers had no work to do they basically did this:

1. Lock mutex.
2. Do a cond_timedwait waiting for work. (this has it's own mutex, not the one from above)
3. Unlock mutex.
4. Goto 1. Do not sleep, pass go, collect £200, anything ...

All of which worked fine until I ran it on a single core linux box, at which point a second thread waiting for the lock would occasionally never get it, it would wait, and wait, and never receive the lock, while the worker thread span happily around releasing and acquiring it,a sort of 'soft deadlock' - if you like?

Once I figured out what was going on I stripped it down to a test case and discovered it was happening about half the time, a little more if optimised.

Some weird scheduler problem then. I was under the impression that a thread should yield when it releases a contended mutex but a kernel hacker friend disabused me of that notion, it can do what it likes. Whether or not my waiting thread ever gets the lock is entirely down to luck.

The mutex here is protecting the work sources (including the cond var that the worker is waiting on) which are dynamic and can come and go. The proper solution then is to create some other static secondary work source that is checked between steps 3 and 4 and onto which I can post changes to the primary work sources, but then I would also have to create some response mechanism so that the worker pool knew when it was done and was safe to do the delete - all of which adds complexity and risk to an already complex and risk prone area of the code; all for a code path that will happen very rarely, in fact hardly ever.

Instead I did something truly abhorrent with a volatile flag and a call to sched_yield which I'm not proud of but is unobtrusive and works.

Anyway, an interesting gotcha. Or at least I thought it was interesting.