SandyWalsh.com

Python Halifax!

2011-11-08T11:28:00.002-04:00

A one-day Python gathering here in Halifax? Say it ain't so?!

It's true ... register now!

http://pythonhalifax.org/

The Pain of Unit Tests and Dynamically Typed Languages

2011-08-30T22:44:00.000-03:00

Don't get me wrong. I love me some Python and I love me some unit tests. The problem is, I find strict unit tests in dynamically typed languages aren't nearly as useful as they are in statically typed languages.

I can hopefully better illustrate this with an example. Let's say we have some code like below. Note: Don't worry about what's actually happening (I completely made it up), but just be aware that decide() calls on compute() and both have conditionals/exceptions/loops, etc which make them good candidates for unit testing … in that, they could easily bust our application if someone starts messing with either of them.

Also note that decide() is dependent on a bunch of other functions: get_seed(), get_first()/second()/third() … which we will assume are equally unit tested.

So, we write our unit tests and they're really good. They mess with the fence post cases of compute(). They make get_whatever() exception out to test the retry code. All of this is done with Mocks and Fakes so we're not dependent on the other code. These are unit tests after all; we shouldn't be calling external functions/methods. This means that in our decide() unit tests, we've mocked out compute(). We green bar our test suites and all is good.

Until someone changes compute() to something like this:

Now it's some weird function that takes a boolean (or something that can be coerced into a boolean) and returns a string. Fortunately the author fixed the unit tests for compute() and everything green bars again.

But the application no longer works. decide() is broken … and we won't discover it until decide() gets called.

Why? Because with most dynamically typed languages, the contracts between functions or methods are decided at runtime. When compute() is called by decide() the virtual machine will try to coerce the values for first and second as best as it can. Likewise, when the response from compute() comes back to decide() the virtual machine will try to apply the +100 to it the best way it can. Maybe it will work, maybe it won't. Almost certainly it won't be what the developers intended.

If we are using a statically typed language such as C/C++/Java/C# things are a little easier. Our function would probably look something like

So, when compute() changes to

… the compiler can catch it and complain long before the unit tests are ever run.

What does this mean for us Python/Ruby/PHP developers? It means we have to start moving towards integration tests in addition to our unit tests. Unit tests alone are not enough to let us sleep at night.

As I've said before, integration tests, while great, are a royal pain in the buttocks. They are fragile and difficult to maintain. Do we need to go full-on end-to-end integration testing? No. A normal unit test has a call-out depth of zero. We only test the function in question. But, what we can do is start to write some 1-depth unit tests (or baby integration tests, if you prefer). A 1-depth unit test would allow decide() to call compute() (and get_first/second/third/seed, etc) but no further. All calls beyond the 1-function call depth would be stubbed out as normal.

How do we decide where to place our 1-depth unit tests? I find that file or module boundaries are good places; places where it's not really easy to scan with your eyes. I will also skip third party libraries. I'm really just worried about the code I can immediately control. Or perhaps you might want to write them between highly dependent and equally complex functions?

Do you need to include every external call? Probably not. Make it a judgment call. Is the return type from a call sufficiently complex that it's likely to change? That's a great place to push your call depth down a level. Does one of your functions parameters have a dictionary of assorted types? If so, that sounds suitably fragile. Use common sense, there's no one-size-fits-all rule to this.

But, I would strongly encourage you to place these tests in a separate directory away from your strict unit tests. Developers should know what they're getting into when they open those files.

I look forward to hearing your thoughts on this.

Effective Units Tests and Integration Tests

2011-06-30T11:14:00.002-03:00

A unit is the smallest testable part of an application.

More often than not, I see unit tests that forget this core tenet. Picture a method or function as a sandbox. When you are writing a unit test, never let a call get outside of your sandbox.

You unit test should be testing the conditional logic and correctness of the code in your sandbox. Let another test worry about all that external/dependent code. When writing a suite of unit tests we're interested in code coverage. We want to exercise as much of our code base as possible. That's a lot of tests. We need to make sure these tests are not fragile, difficult to write or hard to maintain. And we need to make sure they're quick to execute.

The code in your sandbox really only has one or two purposes:

To execute some external operation, and/or
To compute some result

Your unit test should be testing all possible code paths through your sandbox. What can cause your code path to change?

Each of these constructs can make one pass through your code different from the next one. That's the stuff your unit tests should be exercising.

Computing a result includes:

Checking return values
Checking thrown exceptions
Checking set member variables and/or globals

Your unit test should be throwing all kinds of input at your code to ensure the return values are correct. Also that your code throws the appropriate exceptions on bad input. If your code doesn't return any values you may need to check that any global or member variables were correctly set.

How to stay in your sandbox
Nearly all programming languages have some sort of Fake, Mock or Stub library available for it. The purpose of these libraries is to short-circuit the external code and replace it with your test code. Something that:

Always returns an expected value
Has no state
Has no external state dependencies (i.e. a database initialized to known values)
Returns nearly instantly
Has no conditional logic in it

In "XUnit Test Patterns", Gerard Meszaros has a nice breakdown of Fakes, Mocks, Stubs and Dummies:

Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).
Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it 'sent', or maybe only how many messages it 'sent'.
Mocks are objects pre-programmed with expectations which form a specification of the calls they are expected to receive.

I try to only use Dummies and Mock objects for unit tests. I don't like storing state in my test doubles ... after a while you spend so much time making an effective stub you don't know if you're testing the stub or the code in the sandbox. That's why I like mock objects. They usually just consist of "return value" or "raise exception" ... simple, maintainable and readable.

If you find yourself putting an If statement in a Mock, you're doing something wrong.

How can you verify an If-Then statement?
Most of the time you just want to be sure that an external method or function was actually called. Do you need to actually call that method? Hell no. Not in a unit test. Many of these Mock libraries also have provisions for "Was Called" semantics. Essentially a flag is set if a Mock method was called. If your library doesn't have one, it's pretty darn simple to simple to set one up:

So, rather than let your code call outside of your sandbox, replace the external method (via monkey-patching, dependency injection or some other mechanism) to use your mock instead.

Does all code need a Unit Test?
No. Consider the following gist:

What would a unit test accomplish here? Nothing. Just make sure you have tests around each of the methods within foo().

When Unit Tests aren't sufficient
Let's say we did write a unit test for foo() in the above example. Would it accomplish anything? No, because we aren't checking the semantic ordering of the calls within foo(). Putting logic in to ensure that do_this() was called before do_that() and do_something_else() would just be a waste of time.

What if foo() was changed to this:

Our unit test would still pass just fine. But it would be wrong. What we need here is an Integration Test. Integration tests do check the semantic ordering of statements. With an integration test we allow ourselves to step outside of our sandbox. We may even use test doubles to help us build our integration tests (including Fakes and Stubs).

But now our mandate for these tests have changed: No longer are we concerned with code coverage, but instead we're interested in testing specific usage scenarios that are critical to our customers acceptance of the software. Other than manual testing, integration testing is the only way to ensure we're building software that will work for the customer.

We don't need 100% code coverage, instead we want the 80% of the every day scenarios the user will be experiencing when they use our software. If we were making a word processor, do we need integration tests for mail merge, form builder and the math editor? Probably not. Sure, it would be nice to have, but it's not critical. What we absolutely need are integration tests for: launch, enter some text, setting basic styles like bold, italics, etc, printing, saving and loading, page flow, etc.

Do we need to do integration testing for every edge case where an error might occur? No (again, it would be nice to have) ... but realistically we need to test: disk full, disk failed, out of memory, out of paper, etc. The most common scenarios.

But, if integration testing is so good and so important, why not just do integration testing?

Because integration testing is hard, slow, brittle, hard to read and hard to maintain. Everything unit tests aren't. In other words ... they're a pain in the ass. Integration testing is sometimes frowned upon because of these limitations, but they're being compared to unit tests. Integration tests and unit tests are very different animals. They serve two different purposes and give two different levels of comfort to the developer. I don't even think they're comparable at all. The worst mistake a development team can make (regarding testing) is to mix and match their integration tests with their unit tests. You get the worst of both worlds. Keep them clearly separate.

Also, treat your integration tests like your core code base. Unit tests can be hacky since they're so small. But integration tests are complicated and need to be carefully maintained. They need to be documented properly. They need fantastic logging capabilities with rich output. They need to follow the same coding styles as your core code base. They need to be structured, refactored and updated so that they're always easily readable. Unit tests are rarely refactored, since they're so small and atomic, there's usually nothing to change.

So, to grow an effective body of Unit and Integration tests for your application, remember these rules:

Don't step outside your sandbox for unit tests
Use Mocks and Dummies for your unit tests
Check all branching logic in your unit tests
Go for very high code coverage for your unit tests
Integration tests are difficult beasts. Go for high-impact user stories in your integration testing. Mostly Happy Day scenarios.
Don't spend a lot of time on the edge-case failure conditions in your integration tests
Keep your integration test code clean and maintainable. Refactor as frequently as your core code base.
Set up a dedicated integration test server that runs on every commit to trunk (they're slow and hard to set up remember)

Technical Rabbit Holes …

2011-06-22T20:48:00.000-03:00

Last time we had a good discussion about iterations and how there's a potential for dead-air and confusion near the end of a time-box. Tricky problem with lots of nuances. But there's another place where we can get fooled by the perceived safety of time boxes and iterations. Complex code.

Most interesting software is complex. You're not really treading the high-wire when you're cookie cutting variations on the same app over and over again. The fun stuff comes in when you're solving new or hard problems.

The issue with difficult problems is it's easy to go down technical rabbit holes. To me, technical rabbit holes are when there's a gap between the what the developer is working on and the understanding of it by the customer. It's not that developer is doing something dishonest and obscuring their work. It's not that the customer is unable to understand the problem. The problem is trust. There's too much of it. The developer has just enough rope to potentially hang his part of the project and the customer is willing to let it happen.

And, I know I like to pick on Scrum, but this is not Agile's fault. Agile does a great job of forcing small units of work to be produced and highlights customer interaction and feedback. Agile solved the old problem of “Going Dark” as Steve McConnell correctly describes it: The customer anxiously waiting at the end of a pipe for a product to drop out.

So, then, how can a technical rabbit hole occur?

Well, we know what a good project should look like. We may go off course a little from iteration to iteration, but the customer is there to keep us on track and reach the goal.

When there is no clearly accessible customer or a poor customer proxy we know it's easy to go off course for real. We keep hitting our iterations, we keep delivering code, but it's not the right code (it's not solving the customers problem.)

Dev shops often forget that there are two aspects of writing software: Research and Development. Too often these functions are rolled together into just “Development” and sadly the total becomes less than the sum of the parts. The customer needs to understand whether the developer is doing R or D.

In the above illustration, the Y axis is the customers understanding of what the developer is doing. Green = “I see what's happening.” Red = “I don't really understand, but you say it's important.” You can see that for nearly two iterations, the customer wasn't really comfortable in what the developer was doing. Two iterations. Probably in the best case, that's four weeks of time. That's a long time.

Can you see how this could go bad?

How many iterations do you have to go before the customer finally says “Stop!”?

And, again, it's not that the developer is doing something bad. He's not being tricky or deceitful. He's mostly likely really chasing down some complex code. The problem is he's doing in a manner that abandons the customer. Agile development means developing code together with the customer. It's not Us vs. Them. The customer put too much trust in the developer.

Many times these things work out. But there are times when they don't, when the rabbit hole goes too deep and the developer gets lost. Now we've just spent a lot of time and money and we aren't any closer to the goal for it.

What can we do to identify and prevent technical rabbit holes?

Get a Second Opinion

As with any possible fatal disease, you should get a second opinion. Bring in another developer or two to bounce your approach off. Go deep. Talk about all the nitty gritty issues you foresee. Make them own the problem as much as you do. Ask questions, do they understand your vision? Do they understand your development plan? Do they have any concerns about your approach?

It's these people that are going to help you stay in the light. Perhaps they can articulate your plan to the customer better than you can? Perhaps they can see other ways to deliver on your vision without requiring the customer to put blind faith in you?

Never swim alone in deep water.

Incremental Completeness

Technical rabbit holes can easily occur in any project with some degree of technical complexity. What we should be striving for is Incremental Completeness. Delivering small chunks of working code (with tests) to trunk that, not only help the developer with the R part of Research and Development but, show the customer that we understand where the destination is.

Every iteration, try to put something in the code drop that the customer can see the results of. Perhaps even a simple command line tool the user can run on the release to play with the feature in some fashion (the customer is getting a working build every iteration, right?)

Which gets us to our next point ...

Educate the Customer

Don't assume the customer knows anything about what you're building. They may be able to spout off all kinds of technical mumbo jumbo, but is it code? If they don't understand the piece your building, how can they expect to confirm they have it at the end of the iteration?

Use analogies for the data structures you're building. Try to put it in terms they'll understand. Draw pictures. Share the whiteboard. Explain the complexity.

You may not get it at the first sitting and don't try to force it in a single go. You're telling them a story. Put it in their terms. You're bringing them on a journey, take your time. Just make sure they're not just nodding mindlessly at you.

You're not trying to turn them into computer scientists, but you want to get to the point where the customer can call YAGNI. You want them to tighten the requirements to the point where they can say “we're not going to hit that condition frequently enough … so just print an error message.”

Always Commit

This probably goes without saying, assuming you're using a modern day revision control system, but you have to commit and push your code. Do it frequently. Ideally, no later than every 3 days. Get others to look at your branch. Keep it in the light. Let other developers see what you've been up to. Tell them the same story you told the customer … what do they think? Do they agree that your description is a fair explanation of what you're actually doing? Do they think you're on course?

Don't lose sight of trunk

Merge. Merge. Merge. Pull from trunk frequently . Keep those tests green. If you're doing something that is going to break trunk, use a scaffolding technique: build around the existing mechanism and provide a means to turn it on and off. Later, once your implementation is complete, you can remove the dead code. But, until then, you still have working code. Each iteration you should be pushing something working (with tests) back into trunk.

Letting trunk get a way from you is a sure way to get lost in a technical rabbit hole. If you're adding new code when your tests don't run, you're in trouble. Do you remember what you changed that broke the tests? Are you getting closer or further away from them working again?

As a developer, it's probably your greatest risk.

Mr. Customer: Share the vision

This one is for the customer ... it's ok to sound like a broken record. Keep talking about what the final product will look like. Give lots of detail. Make sure everyone knows where they are going. Make sure everyone knows what's important and what's not. Help the development team understand your business, who your customers are, what they want to experience from this software.

If you keep everyone focused on the goal (and the deadline), it's harder to wander off course. You want the developer to say “You know, I'd like to do this differently, but there won't be time right now. Let me find an intermediate step.”

TL;DR

Technical Rabbit-Holes can be prevented with Eyeballs and Education. Show people what your doing and explain how you're doing it. Don't let too much trust get the better of you.

Good luck … and keep your head above ground!

Iterations and Time-boxing are (Mostly) Useless

2011-04-28T19:23:00.001-03:00

Sorry, this is a rather long post, but I've been thinking a lot about Scrum and iterations lately and haven't had time to blast this out in smaller chunks.

If you know me, you know I'm not a big fan of Scrum. I don't like the hand-waving concept of Velocity and I don't like the fact that it places a priority on process over code. XP, I feel, is a much more important for software development. Agile (aka customer responsiveness) can be achieved without the dogma. But that's not the point of this post. In this post I want to question the value of timeboxing and the benefits that modern revision control systems grant developers. So, stick with it, I spend some time setting it up before getting into the meat of the post.

Back in the 90's I was using Spiral Development Model on a project. Spiral is Waterfall with shorter, fixed-length, time spans between milestones. In XP, this idea was formalized as Iterations by dropping the Design phase and using automated testing to replace the long, drawn-out, testing phase.

The rationale behind iterations and Spiral was simple: Estimating is hard. The further out you look, the greater your error rate.

Spiral's insight was to reduce risk by keeping the milestones close together (and ensuring that each milestone is a shippable product).

Scrum expanded on this idea and permitted the functionality of the project to change from iteration to iteration. It does this by bringing the customer back into the picture each iteration for sign off and review of “what's most important now”. All good stuff.

At the start of the project, the customer has a general idea of what they want from the development effort. Working with the team, these requirements manifest themselves as Stories (aka Requirements ... friggin' Scum wants to rename everything)

Now the developers enter the picture and give some guesses about how difficult each story is. This is usually done as T-Shirt Sizes (Small, Medium, Large) or some other metric. In a perfect world, these sizings are not equated to Time ... they are relative efforts and no planning occurs more than one iteration out. But in reality, the customer needs to have some idea for when to expect their product. It could be as coarse a resolution as 1^st Half of Next Year or 4^th Calendar Quarter, etc.

So ... we're back to using Time again. T-Shirt sizes get equated to some unit of time (let's say, Small = 1 day, Medium = 2 days, Large = 3 days). Iteration sizes are set into something manageable such as 2-3 weeks each.

Finally, given the number of developers it's a pretty easy exercise to give a Wild Assed Guess (WAG) as to when the product will ship. If everyone is happy, the dance can begin with the first iteration.

When the developers get into the first iteration they start coding on the most important stories (as selected by the customer). Things are wonderful. But, when they get into the second iteration some things may have changed. The priority of the stories may have changed, the first iteration dev effort may have uncovered new stories or perhaps, there is already technical debt that needs to be addressed. Regardless, the original WAG deadline is likely changing (for better or worse).

The reality is that time-boxing the iteration introduces waste near the deadline. Let's say I'm 2 weeks into a 3 week iteration and I've finished all my work early, what should I do? Should I start working on another story from the next iteration (which includes bug fixes)? Should I perform some other non-development task such as manual testing or documentation? Should I refactor?

Surprisingly, the Source Revision Control System (RCS) that you are using can have a significant impact on your decision. Personally, I think Distributed Revision Control Systems (DRCS) such as git, bazaar or mercurial are the most significant change in software engineering within that last 10 years. Not because having a non-central repository is so revolutionary, but because branching and merging has become such a low-cost operation that the speed of software development has increased dramatically. Development shops that utilize these tools can see some big improvements in how they deal with the “Rough Edge” at the end of the sprint.

Let's compare dev shops that use a RCS that supports fast/low-cost branching and merging to those at don't.

As I mentioned above, having rigid deadlines means potentially having dead time on your hands.

You either have to find small, low-risk stories that you can take on within the remaining time or do busy-work to fill the gaps. If your customer is more willing to let the deadline slip to get the functionality they desire, things aren't much better. You're simply moving the dead time out a little further.

But if you're using a RCS that has low cost branches and good merge capabilities, things get much easier because the developers can keep working on upcoming stories without affecting the current iteration.

“But I can branch and merge with svn or ” I hear you say. Yes, you can, but not quickly. When keeping in sync with trunk is expensive, it doesn't happen. When you don't sync with trunk the costs of your merges increase, merges are larger and your unit test maintenance efforts increase. When branching and merging costs are low, it's easy for developers to start and new branch and keep working without upsetting trunk or the deliverable for the current iteration. Look at the NVIE branching model as your reference standard.

So, the million dollar question ... if I can keep working and still ship my promised set of stories as they become available, why time box in the first place?

Pull models such as Kanban don't really rely on timeboxing. Instead, developers pull stories from the backlog when they need to. The customer can re-prioritize the backlog as they desire. Finally, when a feature is completed it's merged into trunk and becomes part of the deliverable. The customer can decide which version of trunk they want to use and don't need to wait for an N week deadline to pass before grabbing the product. Developers are encouraged to keep the Work In Progress (WIP) to a minimum to prevent having lots of half-finished efforts.

We, as developers, can still give estimates on each feature and we can still do our burn up/down charts to track progress, but we don't need to actually block out time frames. Actually what we're doing is taking the idea of iterations to the logical extreme: each story is an iteration.

I can think of only one situation where time boxing is useful and that's in Scrum of Scrums. When the senior managers/customers have to monitor more than one team of developers they need to be able to snapshot the state of progress across teams. The ideal situation however, would be a Scrum of Kanbans:

Senior stakeholders get a top-down view of the state of the development process without slowing down the developers every 2-3 weeks. For them, this is a simple synchronization mechanism for keeping the cats herded.

This gets us back to one last point on estimating. As I've said before, estimating sucks. Developers hate having to break down tasks to such a fine resolution that they can be tracked in 4-8 hr intervals. If I have to think about a problem to 4hr resolution I may as well just code it. Why not 1 hour? Why not task out to every 30 minutes? Simple, it's a case of diminishing returns. The time it takes to document the tasks outweighs the effort itself.

I think a better approach is to not to track tasks at high resolution. Instead keep things in the 1-3 day range with a clear deliverable and simply track the sentiment of the developer towards the story. On the first day of the iteration, the developer should be very positive that they can deliver this story in this iteration (otherwise, why did they sign up for it?) ... but as the iteration progresses their sentiment may change. And it's going to North or South very quickly. “Oh, that's not good”, “This is getting bad.”, “I'm screwed”.

This is what we should be getting from our Standup Meetings. When we report status and obvious impediments to our peers we should also be giving a vibe on our ability to deliver.

This gets ever better for the Senior Managers, not only can they see their Scrum of Kanban stories getting executed, but they can boil up the sentiment from each of the teams to see how the iteration is going.

What do you think? Would your daily development process be better if you didn't have to break down tasks to super-fine resolution? As a manager, could you better estimate ability-to-deliver based on higher-level sentiment vs. tasks completed?

UPDATE: Glen Campbell was nice enough to give his perspective on this post. Jump over there to read his post and the comments to see my response.

WebOS: Horizontally and Vertically centering an image.

2011-03-01T00:32:00.000-04:00

Ok, this shouldn't have been too difficult. I just wanted to horizontally and vertically center an image using HTML5/CSS3 for WebOS. There's lots of ways to do this, but most solutions have fixed coordinates and wouldn't work well moving to different resolutions. Turns out it was a little tricker than I thought.

Here's the proper way (colored borders added so you can see what's what):

Palm Pre 2 - First Impressions from an iPhone user

2011-02-27T15:41:00.006-04:00

I've been an iPhone user for about 3yrs now. And, while it's a great phone, I've never developed any software for it. Why? Well, I didn't really want to learn Obj-C (no good reason), I'm an Ubuntu user and I could never justify the Mac hardware prices. I'd prefer to stick with Python or HTML/CSS/JS. Yes, I could do C/C++ and have support on nearly all the platforms, but I think HTML5/CSS3/JS is a more versatile cross platform approach.

I've been really riding the fence on Android, but I really couldn't get myself excited about Java development again. It would take all the fun out of it.

Anyway ... HP WebOS caught my eye. I noticed Palm has a $200 rebate for app developers to go towards a Palm Pre 2. But it's only for the US. I pleaded my case and HP was very quick to get me into the program. Naturally I had to prove I could find my way around the emulator, build tools, SDK, PDK and app store registration stuff. These were all easy to do (certainly better than another not-so-well-run developer program). No hidden costs and, since I already have a registered company, signing up for the app store was simple.

It's only been a couple of days and I'm still getting into it. The tools are nice and development is pretty easy. Apparently a Linux emulator is going to be available soon, so I run in a VM image for now. The tutorials are sparse and the videos are arduous, but it's better than a Readme file. I've hit a couple of snags with some trivial things; fortunately the community is supportive, as is the company. I'm looking forward to getting into a little WebOS development now ... and not just because of the incentives. WebOS is actually really cool.

As an iPhone user I was really concerned how the touch interface would compare. There are some definite quirks, but overall it's an easy transition. The Pre products have a "Gesture Area" where the iPhone home button is. You can do some basic swipes in the gesture area to go backwards, etc. But the real strength of the gesture area is in managing your multi-threaded apps. WebOS has a really cool card management UI metaphor that works nicely over all applications. Since the apps are mostly in HTML managing your apps and managing your websites have all the same look and feel to it. Creating a new browser page is just creating a new card in the stack. That page can sit next to your other apps and you can create stacks of these cards around function, not application. It just feels right.

WebOS has a "launcher" which does feel a little clunky though. I prefer to have the icons on the main screen instead of a separate operation. Instead WebOS has a "Just Type" area. You can start entering an email, sms, tweet, etc while the thought is fresh and then say "send this as an email", "tweet this" or "SMS to ..." It's going to take some getting used to for me.

Notifications are handled better on WebOS than the iPhone, but I do miss the "going dim" warning on the iPhone that says the screen is about to shut off. I think I also miss push notifications, but they were always a love/hate thing anyway. And, I'm of two minds on the slide out keyboard. It's a little small for me. Perhaps the Pre 3 would be better? I'm also keen to see more of the stuff WebOS is doing for sharing data between the devices like the HP Tablet with the "tap" metaphor. I will miss Beluga Messenger, my IRC client, TweetDeck and InstaPaper. Skype is coming soon though and Flash is supported.

I was easily able to get my calendars and email synced since they're all on Google. But now I've got to cut the tether on iTunes for my music. I'm not too worried. The Pre seems very open. It has a removable battery, uses a standard USB cable and can act as a fully operational USB drive when plugged in. No magic. And the app store isn't a military lock down zone. So I think making the move will be easy.

Naturally, the app store is a little bare. But hopefully other developers will appreciate this platform and jump in to fill the void. I'll let you know when I get my first app in the app store.

15 Problems with Old School Technical Support

2011-02-12T14:01:00.000-04:00

Last time I checked it was 2011. So why is it I have to deal with these 1990-era call centers when I need customer support? ISP's, cell phone providers, telcos, large retail organizations … it's the same thing every time. I'll just refer to them as BigCo.

You know the flow. Something goes wrong with a product or service and you have to contact the company.

If you can, you go to their website and look for the contact page. Otherwise it's the only time you dig out the Yellow Pages.

Problem 1: The customer support contact page is either so well hidden or written in such a confusing manner you have no idea how to reach the company.

More often than not a phone number is not even available. If you are stuck with leaving an email (assuming you have Internet connectivity) you have no idea when you may hear back or even if the email was received. So, to play it safe, you phone them. No way they can lose a voice call is there?

You call the number and you're greeted with a lovely … robot. Those damn Interactive Voice Response (IVR) systems.

Instantly you added a few minutes to your service effort. If you're like me, you hit 0 immediately to get an operator. The bottom feeders of the customer support business will disable the 0 button and simply repeat the message. Assuming you have to follow the menu, you hang tight while your mind wanders and you miss the fact that option 4 of 7 was the one you needed. God forbid you press the wrong number and need to go backwards. Was it # or * to get the menu again? Then the menu goes 3 levels deep and ultimately doesn't offer anything that helps your cause. Or worse, you get a canned message with no hints of what to do next.

Problem 2: IVR systems were invented in the 70's and have not evolved since 1997. Everyone hates them. BigCo disables the “0 for operator function” or does not permit “type ahead” and force you to listen to the entire message.

Side note, there is only one thing worse than IVR … voice recognition systems. The thinking is that it's more “conversational” and less robotic. It sucks. Often, it doesn't work and you have a long delay while the system times out and you get the annoying “Sorry I didn't understand that. ”. Even when it does work it only goes so many levels deep and you are forced to go back to touch-tone options regardless. Why bother? Perhaps it's intended for old people with rotary phones. In which case, wouldn't a human be the best approach?

Problem 2a: Voice Recognition Systems don't work.

Let's assume you get an operator. More than likely your dealing with a BigCo call center. Unless you've been living on desert island with no phone service for the last 20 years, you know that call centers are caves deep underground that keep slaves (aka Customer Service Agents or CSA's) chained to small rocks (called 'cubicles') and force them to answer phones.

The actions these slaves perform are timed to within four decimal places of accuracy. They are told when they can sit, stand, use the bathroom and eat. Most importantly though, they are told what to say and are not allowed to think for themselves. We should all feel for these poor souls, they take all of our crap when the real problem is higher up.

The things they are allowed to say are dictated in Scripts called “Policy Manuals”. Deviating from the Policy Manual is grounds for keel-hauling.

Problem 3: Call centers do not encourage individuality or self-thought. But instead they encourage compliance and uniformity (think fast-food joints). This is really too bad since the real task, servicing the customer, can often be accomplished very quickly if the agent is permitted to think and act outside the box.

Maybe your problem was part of the 70% of problems that can be solved by a simple phone call. Your irritation was minimal. You lost a few minutes navigating the system, but you write it off as the cost of doing business. Life goes on.

If you're in the 10% of problems that are not easily addressed by the Policy Manuals your journey is a little more complicated. You were likely transferred to 1, 2 or 3 other agents. Probably your call was dropped during the transfer. Maybe you had to call back in and repeat the whole process.

Problem 4: The Call Center does Blind Transfers. They forward a call to another agent or department and do no stay on the line with all parties until the transfer has been conducted successfully.

Problem 5: No trouble ticket. At the start of your call you should be given a trouble ticket number and a priority callback number. In the event you get cut off or dropped you should be able to circumvent all the conventional mechanisms and get back the agent you were dealing with immediately.

Problem 6: CSA's cannot make outgoing calls. When the call ends, the call center queuing system sends another call down the pipe. Unless your call is resolved during that call you better hope your phone batteries don't go dead or you're heading back to start over again.

Regardless, on every transfer you have to repeat your story again. Start again from the top. “What is your name?” “I have a few security questions for your safety. ” “Tell me about the problem”. Now you've just doubled your time on the call while the CSA refers to their copy of the Policy Manual.

Problem 7: Don't make us repeat ourselves. Don't ask inane security-theater questions over and over again when the call came internally.

But maybe your problem was more complicated than that. Maybe your problem required some free thought? Maybe your problem had extenuating circumstances? Maybe your problem isn't documented in the Policy Manuals.

God forbid your problem requires Field Service. If it does, well hang tight 'cause you're screwed.

Field Service technicians and Level 2+ NOC technicians are BigCo royalty. You know the drill “we will send a field technician out between 8am and 12pm” … no finer resolution than that. And you had better be home when they arrive, or your name will do in the Black Book and never get service again.

Problem 8: Hire skilled staff. Don't put your Field Technicians and Level 2 line of defense in glass towers where they are handled like Fabergé eggs. Increase the skill levels of your call center workers and Level 1 operators. Give the skilled service that is needed earlier in the process … not later.

The problem with the old-school call center is how performance is measured. Customers view success as the companies ability to quickly resolve our problem with a minimum of fuss and repetition. Call Centers like to give the impression that these things are important. They conduct customer satisfaction surveys. Some even perform follow up phone calls inquiring if our problem was met to our satisfaction.

Unless BigCo really screwed up, most people will give a “ok”, “fine”, “whatever” … because everyone hates telemarketers and spam more than in-bound call centers! They want to get off the phone. They delete the spam email. JUST GO AWAY. Especially if the problem was resolved. It's Consumer Post Traumatic Stress Syndrome. We don't want to think about the pain we had to suffer dealing with your company. Lord knows we'll never say to our friends “You know I dealt with BigCo last week on a complicated issue and they solved it without any issues and I felt good at the end of it.”

For me, just the thought of having to deal with BigCo makes my blood pressure go up.

Problem 9: These follow up calls and customer satisfaction surveys are just more spam. They waste more of our already precious time and you can't trust their results. Find other, less intrusive, ways to gauge customer satisfaction … like # of positive mentions on Facebook/Twitter or permitting BigCo discussion forums or chat rooms. Bring BigCo customers together to solve problems. Don't keep us apart.

No, call centers don't really care about customer satisfaction … you know what they do care about? Call Metrics. They care about Utilization Rates and Inbound Calls Handled and Abandon Rate. Do those things sound they affect us? Hell no. Inbound Calls Handled is an interesting one because it means “Get them off the line as quickly as possible.” Ugh.

Problem 10: Call centers measure the wrong things. They should be forming customer relationships, not just solving technical problems. Drop the green bar reports.

Here is the crux of the problem. There are two groups within BigCo: the Executive Management team (and their immediate lieutenants) and the front line workers. And never the two shall meet.

The gatekeeper on the call center side is The Supervisor. The most hated and dreaded person in the customer service experience. Scrawled in blood on the call center dungeon walls are the words “Do Not Disturb”.

Every perturbed customers asks their CSA to “speak to a supervisor”. Most call centers have electric shock sensors embedded in the chairs such that any agent that says “one moment please” will immediately receive 10,000 low current volts to their nether regions.

“Damn them all”, you say, “I want to speak to your supervisor!”. Background: the Supervisor is usually a battle worn CSA that has heard all the stories before. They have a toughened hide and know, from their many beatings, that there are only four important facts in life:

Keep your metrics in line
Never, under penalty of immediate death, escalate a matter beyond the call center
Life does not exist outside of the Policy Manual
Follow the script

Problem 11: Call Center Supervisor's don't want to talk to customers. They will only after extended periods of duress. And then, they only repeat the same thing the CSA said … they cite the Policy Manual verbatim. Chapter and verse like an old-timey preacher.

Why are these supervisors so cold? Well, we all like to think that BigCo is one big happy family. We like to think that all departments keep in touch with each other and work together for the greater good. The reality is that the Executive Team would never sully themselves by dealing with the front-line workers. Doing so would be like the over privileged working in a soup kitchen on Thanksgiving.

“Yes Muffy, those poor people. Tad and I spent the whole day with them. Listening to their problems and helping out wherever we could. But their problems run so deep, whatever can we do? Charity isn't the answer. One of them asked for my home address to visit, but that wouldn't be right would it? Another martini?”

There is not a razor-wire fence between these groups, there is a gaping chasm.

Problem 12: The Executive Team is so far out of touch with the customer experience they are useless in the customer satisfaction equation. They need more quality time on the front lines or in the trenches.

The Executive Team care about a subset of the Call Center metrics. Mostly Utilization Rate. If one call center is under-performing … outsource to another one. They switch front-line services like a day trader with a penny stock.

So, is it the Executive Management teams fault? Are they the puppet masters behind this grand scheme to piss off the customer? Kind of. But it's not entirely their fault.

The real fault lies with the technology vendors. Back in the 90's the buzz in the call center business was “Integrated Customer Service” (or “Unified Messaging” or 'insert marketing BS here') Previously there was only the phone. AT&T had solved the IVR problem and honed the call center metrics to a precision point. But with the web, there was email. There was live chat. There were multiple contact points. Then we saw the rise of Customer Relationship Management (CRM) – integrated databases that track all customer interactions.

CRM Fact #1: No one ever reads the case history. No one. Ever.

Problem 13: Technology is not the answer. Being a human being is. Ditch the IVR. Ditch the Voice Recognition Systems. Ditch the outsourced call centers.

There's an old saying in software development circles:

“All problems can be solved by introducing a level of indirection.” - Butler Lampson

This is what's happened in the customer service industry. So many levels of indirection have arisen between the business and the customer that service has fallen between the cracks.

David Wheelers corollary to this is “... except for the problem of too many levels of indirection” in other words. “All problems of performance can be solved by removing a level of indirection.”

And that's what we are faced with in the customer service industry … a problem of performance. There are so many layers of indirection that customers are forced repeat themselves endlessly. We get shuffled between groups. We, the customers, get lost in the cracks.

BigCo was sold a vision. They were told “use our products and your customer service levels will increase.” And they were right, but at the cost of customer satisfaction. The call center is a finely tuned machine, capable of handling large call volumes with relative ease. All the while the customers pay the price. Not only in wasted time but in increased service costs. Running these centers cost money. Buying these switches, IVR's, CRM packages and putting bums in seats cost real dollars. But the real cost is the cost to BigCo. Now, BigCo is not just a big company, but BigCo becomes Big-Company-With-Fat-Cat-Managers-That-Hide-Behind-Layers-Of-Customer-Support-And-Are-Out-Of-Touch-With-Their-Client-Base.

And we hate them for it. The technology may help their internal numbers but their outside perception has suffered.

Problem 14: Too many layers of indirection inside of the customer support department. The customer ends up getting bounced around like a ball in pinball machine. Reduce the layers of indirection. CSA's should handle a problem from start to finish, holding your hand the whole time.

It gets worse.

Twitter, Facebook and other forms of Social Media. Oh BigCo is all over them. But it's usually not the call center. It seems like these are Executive management initiatives. Spin offs from the executive management customer support groups or marketing groups are watching the Twitter accounts. And why wouldn't it be? Social Media is hot. No executive is stupid enough not to dust up their resume with “Led BigCo's Social Media initiative and reduced print marketing costs by 7% blah blah blah”

These social media presences try to be more “personable” and put a human face to BigCo. But they have no power. When you need real work done you still have to deal with the old call center. The end result? The frustrated customer has to repeat themselves yet again.

Problem 15: For BigCo, Social Media is a bag hung on the side of the customer support effort. It's a crippled dog that just ends up getting in the way of actually getting your problem resolved. If you're going to use Social Media, go full hog. Replace your call center farms with empowered in-house employees that can handle issues fully. Or keep Social Media as a marketing tool and don't even bother trying to solve our problems in real-time. Your Twitter account isn't fooling anyone. Watch how smaller companies do it.

True Story: I recently had a CSA try to sell me new products after my 10^th call attempting to get service. Really.

Summary

Sadly for us, BigCo often has a near monopoly … there may be only one competitor for consumers to run to, and it's the exact same crap over there too. The devil you know vs. the devil you don't know. There's nowhere to turn.

But there is something BigCo can do. They can try to win back the customer and they can effectively crush their competition in the process.

If BigCo really wants to destroy their nemisis FatShop, it's not going to be by offering streaming movies on demand or fiber to the house or +10mbps … it's going to be by having killer customer support.

Message to BigCo's everywhere: Don't woo the customer on speeds and feeds. Woo us with your killer support. Woo us with attention to detail. Woo us with ability to execute. I'd rather have older technology that works over bleeding edge stuff that you can't support.

Business customers: ask yourself this, Knowing the level of service you get from BigCo as a consumer, would you ever consider running your mission critical business with them?

BigCo needs to get with the times. Modern companies are winning the business from BigCo's all over the world by offering leading customer service. I think of all the products and services I use from all over the world and get stellar service from. Where do I get the worse service? … right here in my hometown. That saddens me.

You know, my phone is a funny device. I use it for everything but making phone calls. When it rings I get startled. Why, because nearly everything I need is available online. Except for BigCo's customer support. There is no Quality of Service, no Service Level Agreement online, no Anticipated Response Time. It's a crap shoot. So I'm stuck having to use their antiquated gauntlet of voice technology. But you know what? The instant I can get a competitive offering from a modern company that allows me to ditch a BigCo service … I'm gone. And customer support will be the deciding factor. I'm done with Dinosaur Customer Support.

Software Craftsmanship - Knife Skills, Tools and Art

2011-01-13T23:27:00.006-04:00

I have a love/hate relationship with the expression Software Craftsmanship. Jason Gorman has an interesting article entitled "Why Software Craftsmanship Is Not Just About 'Beautiful Code'" and I agree with his points.

Don't confuse Craftsmenship with 'Knife Skills'. Cooks with great knife skills in a kitchen are able to try ideas faster because they are not bogged down by the tools. Vim, shell, Git and algorithms are knife skills. Iterations, continuous integration and stand-ups are tools. The Art is applying them. Interpersonal skills are also an Art. Be they talking with your team or coaxing requirements from a customer.

My issue with the expression is when people label themselves a "Software Craftsperson". In my mind, Craftsman implies Master Craftsman. That connotes an image of the grey bearded guru who roams the countryside like Kung Fu looking to impart his wisdom on the "poor unfortunates". His charming tales of daring and adventure, loves won and lost, enrapture the weary apprentices as they strive to achieve that same level of Programming Zen. ... or Bob Vila. Both images cause equal pain.

I think the real Software Craftsmen are the ones that view themselves as the apprentices. Humble. Usually you'll only hear them ask their colleagues "Could you show me how you did that?" They take pride in the work they've done and are happy to show others how they did it, but only from a place of shared excitement. Or, when unhappy with their results, they'll say "Can someone help me clean up this mess I've made?" and respect the feedback given.

For now, I like Software Journeyman more.

Done ...

2010-12-03T08:19:00.006-04:00

There is no such thing as "Done Done". Stop saying it.

You can be:
1. Code Complete (proper comments and debugging code removed)
2. Tests written and all tests pass
3. Pull merge from trunk and integration testing done
4. Pushed / Committed
5. Integrated with Trunk
6. Deployed

1 and 2 may be performed in any order appropriate.
3, 4 and 5 may go many iterations and may mean revisiting 1 and 2.
6 may contain a GOTO 1 in there somewhere.

Then, you are Done. Singular ... hopefully. The beast could wake up again.

Yes, I know the formal definition of "done done" means all this stuff, but it's confusing. Done means done. There is so much that can go wrong between done and done-done that you need to be more specific when reporting to your team. Team communication is too important to discount.

Buying Firewood (aka Becoming a Shirt)

2010-11-22T08:33:00.003-04:00

It gets cold in Nova Scotia. My house uses electric heat which is expensive, so I burn wood on cold days. I burn about four face cords of wood over the winter (a Face Cord is 16" x 4' x 8' of wood)

Wood needs to be cut, seasoned (dried), delivered, split and stacked before the season starts. Wood comes in hardwood (like Maple) or softwood (like Spruce). Hardwood is better for burning. With tax, a face cord of hardwood, cut, split and delivered costs as much as $250 from a reputable supplier. The low end of the price range is about $200, but the moisture content or mix of hard/soft wood may vary. It takes about a year or more for wood to be seasoned enough to be suitable for drying. I can buy 8' lengths of hardwood for about $100/fc. As you can easily deduce, it can cost $1000 for enough wood to last the winter, but it can be much cheaper if you're willing to do some of the work yourself.

Buying unseasoned wood is even cheaper, but it means I need to store two loads of wood on my property. One for the current season, one for the next season. Eight cords of wood takes up a lot of room, so I try not to do that.

Recently, I've been getting wood from friends who don't burn. They have large wooded lots and often lose trees to wind storms, etc. I've been able to get enough wood from them to not have to buy from a supplier. But it comes at a cost. To rent a trailer capable of carrying the weight of a load of wood is about $40/day. You can lug a lot of wood in a day. If the wood isn't cut to 16" lengths, you are also looking at getting the chainsaw out and slicing it up. Assume you're going to destroy a chainsaw chain during this process (resharpening about 5-6 times), so this is about $20. Then you need to split the wood. Rental of a splitter, here, is about $50 for 4 hrs or $80/day. I can split about a face cord of wood per hour. Assume an hour to get the splitter and bring it back. I could buy a splitter for about $2000 and a good trailer for about $2000.

It's a lot of work. Getting fell lumber from friends usually means I have to move it, recut it to size, split it, and stack it. I probably touch each piece of firewood about eight times by the time it gets in the stove. It's tremendous exercise. I like being outdoors in the cool air doing physical labor. It's an added bonus and cheaper than a gym membership. It can, however, be dangerous. You're working with heavy loads of wood, chainsaws, hydraulic splitters and falling trees. It could be done faster with more people, but coordinating the extra manpower is a pain ... unless there's a beer incentive, but that has to be factored into the cost.

I assume I'm going to run out of this supply of free wood from friends in a couple of years. And I'm sure there is a Lost Opportunity Cost given the number of hours I spend on this vs. something else I could be doing that is either making or saving me more money.

The question is, should I continue to scrounge the wood, lug, cut and split it myself or have 8' lengths delivered to process? Should I just pay the price and have cut, split & seasoned wood delivered at the end of each summer ... sort of Just-In-Time Delivery? How should I factor in the obvious exercise component of all of this work? Should I buy unseasoned wood and have a two-year pipeline in the works? Should I invest in a trailer and a splitter to reduce the time travelling to the rental shop and for the convenience of being able to work at my own pace?

Don't know? Ask your Head of Engineering. They should be able to knock this problem out on the back of a napkin in short order. They have to make these decisions all the time. Should we license a third-party component, use Open Source or build it ourselves? Should we hire three more bodies or reorganize the project? Should we tackle this technical debt now or defer until a future iteration?

As programmers, we spend a lot of time stacking firewood. Or splitting firewood. Or moving firewood. We don't take the time to stand back, look at the overall process and wonder if what we're doing is efficient or financially sound. Don't assume that when a developer has "turned into a shirt" that they've lost their problem solving skills. They're just tackling new problems which can be just a challenging. The big difference is that there's often more human interaction (politics) involved in getting a solution.

State Creep

2010-11-05T09:52:00.000-03:00

State kills. It makes concurrency difficult and transactions brittle. State creeps into your program slowly and, over time, makes things difficult to read, understand, manage and maintain.

What is state? State is data. But data by itself is not complicated, nor is it something we can do without. When I talk about state, I mean data that affects business logic. Depending on the state of a variable, the logic of how a module works changes.

The most common form of state is the Boolean:

is_manager
requires_approval
can_see_salaries

When you see a boolean, you should automatically see an IF statement … and, more than likely, many of them.

If two or more booleans are related in such a way that the value of one affects the value of another, your complexity just shot up 2^N. You can expect to see the most nasty construct in your code: The Nested If (and remember that And/Or clauses are just short forms of nested If's). Two related Booleans means four cases to consider. Three means eight. You get the idea.

def show_salaries:
    get_approval = false
    if requires_approval:
        get_approval = not is_manager
    if can_see_salaries:
        if get_approval:
            return redirect('GetApproval')
        else:
            return redirect('SalaryPage')
    return redirect('NotAllowed')

Ugh.

Recommendation: Once you get beyond two linked booleans … Extract to Class. If you can't avoid these nested If statements, at least keep them in one place.

If you're lucky, the state is atomic. In which case the object can only be in one state at a time. Finite State Machines (FSM's) are great way to manage atomic state. The Poor Man's FSM is the Enum.

enum RegistrationState { Ready, EmailSent, Waiting, Approved, Error }

This is usually followed by state transition statements sprinkled throughout your code. Every time you see a branch in your FSM, you should again see an If statement. It gets worse if you need to worry about multiple threads coming into module and attempting to change state. You can see from the diagram above that the Enum does nothing to tell you about valid or invalid transitions from one state to another. You have to read all the code to understand what is acceptable.

Recommendation: PMFSM's are generally bad, but better than nested If's. If you have anything beyond the most trivial PMFSM … Extract to Class. It's nice to keep your state transition logic and locking mechanisms in one place which is easy to test.

But the Poor Man's FSM is only good for state within a single instance of an object. When multiple objects (probably of the same type) need to coordinate you need something a little more industrial strength.

Consider two Employee's that need coordinate on an approval for pending purchases. At first blush you might think the state diagram for the process looks like this:

The PMFSM might look like this:

enum PurchaseState { Ready, Review, Approve, Reject, SendEmail, PlaceOrder, Done}

In reality there is a separation of state between both Employee's. The Role of the Employee in this particular FSM determines which states are applicable to each employee. Bob and Alice are both Employees, but Bob is the Requester and Alice is the Approver. So, keeping the state of the process in any one Employee instance is going to get ugly quickly. It will also limit the number of purchase requests that Bob can make concurrently to 1.

The real FSM might look something like this:

We need to break this FSM out and away from the Employee and make them participants in the transitions of the FSM. Petri-Nets (PN) are excellent data structures for modeling these sorts of FSM. With a Petri-Net you can model concurrency, forks, joins, splits, merges and multiple running jobs … everything needed by most business applications.

PN's are the basis for most modern Business Process Modeling (BPM) / Workflow solutions such as Windows Workflow Foundation, jBPM and a multitude of others. If you live in a big stack like Java or .NET, these are valid options. We're also starting to see many BPM solutions becoming SaaS, so implementation language is becoming less of an issue.

You'll find that for most business applications, having a good BPM solution in your toolkit will make your life so much easier. And you don't need a heavy-weight solution that has graphical editors, connectors and other bloat … just a simple Petri-net implementation will get you a long way to more maintainable code. You'll find one for nearly every language out there.

Recommendation: When you have complex state interactions between multiple objects, consider using a Petri-net based library to keep your code manageable.

To summarize, state is a nasty thing in your code. Always be aware of it and be vigilant to keeping it under control. Hopefully some of the tips I've listed in this article will make that process easier.

Removing "Configuration Flag" Code ...

2010-10-20T11:46:00.002-03:00

I had the pleasure of working on some code with @edleafe recently. It was a change in some existing functionality. This change could be turned from "old way" to "new way" with a configuration flag, which is pretty normal practice and encouraged.

The issue with "Configuration Code" is that after a while you have two blocks of code that you have to maintain. One for the old way, one for the new way. And, hopefully, there are tests around the old code and the new code.

What has always bothered me that once the flag gets set and its state stabilizes, the old code should be ripped out and the flag removed. The reality is, this never seems to happen. When was the last time you grep'ed for TODO? So you end up with these Conditional Code blocks all over the place. It's worse when your flag touches many parts of the code.

Ed had a great idea. Add a ticket to your bug tracker to remove the conditional code and the flag. Product management can review the bug from time to time and determine when the flag is no longer needed. It becomes a thorn in everyone's side until it gets removed.

Simple solution and maybe it's common practice, but I had never considered or heard of it before. I like it.

The Joy of Getting Cardinality Assumptions Wrong ...

2010-10-17T13:59:00.001-03:00

(Sorry for the infrequent posts as of late ... just getting back into it. Lots of new fuel to write about. This is just a little note to prime the pump again.)

As a developer, I find that some of the biggest code changes arise when my assumptions of "how many" of something I can expect are wrong.

I thought there would only be 1 when there are going to be N items.
I thought there would be N when there are going to be N²items.
I thought there would be N² when there are going to be 2^N items.
Or, I just guessed N wrong.

Each change in magnitude calls for a radical change in architecture. You have to go from a single variable to data structure, or from a list to a tree or from a tree to a database or from a single database to a partitioned database. Everything changes. Often, the clients of the code break because the access mechanisms change. They don't get back a single item, they get a list. They can't get all the items in a single query, but have to page the results, they have to give better qualified queries, etc.

User Interface is perhaps affected the most by these changes. Going from showing 1 result to N results is a big change. When N gets beyond a manageable number you start thinking about trees ... more UI changes. If N is too large for a tree you start thinking about tagging/search mechanisms. The UI changes again.

Big development costs.

The multi-tenant problem is a good example of getting bitten by these assumptions. You assume you're going to have one customer per installation of your code. But then you evolve into a hosted application so one installation has many customers and no customer can see another customer. Then you support resellers, so a customer is not an end-customer, but a meta-customer than has many sub-customers. The reseller can see all of their clients, but no client can see each other. Ugh. These sorts of changes hurt. They usually take considerable time to implement and it's tough to get right.

So, when you are talking to your customers (or prospective customers), really try to get to get a sense for the cardinality of the things they talk about. Think about how big a change your code will undergo if you had to move from "one of these" to "N of these". Ask them explicitly "How many of these can you foresee?"

Personally I think these sorts of design problems are so painful that I'm willing to violate YAGNI and opt for the larger data structure.

Bug Triage & Backlogs ... Unseen Broken Windows

2010-07-06T11:31:00.001-03:00

Bug reviews are a necessary evil. Your application is out in the wild and, inevitably, bugs will be found. If you're busy working on other things, these bugs usually get stuffed in the bug tracking system for later review. At some point you will sit down with all the product stakeholders and decide what sort of priority to put on these bugs. Everyone needs to agree which bugs are going to get fixed and which are going to get deferred.

As a developer, this is generally "yet another meeting". We show up. We give our thoughts on the bugs. Is this one difficult to fix or can you fix it during the meeting (kidding)? We don't assign priorities ... that's generally the job of the product owners. The product owner might be a direct customer or it could be an abstracted customer like a product manager or the sales team (the Customer Proxy in Agile parlance). We just throw in our $0.02

Now the real show begins. Sales flips out because this bug or feature request is holding up revenue. Product management may veto a ranking because of a more strategic matter. Marketing will scream that the competitors can make a Square Circle and that our product needs one too. There are feats of strength, ruffling of plumage, bun fights and kiva sits until eventually the bug list is placed in an ordered fashion. Bugs are ranked in some Priority and Severity such as Blocker, Critical, Major, Minor or Trivial / number codes / whatever. If you're really lucky, some bugs will be removed from the list as being no longer relevant or already fixed.

Then it's our job. Development will do whatever sort of voodoo it needs to do to estimate, assign and work on these tasks. A new release is issued and the process repeats. Ideally the cycle time between bug review and product release is suitably short so that the bug rankings don't shift too much. No real magic here.

And, as we all know and every Agile pundit will evangelize, the priorities will shift.

If you get your priorities from Sales, your priorities will shift on a near-daily basis. The hottest customer will get the most attention. Depending on the length of your sales cycle, don't plan on taking on any ambitious new features or refactorings because you're likely to get pulled in a new direction tomorrow. One thing I like about having a good product manager to work with is they can temper the rapid mood swings of sales people and boil them down to a common theme or essence.

Regardless ... not all of the bugs can get fixed. You build up a bug backlog. After a while, in an attempt to keep the meeting short, you stop reviewing the Minor or Trivial bugs and just focus on Blockers and Criticals. After a longer period of time, you can't manage the duplicates anymore so the list grows even longer. It's a Broken Window. Because of the organizations inability to effectively manage these bugs the overall effect to the end-user is a rash of problems that are simply never addressed. Your application is dying and you can't prevent it.

Here's the real kicker: it may have nothing to do with the quality of the code or the process. Your code could be clean and readable, but simply because of market forces there is an unmanageable backlog. What can you do?!

It's a hard problem, but here are some suggestions I would make to help get you back on track:

Group your bugs and look for architectural commonality.
You may notice that large collections of bugs stem from a common architectural place in the code. It may not be obvious at first. In a separate worksheet, try and group your bugs by software sub-modules and look for alternative approaches that might kill many bugs in one swat. You may find that cutting off an arm will save the body.

Freeze all new feature development and do Bug Sprints
Push back on new feature requests or enhancements. Use all of your political powers to rally support about the importance of getting the bug list reduced. Try alternating Bug Sprints between normal iterations in an attempt to get the bug count down.

Remove functionality
This is often not as hard as it sounds. Is the bug related to a legacy feature? How many customers will be affected if that feature is no longer there? Can they live with a work-around? Removing lesser-used functionality means less code. Less code means happier developers.

Shore up your tests
Are you lacking some critical code coverage in your unit or functional tests? How many of these bugs could have been eliminated with better tests?

Hire a usability expert
Are the bugs related to confusing UI, complicated configuration steps or poor documentation? Could the product benefit from a usability review? Your product might be stable, but due to poor usability you are seeing a high number of bug reports?

Replace third-party components (or start using them)
Do your bugs stem from a flaky piece of third party code? Can you replace it? Alternatively, do your bugs come from a piece of code that was developed in-house but can now be replaced by an off-the-shelf alternative?

Start hiring or Split the Team
Perhaps your growing bug list is a symptom of not enough resources? Should your company make the investment in another developer to help share the load? If so, can you ensure that they won't be used by the Customer as a means of slamming even more functionality into the product? If not, consider splitting the team so that the Customer deals with one group just for new functionality and the other group just for bug triage and repair. Balance the size of the teams according to how dire your situation is.

And then ...
... fight tooth and nail to keep the bug count at zero. Always fix bugs first. You will get incredible pressure from sales to sweep them under the rug and some bugs can be notoriously difficult to find. Ask for help. Hire a highly skilled consultant to help you fix the bug if need be. But try to keep the bug count to zero or these Backlog Broken Windows may kill your productivity (if not your product).

Next time, I'll try to give some of my thoughts on how Kanban may or may not be able to help with this problem. Until then, I look forward to your thoughts.

The Problem with Java and C# ...

2010-06-29T10:10:00.003-03:00

I'm going to stick with saying "Java" in this article, but the same argument applies to C# as well. Replace accordingly.

There seems to be a lot of bad sentiment aimed at Java these days. Either complaining about the verbosity of the language, the bloat of the surrounding tools or the mentality of the developers that use it. Other articles seems to talk about how rapidly that community is shrinking or the futility of the innovation in the community.

The problem with Java is not the language, it's the hiring mentality that goes behind it.

When companies decide to play in the "enterprise" space, they feel they need to use programming languages that best fit with those environments. This usually means Java. This is nonsense. Facebook, Twitter, Google and other hot new companies have incredibly scalable architectures, many without SQL, LDAP and Application Servers. We know there are other/better ways to solve the scalability problem. Also, you have compatibility with legacy systems without requiring an Oracle Application Server. If you need to have an army of Operations people to tune your app server, you might want to reconsider your architecture.

Another factor in this rationalization is the ability to hire from a wider gene-pool of developers. There are certainly more Java developers out there than nearly any other programming language. When hiring a Java developer many of these companies hire tactical skills, not general skills. Do you know Struts, JSF, Hibernate, Websphere, etc.? And I can understand why. If you build a product that is going to have a long life-cycle, you have to assume the software is going to have to live beyond the developers that assembled it. If the application is written on niche or trendy languages or tools it will be harder to replace these developers later.

This is broken thinking.

People will do what you reward. If you reward blindly learning software packages without understanding what their benefits and limitations are you will end up with a development team full of automatons. If you reward understanding computer science fundamentals, the best tool for the job, code simplicity and readability you will end up with a better software product and it will take less developers to build / maintain it.

And don't read this the wrong way. I'm not implying that all Java developers are sheep. There are many, many awe-inspiring Java developers out there. Sadly, there are far more sheep.

Can you blame the developer? No. The common sentiment is "If I stick with Java, I'll have a job." ... and it's mostly true. This is usually followed by "Do I like my job? Meh."

Yes, it is harder to hire these sorts of developers. Your interview process will be lengthy. You will eliminate more candidates. But do you want to work in a brain shop or a body shop? Which is harder: reading clean understandable code in a new language or reading verbose bloat code in an understood language? Which is easier: reading a new DSL or pages of XML?

What is the impact of developer farms that are hired for tactical skill sets instead of core understanding and passion? An uninspired workspace. A breeding ground for "[Flavor-of-the-Day] Coaches" who feed off the malaise of the corporation. "Certified [SilverBullet] Masters". Higher turnover. More developers pounding in Feature X, Y & Z without a care for the big picture.

Let's not throw out the baby with the bathwater. Don't diss the language. Diss the companies that foster this lousy work environment on us developers. Developers, work on your core skills. Don't let yourself fall into the "must master the Java stack" trap. Learn other languages and see how they tackle the problems. Look under the hood at how the libraries you use today work and what the alternatives are. Learn new API's, but then learn their implementations. Review your algorithm books and decide if there are better ways to solve the problem.

Don't go wide, go deep and don't work for body shops.

Should You Always Explain Why?

2010-06-07T09:08:00.000-03:00

When mentoring apprentice software craftsman there are times when rationale gets lost. You are trying to teach them caution. Caution means slowing down and being attentive to your surroundings so you don't do something dangerous. But if they never get hurt can they really understand why they should be cautious?

It's different with children. You don't want them to use the stove or run with scissors because the cost of not being cautious is too high. But you let them spin around like a top until they fall down, potentially getting a bop in the head. You let them have a pillow fight, knowing that someone is going to come screaming in a few minutes. Why? Because when they get hurt they learn a valuable lesson.

In software, how do you gauge what is an acceptable risk? I remember several years ago working with a group of junior developers and getting them to write unit tests. I was a broken record: "You have to write your tests." "Did you write your tests?" "You can't refactor without tests." "Did you run the tests before checking in?" "Did you run the tests as soon as you checked out?" ... but they had never really felt the pain of not having tests or dealing with poorly written tests. It was nearly three years later when one of the developers came to me and said "You know I finally get this unit testing thing now. I just did it before because we were told to, but I never really understood why. Now I do." Almost instantly, all the light bulbs went on for him. Short iterations, the importance of enforcing a coding style, the use of Fakes & Mocks ... it was easy to see he was no longer a junior developer. Testing was a part of his craft now.

Is there anything more I could have said to make this more evident? Probably. But I think the words would have been lost or come across as more preachy than it always was. He needed to feel the pain for himself to learn.

But ... as a manager, could we afford to have developers not doing tests for three years? not complying to the coding standard for three years? No. We had felt the pain. We know what the costs of not doing that stuff is. I'm sure many of the tests written during his journeyman-ship weren't very good ... but they were building muscle memory. I'm sure many master carpenters, glassblowers and potters do things by rote for a long time before really understanding the subtlety of their actions.

Being a mentor, just as being a parent, you have to make the hard decisions of when your charges are really going to hurt themselves and when it's ok for them to take a smack in the head. And, often times, explaining why beforehand gives no benefit.

How long can you work on a programming team before things get stale?

2010-06-04T10:25:00.003-03:00

Assuming it involves lots of new learning initially, my gut tells me it's about 1-2 years. Once a developer understands the problem well enough they can start to get antsy.

I have a theory that 3yrs is the maximum amount of time a programmer can work in a given problem space before things get routine. It doesn't have much to do with your team or the company ... it's the challenge. The rest is an "exercise left for the reader."

Certainly there are exceptions. People can last longer but, to me, it seems generally they become ghosts in the machine. All the good programmers will tell you the same thing: "program on the side", "learn a new language", "experiment with an API", "moonlight".

But are your extra-curricular activities aligned with those who write you a check? A recent trend is for programmers to be let go for "not being engaged" ... that's scary.

Or, are you one of the really lucky ones? Is your work so in-depth and challenging that you constantly have to research new techniques to solve them ... and you are given the time to do so?

I think the quote on "Beards and Keyboards" regarding Change says it best:

So when you hear people say that change is hard because people are lazy or resistant, that’s just flat wrong. In fact, the opposite is true: Change is hard because people wear themselves out. And that’s the second surprise about change: What looks like laziness is often exhaustion.

— The first few pages of the book “Switch, how to change things when change is hard” are priceless. Get it! By Chip Heath & Dan Heath

Packaging a Django Application as a Stand-alone Application

2010-05-18T09:53:00.000-03:00

Here is a little diversion from my normal Software Craftsmanship ramblings. Recently, I needed to package a Django web app as a stand-alone Windows Application with an installer (see below for the rationale)

In order to make this work I needed to get around a few hurdles:

Since I'm using Django, I didn't want to customer to have to install Python, so I needed to turn the Python code into an .exe. Py2exe is great for this.
I also use a bunch of other python libraries such as rpyc, Python Win32 extensions and have a bunch of custom C++ components that interface with Python via Swig. These all needed to get wrapped up and included as well.

I didn't want the customer to have to deal with installing each of these components and sewing them together. Assume no IT person is available. I also didn't want to show the installer spawning off all these other sub-installers. It needs to install and appear in the menus like every other Windows application.

"Well", I thought, "best get started. Let's get py2exe running on manage.py (the django project startup utility) and make a .exe out of it." And that's when the fight started.

Django does some funky magic with imports under the hood that breaks the component discovery mechanism of py2exe. The first thing I needed to do was eliminate manage.py. There were a couple of good blog posts that talk about this very problem. The trick is mapping your inside and outside directories so you can keep your database and template files outside of the package that py2exe fabricates.

Following their lead, I had a bootstrap program I could use to launch my application as an executable. All I had to do then was throw an install script around it with InnoSetup we were off to the races.

Then there was Part Two: I had my daemon process that did the network monitoring. This was also Python and used the rpyc library to communicate with the Django server. It also talked to a bunch of C++ programs that were python-wrappered using Swig, so it was .dll hell. This was going to be my other .exe.

Py2exe normally does a great job of dealing with .pyd files (essentially Python extension dll's), but it has no way of knowing when a .pyd references a .dll which may reference many other dll's since they are loaded dynamically at runtime ... as their name suggests. The result was a crashing application in my test VM.

In order to track down all the dll dependencies I used Dependency Walker to see which dll's were being accessed and which were failing. By using the "Profiling" option, you are able to see when a LoadLibrary call is made and where it fails.

Happy to say ... it works like a charm.

Why do this?

This is an experiment. There's a part of me that likes the idea of packaging a web-application as a standalone application. While this is only applicable to certain class of applications, using a lightweight HTTP server and a tiny database like SqlLite to deploy is pretty sweet. Why? A browser is a familiar experience and it's way faster to write UI for web than GUI frameworks. People are familiar with web UI semantics. They know what happens when you close a browser window, when you hit the refresh or back buttons, when you click on a link. GUI products require new learning. Most of the users work day is in a browser. Why make them switch context to another tool?

I can hear you now "Why not just make it Software as a Service (SaaS) like everyone else?"

Normally I would. 99.9% of the time SaaS is the way to go. But for this application I need to have a continuously running process (which listens on the internal network) while the user interface is running. I could still do this with Saas, but it would require people trusting to install the daemon process inside their network and let it call home. And, for our customers, often times their production network does not have access to the WAN.

Down the road, they can buy a SaaS version if we deem the product worthy of that effort, or upgrade to more robust components. But for now, it's an easy way to get started. And their entire team can try the application without every having to install it and futz with networking. This also falls into the "release early, release often" camp of product development. I need to get feedback from our early adopter customers and this is a simple way to get started.

Moreover, look at all the other tools I can integrate with a simple link: I'm using UserVoice to capture customer feedback and jquery for some nice UI. I can use Google Analytics to see usage patterns. There is so much more I can do with a local website than is remotely available to a conventional fat-app. I can easily put a REST API on the product if need be and online help is a simple addition.

The obvious issue, in the short term, is that someone needs to remember to launch the server before hitting the web page. This could be an issue. There's also no name resolution for the person that installed the application. It's essentially a localhost website. And, there are known security issues with this particular http server. I may need to use something a little more secure soon ... but still lightweight. We don't always want to notify the IT department, get new databases created, add new modules to the application server. This, to me, is a middle ground ... well, at least I hope so. I may regret those words.

I'll let you know later how this experiment goes.

The new Facebook features and how they affect your privacy ...

2010-04-21T22:53:00.006-03:00

This isn't a note for you geeks, it's for my friends that aren't geeks and use Facebook.
----

I know many of you don't follow this stuff, so here's the Readers Digest summary of the new Facebook changes made recently, what you can do with it and how private your actions are.

As of today, if you visit sites like CNN or IMDB you can now "Like" or "Recommend" articles, products, movies, etc. directly from their site. If you do, these actions show up on your Profile (under Info | Pages) and cannot be filtered with your privacy settings.

The only way to remove it is to go back to that page and un-like or un-recommend it. You can't remove it from within Facebook.

This means that any person or application can see your Likes and Recommendations. This could be an issue if you "Like" the Marijuana Party or similar fiery topics. Everyone who sees your profile will see this stuff, even if they're not your friend! And, anyone outside Facebook can see ALL the people that endorse that thing too. You've just published personal stuff outside the gates of Facebook. Be aware.

Also announced today, non-Facebook applications (like Mafiawars, quizzes, Scrabble, etc.) do NOT have to delete their copies of your personal data any more. Previously they could only keep it 24 hours and we all hoped they were honest.

So, don't Like or Recommend something that could get you in trouble now or 10 years down the road (especially you crazy kids in university). The web never forgets. And don't install those stupid applications. You have no idea who runs these things and what they are doing with your personal data.

Edit: Thanks for question from @BradFraser. "Can you turn it off?" ... No, you can't turn it off, unless you log out whenever you leave FB. This also means that simply by looking at an "enabled" external page Facebook knows it. There is a new privacy option under "Privacy | Applications & Websites | Instant Personalization" that is enabled by default, you should turn it off so the site doesn't read your profile on visit. This can't stop Facebook however.

Update: regarding my comment that "And, anyone outside Facebook can see ALL the people that endorse that thing too." You can turn this off by making sure "Privacy Settings | Profile Information | Likes and Interests" is not set to Everyone. I'm not sure if this will cover Recommendations too?

Hope it helps!
-Sandy

PS> Geek postings will resume shortly.

Am I a Bad Man for Naming my Classes after Patterns?

2010-04-09T14:40:00.001-03:00

A couple of weeks ago a rather innocuous tweet from Nat Pryce (@natpryce) flashed on my screen which read:

“Pattern names should be obscene so that programmers will not be allowed to name classes after patterns.”

It was quickly picked up by the tweet-o-sphere as programmers from all over echoed the chant. But wait … I name my classes after patterns?! Was I committing some sort of programming faux pas? For years, in discussions with other developers over design issues or implementation details, I would always try to reference the appropriate pattern where applicable. And these Pattern-based conversations inevitably made their way back into the code.

“That server uses a Reactor Pattern so it's not going to perform as well on a multi-core machine. Perhaps you should consider one based on Leader-Follower?”

“Each instance of that class is going to occupy a lot of memory, but we need to cache the most popular ones. Would the Evictor Pattern be the best choice here?”

If you go back and look at the original GOF book the rationale given for Patterns was based on a quote by Christopher Alexander who, speaking of buildings and towns, said,

"Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice"

Speaking in Patterns does something very important between programmers … it raises the level of the conversation. If I talk to another developer about a Strategy Pattern, I don't need to say “Create an interface and move the specific implementation for each way to lay out the nodes into each derivation of it. That way, if we can't lay out the nodes one way, we can just change that class and try another way. Get it?” If my colleague knows what a Strategy Pattern is, he'll just say “Yup, good idea.” (or "Are you insane?")

I love whiteboards, but I don't need to write on them every day. That's where we would be without Patterns.

When I name my classes with a Pattern name, I'm doing the same thing, I'm telling you how I'm attempting to solve a problem without requiring you to look into the nasty implementation details. A comment on a class or interface definition is not the same thing. It's not going to follow everywhere the class or interface is referenced so the details get lost.

If I see a package that has three classes: FooStrategyFactory, FooStrategy and FooState I have a very good idea what's going on in that module. I don't need to look at the code.

This was the basis for my discussion with Nat when I asked “Why is this a bad thing?” We chatted back and forth (as best you can on Twitter) about my position and his first objection was “Why do you need to call it FooStrategy? If there is an Interface you can assume there are going to be more than one of them.”

A good point. Perhaps there is something to his argument, consider this developer conversation:

“Oh, there are going to be several different ways to lay out those nodes in the graph?”
“Yes, I'm going to switch between then using a Strategy Pattern.”

And I set off to coding. I start with a LayoutStrategy interface and move down into the implementation definitions I may create concrete classes like LinearLayout, CircularLayout, SquareLayout. I will concede that you don't need to repeat the pattern name in these derived classes. There is no need to call them LinearLayoutStrategy, CircularLayoutStrategy, etc. because you'll see that in the class definition:

interface LayoutStrategy ...

class LinearLayout implements LayoutStrategy …

And, in this scenario, you probably never should see these derived class names anyway. They are likely being instantiated from a Factory or being “injected” into some other class. A common calling scenario might look something like:

LayoutStrategy layout = layout_strategy_factory.get_layout_strategy(“circular”);

renderer = new GraphRenderer(layout);

And, arguably, it's a little verbose. But, how does it look without the Pattern name in the class?

Layout layout = layout_strategy_factory.get_layout_strategy(“circular”);

renderer = new GraphRenderer(layout);

That looks pretty vague to me. I can't infer how to interact with the class based on the name like I can with LayoutStrategy. How about we put the pattern in the instance variable?

Layout layout_strategy = layout_strategy_factory.get_layout_strategy(“circular”);

renderer = new GraphRenderer(layout_strategy);

Uggh! Now the developer has to know what the implementation approach is in order to correctly name the local variable. No way.

Personally I think the first snippet offers the most value. But I find all of the above snippets equally useless. If I was a first time reader of that code, I wouldn't know if Layout objects should simply position the nodes or render them as well? What is the purpose of each class? I have a hint about their purpose and I have a good idea what's under the hood, but I have no idea about the separation of duties.

And this seems to be one of Nat's concerns as well. Some other interesting points he mentioned related to this were:

that classes should be named after their purpose and not their implementation, and
that every class applies and/or plays roles in many overlapping patterns. Picking just one for the name ignores all others.

But I'm not sure I agree. I'm a big fan of the Single Responsibility Principle. That a class should do one thing and one thing only. If a class is implementing more than one Pattern it should be a Composite of the separate functions. Perhaps his concern is that some programmers name a class after a Pattern and then the class ends up doing something different? Bait-and-Switch naming is a viable concern. But that's something that should be caught in a code review and not a reason for a blanket coding style guide such as “Don't Name Classes After Patterns”.

Class Naming becomes an issue for two different groups of developers:

Consumers of the classes
Maintainers of the classes

And most frequently, these are different groups with different expectations from the class name. Knowing the Pattern a class follows helps both groups. If it's a Strategy, I'm going to look for a place where work occurs. If it's a Leader-Follower, I'm going to look for the event handler and the thread dispatcher. If it's a State object, I'm going to try and find the Finite State Machine and methods for transitioning between states. Those are important clues when you have to look at legacy code or use an API for the first time. I don't even need to read the docs to know what I'm looking for.

So, the problem seems to be this:

If the classes are granular enough, naming them after Patterns is useful since it provides an indication of Behavior. But it is not expressive enough to convey Purpose (in the context of the problem).

How can we name a class so as to convey behavior and purpose? I'm certain the wrong way would be to give classes names like LayoutButDoNotRenderStrategy and RenderNodesButDoNotPositionThem.

But if the class is named after the behavior, perhaps the package/module names should convey purpose and context? The package name describes what the Factory, Strategy and State objects are acting on.

In our toy example, we need the the GraphRenderer class to live in a package that screams “I draw stuff!” and the Layout classes live in a package that screams “I position stuff!” Perhaps the following works:

package Graph.Node.Placement;

class Factory;

interface Strategy;

and

package Graph.Rendering;

class Renderer;

Our calling code now looks like:

import Graph.Node.Placement.Factory;

import Graph.Node.Placement.Strategy;

import Graph.Rendering.Renderer;

Factory strategies = new Factory(...);

Strategy layout = strategies.get_layout_strategy(“circular”);

renderer = new Renderer(layout);

Personally I find this more than a little sterile but immensely readable. If you want to make it like a hammer-to-the-head you could do:

Factory strategies = new Graph.Node.Placement.Factory(...);

Strategy layout = strategies.get_layout_strategy(“circular”);

renderer = new Graph.Rendering.Renderer(layout);

Works for me. Am I still a bad man for my rationale? What am I missing here? What are some viable alternatives? What works for you? It would be interesting to try this experiment with other Patterns, but I think single implementation Patterns are the easy cases. Overall, it's a tricky topic and class/package naming is more Art than Science.

Finally, some side notes:

In order for any of this to work, your team needs to agree on the reference sources for their Patterns. If people start pulling Pattern names out of thin air the conversation doesn't benefit. For example, you could state that your project can only reference patterns from the GOF, Pattern-Oriented Software Architecture Vol 2 and Core Security Patterns. At least then the developers have somewhere definitive to start their searches. (see my previous article on Coding Standards)
We do agree that calling things Manager is bad. But this is a simple problem to solve since there is no Manager Pattern. Did the developer mean Factory? Calling a class a Manager is a definite code smell and should be avoided.
Having Python as my weapon of choice for the last ten years it kills me to have to make all these verbose module definitions. But for projects of any meaningful size you need to be regimented in your code organization.

Look forward to your feedback in the comments!

A Tip for Closed Source Software Houses: Hold off on giving Commit Access

2010-03-16T13:18:00.002-03:00

In the preface of Kent Beck's fantastic "Extreme Programming Explained" book he spells out the core principles of XP:

If code reviews are good, we'll review code all the time (pair programming).
If testing is good, everybody will test all the time (unit testing), even the customers (functional testing).
If design is good, we'll make it part of everybody's daily business (refactoring).
If simplicity is good, we'll always leave the system with the simplest design that supports its current functionality (the simplest thing that could possibly work).
If architecture is important, everybody will work defining and refining the architecture all the time (metaphor).
If integration testing is important, then we'll integrate and test several times a day (continuous integration).
If short iterations are good, we'll make the iterations really, really short—seconds and minutes and hours, not weeks and months and years (the Planning Game).

As someone who is, not only a programmer, but responsible for making sure the correct products make it out the door on time, only three of these are critically important to me:

If code reviews are good, we'll review code all the time.
If testing is good, everybody will test all the time.
If integration testing is important, then we'll integrate and test several times a day

The others are important, but for different reasons and we'll touch on them later in this blog.

The list is very similar to a story Anthony Robbins tells in "Awaken The Giant Within". Someone asked him "How do you become a great public speaker?" and he essentially said "Talk in public all the time". Poor testing and Integration are, without a doubt, the two things under our control that will make a software project go off the rails. Crappy code is the other. Test all the time. Integrate all the time. Read the code all the time. But how can you do this effectively? TDD and Continuous Integration are well solved problems. The real challenge is getting others to look at your code.

Many suggest Peer Programming. Most developers I know don't like it or can only handle it in small doses and management often have a hard time rationalizing it. Personally I'm not a fan. I find too much time is lost because the development thought process hasn't fully gelled yet. It's gratuitous programming. But the motivation is sound. In the Open Source world, Linus' Law states "With Enough Eyeballs, All Bugs Are Shallow." We need to have visibility on our code and, I think, Eric Sink puts it most succinctly: "Read the diffs!" This is a great way to read the code after the developer has had time to get their thoughts together. My issue is that it happens too late. Once the code is in the revision control system you have to make immediate decisions on how to rectify any problems you find during the review. Revert the code and repair or wait for the repair with broken windows in your source?

I prefer Jacob Kaplan-Moss's perspective on granting commit bits ... don't. If lower level developers cannot perform commits it forces the higher-level developers (the ones trying to enforce the religion of the project) to review their code. They are looking for your use of coding style, of enforced idioms and documentation. If ... IF ... you can consistently code to the norms of the company THEN you are granted commit bits to the revision control system. You can then review other developers code, merge their patches and mentor them on what is important. So, I would make a slight variation to Eric's article and change it to "Read the Patches".

For the corporate software houses that never let their source outside the door, you can still have the benefits of the open source world by making code review a mandatory part of your development process. No projectors, no print outs, no meeting rooms. Just turn off commit access. Right now ... how many people on your corporate development project should have their commit bits revoked?

Revoke and Review is my new mantra.

How Do You Hire Software Developers?

2010-03-11T12:15:00.001-04:00

I want to learn more about the s/w dev hiring practices used by software companies.

When the time has come to bring a new body to the team, how do you execute that process?

First, let's consider the job description ... do you write this up in an "ad" format? If so:

Does the hiring manager write the skills requirements?
Do you focus on tactical skills or fundamental skills (i.e. Joomla vs. core OOAD)
Can you give an example of how you separate your must-have and your nice-to-have skills?
Do you prefer a strictly business format or hyper / jumped-up / sell-sell-sell ad? Do you find it makes a difference?
Do you have any examples of great job ads you've seen, written or responded to?

With your fresh list of requirements, how do you get new resumes?

We just put an add in the 'Careers' section of our website and they appear
We put ad in traditional print media
We pay for ads on generic job websites
We pay for ads on IT-specific job websites
We pay for ads on Language or Technology-specific websites (django/php/etc.)
We attend job fairs
We promote through social media
We recruit from the open source community
We recruit from our open source community
We only use contractors

Which gives the highest response rate? Which gives the best candidates?

Now that you've got a stack of shiny new resumes and you've filtered out all the noise, what's next? Some people like to do a phone interview. For the purpose of this post, I'm not interested in communications skills, availability, salary expectations, previous employment red-flags ... I'm interested in how you test their technical ability over the phone. The phone can be a great way to get some high-level data on the candidate, including:

Some generic technical questions to weed out the weak ones.
Team dynamic questions about difficult interpersonal situations
Thinking process for debugging
Logic/lateral thinking tests
General understanding of algorithms or major systems

Sometimes companies will ask the candidate to do a take-away programming quiz. This is a relatively simple programming challenge that most developers should be able to solve in a few hours. The intention is not to focus on ability to execute under pressure but, assuming you have have access to all your normal reference material and resources, does the candidate exhibit good code craftsmanship? Can they following a coding style guide? Do they have bad coding habits?

Then do you bring them in for a face-to-face interrogation? If so, what is the agenda? Is this a single person interview or do you take them through a gauntlet of technical and business interviews? Do you issue more "think on your feet" programming or logic problems?

Do you think a take-away programming challenge gives better or worse results than a "think on your feet" test?

What do you think of the Google/Facebook approach where the key to entry is to compete in their programming challenges before getting an interview?

Ultimately ... how do you hire developers? Is it working? Are you getting the best developers possible or getting stuck with duds? What would you change?

Developer Skill Sets: There is no such thing as an Intermediate Software Developer ...

2010-03-05T11:00:00.001-04:00

At every company I've worked, the performance reviews have been structured nearly the same way. Software developers are graded on perhaps five different traits:

Technical skills - How well do you code?
Debugging skills - How well can you do maintenance work?
Communication skills - Can you write effective email, documentation and talk to your team?
Business/Marketing awareness - How well do you know the market this company plays in and the competitors?
Time management - Can you focus on the task at hand?

The weight applied to each category varied depending on the developers position of Junior, Intermediate or Senior, Team Lead, etc.

While these are not bad, in the general sense, there are two very large buckets in the above list where a multitude of sins can occur: technical skills and debugging. Fine granularity is important to software developers when it comes to performance reviews. If you want a developer to exhibit great code craftsmanship, then you need to be explicit on what you value and reward.

What Can We Do To Revitalize The Old Developers Performance Review?

Let's look at the table stakes; skills every developer you hire should be able to exhibit before you hire them! These are not skills you want to teach on the job:

Solid core CS skills - algorithms, complexity, architecture, underpinnings, etc.
Operating system essentials - scripting, backup/restore, boot process, device management, remote access
Programming as a passion - They program recreationally and perhaps have a few side projects on the go. They never got into CS just because they thought it was a sound career choice.

I assume I'm getting these skills from a good programmer right out of school. More is nice, but I don't assume much more. Code craftsmanship is rarely taught in school. It comes from the school of hard knocks. Once you've been beaten into submission by a chunk of code you can start to think about how to do things better.

If we break Technical Skills and Debugging Skills down, what could this list look like? Here are the things I like to see in every developer that I work with:

Adheres to coding style
Correctly applies our core development idioms
Follows language best practices (knows "Effective [Foo]")
Maintains useful documentation
Writes effective tests
Writes tests that act as teaching aids
Is able to focus
Contributes constructively in development discussions
Experience with dynamically typed, static typed and functional languages
Can grok new topics quickly
Understands scalability concerns
Understands usability concerns
Effectively uses Patterns
Refactors in a timely fashion

(What would your list look like?)

You give me a team of developers that have those skills and we will conquer the world! Obviously not everyone, especially anyone straight out of school, is going to have that skill set. The above list is really a road map to great code craft.

Personally, if over the years, I was given this list in my performance reviews it would have given me a lot more career direction than just "better technical and debugging skills". I would have loved to learn that I need to focus more on "writing tests that act as teaching aids" and that I'm considered great at "groking new topics quickly".

Given a suitable list, any company should be able to delineate what they think it means to be a Junior, Intermediate or Senior developer.

Perhaps:

Junior: < top grade in 5 topics,
Intermediate: top grade in 5 to 9 areas and
Senior: top grade in > 9 topics?

(What would work for you?)

I have a theory about this: There is no such thing as an Intermediate developer! You are either Junior or quickly on your way to becoming Senior. If a developer stays in the Intermediate designation for too long the company should re-evaluate their role in the company. Sounds a little harsh but, assuming they are being given every opportunity to advance, why are they not? Not everyone can be an astronaut.

More so, these skills are applicable to nearly any company. Most programmer job ads have stuff like this:

Knows Struts, Hibernate and JSP
3yrs experience with Powerbuilder 6.x and Oracle Form Blaster
Has Cake PHP experience

While I understand the motivation of getting a developer that can hit the ground running when you hire them, I would argue that a developer with the craftsmanship skill set listed above would be handle any of those specific domains with little problem. I would also argue they could probably do it better than developers that only focus on those specific tools. There needs to be a balance. Yes, confirm that the developer has the specific skills needed by your company, but more importantly, make sure they have the Code Craft skills to not do more harm than good!

Would you hire a developer to work on your desktop graphical C# application if they only have Ruby on Rails experience? Would you change your mind if they could demonstrate great code craft and general language diversity?

This is a hard problem. How can a developer prove they have good code craft? Sadly, I don't have a good answer for that right now. I don't think certification is the answer. The closest thing I can think is their demonstrable participation in an open source project. I'm open to suggestions here and will continue to post my thoughts on the topic as they come.

Is Code Craftsmanship The Only Thing?

If code craftsmanship was the only thing that mattered to a company we would all be contractors … disposable drones brought in to do a task as needed. But we know this is not the case. The longer you work at a company, the more you should learn about how the company makes money. Who are their competitors? How do their products differentiate? Who are the key customers? How do customers use the products you create?

A developers value in a company is multiplied greatly by their awareness of these factors. The other part the performance evaluation and hiring criteria needs to take these factors into account. A very valuable software developer is one that:

Designs with product vision in mind
Identifies pending delivery roadblocks in a timely fashion
Researches new technologies, languages and tools applicable to our product mix
Helps improve our operational efficiency
Contributes to our competitive advantage

Software developers that write solid code and understand the business are your greatest assets. Guard them accordingly!

Your thoughts?

Enforce Idioms - Get Religion

2010-02-24T11:31:00.007-04:00

In the last series of articles we talked about how consistent coding style and great documentation can help a code base stay manageable. Beyond coding style and documentation, I think the most important thing Project 2 could have done would be to clearly define the coding idioms they deemed to be important.

Idioms and coding religion are a great way to boil down the things that the designers, architects and programmers of the original code base deem to be important. Some things are about how code should be written and others are related to business aspects of the project. Often they reflect, at a code level, what the functional specifications of the project are. They all should be addressed as early into the project as possible.

Perhaps you are making a web framework? What are the important characteristics that framework should exhibit. Look at how Django and Rails have defined their design philosophies.

What if you are making an asynchronous communications library? Look at what Ice deems to be important. Compare this to the Twisted library.

What if you are writing a programming language? Then perhaps a whole other set of things are important to you? Consider The Zen of Python.

Just will a little thought (and a couple of beers) I assembled this short list of things I view as important considerations when tackling a software project.

Low Coupling, High Cohesion
Less code
Quick development
Don't Repeat Yourself
Explicit is better than Implicit
Consistency
Prefer Stateless Classes
Separate Logic from Presentation
All Code must have Tests
Internationalization must be addressed in all cases
MVC
SOA (RESTful?)
Getting Real - https://gettingreal.37signals.com/
Convention over Configuration - No XML Situps
Principle of Least Surprise
Principle of Least Knowledge
What are the accepted licenses
What are the accepted libraries
What are the accepted languages
No commented out code
Remove dead classes
Compile Clean
Prefer Data Driven Design
Don't treat Exceptions like Booleans
Don't Mask Exceptions
Know your Exception Hierarchy
Prefer Aggregation over Inheritance
Prefer Event-driven over Multi-threaded (or vise-versa)
Prefer Reaping over Explicit Deletion
Prefer Immutable Data
Prefer Finite State Machines to Boolean Nests
No Switch Statements
No Singletons (includes Globals)
No “Regions” in C#
Avoid state / Offload state to the caller
Cache everything / Cache nothing
Prefer Lazy-loading / Lazy initialization
Don't make up new names for accepted practices

Now I'm sure some of you are looking at this list and going “Well, duh!” but don't laugh too soon. You would not believe how many of these common sense coding philosophies are violated daily. If anything these rules can serve as a training tool for new developers to learn the difference between good and bad code. Reviewing one ortwo of these during lunch sessions is a great way to ramp up your teams overall coding skills.

I'm sure, even without a few beers, you could add a ton of your own rules to this list. The c2.com website is a great place for ideas and discussion.

The question is … how many rules should you set?

The answer is, of course, it depends. If you are running an open source project and are looking to recruit more developers onto your project, then perhaps you don't want too many barriers to adoption. If you are in a corporate environment and give RCS commit privileges immediately to new hires (bad), you might want to set more ground rules.

Just be aware of one important thing: When you add or change a rule, you instantly incur technical debt! If it was previously acceptable to mask exceptions and then you set a project rule that exceptions cannot be masked, you just took on a chunk of technical debt. Your first step in adopting this rule has to be cleaning up the debt. Don't start building your church with broken windows.

If your programmers can follow and maintain an agreed upon set of programming idioms you are likely well on your way to producing a beautiful body of code.

Knowing this sort of thing comes from experience. Next time we will talk about developer skills and what the good ones seem to have in common.

Self Documenting Code ...

2010-02-09T12:26:00.003-04:00

I hope you read Jacob Kaplan-Moss's article "What to Write" ... because I'm not going to talk a lot more about that topic. He does an amazing job of explaining what users are looking for when they approach a code base as well as what a waste of time embedded documentation is.

My perspective on documenting code builds on Jacob's observations. As I've mentioned previously in Your Code Is The Other Team Member, code should read like a book. Your eyes should be able to scan the source unencumbered to glean the meaning. Don Knuth says code should be written primarily to communicate its purpose to humans [not to make the compilers happy].

In order to read like a book, code should be Self-Documenting. And by self-documenting I do not mean Doxygen or Javadoc. I mean, does the code clearly express its purpose in and of itself?

When you write code you need to constantly ask yourself "Am I sabotaging the readability of this code?"

How can we sabotage the readability of our code?

Don't document the literal code, but rather, the intention of the code

Who should I believe in this snippet of code?

// Do foo if check is True ...
if ( ! check_thing_has_occurred())
{
foo();
}

You just broke my brain. Now I've got to drill through the logic to find out if the comment or the code is right.

IDE Code Snippet Macros are an invention of the Devil

What value does this comment bring to this method?

/*********************
Detail: Constructor
**********************/
CCPasswordDlg::CCPasswordDlg(CWnd* pParent)
{
...

How about this?

/*********************
Detail: Sets the quality level based on a 1-10 variant
Pram: iQuality - quality setting (1-10)
Date: Sunday, April 06, 2008
By: Joe Developer
Note: Returns VIDEO_CODEC_SUCCESS if successful, VIDEO_CODEC_INCORRECT_PARAMETER otherwise.
**********************/
int CVideoEncoder::SetQuality( int iQuality )
...

So much of this fluff is auto-generated by some keystroke macro from the IDE. It's a complete waste of space. It breaks the flow of the code and creates more places for the reader to second guess themselves. Even worse, it's not even metadata for a doc generator. It's double junk.

Why do we need a date time stamp in the source? Why do we need the developers name? All that stuff is in the RCS. Don't duplicate it. Why do we need the /***...***/? Modern IDE's highlight method names perfectly. Isn't the bounds check more useful in the code? Can't we look at the source to see the return types? Sure, if this is a public-facing API, then perhaps the summary, variable information and return types are valid ... but not for internal code. Again, I would much prefer to see a working unit test or tutorial that explains how this method should be called in the context of a proper use case. That gives me far more benefit than an auto-generated html page will ever provide.

This is just the low-hanging fruit. The C2 wiki entry on Self Documenting Code gives an excellent summary of other things you can do to make your code more readable. Think about all of these points before you start sprinkling doc sugar all over your source.

If you focus on a consistent coding style and follow these rules for making your code base self-documenting you are well on your way to creating a code base that will out live your team or process. There is one more thing that really make it truly timeless ... establish a religion. We'll discuss this next time.

Your Code is the Other Team Member ...

2010-02-05T09:32:00.009-04:00

One of my big complaints with the code base of Project 2 was the complete lack of a consistent coding style guide.

I'm of the belief that code should read like a book. You should be able to look at a function, class or method and read it from top to bottom like a story. It should have a clear beginning, middle and end. How would you like to a read a book where the formatting was completely inconsistent and you had to struggle with the typesetting to get to the content. That's what happens when source doesn't follow a style guide. Your brain can't move up the semantic ladder because it's mired in syntactic noise.

In order to get to the goal of Collective Code Ownership, everyone needs to see things the same way. Formatting and style is the first step to this goal.

There are a many different code style guides already available. Most are for specific languages: C, C++, C#, Java, Python. While some are variations on these for specific libraries or packages: Gnome, Django, Rails, Linux. And others are company specific style guides like the Google C++ guide and Microsoft Hungarian Notation.

A good coding style should include more than just brace style, comment form and indent conventions, but also things such as:

Directory naming
File naming
Package/Module naming
Class, Function and Variable naming (CamelCase? all_lower? AbbrAllowed?)
Member variable referencing (m_ vs. _ vs. this.?)
Member variable ordering (public, protected, private? Top of class vs. End of class?)
Method ordering (alphabetical? public, protected, private?)
Import rules and organization (Should all external references be fully qualified? Are blanket includes allowed?)
Variable shadowing rules?
Versioning standards
Copyright and IP ownership notifications

These all need to be addressed! It's very hard to find one standard that will address all of these points for your specific project, so you can expect to roll up your sleeves. Put it on a wiki and have all the developers get notified whenever it changes. Try and pick one that is as close as possible to an existing, widely enforced standard.

With so many definitions it can be a little daunting to pick one. There is one rule you must follow: BE CONSISTENT. While we all have preferences on whether braces should go at the end of the line or the next line, it's far more important to pick one and stick with it.

I saw a great tweet the other day from @sbastn: "I am never going to use the word team unless I really mean it. The word group seems my best replacement"

Your code is your other group member. I don't think you can even think about transitioning from a "group" to a "team" until your code is playing the game too. So, how can you get your group to start considering your code as the other team member?

Many places I've worked have won't permit code reviews to start until the code meets the style guide. "Fix it and I'll come back." Personally I like that rule, but it can frustrate some developers. Enforcing code style can be a daunting, laborious, thankless task ... trust me. Your group is on its way to becoming a team when there is no enforcement required ... everyone just does it because they see the value in it. So, following the mantra of "Automate Everything You Can" (and knowing you can't automate everything) ... there are things you can do with respect to code.

Automated Reformatting

Some groups perform nightly automated reformat of the code base using pretty printers likePyIndent, Jalopy and AStyle. I don't like this approach. If the code changes after you have checked it in, it will look foreign to you when you next look at it. These tools are best used initially to get your legacy code to a good base state.

No Commit Until Compliant

With most revision control systems you can install pre-commit hooks that can verify code compliance first before letting the code into production. I prefer this approach because it makes the developer think about the code style as part of the check-in checklist ... right up there with making sure the tests are written.

Try to Convert the Legacy Codebase in One Fell Swoop

Picking away at a code base to make it match your new style can take forever. You may need to write some custom scripts to help with this (especially when renaming files), but formatting changes are one of the few modifications you can make to a legacy code base where behavior should not change. You have to get into the new code style mindset as quickly as possible and doing this piecemeal only makes it harder.

In summary, establishing and enforcing a coding style on your project should be one of your first steps to fixing a legacy code base. Use automated tools or scripts to help with this project. If you can't automate the effort ask developers not to make the problem worse and to help clean up the mess when they're in fixing something else.

Another thing your programming style guide should address is the documentation style. We'll go over this next time, but in preparation you should read this great series of articles on the topic by Jacob Kaplan-Moss: What to Write, Technical Style and, for large doc efforts, You Need an Editor.

Before sitting down to Eat the Elephant ...

2010-02-02T14:55:00.003-04:00

In 'A Tale of Two Code Bases' I described two legacy code bases and how one (Project 1) was a pleasure to work on and the other (Project 2) was a nightmare. I promised to follow up with some hints for what Project 2 could have done to make their code more manageable.

About a year ago I was looking into wind generators and solar devices for my house ... it can get cold in Nova Scotia. The store owner told me "get your house fully insulated first, then we can talk about efficiency". Great advice. The same applies for Code Management. I think "The Joel Test" is a great litmus for whether your house is insulated enough. You shouldn't be struggling with bad source control/development tools, slow development PC's or perpetually looming deadlines before you go into this. You should also have some degree of buy-in from your superiors that you can regularly dedicate a little time to making your code base better. The intention is not to upset the apple-cart, boil the ocean or reinvent the wheel (choose your analogy) ... but rather Eat the Elephant.

I don't want to rehash topics that have already been exhaustively covered, such as TDD or the particulars of writing effective code in a given programming language. Well, I don't want to rehash them in any great detail. My intention is to make these posts about Effective Code Management and not about software development methodologies. That is, not about requirements gathering, feature definition, scheduling or estimating. I feel strongly that without a solid code management foundation whatever development methodology you use will fail over time.

So, I'll assume you've read one of the hundreds of TDD books out there and at least one of the "Effective [Foo]" books out there that apply to the programming language you use daily. C++/STL, C#, Java, etc. And, because I'm going to assume you will be working with code that is already in the field, I'll also recommend Michael Feathers 2004 book "Working Effectively with Legacy Code" which gives some nice guidelines for anyone faced with an elephant-meat diet.

Next time, we'll take our first nibble with a look at coding style guides ...

A Tale of Two Code Bases

2010-01-26T10:03:00.002-04:00

On a recent project, I inherited a rather large code base. The developers had long since moved on and all the in-house knowledge of the application left with them. Before I could even really get down to business I had some logistical problems to work out. The source control system in place wasn't accessible from my remote location and the build system was very dependent on the network layout. The code had been under development for nearly 10 years and was a mix of PC-based code and cross-platform shared code with the firmware for several hardware products. It was pretty hard to know where one product ended and another began. It was a mix of C++, Python and C. And this code was very technical. Lots of concurrency, thread management, parts made heavy use of COM, rendering stuff, low-level bit-twiddling stuff, you name it.

But, the code was neat. There was a consistent coding style. It was well structured with clear delineation between modules. In several places there were API's established for third-party integration. There was an architecture and good attempts at getting some automated tests in place. Pattern names were used and standard development idioms were followed. Sure, much of it was organic growth and you could see where the developers had some crosses to bear, but there was clear evidence of intelligent life and a passion for the code base. All in all, it was pretty darn impressive.

One of my first tasks was to get the code running again. I needed to get it moved over to a new version control system and then get it working with the latest compilers and libraries. From there I could work on getting the much needed new features in place that the the customer needed. It was a big task, but it was fun to dig through a nice code base and see how the previous people did it. It was like discovering a nice old house and walking through the rooms to admire the craftsmanship without a guide.

With that one under my belt, I got the "lucky" task of being given another legacy code base. This one wasn't as big as the previous. It wasn't as technically complicated either. Yes, there was multi-threading, interprocess communications and some tricky GUI stuff going on there, but all in it nothing that you couldn't wrap your head around. It was just C++ with MFC. Very little COM. Windows only. I had a little bit of guidance from a couple of the last developers to work on it, but once they left I was really on my own to understand all the nooks and crannies of the product. Again, I had to move it over to the new source control system and update the compilers and libraries and make the build system less rigid.

But this second code base was a mess. There was no coding style. Variable names were cryptic with a mishmash of Hungarian Notation forms. Every class (when classes were used) was tightly coupled to each other. There was no modularity to the architecture. There was no structured exception handling, no provisions for automated tests, no organized logging to speak of. Constants changed. Code was copy pasted everywhere it was needed. One feature I needed to remove, which was essentially a boolean check, touched over 40 files across four applications. Data lived everywhere and had no core data dictionary of usage, so you had to trace the code to find out what the intended use was.

Basically, the second code base was buried in technical debt.

This is not to blame the developers or the management. There are no fingers to point here ... there's no one left to point them at anyway. It's like an archaeologist stumbling on a lost city and learning that the citizens died of a common disease ... it wasn't their fault. But we now know that, with some basic medicine and health practices, it could have been avoided.

I'm an Agile guy. I love the innovations in the software development process of the last ten years. But this experience opened my eyes to where the real work of software development needs to be done ... in the code. I have no idea what the development methodologies used were when these code bases were being developed. Perhaps they were Waterfall? Perhaps they were some Scrum/XP hybrid? Who knows? Who cares. The code is the only thing left that matters. Developers need to remember that the process, the methodology, should not take precedence over code craftsmanship. Don't get so mired in the busy work to lose track of what you're doing in the first place ... writing code. The code is paramount.

I'll follow up this post soon with some of my other insights on this experience and recommend some steps that could have been taken to help Project 2 out.

Installing IronPython 2.6RC1

2010-01-08T10:37:00.002-04:00

I'm not a .NET guy at all. I'm in the process of working with a third party C# library, but would rather use IronPython instead of C#, so I tried to install IronPython 2.6 RC1 ... it didn't go well. The installer would get to about 90% complete, but then fail and rollback without any info as to what went wrong.

By manually running the .msi installer we can get a log file:

msiexec /i IronPython-2.6.msi /log install.log

The offending operation seems to be:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\ngen.exe install C:\Program Files\IronPython 2.6\Microsoft.Scripting.dll /queue:1

The /queue parameter is to kick off the .NET Optimizing compiler service ... but this seemed to be failing. I could verify this when I ran the ngen.exe command directly:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727>ngen queue status
Microsoft (R) CLR Native Image Generator - Version 2.0.50727.3053
Copyright (c) Microsoft Corporation. All rights reserved.
The specified service does not exist as an installed service. (Exception from HRESULT: 0x80070424)

Hmm, where's my JIT?

Turns out my .NET Runtime Optimization Service v2.0.50727_X86 service is either not running or not installed. In my case it was not installed. Very odd as I have .NET 2.0 -> 3.5.1 installed and everything else works. Oh well.

After some poking around I found you can register the service manually via:

%WINDIR%\Microsoft.NET\Framework<64>\v2.0.Number\mscorsvw.exe -i

Or, in my case, C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727>mscorsvw.exe -i

That got the service installed and the IronPython install worked fine afterwards.

Hope this saves someone some time.

Interfacing multi-threaded C API's with Python using SWIG ...

2009-10-08T13:52:00.004-03:00

Ran into an interesting problem yesterday. There is a third party C library that I need to interface to using Python. Normally, this isn't a problem. Just use SWIG to make a Python Wrapper on the desired API functions and call away. But this library was a little nastier because it spawned other threads and notified the caller via callbacks. Callbacks on their own aren't really a big deal for SWIG, you just have to handle the delegation from C to Python yourself, which is easy to do in the .i file. But when the callback is coming from another thread there are some twists.

CPython is notorious for the Global Interpreter Lock (GIL). You have to release the GIL whenever you do blocking I/O and re-obtain it when you are done. With a callback it's a little different, you want to acquire the lock, call Python and release it again when your back in C-land.

Starting off, my .i file looked something like this:

%module probe
%{
#include "myapi.h"
%}

%include "windows.i"

typedef void __stdcall (*HandleDataCallBack)(char*, int);

int Start(HandleDataCallBack);
int Stop();
int SendInquiry();

%{
PyObject* python_callback = 0;

void __stdcall api_start_callback(char* foo, int blah)
{
PyObject *arglist;
PyObject *result;
PyGILState_STATE gstate;

if (!PyCallable_Check(python_callback))
{
PyErr_SetString(PyExc_TypeError, "Python callback must be callable");
return;
}
arglist = Py_BuildValue("(si)", foo, blah);

gstate = PyGILState_Ensure();

result = PyEval_CallObject(python_callback, arglist);

Py_XDECREF(arglist);
Py_XDECREF(result);

PyGILState_Release(gstate);
}

void alt_start(PyObject *pyfunc)
{
python_callback = pyfunc;
Start(api_start_callback);
Py_INCREF(pyfunc);
}
%}

void alt_start(PyObject*);

Essentially what I'm doing here is defining a new function alt_start() to replace the API Start() function. A well known callback (api_start_callback) is then handed off the the real Start(). api_start_callback() does the heavy lifting of assembling the arguments, acquiring the GIL, calling the Python callback and then giving the GIL back to Python.

For other places where the threading is an issue, SWIG can take of this itself when you run Swig with the -threads option. In my case: swig -python -threads probe.i

The simplified Python code for calling this looks something like this:

import probe
import time

def ProbeCallback(foo, blah):
....print "Foo: %s, Blah: %d" % (foo, blah)

probe.alt_start(ProbeCallback)
probe.SendInquiry()
time.sleep(60)
probe.Stop()

Someday perhaps, I'll be able to call Start() directly and pass my Python callback handler without the need for this special handling code. Until then, the work around isn't too horrible.

Microsoft VC++ SxS Assemblies & Manifests

2009-08-14T09:29:00.004-03:00

This is true madness. In the process of bringing a legacy VC++ project up to date with Visual Studio 2008 there were some real hurdles to get over.

The biggest issue is that starting with Visual Studio 2005 Microsoft introduced the idea of Side-by-Side Assemblies and Manifests. This is a mechanism whereby different versions of the Visual Studio C runtime, ATL libraries, MFC libraries and other assemblies can be run "side-by-side".

So, while your program may compile and run in debug, it may not be able to be copied to a system with no dev environment on it without serious tap-dancing. You'll get cryptic "The application is not configured correctly and should be reinstalled" messages on XP or a vague hint about SxS on Vista.

After a lot of time spent on the MSDN forums I finally figured out how this mess works.

First off, VC++ apps will include \Program Files\Microsoft Visual Studio 9.0\VC\include\crtdefs.h via Stdafx.h. Look at this header file. It contains lines like this:


#include <crtassem.h>

#ifdef _M_IX86
#ifdef _DEBUG

#pragma comment(linker,"/manifestdependency:\"type='win32' " \
      "name='" __LIBRARIES_ASSEMBLY_NAME_PREFIX ".DebugCRT' "         \
      "version='" _CRT_ASSEMBLY_VERSION "' "                          \
      "processorArchitecture='x86' "                                  \
      "publicKeyToken='" _VC_ASSEMBLY_PUBLICKEYTOKEN "'\"")
#else

#pragma comment(linker,"/manifestdependency:\"type='win32' "            \
      "name='" __LIBRARIES_ASSEMBLY_NAME_PREFIX ".CRT' "              \
      "version='" _CRT_ASSEMBLY_VERSION "' "                          \
      "processorArchitecture='x86' "                                  \
      "publicKeyToken='" _VC_ASSEMBLY_PUBLICKEYTOKEN "'\"")
#endif

#endif /* _M_IX86 */

This will put a little piece of code in the .obj which gets picked up by the linker and the manifest tool. So, if _DEBUG if defined anywhere, you'll get manifest entries requiring the debug runtimes. So, double/triple check that you don't have any debug files or .libs in your project. The .manifest file looks like this:


<assembly xmlns="'urn:schemas-microsoft-com:asm.v1'" manifestversion="'1.0'">
<trustinfo xmlns="urn:schemas-microsoft-com:asm.v3">
  <security>
    <requestedprivileges>
      <requestedexecutionlevel level="'asInvoker'" uiaccess="'false'">
      </requestedexecutionlevel>
    </requestedprivileges>  
  </security>
</trustinfo>
<dependency>
  <dependentassembly>
    <assemblyidentity type="'win32'" name="'Microsoft.VC90.CRT'" version="'9.0.30729.4148'" processorarchitecture="'x86'" publickeytoken="'1fc8b3b9a1e18e3b'">
     </assemblyidentity>
  </dependentassembly>
</dependency>
<dependency>
  <dependentassembly>
    <assemblyidentity type="'win32'" name="'Microsoft.VC90.MFC'" version="'9.0.30729.4148'" processorarchitecture="'x86'" publickeytoken="'1fc8b3b9a1e18e3b'">
  </assemblyidentity>
  </dependentassembly>
</dependency>
</assembly>

The big thing to note is the version of the runtime. How does Visual Studio know which version of the runtime to load? Well, it uses \Program Files\Microsoft Visual Studio 9.0\VC\include\crtassess.h to make this decision. Whenever you install a new Service Pack for Visual Studio this file gets updated.

If you define _BIND_TO_CURRENT_VCLIBS_VERSION = 1 in your C++ code, crtassess.h will grab the latest MFC, ATL and CRT libraries, whatever they are. These are the versions that will go into the .manifest file (by the way, set up your code to embed the .manifest in your code to keep things easier).

On the deployment machine, you need to install all the Visual Studio runtime libraries. The best way to do this is by running the latest and greatest vcredist_x86.exe application from the microsoft site. But this has its own problems.

The entire version number of the assembly is important. With VS 2008 SP1, the CRT version was bumped from 9.0.21022.8 to 9.0.30729.4148, but the redist for SP1 installed 9.0.30729.1 and the program wouldn't run on the deployment machine. Turns out there was a Security Update for SP1 and a later redist program for it. I don't know how you're supposed to keep up on this other than watch the msdn forums. Once this was run on the deployment machine the program would run.

Another problem I ran into was the compiler kept putting entries in the manifest for the older 9.0.21022.8 crt even though I had _BIND_TO_CURRENT_VCLIBS_VERSION defined everywhere. The workaround for this is very cryptic. You need to put the following code somewhere in your application (like in stdafx.h)


extern "C" { 
__declspec(selectany)       int _forceCRTManifestRTM;
__declspec(selectany)       int _forceCRTManifestCUR;
};

This will force the compiler to use the latest assemblies.

You can see how the manifests are looked up on your deployment machines by looking in \Windows\WinSxs. You'll see the mapping of the runtime to the .dll's along with the security policies for selecting them. Using this you can quickly spot any discrepancies between your development environment and your deployment environment.

Some Changes ...

2009-08-13T11:20:00.002-03:00

I've nuked my old blog. Since all my family photos and ramblings are on Facebook I'm going to have this site focus on public industry related topics. Stay tuned.