Mapping (Multiple) Legacy Databases With Datamapper

We're starting in on a new project at work and it comes with some unique requirements (don't they all?)

To get things going so the rest of the team can take over and run with it I spent a couple days figuring out Datamapper the ruby ORM and mapping a number of legacy data tables.

This application is unique in that it will run against 3 separate databases. Two of them come from an older version of the application that the client built several years ago, we'll call these databases db1 and db2. The third database is where any custom data will be held for the web interface itself. We'll call it web1.

The first trick to getting datamapper to work in a rails applications is knowing what to actually install. Datamapper is a general purpose ORM and can be used quite effectivley outside of rails, as such the documentation on datamapper.org doesn't really focus on rails. No problem, take a look at the dm-rails gem.

dm-rails overrides a number of ruby on rails default generators, patches rake db:migrate to work with datamapper upgrades and automigrates and even integrates well with rspec and cucumber. Also worth noting that it worked fine on the latest and greatest (as of this blog post) - Rails 3.1.3 with Ruby 1.9.3.

Part of setting up dm-rails is getting your database.yml configured correctly. The documentation on the github page is great but as an added example here is a version of my config (with secure info removed.)

db1_defaults: &db1_defaults
  adapter: mysql
  username: [USERNAME]
  password: [PASSWORD]
  host: localhost
  database: db1

db2_defaults: &db2_defaults
  adapter: mysql
  username: [USERNAME]
  password: [PASSWORD]
  host: localhost
  database: db2

web1_defaults: &web1_defaults
  adapter: mysql
  username: [USERNAME]
  password: [PASSWORD]
  host: localhost
  database: web1

development:
  database: web1
  <<: *web1_defaults
  repositories: 
    db1: 
      <<: *db1_defaults
    db2: 
      <<: *db2_defaults

test: &test
  database: web1_test
  <<: *web1_defaults
  repositories: 
    db1: 
      database: db1_test
      <<: *db1_defaults
    db2: 
      database: db2_test
      <<: *db2_defaults

Once your database is configured you should be able to run rake db:migrate and see success messages. At this point you have no models configured but you can see that it can connect correctly.

The next step is mapping your existing tables.

db1 and db2 have quite a bit of existing data, their own data formats and a somewhat inconsistant naming structure. This resulted in a lot of custom mapping and eventually I just decided to explicitly spell out every mapping for consistency (even if the datamapper conventions would have applied.)

Here's an example model file. Again I've removed business context by renaming some things. I'll call out some pitfalls after.

class Procedure
  include DataMapper::Resource

  def self.default_repository_name
    :db1
  end
  storage_names[:db1] = "procedures"

  property :id, Serial, :field => "id"
  property :name, String, :field => "procnam"
  property :length, Integer, :field => "avgminlen"

  validates_length_of :name, :max => 25
end

The first important thing to note is this method declaration.

  def self.default_repository_name
    :db1
  end

This is how you tell Datamapper which database this model's table is in. In this case Procedures come from the db1 database.

Next thing worth noting is:

  storage_names[:db1] = "procedures"

Which is how you set the table name. In this case the table was pluralized. If you find that all of your tables follow a naming standard (something like tblProcedure) there are ways to override the naming globally. In this application naming standards were fairly inconsistant so I ended up adding this line in every model even when it was technically unnecessary.

Also for each varchar type column I added a length validator. The length of the varchar varied wildly in my application from 1 char fields up to 50 varchar fields. The best way I could come up with to deal with that was to simply validate that it never overflowed. Would love suggestions on any other config flags I could add if need be so datamapper knows what it is structurally.

One last thing to note. The override to default_repository_name MUST BE BEFORE THE PROPERTY DECLARATIONS. Sorry for the all caps. I spent about 2 hours fighting datamapper before figuring that one out. I even made a SO post on it before i figured it out.

Codemash 2.0.1.2

2012 marks my fourth year at Codemash. I've written about this conference before but in short, if you don't go to Codemash you should. For your (or your companies) dollar it is the best value you will get. If you run a company you should send your people to Codemash, or sponsor it, maybe even host a bacon bar (thanks DI).

So the promotion is out of the way, what I really want to do with this year's recap is talk about each of the sessions I attended because they were all phenomenal.

The Scala Koans - Dianne Marsh & Daniel Hinojosa


The first precompiler was a half day session that covered the Scala Koans. The Scala Koans are a Scala implementation of the Ruby Koans.

Koans, in the context of computer programming, are "short, enigmatic stories that usually involve some kind of epiphany. (Jim Weirich)." The idea of a language koan exercise is that by completing a series of short questions you will learn the core of a language. Jim Weirich and Joe O'Brien really started this approach to leaning programming languages with the aforementioned Ruby Koans but now implementations exist in Scala, Objective-C, C# and many other languages.

Scala is a pseudo-functional, pseudo-object oriented language that has static typing with a dynamic style syntax. So.. it's a Mutt, but a very interesting Mutt. As a ruby / C# dev I saw a lot of promise in having the power of a runtime like the JVM and static typing with the comfort of ruby-style syntax.

If nothing else it's worth checking out the Scala Koans to get a really quick overview of another language.

Advanced Patterns with Ruby on Rails - Jeff Casimir

This precompiler was targeted at an existing ruby developer with a few projects under their belt that wanted to learn more about applying various design patterns or concepts to their ruby (and rails) applications.

Among the topics covered was isolating business logic and limiting law of demeter violations with Draper by using the decorator pattern.  Also slimming down controllers by refactoring calls to activerecord into one method and limiting the number of instance variables used in the view.

Keynote: Rethinking Enterprise - Ted Neward


Thursday's morning keynote focused on how enterprise software needs to reevaluate what it values. How reuse is no longer as significant as it was in the past, how we should be not only looking at the tools we are comfortable but looking to find the best tool even if it isn't part of out "enterprise stack." It warned agains being technist "This platform is always better than that platform."

It's the Little Things - Brad Colbow


This talk focused on the little issues in UX design and how when put together they can add up to larger problems with your applications. He used the example of the Cleveland Public Library (and really Overdrive.) He previously did a comic strip to express his frustrations with their interface for downloading audiobooks which became quite a popular strip. How taken individually the issues he ran into were minor but as a collection completely infuriating. He advocated small changes like remembering to turn off autocorrect or autocapitalize on a login form for iphone users.


Brad also talked a bit about his experience building an iPad magazine and how so many of the other iDevice magazines tend to make bad assumptions about their audience and use custom gestures that are confusing and easily forgot.

Building Windows 8 Applications with HTML5 and jQuery - Rich Dudley


Rich covered the new Windows 8 developer preview and how to use HTML5 to craft a metro style application that can be deployed via the Windows Store.

Regardless of what you think about Windows 8 I'm really excited about the prospect of using HTML5 and CSS to layout the UI of a client application. I've always thought that Microsoft's previous attempts to handle UI layout were really poorly implemented. HTML in my mind is a more intuitive way to make a UI and I prefer working at a markup level than using a designer (yes i know you could layout xaml in an text editor too.)

It looks like Microsoft is serious about not breaking the standards based nature of the markup too, which is encouraging. Looking at the examples shown they make liberal use of the data- attributes in order to bind data and actions but all of that is standards compliant. I saw no custom metro only tags only attributes and data attributes. WinRT is implemented through a javascript layer that can be granted system level access.

A number of precautions are taken with the applications including making the developer select one of two modes, essentially net or client. Net mode sandboxes the application and it's data and prevents access to most system resources. Client mode provides system access but isolated most network calls. The apps run like mobile apps, they do not remain in memory when switched away from but instead must have their state stored and recovered when focus changes.

In short metro looks promising, especially for non-.net devs who want to build something on windows but don't want to get too far out of the comfort zone of web development.

Mastering Change with Mercurial - Kevin Berridge

So to be perfectly honest I wasn't going to go to this talk except Kevin's an old college friend of mine and I wanted to be there to support his first talk at a larger conference like Codemash. Before I left though I was completely blown away and realized I was very much being a technist like Ted Neward had described.

See there's a bit of a git / mercurial rivalry. Kind of like vim / emacs or other opposing tools. Developers love rivalries and I very clearly have fallen on the git side. Truth be told though that has more to do with Github being awesome than git itself. So i spent most of Thursday harassing kevin (jokingly) about mercurial and asking if he was going to admit at the end of his talk that git was simply better. Instead he came out swinging and knocked everything I knew about distributed version control on it's proverbial ass.

Mercurial is awesome... dammit... it pains me to admit it and then publicly post it in this blog but it really is.

So first off kevin did the best "intro to the thing I'm about to talk about" that I've ever seen. Most "advanced" talks end up spending 30 minutes doing an intro and barely ever get into advanced content. Kevin played a minute long video and narrated it and if you had any experience with distributed version control (including git) you were now up to speed. Seriously i basically never used hg but the video was all i needed to make the connection to my normal workflow in git.

He then talked about stuff like viewing and searching history, simple aliases or tricks that can make your life better, and ended with some of the coolest history management stuff I've ever seen using hgs queue's extension. I may also be convinced that hg's way of labeling branches (by default) is better than how git does it.

The plan now is to get Kevin's slides, convert them to git and then jointly present the topic at burning river devs as how to do these things in both worlds.

CoffeeScript is for Closers - Brandon Satrom

Brandon's talk was an honest evaluation of why Coffeescript is worth looking at. He made every attempt to avoid the CoffeeScript > Javascript false debate people tend to have online. Instead he made a number of points as to why, even if you stick with normal JS for your daily work, you should give coffeescript a shot. The argument that best resonated with me was that using CS and then looking through the JS code is compiles into you can get an idea of how to be a better JS programmer. CS doesn't compiled into impossible to read minified JS code, instead it compiles into very readable code that makes every attempt to follow best practices. If you're a bit of a JS noob it might be worth rewriting some of your stuff in CS, compile it and then see how it turns out.

Dealing with Information Overload - Scott Hanselman

So i went to this talk because Scott Hanselman was speaking. I really didn't even read the title or abstract. It was added as a talk kind of last minute because of some other cancellations.

Scott is a freaking great presenter, which makes sense as that's what he does. This talk could have been a keynote. It focues on real world hacks to make your life easier to handle. Things like not answering email until a set time so that you can establish the precident that you work in the morning when your most "fresh." One hack I actually did right away was setting up an inbox rule to redirect email where I am cced into another folder meaning I focus on the email that I am directly referenced on.

Be The Input - Kinect With Your Computer - Ben Barefield

Ben's talk was on the Microsoft Kinect API for windows. Ben was lead dev on a small racing game that uses the Kinect sensor to control the carts on the screen. His company, SRT Solutions, had the game on display at their booth.

The thing that most surprised me about this talk was how easy it is to get up and running with both Kinect and the XNA platform for game development.  Ben was able to build a small test app that tracked a user's hand on the screen using Kinect in about 20 minutes during the talk. A really impressive demo and now I need to learn a bit more about this platform.

Everything Else


Some of the best parts of a conference like this don't happen in the session but rather in the side conversations you have with friends, peers and strangers in the hallways. I learned about dynamic vs static typing on a more fundamental level, closures, JVM backed languages, the logistics of spinning up dynamic servers with chef and vagrant and a million more things meeting with people in the halls and chatting over beers at the evening parties.

Codemash is worth it for just the inbetweens. The 20 mins between sessions and the events that occur between Thursday and Friday. You can't possibly make it to everything but the joy of codemash is you can always go next year.




Edit: Left off the speakers for the precompilers. Sorry.

Yet another move

So my year with Squarespace is up and they are coming knocking to renew. I don't really want to pay for a blog I update maybe 3 times a year so I'm moving this all over to blogspot.

I migrated all the posts (manually as the export thingy didn't work). I even moved over comments that weren't spam. You'll notice a lot of "Originally from ...." if you look at the comments. There's no way for me to add a comment on someone's behalf it seems so I just added them all myself.

Hopefully I'll have more to say soon.

Mocking Out Omniauth for Cucumber Testing

A few months ago some friends and I got together for Cleveland Startup Weekend and started building our "next big thing" DreamKumo. 

We had a couple challenges. First we knew from the get go that we wanted the site to be 100% facebook integrated. That meant implementing Facebook Connect and the Facebook Graph API. We also are a group of people who throughly believe in ATDD, TDD and agile development principals (and practices) in general. This led us to building up cucumber features for every area of the site. 

We hit a snag though when we tried to cuke our login process. We're asking our users to login through Facebook connect, which means when they click the login button they are redirected to facebook.com, and then sent back to us. Implementing the requirement was pretty simple using the omniauth gem with a little help from the folks at Rails Rumble. Cuking it is another story. Since facebook connect actually redirects the user we can't mock out the request using fakeweb and we don't want to cuke what happens on Facebook.com since 1) we can't control it, 2) it isn't consistent (sometimes you get a screen to accept the permissions that your app is requesting, sometimes you get the facebook login screen and sometimes you get redirected straight through) and 3) we don't want to setup a dummy facebook account for our cukes to use. 

So we needed to prevent the site from redirecting to facebook, and dummy up the site cookie so it looked like we were authenticated. We also needed the failure case to test what would happen if our user was sent to facebook and they decided they didn't want to grant us permission to their account. 

Enter monkey patching. 

Ruby has this wonderfully powerful ability to override the native functionality of a method or class at runtime. It's part of it being a dynamic language and it let's you do some really cool things, like manipulate the functionality of any class, including library classes like those in omniauth, at any time. So we dug through omniauth's code to find the point where it did the redirect to facebook and mocked out a response. We then had it immediatley redirect back to the callback url, or the url in our application that responds when facebook redirects back to us. So the result is we never hit facebook and omniauth sends along what we need to determine if the authentication was successful or not. 

Ok enough talk - here's the code. 

These steps are in a generic_context_steps.rb file in our step_definitions

 

Given /^I have valid facebook credentials$/ do
  module OmniAuth
    module Strategies
      class OAuth2
        def request_phase
          redirect callback_url
        end
        def callback_phase
          @env['rack.auth'] = {"provider"=>"facebook", "uid"=>"123", "user_info"=>{"first_name"=>"John", "last_name"=>"Doe", "nickname"=>"JohnDoe", "email"=>"john.doe@test.com"}, "credentials"=>{"token"=>"abc123"}}
          call_app!
        end
      end
    end
  end
end

Given /^I have invalid facebook credentials$/ do
  module OmniAuth
    module Strategies
      class OAuth2
        def request_phase
          redirect callback_url
        end
        def callback_phase
          fail!(:invalid_credentials, nil)
        end
      end
    end
  end
end

We simply call those in any feature where we want to authenticate the user and it continues on with the test as intended. 

There is a slight ATDD tradeoff. You aren't actually testing everything exactly as your user would see it since you're skipping the Facebook part. But In this case we borrow a bit from our TDD book and justify this by saying we only test our own code, and we didn't write facebook.com.

Hope this helps. 

 

Codemash 2011 Recap

In mid January I attended Codemash 2011. Codemash is a semi-regional software conference that, in my opinion, holds its own against any of the larger, more funded, and considerably more expensive to attend conferences that I have been to in years past. 

I say semi-regional because while the conference is mostly geared toward the mid-west attendees ultimatley come from all over the country and several are international. 

Every year a great deal of care is put in by the organizers to ensure a diverse selection of topic spanning multiple technology stacks and technical communities. I usually split my time between reinforcing the skills I find valuable in my own career and looking into new (to me) technologies that I may not have given enough attention to in the past. This year my "comfort zone" sessions were on git version control, ruby on rails and a couple talks on utilizing agile in somewhat untraditional ways. The "new" stuff for me was a fantastic talk by Gary Bernhardt that was really on the value of Bash and Unix (like i said "new"), android development and javascript BDD. 

Gary's "A Modern Open Source Development Environment" talk was the most interesting to me. His overview on the tools and utilities he uses to code efficiently brought to light a number of areas i could stand to improve in on unix, oddly identifying your inadequacies is a common trend when watching Gary code - and I say that with the utmost respect. In the talk gary cited a study that indicated the usefulness of a test suite is inversely correlated to how long it takes to run. It inspired me to go back and look at how I could make our test suite faster on a .NET project I'm on. 

Jim Weirich's session on writing and maintaining better tests brought answers to a number of questions I have about test suites in the projects I write and maintain. It also introduced a few new tools I didn't know existed in the Ruby testing space.

Jon Stahl's Agile Enterprise talk was fantastic as well. The general concept being that enterprises have started to adapt agile for their development and IT teams but a lot can be gained in applying many of the same principals of transparency, collaborative work flow and open communication to executive level management and management in general. Organizations Jon has worked with now hold standups for their managers, post information publicly including such sacred and secret information as salaries and perceived level of experience. Most importantly though he highlighted the fact that practice by itself does not make an organization agile. Agile is a combination of culture and practice and without both it really doesn't work. 

Of the best keynote presentations, thus far (there is one more to go as i write this) Chad Fowler's keynote on process, apprenticeship and offshoring was the best. A lot of the topics were taken from his book The Passionate Programmer (which I previously reviewed) 

Scott Chacon from Github also gave a exceptional keynote on how an open source business operates, giving examples from Github's organization and how it has helped them see success. In many way's Github is the anti-Agile shop. That isn't to say they are rigid or slow to react, quite the opposite, that respect they remain agile but they are very much not Agile. Cap-A Agile usually indicates a very specific process which in itself is designed to make change easy, but still manages projects in a careful way. Think Scrum, XP, Kanban. Github simply lacks established and firm process. They don't commit to dates, they don't follow any sort of project roadmap and they don't hard assign developers to projects, their teams are purely self-organizing. So.. github is agile, the right kind of agile without being Agile. 

2011 Goals

Well a week in and I'm off to a great start. Said I'd post a follow up on Wed the 5th, it's now Wed the 12th. Awesome. 

So as mentioned in my last post, in the beginning of 2010 I posted a list of goals for the coming year. That list became a guide for the year, some worked out, others didnt. Either way I found value in the post so to start 2011 it's time to do it again. 

So first off:

Renewed Goals

1) Update this blog twice a month. 
2) Seal the floor in the garage
3) Pay off the credit card(s)
4) Take a vacation outside of a border state 

All of these are pretty much the same as last year. 

Ammended Goals

1) Continue speaking, but more importantly prepare a talk that can be presented multiple times for multiple audiences.

This will probably be an intro to ruby or rails talk which I can give for work and possibly the CleRB group if we want to do another newbie night sometime soon. But If i can polish it and there's still interest in that basic of a talk I'd be happy to present elsewhere.

2) Learn some new stuff in .NET

This years list: Fluent NH or ActiveRecord NH, Dynamics and the DLR, MVC3, Cuke4Nuke or Specflow

3) Diet plan. I'm working with a personal trainer it's time to get better at nutrition. I'm terrible at it (food is kinda amazing.)

New Goals (Professional)

1) Build a series of blog posts or a mini-book on using the techniques i've learned in Ruby development and applying them to .NET

2) Be a good team lead. I have 2 guys working with me now and it's pretty new. So far it seems to be going well but theres a fine line to walk between being adamant about quality and being a jerk. Hoping I can manage the tight rope. 

3) Get serious about javascript testing. It's a first class and hugely important language and it's irresponsible of me to spend a ton of time testing c# or ruby or php and not pay attention to javascript. 

4) Get serious about ATDD. For .NET projects it will be cuke4nuke or specflow (or possibly cucumber but probably not), for ruby its cucumber. 

5) Finish and launch Dreamkumo. More about this one at a later date. 

New Goals (Personal)

1) Get a budget in order and start tracking expenses. I've tried this a couple times using mint but so far I've given up after a short period of time. Need to get the wife on board with this one cause its a bit bad to not really know where your money is going. 

2) Finish the upstairs flooring. Seriously they are plywood floors right now, kinda embarrassing if this doesn't get done in 2011 there's seriously something wrong. 

I think that's all for now. Time to get back to codemash (another upcoming post.) 

 

2010 Wrap Up

So at the end of 2009 I posted a list of goals I wanted to accomplish in 2010. This post serves as a recap of what I accomplished and didn't accomplish. Tomorrow I'll post my goals for 2011. Yes it's a bit cliché but it's good to have a picture of what you want to do. 

2010 Recap

1) (Partial Credit) Update this blog with a new topic at least twice a month. I managed this for about 3 months, then it vanished. As usual when the schedule got hectic this fell by the wayside. So this goal gets renewed for 2011. I just moved the site over to squarespace (and i'm loving it) and really want to keep it useful.

2) (Success) Speak at something, anything, and codemash doesn't count. I pulled this one off. I spoke at CleRB once this year about geolocation in rails and also pitched at Startup weekend. Not huge events by any means but its the start of public speaking. I need to try and get out there more this year, but its surprisingly hard coming up with interesting topics. 

3) (Success) Continue developing a Kanban approach for consultancies and apply it to my current job. Well so kinda. I didn't pull this one off at the "current job" but i did switch jobs and have moved to a company where helping establish Kanban was one of my first priorities. I'm very happy to report we now have a working process with a mostly remote team and all seems to be going well. My goal is to expand on this in 2011 and bring in more Agile techniques to our consulting practice. 

4) (Success) Learn some of the new development techniques and technologies in the .NET world. So of my original list I can say that I now have experience in NHibernate, Fluent NHibernate, WPF and an academic knowledge of MVC. I'm confident in the next few months MVC will move into the "very comfortable" stage. In addition to that I've learned quite about about MVP pattern and EF4.

5) (Success) Continue to get to know Ruby. I'm really proud of the experience I've gained in ruby this year. It's truly gone from a hobby to something I am comfortable using professionally and recommending to clients. The specific bullet points from last years list aren't necessarily all filled in yet but overall I'm happy standing in as a defacto ruby expert on projects. Goal for 2011 is simply keep going, i love the platform more and more as i use it and there are some great rubiests in cleveland to learn from.

6) (Complete, not really success) Get things squared away with Greenfield Studios. Well Greenfield Studios is no longer in use but for quite a few months i used the name Greenfield Consulting (and technically still do) for side project work. The new gig has brought a bit more intensity so I've stopped doing side work for now but things are straight on this one. 

7) (Epic Fail) Finish EventCasts and one other side project. Eventcasts was shuttered and the codebase made open source on github. Started a new project, Questering, but that too has languished. Now I'm working with a fantastic team on the project we started for Startup Weekend - Dreamkumo. This one seems to have legs, we'll see how 2011 goes.

House /Personal Goals

These didnt go so well

1) (Fail) Finish remodeling the basement Nothing has really happened in a year. The tiles up but that's about it. Too much time & month for right now. 

2) (Fail) Seal the floor in the garage - I forgot about this one

3) (Depressing Fail) Take a vacation somewhere outside of Ohio or any state that borders Ohio. This is my most depressing failure. In 2011 we are going to Boston for a friends wedding so that could serve as a kickoff for a decent vacation though, we'll see.

4) (Consistent Fail) Pay off my friends at Discover Card. Ya.... i'm going to be paying these people forever. The notion of debt reduction doesn't seem to be in my vocabulary.

5) (Shocking Fail) Figure out a "diet plan" and stick to it, lose 10 lbs a month. Ya that was sarcasm (the shocking part) this didn't happen. I did start working out with a trainer and am doing pretty well, i think, but haven't really lost much weight. 

So that's 2010. Tomorrow I'll post what I want to do in 2011 including a new blog series I want to write and some other goals.