This is a post that I started writing but never completed. It must have been about eight months ago because I don’t do the job described here. I still work for the same company and I do similar work though so most of it still applies.
This is probably one of the only times that you’ll see me posting specifics about work. There are a few reasons for that. Some people, like Aviatrix, Dave and Greybeard can do it and maintain and maintain a level of anonymity for the company and the people that they fly with. In fact, of the three, I can only tell you exactly which aircraft one of them flies and it took months to figure that out.
Like I said though, I’m not good at that so let’s shortcut the process. I work for a major national bank that has made a lot of news lately. There have been calls for all sorts of silly stuff up to and including requests for the resignation or firing of the CEO. Let’s be clear, I’m a technical guy. I’m not a "banker" and I’m not a "trader". I don’t understand the "high finance" side of the business because, honestly, it bores me. There are a lot of people like me here. They know their jobs and many are considered subject matter experts not just around the office but in the specific industries where their skills apply. Many of them, in fact, literally wrote the book on the subjects that they work with. It is highly gratifying to work with people who are experts in a particular field. Mistakes were made recently… it’s not yet clear who made them but does that really matter? I continue to have great confidence in both the stability of the bank and the leadership. Let me explain why from my own perspective…
It’s tough to describe the complexity that exists within a major international organization. There are people that do everything and there’s a lot to be done. Some of the most valuable people in this organization aren’t the guys who can do things but the guys who know who to call to get things done. A subtle but important difference. The software that I primarily work with is an externally provided application. This is also known as "third-party" or vendored application. It’s the only software in the world that does what it does and honestly, it does it rather well. The problem is that there can sometimes be quality issues with new versions so we put a lot of work into making sure that it works the way it should.
For months at a time we work with the vendor to define changes that are need in the application. Thousands of hours go into making sure that the next version works the way it should. After all of that work is complete we get a link from the vendor and are told where to download the product. That’s when I get to play Maestro.
The first thing I do is download the software to a secure place and make sure it works in a basic way. That means you can install it without errors and connect the client to the server. Minimal stuff. When I’m sure it does I move the packaged software somewhere that that the folks that will need to can get at the bits they need.
I ask the database administrators to make copies of the "production" data so that we have something to test the product against. Depending on what we’re testing this can be as many as six copies of the database which is about 15 Gig of data so you’re talking about moving around 90 Gigabytes of stuff… that’s a lot. This process takes at least "overnight" because the team that does the copies is in India. To get them to do it they need a request form filled out. That form can be complex and if it’s not right they won’t do the work, you’ll have to do the form again and you’ll have to wait another 24 hours. It seems pedantic and mean but it makes sense. A lot depends on getting data in the right places securely at a bank and they’re not going to make a mistake because I was lazy in filling out forms and didn’t explain exactly what I want.
That done, my Indian counterpart and I start putting the servers together. Mostly, this is an easy process. We have three servers to do testing with but at any given time they can be in use for different things. It’s a dance to make sure that servers with the right versions of the software are available to the people that need them at just the right time. We get the servers installed and pointing to the right databases for the specific tests. There are, however, a couple of the servers that we don’t have access to. For those it’s filling out another form with specific instructions on where to get the software, how to install it and how to configure it. Here again, if something is wrong you’re in for a 24 hour delay to make it right.
Next, is making sure the client side is set up. A "client" is the tool that you use to connect to the server. In Internet terms, a webserver would be considered the "server" part of the application and your web browser would be considered the "client" part. In order to test the client part we need to desktops to install them on. We can’t have this "test" version installed on our own workstations because the potential for error is too great. A lot of the test involve changing data and generally trying to break things. Accidentally pointing a test client at the live or production servers would be a bad thing(tm) so to prevent that we never install the test version on the same desktop as a live version. That means we need at least six or eight test desktops. They all have to be installed with the proper software and configured in such a way that they can never talk to the live servers. This takes another form which again must be prefect err… perfect.
With these parts done, you’re already up to three days worth of "wait time" just to get things set up. In order to speed up the process we do it all at once (databases, clients, servers, etc) and can have an entire test environment set up over night. The problem with doing that is that it all has to be right on the first go or it has to be done again. Doing it again means that I’m affecting other people. The testers are standing by to test for us and until we get it right they’re sitting on their hands. The project managers are waiting patiently (yeah right) and the end users… the end users are waiting for us to all get done so that they can get all of the cool new features.
It’s complex but fantastically fulfilling when it all comes together perfectly and I realize that there are probably only two or three people who could have done it as fast…
… that’s where the post ended. I can’t remember now the thought that I wanted to share here but it does give a bit of insight into what I do, or at least did.