Don't Pave the Cow Path
- 18 Apr 13
Lately at work I've been having to remind myself lately not to pave the cow path -- not to bring existing business processes and technology forward just because that's how we've always done it. Here's the poem that the expression comes from.
by Sam Walter Foss (1858-1911)
One day, through the primeval wood,
A calf walked home, as good calves should;
But made a trail all bent askew,
A crooked trail, as all calves do.
Since then three hundred years have fled,
And, I infer, the calf is dead.
But still he left behind his trail,
And thereby hangs my moral tale.
The trail was taken up next day
By a lone dog that passed that way;
And then a wise bellwether sheep
Pursued the trail o’er vale and steep,
And drew the flock behind him, too,
As good bellwethers always do.
And from that day, o’er hill and glade,
Through those old woods a path was made,
And many men wound in and out,
And dodged and turned and bent about,
And uttered words of righteous wrath
Because ’twas such a crooked path;
But still they followed — do not laugh —
The first migrations of that calf,
And through this winding wood-way stalked
Because he wobbled when he walked.
This forest path became a lane,
That bent, and turned, and turned again.
This crooked lane became a road,
Where many a poor horse with his load
Toiled on beneath the burning sun,
And traveled some three miles in one.
And thus a century and a half
They trod the footsteps of that calf.
The years passed on in swiftness fleet.
The road became a village street,
And this, before men were aware,
A city’s crowded thoroughfare,
And soon the central street was this
Of a renowned metropolis;
And men two centuries and a half
Trod in the footsteps of that calf.
Each day a hundred thousand rout
Followed that zigzag calf about,
And o’er his crooked journey went
The traffic of a continent.
A hundred thousand men were led
By one calf near three centuries dead.
They follow still his crooked way,
And lose one hundred years a day,
For thus such reverence is lent
To well-established precedent.
A moral lesson this might teach
Were I ordained and called to preach;
For men are prone to go it blind
Along the calf-paths of the mind,
And work away from sun to sun
To do what other men have done.
They follow in the beaten track,
And out and in, and forth and back,
And still their devious course pursue,
To keep the path that others do.
They keep the path a sacred groove,
Along which all their lives they move;
But how the wise old wood-gods laugh,
Who saw the first primeval calf!
Ah, many things this tale might teach —
But I am not ordained to preach.
Changing your PS Database Platform: Cutover
- 17 Feb 13
You’ll probably want to do at least 4 mock cutovers. One to build the initial development environment on the new hardware. One to start System Test. One to start User Acceptance Testing, and a “dress rehearsal” to prove out your cutover plan/strategy.
Start the cutover plan when you do your first migration. Capture tasks and timing. And continue to refine it with each additional mock cutover.
For the 3rd mock cutover, include items in the cutover plan for communication, external systems that will need to be modified and moved in parallel, shutdown sequence for batch, expected timeline, contact lists, etc. By now your communication plan should be fairly explicit and there should be no surprises from the extended IT team or the business as to what will happen and when.
One to two weeks prior to cutover, execute a “dress rehearsal” where you actually move your production database in as realistic of a fashion as possible. Validate your final timings and make sure nothing was missed.
Two words about cutover communications: They’re important. You need to keep all of your stakeholders informed of where you are in the cutover, raise any issues quickly, and insure all of the hand offs are executed cleanly with no loss of time. Identify a single point of contact (or contacts if you’ll be running cutover around the clock) who can get status from the team members without bugging them too much and prepare regular communications to the interested stakeholders.
In addition, you’ll probably want to maintain two open conference call bridge lines: One for executive/stakeholder updates, and another to allow your technical teams to quickly collaborate on handoffs or issues that arise.
A good cutover plan will include a final “Go/No-Go” decision point prior to starting any cutover activities. If you have no “Severity 1” or “Showstopper” issues the cutover should proceed on schedule.
Now the cutover plan becomes the script for everything over the next hours and days. A common scenario follows: Users close out transactions. Batch schedule is stopped in a controlled manner. Final interface files are sent. Validation reports are run that users will use to validate the system when it comes back up. Finally user accounts are disabled, the application is stopped, and the DBA team (who is hopefully caught up on sleep) takes over.
Now the DBA team executes the data migration using whatever tool you decided on. Row count reports and other validation will be executed when it’s complete and the PeopleTools upgrade will start on the database. This can be the longest part of the process. Then all of your customizations are migrated in, the application is configured and a non-destructive technical checkout is conducted.
It’s typical at this point to allow a limited number of users log in and enter and process real production transactions. This allows any problems to be identified and resolved before the system is turned over to the larger user population.
Finally we’re ready to go. Get your project sponsors and executives on the phone for a final Go/No-Go decision. Once you get the green light, unlock all of the users and start your batch schedule back up in a controlled manner. Congratulations! This is a big accomplishment!!
Changing your PS Database Platform: The Test Phase
- 11 Feb 13
Just because you’re only changing some SQL around doesn’t mean that you can take shortcuts with testing. You’ll want to run an entire stem-to-stern test of your PeopleSoft system to insure that everything works as expected.
One thing to keep in mind: 99% of your defects will be with custom and customized code. The delivered code won’t generate nearly as many problems, so if you’re deciding where to spend your testing resources definitely focus on the custom processes.
As I mentioned in the Build phase, Unit Testing is critical. It’s arguably the most important testing that you will do so come up with a mechanism to track unit test results with the objects modified and make sure you have 100% custom code coverage.
Testing the data migration process is important too. Databases differ in subtle ways and you’ll want to make sure your normal and extended character sets make it across correctly. Naturally you’ll want to run row count reports, but you’ll need to go beyond that. You’ll want to make sure the data in the underlying tables are identical and nothing has been lost in translation. One simple way is to use ODBC to join to tables in the source and target databases and create queries that insure the individual columns are identical. Unfortunately that approach is extremely slow. Another approach is to run hash algorithms on key character fields and summarize the totals, comparing the results on both the source and target database.
System/Integration Testing is also very important. For one thing, you’ll want to have confidence that the system behaves the way you expect it to. And interfaces will generate problems themselves. One common problem is that your interface programs probably assume the default date format for a database platform, and interfaces that don’t specify a date format can choke when the incoming date format doesn’t match what’s expected, or they can send the wrong date format to an output file. Switching from a Non-Unicode database platform to Unicode can cause other problems. You’ll want to execute all of your interfaces and make sure results are valid.
User Acceptance Testing is important as well. Users who know the processes should spend some time in the system making sure all of the critical functionality works and they feel comfortable everything is working as expected. They’ll need to be on the lookout for date format issues, performance issues, data entry discrepancies, etc. They should also spend quality time with their reporting to make sure no new errors were introduced during the build phase.
And finally Performance Testing should be conducted to flesh out any new problems introduced by the new platform. The performance testing should include Online Performance testing, Batch performance testing, and if possible data from PeopleSoft Performance Monitor should be captured and compared to baseline performance data from the old system.
Online performance testing is typically conducted using a tool like LoadRunner or Rational Performance Tester. You record scripts based on a set of highly used business processes and play them back in volume. While the test is executing your admin team will monitor system performance on the various servers and look for bottlenecks. At the end of the day this is necessary for a performance testing effort (especially if you’re making multiple changes like migrating to new hardware and/or a PeopleTools release). However, it’s not sufficient to identify all performance issues.
One of the big deficiencies of online performance testing is that it doesn’t test batch jobs, or if it does test them it is very limited. For batch testing, you’ll want to define a testing mechanism that is realistic according to real, observed jobs that run in your production environment. You’ll want to make sure that the jobs have real data to process. And you’ll want to make sure the jobs are sequenced in such a way that dependencies are tracked to some extent. After all of that, you’ll want to come up with a way to track and report results. I’ll write more about the approach and toolset I’ve used to get realistic batch tests in the future so stay tuned.
The other deficiency of online performance testing is that PeopleSoft is just too complicated of an application to expect a robot to crawl all of the web pages looking for slowness. Sure, during your System and User Acceptance testing phases users will have manually gone through all of the important online pages, but if they didn’t report performance problems it’s likely performance problems will end up in Production. That’s another place where having PeopleSoft Performance Monitor running in the background during testing can be helpful. You can run reports against the various SQL and PeopleCode that gets executed during testing and compare them to your baseline. If certain items are too much slower, you can log a defect and fix them.