Dynamic Enumerations, revisited

Announcer: Now, it’s time for the custodian of clean code, the man who codes what he means and means what he codes, El Moffdo!

That’s correct Mr. Announcer, I am your host, designing with a mere pad and pen just to give the bugs a chance. This is the long-awaited sequel to the oft-searched topic of dynamic enumerations. Let’s go.

First, why is this sequel being written? I have learned that the general problem of a variable’s allowed values being restricted to a finite set that can grow or shrink at run-time is not a unique one; it goes by the alias of the Allowed Value Table (AVT). The quest for solutions to this problem is the source of many of the search engine referrals to this blog.

Home-made Stress Testing

Software engineering nowadays necessitates the use of tools. From IDEs to the xUnit family of unit testing to aspect-oriented frameworks to performance testing tools, you will be hard-pressed to find a decent development shop that does not use tools at every opportunity…except on The Project.

Yes, you see, I am still waiting for The Company to send me my check for my last week of work. Meanwhile, I have already been paid for my first week of work at my new job.

So, as you can imagine, dumping a stack of purchase orders on my boss’s desk was not going to get me anywhere. This blog has chronicled the results of this frugality:

• Coloring DataGrid rows in .NET 1.1
• Schedule estimation by hand
• Ridiculous timelines

But to their credit, I was provided with four test machines (not behemoths, but regular-use workstations) for me to use as I saw fit.

What, then, is a poor chap to do when it comes time for performance testing?

(this is how easy it should be)

Simple; work from the ground up. What was the problem? I had to simulate 300 simultaneous users in 100 buildings spread across the American northeast hitting the same server and somehow measure the response time in each of our use cases.

(ready…click!)

My first crack at this was to strip the UI layer from the system and replace it with a console app. Hey, layers came in handy after all. After obtaining an operational profile, I essentially wrote a script in the console app that acted out this profile.

Then, another executable spawned off 70 copies of this console app. Run that on my four test machines and 70*4=280, which was close enough for us.

In the script, I added code to time how long each operation of interest took. This proved to be inaccurate because of the overhead of console output and exception handling. I finally bit the bullet and added more fine-grained time measurement code within the code-under-test itself. Yes, this did spawn off a “test” branch version of some files, but the cost and inconvenience was minimal.

The time measurement code would dump the statistics into Excel sheets, and a quick VBA script tallied everything up and gave us an average response time for the system.

For a few sprints, this worked out fine, and I was meeting the goal set forth in the requirements document. But this model was fundamentally flawed, and it soon showed when more and more operations were added to the profile. The measured response time skyrocketed, and the problem was not in the new code.

The problem: in the real operating environment, each user would be on their own machine. That means each one would have its own CPU, memory, hard drive, and network connection. The effect of 70 processes executing identical code at the same time and sharing these four resources was warping the stress test. And the hardware wasn’t built for large-scale processing. These were regular plain workstations.

The new problem is now how to distribute the remaining 69 processes in a such a way to not distort the measurements. And gaining access to a test machine in each of the 100 buildings was not going to happen, nor would it be manageable.

The big flash of insight came when I theorized about the layout of The Company’s WAN. A logical person would give each of the 100 buildings its own router, and traffic in building A would not be routed through building B, thus not affecting building B’s router. The traffic would build up on the server side only.

(WAN, LAN…oh man…)

After confirming this theory, I came up with my final home-made stress testing method:

• Keep the console script and keep spawning it 50 times, but only once. There were, incidentally, around 50 users in my home building. This would simulate 50 users hitting the server out of our building. I dumped the timing code and used the working copy of the system.
• For the remaining 210 users, simulate this traffic on the server in order to tie of server resources. This is the tricky part. One of the constraints of The Project was that I had no privilege whatsoever to install any executable on our server. I had a SQL Server, and that was it.

  So I had to bite the bullet and write a script in Transact-SQL. Not only that, but the operational profile had to be translated into stored-procedure calls. It was ugly, but there was no other choice.

• Finally, modify the UI to display measured response time on the UI only. This consisted of simpler timing code and a text box on each screen to display the elapsed time between a click and the conclusion of the operation. These values were used to calculate the response time.

To put it all in action, fire up the server-side script, the local 70-process script, and a copy of the system on my own machine. Run through the use cases and record the response times displayed on teh screen.

This new method successfully modeled the intended operating environment. It also uncovered some silly performance bugs, like queries that used the LIKE operator to match on a primary key. Hey, I never claimed to be a super uber guru, and I probably never will. It also uncovered the performance issue associated with sorting colored DataGrid rows.

It can be done, and it can be done cheaply. But, to be honest, the scalability of a solution like this would be directly related to how messy the server-side T-SQL script became. And I have a feeling that this is limited, at best.

Highlighting DataGrid Rows

I have a story about one instance in which The Project‘s management’s refusal to upgrade from an old version of .NET cost them valuable developer time on a feature that is trivialized in subsequent versions.

When showing all of the equipment of a certain type, there were certain statuses, like “In use” or “Discarded”, that were not to contribute to the equipment count of a room. And in order to keep the counts consistent with the full listing, it would make sense not to show such equipment in the listings. But then you had no easy way to change something from “In use” back to “Spare”.

The solution was to show them anyway in the listing, but highlight their rows to set them apart from the rest of the equipment. Since the natural Windows Forms control for this task is a DataGrid, I set out on figuring out how to highlight specific rows of a DataGrid in .NET 1.1.

The version number is key. Being restricted to .NET 1.1, I was unable to use a DataGridView, which is a customizable DataGrid. This cut me off from handling the CellFormatting event of DataGridView and implementing a rather painless solution to the problem.

After a long search, I found an article that describes the solution. Since the article is fairly detailed and contains lots of other examples, I will summarize how to color a specific row of a .NET 1.1 DataGrid (refer to the article itself for code examples):

• Define a new event with a new data type that will keep track of a cell’s row, column, background color, and font color. Make sure the new data type is passed by reference.
• Handle this event in the form that contains your DataGrid. Do not use the handles keyword. This way, you can re-use the handler and you aren’t tied to a specific number of columns.

  In this handler, you provide the logic that determines which rows are colored. For us, we assume we have this info in a Hashtable somewhere. So we compare if the event’s row shows up in our Hashtable and is not selected. If so, set the background and font color properties of the event.

• Inherit from DataGridTextBoxColumn and override the seven-argument Paint method. This method will get called every time a cell is drawn on the screen.
• Now you fire the event you earlier defined, providing it with the current row and column, which you can get from inside the overridden Paint() and from Me.DataGridTableStyle.GridColumnStyles.IndexOf(Me).
• Finally, go back to your form and replace instances of DataGridTextBoxColumn with your derived column. Use the AddHandler statement to add the event handler to handle the derived column’s newly-defined event.

  The event is handled in the form and its argument gets altered. After the RaiseEvent statement, you can inspect the argument to see if it was altered and set the appropriate brush color to pass to MyBase.Paint().

All of that to highlight a lousy row.

Problems

Well, fine, it might be an annoying exercise in polymorphism and event handling for me, but it did get the job done.

But there was a problem. The columns of this DataGrid had to be sortable. As it stood after the above steps, if you clicked on a column, the grid would sort based on that column, but there wouldn’t be a re-painting of the rows, so the wrong rows would be highlighted after a sort.

Solution:

• Keep track of a need-to-repaint flag.
• Handle the DataGrid’s Paint event. Check the flag and re-paint if the flag is true. By “re-paint”, I mean iterate through each row, examine the correct column(s) value(s) that determine if a row needs to be highlighted, and add that row number to the Hashtable. Be sure to set the flag back to false or you’ll be painting forever.
• Handle the DataGrid’s OnClick event, not its OnMouseDown event. Perform a HitTest and see if the click happened on a column heading. If yes, set need-to-repaint to true.

Why not handle OnMouseDown? I was never fully sure if I correctly understood this MouseClick article:

Depressing a mouse button when the cursor is over a control typically raises the following series of events from the control:

1. MouseDown event.
2. Click event.
3. MouseClick event.
4. MouseUp event.

and this MouseDown article:

Mouse events occur in the following order:

1. MouseEnter
2. MouseMove
3. MouseHover / MouseDown / MouseWheel
4. MouseUp
5. MouseLeave

but I think it is because MouseDown occurs before MouseClick, so in MouseDown, let the re-sort happen with no re-paint. Then, in MouseClick, do a re-paint; this avoids re-painting too early.

OK, that is solved, but now whenever I have a substantial amount of data in a grid, there is a “flickering” effect whenever the grid is loaded or a column is sorted on.

Solution: in DataGrid’s Paint handler, hide the grid before doing the re-paint and then re-show it once finished.

OK, that is solved, but now, whenever a very substantial amount of data is in a grid, sorting by a column introduces a considerable multi-second delay before the grid is visible again.

Solution: paginate the data. Problem: we are in Windows Forms, not ASP.NET.

So we’ve followed this to its logical conclusion: now we’ll have to implement a custom paging solution in order to side-step this performance problem. I never actually got around to implementing this feature, and possible solutions are the topic of a separate post.

So there you have it. Refusal to upgrade to a newer version of .NET, even though the platform itself is rather high-level, cost us substantial development and test time. Don’t underestimate the value of upgrades.