Archive for February, 2011

Many hands make light work

The last posting on collections, whilst being a bit on the long side was fairly straightforward to follow. Today’s posting is a little more, umm, challenging … The reason is I really want to show you what’s possible in the .NET Framework 4.0 that means you will be able to make your apps scream along on modern, multi-cored processors with minimal effort.

First off, a little theory. Most programming languages provide the ability to execute external functionality through transferring control to code referenced by a pointer. Modern COBOL in fact uses procedure pointers to provide this functionality (it just wasn’t possible in OS/VS COBOL). The problem with this type of transfer of control is that it is not safe: you can’t guarantee that you are passing the correct parameters or will receive the correct data back. The .NET Framework gets around this problem through the use of Delegates. Delegates are a type safe way of invoking functionality. This invocation may be synchronous or asynchronous. A delegate describes what some function looks like without defining it. In order to use a delegate you must go through three steps:

  1. define the delegate or method signature
  2. instantiate the delegate
  3. invoke the delegate

In this particular posting, I have to admit we’ll be going through this (to use the vernacular), arse about face. The only reason for this is I had a bit of fun writing this particular bit of code and I want to get it out there and then get on to explaining delegates better!

In this example we’re going to use a generic delegate already defined in the .NET Framework: System.Action. If we look at the code for the new DirectoryCollection (nothing else was modified), I’ve highlighted the new code and commented out the replaced code: 

class-id DirectoryUtility.DirectoryCollection  
  inherits type System.Collections.Generic.List[type System.IO.FileInfo].

working-storage section.

01  DirectoryName   string as "DirectoryName" public property with no set.
   
method-id New.
local-storage section.

01  dirInfo         type System.IO.DirectoryInfo.
01  fiArray         type System.IO.FileInfo[].
01  fInfo           type System.IO.FileInfo.
    
procedure division using by value directoryName as string.

    Set DirectoryName to directoryName.
    Set dirInfo to new type System.IO.DirectoryInfo(DirectoryName).          
    Set fiArray to dirInfo::GetFiles("*.cbl").
    
    Perform varying fInfo through fiArray
        Invoke self::Add(fInfo)
    End-Perform.

    goback.
end method.

method-id Compile public.

local-storage section.

*>01  fInfo           type System.IO.FileInfo.
01 actionDelegate   type System.Action[type System.IO.FileInfo].     
       
procedure division.

    Invoke type DirectoryUtility.CobolUtilities::Initialise().
*>   Perform varying fInfo through self
*>       Invoke type DirectoryUtility.CobolUtilities::
*>                   CompileProgram(
*>                       fInfo::FullName, 
*>                       fInfo::DirectoryName
*>                                 )
*>   End-Perform.          

Set actionDelegate to new type Action[type System.IO.FileInfo]
(
delegate using fInfo as type
System.IO.FileInfo
invoke type DirectoryUtility.CobolUtilities::

                            CompileProgram(
                                fInfo::FullName, 
                                fInfo::DirectoryName
                                          )
             end-delegate
        ).       
 
    Invoke type System.Threading.Tasks.Parallel::ForEach
                    (self, actionDelegate).
                     
    goback.
    
end method.

end class.

Firstly, I define actionDelegate as a System.Action generic delegate of type System.IO.FileInfo. Then I instantiate actionDelegate using the anonymous delegate syntax (the delegate … end-delegate stuff) with the identical code that had been wrapped previously in the Perform varying … End-Perform. I can then invoke the delegate for each System.IO.FileInfo object within the collection using System.Threading.Tasks.Parallel’s ForEach method. This function, part of the Task Parallel Library of .NET Framework 4.0 knows about your hardware and is  able to schedule processing of tasks in parallel to optimise the usage of that hardware. So is it worth it? You be the judge:

 

SNAGHTMLba62613

Look at the duration there: 14 minutes 22 seconds (or 862 seconds). Sequentially, this took 46 minutes 18 seconds (or 2778 seconds). That’s a reduction in duration of 69%. This program was running whilst I was editing this posting, had several Internet Explorer windows open, Outlook, Visual Studio and lord knows what else, at no time losing responsiveness.  Look at Task Manager for some of those programs below. Notice the number of compiler processes running at the top …

image

 

Now I do admit my laptop is a quad core, but that’s what those cores are for, isn’t it? The System.Threading.Tasks functionality drives those cores hard without overwhelming them. Check out the Performance tab on Task Manager during the running of the program. Notice the workout each CPU is getting.

image

The program isn’t production strength and could definitely do with some polishing (the displays come out at a different point to the compilation output due to the fact that the threads are running concurrently), but still not bad for an hour’s work (in total – including the previous post’s code)…

The code can be found here:

http://cid-29bd479bafba9591.office.live.com/embedicon.aspx/COBOL%20School/Multi-threaded%20Cobol%20Directory%20Collection.zip

Welcome to the worker’s collective

The code has been written for a while but I’ve been struggling with writer’s block on what to say about it for a couple of days … Code comes easy but words fail me? Sounds like I might end up being condemned to the back room …

This post is a bit of an introduction to a feature of the .NET Framework called Generic Collections. They can be simply described as COBOL arrays on steroids plus a whole lot more. Just about any way you can group objects has already been written: Lists, Dictionaries, Stacks and many more.

Within the .NET Framework, generics provide a mechanism for writing classes using placeholders rather than specific types so that the developer may write type-safe general classes. Most of the generic classes you are ever likely to need are already written and are available as part of the Framework. Since generics were introduced in the .NET Framework 2.0 in early 2006, I haven’t written a single generic class in anger (although I have played a little in order to understand them).

Once upon a time it was a time-consuming task to write a collection doing complex things like getting enumerators correct. Those days are gone, now we can simply inherit from the appropriate generic collection, add a little specialised behaviour and plough on with the real work. Let me show you … We’re writing a little application to compile all the COBOL programs in a directory so let’s create a collection of directory entries that we may iterate through to compile everything.

First, here’s our solution structure:

image

There are two projects in the solution: CompilePrograms a simple COBOL Console application that references the second project, DirectoryUtility, a COBOL Class Library. I try to put anything I might reuse into class libraries that I might reference in simple top level programs that may define a user interface. With little effort I would be able to create a Windows Forms interface, a Windows Presentation Framework or even a Silverlight interface to the same functionality.

The core of the application is in:

class-id DirectoryUtility.DirectoryCollection  
   inherits type System.Collections.Generic.List[type System.IO.FileInfo].

working-storage section.

01  DirectoryName   string as "DirectoryName" public property with no set.  

method-id New.
local-storage section.

01  dirInfo         type System.IO.DirectoryInfo.
01  fiArray         type System.IO.FileInfo[].
01  fInfo           type System.IO.FileInfo.
    
procedure division using by value directoryName as string.

    Set DirectoryName to directoryName.
    Set dirInfo to new type System.IO.DirectoryInfo(DirectoryName).          
    Set fiArray to dirInfo::GetFiles("*.cbl").
    
    Perform varying fInfo through fiArray
        Invoke self::Add(fInfo)
    End-Perform.

    goback.
end method.

method-id Compile public.

local-storage section.

01  fInfo           type System.IO.FileInfo.
       
procedure division.

    Invoke type DirectoryUtility.CobolUtilities::Initialise().

    Perform varying fInfo through self
        Invoke type DirectoryUtility.CobolUtilities::
                    CompileProgram(fInfo::FullName, fInfo::DirectoryName)
    End-Perform.

    goback.
    
end method.

end class.

The first thing to notice is that in the class definition I’ve inherited from  System.Collections.Generic.List[type System.IO.FileInfo]. This class will be a strongly typed collection of System.IO.FileInfo objects but could as easily be a Dictionary (for fast keyed access) or a Stack (to push and pull objects from). I’ve overridden the new method to populate the list when you create the collection by merely passing a string containing the directory name (I probably should generate an exception if the directory doesn’t exist, but you get the idea). I’ve also created a new method Compile, that will invoke a utility class to initialise the Micro Focus Visual COBOL environment and invoke the compiler for every item in the collection. Note the syntax to iterate through the collection:

Perform varying fInfo through self

..

End-Perform.

We’re processing every fInfo (System.IO.FileInfo) in self , or the collection itself. No indexes to maintain or danger of going beyond the bounds. I’d suggest that in the .NET world no COBOL program should ever create a traditional array ever again. This is simple, powerful stuff.

You probably want to check out the CobolUtilities class out of curiosity:

class-id DirectoryUtility.CobolUtilities public.

static.

working-storage section.

method-id Initialise public.
local-storage section.

78  EnvLocn 
value "SOFTWARE\Wow6432Node\Micro Focus\Visual COBOL\1.2\COBOL\Environment".
01  copyDir     string.
01  settings    type DirectoryUtility.Properties.Settings.
01  regKey      type Microsoft.Win32.RegistryKey.
01  subKey      type Microsoft.Win32.RegistryKey.
01  keys        string occurs any.
01  aKey        string.
01  aValue      string.
       
procedure division.

    Set settings to new type DirectoryUtility.Properties.Settings.
    Set copyDir  to settings::CopyDirectories.
    Invoke self::SetVariable("COBCPY", copyDir).
    
    Set regKey      to type Microsoft.Win32.Registry::LocalMachine.
    Set subKey      to regKey::OpenSubKey(EnvLocn).
    
    If  subKey not equal null
        Set keys to subKey::GetValueNames() 
        Perform varying aKey through keys
            Set aValue to subKey::GetValue(aKey)
            Invoke self::SetVariable(aKey, aValue)
        End-Perform
    End-If.          
    
    goback.
    
end method.

method-id CompileProgram public.

local-storage section.

78  compileLine     value quote & "{0}" & quote 
                          & " ILGEN NOQUERY RAWLIST;".     
01  compileOptions  string.
01  pInfo           type System.Diagnostics.ProcessStartInfo.     
01  aProcess        type System.Diagnostics.Process.  
01  compilerOutput  string.

procedure division using by value programName       as string
                                  programDirectory  as string.

        Display "Compiling " programName.
        Set compileOptions to type 
                           System.String::Format(compileLine, programName)
        Set pInfo to new System.Diagnostics.ProcessStartInfo("cobol.exe")
        Set pInfo::Arguments to compileOptions
        Set pInfo::WorkingDirectory to programDirectory
        Set pInfo::UseShellExecute to false
        Set pInfo::RedirectStandardOutput to true
        Set aProcess to new System.Diagnostics.Process
        Set aProcess::StartInfo to pInfo
        Invoke aProcess::Start()
        Set compilerOutput to aProcess::StandardOutput::ReadToEnd()
        Invoke aProcess::WaitForExit()
        Display compilerOutput

    goback.
    
end method.

method-id SetVariable private static.

local-storage section.

procedure division using by value   envName as string
                                    envValue as string.
                                    
    Invoke type 
           System.Environment::SetEnvironmentVariable(envName, envValue).       

    goback.
    
end method.
end static.
end class.

First off, what the heck am I doing in the Initialise method? Well, initially (sorry for the pun) I’m getting from the application configuration file (an XML file associated with the executable) the setting called CopyDirectories and setting the environment variable COBCPY to contain the setting value (old Micro Focus users will be familiar with this environment variable to set the path for finding COBOL copybooks). To set up application settings easily, right click on your COBOL project (or for that matter C# or VB), click on the Settings tab, give the setting a name, give it a type (in this case, System.String, but it could be an integer or a date for example), and finally give it a value:

SNAGHTML66d09f0

Visual Studio will then automatically generate an accessor class for you to easily read the setting, so you can just type:

Set settings to new type DirectoryUtility.Properties.Settings.

Set copyDir to settings::CopyDirectories.

to read the XML file.

I’m then in the Initialise method, off to the Windows Registry to the key  “HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Micro Focus\Visual COBOL\1.2\COBOL\Environment” (I’m running on a 64 bit version of Windows 7; if you’re running 32 bit you’d use “HKEY_LOCAL_MACHINE\SOFTWARE\Micro Focus\Visual COBOL\1.2\COBOL\Environment” … If I was smart I could even detect which version etc.). I then iterate through all the values within this key and set all the environment variables that Micro Focus requires to run the 32 bit compiler (like COBDIR and PATH).

We then have the Compile method that takes two strings, program name and directory. It then sets up a System.Diagnostics.ProcessStartInfo object with the parameters to pass to the compiler (cobol.exe), creates a System.Diagnostics.Process to execute the compiler, redirects output from the compiler to a stream, waits for the compilation to complete and reads the stream and displays the output.

Finally, to execute the functionality we have a little program:

Program-id CompilePrograms.MainProgram.

working-storage section.

01  Parms-Passed        Pic X(100).
01  dirCollection       type DirectoryUtility.DirectoryCollection.
01  startTime           type System.DateTime.
01  endTime             type System.DateTime.
01  duration            type System.TimeSpan.

procedure division.

    Accept Parms-Passed from Command-Line.
    
    If  type System.IO.Directory::Exists(Parms-Passed) 
        Set startTime to type System.DateTime::Now
        Display "Commencing compilation at " startTime
        Set dirCollection to new 
                    DirectoryUtility.DirectoryCollection(Parms-Passed)               
        Invoke dirCollection::Compile()
        Set endTime to type System.DateTime::Now               
        Display "Completed compilation at " endTime
        Display dirCollection::Count " programs were compiled"
        Set duration to endTime - startTime
        Display "Duration of compilation is " duration
        Display "Hit enter to continue"
        Accept Parms-Passed
    Else
        Display "Directory does not exist: " Parms-Passed
    End-If

    goback.

end program.

which reads the command line for the directory name, checks that the directory exists, takes a time stamp, creates the collection, compiles the cobol programs and takes the duration and prints the duration, ala:

CompileAllSequential

Note those stats: 46 minutes 18 seconds to compile 3044 programs. Not bad, eh? Let’s see what we can do next post to improve this!

The code can be found here.

Missing in combat

Sorry about the pause over the last couple of weeks … I visited Ireland for a week to work with a client and then had a week of everything going wrong. First the extremely hot weather broke my server badly, then it took a couple of days for a visit from the maintenance engineer to visit with replacement parts, only to find that other parts were needed and a pause for another couple of days. All the while NOT getting emails. Also, just prior to this period of electronic isolation finding my Visual COBOL license had expired I requested from a kindly soul a new license, but of course could not see it … All’s well now and I have my license so normal service is about to be resumed.

I’ve decided to briefly digress from the school project to create a little application that will compile every COBOL program in a directory. The aim of the program is to illustrate a fantastic feature of the .NET Framework: Collections and along the way show the Task Parallel Library in .NET 4.0 and spring delegates on you … I’ll do some coding today and have something for you soon