The Dyalog Cookbook

Chapter 1:

Introduction

You want to write a Windows [1] application in Dyalog APL. You have already learned enough of the language to put some of your specialist knowledge into functions you can use in your work. The code works for you. Now you want to turn it into an application others can use. Perhaps even sell it.

This is where you need professional programming skills. How to install your code on an unknown computer. Have it catch, handle and log errors. Manage the different versions of your software as it evolves. Provide online help.

You are not necessarily a professional programmer. Perhaps you don't have those skills. Perhaps you need a professional programmer to turn your code into an application. But you’ve come a long way already. Perhaps you can get there by yourself - with The Dyalog Cookbook. Alternatively, you might be a professional programmer wondering how to solve these familiar problems in Dyalog APL.

The Dyalog Cookbook is about how to turn your Dyalog code into an application. We’ll cover packaging your code into a robust environment. And we’ll introduce some software development tools you’ve managed without so far, which will make your life easier.

You might continue as the sole developer of your application for a long time yet. But if it becomes a successful product you will eventually want other programmers collaborating on it. So we’ll set up your code in a source-control system that will accommodate that. Even while you remain a sole developer, a source-control system will also allow you to roll back and recover from your own mistakes.

Not so long ago it was sufficient for an application to be packaged as an EXE that could be installed and run on other PCs. Nowadays many corporate clients run programs in terminal servers or in private clouds. So we’ll look at how to organise your program to run as tasks that communicate with each other.

Many applications written in Dyalog focus on some kind of numerical analysis, and can have CPU-intensive tasks. We'll look at how such tasks can be packaged to run either in the background or on remote machines.

1. Method

It’s conventional in this context for authors to assure readers that the techniques expounded here have been hammered out, proven and tested in many successful applications. That is true of individual components here, particularly of scripts and applications from the APLTree [2] library.

But the development tools introduced by Dyalog in recent years are still finding their places with development teams. Some appear here in print for the first time. This book is the first sustained attempt to combine all the current Dyalog tools into an integrated approach.

Many of the issues addressed here are entangled with each other. We’ll arrive at our best solutions by way of interim solutions. Proposing some wickedly intricate ‘complete solution’ framework does little to illuminate the problems it solves. So we’ll add features – INI files, error handling, and so on – one at a time, and as we go we’ll find ourselves revisiting the code that embeds the earlier features.

We will also improve the code along the way while explaining why exactly the changes are improvements.

That is the method for chapters 1 – 14. Later chapters stand on their own.

If you are an experienced Dyalog developer, you may be able to improve on what is described here. For this reason The Dyalog Cookbook remains for now an open-source project on GitHub.

Working through the book, you get to understand how the implementation issues, and the solutions to them, work. In the first chapters you will find ‘framework’ code for your application, growing more complex as the book progresses. You can find scripts for these interim versions in the code folder on the book website. Watch out: they are interim solutions, constantly improved along the way.

You are of course welcome simply to copy and use the last version of the scripts. But there is much to be learned while stumbling.

Later on we’ll introduce some professional writing techniques that might make maintaining your code easier – in what we hope will be a long useful future for it. This includes third-party tools, configuring your development environment and discussing user commands.

2. What you need to use the Dyalog Cookbook

We have not attempted to ‘dumb down’ our use of the language for readers with less experience. In some cases we stop to discuss linguistic features; mostly not.

If you see an expression you cannot read, a little experimentation and consultation of the reference material should show you how it works.

But we have not tried to be smart either. Code should be as terse as reasonable, but should always be readable, maintainable and traceable.

We encourage you to take the time to do this. Generally speaking – not invariably – short, crisp expressions are less work for you and the interpreter to read. Learn them and prefer them.

In case you still need help the Dyalog Forum provides access to a competent and friendly community around Dyalog.

3. Conventions

Information

Note that we assume ⎕IO←1 and ⎕ML←1, not because we are making a statement, but because that’s the Dyalog default. That keeps the Cookbook in sync with the Dyalog documentation.

Getting deeper

In case we want to discuss a particular issue in more detail but we are not sure whether the reader is ready for this, now or ever, we format the information this way.

Warning

Sometimes we need to warn you, for example in order to avoid common traps. This is how that would look like.

Tip

Sometimes we want to provide a tip, and this is how that looks.

When we refer to a text file, e.g. something with the extension .txt, we refer to it as a TXT. We refer to a dyalog script (*.dyalog) as a DYALOG. We refer to a dyapp script (*.dyapp) as a DYAPP. You get the pattern.

4. Acknowledgements

We are deeply grateful for contributions, ideas, comments and outright help from our colleagues, particularly from (in alphabetical order) Gil Athoraya, Morten Kromberg, Paul Mansour, Nick Nickolov, Andy Shiers and Richard Smith.

We jealously claim any errors as entirely our own work.

Kai Jaeger & Stephen Taylor


Footnotes

  1. Perhaps one day you would like it to ship on multiple platforms. Perhaps one day we’ll write that book too. Meanwhile, Microsoft Windows.

    You will, however, find that whenever possible we keep the code platform independent. If we use platform-dependent utilities we mention it and explain why; we might also mention alternatives available on other platforms.

  2. APLTree is the name of an open-source library that offers robust, tested and well-documented solutions to many everyday problems you face when addressing the tasks discussed in this book.

    We will use this library extensively and discuss it in detail. More at the source:
    https://github.com/aplteam/apltree/wiki/Members. You can also search for “apltree” on GitHub.

  3. These days seasoned programmers often have strong opinions about whether to use an object-oriented approach or a functional approach or to mix them both.

    We have seen friendships broken on these issues. In this book we take a mixed approach.

Chapter 2:

Structure

In this chapter we consider your choices for making your program available to others, and for taking care of the source code, including tracking the changes through successive versions.

To follow this, we’ll make a very simple program. It counts the frequency of letters used in one or multiple text files. (This is simple, but useful in cryptanalysis, at least at hobby level.) We’ll put the source code under version control, and package the program for use.

Some of the things we are going to add to this application will seem like overkill, but keep in mind that we use this application just as a very simple example for all the techniques we are going to introduce.

Let’s assume you've done the convenient thing. Your code is in a workspace. Everything it needs to run is defined in the workspace. Maybe you set a latent expression, so the program starts when you load the workspace.

We shall convert a DWS to some DYALOG scripts and introduce a DYAPP script to assemble an active workspace from them.

Using scripts to store your source code has many advantages: you can use a traditional source code management system rather than having your code and data stored in a binary blob.

Changes that you make to your source code are saved immediately, rather than relying on you to remember to save the workspace at some suitable point in your work process.

Finally, you don’t need to worry about crashes in your code or externally called modules and also any corruption of the active workspace which might prevent you from saving it.

Corrupted workspaces

The workspace (WS) is where the APL interpreter manages all code and all data in memory. The Dyalog tracer/debugger has extensive edit-and-continue capabilities; the downside is that these have been known to corrupt the workspace occasionally.

The interpreter checks WS integrity every now and then; how often can be influenced by setting certain debug flags; see the “Appendix 3 — aplcores and WS integrity” for details. More details regarding aplcores are available in the appendix “Aplcores”.

1. How can you distribute your program?

1.1. Send a workspace file (DWS)

Could not be simpler. If your user has a Dyalog interpreter, she can also save and send you the crash workspace if your program hits an execution error. But she will also be able to read your code – which might be more than you wish for.

Crash workspaces

A crash workspace is a workspace that was saved by a function that was initiated by error trapping, typically by setting ⎕TRAP. It’s a snapshot of the workspace at the moment an unforeseen problem triggered error trapping to take over. It’s usually very useful to analyse such problems.

Note that a workspace cannot be saved when more than one thread is running.

If she doesn’t have an interpreter and you are not worried about her someday getting one and reading your code, and you have a Run-Time Agreement with Dyalog, you can send her the Dyalog Run-Time interpreter with the workspace. The Run-Time interpreter will not allow the program to suspend, so when the program breaks the task will vanish, and your user won’t see your code. All right so far. But she will also not have a crash workspace to send you.

If your application uses multiple threads, the thread states can’t be saved in a crash workspace anyway.

You need your program to catch and report any errors before it dies, something we will discuss in the chapter Handling errors.

1.2. Send an executable file (EXE)

This is the simplest form of the program to install because there is nothing else it needs to run: everything is embedded within the EXE. You export the workspace as an EXE, which can have the Dyalog Run-Time interpreter bound into it. The code cannot be read. As with the workspace-based runtime above, your program cannot suspend, so you need it to catch and report any errors before dying.

We’ll do that!

2. Where should you keep the code?

Let’s start by considering the workspace you will export as an EXE.

The first point is: PCs have a lot of memory relative to your application code volume. So all your Dyalog code will be in the workspace. That’s probably where you have it right now anyway.

Your workspace is like your desktop – a great place to get work done, but a poor place to store things. In particular it does nothing to help you track changes and revert to an earlier version.

Sometimes a code change turns out to be for the worse, and you need to undo it. Perhaps the change you need to undo is not the most recent change.

We’ll keep the program in manageable pieces – ‘modules’ – and keep those pieces in text files under version control.

For this, there are many source-control management (SCM) systems and repositories available. Subversion, Git and Mercurial are presently popular. These SCMs support multiple programmers working on the same program, and have sophisticated features to help resolve conflicts between them.

Source code management with acre Desktop

Some members of the APL community prefer to use a source-code management system that is tailored to solve the needs of an APL programmer, or a team of APL programmers: acre Desktop.

APL code is very compact, teams are typically small, and work on APL applications tends to be oriented towards functions rather than modules such as classes.

acre Desktop can be used as a source-code management system in its own rights= together with acre Server, but it can use other code management systems like Git or SubVersion as well. Both acre Desktop and acre Server are available as open source software. We will discuss acre in its own appendix.

Whichever SCM you use (we used GitHub for writing this book and the code in it) your source code will comprise class and namespace scripts (DYALOGs) for the application. The help system will be an ordinary – non-scripted – namespace. We use a build script (DYAPP) to assemble the application and the development environment.

You’ll keep your local working copy in whatever folder you please. We’ll refer to this working folder as Z:\ but it will of course be wherever suits you.

3. The LetterCount workspace

We suppose you already have a workspace in which your program runs. We don’t have your code to hand so we’ll use ours. We’ll use a very small and simple program, so we can focus on packaging the code as an application, not on writing the application itself.

So we’ll begin with the LetterCount workspace. It’s trivially simple but for now it will stand in for your code. You can download it from the book’s web site: https://cookbook.dyalog.com.

On encryption

Frequency counting relies on the distribution of letters being more or less constant for any given language. It is the first step in breaking a substitution cypher.

Substitution cyphers have been superseded by public-private key encryption, and are mainly of historical interest, or for studying cryptanalysis. But they are also fun to play with.

We recommend The Code Book: The secret history of codes & code-breaking by Simon Singh and In Code by Sarah Flannery as introductions if you find this subject interesting.

4. Versions

In real life, you will produce successive versions of your program, each better than the last. In an ideal world, all your users will have and use the current version. In that ideal world, you have only one version to maintain: the latest.

In the real world, your users will have and use multiple versions. If you charge for upgrading to a newer version, this will surely happen. And even in your ideal world, you have to maintain at least two versions: the current and the next.

What does it mean to maintain a version? At the very minimum, you keep the source code for it, so you could recreate its EXE from scratch, exactly as it was distributed. There will be things you want to improve, and perhaps bugs you must fix. Those will all go into the next version, of course. But some you may need to put into the released version and re-issue it to current users as a patch.

So in The Dyalog Cookbook we shall develop in successive versions until we manage to create an installer that is capable of installing the application on any machine running Windows 10. What’s needed to achieve that is discussed in the chapters 1-16. Later chapters are independent.

Our ‘versions’ are not ready to ship, so are probably better considered as milestones on the way to version 1.0. You could think of them as versions 0.1, 0.2 and so on. But we’ll just refer to them as Versions 1, 2, and so on.

Our first version won’t even be ready to export as an EXE. It will just create a workspace MyApp.dws from scripts: a DYAPP and some DYALOGs. We’ll call it Version 1.

Load the LetterCount.dws workspace from the code\foo folder on the book website. Again, this is just the stand-in for your own code. Here’s a quick tour.

4.1. Investigating the workspace LetterCount

Let’s load the workspace LetterCount and investigate it a bit.

Function TxtToCsv takes the filepath of a TXT and writes a sibling CSV [1] containing the frequency count for the letters in the file. It uses function CountLetters to produce the table.

      ∆←'Now is the time for all good men'
      ∆,←' to come to the aid of the party.'
      CountLetters ∆
N 2
O 8
W 1
I 3
S 1
T 7
H 3
E 6
M 3
F 2
R 2
A 3
L 2
G 1
D 2
C 1
P 1
Y 1
Information

Note that we use a variable here. Not exactly a memorable or self-explaining name. However, we use whenever we collect data for temporary use.

CountLetters returns a table of the letters in ⎕A and the number of times each is found in the text. The count is insensitive to case and ignores accents, mapping accented to unaccented characters:

      Accents
ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ
AAAAAACDEEEEIIIINOOOOOOUUUUY

That amounts to five functions. Two of them are specific to the application: TxtToCsv and CountLetters. The other three –– toUppercase, join and map — are utilities of general use.

Note that we have some functions that start with lowercase characters while others start with uppercase characters. In a larger application you might want to be able to tell data from calls to functions and operators by introducing consistent naming conventions. Which one you settle for is less important than choosing something consistent. And remember to put it into a document any programmer joining the team can read.

toUppercase uses the fast case-folding I-beam introduced in Dyalog 15.0 (also available in 14.0 & 14.1 from revision 27141 onwards).

TxtToCsv uses the file-system primitives ⎕NINFO, ⎕NGET, and ⎕NPUT introduced in Dyalog 15.0.

4.2. How to organise the code

To expand this program into distributable software we’re going to add features, many of them drawn from the APLTree library. To facilitate that we’ll first organise the existing code into script files, and write a build script to assemble a workspace from them.

Information

The APLTree library is an open-source project hosted on GitHub. It offers solutions for many every-day problems a Dyalog APL programmer might run into. In the Cookbook we will use many of its members. For details see https://github.com/aplteam/apltree/wiki.

Start at the root namespace (#). We’re going to be conservative about defining names in #. Why? Right now the program stands by itself and can do what it likes in the workspace. But in the future your program might become part of a larger APL system. In that case it will share # with other objects you don’t know anything about right now.

So your program will be a self-contained object in #. Give it a distinctive name, not a generic name such as Application or Root. From here on we’ll call it MyApp. (We know – almost as bad.)

But there are other objects you might define in #. If you’re using classes or namespaces that other systems might also use, define them in #. For example, if MyApp should one day become a module of some larger system, it would make no sense for each module to have its own copy of, say, the APLTree class Logger.

With this in mind, let’s distinguish some categories of code, and how the code in MyApp will refer to them.

General utilities and classes
For example, the APLTreeUtils namespace and the Logger class. (Your program doesn't yet use these utilities.) In the future, other programs, possibly sharing the same workspace, might use them too.
Your program and its modules
Your top-level code will be in #.MyApp. Other modules and MyApp-specific classes may be defined within it.
Tools and utility functions specific to MyApp
These might include your own extensions to Dyalog namespaces or classes. Define them inside the app object, e.g. #.MyApp.Utils.
Your own language extensions and syntax sweeteners
For example, you might like to use functions means and else as simple conditionals. These are effectively your local extensions to APL, the functions you expect always to be around. Define your collection of such functions into a namespace in #, eg #.Utilities.

The object tree in the workspace might eventually look something like:

#
|-⍟Constants
|-⍟APLTreeUtils
|-⍟Utilities
|-○MyApp
| |-⍟Common
| |-⍟Engine
| |-○TaskQueue
| \-⍟Utils
\-○Logger
\-⍟UI
Information

denotes a namespace, a class. These are the characters (among others) you can use to tell the editor what kind of object you wish to create, so for a class )ed ○ Foo. Press F1 with the cursor on )ed in the session for details.

Note that we keep the user interface (UI) separate from the business logic. This is considered good practice because whatever you believe right now, you will almost certainly one day consider exchanging a particular type of UI (say .NET Windows forms) for a different one (say HTML+JavaScript). This is difficult enough, but a bit easier when you separate them right from the start. However, our application is so simple that we collect all its code in a namespace script MyApp in order to save one level in the namespace hierarchy.

If this were to be a serious project then you would not do this even if the amount of code is small: applications change and grow over time, sometimes significantly. Therefore you would be better prepared to have, say, a namespace MyApp that contains, say, a namespace script engine with all the code.

The objects in # are ‘public’. They comprise MyApp and objects other applications might use; you might add another application that uses #.Utilities. Everything else is encapsulated within MyApp. Here’s how to refer in the MyApp code to these different categories of objects.

  1. log←⎕NEW #.Logger
  2. queue←⎕NEW TaskQueue
  3. tbl←Engine.CountLetters txt
  4. status←(bar>3) #.Utilities.means 'ok' #.Utilities.else 'error'

The last one is pretty horrible. It needs some explanation.

Many languages offer a short-form syntax for if/else, eg (JavaScript, PHP, C…)

status = bar>3 ? 'ok' : 'error' ;

Some equivalents in Dyalog:

What style you prefer is mainly a matter of personal taste, and indeed even the authors do not necessarily agree on this. There are however certain rules you should keep in mind:

4.2.1. Execution time

status←(bar>3) U.means 'ok' U.else 'error'

In this approach two user-defined functions are called. Not much overhead but don’t go for this if the line is, say, executed thousands of times within a loop.

4.2.2. Keep the end user in mind

The authors have done pair programming for years with end users being the second party. For a user a statement like:

taxfree←(dob>19491231) U.means 35000 U.else 50000

is easily readable despite it being formed of APL primitives and user-defined functions. This can be a big advantage in an agile environment where the end user reviews business logic with the implementors.

For classes, however, there is another way to do this: include the namespace #.Utilities. In order to illustrate this let’s assume for a moment that MyApp is not a namespace but a class.

:Class MyApp
:Include Utilities
…
:EndClass

This requires the namespace #.Utilities to be a sibling of the assumed class MyApp. Now within the class you can do

status←(bar>3) means 'ok' else 'error'

yet Shift+Enter in the Tracer or the Editor still works, and any changes made to the utilities would go into #.Utilities.

More about :Include

When a namespace is included, the interpreter will execute functions from that namespace as if they had been defined in the current class. However, the actual code is shared with the original namespace. For example, this means that if the code of means or else is changed while tracing into it from the MyApp class those changes are reflected in #.Utilities immediately (and any other classes that might have included it).

Most of the time, this works as you expect it to, but it can lead to confusion, in particular if you were to )COPY #.Utilities from another workspace. This will change the definition of the namespace, but the class has pointers to functions in the old copy of #.Utilities, and will not pick up the new definitions until the class is fixed again.

If you were to edit these functions while tracing into the MyApp class, the changes will not be visible in the namespace. Likewise, if you were to )ERASE #.Utilities, the class will continue to work until the class itself is edited, at which point it will complain that the namespace does not exist.

Let’s assume that in a WS C:\Test_Include we have just this code:

:Class Foo
:Include Goo
:EndClass

:Namespace Goo
∇ r←Hello
    :Access Public Shared
      r←'World'
    ∇
:EndNamespace

Now we do this:

Foo.Hello
world
      )Save
Saved...
      ⎕EX 'Goo'
      Goo
VALUE ERROR
      Foo.Hello
world
)copy c:\Test_Include Goo
copied...

If at this stage you were to edit Goo and change 'world' to 'Universe', and then again call Foo.Hello it would still print world to the session.

If you encounter this, re-fix your classes (in this case Foo). Rebuilding the WS from the source files would be even better.

4.2.3. Be careful with diamonds

The :If - :Then - :else solution could have been written this way:

:If bar>3 ⋄ status←'ok' ⋄ :Else ⋄ status←'error' ⋄ :EndIf

There is one major problem with this: when executing the code in the Tracer the line will be executed in one go. If you think you might want to follow the control flow and trace into the individual expressions, you should spread the control structure over 5 lines.

In general: if you have a choice between a short and a long expression then go for the short one – unless the long one offers an incentive such as improved readability, better debugging or faster execution speed.

Diamonds can be useful in some situations. In general it’s better to avoid them.

Diamonds

In some circumstances diamonds are quite useful:

4.2.4. Why not use ⎕PATH?

⎕PATH tempts us. We could set ⎕PATH←'#.Utilities'. The expression above could then take its most readable form:

status←(bar>3) means 'ok' else 'error'

Trying to resolve the names means and else, the interpreter would consult ⎕PATH and find them in #.Utilities. So far so good: this is what ⎕PATH is designed for. It works just fine in simple cases but quickly leads to confusion about which functions are called or edited, and where new names are created. Avoid ⎕PATH if reasonable alternatives are available.

4.3. Convert the WS LetterCount into a single scripted namespace.

If your own application is already using scripted namespaces and/or classes then you can skip this, of course.

Download the WS and save it as Z:\code\v00\LetterCount.

Everything in that WS lives in #. We have to move it into a single namespace MyApp. Execute the following steps:

  1. Start an instance of Dyalog
  2. Execute )ns MyApp to create a namespace MyApp in the workspace.
  3. Execute )cs MyApp to make MyApp the current namespace.
  4. Execute )copy Z:\code\v00\LetterCount to copy all the functions and the single variable into the current namespace, #.MyApp.
  5. Execute )copy Z:\code\v00\LetterCount ⎕IO ⎕ML

    This ensures we really use the same values for important system variables as the WS does by copying their values into the namespace #.MyApp.

  6. Execute ]save #.MyApp Z:\code\v01\MyApp -makedir -noprompt

The last step will save the contents of the namespace #.MyApp into Z:\code\v01\MyApp.dyalog. If the folder v01 or any of its parents do not already exist the -makedir option will create them. -noprompt makes sure that ]save does not ask any questions.

This is how the script would look like:

:Namespace MyApp
⍝ === VARIABLES ===

Accents←2 28⍴'ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝAAAAAACDEEEEIIIINOOOOOOUUUUY'


⍝ === End of variables definition ===

(⎕IO ⎕ML ⎕WX ⎕PP ⎕DIV)←1 1 3 15 1

 CountLetters←{
     ⍝ Table of letter frequency in txt
     {⍺(≢⍵)}⌸⎕A{⍵/⍨⍵∊⍺}(↓Accents)map toUppercase ⍵
 }

∇ noOfBytes←TxtToCsv fullfilepath;NINFO_WILDCARD;NPUT_OVERWRITE;tgt;files;path;stem;txt;enc;nl;lines;csv
     ⍝ Write a sibling CSV of the TXT located at fullfilepath,
     ⍝ containing a frequency count of the letters in the file text.
 NINFO_WILDCARD←NPUT_OVERWRITE←1 ⍝ constants
 fullfilepath~←'"'
 csv←'.csv'
 :Select 1 ⎕NINFO fullfilepath
 :Case 1 ⍝ folder
     tgt←fullfilepath,'\total',csv
     files←⊃(⎕NINFO⍠NINFO_WILDCARD)fullfilepath,'\*.txt'
 :Case 2 ⍝ file
     (path stem)←2↑⎕NPARTS fullfilepath
     tgt←path,stem,csv
     files←,⊂fullfilepath
 :EndSelect
     ⍝ assume txt<<memory
 (txt enc nl)←{(⊃,/1⊃¨⍵)(1 2⊃⍵)(1 3⊃⍵)}⎕NGET¨files
 lines←','join¨↓⍕¨CountLetters txt
     ⍝ use encoding and NL from first source file
 noOfBytes←(lines enc nl)⎕NPUT tgt NPUT_OVERWRITE
     ⍝Done
∇

 join←{
     ⍺←⎕UCS 13 10
     (-≢⍺)↓⊃,/⍵,¨⊂⍺
 }

 map←{
     (old new)←⍺
     nw←∪⍵
     (new,nw)[(old,nw)⍳⍵]
 }

 toUppercase←{1(819⌶)⍵}

:EndNamespace

There might be minor differences, depending on the version of the ]save user command and the version of SALT you are using.

This is the easiest way to convert an ordinary workspace into one or more scripted namespaces.

Now we start improving it.

5. Project Gutenberg

We’ll raid Project Gutenberg for some texts to read.

We’re tempted by the complete works of William Shakespeare but we don’t know that letter distribution stayed constant over four centuries. Interesting to find out, though, so we’ll save a copy as Z:\texts\en\shakespeare.dat. And we’ll download some 20th-century books as TXTs into the same folder. Here are some texts we can use.

      ↑⊃(⎕NINFO⍠'Wildcard' 1) 'z:\texts\en\*.txt'
z:/texts/en/ageofinnocence.txt
z:/texts/en/dubliners.txt
z:/texts/en/heartofdarkness.txt
z:/texts/en/metamorphosis.txt
z:/texts/en/pygmalion.txt
z:/texts/en/timemachine.txt
z:/texts/en/ulysses.txt
z:/texts/en/withthesehands.txt
z:/texts/en/wizardoz.txt

6. MyApp reloaded

We’ll first make MyApp a simple 'engine' that does not interact with the user. Many applications have functions like this at their core. Let’s enable the user to call this engine from the command line with appropriate parameters. By the time we give it a user interface, it will already have important capabilities, such as logging errors and recovering from crashes.

Our engine will be based on the TxtToCsv function. It will take one parameter, a fully qualified filepath for a folder or file. If it is a file it will write a sibling CSV. If it is a folder it will read all the TXTs in the folder, count the letter frequencies and write them as a CSV sibling to the folder. Simple enough. Here we go.

7. Building from a DYAPP

In your text editor open a new document.

You need a text editor that handles Unicode. If you’re not already using a Unicode text editor, Windows’ own Notepad will do for occasional use. (Set the default font to APL385 Unicode)

For a full-strength multifile text editor Notepad++ works well but make sure that the editor converts Tab into space; by default it does not, and Dyalog does not like Tab characters.

You can even ensure Windows calls Notepad++ when you enter notepad.exe into a console window or double-click a TXT icon: Google for “notepad replacer”.

Here’s how the object tree will look like:

#
|-⍟Constants
|-⍟Utilities
\-⍟MyApp

We’ve saved the very first version as z:\code\v01\MyApp.dyalog. Now we take a copy of that and save it as z:\code\v02\MyApp.dyalog. Alternatively you can download version 2 from the book’s website.

Note that compared with version 1 we will improve in several ways:

The file tree will look like this:

z:\code\v02\Constants.dyalog
z:\code\v02\MyApp.dyalog
z:\code\v02\Utilities.dyalog
z:\code\v02\MyApp.dyapp

MyApp.dyapp looks like this if we take the simple approach:

Target #
Load Constants
Load Utilities
Load MyApp

This is the Constants.dyalog script:

:Namespace Constants
    ⍝ Dyalog constants
    :Namespace NINFO
        ⍝ left arguments
        NAME←0
        TYPE←1
        SIZE←2
        MODIFIED←3
        OWNER_USER_ID←4
        OWNER_NAME←5
        HIDDEN←6
        TARGET←7
        :Namespace TYPES
            NOT_KNOWN←0
            DIRECTORY←1
            FILE←2
            CHARACTER_DEVICE←3
            SYMBOLIC_LINK←4
            BLOCK_DEVICE←5
            FIFO←6
            SOCKET←7
        :EndNamespace
    :EndNamespace
    :Namespace NPUT
        OVERWRITE←1
    :EndNamespace
:EndNamespace

Note that we use uppercase here for the names of the ‘constants’. (They are of course not really constants but ordinary variables.) It is a common programming convention to use uppercase letters for constants.

Information

Later on we’ll introduce a more convenient way to represent and maintain the definitions of constants. This will do nicely for now.

This is the Utilities.dyalog script:

:Namespace Utilities
      map←{
          (old new)←⍺
          nw←∪⍵
          (new,nw)[(old,nw)⍳⍵]
      }
    toLowercase←0∘(819⌶)
    toUppercase←1∘(819⌶)
:EndNamespace

Finally the MyApp.dyalog script:

:Namespace MyApp

   (⎕IO ⎕ML ⎕WX ⎕PP ⎕DIV)←1 1 3 15 1

⍝ === Aliases

    U←##.Utilities ⋄ C←##.Constants

⍝ === VARIABLES ===

    Accents←↑'ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ' 'AAAAAACDEEEEIIIINOOOOOOUUUUY'

⍝ === End of variables definition ===

      CountLetters←{
          {⍺(≢⍵)}⌸⎕A{⍵⌿⍨⍵∊⍺}(↓Accents)U.map U.toUppercase ⍵
      }

    ∇ noOfBytes←TxtToCsv fullfilepath;csv;stem;path;files;lines;nl;enc;tgt;tbl
   ⍝ Write a sibling CSV of the TXT located at fullfilepath,
   ⍝ containing a frequency count of the letters in the file text.
      fullfilepath~←'"'
      csv←'.csv'
      :Select C.NINFO.TYPE ⎕NINFO fullfilepath
      :Case C.TYPES.DIRECTORY
          tgt←fullfilepath,'total',csv
          files←⊃(⎕NINFO⍠'Wildcard' 1)fullfilepath,'\*.txt'
      :Case C.TYPES.FILE
          (path stem)←2↑⎕NPARTS fullfilepath
          tgt←path,stem,csv
          files←,⊂fullfilepath
      :EndSelect
      (tbl enc nl)←{(⊂⍪⊃⍵)1↓⍵)}(CountLetters ProcessFiles) files
      lines←{⍺,',',⍕⍵}/⊃{⍺(+/⍵)}⌸/↓[1]tbl
      noOfBytes←(lines enc nl)⎕NPUT tgt C.NPUT.OVERWRITE
    ∇

    ∇(data enc nl)←(fns ProcessFiles) files;txt;file
   ⍝ Reads all files and executes `fns` on the contents. `files` must not be empty.
      data←⍬
      :For file :In files
          (txt enc nl)←⎕NGET file
          data,←⊂fns txt
      :EndFor
    ∇

:EndNamespace

This version comes with a number of improvements. Let’s discuss them in detail:

Warning

If you see any namespaces called SALT_Data ignore them. They are part of how SALT manages meta data for scripted objects.

We have converted the saved workspace to text files, and made a DYAPP that builds the workspace from the DYALOGs. But we have not saved a workspace: we will always build a workspace from scripts.

Launch the DYAPP by double-clicking on its icon in Windows Explorer. Examine the active session. We see

- Constants
  - NINFO
    - NAME
    - ...
    - TYPES
      - NOT_KNOWN
      - DIRECTORY
      - ...
  - NPUT
    - OVERWRITE
- MyApp
  - Accents
  - C
  - CountLetters
  - TxtToCsv
  - U
- Utilities
  - map
  - toLowercase
  - toUppercase

Note that MyApp contains C and U. That shows the code in the script got executed when the WS was built – otherwise they wouldn’t exist. This is nice: when you type #.MyApp.C. then autocomplete pops up and suggests all the names contained in Constants.

We have reached our goal:


Footnotes

  1. With version 16.0 Dyalog has introduced a system function ⎕CSV for both importing from and exporting to CSV files.

Chapter 3:

Package MyApp as an executable

Now we will make some adjustments to prepare MyApp for being packaged as an EXE. It will run from the command line and it will run ‘headless’ – without a user interface (UI).

Copy all files in z:\code\v02\ to z:\code\v03\. Alternatively you can download version 3 from https://cookbook.dyalog.com.

1. Output to the session log

In a runtime interpreter or an EXE, there is no APL session, and output to the session which would have been visible in a development system will simply disappear.

Information

Note that output assigned to or does not stop the runtime executable.

However, when the result of a function is neither consumed by another function nor assigned to a variable then you will see the message “This Dyalog APL runtime application has attempted to use the APL session and therefore be closed.”, and that will be the end of it.

If we want to see this output, we need to write it to a log file. But how do we find out where we need to make changes? We recommend you think about this from the start, and ensure that all intentional output goes through a log function, or at least use an explicit ⎕← so that output can easily be identified in the source.

Unwanted output to the session

What can you do if you have output appearing in the session and you don’t know where in your application it is being generated? The easiest way is to associate a callback function with the SessionPrint event as in:

   '⎕se' ⎕WS 'Event' 'SessionPrint' '#.Catch'
   #.⎕FX ↑'what Catch m'  ':If 0∊⍴what' '. ⍝ !' ':Else' '⎕←what' ':Endif'
   ⎕FX 'test arg'  '⎕←arg'
   test 1 2 3
⍎SYNTAX ERROR
Catch[2] . ⍝ !

You can even use this to investigate what is about to be written to the session (the left argument of Catch) and make the function stop when it reaches the output you are looking for. In the above example we check for anything that’s empty.

Notes:

TxtToCsv has a shy result, so it won't write its result to the session. That’s fine.

2. Preparing the application

TxtToCsv needs an argument. The EXE we are about to create must fetch it from the command line. We’ll give MyApp a function StartFromCmdLine.

We will also introduce SetLX: the last line of the DYAPP will run it to set ⎕LX:

Target #
Load Constants
Load Utilities
Load MyApp
Run #.MyApp.SetLX ⍬

In MyApp.dyalog:

:Namespace MyApp

(⎕IO ⎕ML ⎕WX ⎕PP ⎕DIV)←1 1 3 15 1

    ∇r←Version
    ⍝ * 1.0.0
    ⍝   * Runs as a stand-alone EXE and takes parameters from the command line.
      r←(⍕⎕THIS) '1.0.0' 'YYYY-MM-DD'
    ∇
    ...
    ⍝ === VARIABLES ===

    Accents←'ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ' 'AAAAAACDEEEEIIIINOOOOOOUUUUY'

⍝ === End of variables definition ===

      CountLetters←{
          {⍺(≢⍵)}⌸⎕A{⍵⌿⍨⍵∊⍺}Accents map toUppercase ⍵
      }
    ...
    ∇ {r}←SetLX dummy
    ⍝ Set Latent Expression (needed in order to export workspace as EXE)
     #.⎕IO←1 ⋄ #.⎕ML←1 ⋄ #.⎕WX←3 ⋄ #.⎕PP←15 ⋄ #.⎕DIV←1
     r←⍬
     ⎕LX←'#.MyApp.StartFromCmdLine #.MyApp.GetCommandLineArgs ⍬'
    ∇

    ∇ {r}←StartFromCmdLine arg
    ⍝ Run the application; arg = usually command line parameters .
       r←⍬
       r←TxtToCsv arg~''''
    ∇

    ∇ r←GetCommandLineArgs dummy
       r←⊃¯1↑1↓2 ⎕NQ'.' 'GetCommandLineArgs' ⍝ Take the last one
    ∇

:EndNamespace

Changes are emphasised.

3. Conclusions

Now MyApp is ready to be run from the Windows command line, with the name of the file to be processed following the command name.

Notes:

Warning

Inheriting system variables

A common source of confusion is code that relies on system variables having expected values. Your preferred values for those system variables are set in the Dyalog configuration.

Whenever you execute then, say, #.⎕NS '' you can expect the resulting namespace to inherit those settings from the hosting namespace. That’s fine.

But if you send your WS elsewhere then somebody with different values in their Dyalog configuration might load and run your WS. In this environment #.⎕NS '' creates a namespace with different values for system variables: a recipe for disaster.

4. Exporting the application

We’re now nearly ready to export the first version of MyApp as an EXE.

  1. Double-click the DYAPP to create the WS.
  2. From the File menu pick Export.
  3. Pick Z:\code\v03 as the destination folder [1].
  4. From the list Save as type pick Standalone Executable.
  5. Set the File name as MyApp.
  6. Check the Runtime application checkbox.
  7. Clear the Console application checkbox.
  8. Click Save.

You should see a message: File Z:\code\v03\MyApp.exe successfully created. This occasionally (rarely) fails for no obvious reason. If it does fail just try again and you should be fine.

If it keeps failing then the by far most common reason is that the EXE is running – you cannot replace an EXE while it is running.

Information

Although you cannot replace a running EXE what you can do is to rename it; that is possible. You can then create a new EXE with the original name.

In case you wonder what a “Console application” is:

Note that it catches the return code and assigns it to the environment variable “ERRORLEVEL” in any case.

Note that you cannot really debug a console application with Ride; for details see the Debugging a stand-alone EXE chapter.

If you do not check Console application, the program is started as a separate process and you cannot catch the return code.

We therefore recommend you clear the Console application checkbox unless you have a good reason to do otherwise.

Tip

Use the Version button to bind to the EXE information about the application, author, version, copyright and so on. These pieces of information will show in the Properties/Details tab of the resulting EXE.

Note that to use the cursor keys or Home or End within a cell the Version dialog box requires you to enter ‘in-cell’ mode by pressing F2.

Tip

You could specify an icon file to replace the Dyalog icon with your own one.

5. Running the stand-alone EXE

Let’s run it. From a command line:

Z:\code\v03\MyApp.exe texts\en

Looking in Windows Explorer at Z:\texts\en.csv, we see its timestamp just changed. Our EXE works!


Footnotes

  1. Note that in the Dyalog Cookbook the words folder and directory are used interchangeably.

Chapter 4:

Logging what happens

MyApp 1.0 is now working but handles errors poorly. See what happens when we try to work on a non-existent file/folder:

Z:\code\v03\MyApp.exe Z:\texts\Does_not_exist

We see an alert message: This Dyalog APL runtime application has attempted to use the APL session and will therefore be closed.

MyApp failed because there is no file or folder Z:\texts\Does_not_exist. That triggered an error in the APL code. The interpreter tried to display an error message and looked for input from a developer from the session. But a runtime task has no session, so at that point the interpreter popped the alert message and MyApp died.

CONTINUE workspaces

Prior to version 16.0, as soon as you close the message box a CONTINUE workspace was created in the current directory. Such a CONTINUE WS can be loaded and investigated, making it easy to figure out what the problem is. (However, this is true only if it is a single-threaded application, since workspaces cannot be saved when more than one thread is running.)

With version 16.0 you can still force the interpreter to drop a CONTINUE workspace by enabling the old behaviour with 2704⌶ 1, while 2704⌶ 0 would disable it again.

For analysis, load a CONTINUE workspace in an already running Dyalog session – don’t double-click a CONTINUE! The reason is that ⎕DM and ⎕DMX are overwritten in the process of booting SALT, meaning that you lose the error message.

You might recreate them by re-executing the failing line – but that has other dangers, or might fail in a new way.

Note also that the CONTINUE is always saved in the current directory; in version 16.0 there is no way to tell the interpreter to save the CONTINUE workspace elsewhere.

That is limiting, as it will fail for your own stand-alone EXEs if they are installed in the standard folders for executables under Windows, C:\Program Files (64-bit programs) and C:\Program Files (x86) (32-bit programs): even as an admin you cannot write to those folders or subfolders.

But Windows saves it anyway! If a program attempts to write to a banned location Windows tells them “Sure, no problem” but saves them in a e.g. "C:\Users\kai\AppData\Local\VirtualStore\Program Files\Dyalog\Dyalog APL-64 16.0 Unicode\CONTINUE.dws" where you are running Dyalog APL 64-bit Unicode version 16.0.

The next version of MyApp will improve by logging what happens when it runs.

Save a copy of Z:\code\v03 as Z:\code\v04 or copy v04 from the Cookbook website.

1. Include the “Logger” class

We’ll use the APLTree Logger class, which we’ll now install in the workspace root. If you’ve not already done so, copy the APLTree library folder into Z:\code\apltree.[1] Now edit Z:\code\v04\MyApp.dyapp to include some library code:

Target #
Load ..\AplTree\APLTreeUtils
Load ..\AplTree\FilesAndDirs
Load ..\AplTree\OS
Load ..\AplTree\Logger
Load Constants
Load Utilities
Load MyApp
Run #.MyApp.SetLX ⍬

and run the DYAPP to recreate the MyApp workspace.

Help for the APLTree namespaces

You can get detailed documentation on an APLTree class or namespace by executing e.g.:

]ADoc APLTreeUtils

You’ll find more about ADoc in the chapter Documentation – the Doc is in.

The Logger class and its dependencies will now be included when we build MyApp:

Let’s get the program to log what it’s doing. Within MyApp, some changes. First we introduce aliases for the new modules:

⍝ === Aliases (referents must be defined previously)

    F←##.FilesAndDirs ⋄ A←##.APLTreeUtils   ⍝ from the APLTree lib

Note that APLTreeUtils comes with the functions Uppercase and Lowercase. We have those already in the Utilities namespace. This violates the DRY principle. We should get rid of one version and use the other everywhere. But how to choose?

First of all, almost all APLTree projects rely on APLTreeUtils. If you want to use this library then we cannot get rid of APLTreeUtils.

The two different versions both use the Dyalog function, so functionality and speed are the same.

However, APLTreeUtils is in use for more than 10 years now, it comes with a comprehensive set of test cases and it is documented in detail. That makes the choice rather easy.

Therefore we remove the two functions from Utilities and change CountLetters:

      CountLetters←{
          {⍺(≢⍵)}⌸⎕A{⍵⌿⍨⍵∊⍺}Accents U.map A.Uppercase ⍵
      }

That works because the alias A we've just introduced points to APLTreeUtils.

2. Where to keep the logfiles?

Where is MyApp to write the logfile? We need a folder we know exists. That rules out fullfilepath. We need a logfile even if fullfilepath isn’t a valid path.

We'll write logfiles into a subfolder of the current directory, which we can be sure exists. Where will that be? When the EXE launches, the current directory is set:

Z:\code\v04\MyApp.exe Z:\texts\en

Current directory is Z:\ and that’s where the logfiles will appear.

If this version of MyApp were for shipping that would be a problem. An application installed in C:\Program Files cannot rely on being able to write logfiles there. That is a problem to be solved by an installer. We’ll come to that later.

But for this version of MyApp the logfiles are for your eyes only. It’s fine to have the logfiles appear wherever you launch the EXE. You just have to know where they are. We will put them into a subfolder Logs within the current directory.

In developing and testing MyApp, we create the active workspace by running MyApp.dyapp. The interpreter sets the current directory of the active workspace as the DYAPP’s parent folder. That, too, is sure to exist.

      #.FilesAndDirs.PWD
Z:\code\v04

3. Setting up parameters for Logger

Now we set up the parameters needed to instantiate the Logger class. First we use the Logger class’ shared CreateParms method to get a parameter space with an initial set of default parameters. You can use the built-in method ∆List to display its properties and their defaults:

      #.Logger.CreateParms.∆List''
  active                   1
  autoReOpen               1
  debug                    0
  encoding              ANSI
  errorPrefix      *** ERROR
  extension              log
  fileFlag                 1
  filename
  filenamePostfix
  filenamePrefix
  filenameType          DATE
  path
  printToSession           0
  timestamp

We shall modify them to match our needs and use the parameter namespace to create the Logger object.

4. Implementing the logging function

For this we create a function OpenLogFile:

∇ instance←OpenLogFile path;logParms
  ⍝ Creates an instance of the "Logger" class.
  ⍝ Provides methods `Log` and `LogError`.
  ⍝ Make sure that `path` (that is where log files will end up) does exist.
  ⍝ Returns the instance.
  logParms←##.Logger.CreateParms
  logParms.path←path
  logParms.encoding←'UTF8'
  logParms.filenamePrefix←'MyApp'
  'CREATE!'F.CheckPath path
  instance←⎕NEW ##.Logger(,⊂logParms)
∇

Notes:

5. Initializing “Logger”

We create a function Initial (short for “Initialize”) which calls OpenLogFile and returns the Logger instance:

∇ {MyLogger}←Initial dummy
⍝ Prepares the application.
⍝

At this point Initial does nothing; that will change soon.

6. Get it to work

We also need to change ProcessFile:

∇ data←(fns ProcessFiles)files;txt;file
⍝ was: (data enc nl)←(fns Pe processFiles)files;txt;file
⍝ Reads all files and executes `fns` on the contents.
   data←⍬
   :For file :In files
       txt←'flat' A.ReadUtf8File file
       ⍝ was: (txt enc nl)←⎕NGET file
       data,←⊂fns txt
   :EndFor
∇

We use APLTreeUtils.ReadUtf8File rather than ⎕NGET because it optionally returns a flat string without a performance penalty, although that is only an issue with really large files. This is achieved by passing 'flat' as the left argument to ReadUtf8File.

We ignore encoding and the newline character and allow it to default to the current operating system.

As a side effect ProcessFiles won’t crash anymore when files is empty because enc and nl have disappeared from the function.

Now we have to make sure that Initial is called from StartFromCmdLine:

∇ {r}←StartFromCmdLine arg;MyLogger
⍝ Needs command line parameters, runs the application.
   r←⍬
   MyLogger←Initial ⍬
   MyLogger.Log'Started MyApp in ',F.PWD
   MyLogger.Log #.GetCommandLine
   r←TxtToCsv arg~''''
   MyLogger.Log'Shutting down MyApp'
∇

Note that we now log the full command line. In an application that receives its parameters from the command line, this is important to do.

7. Improvements to our code

We take the opportunity to move code from TxtToCsv to a new function GetFiles. This new function will take the command-line argument and return a list of files which may contain zero, one or many filenames:

 ∇ (target files)←GetFiles fullfilepath;csv;target;path;stem
 ⍝ Investigates `fullfilepath` and returns a list with files
 ⍝ May return zero, one or many filenames.
   fullfilepath~←'"'
   csv←'.csv'
   :If F.Exists fullfilepath
       :Select C.NINFO.TYPE ⎕NINFO fullfilepath
       :Case C.TYPES.DIRECTORY
           target←F.NormalizePath fullfilepath,'\total',csv
           files←⊃F.Dir fullfilepath,'\*.txt'
       :Case C.TYPES.FILE
           (path stem)←2↑⎕NPARTS fullfilepath
           target←path,stem,csv
           files←,⊂fullfilepath
       :EndSelect
       target←(~0∊⍴files)/target
   :Else
       files←target←''
   :EndIf
 ∇

We have to ensure GetFiles is called from TxtToCsv. Note that moving code from TxtToCsv to GetFiles allows us to keep TxtToCsv nice and tidy and the list of local variables short. In addition we have added calls to MyLogger.Log in appropriate places:

∇ rc←TxtToCsv fullfilepath;files;tbl;lines;target
⍝ Write a sibling CSV of the TXT located at fullfilepath,
⍝ containing a frequency count of the letters in the file text
   (target files)←GetFiles fullfilepath
   :If 0∊⍴files
       MyLogger.Log'No files found to process'
       rc←1
   :Else
       tbl←⊃⍪/(CountLetters ProcessFiles)files
       lines←{⍺,',',⍕⍵}/{⍵[⍒⍵[;2];]}⊃{⍺(+/⍵)}⌸/↓[1]tbl
       A.WriteUtf8File target lines
       MyLogger.Log(⍕⍴files),' file',((1<⍴files)/'s'),' processed:'
       MyLogger.Log' ',↑files
       rc←0
   :EndIf
∇

Notes:

Finally we change Version:

∇r←Version
⍝ * 1.1.0:
⍝   * Can now deal with non-existent files.
⍝   * Logging implemented.
⍝ * 1.0.0
⍝   * Runs as a stand-alone EXE and takes parameters from the command line.
  r←(⍕⎕THIS) '1.1.0' '2017-02-26'
∇

The foreseeable error that aborted the runtime task – an invalid filepath – has now been replaced by a message saying no files were found.

We have also changed the explicit result. So far it has returned the number of bytes written. In case something goes wrong (file not found, etc.) it will now return ¯1.

We can now test TxtToCsv:

      #.MyApp.TxtToCsv 'Z:\texts\en'
      ⊃(⎕NINFO⍠1) 'Logs\*.LOG'
 MyApp_20160406.log
      ↑⎕NGET 'Logs\MyApp_20160406.log'
2016-04-06 13:42:43 *** Log File opened
2016-04-06 13:42:43 (0) Started MyApp in Z:\
2016-04-06 13:42:43 (0) Source: Z:\texts\en
2016-04-06 13:42:43 (0) Target: Z:\texts\en.csv
2016-04-06 13:42:43 (0) 244 bytes written to Z:\texts\en.csv
2016-04-06 13:42:43 (0) All done
Information

Alternatively you could set the parameter printToSession – which defaults to 0 – to 1. That would let the Logger class write all the messages not only to the log file but also to the session. That can be quite useful for test cases or during development. (You can even stop the Logger class writing to the disk at all by setting fileFlag to 0.)

Information

The Logger class is designed never to break your application – for obvious reasons. The drawback of this is that if something goes wrong, such as the path becoming invalid because the drive got removed, you would notice only by trying to examine the log files.

You can tell the Logger class that it should not trap all errors by setting the parameter debug to 1. Then Logger will crash if something goes wrong.

Let’s see if logging works also for the exported EXE. Run the DYAPP to rebuild the workspace. Export as before and then run the new MyApp.exe in a Windows console.

Z:\code\v04\MyApp.exe Z:\texts\en

Yes! The output TXT gets produced as before, and the work gets logged in Z:\Logs.

Let’s see what happens now when the filepath is invalid.

Z:\code\v04\MyApp.exe Z:\texts\de

No warning message – the program made an orderly finish. And the log?

      ↑⎕NGET 'Logs\MyApp_20160406.log'
2017-02-26 10:54:01 *** Log File opened
2017-02-26 10:54:01 (0) Started MyApp in Z:\code\v04
2017-02-26 10:54:01 (0) Source: G:\Does_not_exist
2017-02-26 10:54:01 (0) No files found to process
2017-02-26 10:54:26 *** Log File opened
2017-02-26 10:54:26 (0) Source: "Z:\texts\en\ageofinnocence.txt"
2017-02-26 10:54:26 (0) Started MyApp in Z:\code\v04
2017-02-26 10:54:26 (0) 1 file processed.
2017-02-26 10:58:07 (0) Z:/texts/en/ageofinnocence.txt
2017-02-26 10:54:35 *** Log File opened
2017-02-26 10:54:35 (0) Started MyApp in Z:\code\v04
2017-02-26 10:54:35 (0) Source: "Z:\texts\en\"
2017-02-26 10:54:35 (0) 9 files processed.
2017-02-26 10:58:07 (0) Z:/texts/en/ageofinnocence.txt
...
Information

In case you wonder what the (0) in the log file stands for: this reports the thread number that has written to the log file. Since we do not use threads, this is always (0) = the main thread the interpreter is running in.

One more improvement in MyApp: we change the setting of the system variables from

:Namespace MyApp

    (⎕IO ⎕ML ⎕WX ⎕PP ⎕DIV)←1 1 3 15 1
    ....

to the more readable:

:Namespace MyApp

    ⎕IO←1 ⋄ ⎕ML←1 ⋄ ⎕WX←3 ⋄ ⎕PP←15 ⋄ ⎕DIV←1
    ....

8. Watching the log file with LogDog

So far we have used modules from the APLTree project: class and namespace scripts that might be useful when implementing an application.

APLTree also offers applications that support the programmer during her work without becoming part of the application. One of those applications is the LogDog.

Its purpose is simply to watch a log file and reflect any changes immediately in the GUI. This is useful for us, as the log file is now our best view of how the application is doing.

In order to use LogDog you first need to download it from http://download.aplwiki.com. Download it into the default download location. For a user JohnDoe that would be C:\Users\JohnDoe\Downloads.

LogDog does not come with an installer. All you have to do is to copy it into a folder where you have the right to add, delete and change files. That means C:\Program Files and C:\Program Files (x86) are not options.

If you want to install the application just for your own user ID then this is the right place:

"C:\Users\JohnDoe\AppData\Local\Programs\LogDog

If you want to install it for all users on your PC then we suggest that you create this folder:

"C:\Users\All users\Local\Programs\LogDog

Of course C:\MyPrograms\LogDog might be okay as well.

You start LogDog by double-clicking the EXE. You can then consult LogDog’s help for how to open a log file.

We recommend the Investigate folder option. The reason is: every night at 24:00 a new log file with a new name is created. To display any new(er) log file, issue the Investigate folder menu command again.

Once you have started LogDog on the MyApp log file you will see something like this:

LogDog GUI

Note that LogDog comes with an auto-scroll feature, meaning that the latest entries at the bottom of the file are always visible. If you don't want this for any reason just tick the Freeze checkbox.

From now on we will assume you have LogDog always up and running, so that you will get immediate feedback on what is going on when MyApp.exe runs.

9. Where are we

We now have MyApp logging its work in a subfolder of the application folder and reporting any problems it has anticipated.

Next we need to consider how to handle and report errors we have not anticipated. We should also return some kind of error code to Windows. If MyApp encounters an error, any process calling it needs to know. But before we are doing this we will discuss how to configure MyApp.

Destructors versus the Tracer

When you trace through TxtToCsv, the moment you leave the function the Tracer shows the function Cleanup of the Logger class. The function is declared as a destructor.

Why that is: a destructor (if any) is called when the instance of a class is destroyed (or shortly thereafter).

MyLogger is localized in the header of TxtToCsv, meaning that when TxtToCsv ends, this instance of the Logger class is destroyed and the destructor is invoked. Since the Tracer was up and running, the destructor makes an appearance in the Tracer.


Footnotes

  1. You can download all members of the APLTree library from the APL Wiki http://download.aplwiki.com/ or from the project pages on GitHub:
    search for “apltree” to get a full list. Note that all apltree projects are owned by “aplteam”.

  2. Details regarding the BOM:
    https://en.wikipedia.org/wiki/Byte_order_mark

Chapter 5:

Configuration settings

We want our logging and error handling to be configurable. In fact, we will soon have lots of state settings. Thinking more widely, an application’s configuration includes all kinds of state: e.g. folders for log files and crashes, a debug flag, a flag for switching off error trapping, an email address to report to – you name it.

Several mechanisms are available for storing configuration settings. Microsoft Windows has the Windows Registry. There are also cross-platform file formats to consider: XML, JSON – and good old INI files.

1. The Windows Registry

The Windows Registry is held in memory, so it is fast to read. It has been widely used to store configuration settings. Some would say, abused. However, for quite some time it was considered bad practice to have application-specific config files.

Everything was expected to go into the Windows Registry. The pendulum started to swing back the other way now for several years, and application-specific config files become ever more common. We follow a consensus opinion that it is best to minimise the use of the Registry.

Settings needed by Windows itself have to be stored in the Registry. For example, associating a file extension with your application, so that double-clicking on its icon launches your application.

The APLTree classes WinRegSimple and WinReg provide methods for handling the Windows Registry. We will discuss them in their own chapter.

MyApp doesn’t need the Windows Registry at this point. We’ll store its configurations in configuration files.

Information

The Windows Registry is still an excellent choice for saving user-specific stuff like preferences, themes, recent files etc. However, you have to make sure that your user has permission to write to the Windows Registry – that's by no means a certainty.

2. INI, JSON, or XML configuration files?

Three formats are popular for configuration files: INI, JSON and XML. INI is the oldest, simplest, and most crude. The other formats offer advantages: XML can represent nested data structures, and JSON can do so with less verbosity.

Both XML and JSON depend upon unforgiving syntax: a single typo in an XML document can render it impossible to parse.

We want configuration files to be suitable for humans to read and write, so you might consider the robustness of the INI format an advantage. Or a disadvantage: a badly-formed XML document is easy to detect, and a clear indication of an error.

Generally, we prefer simplicity and recommend the INI format where it will serve.

By using the APLTree class IniFiles we get as a bonus additional features:

We will discuss these features as we go along.

3. INI files it is!

3.1. Where to save an INI file

In the chapter on Logging, we considered the question of where to keep application logs. The answer depends in part on what kind of application you are writing. Will there be single or multiple instances?

For example, while a web browser might have several windows open simultaneously, it is nonetheless a single instance of the application. Its user wants to run just one version of it, and for it to remember her latest preferences and browsing history.

But a machine may have many users, and each user needs her own preferences and history remembered.

Our MyApp program might well form part of other software processes, perhaps running as a service. There might be multiple instances of MyApp running at any time, quite independently of each other, each with quite different configuration settings.

Where does that leave us? We want configuration settings:

As defaults for the application in the absence of any other configuration settings, for all users

These must be coded into the application (‘Convention over configuration’), so it will run in the absence of any configuration files.

But an administrator should be able to revise these settings for a site. So they should be saved somewhere for all users. This filepath is represented in Windows by the ALLUSERSPROFILE environment variable. So we might look there for a MyApp\MyApp.ini file.

For invocation when the application is launched
We could look in the command-line arguments for an INI.
As part of the user’s profile

The Windows environment variable APPDATA points to the individual user’s roaming profile, so we might look there for a MyApp\MyApp.ini file. Roaming means that no matter which computer a user logs on to in a Windows Domain [1], her personal settings, preferences, desktop etc. roam with her.

The Windows environment variable LOCALAPPDATA on the other hand defines a folder that is saved just locally. Typically APPDAATA points to something like C:\Users\{username}\AppData\Roaming and LOCALAPPDATA to C:\Users\{username}\AppData\Local.

Information

Note that when a user logs on to another computer all the files in APPDATA are synchronised first. Therefore it is not smart to save in APPDATA a logfile that will eventually grow large – put it into LOCALAPPDATA.

From the above we get a general pattern for configuration settings:

  1. Defaults in the program code
  2. Overwrite from ALLUSERSPROFILE if any
  3. Overwrite from USERPROFILE
  4. Overwrite from an INI specified on the command line
  5. Overwrite with the command line

However, for the Cookbook we keep things simple: we look for an INI file that is a sibling of the DYAPP or the EXE for now but will allow this to be overwritten via the command line with something like INI='C:\MyAppService\MyApp.ini.

We need this when we make MyApp a Windows Scheduled Task, or run it as a Windows Service.

3.2. Let’s start

Save a copy of Z:\code\v04 as Z:\code\v05 or copy v05 from the Cookbook website. We add one line to MyApp.dyapp:

...
Load ..\AplTree\FilesAndDirs
leanpub-insert-start
Load ..\AplTree\IniFiles
leanpub-insert-end
Load ..\AplTree\OS
...

and run the DYAPP to recreate the MyApp workspace.

You can read the IniFiles documentation in a browser with ]ADoc #.IniFiles.

3.3. Our INI file

This is the content of the newly introduced code\v05\MyApp.ini:

localhome = '%LOCALAPPDATA%\MyApp'

[Config]
Debug       = ¯1    ; 0=enfore error trapping; 1=prevent error trapping;
Trap        = 1     ; 0 disables any :Trap statements (local traps)

Accents     = ''
Accents     ,='ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ'
Accents     ,='AAAAAACDEEEEIIIINOOOOOOUUUUY'

[Folders]
Logs        = '{localhome}\Log'
Errors      = '{localhome}\Errors'

If you have not copied v05 from the website make sure you create an INI file with this content as a sibling of the DYAPP.

Notes:

3.4. Initialising the workspace

We create a new function CreateConfig for that:

∇ Config←CreateConfig dummy;myIni;iniFilename
⍝ Instantiate the INI file and copy values over to a namespace `Config`.
  Config←⎕NS''
  Config.⎕FX'r←∆List' 'r←{0∊⍴⍵:0 2⍴'''' ⋄ ⍵,[1.5]⍎¨⍵}'' ''~¨⍨↓⎕NL 2'
  Config.Debug←A.IsDevelopment
  Config.Trap←1
  Config.Accents←'ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ' 'AAAAAACDEEEEIIIINOOOOOOUUUUY'
  Config.LogFolder←'./Logs'
  Config.DumpFolder←'./Errors'
  iniFilename←'expand'F.NormalizePath'MyApp.ini'
  :If F.Exists iniFilename
      myIni←⎕NEW ##.IniFiles(,⊂iniFilename)
      Config.Debug{¯1≡⍵:⍺ ⋄ ⍵}←myIni.Get'Config:debug'
      Config.Trap←⊃Config.Trap myIni.Get'Config:trap'
      Config.Accents←⊃Config.Accents myIni.Get'Config:Accents'
      Config.LogFolder←'expand'F.NormalizePath⊃Config.LogFolder myIni.Get'Folders:Logs'
      Config.DumpFolder←'expand'F.NormalizePath⊃Config.DumpFolder myIni.Get'Folders:Errors'
  :EndIf
  Config.LogFolder←'expand'F.NormalizePath Config.LogFolder
  Config.DumpFolder←'expand'F.NormalizePath Config.DumpFolder
∇

What the function does:

Notes:

The built-in function ∆List is handy for checking the contents of Config:

      Config.∆List
 Accents      ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ  AAAAAACDEEEEIIIINOOOOOOUUUUY
 Debug                                                                  0
 DumpFolder                          C:\Users\kai\AppData\Local\MyApp\Log
 LogFolder                           C:\Users\kai\AppData\Local\MyApp\Log
 Trap                                                                   1

Now that we have moved Accents to the INI file we can lose these lines in the MyApp script:

⍝ === VARIABLES ===
    Accents←'ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ' 'AAAAAACDEEEEIIIINOOOOOOUUUUY'
⍝ === End of variables definition ===

Where should we call CreateConfig from? Surely that has to be Initial:

leanpub-start-insert
∇ (Config MyLogger)←Initial dummy
⍝ Prepares the application.
  Config←CreateConfig ⍬
  MyLogger←OpenLogFile Config.LogFolder
  MyLogger.Log'Started MyApp in ',F.PWD
  MyLogger.Log #.GetCommandLine
  MyLogger.Log↓⎕FMT Config.∆List
∇

Note that we also changed what Initial returns: a vector of length two, the namespace Config but also an instance of the MyLogger class.

Initial was called within StartFromCmdLine, and we are not going to change this but we must change the call as such because now it returns something useful:

leanpub-start-insert
∇ {r}←StartFromCmdLine arg;MyLogger;Config
⍝ Needs command line parameters, runs the application.
  r←⍬
  (Config MyLogger)←Initial ⍬
  r←TxtToCsv arg~''''
∇

Although both MyLogger and Config are global and not passed as arguments, it’s good practice to assign them this way rather than bury their creation somewhere down the stack. This way it’s easy to see where they are set.

Specifying an INI file on the command line

We could pass the command line parameters as arguments to Initial and investigate whether it carries any INI= statement. If so the INI file specified this way should take precedence over any other INI file. However, we keep it simple here.

We now need to think about how to access Config from within TxtToCsv.

3.5. What we think about when we think about encapsulating state

The configuration parameters, including Accents, are now collected in the namespace Config. That namespace is not passed explicitly to TxtToCsv but is needed by CountLetters which is called by TxtToCsv.

We have two options here: we can pass a reference to Config to TxtToCsv, for example as left argument, and TxtToCsv in turn can pass it to CountLetters. The other option is that CountLetters just assumes the Config is around and has a variable Accents in it:

CountLetters←{
    {⍺(≢⍵)}⌸⎕A{⍵⌿⍨⍵∊⍺}Config.Accents U.map A.Uppercase ⍵
}

Yes, that’s it. Bit of a compromise here. Let’s pause to look at some other ways to write this.

Passing everything through function arguments does not come with a performance penalty. The interpreter doesn’t make ‘deep copies’ of the arguments unless and until they are modified in the called function (which we hardly ever do) – instead the interpreter just passes around references to the original variables.

So we could pass G as a left argument of TxtToCsv, which then simply gets passed to CountLetters.

No performance penalty for this, as just explained, but now we’ve loaded the syntax of TxtToCsv with a namespace it makes no direct use of, an unnecessary complication of the writing. And we’ve set a left argument we (mostly) don't want to specify when working in session mode.

The matter of encapsulating state – which functions have access to state information, and how it is shared between them – is very important. Poor choices lead to tangled and obscure code.

From time to time you will be offered (not by us) rules that attempt to make the choices simple. For example: never communicate through global or semi-global variables. [2].

There is some wisdom in these rules, but they masquerade as satisfactory substitutes for thought, which they are not.

Just as in a natural language, any rule about writing style meets occasions when it can and should be broken.

Following style ‘rules’ without considering the alternatives will from time to time have horrible results, such as functions that accept complex arguments only to pass them on unexamined to other functions.

Think about the value of style ‘rules’ and learn when to follow them.

One of the main reasons why globals should be used with great care is that they can easily be confused with local variables with similar or – worse – the same name.

If you need to have global variables then we suggest encapsulating them in a dedicated namespace Globals. With a proper search tool like Fire [3] it is easy to get a report on all lines referring to anything in Globals.

Sometimes it’s only after writing many lines of code that it becomes apparent that a different choice would have been better.

And sometimes it becomes apparent that the other choice would be so much better than it’s worth unwinding and rewriting a good deal of what you’ve done. (Then rejoice that you’re writing in a terse language.)

We share these musings here so you can see what we think about when we think about encapsulating state; and also that there is often no clear right answer.

Think hard, make your best choices, and be ready to unwind and remake them later if necessary.

3.6. The IniFiles class

We have used the most important features of the IniFiles class, but it has more to offer. We just want to mention some major topics here.

3.7. Final steps

We need to change the Version function:

∇ r←Version
   ⍝ * 1.2.0:
   ⍝   * The application now honours INI files.
   ⍝ * 1.1.0:
   ⍝   * Can now deal with non-existent files.
   ⍝   * Logging implemented.
   ⍝ * 1.0.0
   ⍝   * Runs as a stand-alone EXE and takes parameters from the command line.
      r←(⍕⎕THIS)'1.2.0' '2017-02-26'
∇

And finally we create a new standalone EXE as before and run it to make sure that everything keeps working. (Yes, we need test cases)


Footnotes

  1. https://en.wikipedia.org/wiki/Windows_domain

  2. So-called semi-globals are variables to be read or set by functions to which they are not localised. They are semi-globals, rather than globals, because they are local to either a function or a namespace. From the point of view of the functions that do read or set them, they are indistinguishable from globals – they are just mysteriously ‘around’.

  3. Fire stands for Find and Replace. It is a powerful tool for both search and replace operations in the workspace. For details see https://github.com/aplteam.Fire. Fire is discussed in the chapter Useful user commands.

Chapter 6:

Debugging a stand-alone EXE

Imagine the following situation: MyApp is started with a double-click on the DYAPP and, when tested, everything works just fine. Then you create a stand-alone EXE from the DYAPP and execute it with some appropriate parameter, but it does not create the CSV files.

In this situation, obviously you need to debug the EXE. In this chapter we’ll discuss how to achieve that. In addition we will make MyApp.exe return an exit code.

For debugging we are going to use Ride. (See the Dyalog manuals for information about Ride.) If enabled, you can use Ride to hook into a running interpreter, interrupt any running code, investigate, and even change that code.

1. Going for a ride

We introduce a [RIDE] section into the INI file:

[Ride]
Active      = 1
Port        = 4599
Wait        = 1

By setting Active to 1 and defining a Port number for the communication between Ride and the EXE you can tell MyApp that you want ‘to take it for a ride’. Setting Wait to 1 lets the application wait for a ride. That simply means it enters an endless loop.

That’s not always appropriate of course, because it allows anybody to read your code.

If that's something you need to avoid, you have to find other ways to make the EXE communicate with Ride, perhaps by making temporary changes to the code.

The approach would be the same in both cases. In MyApp we keep things simple and allow the INI file to rule whether the user may ride into the application or not.

Copy Z:\code\v05 to Z:\code\v06 and then run the DYAPP to recreate the MyApp workspace.

Information

Note that 4502 is Ride’s default port, and that we’ve settled for a different port, and for good reasons. Using the default port leaves room for mistakes.

Using a dedicated port rather than using the default minimises the risk of connecting to the wrong application.

2. The Console application flag

If you exported the EXE with the Console application checkbox ticked there is a problem. You can connect to the EXE with Ride, but all output goes into the console window.

That means you can enter statements in Ride but any response from the interpreter goes to the console window rather than Ride.

For debugging we therefore recommend creating the EXE with the check box cleared.

3. Code changes

3.1. Making Ride configurable

We want to make the ride configurable. That means we cannot do it earlier than after having instantiated the INI file. But not long after either, so we change Initial:

∇ (Config MyLogger)←Initial dummy
⍝ Prepares the application.
  Config←CreateConfig ⍬
  CheckForRide Config.(Ride WaitForRide)
  MyLogger←OpenLogFile Config.LogFolder
  MyLogger.Log'Started MyApp in ',F.PWD
  MyLogger.Log #.GetCommandLine
  MyLogger.Log↓⎕FMT Config.∆List
∇

We have to ensure Ride makes it into Config, so we establish a default 0 (no Ride) and overwrite with INI settings.

∇ Config←CreateConfig dummy;myIni;iniFilename
  Config←⎕NS''
  Config.⎕FX'r←∆List' 'r←{0∊⍴⍵:0 2⍴'''' ⋄ ⍵,[1.5]⍎¨⍵}'' ''~¨⍨↓⎕NL 2'
  Config.Debug←A.IsDevelopment
  Config.Trap←1
  Config.Accents←'ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ' 'AAAAAACDEEEEIIIINOOOOOOUUUUY'
  Config.LogFolder←'./Logs'
  Config.DumpFolder←'./Errors'
  Config.Ride←0        ⍝ If not 0 the app accepts a Ride & treats Config.Ride as port number.
  Config.WaitForRide←0 ⍝ If 1 `CheckForRide` will enter an endless loop.
  iniFilename←'expand'F.NormalizePath'MyApp.ini'
  :If F.Exists iniFilename
      myIni←⎕NEW ##.IniFiles(,⊂iniFilename)
      Config.Debug{¯1≡⍵:⍺ ⋄ ⍵}←myIni.Get'Config:debug'
      Config.Trap←⊃Config.Trap myIni.Get'Config:trap'
      Config.Accents←⊃Config.Accents myIni.Get'Config:Accents'
      Config.LogFolder←'expand'F.NormalizePath⊃Config.LogFolder myIni.Get'Folders:Logs'
      Config.DumpFolder←'expand'F.NormalizePath⊃Config.DumpFolder myIni.Get'Folders:Errors'
      :If myIni.Exist'Ride'
      :AndIf myIni.Get'Ride:Active'
          Config.Ride←⊃Config.Ride myIni.Get'Ride:Port'
          Config.WaitForRide←⊃Config.Ride myIni.Get'Ride:Wait'
      :EndIf
  :EndIf
  Config.LogFolder←'expand'F.NormalizePath Config.LogFolder
  Config.DumpFolder←'expand'F.NormalizePath Config.DumpFolder
∇

As a result Config.Ride will be 0 if the INI rules that no Ride is permitted, otherwise the port number to be used by Ride.

3.2. Allowing a Ride

We add a function CheckForRide:

∇ {r}←CheckForRide (ridePort waitFlag);rc;init;msg
 ⍝ Depending on what's provided as right argument we prepare for a Ride
 ⍝ or we don't. In case `waitFlag` is 1 we enter an endless loop.
  r←1
  :If 0<ridePort
      {}3502⌶0                     ⍝ Switch Ride off
      init←'SERVE::',⍕ridePort     ⍝ Initialisation string
      rc←3502⌶ini                  ⍝ Specify INIT string
      :If 32=rc
          11⎕Signal⍨'Cannot Ride: Conga DLLs are missing'
      :ElseIf 64=rc
          11 ⎕Signal⍨'Cannot Ride; invalid initialisation string: ',ini
      :ElseIf 0≠rc
          msg←'Problem setting the Ride connecion string to SERVE::'
          msg,←,(⍕ridePort),', rc=',⍕rc
          11 ⎕SIGNAL⍨msg
      :EndIf
      rc←3502⌶1
      :If ~rc∊0 ¯1
          11 ⎕SIGNAL⍨'Switching on Ride failed, rc=',⍕rc
      :EndIf
      {}{_←⎕DL ⍵ ⋄ ∇ ⍵}⍣(⊃waitFlag)⊣1  ⍝ Endless loop for an early RIDE
  :EndIf
∇

Notes:

Finally we amend the Version function:

∇r←Version
   ⍝ * 1.3.0:
   ⍝   * MyApp gives a Ride now, INI settings permitted.
   ...
∇

Now you can start Ride, enter both 'localhost' and the port number as parameters, connect to the interpreter or stand-alone EXE etc., and then pick Strong interrupt from the Actions menu to interrupt the endless loop; you can then start debugging the application.

Note that this does not require the development EXE to be involved: it may well be a runtime EXE.

NB You need a development licence to be legally entitled to Ride into an application run by the RunTime EXE (DyalogRT.exe).

4. DLLs required by Ride

Prior to version 16.0 one had to copy these files :

or these:

as siblings of the EXE. From 16.0 onward you must copy the Conga DLLs instead.

Neglecting that will make 3502⌶1 fail. Note that 2.7 refers to the version of Conga, not Ride.

Prior to version 3.0 of Conga every application (interpreter, Ride, etc.) needed their own copy of the Conga DLLs, with a different name.

Since 3.0 Conga can serve several applications in parallel. We suggest you copy the 32-bit and the 64-bit DLLs as siblings of your EXE.

If you forgot to copy the DLLs you will see an error Can't find Conga DLL. This is because the OS does not bother to tell you about dependencies.

You need a tool like DependencyWalker for finding out exactly what’s missing. (We said OS because this is not a Windows-only problem.)

Restartable functions

Not only do we try to exit functions at the bottom, we also like them to be restartable.

What we mean by that is that we want if possible a function – and its variables – to survive →1. That is not possible for all functions: for example, a function that starts a thread and must not start a second one for the same task, or a file was tied etc. But most functions can be restartable.

That means that something like this should be avoided:

∇r←MyFns arg
r←⍬
:Repeat
    r,← DoSomethingSensible ⊃arg
:Until 0∊⍴arg←1↓arg

This function does not make much sense but the point is that the right argument is overwritten; so one cannot restart this function with →1. Don’t do overwrite an argument without a very good reason. In this example, a counter is a better way to iterate. (Faster, too.)

Chapter 7:

Handling errors

MyApp already anticipates, tests for and reports certain foreseeable problems with the parameters. We’ll now move on, to handle errors more comprehensively.

1. What are we missing?

  1. Other problems are foreseeable. The file system is a rich source of ephemeral problems and displays. Many of these are caught and handled by the APLTree utilities.

    They might make several attempts to read or write a file before giving up and signalling an error. Hooray. We need to handle the events signalled when the utilities give up.

  2. The MyApp EXE terminates with an all-OK zero exit code even when it has caught and handled an error.

    It would be a better Windows citizen if it returned custom exit codes, letting a calling program know how it terminated.

  3. By definition, unforeseen problems haven’t been foreseen.

    But we foresee there will be some! A mere typo in the code could break execution. We need a master trap to catch any events that would break execution, save them for analysis, and report them in an orderly way.

We'll start with the second item from the list above: quitting and passing an exit code.

2. Inspecting Windows exit codes

How do you see the exit code returned to Windows? You can access it in the command shell like this:

Z:\code\v05\MyApp.exe Z:\texts\en

echo Exit Code is %errorlevel%
Exit Code is 0

MyApp.exe Z:\texts\does_not_exist

echo Exit Code is %errorlevel%
Exit Code is 101

but only if you ticked the checkbox Console application in the Export dialog box. We don’t want to do this if we can help it, because we cannot ride into an application with this option active. Therefore we are going to execute our stand-alone EXE from now on with the help of the APLTree class Execute.

Copy Z:\code\v06 to Z:\code\v07.

For the implementation of global error handling we need APLTree's HandleError class. For calling the exported EXE we need the Execute class. Therefore we add both to the DYAPP. Edit Z:\code\v07\MyApp.dyapp:

Target #
Load ..\AplTree\APLTreeUtils
Load ..\AplTree\FilesAndDirs
Load ..\AplTree\HandleError
Load ..\AplTree\Execute
Load ..\AplTree\Logger
Load Constants
Load Utilities
Load MyApp
Run MyApp.SetLX

3. Foreseen errors

For foreseen errors we check in the code and quit when something is wrong, and pass an error code to the calling environment.

First we define in #.MyApp a child namespace of exit codes:

    :Namespace EXIT
        OK←0
        INVALID_SOURCE←111
        SOURCE_NOT_FOUND←112
        UNABLE_TO_READ_SOURCE←113
        UNABLE_TO_WRITE_TARGET←114
          GetName←{
              l←' '~¨⍨↓⎕NL 2
              ind←({⍎¨l}l)⍳⍵
              ind⊃l,⊂'Unknown error'
          }
    :EndNamespace

We define an OK value of zero for completeness; we really are trying to eliminate from our functions numerical constants that the reader has to interpret. In Windows, an exit code of zero is a normal exit.

All the exit codes are defined in this namespace. The function code can refer to them by name, so the meaning is clear. And this is the only definition of the exit-code values.

We can convert the numeric value back to the symbolic name with the function GetName:

      EXIT.GetName EXIT.INVALID_SOURCE
INVALID_SOURCE

This is useful when we want to log an error code: the name is telling while the number is meaningless.

Information

We could have defined EXIT in #.Constants, but we reserve that script for Dyalog constants, keeping it as a component that could be used in other Dyalog applications. The exit codes defined in EXIT are specific to MyApp, so are better defined there.

3.1. Passing an exit code to the caller

Now the result of TxtToCsv gets passed to Off to be returned to the operating system as an exit code.

∇ StartFromCmdLine;exit;args;rc
 ⍝ Read command parameters, run the application
  args←⌷2 ⎕NQ'.' 'GetCommandLineArgs'
  rc←TxtToCsv 2⊃2↑args
  Off rc

Note that in this particular case we set a local variable rc. Strictly speaking, this is not necessary. We learned from experience not to call several functions on a single line with the left-most being Off. (If you do, you will regret it one day.)

3.2. The function Off

Now we introduce a function Off:

∇ Off exitCode
  :If 0<⎕NC'MyLogger'
      :If exitCode=EXIT.OK
          MyLogger.Log'MyApp is closing down gracefully'
      :Else
          MyLogger.LogError exitCode('MyApp is unexpectedly shutting down: ',EXIT.GetName exitCode)
      :EndIf
  :EndIf
  :If A.IsDevelopment
      →
  :Else
      ⎕OFF exitCode
  :EndIf
∇
Information

In case you wonder about ⎕OFF: that's actually a niladic function. Being able to provide a ‘right argument’ is therefore a kind of cheating because there can’t be any. This is a special case in the Dyalog parser.

Note that ⎕OFF is actually only executed when the program detects a runtime environment, otherwise it just quits. Although the workspace is much less important in these days of scripts you still don’t want to lose it by accident.

We modify GetFiles so that it checks its arguments and the intermediary results:

leanpub-start-insert
∇ (rc target files)←GetFiles fullfilepath;csv;target;path;stem;isDir
⍝ Checks argument and returns a list of files (or a single file).
   fullfilepath~←'"'
   files←target←''
   :If 0∊⍴fullfilepath
       rc←EXIT.INVALID_SOURCE
       :Return
   :EndIf
   csv←'.csv'
   :If 0=F.Exists fullfilepath
       rc←EXIT.SOURCE_NOT_FOUND
   :ElseIf ~isDir←F.IsDir fullfilepath
   :AndIf ~F.IsFile fullfilepath
       rc←EXIT.INVALID_SOURCE
   :Else
       :If isDir
           target←F.NormalizePath fullfilepath,'\total',csv
           files←⊃F.Dir fullfilepath,'/*.txt'
       :Else
           (path stem)←2↑⎕NPARTS fullfilepath
           target←path,stem,csv
           files←,⊂fullfilepath
       :EndIf
       target←(~0∊⍴files)/target
       rc←EXIT.OK
   :EndIf
∇

Note that we have replaced some constants by calls to functions in FilesAndDirs. You might find this easier to read.

In general, we like functions to start at the top and exit at the bottom. Returning from the middle of a function can lead to confusion, and we have acquired a great respect for our capacity to get confused.

However, here we don’t mind exiting the function with :Return on line 5. It’s obvious why that is, and it saves us one level of nesting in the control structures. Also, there is no tidying up at the end of the function that we would miss with :Return.

3.3. Trapping errors

ProcessFile now traps some errors:

∇ data←(fns ProcessFiles)files;txt;file
 Reads all files and executes `fns` on the contents.
  data←⍬
  :For file :In files
      :Trap Config.Trap/FileRelatedErrorCodes
          txt←'flat'A.ReadUtf8File file
      :Case
          MyLogger.LogError'Unable to read source: ',file
          Off EXIT.UNABLE_TO_READ_SOURCE
      :EndTrap
      data,←⊂fns txt
  :EndFor
∇

In the line with the :Trap we call a niladic function (an exception to our rule!) which returns all error codes that are related to problems with files:

∇ r←FileRelatedErrorCodes
⍝ Returns all the error codes that are related to files and directories.
⍝ Useful to trap all those errors.
  r←12 18 20 21 22 23 24 25 26 28 30 31 32 34 35
∇

Doesn’t that breach our policy of avoiding unintelligible constants in the code? It does indeed.

Let’s fix this. There is a class EventCodes available on the APLTree that contains symbolic names for all these error numbers. The symbolic names are taken from the help page you get when you press F1 on ⎕TRAP. Add this class to your DYAPP file:

...
Load ..\AplTree\Logger
Load ..\AplTree\EventCodes.dyalog
Load Constants
Load Utilities
Load MyApp
Run #.MyApp.SetLX ⍬

The EventCodes class comes with a method GetName that, when fed with an integer, returns the corresponding symbolic name. We can use that to convert return codes to meaningful names:

      #.EventCodes.GetName¨ #.MyApp.FileRelatedErrorCodes
HOLD_ERROR  FILE_TIE_ERROR  FILE_INDEX_ERROR  FILE_FULL  FILE_NAME_ERROR...

We can convert this into something that will be useful when we change the function FileRelatedErrorCodes:

      ⍪'r,←E.'∘,¨#.EventCodes.GetName¨#.MyApp.FileRelatedErrorCodes
 r,←E.HOLD_ERROR
 r,←E.FILE_TIE_ERROR
 r,←E.FILE_INDEX_ERROR
 r,←E.FILE_FULL
 ...

Now we can change FileRelatedErrorCodes by copying what we've just printed to the session into the function:

∇ r←FileRelatedErrorCodes;E
⍝ Returns all the error codes that are related to files and directories.
⍝ Useful to trap all those errors.
  r←''
  E←##.EventCodes
  r,←E.HOLD_ERROR
  r,←E.FILE_TIE_ERROR
  r,←E.FILE_INDEX_ERROR
  r,←E.FILE_FULL
  r,←E.FILE_NAME_ERROR
  r,←E.FILE_DAMAGED
  r,←E.FILE_TIED
  r,←E.FILE_TIED_REMOTELY
  r,←E.FILE_SYSTEM_ERROR
  r,←E.FILE_SYSTEM_NOT_AVAILABLE
  r,←E.FILE_SYSTEM_TIES_USED_UP
  r,←E.FILE_TIE_QUOTA_USED_UP
  r,←E.FILE_NAME_QUOTA_USED_UP
  r,←E.FILE_SYSTEM_NO_SPACE
  r,←E.FILE_ACCESS_ERROR_CONVERTING_FILE
∇

Why don’t we just :Trap all errors?

:Trap 0 would trap all errors - way easier to read and write, so why don’t we do this?

Well, for a very good reason: trapping everything includes such basic things as a VALUE ERROR, most likely introduced by a typo, or removing a function you thought not called anywhere.

We don’t want to trap those errors during development. The sooner they come to light the better. For that reason we restrict the errors to be trapped to whatever might pop up when it comes to dealing with files and directories.

Your shipped system must trap all errors. (See Unforeseen errors below.) When you have to trap all errors, use a global flag which will allow you to switch it off in development. :Trap trapFlag/0: if trapFlag is 1 then the trap is active, otherwise it is not.

Back to ProcessFiles. Note that in this context the :Trap structure has an advantage over ⎕TRAP. When it fires, and control advances to its :Else fork, the trap is immediately cleared.

This neatly avoids the following pitfall: a trap fires and invokes a handling expression. But the handling expression also breaks, re-invoking the trap in what now becomes an open loop.

So with :Trap there is no need to reset the trap to avoid an open loop. But you must still consider what might happen if you call other functions in the :Else fork: if they crash the :Trap would fire again!

The handling of error codes and messages can easily obscure the rest of the logic. Clarity is not always easy to find but is worth striving for. This is particularly true where there is no convenient test for an error, only a trap for when it is encountered.

Note that here for the first time we take advantage of the [Config]Trap flag defined in the INI file, which translates to Config.Trap at this stage. With this flag we can switch off all ‘local’ error trapping, a measure we sometimes need to take to get to the bottom of a problem.

Finally we need to amend TxtToCsv:

    ∇ exit←TxtToCsv fullfilepath;∆;isDev;Log;LogError;files;target;success
     ⍝ Write a sibling CSV of the TXT located at fullfilepath,
     ⍝ containing a frequency count of the letters in the file text
     ⍝ Returns one of the values defined in `EXIT`.
      (rc target files)←GetFiles fullfilepath
      :If rc=EXIT.OK
          :If 0∊⍴files
              MyLogger.Log'No files found to process'
          :Else
              tbl←⊃⍪/(CountLetters ProcessFiles)files
              lines←{⍺,',',⍕⍵}/{⍵[⍒⍵[;2];]}⊃{⍺(+/⍵)}⌸/↓[1]tbl
              :Trap Config.Trap/FileRelatedErrorCodes
                  A.WriteUtf8File target lines
                  success←1
              :Case
                  MyLogger.LogError'Writing to <',target,'> failed, rc=',(⍕⎕EN),'; ',⊃⎕DMX
                  rc←EXIT.UNABLE_TO_WRITE_TARGET
                  success←0
              :EndTrap
              :If success
                  MyLogger.Log(⍕⍴files),' file',((1<⍴files)/'s'),' processed:'
                  MyLogger.Log' ',↑files
              :EndIf
          :EndIf
      :EndIf
    ∇

Note that the exit code is tested against EXIT.OK. Testing 0=exit would work and read as well, but relies on EXIT.OK being 0. The point of defining the codes in EXIT is to make the functions relate to the exit codes only by their names.

Logging file-related errors

Logging errors related to files in a real-world application requires more attention to detail: ⎕DMX provides more information that can be very useful:

4. Unforeseen errors

Our code so far covers the errors we foresee: errors in the parameters, and errors encountered in the file system. There remain the unforeseen errors, chief among them, errors in our own code.

If the code we have so far breaks, the EXE will try to report the problem to the session, find no session, and abort with an exit code of 4 to tell Windows “Sorry, it didn’t work out.”

If the error is replicable, we can easily track it down using the development interpreter.

But the error might not be replicable. It could, for instance, have been produced by ephemeral congestion on a network interfering with file operations. Or the parameters for your app might be so complicated that it is hard to replicate the environment and data with confidence. What you really want for analysing the crash is a crash workspace, a picture of the ship when it went down.

4.1. Global trapping

For this we need a high-level – or global – trap to catch any event not trapped by any specific :Trap statements. We want it to save the workspace for analysis. We might also want it to report the incident to the developer – users don’t always do this! For this we’ll use the HandleError class from the APLTree.

Define a new EXIT code constant:

    ....
    OK←0
    APPLICATION_CRASHED←104
    INVALID_SOURCE←111
    ...
Information

104? Why not 4, the standard Windows code for a crashed application? The distinction is useful. An exit code of 104 will tell us MyApp’s trap caught and reported the crash. An exit code of 4 tells you even the trap failed!

We want to establish general error trapping as soon as possible, but we also need to know where to save crash files etc. That means we start right after having instantiated the INI file, because that’s where we get this kind of information from. For establishing error trapping we need to set ⎕TRAP.

Because we want to ensure any function down the stack can pass a certain error up to the next definition of ⎕TRAP (see the ⎕TRAP help, options C and N) it is vitally important not only to set but also to localise ⎕TRAP in StartFromCmdLine

leanpub-start-insert
∇ {r}←StartFromCmdLine arg;MyLogger;Config;rc;⎕TRAP
⍝ Needs command line parameters, runs the application.
   r←⍬
   (Config MyLogger)←Initial ⍬
   ⎕WSID←'MyApp'
   ⎕TRAP←(Config.Debug=0) SetTrap Config
   rc←TxtToCsv arg~''''
   Off rc
∇

We need to set ⎕WSID because the global trap will attempt to save a workspace in the event of a crash.

4.2. Trap parameters

We set ⎕TRAP by assigning the result of SetTrap, so we need to create that function now:

∇ trap←{force}SetTrap Config
⍝ Returns a nested array that can be assigned to `⎕TRAP`.
  force←{0<⎕NC ⍵:⍎⍵ ⋄ 0}'force'
  #.ErrorParms←##.HandleError.CreateParms
  #.ErrorParms.errorFolder←⊃Config.Get'Folders:Errors'
  #.ErrorParms.returnCode←EXIT.APPLICATION_CRASHED
  #.ErrorParms.logFunction←MyLogger.Log
  #.ErrorParms.windowsEventSource←'MyApp'
  #.ErrorParms.addToMsg←' --- Something went terribly wrong'
  trap←force ##.HandleError.SetTrap '#.ErrorParms'
∇

Notes:

Let’s investigate how this will work; trace into #.MyApp.StartFromCmdLine ''. When you reach line 4, Config exists, so now you can call MyApp.SetTrap with different left arguments:

      SetTrap Config
 0 1000 S
      0 SetTrap Config
 0 1000 S
      1 SetTrap Config
 0 E #.HandleError.Process '#.ErrorParms'
      #.ErrorParms.∆List
 addToMsg
 checkErrorFolder                                                      1
 createHTML                                                            1
 customFns
 customFnsParent
 enforceOff                                                            0
 errorFolder                     C:\Users\kai\AppData\Local\MyApp\Errors
 logFunction                                                         Log
 logFunctionParent   [Logger:C:\Users\...\MyApp_20170305.log(¯70419218)]
 off                                                                   1
 returnCode                                                          104
 saveCrash                                                             1
 saveErrorWS                                                           1
 saveVars                                                              1
 signal                                                                0
 trapInternalErrors                                                    1
 trapSaveWSID                                                          1
 windowsEventSource

4.3. Test the global trap

We can test this by inserting a line with a full stop[1] into, say, CountLettersIn.

But that would be awkward. We don’t really want to change our source code in order to test error trapping. (Many an application crashed in production because a programmer forgot to remove a breakpoint before going live.) So we put another setting in the INI file:

[Config]
Debug       = ¯1    ; 0=enfore error trapping; 1=prevent error trapping;
Trap        = 1     ; 0 disables any :Trap statements (local traps)
ForceError  = 1     ; 1=let TxtToCsv crash (for testing global trap handling)
...

That requires two minor changes in CreateConfig:

∇ Config←CreateConfig dummy;myIni;iniFilename
...
Config.ForceError←0
      iniFilename←'expand'F.NormalizePath'MyApp.ini'
      :If F.Exists iniFilename
          myIni←⎕NEW ##.IniFiles(,⊂iniFilename)
          Config.ForceError←myIni.Get'Config:ForceError'

We change TxtToCsv so that it crashes in case Config.ForceError equals 1:

∇ rc←TxtToCsv fullfilepath;files;tbl;lines;target
⍝ Write a sibling CSV of the TXT located at fullfilepath,
⍝ containing a frequency count of the letters in the file text.
⍝ Returns one of the values defined in `EXIT`.
   MyLogger.Log'Source: ',fullfilepath
   (rc target files)←GetFiles fullfilepath
   {~⍵:r←⍬ ⋄ 'Deliberate error (INI flag "ForceError"'⎕SIGNAL 11}ForceError
...

The dfns {~⍵:r←⍬ ⋄ … uses a guard to signal an error in case is true and otherwise does nothing but return a shy result. In order to test error trapping we don’t need even to create and execute a new EXE. Instead we just set ForceError to 1 and then call #.MyApp.StartFromCmdLine from within the WS:

      #.MyApp.StartFromCmdLine 'Z:\texts\ulysses.txt'
⍎SYNTAX ERROR
TxtToCsv[6] . ⍝ Deliberate error (INI flag "ForceError")
           ∧

That’s exactly what we want! Error trapping should not interfere when we are developing.

To actually test error trapping we need to set the Debug flag in the INI file to 0. That will tell MyApp we want error trapping active, no matter what environment we are in. Change the INI file accordingly and execute it again.

      )reset
      #.MyApp.StartFromCmdLine 'Z:\texts\ulysses.txt'
HandleError.Process caught SYNTAX ERROR

Note that HandleError has not executed ⎕OFF because we executed this in a development environment.

That’s all we see in the session, but when you check the folder #.ErrorParms.errorFolder you will find that indeed there were three new files created in that folder for this crash.

Note that had you traced through the code there would be just two files: the workspace would be missing.

The reason is: with the Tracer active the current workspace cannot be saved. Generally there are two reasons for no workspace being saved:

This is not strictly true. When HandleError detects multiple threads it tries to kill all of them. By definition that won’t work because (a) it cannot kill the main thread (0) and (b) it cannot kill its own thread.

However, if it happens to run in the main thread at that very moment it will get rid of all other running threads and be able to save a crash workspace afterwards as a result.

Because we’ve defined a source for the Windows Event Log, HandleError has reported the error accordingly:

Windows Event Log

See the discussion of the Windows Event Log in a later chapter.

We also find evidence in the log file that something broke; see LogDog:

The log file

This is done for us automatically by the HandleError class because we provided the name of a logging function, and a ref pointing to the instance where that log function lives.

We also have an HTM with a crash report, an eponymous DWS containing the workspace saved at the time it broke, and an eponymous DCF whose single component is a namespace of all the variables visible at the moment of the crash. Some of this has got to help.

Note that the crash file names are simply the WSID and the timestamp prefixed by an underscore:

      ⍪{⊃,/1↓⎕NPARTS⍵}¨⊃#.FilesAndDirs.Dir #.ErrorParms.errorFolder,'\'
 MyApp_20170307111141.dcf
 MyApp_20170307111141.dws
 MyApp_20170307111141.html

Save your work and re-export the EXE.

4.4. The crash files

What's in those crash files?

The HTM contains a report of the crash and some key system variables:

MyApp_20170307111141

Version:   Windows-64 16.0 W Development
⎕WSID:       MyApp
⎕IO:       1
⎕ML:       1
⎕WA:       62722168
⎕TNUMS:       0
Category:
EM:           SYNTAX ERROR
HelpURL:
EN:           2
ENX:       0
InternalLocation:    parse.c 1739
Message:
OSError:   0 0
Current Dir:    ...code\v07
Command line:    "...\Dyalog\...\dyalog.exe" DYAPP="...code\v07\MyApp.dyapp"
Stack:

#.HandleError.Process[22]
#.MyApp.TxtToCsv[6]
#.MyApp.StartFromCmdLine[6]
Error Message:

⍎SYNTAX ERROR
TxtToCsv[6] . ⍝ Deliberate error (INI flag "ForceError")
           ∧

More information is saved in a single component – a namespace – on the DCF.

      (#.ErrorParms.errorFolder,'/MyApp_20160513112024.dcf') ⎕FTIE 1
      ⎕FSIZE 1
1 2 7300 1.844674407E19
      q←⎕FREAD 1 1
      q.⎕NL ⍳10
AN
Category
CurrentDir
DM
EM
EN
ENX
HelpURL
InternalLocation
LC
Message
OSError
TID
TNUMS
Trap
Vars
WA
WSID
XSI
      q.Vars.⎕NL 2
ACCENTS
args
exit
files
fullfilepath
i
isDev
tbl
tgt

The DWS is the crash workspace. Load it. The Latent Expression has been disabled to ensure MyApp does not attempt to start up again.

      ⎕LX
⎕TRAP←0 'S' ⍝#.MyApp.StartFromCmdLine

The state indicator shows the workspace captured at the moment the HandleError object saved the workspace. Your real problem – the full stop in MyApp.TxtToCsv – is some levels down in the stack.

      )SI
#.HandleError.SaveErrorWorkspace[7]*
#.HandleError.Process[28]
#.MyApp.TxtToCsv[6]*
#.MyApp.StartFromCmdLine[6]

You can clear HandleError off the stack with a naked branch arrow; note the * on the first and third line. When you do so, you'll find the original global trap restored. Disable it. Otherwise any error you produce while running code will trigger HandleError again!

      →
      )SI
#.MyApp.TxtToCsv[6]*
#.MyApp.StartFromCmdLine[6]
      ⎕TRAP
  0 E #.HandleError.Process '#.ErrorParms'
      ⎕TRAP←0/⎕TRAP

We also want to check whether the correct return code is returned. For that we have to call the EXE, but we don’t do this in a console window for the reasons we discussed earlier. Instead we use the Execute class which provides two main methods:

      ⎕←2⊃#.Execute.Application 'Myapp.exe '"Z:\texts\ulysses.txt"'
104

In development you’ll discover and fix most errors while working from the APL session. Unforeseen errors encountered by the EXE will be much rarer. Now you’re all set to investigate them!

4.5. About #.ErrorParms

We’ve established #.ErrorParms as a namespace, and we have explained why: HandleError.Process needs to see ErrorParms, no matter the circumstances, otherwise it cannot work. Since we construct the workspace from scratch when we start developing it cannot do any harm because we quit as soon as the work is done.

Or can it? Let’s check. First, change the INI file so that it reads:

...
Trap        = 1    ; 0 disables any :Trap statements (local traps)
ForceError  = 0    ; 1=let TxtToCsv crash (for testing global trap handling)
...

Now double-click the DYAPP, call #.MyApp.StartFromCmdLine '' and execute:

      ⎕nnames
C:\Users\kai\AppData\Local\MyApp\Log\MyApp_20170309.log

The log file is still open! Now that’s what we expect to see as long as MyLogger lives, but that is kept local in #.MyApp.StartFromCmdLine, so why is this? The culprit is ErrorParms! In order to allow HandleError to write to our log file we’ve provided not only the name of the log file but also a reference pointing to the instance the log function is living in:

      #.ErrorParms.logFunctionParent
[Logger:C:\Users\kai\AppData\Local\MyApp\Log/MyApp_20170309.log(¯76546889)]

In short: we have a good reason to get rid of ErrorParms once the program has finished – but how? ⎕SHADOW to the rescue! With ⎕SHADOW we can declare a variable to be local from within a function. Mainly useful for localising names that have been constructed by other expressions, we can use it to make ErrorParms local within StartFromCmdLine. For that we add a single line:

∇ {r}←StartFromCmdLine arg;MyLogger;Config;rc;⎕TRAP
⍝ Needs command line parameters, runs the application.
  r←⍬
  #.⎕SHADOW'ErrorParms'
  ⎕WSID←'MyApp'
....

Note that we put #. in front of ⎕SHADOW; that is effectively the same as having a header StartFromCmdLine;#.ErrorParms – but that is syntactically impossible to do. With #.⎕SHADOW it works. When you now try a double-click on the DYAPP and call #.MyApp.StartFromCmdLine you will find that no file is tied anymore, and that #.ErrorParms is not hanging around either.

4.6. Very early errors

There is a possibility of MyApp crashing without the global trap catching it. This is because we establish the global trap only after instantiating the INI file. Only then do we know where to write the crash files, how to log the error, etc.

But an error might occur before that!

Naturally there is no perfect solution available here but we can at least try to catch such errors.

For this we establish a ⎕TRAP with default settings very early, and we make sure that ⎕WSID is set even earlier, otherwise any attempt to save the crash WS will fail.

∇ {r}←StartFromCmdLine arg;MyLogger;Config;rc;⎕TRAP
⍝ Needs command line parameters, runs the application.
  r←⍬
  ⎕WSID←'MyApp'
  ⎕SIGNAL 0
  ⎕TRAP←1 #.HandleError.SetTrap ⍬
  .
  #.⎕SHADOW'ErrorParms'
  ....

Note that we use the SetTrap function in HandleError. It accepts a parameter space as right argument, but it also accepts an empty vector, when it falls back to the defaults.

Resetting the diagnostic message and the event number

We take the opportunity to initialize both ⎕DM and ⎕EN: with ⎕SIGNAL 0 we ensure

      .
.
SYNTAX ERROR
     .
    ∧
      ⎕DM
SYNTAX ERROR
     ⎕EN
2
     ⎕SIGNAL 0
     ⎕DM

     ⎕EN
0

For testing purposes we have provided a 1 as the left argument, which enforces error trapping even in a development environment. In the following line we break the program with a full stop.

When you now call #.MyApp.StartFromCmdLine '' the error is caught. Of course no logging will take place but it will still try to save the crash files. Since no better place is known it will try to create a folder MyApp\Errors in %LOCALAPPDATA%.

You can try this now but make sure that when you are ready you remove the line with the full stop from MyApp.StartFromCmdLine and also remove the 1 provided as the left argument to HandleError.SetTrap.

5. HandleError in detail

HandleError can be configured in many ways by changing the defaults provided by the CreateParms method. There is a table with documentation available; execute ]ADoc #.HandleError and scroll to CreateParms. Most of the parameters are self-explaining but some need background information.

    #.HandleError.CreateParms.∆List
addToMsg
checkErrorFolder          1
createHTML                1
customFns
customFnsParent
enforceOff                0
errorFolder         Errors/
logFunction
logFunctionParent
off                       1
returnCode                1
saveCrash                 1
saveErrorWS               1
saveVars                  1
signal                    0
trapInternalErrors        1
trapSaveWSID              1
windowsEventSource
signal

By default, HandleError executes ⎕OFF in a runtime environment. That’s not always the best way to deal with an error.

In a complex application it might be the case that just one command fails, but the rest of the application is doing fine. In that case we would be better off by setting off to 0 and signalling a numeric code that can be caught by yet another ⎕TRAP that simply allows the user to explore other commands in the application.

trapInternalErrors
This flag allows you to switch off any error trapping within HandleError. This can be useful in case something goes wrong. It can be useful when working on or debugging HandleError itself.
saveCrash, saveErrorWS and saveVars
While saveCrash and saveVars are probably always 1, setting saveErrorWS to 0 is perfectly reasonable if you know any attempt to save the error WS will fail, for example, because your application is multi-threaded. (Another good reason to not save a workspace is to keep your code from spying eyes.)
customFns and customFnsParent
This allows you to have HandleError call a function of your choice. For example, you can use this to send an email or a text to a certain address.
windowsEventSource
This defaults to an empty vector, meaning that HandleError does not attempt to write to the Windows Event Log. Writing to the Windows Event Log is discussed in its own chapter.

Footnotes

  1. The English poets among us love that the tersest way to bring a function to a full stop is to type one. (American poets will of course have typed a period and will think of it as calling timeout.)

Chapter 8:

Testing – the sound of breaking glass

Our application here is simple – just count letter frequency in text files.

All the other code has been written to configure the application, package it for shipment, and to control how it behaves when it encounters problems.

Developing code refines and extends it. We have more developing to do. Some of that developing might break what we already have working. Too bad. No one’s perfect.

But we would at least like to know when we’ve broken something – to hear the sound of breaking glass behind us. Then we can fix the error before going any further.

In our ideal world we would have a team of testers continually testing and retesting our latest build to see if it still does what it’s supposed to do. The testers would tell us if we broke anything. In the real world we have programs – tests – to do that.

What should we write tests for? “Anything you think might break,” says Kent Beck[1], author of Extreme Programming Explained. We’ve already written code to allow for ways in which the file system might misbehave. We should write tests to discover if that code works. We’ll eventually discover conditions we haven’t foreseen and write fixes for them. Then those conditions too will join the things we think might break, and get added to the test suite.

1. Why you want to write tests

1.1. Notice when you break things

Some functions are more vulnerable than others to being broken under maintenance. Many functions are written to encapsulate complexity, bringing a common order to a range of different arguments.

For example, you might write a function that takes as argument any of a string[2], a vector of strings, a character matrix or a matrix of strings.

If you later come to define another case, say, a string with embedded line breaks, it’s easy enough inadvertently to change the function’s behaviour with the original cases.

If you have tests that check the function’s results with the original cases, it’s easy to ensure your changes don't change the results unintentionally.

1.2. More reliable than documentation

No, tests don’t replace documentation. They don’t convey your intent in writing a class or function. They don’t record your ideas for how it should and should not be used, references you consulted before writing it, or thoughts about how it might later be improved.

But they do document with crystal clarity what it is known to do. In a naughty world in which documentation is rarely complete and even less often revised when the code is altered, it has been said the only thing we know with certainty about any given piece of software is what tests it passes.

1.3. Understand more

Test-Driven Design (TDD) is a high-discipline practice associated with Extreme Programming. TDD tells you to write the tests before you write the code. Like all such rules, we recommend following TDD – thoughtfully.

The reward from writing an automated test is not always worth the effort. But it is a very good practice and we recommend it given that the circumstances are right.

For example, if you know from the start exactly what your program is supposed to do then TDD is certainly an option. If you start prototyping in order to find out what the user actually wants the program to do, TDD is no option at all.

If you are writing the first version of a function, writing the tests first will clarify your understanding of what the code should be doing. It will also encourage you to consider boundary cases or edge conditions: for example, how should the function above handle an empty string? A character scalar?

TDD first tests your understanding of your task. If you can't define tests for your new function, perhaps you’re not ready to write the function either.

If you are modifying an existing function, write new tests for the new things it is to do. Run the revised tests and see that the code fails the new tests. If the unchanged code passes any of the new tests… review your understanding of what you’re trying to do!

2. Readability

Reading and understanding APL code is more difficult than in other programming languages due to the higher abstraction level and the power of APL’s primitives. However, as long as you have at least one example with correct arguments, it’s always possible to decipher the code.

Things can become very nasty indeed if an application crashes because inappropriate data arrives at your function. However, before you can figure out whether the data is appropriate or not you need to understand the code – a chicken-egg problem.

That’s when test cases can be very useful as well because they demonstrate which data a function is expected to process. It also emphasises why it is important to have test cases for all the different types of data (or parameters) a function is supposed to process. In this respect test cases should be exhaustive.

2.1. Write better

Writing functions with a view to passing formal tests will encourage you to write in functional style. In pure functional style, a function reads only the information in its arguments and writes only its result. No side effects or references.

  ∇ Z←mean R;r
   [1] Z←((+/r)÷≢r←,R)
  ∇

In contrast, this line from TxtToCsv reads a value from a namespace external to the function (EXIT.APPLICATION_CRASHED) and sets another: #.ErrorParms.returnCode.

    #.ErrorParms.returnCode←EXIT.APPLICATION_CRASHED

In principle, TxtToCsv could be written in purely functional style. References to classes and namespaces #.HandleError, #.APLTreeUtils, #.FilesAndDirs, EXIT, and #.ErrorParms could all be passed to it as arguments.

If those references ever varied – for example, if there were an alternative namespace ReturnCodes sometimes used instead of EXIT – that might be a useful way to write TxtToCsv.

But as things are, cluttering up the function’s signature – its name and arguments – with these references harms rather than helps readability. It is an example of the cure being worse than the disease.

You shouldn’t write everything in pure functional style but the closer you stick to it, the better your code will be, and the easier to test. Functional style goes hand in hand with good abstractions, and ease of testing.

3. Why you don’t want to write tests

There is nothing magical about tests. Tests are just more code. The test code needs maintaining like everything else. If you refactor a portion of your application code, the associated tests need reviewing – and possibly revising – as well.

In programming, the number of bugs is generally a linear function of code volume. Test code is no exception to this rule. Your tests are both an aid to development and a burden on it.

You want tests for everything you think might break, but no more tests than you need.

Beck’s answer – test anything you think might break – provides useful insight. Some expressions are simple enough not to need testing. If you need the indexes of a vector of flags, you can see that {⍵/⍳≢⍵} [5] will find them. It’s as plain as 2+2 making four. You don’t need to test that.

APL’s scalar extension and operators such as outer product allow you to replace nested loops (a common source of error) with expressions which don’t need tests. The higher level of abstraction enabled by working with collections allows not only fewer code lines but also fewer tests.

Time for a new version of MyApp. Make a copy of Z:\code\v07 as Z:\code\v08.

4. Setting up the test environment

We’ll need the Tester class from the APLTree library. And a namespace of tests, which we’ll dub #.Tests.

Create Z:\code\v08\Tests.dyalog:

    :Namespace Tests

    :EndNamespace

Save this as Z:\code\v08\Tests.dyalog and include both scripts in the DYAPP:

    Target #
    Load ..\AplTree\APLTreeUtils
    Load ..\AplTree\FilesAndDir
    Load ..\AplTree\HandleError
    Load ..\AplTree\IniFiles
    Load ..\AplTree\Logger
    Load ..\AplTree\Tester
    Load Constants
    Load Utilities
    Load Tests
    Load MyApp
    Run MyApp.Start 'Session'

Run the DYAPP to build the workspace. In the session you might want to execute ]ADoc #.Tester to see the documentation for the Tester class.

5. Unit and functional tests

Information

Unit tests tell a developer that the code is doing things right; functional tests tell a developer that the code is doing the right things.

It’s a question of perspective. Unit tests are written from the programmer’s point of view. Does the function or method return the correct result for given arguments?

Functional tests, on the other hand, are written from the user’s point of view. Does the software do what its user needs it to do?

Both kinds of tests are important. If you are a professional programmer you need a user representative to write functional tests. If you are a domain-expert programmer [3] you can write both.

In this chapter we'll tackle unit tests.

6. Speed

Unit tests should execute fast: developers often want to execute them even when still working on a project in order to make sure that they have not broken anything, or to find out what they broke. If executing the test suite takes too long it defeats the purpose.

Sometimes it cannot be avoided that tests take quite a while, for example when testing GUIs. In that case it might be an idea to create a group of tests that comprehend not all, but just the most important ones.

Those can then be executed while actually working on the code base while the full-blown test suite is only executed every now and then, maybe only before checking in the code.

7. Preparing: helpers

The first thing we are going to do is to establish a number of helpers in Tests that the Tester class provides. We can simply call Tester.EstablishHelpersIn and provide a ref to the namespace hosting our test cases as right argument:

    )cs #
#
    Tester.EstablishHelpersIn #.Tests
    #.Tests.⎕nl 3
FailsIf
G
GoToTidyUp
L
ListHelpers
PassesIf
Run
RunBatchTests
RunBatchTestsInDebugMode
RunDebug
RunThese
∆Failed
∆Inactive
∆LinuxOnly
∆LinuxOrMacOnly
∆LinuxOrWindowsOnly
∆MacOnly
∆MacOrWindowsOnly
∆NoAcreTests
∆NoBatchTest
∆OK
∆WindowsOnly

The helpers can be categorized:

Some of the helpers (G, L and ListHelpers) are just helpful while others, like all the Run* functions and the flow control functions, are essential. We need them to be around before we can execute any test case.

The fact that we had to establish them with a function call upfront contradicts this. But there is an escape route: we add a line to the DYAPP:

...
Run #.MyApp.SetLX ⍬
Run #.Tester.EstablishHelpersIn #.Tests

Of course we don’t need this when the DYAPP is supposed to assemble the workspace for a productive environment; we will address this problem later.

We will discuss all helpers in detail, and we start with the flow control helpers.

7.1. Flow control helpers

Let’s look at an example: FailsIf takes a boolean right argument and returns either 0 in case the right argument is 1 or an empty vector in case the right argument is 0:

      FailsIf 1
0
      FailsIf 0

      ⍴FailsIf 0
0

That means that the statement →FailsIf 1 will jump to 0, exiting the function carrying the statement.

Since GoTo statements are rarely used these days because under most circumstances control structures are far better, it is probably worthwhile to mention that →⍬ – as well as →'' – makes the interpreter carry on with the next line.

In other words the function just carries on. That’s exactly what we want when the right argument of FailsIf is a 0 because in that case the test has not failed.

PassesIf is exactly the same thing but just with a negated argument: it returns a 0 when the right argument is 0 and an empty vector in case the right argument is 1.

GoToTidyUp is a special case. It returns an empty vector when the right argument is 0. If the right argument is 1 by convention the function that calls it has a line labelled ∆TidyUp; the line number of that label is then returned.

This is useful in case a test function needs to do some cleaning up, no matter whether it has failed or not. Imagine you need a temporary file for a test but want to delete it after carrying out the test case. In that case the bottom of your test function might look like this:

...
∆TidyUp:
    #.FilesAndDirs.DeleteFile tempFilename

When everything goes according to plan the function would eventually execute these lines anyway, but when a test case fails you need this:

    →GoToTidyUp expected≢result

Like FailsIf the test function would just carry on in case expected≢result returns a 0 but jump to the label ∆TidyUp in case the test fails (=the condition is true).

But why are we using functions for all this anyway? We could do without, couldn’t we? Yes, so far we could, but there is just one more thing.

8. Writing unit tests

We have automated the way the helpers are established in Tests. Now we are ready to implement the first test case.

Utilities are a good place to start writing tests. Many utility functions are simply names assigned to common expressions. Other utilities encapsulate complexity, making similar transformations of different arguments.

We’ll start with map in #.Utilities. We know by now that in general it works although even that needs confirmation by a test of course. What we don’t know yet is whether it works under all circumstances. We also need to ensure it complains when given inappropriate arguments.

To make writing test cases as easy as possible you can ask Tester to provide a test case template.

    ⎕←⍪#.Tester.GetTestFnsTemplate
  R←Test_000(stopFlag batchFlag);⎕TRAP
 ⍝ Model for a test function.
  ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
  R←∆Failed

 ⍝ Preconditions...
 ⍝ ...

  →PassesIf 1≡1
  →FailsIf 1≢1
  →GoToTidyUp 1≢1
  R←∆OK

 ∆TidyUp: ⍝ Clean up after this label
           ⍝ ...

The template covers all possibilities, and we will discuss all of them. However, for the time being we want to keep it simple, so we will delete quite a lot and also add three more functions:

:Namespace Tests
⎕IO←1 ⋄ ⎕ML←1
∇Initial
  U←##.Utilities ⋄ F←##.FilesAndDirs ⋄ A←##.APLTreeUtils
∇
∇ R←Test_001(stopFlag batchFlag);⎕TRAP
 ⍝ Is the length of the left argument of the `map` function checked?
  ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
  R←∆Failed
  :Trap 5
      {}(⊂⎕A)U.map'APL is great'
      →FailsIf 1
  :Else
      .
  :EndTrap
  R←∆OK
∇
∇ {r}←GetHelpers
  r←##.Tester.EstablishHelpersIn ⎕THIS
∇
∇ Cleanup dummy
  ⎕EX¨'AFU'
∇
:EndNamespace

What we changed:

You might have noticed we address, say, Utilities with ##.Utilities rather than #.Utilities. Making this a habit is a good idea: currently it does not make a difference, but when you later decide to move everything in # into, say, a namespace #.Container (you never know!) then ##. would still work while #. wouldn't.

The :Else part is not ready yet; the full stop will prevent the test function from carrying on when we get there.

Notes:

Ordinary namespaces versus scripted ones

There’s a difference between an ordinary namespace and a scripted namespace: imagine you've called #.Tester.EstablishHelpersIn within an ordinary namespace.

Now you change/add/delete test functions; that would have no effect on anything else in that namespace. In other words, the helpers would continue to exist.

When you change a namespace script, on the other hand, the namespace is re-created from the script, and that means that our helpers will disappear because they are not a part of the Tests script.

Let’s call our test case. We do this by running the Run method first:

Run
--- Test framework "Tester" version 3.5.0 from 2017-07-16 ---------------------------------
Searching for INI file Testcases.ini
  ...not found
Searching for INI file testcases_APLTEAM2.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:ss on #.Tests ---------------------------------------
# Test_001 (1 of 1) : Is the length of the left argument of the `map` function checked?
 ------------------------------------------------------------------------------------------
   1 test case executed
   0 test cases failed
   1 test case broken
Time of execution recorded on variable #.Tests.TestCasesExecutedAt in: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and executed.
*** Tests done

That’s what we expect.

Information

Note that there are INI files mentioned. Ignore this for the time being; we will discuss this later on.

What is a test case?!

You might wonder how Run established what is a test case and what isn’t: that’s achieved by naming conventions. Al test functions start their names with Test_. After that there are two possibilities:

  1. In the simple case the _ is followed by nothing but digits. All these qualify as test cases: Test_1, Test_01, Test_001 and so on. (Test_01A however does not.)
  2. If you have a large number of test cases you most probably want to group them. You can insert a group name between two underscores, followed by one or more digits. So Test_map_1 is recognized as a test case, and so is Test_Foo_9999. Test_Foo_Goo_1 however is not.

What if we want to look into a broken or failing test case? Of course in our current scenario – which is extremely simple – we could just trace into Test_001 and find out what’s going on, but if we take advantage of the many features the test framework offers, we cannot do this. (Soon to become clear why.)

However, there is a way to do this no matter whether the scenario is simple, reasonably complex or extremely complex: we call RunDebug:

RunDebug 0
--- Test framework "Tester" version 3.6.0 from  -------
Searching for INI file testcases_{computername}.ini
  ...not found
Searching for INI file Testcases.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:ss on #.Tests -------------
SYNTAX ERROR
      . ⍝ Deliberate error
     ∧
      )si
#.Tests.Test_001[6]*
⍎
#.Tester.ExecuteTestFunction[6]
#.Tester.ProcessTestCases[6]
#.Tester.Run__[39]
#.Tester.RunDebug[17]
#.Tests.RunDebug[3]

It stopped in line 6. Obviously the call to FailsIf has something to do with this, and so has the ⎕TRAP setting because apparently that’s where the “Deliberate error” comes from.

This is indeed the case. All three flow-control functions, FailIf, PassesIf and GoToTidyUp check whether they are running in debug mode; if so, rather than return a result that indicates a failing test case, they ⎕SIGNAL 999, which is then caught by the ⎕TRAP, which in turn first prints ⍝ Deliberate error to the session and then hands over control to the user.

You can now investigate variables or start the Tracer, etc. to investigate the problem.

The difference between Run and RunDebug is the setting of the first of the two flags provided as right argument to the test function: stopFlag. This is 0 when Run executes the test cases, but it is 1 when RunDebug is in charge. The three flow-control functions FailsIf, PassesIf and GoToTidyUp all honour stopFlag – that’s how it works.

Now sometimes you don’t want the test function to go to the point where the error actually appears, for example if the test function does a lot of precautioning, and you want to check this upfront because there might be something wrong with it, causing the failure.

Note that so far we passed a 0 as right argument to RunDebug. If we pass a 1 instead, then the test framework would stop just before executing the test case:

      RunDebug 1
--- Test framework "Tester" version 3.6.0 from YYYY-MM-DD -------
Searching for INI file Testcases.ini
  ...not found
Searching for INI file testcases_APLTEAM2.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:ss on #.Tests -------------

ExecuteTestFunction[6]
      )si
#.Tester.ExecuteTestFunction[6]*
#.Tester.ProcessTestCases[6]
#.Tester.Run__[39]
#.Tester.RunDebug[17]
#.Tests.RunDebug[3]

You could now trace into Test_001 and investigate. Instead, enter →0. You should see something like this:

* Test_001 (1 of 1) : Is the length of the left argument of the `map` function checked?
 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   1 test case executed
   1 test case failed
   0 test cases broken
Time of execution recorded on variable #.Tests.TestCasesExecutedAt in: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and executed.
*** Tests done

Let’s have map check its left argument:

:Namespace Utilities
      map←{
          (,2)≢⍴⍺:'Left argument is not a two-element vector'⎕SIGNAL 5
          (old new)←⍺
          nw←∪⍵
          (new,nw)[(old,nw)⍳⍵]
      }
:EndNamespace

Now run RunDebug 1. Trace into Test_001 and watch whether now any error 5 (LENGTH ERROR) is trapped, You should end up on line 8 of Test_001. Exchange the full stop by:

    →PassesIf'Left argument is not a two-element vector'≡⎕DMX.EM

⎕DM versus ⎕DMX

You have always used ⎕DM, and it was fine, right? No need to switch to the (relatively) new ⎕DMX, right? Well, the problem with ⎕DM is that it is not thread-safe, while ⎕DMX is. That’s why we suggest you stop using ⎕DM and use just ⎕DMX. It also provides more, and more precise, information.

This checks whether the error message is what we expect. Trace through the test function and watch what it is doing. After having left the test function you may click the green triangle in the Tracer. (Continues execution of all threads.)

Now what if you’ve executed, say, not one but 300 test cases with Run, and just one failed, say number 289? You expected them all to succeed; now you need to check on the failing one.

Calling Run as well as RunDebug would always execute all test cases found. The function RunThese allows you to run just the specified test functions:

      RunThese 1
--- Test framework "Tester" version 3.5.0 from 2017-07-16 --------------------------------
Searching for INI file Testcases.ini
  ...not found
Searching for INI file testcases_APLTEAM2.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:ss on #.Tests --------------------------------------
  Test_001 (1 of 1) : Process a single file with .\MyApp.exe
 -----------------------------------------------------------------------------------------
   1 test case executed
   0 test cases failed
   0 test cases broken
Time of execution recorded on variable #.Tests.TestCasesExecutedAt in: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and executed.
*** Tests done

This would run just test case number 1. If you specify it as ¯1 it would stop just before actually executing the test case. Same as before since we have just one test function yet but take our word for it, it would execute just Test_001 no matter how many other test cases there are.

We have discussed the functions Run, RunDebug and RunThese. That leaves RunBatchTests and RunBatchTestsInDebugMode; what are they for?

Imagine a test that would either require an enormous amount of effort to implement – or alternatively you just build something up and ask the human in front of the monitor: Does this look alright?.

That’s certainly not a batch test case because it needs a human sitting in front of the monitor. If you know upfront that there won’t be a human paying attention then you can prevent non-batch test cases from being executed by calling either RunBatchTests or RunBatchTestsInDebugMode.

How does this work? We already learned that stopFlag, the first of the two flags passed to any test case as the right argument, governs whether any errors are trapped or not.

The second flag is called batchFlag, and that gives you an idea of what it’s good for. If you have a test that interacts with a user (i.e. cannot run without a human) then your test case would typically look like this:

 R←Test_001(stopFlag batchFlag);⎕TRAP
⍝ Check ...
 ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
 R←∆Failed
 :If batchFlag
     ⍝ perform the test
     R←∆OK
 :Else
     R←∆NoBatchTest
 :EndIf

The test function checks the batchFlag and sees from the explicit result that it did not execute because it is not suitable for batch testing.

One can argue whether the test case we have implemented makes much sense, but it allowed us to investigate the basic features of the test framework. We are now ready to investigate the more sophisticated features.

Of course we also need a test case that checks whether map does what it’s supposed to do when appropriate arrays are passed as arguments, therefore we add this to Tests:

Namespace Tests

∇ R←Test_001(stopFlag batchFlag);⎕TRAP
...
∇

∇ R←Test_002(stopFlag batchFlag);⎕TRAP;Config;MyLogger
  ⍝ Check whether `map` works fine with appropriate data
  ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
  R←∆Failed
  (Config MyLogger)←##.MyApp.Initial ⍬
  →FailsIf'APL IS GREAT'≢Config.Accents U.map A.Uppercase'APL is great'
  →FailsIf'UßU'≢Config.Accents U.map A.Uppercase'üßÜ'
  R←∆OK
∇
...
Information

Note how using the references U and A here simplifies the code greatly.

Now we try to execute these test cases:

      #.Tests.GetHelpers
      RunThese 2
--- Test framework "Tester" version 3.6.0 from YYYY-MM-DD ----------------
Searching for INI file testcases_{computername}.ini
  ...not found
Searching for INI file Testcases.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:ss on #.Tests ----------------------
  Test_002 (1 of 1) : Check whether `map` works fine with appropriate data
 -------------------------------------------------------------------------
   1 test case executed
   0 test cases failed
   0 test cases broken

Works fine. Excellent.

Now let’s make sure the workhorse is doing okay; for this we add another test case:

:Namespace Tests
...
    ∇ R←Test_002(stopFlag batchFlag);⎕TRAP
...
    ∇ R←Test_003(stopFlag batchFlag);⎕TRAP;Config;MyLogger
    ⍝ Test whether `TxtToCsv` handles a non-existing file correctly
      ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
      R←∆Failed
      rc←##.MyApp.TxtToCsv 'This_file_does_not_exist'
      →FailsIf ##.MyApp.EXIT.SOURCE_NOT_FOUND≢rc
      R←∆OK
    ∇
...

Let’s call this test:

      )CS #.Tests
#.Tests
      GetHelpers
      RunThese 3
...
VALUE ERROR
TxtToCsv[4] MyLogger.Log'Source: ',fullfilepath
            ∧

Oops. MyLogger is undefined. In the envisaged use in production, it is defined by, and local to, StartFromCmdLine. That design followed Occam’s Razor[4]: (entities are not to be needlessly multiplied) in keeping the log object in existence only while needed. But it now prevents us from testing TxtToCsv independently. So we’ll refactor:

:Namespace Tests
...
    ∇ R←Test_003(stopFlag batchFlag);⎕TRAP
    ⍝ Test whether `TxtToCsv` handles a non-existing file correctly
      ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
      R←∆Failed
      ##.MyApp.(Config MyLogger)←##.MyApp.Initial ⍬
      rc←##.MyApp.TxtToCsv 'This_file_does_not_exist'
      →FailsIf ##.MyApp.EXIT.SOURCE_NOT_FOUND≢rc
      R←∆OK
    ∇
...

Note that now both Config and MyLogger exist within MyApp, not in Tests. Therefore we don't even have to keep them local within Test_003. They are however not part of the script, so will disappear as soon as the script Tests is fixed again, very much like the helpers.

Let’s try again:

      RunThese 3
--- Test framework "Tester" version 3.6.0 from YYYY-MM-DD -------------------------
Searching for INI file testcases_{computername}.ini
  ...not found
Searching for INI file Testcases.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at 2017-03-22 15:56:41 on #.Tests -------------------------------
  Test_003 (1 of 1) : Test whether `TxtToCsv` handles a non-existing file correctly
 ----------------------------------------------------------------------------------
   1 test case executed
   0 test cases failed
   0 test cases broken
Time of execution recorded on variable #.Tests.TestCasesExecutedAt: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and executed.

Clearly we need to have one test case for every result the function TxtToCsv might return but we leave that as an exercise to you. We have more important test cases to write: we want to ensure whenever we create a new version of the EXE that it will keep working.

Let’s rename the test functions we have so far:

The new test cases we are about to add will be named Test_exe_01, etc. For our application we could manage without grouping, but once you have more than, say, 20 test cases, grouping is a must. So we demonstrate now how this can be done.

8.1. The “Initial” function

We've already introduced a function Initial for establishing the references A, U and F before we execute any test cases. For testing the EXE we need a folder where we can store files temporarily. We add this to Initial:

:Namespace Tests
⎕IO←1 ⋄ ⎕ML←1
 ∇ R←Initial;list;rc
   U←##.Utilities ⋄ F←##.FilesAndDirs ⋄ A←##.APLTreeUtils
   ∆Path←F.GetTempPath,'\MyApp_Tests'
   F.RmDir ∆Path
   'Create!'F.CheckPath ∆Path
   list←⊃F.Dir'..\..\texts\en\*.txt'
   rc←list F.CopyTo ∆Path,'\'
   :If ~R←0∧.=⊃rc
       ⎕←'Could not create ',∆Path
   :EndIf
 ∇
...

Initial does not have to return a result but if it does it must be a Boolean. For “success” it should return a 1 and otherwise a 0. If it does return 0 then no test cases are executed, but if there is a function Cleanup it will be executed. Therefore Cleanup should be ready to clean up in case Initial was only partly or not at all successful.

We have changed Initial so that it now returns a result because copying the files over might fail for all sorts of reasons – and we cannot do without them.

Initial may or may not accept a right argument. If it does it will be passed a namespace that holds all the parameters.

What to do in Initial, apart from creating the references:

Machine-dependent initialisation

What if you need to initialise something (say a database connection) but it depends on the machine the tests are executed on – its IP address, user-id, password…?

The test framework looks for two different INI files in the current directory: First it looks for testcase.ini. It then tries to find testcase_{computername}.ini. computername here is what you get when you execute ⊣ 2 ⎕nq # 'GetEnvironment' 'Computername'.

If it finds any of them (or both) it instantiates the IniFile class as INI on these INI files within the namespace that hosts your test cases. In the case of a clash, the setting in testcase_{computername}.ini prevails.

Now we are ready to test the EXE; create it from scratch. Our first test case will process the file ulysses.txt:

:Namespace Tests
...
    ∇ R←Test_exe_01(stopFlag batchFlag);⎕TRAP;rc
      ⍝ Process a single file with .\MyApp.exe
      ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
      R←∆Failed
     ⍝ Precautions:
      F.DeleteFile⊃F.Dir ∆Path,'\*.csv' ⍝ (1)
      rc←##.Execute.Application'MyApp.exe ',∆Path,'\ulysses.txt' ⍝ (2)
      →GoToTidyUp ##.MyApp.EXIT.OK≠⊃rc ⍝ (3)
      →GoToTidyUp~F.Exists ∆Path,'\ulysses.csv' ⍝ (4)
      R←∆OK
     ∆TidyUp:
      F.DeleteFile⊃F.Dir ∆Path,'\*.csv' ⍝ (5)
    ∇
...

Notes:

  1. Ensure there are no CSVs in ∆Path.
  2. Call the EXE with ulysses.txt as a command line parameter.
  3. Check the return code and jump to ∆TidyUp if it’s not what we expect.
  4. Check whether there is now a file ulysses.cvs in ∆Path.
  5. Clean up and delete (again) all CSV files in ∆Path.

Let’s run our new test case:

      GetHelpers
      RunThese 'exe'
--- Test framework "Tester" version 3.6.0 from YYYY-MM-DD -----
Searching for INI file testcases_{computername}.ini
  ...not found
Searching for INI file Testcases.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:ss on #.Tests -----------
  Test_exe_01 (1 of 1) : Process a single file with .\MyApp.exe
 --------------------------------------------------------------
   1 test case executed
   0 test cases failed
   0 test cases broken
Time of execution recorded on variable #.Tests.TestCasesExecutedAt in: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and executed.

We need one more test case:

:Namespace Tests
...
∇ R←Test_exe_01(stopFlag batchFlag);⎕TRAP;rc
...
∇ R←Test_exe_02(stopFlag batchFlag);⎕TRAP;rc;listCsvs
  ⍝ Process all TXT files in a certain directory
  ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
  R←∆Failed
  ⍝ Precautions:
  F.DeleteFile⊃F.Dir ∆Path,'\*.csv'
  rc←##.Execute.Application'MyApp.exe ',∆Path,'\'
  →GoToTidyUp ##.MyApp.EXIT.OK≠⊃rc
  listCsvs←⊃F.Dir ∆Path,'\*.csv'
  →GoToTidyUp 1≠⍴listCsvs
  →GoToTidyUp'total.csv'≢A.Lowercase⊃,/1↓⎕NPARTS⊃listCsvs
  R←∆OK
 ∆TidyUp:
  F.DeleteFile⊃F.Dir ∆Path,'\*.csv'
∇
...

This one will process all TXTs in ∆Path and write a file total.csv. We check whether this is the case and we are done. Almost: in a real-world application we most likely would also check for a path that contains spaces in its name. We don’t do this, instead we execute the full test suite:

      GetHelpers
      ⎕←⊃Run
--- Test framework "Tester" version 3.6.0 from YYYY-MM-DD ----------------------------
Searching for INI file testcases_{computername}.ini
  ...not found
Searching for INI file Testcases.ini
  ...not found
Looking for a function "Initial"...
  "Initial" found and successfully executed
--- Tests started at YYYY-MM-DD hh:mm:dd on #.Tests ---------------------------------------
  Test_TxtToCsv_03 (1 of 5) : Test whether `TxtToCsv` handles a non-existing file correctly
  Test_exe_01 (2 of 5)      : Process a single file with .\MyApp.exe
  Test_exe_02 (3 of 5)      : Process all TXT files in a certain directory
  Test_map_01 (4 of 5)      : Is the length of the left argument of the `map` function checked?
  Test_map_02 (5 of 5)      : Check whether `map` works fine with appropriate data
 ------------------------------------------------------------------------------------------
   5 test cases executed
   0 test cases failed
   0 test cases broken
Time of execution recorded on variable #.Tests.TestCasesExecutedAt in: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and executed.
0

Note that the function Run prints its findings to the session but also returns a result. That’s a two-item vector:

  1. Is a return code. 0 means OK.
  2. Is a vector of vectors, identical to what’s printed to the session.

8.2. Cleaning up

Although we have been careful and made sure that every single test case cleans up after itself (in particular those that failed), we have not removed the directory that ∆Path points to. We add some code to the Cleanup function in order to achieve that:

:Namespace Tests
...
∇ Cleanup dummy
  ⎕EX¨'AFU'
  :If 0<⎕NC'∆Path'
      ##.FilesAndDirs.RmDir ∆Path
      ⎕EX '∆Path'
  :EndIf
∇

:EndNamespace

This function now checks whether a global ∆Path exists. If so, the directory it points to is removed and the global variable deleted. The Tester framework checks whether there is a function Cleanup.

If so, the function is executed after the last test case has been executed. The function must be either monadic or niladic; if it is a monadic function the right argument will be . It must either return a shy result (ignored) or no result at all.

8.3. Markers

We’ve already mentioned elsewhere that it is useful to mark code in particular ways, like ⍝FIXME⍝ or ⍝TODO⍝. It is an excellent idea to have a test case that checks for such markers. Before something makes it to a customer such strings should probably be removed from the code.

8.4. The “L” and “G” helpers

Now that we have three groups we can take advantage of the G and the L helpers:

      G
exe
map
TxtToCsv
      L''
 Test_exe_01       Process a single file with .\MyApp.exe
 Test_exe_02       Process all TXT files in a certain directory
 Test_map_01       Is the length of the left argument of the `map` function checked?
 Test_map_02       Check whether `map` works fine with appropriate data
 Test_TxtToCsv_01  Test whether `TxtToCsv` handles a non-existing file correctly
      L'ex'
 Test_exe_01  Process a single file with .\MyApp.exe
 Test_exe_02  Process all TXT files in a certain directory

9. TestCasesExecutedAt

Whenever the test cases were executed Tester notifies the time on a global variable TestCasesExecutedAt in the hosting namespace. This can be used in order to find out whether part of the code has been changed since the last time the cases were executed.

However, in order to do this, you have to make sure that the variable is either saved somewhere or added to the script Tests. For example, it could be handled by a cover function that calls any of Testers Run* functions and then handled that variable.

10. Conclusion

We have now a test suite available that allows us at any stage to call it in order to make sure that everything still works. Invaluable.

11. The sequence of tests

Please note that there is always the possibility of dependencies between test cases, however you try to avoid that. That might be a mistake – or due to an unnoticed side effect.

That doesn’t mean that you shouldn’t aim for making all test cases completely independent from one another. A future version of Tester might have an option to shuffle the test cases before executing them. That would help find dependencies.

12. Testing in different versions of Windows

When you wrote for yourself, your code needed to run only on the version of Windows you use yourself. To ship it as a product, you will support it on the versions your customers use.

You need to pick the versions of Windows you will support, and run your tests on all those versions. (If you are not already a fan of automated tests, you are about to become one.)

For this you will need one of:

What VM software should you use? One of us has had good results with Workstation Player from VMware.

If you use VM software you will save a machine image for each OS. Include in each machine image your preferred development tools, such as text editor and Dyalog APL. You will need to keep each machine image up to date with fixes and patches to its OS and your tools.

The machine images are large, about 10 GB each. So you want several hundred gigabytes of fast SSD (solid-state drive) on your test machine. With this you should be able to get a machine image loaded in 20 seconds or less.

13. Testing APLTree modules

By now we are using quite a number of modules from the APLTree project. Shouldn’t we test them as well? After all, if they break, our application will stop working! Well, there are pros and cons:

Pro

The modules have their own unit tests, and those are exhaustive. An update is published only after all the test cases have passed.

The modules are constantly adapted to new demands or changes in the environment, etc. Therefore a new version of Windows or Dyalog won’t break them, although you need to allow some time for this to happen. “Some time” just means that you cannot expect the APLTree modules to be ready on the day a new version of either Windows or Dyalog becomes available.

Contra
We cannot know whether those test cases cover the same environment/s (different versions of Windows, different versions of Dyalog, domain-managed network or not, network drives or not, multi-threaded versus single-threaded, you name it) our application will run in.

That suggests we should incorporate the tests the modules come with into our own test suite, although we are sure that not too many people/companies using modules from the APLTree library are actually doing this.

It’s not difficult to do: every module has a workspace saved on GitHub that comes with everything needed to run the test cases.

All it requires is

If it’s not 0:


Footnotes

  1. Kent Beck, in conversation with one of the authors.

  2. APL has no string datatype. We use the word as a casual synonym for character vector.

  3. An expert in the domain of the application rather than an expert programmer, but who has learned enough programming to write the code.

  4. Non sunt multiplicanda entia sine necessitate.

  5. With version 16.0 the same can be achieved with the new primitive .

Chapter 9:

Documentation — the Doc is in

Documentation is the bad mother of software. Programmers learn early that we depend on it but must not trust it. On the one hand we need it for the software we use; on the other hand we learn a great wariness of it for the software we develop. Understanding why this is so will help us see what to do about documenting MyApp.

It helps to distinguish three quite different things people refer to as documentation.

1. Instructions on how to use the application

Unless you are writing a tool or components for other developers to use, all software is operated through a graphical user interface. Users know the common conventions of UIs in various contexts. The standard for UIs is relatively demanding.

If you know what the application is for, it should be obvious how to use its basic features. The application might help you with wizards (dialogue sequences) to accomplish complex tasks. A user might supplement this by consulting what the Help menu offers. She might search the Web for advice. The last thing she is likely to do is to go looking for a printed manual.

We’ll come in a later chapter to how to offer online help from a Help menu. For now, we mention Help to exclude it from what we mean by documentation.

2. A description of what the application does

This is a useful thing to have, perhaps as a sales document. One or two pages suffice. Including limitations is important: files in certain formats, up to certain sizes. Perhaps a list of Frequently Asked Questions [1] and their answers.

Beyond that, you have the formal tests. This is what you know the system does. It passes its tests. Especially if you’re supporting your application on multiple versions of Windows, you’ll want those tests to be extensive.

3. A description of how the application works

This is what you want when you revisit part of the code after six months – or six days in some cases. How does this section work? What’s going on here?

In the best case the code explains everything. Software is a story told in two worlds. One world is the domain of the user, for example, a world of customer records. The other world is the arrays and namespaces used to represent them.

Good writing achieves a double vision. The transformations described by the code make sense in both worlds. Ken Iverson once coined the term expository programming for this writing. Expository programs reveal their workings to the reader.

They also discover errors more easily, making it possible to “stare the bugs out”. (David Armstrong liked to say the best writing style for a philosopher lets him see his errors before his colleagues do.)

APL requires little ‘ceremonial code’ – e.g. declarations of data type – and so makes high levels of semantic density achievable. It is perhaps easier to write expository code than in more commonly-used languages. But we have learned great respect for how quickly we can forget what a piece of code does. Then we need documentation in its third sense.

It’s in this third sense that we’ll discuss documentation.

4. The poor relation

We write software for people and people press us for results, which rarely include documentation. No one is pressing us for documentation.

Documentation is for those who come after us, including our future selves. Since 80% of the lifetime costs of software are spent on maintenance, documentation is a good investment. If the software is ours, we’re more likely to make that investment. But there will be constant pressure to defer writing it.

The common result of this pressure is that application code has either no documentation, or its documentation is not up to date. Out-of-date documentation is worse than having none. If you have no documentation you have no help with the code. You have to read it and run it to understand what it does.

But however difficult that is, it is utterly reliable. Out-of-date documentation is worse: it will mislead you and waste your time before sending you back to the code. Even if the relevant part of it is accurate, once you learn to distrust it, its value is mostly gone.

The only place worth writing documentation is in the code itself. Maintaining documentation separately adds the uncertainty of matching versions. Writing the documentation as comments in the code encourages you to keep it in step with changes to the code. We write comments in three ways, serving slightly different purposes.

Header comments
A block of comments at the top of a function serves as an abstract, describing argument/s and result and the relationship between them.
Heading comments
Heading comment lines serve exactly as headings in a book or document, helping the reader to navigate its structure.
Trailing comments

Comments at the ends of lines act as margin notes. Do not use them as a running translation of the code. Instead aim for expository style, and code that needs no translation. On lines where you’re not satisfied you’ve achieved expository style, do write an explanatory comment.

Better to reserve trailing comments for other notes, such as ⍝FIXME⍝ slow for >1E7 elements [3]. (Using a tag such as ⍝FIXME⍝ makes it easy to bookmark lines for review.) Aligning trailing comments to begin at the same column makes them easier to scan, and is considered “OCD compliant” [2].

The above conventions are simple enough and have long been in wide use.

Information

The Dyalog editor offers a special command for aligning comments: AC. You can assign a keystroke to this command: open the Configuration dialog (Options / Configure…), select the Keyboard Shortcuts tab and sort the table with a click on the Code column, then look for AC.

If you are exporting scripts for others to use – for example, contributing to a library – then it’s worth going a step further. You and other authors of a script need to read comments in the context of the code, but potential users of a script will want to know only how to call its methods.

Automatic documentation generation will extract documentation from your scripts for other users. Just as above, the documentation is maintained as comments in the code. But now header comments are presented without the code lines.

5. ADoc

ADoc is an acronym for automatic documentation generation. It works on classes and namespaces.

In its most basic function, it lists methods, properties and fields (functions, operators and variables) and requires no comments in the code.

In its more powerful function, it composes an HTML page from header comments in the code. Honouring Markdown conventions, it provides all the typographical features you need for documentation.

If Markdown is new to you, see the Markdown article on Wikipedia [4] and Markdown2Help’s own help file. Your time will be well spent: these days Markdown is used very widely.

Previously only found as a class in the APLTree library, ADoc is now shipped in Dyalog Version 16.0 as a user command.

5.1. Get a summary

Lists the methods and fields of a class. (Requires no comments.)

          ]ADoc #.HandleError -summary
    *** HandleError (Class) ***

    Shared Methods:
      CreateParms
      Process
      ReportErrorToWindowsLog
      SetTrap
      Version

For a more detailed list with arguments and results specify ]ADoc #.HandleError -summary=full.

5.2. Get it all

    ]ADoc #.HandleError

Using ADoc to browse the HandleError class

Composes a documentation page in HTML and displays it in your default browser.

5.3. Getting help

To get basic help on ADoc enter ]?adoc. For more detail, enter ]??adoc. For the full picture enter ]???adoc. The underlying ADOC class then processes itself and creates an HTML page with detailed information.

ADoc’s own documentation

6. ADoc for MyApp

How might ADoc help us? Start by seeing what ADoc has to say about MyApp as it is now:

    ]ADoc #.MyApp

Using ADoc to browse the MyApp namespace

ADoc has found and displayed all the functions within the MyApp namespace. If MyApp contained operators or variables you would find them in the document as well.

We can improve this in several ways. Time for a new version of MyApp! Make a copy of Z:\code\v08 as Z:\code\v09.

6.1. Leading comments: basic information

First we edit the top of the script to follow ADoc’s conventions:

:Namespace MyApp
    ⍝ Counting letter frequencies in text.\\
    ⍝ Can do one of:
    ⍝ * calculate the letters in a given document.
    ⍝ * calculate the letters in all documents in a given folder.
    ⍝
    ⍝ Sample application used by the Dyalog Cookbook.\\
    ⍝ Authors: Kai Jaeger & Stephen Taylor.
    ⍝ For more details see <http://cookbook.dyalog.com>
...

6.2. Public interface

Next we specify which functions we want to be included in the document: not all but just those designed to be called from the outside. In a class those are called the public interface, and it’s easy to see why.

For classes ADoc can work out what’s public and what isn’t using the Public Access statements. For namespaces there is no such mechanism.

By default, ADoc considers all functions, operators and variables to be public; it also offers a mechanism to restrict this to what’s really public. For that, ADoc looks for a function Public. It may return an empty vector (nothing is public at all) or a list of names. This list would define what is public.

Let’s define the public functions at the bottom of the script:

...
∇ r←Public
  r←'StartFromCmdLine' 'TxtToCsv' 'SetLX'
∇
:EndNamespace

6.3. Reserved names

ADoc honours five functions in a special way if they exist: Copyright, History, Version, Public and ADOC_Doc. If they exist, their results are treated in a special way.

6.3.1. Version

If Version is niladic and returns a three-item vector, the vector is taken as:

These pieces of information are then integrated into the document.

6.3.2. Copyright

If Copyright is niladic and returns either a simple text vector or a vector of text vectors then the copyright declaration is integrated into the document.

6.3.3. History

History is expected to be a niladic function that does not return a result. Instead it should carry comments with information about the history of the script.

MyApp already had a function Version in place. So far we’ve added comments to it regarding the different versions. Those should go into History instead. So we replace the existing Version function by these three functions:

∇ Z←Copyright
  Z←'The Dyalog Cookbook, Kai Jaeger & Stephen Taylor 2017'
∇

∇ r←Version
  r←(⍕⎕THIS)'1.5.0' 'YYYY-MM-DD'
∇

∇ History
  ⍝ * 1.5.0:
  ⍝   * MyApp is now ADocable (function Public.
  ⍝ * 1.4.0:
  ⍝   * Handles errors with a global trap.
  ⍝   * Returns an exit code to calling environment.
  ⍝ * 1.3.0:
  ⍝   * MyApp gives a Ride now, INI settings permitted.
  ⍝ * 1.2.0:
  ⍝   * The application now honours INI files.
  ⍝ * 1.1.0:
  ⍝   * Can now deal with non-existent files.
  ⍝   * Logging implemented.
  ⍝ * 1.0.0
  ⍝   * Runs as a stand-alone EXE and takes parameters from the command line.
∇

This gives us more prominent copyright and version notices as well as information about the most recent changes. Note that History is not expected to carry a history of all changes, but rather the most recent ones.

The variables inside EXIT are essential for using MyApp; they should be part of the documentation. ADoc has ignored the namespace EXIT but we can change this by specifying it explicitly:

    ]ADoc #.MyApp #.MyApp.EXIT

Browsing the revised MyApp namespace

When you scroll down (or click at Exit in the top-left corner) then you get to the part of the document where EXIT is documented:

Browsing the revised MyApp namespace

6.3.4. Public

See Public interface above.

6.3.5. ADOC_Doc

There is one more reserved name: ADOC_Doc. This is useful when you want to document an unscripted (ordinary) namespace. Just add this as a niladic function carrying comments that returns no or a shy result.

Its comments are then processed in exactly the same way leading comments in scripts are processed.

That will do for now.


Footnotes

  1. Compile those from questions actually asked by users. It's a common mistake to put together “Question we would like our users to ask”.

  2. Thanks to Roger Hui for this term.

  3. Be it ⍝FIXME⍝ or ⍝CHECKME⍝ or ⍝TODO⍝ - what matters is that you keep it consistent and searchable. That implies that the search term cannot be mistaken as something else by accident. For that reason, ⍝TODO⍝ is slighty better than TODO.

  4. https://en.wikipedia.org/wiki/Markdown

Chapter 10:

Make me

It’s time to take a closer look at the process of building the application workspace and exporting the EXE. In this chapter we’ll

At first glance you might think all we need are two versions of the DYAPP, one for development and one for producing the EXE, but there will be tasks we cannot carry out with this approach. Examples are:

We resume, as usual, by saving a copy of Z:\code\v09 as Z:\code\v10. Now delete MyApp.exe from Z:\code\v10: from now on we will create the EXE somewhere else.

1. The development environment

MyApp.dyapp does not need many changes, it comes with everything needed for development. The only thing we add is to execute the test cases automatically. Almost automatically.

In an ideal world we would ensure all test cases pass before the end of each working day. But sometimes that is just not possible, due to the amount of work involved.

In such cases it might be sensible to execute the test cases before you start working: if you know they will fail and there are many of them there is no point in wasting computer resource and your time; better ask.

1.1. Development helpers

For that we are going to have a function YesOrNo, very simple and straightforward. Its right argument (question) is printed to the session and then the user might answer that question.

If she does not enter one of: “YyNn” the question is repeated. If she enters one of “Yy” a 1 is returned, otherwise a 0. Since we use this to ask ourself (or other programmers) the function does not have to be bulletproof; we just use ¯1↑⍞.

But where exactly should this function go? Though it is helpful it has no part in our final application. Therefore we put it into a new script called DevHelpers. We also add a function RunTests to this new script:

:Namespace DevHelpers

∇ {r}←RunTests forceFlag
⍝ Runs the test cases in debug mode, either in case the user wants to
⍝ or if `forceFlag` is 1.
  r←''
  :If forceFlag
  :OrIf YesOrNo'Would you like to execute all test cases in debug mode?'
      r←#.Tests.RunDebug 0
  :EndIf
∇

∇ flag←YesOrNo question;isOkay;answer
  isOkay←0
  ⎕←(⎕PW-1)⍴'-'
  :Repeat
      ⍞←question,' (y/n) '
      answer←¯1↑⍞
      :If answer∊'YyNn'
          isOkay←1
          flag←answer∊'Yy'
      :EndIf
  :Until isOkay
∇

:EndNamespace

1.2. Running test cases first thing in the morning

We add a line to the bottom of MyApp.dyapp:

...
Run #.Tester.EstablishHelpersIn #.Tests
Run #.DevHelpers.RunTests 0

Now a developer who double-clicks the DYAPP in order to assemble the workspace will always be reminded of running all test cases before she starts working on the application. Experience tells us that this is a good thing.

2. MyApp.dyalog

One minor thing needs our attention: because we create MyApp.exe now in a folder MyApp, simply setting ⎕WSID to MyApp does not do anymore. We need to make a change to the StartFromCmdLine function in MyApp.dyalog:

...
∇ {r}←StartFromCmdLine arg;MyLogger;Config;rc;⎕TRAP
   ⍝ Needs command line parameters, runs the application.
      r←⍬
      ⎕TRAP←#.HandleError.SetTrap ⍬
      ⎕SIGNAL 0
      ⎕WSID←⊃{⍵/⍨~'='∊¨⍵}{⍵/⍨'-'≠⊃¨⍵}1↓2⎕nq # 'GetCommandLineArgs'
      #.FilesAndDirs.PolishCurrentDir
...

This change ensures the ⎕WSID will be correct. Under the current circumstances it will be MyApp\MyApp.dws.

Note that we access GetCommandLineArgs as a function call with ⎕NQ rather than referring to #.GetCommandLineArgs; over the years that has proven to be more reliable.

3. Make the application

Information

In most programming languages the process of compiling the source code and putting together an application is done by a utility that's called Make; we use the same term.

At first sight it might seem all we need is a reduced version of MyApp.dyapp, but not so. Soon we will discuss how to add a Help system to our application.

We must then make sure that the Help system is compiled properly when the application is assembled. Later, more tasks will come up. Conclusion: our Make file cannot be a DYAPP; we need more flexibility.

More complex scenarios

In a more complex application than ours you might prefer a different approach. Using an INI file for this is not a bad idea: it gives you scope to define more than just the modules to be loaded, and some code to execute.

Also, if you have not one but several applications to support, it is useful to implement your own generalised user command like ]runmake.

Execute, Tester and Tests have no place in the finished application, nor do we need the test helpers either.

3.1. Batch file for starting Dyalog

For now, we’ll create a DYAPP file Make.dyapp that performs the Make.

However, if you want to specify explicitly the version of Dyalog that should run this DYAPP rather than using whichever version happens to be associated with the file extension DYAPP at the time you double-click it, (also DWS and DYALOG) you need a batch file that starts the correct version of Dyalog.

Create such a batch file as Make.bat:

"C:\Program Files\Dyalog\Dyalog APL{yourPreferredVersion}\Dyalog.exe" DYAPP="%~dp0Make.dyapp"
@echo off
if NOT ["%errorlevel%"]==["0"] (
    echo Error %errorlevel%
    pause
    exit /b %errorlevel%
)

Edit to use your chosen Dyalog version of your choice. You can see the version currently associated on your machine:

'"',(⊃#.GetCommandLineArgs),'"'

You might want to add other parameters like MAXWS=128M (or MAXWS=6G) to the BAT.

Notes:

The current directory

For APLers, the current directory (sometimes called “working directory”) is, when running under Windows, a strange animal. In general, the current directory is where ‘the application’ lives.

That means that for an application C:\Program Files\Foo\Foo.exe the current directory will be C:\Program Files\Foo.

For APLers “the application” is not the DYALOG.EXE, it’s the workspace, whether it was loaded from disk or assembled by a DYAPP. When you double-click MyApp.dyapp the interpreter changes the current directory for you: it’s going to be where the DYAPP lives, which suits an APL application programmer’s point of view.

The same holds true when you double-click a DWS but it is not true when you load a workspace: the current directory then remains what it was before, by default where the Dyalog EXE lives.

So it’s smart to change the current directory yourself at the earliest possible stage after loading a workspace: call #.FilesAndDirs.PolishCurrentDir and you’re covered, no matter what the circumstances are. One of the authors has been doing this for roughly 20 years now, and it has solved several problems without introducing new ones.

3.2. The DYAPP file

Now we need to establish the Make.dyapp file:

Target #
Load ..\AplTree\APLTreeUtils
Load ..\AplTree\FilesAndDirs
Load ..\AplTree\HandleError
Load ..\AplTree\IniFiles
Load ..\AplTree\OS
Load ..\AplTree\Logger
Load Constants
Load Utilities
Load MyApp
Run #.MyApp.SetLX ⍬

Load Make
Run #.Make.Run 1

The upper part (until the blank line) is identical with MyApp.dyapp, without the stuff that’s needed only during development. We then load a script Make and finally we call Make.Run. Here’s Make at this point:

:Class Make
⍝ Puts the application `MyApp` together:
⍝ 1. Remove folder `DESTINATION\` in the current directory
⍝ 2. Create folder `DESTINATION\` in the current directory
⍝ 3. Copy icon to `DESTINATION\`
⍝ 4. Copy the INI file template over to `DESTINATION`
⍝ 5. Creates `MyApp.exe` within `DESTINATION\`
    ⎕IO←1 ⋄ ⎕ML←1

    DESTINATION←'MyApp'

    ∇ {filename}←Run offFlag;rc;en;more;successFlag;F;U;msg
      :Access Public Shared
      F←##.FilesAndDirs ⋄ U←##.Utilities
      (rc en more)←F.RmDir DESTINATION
      U.Assert 0=rc
      successFlag←'Create!'F.CheckPath DESTINATION
      U.Assert successFlag
      (successFlag more)←2↑'images'F.CopyTree DESTINATION,'\images'
      U.Assert successFlag
      (rc more)←'MyApp.ini.template'F.CopyTo DESTINATION,'\MyApp.ini'
      U.Assert 0=rc
      Export'MyApp.exe'
      filename←DESTINATION,'\MyApp.exe'
      :If offFlag
          ⎕OFF
      :EndIf
      ∇
:EndClass

3.3. Assertions

It is common practice in any programming language to inject checks into the code to throw an error if necessary conditions are not met.

Let’s define a function Assert in Utilities:

:Namespace Utilities
      map←{
          (,2)≢⍴⍺:'Left argument is not a two-element vector'⎕SIGNAL 5
          (old new)←⍺
          nw←∪⍵
          (new,nw)[(old,nw)⍳⍵]
      }
       Assert←{⍺←'' ⋄ (success errorNo)←2↑⍵,11 ⋄ (,1)≡,success:r←1 ⋄ ⍺ ⎕SIGNAL errorNo}
:EndNamespace

Notes:

Because it’s a one-liner you cannot trace into Assert. That’s a good thing.

This is an easy way to make the calling function stop when something goes wrong. There is no point in doing anything but stopping the code from continuing since it is called by a programmer. When it fails you want to investigate straight away.

And things can go wrong quite easily. For example, removing DESTINATION might fail simply because another user is looking into DESTINATION with Windows Explorer.

First we create the folder DESTINATION from scratch and then we copy everything that’s needed to the folder DESTINATION: the application icon and the INI file. Whether the function executes ⎕OFF or not depends on the right argument offFlag. Why that is needed will become apparent soon.

3.4. INI files

We don’t copy MyApp.ini into DESTINATION but MyApp.ini.template; therefore we must create this file: copy MyApp.ini to MyApp.ini.template and then check its settings: in particular these settings are important:

...
[Config]
Debug       = ¯1   ; 0=enfore error trapping; 1=prevent error trapping;
Trap        = 1    ; 0 disables any :Trap statements (local traps)
ForceError  = 0    ; 1=let TxtToCsv crash (for testing global trap handling)
...
[Ride]
Active      = 0
...

Those might well get changed in MyApp.ini while working on the project, so we make sure that we get them set correctly in MyApp.ini.template.

However, that leaves us open to another problem. Suppose we introduce a new section and/or a new key and forget to copy it over to the template. To prevent this we add a test case to Tests:

    ∇ R←Test_misc_01(stopFlag batchFlag);⎕TRAP;ini1;ini2
      ⍝ Check if MyApp.ini & MyApp.ini.template have same sections & keys
      ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
      R←∆Failed
      ini1←⎕NEW ##.IniFiles(,⊂'MyApp.ini')
      ini2←⎕NEW ##.IniFiles(,⊂'MyApp.ini.template')
      →PassesIf ini1.GetSections{(∧/⍺∊⍵)∧(∧/⍵∊⍺)}ini2.GetSections
      →PassesIf(ini1.Get ⍬ ⍬)[;2]{(∧/⍺∊⍵)∧(∧/⍵∊⍺)}(ini2.Get ⍬ ⍬)[;2]
      R←∆OK
    ∇

The test simply checks whether the two INI files have the same sections and the same keys; that will alert us if we forget something.

3.5. Prerequisites

3.5.1. Bind types

For the Bind method we can specify different types. We add them to the Constants namespace, in their own subspace:

:Namespace Constants
...
    :EndNamespace
    :Namespace BIND_TYPES
        ActiveXControl←'ActiveXControl'
        InProcessServer←'InProcessServer'
        Library←'Library'
        NativeExe←'NativeExe'
        OutOfProcessServer←'OutOfProcessServer'
        StandaloneNativeExe←'StandaloneNativeExe'
    :EndNamespace
:EndNamespace

Why do this? By listing all available options, it makes the code self-explanatory.

3.5.2. Flags

:Namespace Constants
...
    :EndNamespace
    :Namespace BIND_FLAGS
        BOUND_CONSOLE←2
        BOUND_USEDOTNET←4
        RUNTIME←8
        BOUND_XPLOOK←32
    :EndNamespace
:EndNamespace

3.6. Exporting

Run then calls Export, a new private function in the Make class:

...
    ∇ {r}←{flags}Export exeName;type;flags;resource;icon;cmdline;try;max;success;details;fn
    ⍝ Attempts to export the application
      r←⍬
      flags←##.Constants.BIND_FLAGS.RUNTIME{⍺←0 ⋄ 0<⎕NC ⍵:⍎⍵ ⋄ ⍺}'flags'
      max←50
      type←##.Constants.BIND_TYPES.StandaloneNativeExe
      icon←F.NormalizePath DESTINATION,'\images\logo.ico'
      resource←cmdline←''
      details←''
      details,←⊂'CompanyName' 'My company'
      details,←⊂'ProductVersion'(2⊃##.MyApp.Version)
      details,←⊂'LegalCopyright' 'Dyalog Ltd 2018'
      details,←⊂'ProductName' 'MyApp'
      details,←⊂'FileVersion' (2⊃##.MyApp.Version)
      details←↑details
      success←try←0
      fn←DESTINATION,'\',exeName     ⍝ filename
      :Repeat
          :Trap 11
              2 ⎕NQ'.' 'Bind' fn type flags resource icon cmdline details
              success←1
          :Else
              ⎕DL 0.2
          :EndTrap
      :Until success∨max<try←try+1
      :If 0=success
          ⎕←'*** ERROR: Failed to export EXE to ',fn,' after ',(⍕try),' tries.'
          . ⍝ Deliberate error; allows investigation
      :EndIf
    ∇
:EndClass

Export automates what we’ve done so far by calling the Export command from the File menu. If the Bind method fails, it retries up to 50 times before giving up.

From experience we know that, with the OS, the machine, the network, the filesystem and who knows what else, the command can fail several times before finally succeeding.

Information

Why is there a “ProductVersion” and a “FileVersion”? No idea! On Stack Overflow this was discussed more than once, and it seems that there are very few cases when it might make sense to have them not in sync.

But “FileVersion” is the more important one: the Inno installer for example (see chapter 16 “Creating SetUp.exe”) compares the “FileVersion” of an already installed version with the possibly new version, and if they are not different then it won't overwrite the EXE - you don't want that!

The Bind method

Note that for the Bind method to work as discussed in this chapter you must use at least version 16.0.31811.0 of Dyalog. Before that Bind was not an official method and did not support the details.

Double-click Make.dyapp: a folder MyApp should appear in Z:\code\v10 with, among other files, MyApp.exe.

3.7. Check the result

Open a Windows Explorer (Windows + E), navigate to the folder hosting the EXE, right-click the EXE and select Properties from the context menu, then click on the Details tab.

EXEs properties

As you can see, the fields File version, Product name, Product version and Copyright hold the information we have specified.

Warning

Note that the names we have used are not the names used by Microsoft in the GUI. The MSDN [2] provides details.

4. The tests

Now that we have a way automatically to assemble all the files required by our application we need to amend our tests. Double-click MyApp.dyapp. You don't need to execute the test cases right now because we are going to change them.

We need to make a few changes:

:Namespace Tests
    ⎕IO←1 ⋄ ⎕ML←1
    ∇ Initial;list;rc
      U←##.Utilities ⋄ F←##.FilesAndDirs ⋄ A←##.APLTreeUtils
      ∆Path←F.GetTempPath,'\MyApp_Tests'
      F.RmDir ∆Path
      'Create!'F.CheckPath ∆Path
      list←⊃F.Dir'..\..\texts\en\*.txt'
      rc←list F.CopyTo ∆Path,'\'
      :If ~R←0∧.=⊃rc
          ⎕←'Could not create ',∆Path
      :EndIf
      ⎕SE.UCMD'Load ',F.PWD,'\Make.dyalog -target=#'
      #.Make.Run 0
    ∇
 ...
:EndNamespace

Initial

5. Workflow

With the two DYAPPs and the BAT file, your development cycle now looks like this:

  1. Launch MyApp.dyapp and check the test results.
  2. Fix any errors and rerun #.Tests.Run until it’s fine. If you edit the test themselves, either rerun
    `#.Tester.EstablishHelpersIn #.Tests`

    or simply close the session and relaunch MyApp.dyapp.


Footnotes

  1. Worst Case User, also known as Dumbest Assumable User (DAU).

  2. The MSDN provides more information on what names are actually recognized.

Chapter 11:

Providing help

Users expect applications to provide help in one way or another. One option is to provide the help as a hypertext system. Under Windows, CHM files are the standard way to provide such help. There are powerful applications available that can assist you in providing help; HelpAndManual [1] is just an example.

However, we take a different approach here: rather than using any third-party software we use Markdown2Help from the APL_cation [2] project. That allows us to create a help system that:

This is the simplest way to create a Help system, and it allows you to run the Help system from within your application in order to view either its start page or a particular page as well as viewing the Help system without running your application at all.

While CHM files are Windows specific, Markdown2Help allows you to export a Help system as a web page that can be displayed with any modern browser. That makes it OS-independent. We’ll discuss later how to do this.

1. Getting ready

It’s time to save a copy of Z:\code\v10 as Z:\code\v11.

To use Markdown2Help you need to download it from http://download.aplwiki.com/. We suggest creating a folder Markdown2Help within the folder Z:\code\APLTree. Copy into Z:\code\APLTree\Markdown2Help the contents of the ZIP you’ve just downloaded:

Download target

Within that folder you will find a workspace Markdown2Help (from which we are going to copy the module) and a folder help.

This folder contains in turn a subfolder files (which contains Markdown2Help’s own Help system) and the file ViewHelp.exe. That EXE is the external viewer for viewing your Help system independently from your application.

Double-click ViewHelp.exe in order to see Markdown2Help2’s own Help system:

Markdown2Help's Help

By default, ViewHelp.exe expects to find a folder files as a sibling of itself, and it assumes this folder contains a Help system.

Specify help folder and help page

You can change the folder ViewHelper.exe expects to host the Help system by specifying a command-line parameter helpFolder:

ViewHelp.exe -helpfolder=C:\Foo\Help

You can also tell ViewHelper.exe to put a particular help page on display rather than the default page:

ViewHelp.exe -page=Sub.Foo

However, all these details are discussed in Markdown2Help’s own Help system.

Markdown2Help is an ordinary (non-scripted) namespace. We therefore need to copy it from its workspace. We also need the script MarkAPL, which is used to convert the help pages from Markdown to HTML. You know by now how to download scripts from the APLTree library. Modify MyApp.dyapp so that it loads the module MarkAPL and also copies Markdown2Help:

...
Load ..\AplTree\Execute
Load ..\AplTree\MarkAPL
Run 'Markdown2Help' #.⎕CY '..\apltree\Markdown2Help\Markdown2Help.dws'
Load Tests
...

Double-click the DYAPP to get started.

2. Creating a new Help system

Markdown2Help comes with a function CreateStub that creates a new Help system for us. We need an unused name for it: the obvious candidate is MyHelp.

We want the Help system managed by SALT, for which we need a folder for all the Help files. For that we call CreateParms and then specify the folder in the parameter saltFolder:

parms←#.Markdown2Help.CreateParms ⍬
parms.saltFolder←'Z:\code\v11\MyHelp'
parms.folderName←'Z:\code\v11\Help\Files'
parms #.Markdown2Help.CreateStub '#.MyHelp'

CreateStub will create some pages and a node (or folder) for us; here’s what you should see:

Download target

Notes:

3. Behind the scenes

In the workspace all nodes (in our case MyHelp and Sub) are ordinary namespaces, while the pages are variables. You can check with the Workspace Explorer:

The help system in the Workspace Explorer

This is why the names of nodes and pages must be valid APL names. Those names appear in the Help tree as topics by default, but we can of course improve on that. We’ll come back to this soon.

4. Editing a page

When you right-click on a page like Copyright and then select Edit help page from the context menu (pressing <Ctrl+Enter> will do the same) the APL editor opens and shows something similar to this:

A help page in the editor

This is the source of the help page in Markdown.

Notes:

Make some changes, for example add another paragraph Go to →[Overview], and then press Esc. Markdown2Help takes your changes, converts the Markdown to HTML and shows you the changed page.

This gives you an idea of how easy it actually is to change help pages. Adding, renaming and deleting help pages – and nodes – can be achieved via the context menu.

Note also that →[Overview] is a link. For the link to work Overview must be the name of an existing page. If the title of the page differs from the name, the title will appear as the link text in the help page.

Watch out Read Markdown2Help’s own help file before you start using Markdown2Help in earnest. Some Markdown features are not supported by the Help system, and internal links are implemented in a simplified way.

5. Changing title and sequence

Note that the Copyright page comes first. That’s because by default the pages are ordered alphabetically. You can change this with a right-click on either the Copyright or the Overview page and then selecting Manage ∆TopicProperties.

After confirming this is really what you want to do you will see something like this:

 ∆TopicProperties←{
⍝ This function is needed by the Markdown2Help system.
⍝ You can edit this function from the Markdown2Help GUI via the context menu.
⍝ *** NOTE:
⍝     Make only changes to this function that affect the explicit result.
⍝     Any other changes will eventually disappear because these functions are rebuilt
⍝     under program control from their explicit result under certain circumstances.
⍝        This is also the reason why you should use the `active` flag to hide a topic
⍝     temporarily because although just putting a `⍝` symbol in front of its line
⍝     seems to have the same effect, in the long run that's not true because the
⍝     commented line will disappear in the event of a rebuild.
⍝ ----------------------------------
⍝ r gets a table with these columns:
⍝ [;0] namespace or function name.
⍝ [;1] caption in the tree view. If empty the namespace/fns name is taken.
⍝ [;2] active flag.
⍝ [;3] developmentOnly flag; 1=the corresponding node does not show in user mode.
     r←0 4⍴''
     r⍪←'Copyright' '' 1 0
     r⍪←'Overview' '' 1 0
     r⍪←'Sub' '' 1 0
     r
}

We recommend reading the comments in this function.

You can specify a different sequence of the pages simply by changing the sequence in which the pages are added to r. Here we swap the position of Copyright and Overview:

 ∆TopicProperties←{
     ...
     r←0 4⍴''
     r⍪←'Overview' 'Miller''s overview' 1 0
     r⍪←'Copyright' '' 1 0
     r⍪←'Sub' '' 1 0
     r
 }

We have also changed the title of the Overview page to Miller’s overview. That’s how you can specify an alternative title to the name of the page.

After fixing the function, the Help system is recompiled automatically; and our changes become visible immediately:

The changed help system

What “compiling the help system” actually means is discussed soon.

6. More commands

The context menu has many commands. The first three commands are always available. The other commands are useful for a developer (or shall we say Help system author?) and are available only when the Help system is running in a development version of Dyalog.

The context menu

As a developer you will have no problem mastering these commands.

7. Compiling the help system

Compiling the help system converts

into a single component file (DCF) containing the HTML generated from the Markdown, plus some more pieces of information.

It’s more than just converting Markdown to HTML. For example, the words of all the pages are extracted, ‘dead’ words like and, then, it, etc. are removed (because searching for them does not make too much sense) and the index, together with pointers to the pages they appear on, are saved in a component.

This allows Markdown2Help to provide a very fast Search function. The list is actually saved in two forms, ‘as is’ and with all words lowercased to speed up any case-insensitive search operations.

Without specifying a particular folder, Markdown2Help would create a temporary folder and compile into that folder. It is better to define a permanent location, which avoids having the Help system compile the Markdown into HTML whenever it is called.

Such a permanent location is also the precondition for using the Help system with the external viewer, necessary if your Help system tells how to install your application.

For converting the Markdown to HTML, Markdown2Help needs the MarkAPL class, but once the Help system has been compiled this class is no longer needed. The final version of your application does not need MarkAPL. As MarkAPL comprises roughly 3,000 lines of code, this is good news.

8. Editing the Help system directly

Besides editing a variable with a double-click in the Workspace Explorer, you could also edit it from the session with )ED. Our advice: don't!

The reason is simple: when you change a Help system via the context menu then other important steps are performed. An example is when you have a ∆TopicProperties function in a particular node and you want to add a new help page to that node.

You have to right-click on a page and select the Inject new help page (stub) command from the context menu. You will then be prompted for a valid name and finally the new help page is injected after the page you have clicked at.

But there is more to it than just that: the page is also added to the ∆TopicProperties function. That’s one reason why you should perform all changes via the context menu rather than manipulating the Help system directly.

Maybe even more important: Markdown2Help also executes the necessary steps in order to keep the files and folders in saltFolder in sync with the Help system and automatically recompiles the Help system for you.

The only exception is when you change your mind about the structure of a Help system. If that involves moving around namespaces or plenty of pages between namespaces then it is indeed better to do it in the Workspace Explorer and, when you are done, to check all the ∆TopicProperties functions within your Help system and finally recompile the Help system; unless somebody implements drag-and-drop for the TreeView of the Help system one day…

However, if you do that, you must ensure the Help system is saved properly. That means that you have to invoke the SaveHelpSystemWithSalt method yourself. You also need to call the Markdown2Help.CompileHelpFileInto method to compile the Help system from the source. Refer to Markdown2Help’s own Help system for details.

9. The Developers menu

If the Help system is running under a development version of Dyalog, you will see a Developers menu on the right side of the menubar. This offers commands that support you in keeping your Help system healthy. We discuss just the most important ones:

9.1. Show topic in browserShow topic in browser

Particularly useful when you use non-default CSS and there is a problem with it: all modern browsers offer excellent tools for investigating CSS, supporting you when hunting bugs or trying to understand unexpected behaviour.

9.2. Create proofread documentCreate proofread document

This command creates an HTML document from all the help pages and writes the HTML to a temporary file. The filename is printed to the session.

You can then open that document with your preferred word processor, say Microsoft Word. This will show something like this:

The help system as a single HTML page

This is a great help when it comes to proofreading a document: one can use the review features of the word processor, and also print the document. You are much more likely to spot any problems in a printed copy than on a screen.

9.3. ReportsReports

Several reports identify broken and ambiguous links, ∆TopicProperties functions, and help pages that do not carry any index entries.

10. Export to HTML

You can export the Help system as a website. For that select Export as HTML from the File menu.

The resulting website does not offer all the features the Windows version comes with, but you can read and print all the pages, you have the tree structure representing the contents, and all the links work. <!– That must do under Linux and macOS for the time being. –>

11. Making adjustments

If you have not copied the contents of code\v11\* from the book’s website then you need to make adjustments to the Help system to keep it in sync with the book. We have just two help pages; a page regarding the main method TxtToCsv:

The changed help system

And a page regarding copyright:

The changed help system

12. How to view the Help system

We want to confirm we can call the Help system from within our application. For that we need a new function; its obvious name is ShowHelp.

The function’s vector right argument is the name of the page the Help system should open at; if empty, the first page is shown. It returns an instance of the Help system. The function goes into the MyApp.dyalog script:

:Namespace MyApp
...
∇

∇{r}←ShowHelp pagename;ps
  ps←#.Markdown2Help.CreateParms ⍬
  ps.source←#.MyHelp
  ps.foldername←'Help'
  ps.helpAbout←'MyApp''s help system by John Doe'
  ps.helpCaption←'MyApp Help'
  ps.helpIcon←'file://',##.FilesAndDirs.PWD,'\images\logo.ico'
  ps.helpVersion←'1.0.0'
  ps.helpVersionDate←'YYYY-MM-DD'
  ps.page←pagename
  ps.regPath←'HKCU\Software\MyApp'
  ps.noClose←1
  r←#.Markdown2Help.New ps
∇

∇ r←Public
  r←'StartFromCmdLine' 'TxtToCsv' 'SetLX' 'ShowHelp'
∇

:EndNamespace
Information

A Windows Registry key? The user can mark any help page as a favourite, and this is saved in the Windows Registry. We will discuss the Windows Registry in a later chapter.

This function requires the Help system to be available in the workspace.

Strictly speaking, only the source parameter needs to be specified to get it to work, but best to specify the other parameters too before a client sets eyes on your Help system.

Most of the parameters should explain themselves, but if in doubt you can always start Markdown2Help’s own Help system with #.Markdown2Help.Selfie ⍬ and read the pages under the Parameters node. Here’s what you should see:

The context menu

You can request a list of all parameters with their default values with this statement:

      ⎕←(#.Markdown2Help.CreateParms'').∆List''

Note that CreateParms is one of the few functions in the APLTree library so named that actually requires a right argument. <!– Breaking our rule! –> This right argument may be just an empty vector, but instead it could be a namespace with variables like source or page. In that case CreateParms would inject any missing parameters into that namespace and return it as a result.

Therefore we could rewrite the function ShowHelp:

∇{r}←ShowHelp pagename;ps
  ps←⎕NS ''
  ps.source←#.MyHelp
  ps.foldername←'Help'
  ps.helpAbout←'MyApp''s help system by John Doe'
  ps.helpCaption←'MyApp Help'
  ps.helpIcon←'file://',##.FilesAndDirs.PWD,'\images\logo.ico'
  ps.helpVersion←'1.0.0'
  ps.helpVersionDate←'YYYY-MM-DD'
  ps.page←pagename
  ps.regPath←'HKCU\Software\MyApp'
  ps.noClose←1
  ps←#.Markdown2Help.CreateParms ps
  r←#.Markdown2Help.New ps
∇

This version of ShowHelp would produce exactly the same result.

13. Calling the Help system from your application

14. Adding the help system to “MyApp.dyapp”

Now that we have a Help system that is saved in the right place we have to ensure it is loaded when we assemble a workspace with a DYAPP. First we add a function LoadHelp to the DevHelpers class:

:Namespace DevHelpers
...

    ∇{r}←LoadHelp dummy;parms
    parms←#.Markdown2Help.CreateParms ⍬
    parms.saltFolder←#.FilesAndDirs.PWD,'\MyHelp'
    parms.source←'#.MyHelp'
    parms.folderName←#.FilesAndDirs.PWD,'\Help\Files'
    {}#.Markdown2Help.LoadHelpWithSalt parms
    ∇

:EndNamespace

Calling this function will load the Help system from saltFolder into the namespace #.MyHelp in the current workspace. So we need to call this function within MyApp.dyapp:

...
Load DevHelpers
Run DevHelpers.LoadHelp ⍬
Run #.MyApp.SetLX ⍬
...

15. Enhancing Make.dyapp and Make.dyalog

Now we need to ensure the Make process includes the Help system. First we add the required modules to Make.dyapp:

Target #
Load ..\AplTree\APLTreeUtils
Load ..\AplTree\FilesAndDirs
Load ..\AplTree\HandleError
Load ..\AplTree\IniFiles
Load ..\AplTree\OS
Load ..\AplTree\Logger
Load ..\AplTree\EventCodes
Load ..\APLTree\WinReg
Run 'Markdown2Help' #.⎕CY '..\apltree\Markdown2Help\Markdown2Help.dws'
Load Constants
Load Utilities
Load MyApp
Run #.MyApp.SetLX ⍬

Load Make
Run #.Make.Run 1

Finally we ensure the compiled Help system is copied over together with the standalone Help Viewer:

:Class Make
...
⍝ 5. Creates `MyApp.exe` within `DESTINATION\`
⍝ 6. Copy the Help system into `DESTINATION\Help\files`
⍝ 7. Copy the stand-alone Help viewer into `DESTINATION\Help`
⎕IO←1 ⋄ ⎕ML←1

    DESTINATION←'MyApp'

    ∇ {filename}←Run offFlag;rc;en;more;successFlag;F;U;msg
      :Access Public Shared
      (F U)←##.(FilesAndDirs Utilities)
      (rc en more)←F.RmDir DESTINATION
      U.Assert 0=rc
      successFlag←'Create!'F.CheckPath DESTINATION
      U.Assert successFlag
      (successFlag more)←2↑'images'F.CopyTree DESTINATION,'\images'
      U.Assert successFlag
      (rc more)←'MyApp.ini.template'F.CopyTo DESTINATION,'\MyApp.ini'
      U.Assert 0=rc
      (successFlag more)←2↑'Help\files'F.CopyTree DESTINATION,'\Help\files'
      U.Assert successFlag
      (rc more)←'..\apltree\Markdown2Help\help\ViewHelp.exe'F.CopyTo DESTINATION,'\Help\'
      U.Assert 0=rc
      Export'MyApp.exe'
      filename←DESTINATION,'\MyApp.exe'
      :If offFlag
          ⎕OFF
      :EndIf
    ∇
...
:EndClass

Footnotes

  1. http://www.helpandmanual.com/

  2. https://github.com/aplteam/apltree/wiki/Members

Chapter 12:

Scheduled Tasks

1. What is a Scheduled Task?

Windows offers a task scheduler in order to run applications at specific times. Like Services, Scheduled Tasks are designed for background jobs, meaning that such applications have no GUI, in fact, cannot have a GUI.

The Scheduler allows you to start the application on a specific date and time once, or every day, every week or every month. The user does not have to be logged on (that’s different in older versions of Windows) and it let you run an application in elevated mode (see below).

2. What can and cannot be achieved by Scheduled Tasks

Scheduled Tasks – like Services – are perfect for background tasks. Examples are:

Scheduled Tasks cannot interact with the user: when you try to put up a GUI and ask a question nothing appears on the screen: you just can’t do it.

3. Scheduled Tasks versus Services

If your application needs to run all the time, even with delays between actions, then running as a Service would be more appropriate. Services are typically started automatically when the machine is booted, and they typically keep running until the next boot.

To make this point clear, imagine these two scenarios:

The former is clearly a candidate for a Scheduled Task while the latter is a candidate for a Service.

4. Preconditions for a Scheduled Task

You need either a saved workspace with ⎕LX set or an EXE created from a workspace. An EXE has two advantages compared with an ordinary workspace:

  1. The user cannot investigate the code.
  2. Dyalog is not required on the target system, not even the Runtime EXE.

If neither of these concerns you, an EXE has no advantages over a simple saved workspace; it just adds complexity and should be avoided if there aren’t any advantages. However, if you cannot be sure the required version of Dyalog is installed on the target machine then it must be a stand-alone EXE.

We have already taken care of handling errors and writing to log files, which are the only sources of information in general, and in particular for analysing any problems that pop up when a Scheduled Task runs, or crashes. In other words, we are ready to go.

Our application is an obvious candidate for running as a Service, but we can still run it as a Scheduled Task, so let’s explore that way.

5. Precautions: ensure one instance only

Dealing with Scheduled Tasks you usually don’t want more than one instance of the application running at the same time. One of the most common sources of difficulty investigating Scheduled Tasks turns out to be running a second instance.

For example, you try to ride into it but the port used by Ride is already occupied by an instance started earlier without you knowing of it. We are going to prevent this happening.

In the rare circumstances when you want application instances managed by the Task Scheduler to run in parallel, establish a mechanism that allows you to enforce having just one instance running, if only for debugging purposes. Make it an INI entry (like AllowMultipleInstances) and document it appropriately.

We resume, as usual, by saving a copy of Z:\code\v11 as Z:\code\v12.

In order to force the application to run only once at any given time we add a function CheckForOtherInstances to MyApp.dyalog:

∇ {tno}←CheckForOtherInstances dummy;filename;listOfTiedFiles;ind
 ⍝ Attempts to tie the file "MyAppCtrl.dcf" exclusively and returns the tie number.
 ⍝ If that is not possible then an error is thrown because we can assume that the
 ⍝ application is already running.\\
 ⍝ Notes:
 ⍝ * In case the file is already tied it is untied first.
 ⍝ * If the file does not exist it is created.
   filename←'MyAppCtrl.dcf'
   :If 0=F.IsFile filename
       tno←filename ⎕FCREATE 0
   :Else
       :If ~0∊⍴⎕FNUMS
           listOfTiedFiles←A.dtb↓⎕FNAMES
           ind←listOfTiedFiles⍳⊂filename
       :AndIf ind≤⍴⎕FNUMS
           ⎕FUNTIE ind⊃⎕FNUMS
       :EndIf
       :Trap 24
           tno←filename ⎕FTIE 0
       :Else
           'Application is already running'⎕SIGNAL C.APP_STATUS.ALREADY_RUNNING
       :EndTrap
   :EndIf
∇

Notes:

Since this function will signal an error Constants.APP_STATUS.ALREADY_RUNNING we need to add this to the EXIT namespace in MyApp:

:Namespace EXIT
...
        UNABLE_TO_WRITE_TARGET←114
        ALREADY_RUNNING←115
          GetName←{
        ....
:EndNamespace

We change Initial so that it calls this new function:

∇ (Config MyLogger)←Initial dummy
...
   Config←CreateConfig ⍬
   Config.ControlFileTieNo←CheckForOtherInstances ⍬
   CheckForRide Config.(Ride WaitForRide)
...
∇

We want to untie the file as well. So far we have not paid any attention to how to close the application down properly; now we take this opportunity to introduce a function Cleanup which does that:

∇ {r}←Cleanup
   r←⍬
   ⎕FUNTIE Config.ControlFileTieNo
   Config.ControlFileTieNo←⍬
∇

:EndNamespace

Of course we have to call Cleanup from somewhere:

∇ {r}←StartFromCmdLine arg;MyLogger;Config;rc;⎕TRAP
 ...
   rc←TxtToCsv arg~''''
   Cleanup
   Off rc
∇

After all these changes it’s time to execute our test cases. Execute #.Tests.Run.

Turns out that two of them fail! The reason: when we run Test_exe_01 and Test_exe_02 the control file is already tied. That’s because Test_TxtToCsv runs first, and it calls Initial – which ties the control file – but not Cleanup, which would untie it. The fix is simple: we need to call Cleanup in the test. However, we can’t just do this at the end of Test_TxtToCsv_01:

∇ R←Test_TxtToCsv_01(stopFlag batchFlag);⎕TRAP;rc
...
   →FailsIf rc≢##.MyApp.EXIT.SOURCE_NOT_FOUND
   #.MyApp.Cleanup ⍬
   R←∆OK
∇

If we do this then Cleanup would not be called if the check fails. Let’s do it properly instead:

∇ R←Test_TxtToCsv_01(stopFlag batchFlag);⎕TRAP;rc
...
   →GoToTidyUp rc≢##.MyApp.EXIT.SOURCE_NOT_FOUND
   R←∆OK
  ∆TidyUp:
   ##.MyApp.Cleanup ⍬
Information

Note that we must call MyApp.Cleanup rather than just Cleanup because we are at that moment in Tests – and we don’t want to execute Tests.Cleanup!

We can learn some lessons from the failure of those two test cases:

  1. The sequence in which the tests are executed can have an impact on whether tests fail or not. If Test_TxtToCsv had been the last test case, the problem would have slipped through undetected.
  2. That a test suite runs through OK does not necessarily mean it will keep doing so when you execute it again. If Test_TxtToCsv had been the very last test case the test suite would have passed without a problem but an attempt to run it again would fail because now the control file would have been tied, and Test_exe_01 would have failed.

In our specific case it was actually a problem in the test cases, not in MyApp, but the conclusion stands anyway.

Shuffle test cases

At the time of writing (2017-07) the sequence of the test cases relies on alphabetical order and is therefore predictable. On the to-do list for the Tester class is a topic: Add an option that makes the test framework shuffle the test cases so that the order of execution is not predictable anymore.

6. Create a Scheduled Task

6.1. Start the Scheduler

Press the Win key and type Scheduler. Select Task Scheduler from the list. You will see:

The Windows Task Scheduler

First check whether the fifth point in the Actions pane on the right reads Disable All Tasks History. If it does not you won’t get any details about a Scheduled Task.

The arrow points to the Create Task command – click it.

Create Task

6.1.1. The General tab

Name
Used in the list presented by the Task Scheduler.
Description
Shown in the list presented by the Task Scheduler. Keep it concise.
Run only when the user is logged on
You will almost certainly change this to Run whether the user is logged on or not.
Do not store a password
The password is stored safely, so there is no reason not to provide it.
Running with highest privileges

Unfortunately this check box is offered whether your user account has admin rights or not. If it does not, then ticking the box won’t have any effect.

If your user account has no admin rights but your Scheduled Task needs to run with highest privileges then you need to specify a different user id/password after clicking the Change user or group button.

Whether your application needs to run with highest privileges or not is impossible to say. Experience shows that sometimes a process that fails when – and only when – the application runs as a Scheduled Task will work fine if run as a Schedule Task with highest privileges, although it is by no means clear what those rights are required for.

Configure for
Generally you should select the OS the task is running on.

UAC, admin rights and all the rest

With UAC (User Account Control), users of the admin group have 2 tokens. The filtered token represents standard user rights.

This token is used by default, for example when you create a shell (a console). Therefore you have just standard user rights by default even when using a user account with admin rights. However, when you have admin rights and you click an EXE and select Run as administrator, the full token is used, which contains admin rights.

Notes:

6.1.2. The Trigger tab

The tab holds no mysteries.

6.1.3. The Action tab

After clicking New this is what you get:

New Action

Make sure you use the Browse button to navigate to the EXE/BAT/whatever you want to run as a Scheduled Task. This avoids typos.

Add arguments
allows you to specify something like MAXWS=345M or the name of a workspace in case Program is not an EXE but a Dyalog interpreter. In particular, you should add DYALOG_NOPOPUPS=1. This prevents any dialogs from popping up (aplcore, WS FULL, etc.). You don’t want them when Dyalog is running in the background because there's nobody around to click the OK button…
Start in

is useful for specifying what will become the current (or working) directory for the running program. We recommend setting the current directory in your code as early as possible, so you don’t really need to set this here except that when you don’t you might well get an error code 2147942512.

We will discuss later how such error codes can be interpreted, but for the time being, trust us that it actually means Not enough space available on the disk! When you do specify the Start in parameter it runs just fine.

However, you must not delimit the path with double-quotes. It’s understandable that Microsoft does not require them in this context because by definition any blanks are part of the path, but why they do not just ignore them when specified is less understandable.

6.1.4. The Conditions tab

The tab holds no mysteries.

6.1.5. The Settings tab

Unless you have a very good reason not to, Allow task to be run on demand, which means you have the Run command available on the context menu.

Note you may specify restart parameters in case the task fails. Whether that makes any sense depends on the application.

The combo box at the bottom allows you to select Stop the existing instance, which can be quite useful when debugging the application.

6.2. Running a Scheduled Task

To start the task, right-click on it in the Task Scheduler and select Run from the context menu. Then check the log file. We have tested the application well, we know it works, so you should see a log file that contains something like this:

2017-03-31 10:03:35 *** Log File opened
2017-03-31 10:03:35 (0) Started MyApp in ...\code\v12\MyApp
2017-03-31 10:03:35 (0)  ...\code\v12\MyApp\MyApp.exe MAXWS=370M
2017-03-31 10:03:35 (0)  Accents            ÁÂÃÀÄÅÇÐÈÊËÉÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝ  AAAAAA...
2017-03-31 10:03:35 (0)  ControlFileTieNo   1
2017-03-31 10:03:35 (0)  Debug              0
2017-03-31 10:03:35 (0)  DumpFolder         C:\Users\kai\AppData\Local\MyApp\Errors
2017-03-31 10:03:35 (0)  ForceError         0
2017-03-31 10:03:35 (0)  LogFolder          C:\Users\kai\AppData\Local\MyApp\Log
2017-03-31 10:03:35 (0)  Ride               0
2017-03-31 10:03:35 (0)  Trap               1
2017-03-31 10:03:35 (0) Source: MAXWS=370M
2017-03-31 10:03:35 (0) *** ERROR RC=112; MyApp is unexpectedly shutting down: SOURCE_NOT_FOUND

Since we have not provided a filename, MyApp assumed that MAXWS=370M would be the filename. Since that does not exist the application quits with a return code SOURCE_NOT_FOUND, which is exactly proper.

However, from experience we know the likelihood of the task not running as intended is high. We have already discussed some of the issues that might pop up, and we will now discuss some more we have enjoyed over the years.

7. Tips, tricks, pratfalls

7.1. Riding into a Scheduled Task

Suppose you want to Ride into a Scheduled Task. So in the INI you set the [Ride]Active flag to 1. While the Windows Firewall has still no rules for both this port and this application you won’t see the usual message when you run the application for the very first time, assuming that you use a user ID with admin rights:

Windows Firewall

The application would start, seemingly run for a short period of time and then stop again without leaving any traces: no error codes, no log files, no crash files, nothing.

It is different when you simply double-click the MyApp.exe: in that case the Security Alert dialog box pops up, giving you an easy way to create a rule that allows the application to communicate via the given port.

By the way, when you click Cancel in the Security Alert dialog box, you might expect the Windows Firewall to deny access to the port but not create a rule either. You would be mistaken. The two buttons Allow access and Cancel shouldn’t be buttons at all! Instead there should be a group Create rule with two radio buttons: Allow access and Deny access.

If the user clicks the “Cancel” button a message should pop up saying that although no rule will be created, access to the port in question is denied. That would imply that when the application is started again the Security Alert dialog box will reappear. Instead, when Cancel is clicked a blocking rule for that combination of application and port number is created, and you will not see that dialog box again for this combination.

7.2. The Task Scheduler GUI

Once you have executed the Run command from the context menu the GUI changes the status from Ready to Running. That’s fine. Unfortunately it doesn’t change automatically back to Ready once the job has finished, at least not at the time of writing (2017-03) under Windows 10. For that you have to refresh with F5.

7.3. Re-make MyApp

When you’ve found a bug and run MyApp’s Make.bat again, remember that this will overwrite the INI. So if you’ve changed, say, Ride’s Active flag in the INI from 0 to 1, it will be 0 again after the call to Make.bat, and any attempt to Ride into the EXE will fail. Easy to overlook!

7.4. MyApp crashes with return code 32

If you see this you most probably forgot to copy over the DLLs needed by Ride [1] itself. That’s what triggers the return code 32 which means File not found.

Windows return codes

To translate a Windows return code like 32 into a more meaningful piece of information, download and install the user command GetMsg from the APL wiki. Then you can do this:

      ]GetMsgFrom 32
The process cannot access the file because it is being used by another process.

Sadly, the error messages are not always that helpful. The above message appears if you try to switch on Ride in an application, and the interpreter cannot find the DLLs that Ride needs.

7.5. Binding MyAPP with the Dyalog development EXE

If for some reason you’ve created MyApp.exe by binding the application to the development version of Dyalog rather than the runtime (you can do this by providing a 0 as left argument to the MakeExport function) then you might run into a problem. Our code notices whether it is running under a development EXE or a runtime EXE. Error trapping will be inactive (unless it is enforced via the INI) and ⎕OFF won’t be executed; instead it would execute and hang around but without you being able to see the session.

So don’t do this. Because you have Ride at your disposal the development version has no advantages over the runtime EXE anyway.

7.6. Your application doesn’t do what it’s supposed to do

… but only when running as a task. Start the Task Scheduler and go to the History tab; if this is empty then you have not clicked Enable all tasks history as suggested earlier.

Don’t get fooled by Action completed and Task completed – they don’t tell you whether a task failed or not. Click Action completed: at the bottom you get information regarding that run. You might read something like:

Task Scheduler successfully completed task "\MyApp" , instance "{c7cb733a-be97-4988-afca-a551a7907918}" , action "...\code\v12\MyApp\MyApp.exe" with return code 2147942512.

That tells you task did not run at all. You won’t find either a log file or a crash file, and you cannot Ride into the application.

7.7. Task Scheduler error codes

If the Task Scheduler itself throws an error you will find the error codes of little value – at first sight.

You can provoke such an error quite easily: edit the task we’ve created and changed the contents of the Program/script field in the Edit action dialog to something that does not exist, so the Task Scheduler won’t find such a program to run. Then issue the Run command from the context menu.

Update the GUI by pressing F5 and you will see that errors are reported. The row that reads Task Start Failed in the Task Category columns and Launch Failure in the Operational Code columns is the one we are interested in. When you click on this row you will find that it reports an Error Value 2147942402. What exactly does this mean?

One way to find out is to google for 2147942402. For this particular error this will certainly do, but sometimes you will have to go through plenty of pages when people managed to produce the same error code in very different circumstances, and it can be quite time-consuming to find a page that carries useful information for your circumstances.

Instead we use the user command [3] Int2Hex, based on code written and contributed by Phil Last [2]. With this user command we can convert the value 2147942402 into a hex value:

      ]Int2Hex 2147942402
80070002

Third-party user commands

There are many useful third-party user commands available. For details how to install them see “Appendix 2 — User commands”.

The first four digits, 8007, mean that what follows is a Win32 status code. The last 4 are the status code. This is a number that needs to be converted into decimal:

      ]Hex2Int 0002

but in our case that is of course not necessary because the number is so small that there is no difference between hex and integer anyway, so we can convert it into an error message straight away.

Again we use a user command that is not part of a standard Dyalog installation but because it is so useful we strongly recommend installing it [4]. It translates a Windows error code into meaningful text.

      ]GetMsgFrom
The system cannot find the file specified.

And that’s why it failed.

8. Creating tasks programmatically

It is possible to create Scheduled Tasks by a program, although this is beyond the scope of this book. See

https://msdn.microsoft.com/en-us/library/windows/desktop/bb736357(v=vs.85).aspx

for details.


Footnotes

  1. This topic was discussed in the chapter “Debugging a stand-alone EXE

  2. http://aplwiki.com/PhilLast

  3. For details and download regarding the user commands Hex2Int and Int2Hex see http://aplwiki.com/UserCommands/Hex

  4. For details and download regarding the user command GetMsgFrom see http://aplwiki.com/UserCommands/GetMsgFrom

Chapter 13:

Windows Services

1. What is a Windows Service

While the Windows Task Manager just starts any ordinary application, any application that runs as a Windows Service must be specifically designed to meet certain requirements.

In particular, services communicate by exchanging messages with the Windows Service Control Manager (SCM).

Commands can be issued by the SC.exe (Service Controller) application or interactively via the Services application. This allows the user to start, pause, continue (resume) and stop a Windows Service.

2. Windows Services and the Windows Event Log

Our application is already prepared to write log files and save information in the event of a crash, but that's not sufficient: while applications started by the Windows Task Scheduler might write to the Windows Event Log, applications running as a Windows Service are expected to do that, and for good reasons: when running on a server one cannot expect anybody to be keeping an eye on log or crash files.

In large organisations running server farms it is common to have software in place that scans the Windows Event Logs of all the servers, and raises an alarm (TCP messages, text messages, emails, whatever) if it finds problems.

We won't add the ability to write to the Windows Event Log in this chapter, but rather discuss how to do this in the next chapter.

3. Restrictions

With Dyalog version 16.0 we cannot install a stand-alone EXE as a Windows Service. All we can do is to install a given interpreter and then ask it to load a workspace, which implies running ⎕LX. (This restriction may be lifted in a future version of Dyalog.)

That means that unless you're happy to expose your code, you have a problem. There are solutions:

All three solutions add a level of complexity just to hide the code, but at least there are several escape routes available.

We resume, as usual, by saving a copy of Z:\code\v12 as Z:\code\v13.

4. The ServiceState namespace

To simplify matters we shall use the ServiceState namespace, also from the APLTree project. It requires you to do just two things:

  1. Call ServiceState.Init as early as possible. This function will ensure the Service can communicate with the SCM.

    Do it as early as possible to enure any request from the SCM will be answered promptly. Windows is not exactly patient when it waits for a Service to respond to a Pause, Resume or Stop request: after only 5 seconds you may see an error message complaining that the Service refused to cooperate.

    However, note that the interpreter confirms the Start request for us; no further action is required.

    Create a parameter space by calling CreateParmSpace, and set at least the name of the log function and possibly the namespace (or class instance) the log function is in. This log function is used to log any incoming requests from the SCM. The parameter space is then passed to Init as the right argument.

  2. In its main loop the Service is expected to call ServiceState.CheckServiceMessages.

    This is an operator, so it needs a function as operand: that is, a function that is doing the logging, allowing CheckServiceMessages to log its actions to the log file. (If you don't need a log file then simply passing {⍵} will do.

    When CheckServiceMessages is called if no request of the SCM is pending it will quit straight away and return a 0. If a Pause is pending then it goes into a loop, and continues to loop (with a ⎕DL in between) until either Continue (a.k.a. Resume) or Stop is requested by the SCM.

    If a Stop is requested, the operator will quit and return a 1.

5. Installing and uninstalling a Service.

Note: for installing or uninstalling a Service you need admin rights.

Suppose you have loaded a WS MyService which you want to install as a Windows Service run by the same version of Dyalog you are currently in:

aplexe←'"',(2 ⎕NQ # 'GetEnvironment' 'dyalog'),'\dyalogrt.exe"'
wsid←'"whereEverTheWsLives\MyAppService.DWS"'
cmd←aplexe,' ',wsid,' APL_ServiceInstall=MyAppService DYALOG_NOPOPUPS=1'

cmd could now be executed with the Execute class introduced in the chapter “Handling Errors”. That would do the trick.

DYALOG_NOPOPUPS

Setting this to 1 prevents any dialogs popping up (aplcore, WS FULL etc.). You don't want them when Dyalog is running in the background because there's nobody around to click the OK button.

This also prevents the Service MyAppService successfully installed message from popping up. You don't want to see that when executing tests that install, start, pause, resume, stop and uninstall a Service.

To uninstall the Service, simply open a console window with Run as administrator and enter:

sc delete MyAppService

and you are done.

Pitfalls when installing or uninstalling Windows Services

When you have opened the Services GUI while installing or uninstalling a Windows Service you must press F5 on the GUI to refresh the display.

The problem is not just that the GUI does not update itself – annoying as that may be. You might end up with a Service marked in the GUI as disabled, aftrer which you need to reboot the machine.

This can happen if you try to perform an action from the GUI when it is out of sync with the Service's current state.

SC: Service Control

SC is a command line program that allows a user with admin rights to issue particular commands regarding Windows Services. The general format is:

SC.exe {command} {serviceName}

Commonly used commands are:

6. Obstacles

From experience we know there are pitfalls. In general, there are kinds of problem you might encounter:

  1. The Service appears to start (shows running in the Services GUI) but nothing happens at all.
  2. The Service starts, the log file confirms it, but once you request the Service pauses or stops you get nothing but a Windows error message.
  3. The Service runs, but the application does not do anything, or it does something unexpected.

6.1. The Service seems to be doing nothing at all

If a Service does not seem to do anything when started:

CONTINUE workspaces

The Service might have created a CONTINUE workspace, for various reasons.

Note that, starting with version 16.0, Dyalog no longer drops a CONTINUE workspace by default. You must configure Dyalog to do so.

A CONTINUE cannot be saved if there is more than one thread running – and Services are by definition multi-threaded. However, if it fails very early there might still be a CONTINUE.

Once a second thread is started, Dyalog is no longer able to save a CONTINUE workspace. Establishing error trapping before a second thread is started avoids this problem.

aplcores

Writing to the directory the service is installed in might be prohibited by Windows. That might well prevent a CONTINUE or aplcore from being saved.

While you cannot choose the folder in which a CONTINUE would be saved you can define a folder for aplcores:

`APLCORENAME='/pathToFolder/my_aplcore*`.

This saves any aplcore as “my_aplcore” followed by a running number. For more information regarding aplcores see “Appendix 3 — aplcores and WS integrity”.

6.2. The Service starts but ignores Pause and Stop requests

Here the log file contains everything we expect: calling parameters etc. In such a case we know the Service started and is running.

If these two conditions are met then it's hard to imagine what could prevent the application from reacting to any requests of the SCM, except when you have an endless loop somewhere in your application.

6.3. The application does not do what's supposed to do

The very thought of you And I forget to do Those ordinary things That everyone ought to do

Cole Porter

First and foremost it is worth repeating that any application supposed to run as a Service should be developed as an ordinary application, including test cases. When it passes such test cases you have reasons to be confident the application will run as a Service too.

Having said this, there can be surprising differences between running as an ordinary application and a Service. For example, when a Service runs, not with a user's account, but with the system account (which is quite normal to do) any call to #.FilesAndDirs.GetTempPath results in

"C:\Windows\System32\config\systemprofile\AppData\Local\Apps"

while for a user's account it would be something like

'C:\Users\{username}\AppData\Local\Temp'.

When the application behaves in an unexpected way you need to debug it, and for that Ride is invaluable.

7. Potions and wands

7.1. Ride

First of all we have to make sure that the application will give us a Ride if we need one. Since passing any arguments for a Ride via the command line requires the Service to be uninstalled and installed at least twice we recommend preparing the Service from within the application instead.

If you have trouble Riding into any kind of application, check there is not an old instance of the Service running and occupying the port you need for the Ride.

There are two very different scenarios in which you might want to use Ride:

  1. The Service does not seem to start or does not respond to Pause or Stop requests.
  2. The Service starts fine and responds properly to Pause or Stop requests, but the application is otherwise behaving unexpectedly.

In the former case ensure as soon as possible after startup you allow the programmer to Ride into the Service. The first line of the function in ⎕LX should provide a Ride.

At such an early stage we don't have an INI file instantiated, so we cannot switch Ride on and off via the INI file, we have to modify the code for that.

You might feel tempted to overcome this by doing it a bit later (e.g. after processing the INI file) but beware: if a Service is not cooperating then “a bit later” might be too late to get investigate the problem. Resist temptation.

In the second case, add the call to CheckForRide once the INI file has been instantiated.

Information

Make sure that you have never more than one of the two calls to CheckForRide active. If both are active you would be able to use the first one, but the second one would throw you out!

8. Logging

8.1. Local logging

We want to log as soon as possible any command-line parameters, as well as any message exchange between the Service and the SCM.

Again we advise you not to wait until the folder holding the log files is defined by instantiating the INI file. Instead we suggest making the assumption that a certain folder (“Logs”) will (or might) exist in the current directory which will become where the workspace was loaded from.

If that's not convenient, consider passing the directory that will contain the Logs folder as a command line parameter.

8.1.1. Windows event log

In the next chapter (“The Windows Event Log”) we will discuss why and how to use the Windows Event Log, in particular when it comes to Services.

8.2. How to implement it

8.2.1. Setting the latent expression

First of all we need to point out that MyApp as it stands is hardly a candidate for a Service. We need to make something up. Let's specify one or more folders to be watched by the MyApp Service.

If any files are found then they are processed. Finally the app will store hashes for all the files it has processed. That allows it to recognize any added, changed or removed files efficiently.

For the Service we need to create a workspace that can be loaded by that Service. Therefore we need to set ⎕LX, and for that we create a new function:

 ∇ {r}←SetLXForService(earlyRide ridePort)
   ⍝ Set Latent Expression (needed in order to export workspace as EXE)
   ⍝ `earlyRide` is a flag. 1 allows a Ride.
   ⍝ `ridePort`  is the port number to be used for a Ride.
      #.⎕IO←1 ⋄ #.⎕ML←1 ⋄ #.⎕WX←3 ⋄ #.⎕PP←15 ⋄ #.⎕DIV←1
      r←⍬
      ⎕LX←'#.MyApp.RunAsService ',(⍕earlyRide),' ',(⍕ridePort)
 ∇

The function takes a flag earlyRide and an integer ridePort as arguments. How and when this function is called will be discussed in a moment.

Because we have now two functions that set ⎕LX we shall rename the original one (SetLX) to SetLXForApplication to tell them apart.

8.2.2. Initialising the Service

Next we need the main function for the service:

 ∇ {r}←RunAsService(earlyRide ridePort);⎕TRAP;MyLogger;Config;∆FileHashes
    ⍝ Main function when app is running as a Windows Service.
    ⍝ `earlyRide`: flag that allows a very early Ride.
    ⍝ `ridePort`: Port number used by Ride.
      r←⍬
      CheckForRide earlyRide ridePort
      #.FilesAndDirs.PolishCurrentDir
      ⎕TRAP←#.HandleError.SetTrap ⍬
      (Config MyLogger)←Initial #.ServiceState.IsRunningAsService
      ⎕TRAP←(Config.Debug=0)SetTrap Config
      Config.ControlFileTieNo←CheckForOtherInstances ⍬
      ∆FileHashes←0 2⍴''
      :If #.ServiceState.IsRunningAsService
          {MainLoop ⍵}&⍬
          ⎕DQ'.'
      :Else
          MainLoop ⍬
      :EndIf
      Cleanup ⍬
      Off EXIT.OK
    ∇

Notes:

8.2.3. The “business logic”

Time to change the MyApp.Initial function:

leanpub-start-insert
 ∇ (Config MyLogger)←Initial isService;parms
    ⍝ Prepares the application.
      Config←CreateConfig isService
      Config.ControlFileTieNo←CheckForOtherInstances ⍬
      CheckForRide (0≠Config.Ride) Config.Ride
      MyLogger←OpenLogFile Config.LogFolder
      MyLogger.Log'Started MyApp in ',F.PWD
      MyLogger.Log 2 ⎕NQ'#' 'GetCommandLine'
      MyLogger.Log↓⎕FMT Config.∆List
      :If isService
          parms←#.ServiceState.CreateParmSpace
          parms.logFunction←'Log'
          parms.logFunctionParent←MyLogger
          #.ServiceState.Init parms
      :EndIf

Note that we pass isService as right argument to CreateConfig, so we must amend CreateConfig accordingly:

leanpub-start-insert
 ∇ Config←CreateConfig isService;myIni;iniFilename
   Config←⎕NS''
   ...
   Config.IsService←isService
   ...
       Config.Accents←⊃Config.Accents myIni.Get'Config:Accents'
        :If isService
            Config.WatchFolders←⊃myIni.Get'Folders:Watch'
        :Else
            Config.LogFolder←'expand'F.NormalizePath⊃Config.LogFolder myIni.Get'Folders:Logs'
        :EndIf
        Config.DumpFolder←'expand'F.NormalizePath⊃Config.DumpFolder myIni.Get'Folders:Errors'
   ...
 ∇

Note that WatchFolder is introduced only when the application is running as a Service.

Time to introduce the function MainLoop:

∇ {r}←MainLoop port;S
  r←⍬
  MyLogger.Log'"MyApp" server started'
  S←#.ServiceState
  :Repeat
      LoopOverFolder ⍬
      :If (MyLogger.Log S.CheckServiceMessages)S.IsRunningAsService
          MyLogger.Log'"MyApp" is about to shut down...'
          :Leave
      :EndIf
      ⎕DL 2
  :Until 0
 ⍝Done
∇

Notes:

The function LoopOverFolder:

∇ {r}←LoopOverFolder dummy;folder;files;hashes;noOf;rc
  r←⍬
  :For folder :In Config.WatchFolders
      files←#.FilesAndDirs.ListFiles folder,'\*.txt'
      hashes←GetHash¨files
      (files hashes)←(~hashes∊∆FileHashes[;2])∘/¨files hashes
      :If 0<noOf←LoopOverFiles files hashes
          :If EXIT.OK=rc←TxtToCsv folder
              MyLogger.Log'Totals.csv updated'
          :Else
              LogError rc('Could not update Totals.csv, RC=',EXIT.GetName rc)
          :EndIf
      :EndIf
  :EndFor
∇

This function calls GetHash so we had better introduce it:

 GetHash←{
 ⍝ Get hash for file ⍵
     ⊣2 ⎕NQ'#' 'GetBuildID'⍵
 }

The function LoopOverFiles:

 ∇ noOf←LoopOverFiles(files hashes);file;hash;rc
   noOf←0
   :For file hash :InEach files hashes
       :If EXIT.OK=rc←TxtToCsv file
           ∆FileHashes⍪←file hash
           noOf+←1
       :EndIf
   :EndFor
 ∇

This function finally calls TxtToCsv.

Because of the change we've made to the right argument of Initial we need to change StartFromCmdLine. Here the function Initial needs to be told it is not running as a Service:

∇ {r}←StartFromCmdLine arg;MyLogger;Config;rc;⎕TRAP
...
   (Config MyLogger)←Initial #.ServiceState.IsRunningAsService
...

Two more changes:

 ∇ {r}←Cleanup dummy
   r←⍬
   ⎕FUNTIE Config.ControlFileTieNo
   Config.ControlFileTieNo←⍬
   '#'⎕WS'Event' 'ServiceNotification' 0

This disconnects the handler from the ServiceNotification event.

Finally we redefine which functions are public:

 ∇ r←PublicFns
   r←'StartFromCmdLine' 'TxtToCsv' 'SetLXForApplication' 'SetLXForService' 'RunAsService'

8.2.4. Running the test cases

Now it's time to see if we broke anything. Double-click MyApp.dyapp and type y to answer the question whether you would like to run all test cases. If anything fails, execute #.Tests.RunDebug 0 and fix the problem.

9. Installing and un-installing the Service

In order to install or uninstall the Service we need two BATs: InstallService.bat and Uninstall_Service.bat. We will write these BATs from Dyalog. For that we write a class ServiceHelpers:

:Class ServiceHelpers

    ∇ {r}←CreateBatFiles dummy;path;cmd;aplexe;wsid
      :Access Public Shared
    ⍝ Write two BAT files to the current directory:
    ⍝ Install_Service.bat and Uninstall_Service.bat
      r←⍬
      path←#.FilesAndDirs.PWD

      aplexe←'"',(2 ⎕NQ'#' 'GetEnvironment' 'dyalog'),'\dyalogrt.exe"'
      wsid←'"%~dp0\MyAppService.DWS"'
      cmd←aplexe,' ',wsid,' APL_ServiceInstall=MyAppService'
     ⍝cmd,←' APLCORENAME={foldername}'
     ⍝cmd,←' DYALOG_EVENTLOGNAME={foo}'
      cmd,←' DYALOG_NOPOPUPS=1'
      cmd,←' MAXWS=64MB'
      #.APLTreeUtils.WriteUtf8File(path,'\Install_Service.bat')cmd

      cmd←⊂'sc delete MyAppService'
      cmd,←⊂'@echo off'
      cmd,←⊂'    echo Error %errorlevel%'
      cmd,←⊂'    if NOT ["%errorlevel%"]==["0"] ('
      cmd,←⊂'    pause'
      cmd,←⊂'exit /b %errorlevel%'
      cmd,←⊂')'
      #.APLTreeUtils.WriteUtf8File(path,'\Uninstall_Service.bat')cmd
     ⍝Done
    ∇

:EndClass

Notes:

10. “Make” for the Service

Now it's time to create a DYAPP for the service. For that copy Make.dyapp as MakeService.dyapp and then edit it:

Target #
Load ..\AplTree\APLTreeUtils
Load ..\AplTree\FilesAndDirs
Load ..\AplTree\HandleError
Load ..\AplTree\IniFiles
Load ..\AplTree\OS
Load ..\AplTree\Logger
Load ..\AplTree\EventCodes
Load Constants
Load Utilities
Load MyApp

Load ..\AplTree\ServiceState
Load ..\AplTree\Tester
Load ..\AplTree\Execute
Load ..\AplTree\WinSys
Load TestsForServices
Load ServiceHelpers

Run #.ServiceHelpers.CreateBatFiles ⍬
Run '#.⎕EX''ServiceHelpers'''
Run #.MyApp.SetLXForService 0 4599   ⍝ [1|0]: Ride/no Ride, [n] Ride port number

Load MakeService
Run #.MakeService.Run 0

Notes:

That obviously requires the class MakeService:

:Class MakeService
⍝ Creates a workspace "MyAppService" which can then run as a service.
⍝ 1. Re-create folder DESTINATION in the current directory
⍝ 2. Copy the INI file template over to DESTINATION\ as MyApp.ini
⍝ 3. Save the workspace within DESTINATION
    ⎕IO←1 ⋄ ⎕ML←1
    DESTINATION←'MyAppService'

    ∇ {r}←Run offFlag;en;F;U
      :Access Public Shared
      r←⍬
      (F U)←##.(FilesAndDirs Utilities)
      (rc en more)←F.RmDir DESTINATION
      U.Assert 0=rc
      U.Assert 'Create!'##.FilesAndDirs.CheckPath DESTINATION
      'MyApp.ini.template' CopyTo DESTINATION,'\MyApp.ini'
      'Install_Service.bat' CopyTo DESTINATION,'\'
      'Uninstall_Service.bat' CopyTo DESTINATION,'\'
      ⎕WSID←DESTINATION,'\',DESTINATION
      #.⎕EX⍕⎕THIS
      0 ⎕SAVE ⎕WSID
      {⎕OFF}⍣(⊃offFlag)⊣⍬
    ∇

    ∇ {r}←from CopyTo to;rc;more;msg
      r←⍬
      (rc more)←from F.CopyTo to
      msg←'Copy failed RC=' ,(⍕rc),'; ',more
      msg ⎕signal 11/⍨0≠rc
    ∇
:EndClass

Notes:

Self-deleting code

How it is possible for the function MakeService.Run to delete itself and keep running anyway?

APL code (functions, operators and scripts) is copied onto the stack for execution. You can investigate the stack at any given moment with )SI and )SINL; for details type the command in question into the session and then press F1.

Even if the code of a class executes ⎕EX ⍕⎕THIS or a function or operator ⎕EX ⊃⎕SI the code keeps running because the copy on the stack will exist until the function or operator quits. Scripts might even live longer: only when the last reference pointing to a script is deleted does the script cease to exist.

11. Testing the Service

We have test cases that tell us the “business logig” of MyApp works just fine. What we also need are tests that it runs fine as a Service as well.

Since the two test scenarios are only loosely related we want to keep those tests separate. It is easy to see a way: testing the Service means assembling all the needed stuff, installing the Service, carrying out the tests and finally un-installing the tests and cleaning up.

We don't want to execute all this unless we really have to.

We start by creating a new script TestsForServices which we save alongside the other scrips in v13/:

:Namespace TestsForServices
⍝ Installs a service "MyAppService" in a folder within the Windows Temp directory with
⍝ a randomly chosen name. The tests then start, pause, continue and stop the service.\\
⍝ They also check whether the application produces the expected results.

    ⎕IO←1 ⋄ ⎕ML←1

:EndNamespace

We now discuss the functions we are going to add one after the other. Note that the Initial function is particularly important in this scenario: we need to copy over all the stuff we need, code as well as input files, make adjustments, and install the Service.

This could all be done in a single function but it would be lengthy and difficult to read. To avoid this we split the function into obvious units. By naming those functions carefully we should get away without adding any comments because the code explains itself. Here we go:

∇ r←Initial;rc;ini;row;bat;more
   ∆Path←##.FilesAndDirs.GetTempFilename''
   #.FilesAndDirs.DeleteFile ∆Path
   ∆Path←¯4↓∆Path
   ∆ServiceName←'MyAppService'
   r←0
   :If 0=#.WinSys.IsRunningAsAdmin
       ⎕←'Sorry, but you need admin rights to run this test suite!'
       :Return
   :EndIf
   ∆CreateFolderStructure ⍬
   ∆CopyFiles ⍬
   ∆CreateBATs ⍬
   ∆CreateIniFile ⍬
   ∆InstallService ⍬
   ⎕←'*** Service ',∆ServiceName,' successfully installed'
   r←1

Note that all the sub-function and global variables start their names with . An example is the function ∆Execute_SC_Cmd:

∇ {(rc msg)}←∆Execute_SC_Cmd command;cmd;buff
 ⍝ Executes a SC (Service Control) command
   rc←1 ⋄ msg←'Could not execute the command'
   cmd←'SC ',command,' ',∆ServiceName
   buff←#.Execute.Process cmd
   →FailsIf 0≠1⊃buff
   msg←⊃,/2⊃buff
   rc←3⊃buff
∇

It executes SC commands like start, pause, continue, stop and query by preparing a string and then passing it to Execute.Process.

It analyzes the result and returns the text part of it as well as a return code. While the first four commands aim to change the current status of a Service, query is designed to establish what the current status of a Service actually is.

After having executed the test suite we want to clean up, so we create a function Cleanup.

Remember: if the test framework finds a function Initial, it executes it before executing the actual test cases, while any function Cleanup will be executed after the test cases have been executed.

∇ {r}←Cleanup
   r←⍬
   :If 0<⎕NC'∆ServiceName'
       ∆Execute_SC_Cmd'stop'
       ∆Execute_SC_Cmd'delete'
       ##.FilesAndDirs.RmDir ∆Path
       ⎕EX¨'∆Path' '∆ServiceName'
   :EndIf
∇

We also need ∆Pause:

∇ {r}←∆Pause seconds
   r←⍬
   ⎕←'   Pausing for ',(⍕seconds),' seconds...'
   ⎕DL seconds
∇

We could discuss all the sub functions called by these two functions but it would tell us little. Therefore we suggest you copy the code from the web site. We discuss here just the two test functions:

∇ R←Test_01(stopFlag batchFlag);⎕TRAP;rc;more
  ⍝ Start, pause, continue and stop the service.
  ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
  R←∆Failed

  (rc more)←∆Execute_SC_Cmd'start'
  →FailsIf 0≠rc
  ∆Pause 2
  (rc more)←∆Execute_SC_Cmd'query'
  →FailsIf 0≠rc
  →FailsIf 0=∨/'STATE : 4 RUNNING'⍷#.APLTreeUtils.dmb more

  (rc more)←∆Execute_SC_Cmd'pause'
  →FailsIf 0≠rc
  ∆Pause 2
  →FailsIf 1≠⍴#.FilesAndDirs.ListFiles ∆Path,'\service\Logs\'
  (rc more)←∆Execute_SC_Cmd'query'
  →FailsIf 0=∨/'STATE : 7 PAUSED'⍷#.APLTreeUtils.dmb more

  (rc more)←∆Execute_SC_Cmd'continue'
  →FailsIf 0≠rc
  ∆Pause 2
  (rc more)←∆Execute_SC_Cmd'query'
  →FailsIf 0=∨/'STATE : 4 RUNNING'⍷#.APLTreeUtils.dmb more

  (rc more)←∆Execute_SC_Cmd'stop'
  →FailsIf 0≠rc
  ∆Pause 2
  (rc more)←∆Execute_SC_Cmd'query'○
  →FailsIf 0=∨/'STATE : 1 STOPPED'⍷#.APLTreeUtils.dmb more

  R←∆OK
∇

In order to understand the →FailsIf statements it is essential to have a look at a typical result returned by the ∆Execute_SC_Cmd function, in this case a query:

      ⍴more
328
      ≡more
1
      #.APLTreeUtils.dmb more
SERVICE_NAME: MyAppService TYPE : 10 WIN32_OWN_PROCESS STATE : 4 RUNNING (STOPPABLE, PAUSABLE, ACCEPTS_SHUTDOWN) WIN32_EXIT_CODE : 0 (0x0) SERVICE_EXIT_CODE
       : 0 (0x0) CHECKPOINT

We have removed white space here to increase readability. (The result is richly endowed with them.)

This test starts, pauses, continues and finally stops the Service after having processed some files:

∇ R←Test_02(stopFlag batchFlag);⎕TRAP;rc;more;noOfCSVs;success;oldTotal;newTotal;A;F
  ⍝ Start service, check results, give it some more work to do, check and stop it.
   ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
   R←∆Failed
   (A F)←#.(APLTreeUtils FilesAndDirs)

   (rc more)←∆Execute_SC_Cmd'start'
   →FailsIf 0≠rc
   ∆Pause 1
   (rc more)←∆Execute_SC_Cmd'query'
   →FailsIf 0=∨/'STATE : 4 RUNNING'⍷A.dmb more

   ⍝ At this point the service will have processed all the text files, so there
   ⍝ must now be some CSV files, including the Total.csv file.
   ⍝ We then copy 6 more text files, so we should see 6 more CSVs & a changed Total.
   oldTotal←↑{','A.Split ⍵}¨A.ReadUtf8File ∆Path,'\input\en\total.csv'
   noOfCSVs←⍴F.ListFiles ∆Path,'\input\en\*.csv'
   (success more list)←(∆Path,'\texts')F.CopyTree ∆Path,'\input\'  ⍝ All of them
   {1≠⍵:.}success
   ∆Pause 2
   newTotal←↑{','A.Split ⍵}¨A.ReadUtf8File ∆Path,'\input\en\total.csv'
   →PassesIf(noOfCSVs+6)=⍴F.ListFiles ∆Path,'\input\en\*.csv'
   →PassesIf oldTotal≢newTotal
   oldTotal[;2]←⍎¨oldTotal[;2]
   newTotal[;2]←⍎¨newTotal[;2]
   →PassesIf oldTotal[;2]∧.≤newTotal[;2]

   (rc more)←∆Execute_SC_Cmd'stop'
   →FailsIf 0≠rc
   ∆Pause 2
   (rc more)←∆Execute_SC_Cmd'query'
   →FailsIf 0=∨/'STATE : 1 STOPPED'⍷A.dmb more

   R←∆OK
∇

Though this test starts and stops the Service, its real purpose is to make sure that the Service processes input files as expected.

11.1. Running the tests

First we need to ensure everything is assembled freshly, and with admin rights.

The best way to do that is to run the script MakeService.dyapp from a console that was started with admin rights. (Unfortunately you cannot right-click on a DYAPP and select “Run as administrator” from the context menu.)

You must change the current directory in the console window to where the DYAPP lives before actually calling it.

Console with admin rights.

The best way to start a console window with admin rights:

  1. Press the Windows key.
  2. Type cmd. If you wonder, “where shall I type this into” don't worry - just type.
  3. Right-click on Command prompt and select Run as administrator.

A Dyalog instance is started. In the session you should see something similar to this:

Booting C:\...\v13\MakeService.dyapp
Loaded: #.APLTreeUtils
Loaded: #.FilesAndDirs
Loaded: #.HandleError
Loaded: #.IniFiles
Loaded: #.OS
Loaded: #.Logger
Loaded: #.EventCodes
Loaded: #.Constants
Loaded: #.Utilities
Loaded: #.MyApp
Loaded: #.ServiceState
Loaded: #.Tester
Loaded: #.Execute
Loaded: #.WinSys
Loaded: #.TestsForServices
Loaded: #.ServiceHelpers
#.⎕EX'ServiceHelpers'
Loaded: #.MakeService

In the next step establish the test helpers by calling #.TestsForServices.GetHelpers.

Finally run #.TestsForServices.RunDebug 0. You should see something like this:

#.TestsForServices.RunDebug 0
--- Test framework "Tester" version 3.3.0 from YYYY-MM-DD -----------------------------
Searching for INI file testcases_APLTEAM2.ini
  ...not found
Searching for INI file Testcases.ini
  ...not found
Looking for a function "Initial"...
*** Service MyAppService successfully installed
  "Initial" found and sucessfully executed
--- Tests started at YYYY-MM-DD hh:mm:dd on #.TestsForServices ------------------------
   Pausing for 2 seconds...
   Pausing for 2 seconds...
   Pausing for 2 seconds...
   Pausing for 2 seconds...
  Test_01 (1 of 2) : Start, pause and continue the service.
   Pausing for 2 seconds...
   Pausing for 2 seconds...
   Pausing for 2 seconds...
  Test_02 (2 of 2) : Start service, check results, give it some more work to do, check and stop it.
 --------------------------------------------------------------------------------------------------
   2 test cases executed
   0 test cases failed
   0 test cases broken
Time of execution recorded on variable #.TestsForServices.TestCasesExecutedAt in: YYYY-MM-DD hh:mm:ss
Looking for a function "Cleanup"...
  Function "Cleanup" found and sucessfully executed.
*** Tests done

Chapter 14:

The Windows Event Log

Now that we have managed to establish MyApp as a Windows Service we need to ensure it behaves. We shall make it report to the Windows Event Log.

1. What… is the Windows Event Log?

The Windows Event Log is by no means an alternative to application-specific log files. Most ordinary applications do not write to the Windows Event Log at all, some do only when things go wrong, and a very few, always.

In other words, for ordinary applications you may or may not find useful information in the Windows Event Log.

In contrast, an application that runs as a Windows Service is expected to write to the Windows Event Log when it starts, when it quits and when it encounters problems, and it might add even more information. There are few exceptions.

Similarly, Scheduled Tasks are expected to do the same, although some don't, or report only errors.

2. Is the Windows Event Log important?

On a server, all applications run either as Windows Services (most likely all of them) or as Windows Scheduled Tasks. Since no human is sitting in front of a server we need a way to detect problems on servers automatically.

That can be achieved by using software that constantly scans the Windows Event Log. It can email or text admins when an application that's supposed to run doesn't, or when an application goes astray, drawing attention to that server.

In large companies, which usually manage many servers, it is common to use software that checks the Windows Event Logs of all those servers.

So yes, the Windows Event Log is indeed important. Really important.

3. How to investigate the Windows Event Log

In modern versions of Windows you just press the Win key and type Event. That brings up a list which contains at least Event Viewer.

By default, the Event Viewer displays all Event Logs on the current (local) machine. However, you can connect to another computer and investigate its Event Log, if you have the right permissions. Here we keep it simple, and focus just on the local Windows Event Log.

4. Terms used

From the Microsoft documentation:

Each log in the Eventlog key contains subkeys called event sources. The event source is the name of the software that logs the event. It is often the name of the application or the name of a subcomponent of the application if the application is large.

You can add a maximum of 16,384 event sources to the registry. The Security log is for system use only. Device drivers should add their names to the System log. Applications and services should add their names to the Application log or create a custom log.[1]

5. Application log versus custom log

The great majority of applications that write to the Windows Event Log write into Windows Logs\Application, but if you wish you can create your own log under Applications and services logs.

For creating a custom log you need admin rights. So creating a custom log is something usually done by the installer for your software, since it needs admin rights by definition anyway.

We keep it simple here, and write to the Application log.

6. Let's do it

Copy Z:\code\v13 to Z:\code\v14.

6.1. Preconditions

Note that any attempt to write to the Windows Event Log with the WindowsEventLog class requires the Dyalog .NET bridge to be a sibling of the EXE, be it the Dyalog EXE or a custom stand-alone EXE.

6.2. Loading WindowsEventLog

We are going to make MyApp write to the Windows Event Log only when it runs as a Service. Therefore we need to load the class WindowsEventLog from within MakeService.dyapp (but not MyApp.dyapp):

...
Load ..\AplTree\OS
Load ..\AplTree\WindowsEventLog
Load ..\AplTree\Logger
...

6.3. Modify the INI file

We need to add to the INI a flag that allows us to toggle writing to the Window Event Log:

...
[Ride]
Active      = 0
Port        = 4599
wait        = 1

[WindowsEventLog]
write       = 1 ; Has an affect only when it's running as a Service

Why would this be useful? During development, when you run the Service to see what it's doing, you might not want the application to write to your Windows Event Log, for example.

6.4. Get the INI entry into the “Config” namespace

We modify the MyApp.CreateConfig function so that it creates Config.WriteToWindowsEventLog from that INI entry:

∇ Config←CreateConfig isService;myIni;iniFilename
...
      :If isService
          Config.WatchFolders←⊃myIni.Get'Folders:Watch'
          Config.WriteToWindowsEventLog←myIni.Get'WINDOWSEVENTLOG:write'
      :Else
          Config.LogFolder←'expand'F.NormalizePath⊃Config.LogFolder myIni.Get'Folders:Logs'
          Config.WriteToWindowsEventLog←0
      :EndIf
...
∇

6.5. Functions Log and LogError

For logging we introduce two new functions, Log and LogError. First Log:

∇ {r}←{both}Log msg
 ⍝ Writes to the application's log file only by default.
 ⍝ By specifying 'both' as left argument one can force the fns to write
 ⍝ `msg` also to the Windows Event Log if Config.WriteToWindowsEventLog.
   r←⍬
   both←(⊂{0<⎕NC ⍵:⍎⍵ ⋄ ''}'both')∊'both' 1
   :If 0<⎕NC'MyLogger'
       MyLogger.Log msg
   :EndIf
   :If both
   :AndIf Config.WriteToWindowsEventLog
       :Trap 0    ⍝ Don't allow logging to break!
           MyWinEventLog.WriteInfo msg
       :Else
           MyLogger.LogError'Writing to the Windows Event Log failed for:'
           MyLogger.LogError msg
       :EndTrap
   :EndIf
∇

Note that this function always writes to the application's log file. By specifying 'both' as left argument one can get the function to also write to the Windows Event Log, given that Config.WriteToWindowsEventLog is true.

That allows us to use Log for logging all events but errors, and to specify 'both' as left argument when we want the function to record the Service starting, pausing and stopping. In other words, all calls to MyLogger.Log will be replaced by Log, although some calls require 'both' to be passed as the left argument.

We also introduce a function LogError:

∇ {r}←LogError(rc msg)
 ⍝ Write to **both** the application's log file and the Windows Event Log.
   MyLogger.LogError msg
   :If Config.WriteToWindowsEventLog
       :Trap 0
           MyWinEventLog.WriteError msg
       :Else
           MyLogger.LogError'Could not write to the Windows Event Log:'
           MyLogger.LogError msg
       :EndTrap
   :EndIf
∇

Note that the Logger class traps any errors that occur. The WindowsEventClass does not do this, and the calls to WriteInfo and WriteError might fail for all sorts of reasons: invalid data type, invalid depth, lack of rights – you name it.

Therefore both Log and LogError trap any errors and write to the log file in case something goes wrong. Note also that in this particular case it's okay to trap all possible errors (0) because we cannot possibly foresee what might go wrong. In a real-world application you still want to be able to switch this kind of error trapping off via an INI entry etc.

In the case of an error we now want the function LogError to be called, so we change SetTrap accordingly:

∇ trap←{force}SetTrap Config
...
  #.ErrorParms.returnCode←EXIT.APPLICATION_CRASHED
  #.ErrorParms.(logFunctionParent logFunction)←⎕THIS'LogError'
  #.ErrorParms.windowsEventSource←'MyApp'
...
∇

Now it's time to replace the call to MyLogger.Log by a call to Log in the MyApp class; use the replace feature of the editor in order to achieve that.

There are however three functions where we need to add 'both' as left argument:

∇ {r}←MainLoop dummy;S
  r←⍬
  'both'Log'"MyApp" server started'
  S←#.ServiceState
  :Repeat
      LoopOverFolder ⍬
      :If ('both'∘Log S.CheckServiceMessages)S.IsRunningAsService
          'both'Log'"MyApp" is about to shut down...'
          :Leave
      :EndIf
      ⎕DL 2
  :Until 0
 ⍝Done
∇

Note that use the compose () operator here: only by ‘gluing’ the left argument ('both') to the function name with the compose operator can we ensure everything's passed to the Log function is written not only to the log file but also to the Windows Event Log when ServiceState is managing the communication between the SCM and the application.

The second function to be changed is Off:

    ∇ Off exitCode
      :If exitCode=EXIT.OK
          'both'Log'Shutting down MyApp'
      :Else

Now we change Initial: if the application is running as a service we let Initial create an instance of WindowsEventLog and return it as part of the result.

leanpub-start-insert
∇ r←Initial isService;parms;Config;MyLogger;MyWinEventLog
⍝ Prepares the application.
  Config←CreateConfig isService
  Config.ControlFileTieNo←CheckForOtherInstances ⍬
  CheckForRide(0≠Config.Ride)Config.Ride
  MyLogger←OpenLogFile Config.LogFolder
  Log'Started MyApp in ',F.PWD
  Log 2 ⎕NQ'#' 'GetCommandLine'
  Log↓⎕FMT Config.∆List
  r←Config MyLogger
  :If isService
      MyWinEventLog←⎕NEW #.WindowsEventLog(,⊂'MyAppService')
      parms←#.ServiceState.CreateParmSpace
      parms.logFunction←'Log'
      parms.logFunctionParent←⎕THIS
      #.ServiceState.Init parms
      r,←MyWinEventLog
  :EndIf
∇

Initial is called by RunAsService and StartFromCmdLine, but because the result of Initial remains unchanged if the application is not running as a Service we need to amend just RunAsService.

We localise MyWinEventLog (the name of the instance) and change the call to Initial since it now returns a three-item vector:

leanpub-start-insert
∇ {r}←RunAsService(earlyRide ridePort);⎕TRAP;MyLogger;Config;∆FileHashes;MyWinEventLog
 ⍝ Main function when app is running as a Windows Service.
...
  ⎕TRAP←#.HandleError.SetTrap ⍬
  (Config MyLogger MyWinEventLog)←Initial 1
  ⎕TRAP←(Config.Debug=0)SetTrap Config
...
∇

6.6. Does it still work?

Having made all these changes we should check whether the basics still work:

  1. Double-click Make.bat in order to re-compile the EXE.
  2. Double-click MyApp.dyapp. This assembles the workspace, including the test cases.
  3. Answer with y the question whether all test cases shall be executed.

Ideally the test cases should pass.

Now it's time to run the test cases for the Service:

  1. Open a console window with admin rights.
  2. Navigate to the v13\ folder.
  3. Call MakeService.dyapp.
  4. Execute TestsForServices.GetHelpers.
  5. Call TestsForServices.RunDebug 0.

Now start the Event Viewer; you should see something like this:

The Windows Event Log

You might need to scroll down a bit.

6.7. Adding a test case

We shall add a test case that checks whether the new logging feature works. For that we introduce Test_03:

∇ R←Test_03(stopFlag batchFlag);⎕TRAP;MyWinLog;noOfRecords;more;rc;records;buff
  ⍝ Start & stop the service, then check the Windows Event Log.
  ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
  R←∆Failed

  MyWinLog←⎕NEW #.WindowsEventLog(,⊂'MyAppService')
  noOfRecords←MyWinLog.NumberOfLogEntries

  (rc more)←∆Execute_SC_Cmd'start'
  →FailsIf 0≠rc
  ∆Pause 1
  (rc more)←∆Execute_SC_Cmd'query'
  →FailsIf 0=∨/'STATE : 4 RUNNING'⍷#.APLTreeUtils.dmb more
  ∆Pause 2

  (rc more)←∆Execute_SC_Cmd'stop'
  →FailsIf 0≠rc
  ∆Pause 2

  records←(noOfRecords-10)+⍳(MyWinLog.NumberOfLogEntries+10)-noOfRecords
  buff←↑MyWinLog.ReadThese records
  →PassesIf∨/,'"MyApp" server started '⍷buff
  →PassesIf∨/,'Shutting down MyApp'⍷buff

  R←∆OK
∇

Notes:

  1. First we save the number of records currently saved in the Windows Event Log “Application”.
  2. We then start and stop the server to make sure we get some fresh records written.
  3. We then read the number of records plus 10 (others write to the Windows Event Log as well) and investigate them.

7. Tips, tricks and traps

Feel confident with the Windows Event Log? Well, a few more wrinkles yet:


Footnotes

  1. Microsoft on the Windows Event Log:
    https://msdn.microsoft.com/en-us/library/windows/desktop/aa363648(v=vs.85).aspx

  2. Details about System Restore Point:
    https://en.wikipedia.org/wiki/System_Restore

Chapter 15:

The Windows Registry

1. What is it, actually?

We cannot say it any better than the Wikipedia [1]:

The Registry is a hierarchical database that stores low-level settings for the Microsoft Windows operating system and for applications that opt to use the Registry. The kernel, device drivers, services, Security Accounts Manager (SAM), and user interface can all use the Registry. The Registry also allows access to counters for profiling system performance.

In simple terms, The Registry or Windows Registry contains information, settings, options, and other values for programs and hardware installed on all versions of Microsoft Windows operating systems. For example, when a program is installed, a new subkey containing settings like a program's location, its version, and how to start the program, are all added to the Windows Registry.

The Windows Registry is still the subject of heated discussion between programmers. Most hate it, some like it. Whatever your opinion, you cannot ignore it.

Originally Microsoft designed the database as the source for any configuration parameters, be it for the operating system, users or applications.

The Windows Registry will certainly remain the store for any OS-related information but, for applications, we have seen a comeback of the old-fashioned configuration file, be it as an INI, an XML or a JSON.

Even if you use configuration files to configure your own application, you must be able to read and occasionally also to write to the Windows Registry, if only to configure Dyalog APL to suit your needs.

The Windows Registry can be used by any application to store user-specific data. For example, if you want to save the current position and size of the main form of your application for each user so ou can restore both position and size next time the application is started≤ then the Windows Registry is the perfect place for these facts. The key suggests itself:

HKCU\Software\MyApplication\MainForm\Posn
HKCU\Software\MyApplication\MainForm\Size

Note that HKCU is a short cut for “HKey Current User”. There are others, and we will discuss them.

2. Terminology

Microsoft’s terminology for the Registry varies strikingly from common usage. That is a rich source of confusion, but there is no avoiding it. Has to be mastered.

<!– Find the terminology strange? So do we, but it was invented by Microsoft and defines the standard.

So we use it too. That makes it easier to understand the Microsoft documentation and also to understand others discussing the Windows Registry. It also helps when you Google for Registry tweaks because the guys posting the solution to your problem are most likely using Microsoft speech as well.

What is strange about the terminology? Microsoft gives common words unusual meanings when it comes to the Windows Registry. –>

Here’s an example. This is the definition of the MAXWS parameter for Dyalog 64 bit Unicode version 16:

Definition of maxws in the Windows Registry

The full path is:

Computer\HKEY_CURRENT_USER\Software\Dyalog\Dyalog APL/W-64 16.0 Unicode\maxws

We can omit Computer if it is the local machine, and we can shorten “HKEY_CURRENT_USER” as “HKCU”. That leaves us with:

HKCU\Software\Dyalog\Dyalog APL/W-64 16.0 Unicode\maxws

That looks pretty much like a file path, doesn't it? So what about calling the different parts to the left of maxws folders?

Well, that might be logical, but Microsoft did not do that. Instead they call HKCU a key. The top-level parts are sometimes called root keys. The other parts except maxws are called subkeys; sometimes just keys.

So what's maxws? Well, it holds a value, so why not call it key? Oops, that's already taken. Maybe name or ID? Well, Microsoft calls it a value. That's a strange name because is has an associated value, in our example, the string '64000'.

To repeat: any given path to a particular piece of data in the Windows Registry consists of a key, one or more subkeys and a value that is associated with data:

root key\subkey\subkey\subkey\value = data

Some other things you should know:

3. Datatypes

These days the Windows Registry offers quite a range of datatypes, but most of the time you can manage with just these:

REG_SZ
The string datatype. APLers call this a text vector. Both WinReg as well as WinRegSimple write text vectors as Unicode strings.
REG_DWORD
A 32-bit number.
REG_BINARY
Binary data in any form.
REG_MULTI_SZ
For an APLer this is a vector of text vectors. This datatype was not available in the early days of the Windows Registry which is probably why it is not as widely used as you would expect.
REG_QWORD
A 64-bit number

There are more datatypes available, but they are less common.

4. Root keys

A Windows Registry has just 5 root keys:

Root key Shortcut
HKEY_CLASSES_ROOT HKCR
HKEY_CURRENT_USER HKCU
HKEY_LOCAL_MACHINE HKLM
HKEY_USERS HKU
HKEY_CURRENT_CONFIG HKCC

From an application programmer’s point of view, the HKCU and the HKLM are the most important ones, and usually the only ones to write to.

4.1. 32/64-bit nightmares

This is what the MSDN has to say about 32-bit and 64-bit applications when it comes to the Windows Registry:

On 64-bit Windows, portions of the registry entries are stored separately for 32-bit application and 64-bit applications and mapped into separate logical registry views using the registry redirector and registry reflection, because the 64-bit version of an application may use different registry keys and values than the 32-bit version. There are also shared registry keys that are not redirected or reflected.

The parent of each 64-bit registry node is the Image-Specific Node or ISN. The registry redirector transparently directs an application's registry access to the appropriate ISN subnode. Redirection subnodes in the registry tree are created automatically by the WOW64 component using the name Wow6432Node. As a result, it is essential not to name any registry key you create Wow6432Node.

The KEY_WOW64_64KEY and KEY_WOW64_32KEY flags enable explicit access to the 64-bit registry view and the 32-bit view, respectively. For more information, see Accessing an Alternate Registry View.

Probably best to avoid 32-bit applications and just create 64-bit applications these days if you can. If for any reason you are forced to deliver 32-bit applications then read up the details in the MSDN [2].

The knowledge you have accumulated by now will probably suffice for the rest of your work as an application programmer. If you want to know all the details we recommend the Microsoft documentation [3].

5. The class WinRegSimple

The APLTree class WinRegSimple is a very simple class that offers just three methods:

It is also limited to the two datatypes REG_SZ and REG_DWORD.

The class uses the Windows Scripting Host (WSH) [4]. It is available on all Windows systems although it can be switched off by group policies, though we have never seen this in the wild.

If you want to read just a certain value then this – very small – class might suffice. For example, to read the aforementioned maxws value:

      #.WinRegSimple.Read 'HKCU\Software\Dyalog\Dyalog APL/W-64 16.0 Unicode\maxws'
64000

You can create a new value as well as a new key with Write:

      #.WinRegSimple.Write 'HKCU\Software\Cookbooktests\MyValue' 1200

MyValue

You can also delete a subkey or a value, but a subkey must be empty:

      #.WinRegSimple.Delete 'HKCU\Software\Cookbooktests'
      #.WinRegSimple.Read 'HKCU\Software\Cookbooktests'
      #.WinRegSimple.Read'HKCU\Software\Cookbooktests\MyValue'
1200
      #.WinRegSimple.Delete 'HKCU\Software\Cookbooktests\MyValue'
Unable to open registry key "HKCU\Software\Cookbooktests\MyValue" for reading.
      #.WinRegSimple.Read'HKCU\Software\Cookbooktests\MyValue'
     ∧
      #.WinRegSimple.Delete 'HKCU\Software\Cookbooktests\'

To delete a subkey you must specify a trailing backslash.

You can also write the default value for a key. For that you must specify a trailing backslash as well. The same holds true for reading a default value:

      #.WinRegSimple.Write 'HKCU\Software\Cookbooktests\' 'APL is great'
      #.WinRegSimple.Read 'HKCU\Software\Cookbooktests\'
APL is great

Default values

Whether Write writes a REG_SZ or a REG_DWORD depends on the data: a text vector becomes “REG_SZ” while a 32-bit integer becomes “REG_DWORD” though booleans, as well as smaller integers, are converted to a 32-bit integer. Other datatypes are rejected.

If the WinRegSimple class does not suit your needs then have a look at the WinReg class. This class is much larger but has virtually no limitations at all.

To give you idea here the list of methods:

]adoc WinReg -summary
*** WinReg (Class) ***

Shared Fields:
  ERROR_ACCESS_DENIED
  ...
  REG_SZ
Shared Methods:
  Close
  CopyTree
  DeleteSubKeyTree
  DeleteSubKey
  DeleteValue
  DoesKeyExist
  DoesValueExist
  GetAllNamesAndValues
  GetAllSubKeyNames
  GetAllValueNames
  GetAllValues
  GetDyalogRegPath
  GetErrorAsStringFrom
  GetString
  GetTreeWithValues
  GetTree
  GetTypeAsStringFrom
  GetValue
  History
  KeyInfo
  ListError
  ListReg
  OpenAndCreateKey
  OpenKey
  PutBinary
  PutString
  PutValue
  Version

6. Examples

We will use both the WinReg class and the WinRegSimple class for two tasks:

The functions we develop along the way, as well as the variables we need, can be found in the workspace WinReg in the folder Z:\code\Workspaces\.

6.1. Add user command folder

Let's assume we have a folder C:\MyUserCommands. We want to add this folder to the list of folders holding user commands. For that we must find out the subkeys of all versions of Dyalog installed on your machine:

∇ list←GetAllVersionsOfDyalog dummy
[1] ⍝ Returns a vector of text vectors with Registry subkeys for all
[2] ⍝ versions of Dyalog APL installed on the current machine.
[3]   list←#.WinReg.GetAllSubKeyNames'HKCU\Software\Dyalog'
[4]   ⍝ Get rid of "Installed components" etc:
[5]   list←'Dyalog'{⍵/⍨((⍴⍺)↑[2]↑⍵)∧.=⍺}list
∇
      ↑GetAllVersionsOfDyalog ⍬
Dyalog APL/W 14.1 Unicode
Dyalog APL/W 15.0 Unicode
Dyalog APL/W 16.0 Unicode
Dyalog APL/W-64 13.2 Unicode
Dyalog APL/W-64 14.0 Unicode
Dyalog APL/W-64 14.1 Unicode
Dyalog APL/W-64 15.0
Dyalog APL/W-64 15.0 Unicode
Dyalog APL/W-64 16.0 Unicode

That's step one. In the next step we need to write a function that adds a folder to the list of user command folders:

∇ {r}←path Add version;subkey;folders
   r←⍬
   subkey←'HKCU\Software\Dyalog\',version,'\SALT\CommandFolder'
   'Subkey does not exist'⎕SIGNAL 11/⍨1≠#.WinReg.DoesValueExist subkey
   folders←#.WinReg.GetString subkey
   folders←';'{¯1↓¨⍵⊂⍨';'=¯1↓';',⍵}folders,';'
   folders←(({(819⌶)⍵}¨folders)≢¨⊂(819⌶)path)/folders ⍝ drop doubles
   folders←⊃{⍺,';',⍵}/folders,⊂path
   #.WinReg.PutString subkey folders
∇

Let's check the current status:

      dyalogVersions←AllVersionsOfDyalog ''
      ⍪{#.WinReg.GetValue 'HKCU\Software\Dyalog\',⍵,'\SALT\CommandFolder'}¨dyalogVersions
 C:\...\Dyalog APL 14.1 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\
 C:\...\Dyalog APL 15.0 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\
 C:\...\Dyalog APL 16.0 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\
...
      'C:\MyUserCommands'∘Add¨dyalogVersions
      ⍪{#.WinReg.GetValue 'HKCU\Software\Dyalog\',⍵,'\SALT\CommandFolder'}¨dyalogVersions
      C:\..\Dyalog APL 14.1 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\;C:\MyUserCommands
C:\...\Dyalog APL 15.0 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\;C:\MyUserCommands
C:\...\Dyalog APL 16.0 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\;C:\MyUserCommands
...
      'C:\MyUserCommands'∘Add¨dyalogVersions
      ⍪{#.WinReg.GetValue 'HKCU\Software\Dyalog\',⍵,'\SALT\CommandFolder'}¨dyalogVersions
C:\...\Dyalog APL 14.1 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\;C:\MyUserCommands
C:\...\Dyalog APL 15.0 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\;C:\MyUserCommands
C:\...\Dyalog APL 16.0 Unicode\SALT\Spice;C:\T\UserCommands\APLTeam\;C:\MyUserCommands

Although we called Add twice, the folder C:\MyUserCommands appears only once. This is because we carefully removed it before adding it.

6.2. Configure Dyalog's window captions

In Appendix 4 — The development environment we mention that if you run more than one instance of Dyalog in parallel then you want to be able to associate any dialog box to the instance that issued it. This can be achieved by adding certain pieces of information to certain entries in the Windows Registry. We talk about this subkey of, say, Dyalog APL/W-64 16.0 Unicode:

HKCU\Software\Dyalog\Dyalog APL/W-64 16.0 Unicode\Captions

If that subkey exists (after an installation it doesn't) then it is supposed to contain particular values defining the captions for all dialog boxes that might make an appearance when running an instance of Dyalog.

So to configure all these window captions you have to add the subkey Chapter and the required values in one way or another. This is a list of values honoured by version 16.0:

Editor
Event_Viewer
ExitDialog
Explorer
FindReplace
MessageBox
Rebuild_Errors
Refactor
Session
Status
SysTray
WSSearch

Although it is not a big deal to add these values with the Registry Editor we do not recommend this, if only because when the next version of Dyalog comes along then you have to do it again.

Let's suppose you have a variable captionValues which is a matrix with two columns:

Here's what captionValues might look like:

      ⍴⎕←values
 Editor          {PID} {TITLE} {WSID}-{NSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
 Event_Viewer    {PID} {WSID} {PRODUCT}
 ExitDialog      {PID} {WSID} {PRODUCT}
 Explorer        {PID} {WSID} {PRODUCT}
 FindReplace     {PID} {WSID}-{SNSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
 MessageBox      {PID} {WSID} {PRODUCT}
 Rebuild_Errors  {PID} {WSID} {PRODUCT}
 Refactor        {PID} {WSID}-{SNSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
 Session         {PID} {WSID}-{NSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
 Status          {PID} {WSID} {PRODUCT}
 SysTray         {PID} {WSID}
 WSSearch        {PID} {WSID} {PRODUCT}
13 2

Again, this variable can be copied from the workspace Z:\code\Workspaces\WinReg. We are going to write this data to the Windows Registry for all versions of Dyalog installed on the current machine.

For that we need a list with all versions of Dyalog installed on the current machine. We can use the function GetAllVersionsOfDyalog we developed earlier in this chapter:

   dyalogVersions←GetAllVersionsOfDyalog ''

Now we write a function that takes a version and the variable captionValues as argument and creates a subkey Captions with all the values. This time we use #.WinRegSimple.Write for it:

∇ {r}←values WriteCaptionValues version;rk
[1]  r←⍬
[2]  rk←'HKCU\Software\Dyalog\',version,'\Captions\'
[3]  rk∘{#.WinRegSimple.Write(⍺,(1⊃⍵))(2⊃⍵)}¨↓values
∇

We can now write captionValues to all versions:

       captionValues∘WriteCaptionValues¨dyalogVersions
      ⍝ Let's check:
      rk←'HKCU\Software\Dyalog\Dyalog APL/W-64 16.0 Unicode\Captions'
      #.WinReg.GetTreeWithValues rk
0  HKCU\...\Captions\
1  HKCU\...\Editor          {PID} {TITLE} {WSID}-{NSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
1  HKCU\...\Event_Viewer    {PID} {WSID} {PRODUCT}
1  HKCU\...\ExitDialog      {PID} {WSID} {PRODUCT}
1  HKCU\...\Explorer        {PID} {WSID} {PRODUCT}
1  HKCU\...\FindReplace     {PID} {WSID}-{SNSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
1  HKCU\...\MessageBox      {PID} {WSID} {PRODUCT}
1  HKCU\...\Rebuild_Errors  {PID} {WSID} {PRODUCT}
1  HKCU\...\Refactor        {PID} {WSID}-{SNSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
1  HKCU\...\Session         {PID} {WSID}-{NSID} {Chars} {Ver_A}.{VER_B}.{VER_C} {BITS}
1  HKCU\...\Status          {PID} {WSID} {PRODUCT}
1  HKCU\...\SysTray         {PID} {WSID}
1  HKCU\...\WSSearch        {PID} {WSID} {PRODUCT}

Footnotes

  1. The Wikipedia on the Windows Registry:
    https://en.wikipedia.org/wiki/Windows_Registry

  2. The MSDN on 32-bit and 64-bit applications and the Windows Registry:
    https://msdn.microsoft.com/en-us/library/windows/desktop/ms724072(v=vs.85).aspx

  3. Microsoft on the Windows Registry:
    https://msdn.microsoft.com/en-us/library/windows/desktop/ms724946(v=vs.85).aspx

  4. The Wikipedia on the Windows Scripting Host:
    https://en.wikipedia.org/wiki/Windows_Script_Host

Chapter 16:

Creating SetUp.exe with Inno

1. Defining the goal

Our application is now ready to be installed on a client's machine. What we need is a tool that:

  1. Collects all the files needed on a target machine
  2. Writes an installer SetUp.exe (you could choose a different name) that installs MyApp on the target machine with all its files

There are other things an installer might do, but these are the essential tasks.

2. Which tool

There are quite a number of tools available to write installers. Wix[1] is popular, and a good candidate if you need to install your application in large corporations.

Wix is very powerful, but its power has a price: complexity. We reckon your first customers are unlikely to have the complex installation requirements of a large corporation. You can start with something simpler.

If that’s not so, and you need to install your application in a complex corporate IT environment, consider consulting an IT professional for this part of the work.

Starting smaller allows you to choose a tool that is less complicated and can be mastered fast. Inno has made a name for itself as a tool that combines powerful features with an easy-to-use interface.

To download Inno visit http://www.jrsoftware.org/isdl.php. We recommend the 'QuickStart Pack'. That not only installs the Inno compiler and its help but also Inno Script Studio from Kymoto (https://www.kymoto.org/).

It also comes with an encrypting DLL – although we don't see the point of encrypting the installer: after installation user can access all the files anyway.

The Script Studio not only makes it easier to use Inno, it also comes with a very helpful debugger.

At the time of writing, both packages are free, even for commercial usage. We encourage you to donate to both Inno and Script Studio as soon as you start to make money with your software.

3. Inno and Script Studio

The easiest way to start with Inno is to take an existing script and study it. Trial and error and Inno's old-fashioned-looking but otherwise excellent help are your friends.

4. Sources of information

When you run into an issue or badly need a particular feature, then Googling for it is, of course, a good idea, and can be even better than referring to Inno's help. The help is excellent as a reference – you just type a term you need help with and press F1 – but if you don't know exactly what to search for, Google is your friend.

Often enough Google points you to Inno's help anyway, getting you straight to the right page. In For example, Google does an excellent job when you search for something like Inno src.

We have found that all we needed while getting acquainted with Inno.

5. Considerations

An installer needs admin rights. Installing a program is a potentially dangerous thing.

Information

It is rare these days to install an application in, say, C:\MyFolder. Under such rare circumstances, it might be possible to install an application without admin rights.

However, even installing a font requires admin rights.

Programs are usually installed in one of:

Those directories are protected by Windows, so only an administrator can install programs. An installer might do other things that require admin rights, for example…

Again, you must consider where certain things should be written to. Log files cannot and should not go into either C:\Program Files and C:\Program Files (x86), so they need to go elsewhere.

Let's suppose we want to install an application Foo. Your options are to create a folder Foo within…

The Roaming folder is the right choice if a user wants the application to be available regardless of which computer she logs on to.

About C:\ProgramData

There is only one difference between the AppData and the ProgramData folders: every user has her own AppData folder but there is only a single ProgramData folder, shared by all users.

The folder C:\ProgramData is hidden by default, so you will see it only when you tick the Hidden items checkbox on the View tab of the Windows Explorer.

Of course you can put that folder in any place you want — provided you have the necessary rights — but by choosing one of these two locations you stick to what's usual under Windows.

6. Sample application

We use a very simple application for this chapter: the application Foo just puts up a form:

Sample application 'Foo'

As soon as you press either Enter or Esc or click the Close button it will quit. That's all it does.

The application comes with several files. This is a list of http://cookbook.dyalog.com/code/v16/:

Foo.dws
The workspace from which Foo.exe was created with the File > Export menu command
Foo.exe
The application's EXE, created from the aforementioned workspace
foo.ico
The icon used by the application
Foo.iss
The Inno file that defines how to create the installer EXE. It is this file we are going to discuss in detail.
foo.ini.remove_me
Foo's INI
ReadMe.html

An HTM with basic information about the application

With the exception of Foo.exe and Foo.iss the files are included for illustrative purposes only. The INI, for example, is not processed at all by Foo.

7. Using Inno

Before going into any detail let's look briefly at a typical Inno script.

7.1. Structure of an Inno script

An Inno script, like a good old-fashioned INI, has sections:

Setup
In this section you define all the constants specific to your application. There should be no other place where, say, a path or a filename is specified; it should all be done in the [Setup] section.
Language
Defines the language and the message file
Registry
Information to be written to the Windows Registry
Dirs
Constants that point to particular directories and specify permissions
Files
Files that are going to be collected within SetUp.exe
Icons
Icons required
Run
Other programs to be run, either during installation or afterwards
Tasks
Checkboxes or radio buttons for the installation wizard's windows so that the user can decide whether those tasks should be performed
Code

Defines programs (in a scripting language similar to Pascal) for doing more complex things

Inno’s powerful built-in capabilities allow us to achieve our goals without writing any code, so we won't use the scripting capabilities. Note however that for many common tasks there are scripts available on the Internet.

7.2. The file Foo.iss

Double-clicking the file's icon should open it in Inno Script Studio, the Inno IDE, with its execution and debugging tools.

7.3. Define variables

As a preamble, define as variables all the values you will use in the script.

This makes the script more readable, and guards against typos and conflicts.

#define MyAppVersion "1.0.0"
#define MyAppName "Foo"
#define MyAppExeName "Foo.exe"
#define MyAppPublisher "My Company Ltd"
#define MyAppURL "http://MyCompanyLtd.com/Foo"
#define MyAppIcoName "Foo.ico"
#define MyBlank " "

MyBlank is included to improve readability. It makes it easier to spot a blank character in a name or path.

7.4. The section [Setup]

[Setup]
; NOTE: The value of AppId uniquely identifies this application.
; It's a 36-character long vector called a UUID or GUID.
AppId={{E0DF5CAB-97E5-4935-A2ED-A7D43DD958D9}

AppName="{#MyAppName}"
AppVersion={#MyAppVersion}
AppVerName={#MyAppName}{#MyBlank}{#MyAppVersion}
AppPublisher={#MyAppPublisher}
AppPublisherURL={#MyAppURL}
AppSupportURL={#MyAppURL}
AppUpdatesURL={#MyAppURL}
DefaultDirName={pf32}\{#MyAppPublisher}\{#MyAppName}
DefaultGroupName={#MyAppPublisher}\{#MyAppName}
AllowNoIcons=yes
OutputDir=ReadyToShip\Foo
OutputBaseFilename="SetUp_{#MyAppName}"
Compression=lzma
SolidCompression=yes
SetupIconFile={#MyAppIcoName}
UninstallDisplayIcon={app}\{#MyAppIcoName}

The meaning of much of the above is pretty obvious. All those names must be defined, however: Inno needs them.

Notes:

7.5. The section [Languages]

[Languages]
Name: "english"; MessagesFile: "compiler:Default.isl"; \
  LicenseFile: "License.txt"; \
  InfoBeforeFile: "ReadMe_Before.txt"; \
  InfoAfterFile: "ReadMe_After.txt";

Inno supports multilingual installations but this is beyond the scope of this chapter. We define just one language here. The parameters for a single language must be defined on a single line but you can avoid very long lines by splitting them with a \ at the end of a line as shown above.

While the two parameters Name and MessageFile are required, the other three parameters are optional:

When we execute Setup.exe you will see when exactly their content is displayed. If LicenseFile is defined the user must accept the conditions before the installation completes.

7.6. The section [Registry]

[Registry]
Root: HKLM32; Subkey: "Software\{#MyAppPublisher}"; Flags: uninsdeletekeyifempty
Root: HKLM32; Subkey: "Software\{#MyAppPublisher}\{#MyAppName}"; Flags: uninsdeletekey
Root: HKLM32; Subkey: "Software\{#MyAppPublisher}\{#MyAppName}"; \
  ValueType: string; ValueName: "RecentFiles"; ValueData: ""; Flags: uninsdeletekey

This section allows you to add settings to the Windows Registry.

Notes:

7.7. The section [Dirs]

[Dirs]
Name: "{commonappdata}\{#MyAppName}"; Permissions: users-modify

From the Inno Help:

This optional section defines any additional directories Setup is to create besides the application directory the user chooses, which is created automatically. Creating subdirectories underneath the main application directory is a common use for this section.

With the above line we tell Inno to create a folder {#MyAppName} which in our case will be “My Company Ltd”. Note that commonappdata defaults to ProgramData\, usually on the C:\ drive.

Instead we could have used localappdata, which defaults to C:\Users\{username}\AppData\Local. There are many more constants available; refer to Constants in the Inno Help for details.

We also tell Inno to give any user in the Users group the right to modify files in this directory.

Warning

Of course you must not grant Modify rights to the folder where your application's EXE lives, let alone to folders not associated with your application.

Note that if you install the application again the folder won't be created – and you won't see an error message either.

7.8. The section [Files]

[Files]
Source: "ReadMe.html"; DestDir: "{app}";
Source: "Foo.ico"; DestDir: "{app}";
;Source: "bridge160_unicode.dll"; DestDir: "{app}";
;Source: "dyalognet.dll"; DestDir: "{app}";
Source: "{#MyAppExeName}"; DestDir: "{app}";
Source: "foo.ini.remove_me"; DestDir: "{app}"; DestName:"foo.ini"; Flags: onlyifdoesntexist;
Source: {#MyAppIcoName}; DestDir: "{app}";
Source: "C:\Windows\Fonts\apl385.ttf"; DestDir: "{fonts}"; FontInstall: "APL385 Unicode"; Flags: onlyifdoesntexist uninsneveruninstall
; NOTE: Don't use "Flags: ignoreversion" on any shared system files

; ----------- For Ride: ---------------
;Source: "Conga*.dll"; DestDir: "{app}";
; -------------------------------------

We have included here a number of files quite common in any APL application:

Warning

.NET

If your applications call any .NET methods make sure you include the Dyalog .NET bridge files!

7.9. The section [Icons]

[Icons]
Name: "{group}\Start Foo"; Filename: "{app}\{#MyAppExeName}"; WorkingDir: "{app}\";  IconFilename: "{app}\{#MyAppIcoName}"
Name: "{commondesktop}\{#MyAppName}"; Filename: "{app}\{#MyAppExeName}"; IconFilename: "{app}\{#MyAppIcoName}"; Tasks: desktopicon

The first line inserts group and application name into the Windows Start menu. Read up on group in the Inno help for what the group name means and where it is installed: there are differences between users who install the application with admin rights and those who don't.

7.10. The section [Run]

[Run]
Filename: "{app}\ReadMe.html"; Description: "View the README file"; Flags: postinstall shellexec skipifsilent
Filename: "{app}\{#MyAppExeName}"; Description: "Launch Foo"; Flags: postinstall skipifsilent nowait

Notes:

7.11. The section [Tasks]

[Tasks]
Name: "desktopicon"; Description: "{cm:CreateDesktopIcon}"; GroupDescription: "{cm:AdditionalIcons}";

From the Inno help:

This section is optional. It defines all of the user-customizable tasks Setup will perform during installation. These tasks appear as checkboxes and radio buttons on the Select Additional Tasks wizard page.

In our example we specify only one task, and it's linked to desktopicon (see the [Icons] section).

However, this is a much more powerful feature than it looks like at first glance! For example, you can give the user a choice between installing the…

or any combination of them.

To achieve that you need to add the (optional) section [Components] and list all the files involved there. You can then create additional lines in the [Task] section that link to those lines in [Components].

The user is then presented with a list of checkboxes that allow her to select the options she's after.

Note that cm:CreateDesktopIcon refers to a message CreateDesktopIcon which can be modified if you wish. The cm stands for Custom Message. For that, you would insert the (optional) section [CustomMessages] like this:

[CustomMessages]
CreateDesktopIcon = This and that

That would overwrite the internal message.

7.12. The section [Code]

Inno comes with a built-in script language that allows you to do pretty much whatever you like. However, scripting is beyond the scope of this chapter.

8. Potential problems

8.1. Updating an already installed version

There is no need to take action - Inno will handle this automatically for you. There is, however, one exception and one pitfall:

8.1.1. Files that are not needed any longer

Inno won't (can't) delete those automatically. This is the recommended way to delete files:

[InstallDelete]
Type: files; Name: {app}\foo.bar
Type: files; Name: {app}\baz\quux.txt

Although wildcards are supported they should never be used because you may well delete user files.

8.1.2. The EXE

When an EXE is part of the installation Inno compares the “FileVersion” of what's already installed with the one that is about to be installed. If they match Inno won't take any action. That means that if you forget to set the “FileVersion” (see chapter 10: “Make: Export”) when creating the stand-alone EXE then it will always be 0.0.0.0, so they won't ever differ, and the first installed EXE will never be replaced!

8.2. Analyzing problems

A problem like the aforementioned one won't cause an error message; you just don't get the new EXE installed. Assuming that you have been careful enough to realize that there is a problem, how to get to the bottom of it?

If the installer does not behave as expected add /LOG={filename} to it. Then Inno will report all actions and even conclusions to that file.

In the above scenario you would find something similiar to this in the log file:

2018-05-08 06:26:30.826   -- File entry --
2018-05-08 06:26:30.827   Dest filename: C:\Program Files (x86)\..\Foo.exe
2018-05-08 06:26:30.827   Time stamp of our file: 2018-05-07 13:07:26.000
2018-05-08 06:26:30.827   Dest file exists.
2018-05-08 06:26:30.827   Time stamp of existing file: 2018-05-07 12:51:24.000
2018-05-08 06:26:30.827   Version of our file: 0.0.0.0
2018-05-08 06:26:30.832   Version of existing file: 0.0.0.0
2018-05-08 06:26:30.832   Same version. Skipping.

9. Conclusion

Although Inno is significantly easier to master than the top dog Wix, it provides a large selection of features and options. This chapter only scratches the surface, but it will get you going.


Footnotes

  1. http://wixtoolset.org/:
    Windows Installer

  2. https://blogs.msdn.microsoft.com/oldnewthing/20080627-00/?p=21823/:
    About GUIDs

Chapter 17:

Regular expressions with Dyalog

1. Start here

You should be able to read and understand this chapter without any previous knowledge of regular expressions.

If you are fluent in regular expressions but unfamiliar with Dyalog's implementation, read this chapter. Dyalog's implementation has some unique and extremely powerful features not offered by any other we have seen.

If you are heavily involved in number crunching and never search strings for certain patterns you probably don’t need regular expressions.

Regular expressions are an extremely powerful tool, but the level of abstraction is high. They are hard to master without regular use. If you need to find a pattern in a string twice a year, you are probably better off finding a friend you can ask for advice.

Having said this, it is amazing how few APLers realize how often they actually do search for patterns in strings.

2. What you can expect

In this chapter we explain regular expressions mostly by example. The examples start off simple and grow complex. Along the way we introduce more features of regular expressions in general and Dyalog's implementation in particular.

Mastering even the very basic features of regular expressions might improve your APL applications.

Your best strategy is to read the following from start to end, not necessarily in one sitting. It will introduce you to the basic concepts and provide you with the necessary knowledge to become a keen amateur.

From there frequent usage of regular expressions — and some research on the Web – will render you an expert. This time and effort is well invested.

This chapter is by no means a comprehensive introduction to regular expressions. It should get you to a point where you can take advantage of examples, documents and books that are not specific to Dyalog's implementation.

3. What exactly are regular expressions?

Regular expressions allow you to find the position of a string in another string. They also allow you to replace a string by another one.

4. Background

Dyalog uses the PCRE implementation of regular expressions. There are other other implementations available, and they differ in their use and performance characteristics. So you need to know what kind of engine you are using to find the right advice, examples and solutions on the Web.

Dyalog 16.0 uses PCRE version 8. PCRE is considered one the most complete and powerful implementations of regular expressions.

5. RegEx in a nutshell

5.1. Search a string in a string

      ⍴'notfound' ⎕s 0⊣ 'the cat sat on the medallion'
0
      'cat' ⎕s 0⊣ 'the cat sat on the medallion'
 4

Operators, operands, derived functions and arguments

⎕S is, like ⎕R, an operator. An operator takes either just a left operand (monadic) or a left and right operand (dyadic) and forms a so-called derived function. For example, the operator / when given a left operand + forms the derived function sum.

In the example, the 0 is the right operand. With ⎕S the right operand can be one or more of 0, 1, 2 and 3 (the transformation codes) or a user-defined function, discussed later.

The transformation codes 2 and 3 will be discussed later.

The right argument to the derived function is the string 'the cat sat on the medallion'. The operand and the string are separated by the function. Instead we could have used parentheses, with the same result

      ('notfound' ⎕s 0) 'the cat sat on the medallion'`.

In the first expression the result is empty because the string notfound was not found. In the second expression cat was actually found. So we could say:

      'cat' {⍵↓⍨⍺ ⎕s 0 ⊣ ⍵}'the cat sat on the medallion'
cat sat on the medallion

That was easy, wasn't it! Regular expressions are nothing to be afraid of.

5.2. What's between the double quotes

Let's look at another example: find out what's between double quotes. First attempt:

      '"' ⎕S 0 ⊣ 'He said "Yes"'
 8 12

That gives us the offset of the two double quotes, but what if we want to have the offset and the length of any string found between — and including — the double quotes?

For that we need to introduce the metacharacter dot (.), which has a special meaning in a regular expression: it represents any character. (Not quite: we will soon discuss the one and only exception, the NewLine character).

Metacharacters

Metacharacters, sometimes called special characters, are characters with a special meaning in a regular expression.

To specify a character that serves as a metacharacter, we escape it with a backslash. So, yes, backslash is another metacharacter. To represent a dot followed by a backslash we would write \.\\

We’ll encounter more metacharacters as we go.

So we try:

      '"."' ⎕S 0 1 ⊣ 'He said "Yes"'

Oops - no hit. To understand this we have to know exactly what the regular expression engine did:

  1. It starts at the beginning of the string; that is actually one to the left of the “H”! That position can only be matched by the metacharacter ^ which represents the start of a line.
  2. Since we did not specify a leading ^ the RegEx engine moves to the first character.
  3. It then tries to match " to H. Since there is no match…
  4. It carries on until it arrives at the "; this is called “consuming a character”. Now there is a match!
  5. The engine now tries to match the . against the Y. Since the. matches any character this is a match, too.
  6. It then moves forward one more character and tries to match the " with e - that's not a match.
  7. The engine forgets what it has done, goes back to where it started from (that was the ") and moves one character forward.
  8. It now tries to match the " with the Y

You can now see why it does not report any hit - it would only work on any single character that is embraced by two double quotes, for example on 'He said "Y"'.

5.3. Repeating a search pattern (quantifiers)

What we need is a way to tell the engine that it should try to match the . more than once. There is a general way of doing this and three abbreviations that cover most cases. First, the general way.

For example, in order to match a minimum of one to a maximum of three underscore characters:

      '_{1,3}' ⎕S 0 ⊢'_one__two___three____four'
0 4 9 17 20

It is actually easier to check the result by replacing the hits with something that stands out:

      '_{1,3}' ⎕R '⍈' ⊢'_one__two___three____four'
⍈one⍈two⍈three⍈⍈four
      '_{2,3}' ⎕R '⍈' ⊢'_one__two___three____four'
_one⍈two⍈three⍈_four

The right operand of ⎕R

⎕R takes one or more replace strings as the right operand – or a user defined function, discussed later. You cannot mix replace strings with user defined functions.

What's between the curly braces ({}) – those are metacharacters as well – defines how many instances are required: minimum and maximum. This is called a quantifier.

Then there are the abbreviations that make life a bit easier:

So:

      '".*"' ⎕R '⍈' ⊣ 'He said "Yes"!'
He said ⍈!
      '".*"' ⎕R '⍈' ⊣ 'He said "Yes" and "No"!'
He said ⍈!

Just one hit, and that hit spans "Yes" and "No"?!

5.4. Greedy and lazy

By default the engine is greedy as opposed to lazy:

  1. First it tries to match the . against as many characters as it can.

    That means it will stop only at the end of the line, because the dot will match everything but the end-of-line. as the asterisk repeats it over and over again.

  2. Then the RegEx engine will backtrack and try to find a double quote, coming from the right; once it finds one it's done the job.

The same is true for the {x,y} quantifiers: by default they are all greedy rather than lazy, so repeating y times is done first before reducing the repetition to x times.

What we need instead is a lazy search:

  1. First it tries to match the . against the current character which is the “Y”. That's a match.
  2. It then tries to match the double quote against the next character. That fails, therefore the engine backtracks and tries again to match the dot against the “e”.
  3. That's a match, so it again tries to match the double-quote with the “s”…

You can see how we end up at the first double quote rather than the last one.

We can achieve that by specifying the “Greedy” option with a zero (default is one):

      '".*"' ⎕R '⍈' ⍠('Greedy' 0) ⊣ 'He said "Yes" and "No"!'
He said ⍈ and ⍈!

Specifying options

Various options are available to control aspects of Search and Replace; these are selected using the variant operator .

Note that if you use Classic Dyalog, or just want to ensure your code runs in a classic interpreter, ⎕OPT is a synonym for .

Between the right operand and the right argument you may specify options. They are marked by the operator. These are the options available:

IC, Mode, DotAll, EOL, ML, Greedy, OM, InEnc, OutEnc, Enc, ResultText, UCP

IC – ignore case – is the principal option. That means that if no other option needs to be specified you can omit the 'IC'.

So this would do: ⎕S 0 ⍠ 1 ⊢ 'whatever'. A 1 would make a search pattern case insensitive. The default is 0.

In this chapter we’ll discuss Mode, DotAll, Greedy and UCP: these are the most important options. For the others refer to ⎕R's help page.

Let's repeat our findings because this is so important:

5.5. Lazy quantifiers

The quantifiers we discussed earlier are all greedy, but you can make them lazy by adding a ? as shown here:

      .{0,}?
      .*?
      .+?
      .??

5.6. The + can be dangerous!

But wouldn't it be better to use + rather than * here? After all, we are not interested in "" because there is nothing between the two double quotes? Good point except that it does not work:

      '".+"' ⎕R '⍈' ⍠('Greedy' 0) ⊣ 'He said "" and ""'
He said ⍈"

That's because the engine would perform the following steps:

  1. Investigate until we find a ".
  2. Investigate the character right after the first double quote. That is the second double quote so that is consumed because the regular expression required at least one. Since all other characters are a match as well the .+ consumes all characters to the very end of the input string.
  3. Because it is lazy it then goes back the current position which by then is the space to the right of the second double-quote!

    From there it carries on until it finds a ". That is the first " after the word «and». All that is then replaced by the character.

  4. The engine then carries on but because there is only a single " left there is no other match.

If we want to ignore any "" pairs then we need to use a look-ahead, something we will discuss soon.

5.7. Negate

Here is another — and better — way to solve our problem:

      '"[^"]*"'⎕R'⍈'⊣'He said "" and ""'

Note that this expression has two advantages:

  1. It is faster than our first attempt although in our example the difference is miniscule.
  2. It's independent of the setting of Greedy.

5.8. Garbage in, garbage out

Let's modify the input string:

      is←'He said "Yes"" and "No"'  ⍝ define "input string"

There are two double quotes after the word “Yes”; that seems to be a typo. Watch what our RegEx is making of this:

      '".*"' ⎕R '⍈' ⍠('Greedy' 0) ⊣ is
 He said ⍈⍈No"

This example highlights a potential problem with input strings: many regular expressions work perfectly well as long as the input string is syntactically correct (or matches your expectations).

That's the reason (among others) why regular expressions are not recommended for processing HTML because HTML is very often full of syntactical errors.

However, if you can be certain that the HTML you have to deal with is syntactically correct and the piece of HTML is short and not nested then you can use regular expressions to process HTML.

Regular expressions and HTML

There are claims that you cannot parse HTML with regular expressions because a regular expression engine is a Finite Automata while HTML can be nested indefinitely. This is partly wrong and partly misleading.

5.9. Metacharacters: the full list

By now we've met quite a number of metacharacters; how many do we have to deal with? Well, quite a lot:

Metacharacter Symbol Meaning
1. Backslash \ Escape character
2. Caret ^ Start of line
3. Dollar sign $ End of line
4. Period or dot . Any character but NewLine
5. Pipe symbol | Logical “OR”
6. Question mark ? Extends meaning of (; 0 or 1 quantifier (=optional); make it lazy
7. Asterisk or star * Repeat 0 to many times
8. Plus sign + Repeat 1 to many times
9. Opening parenthesis ( Start sub pattern
10. Closing parenthesis ) End sub pattern
11. Opening square bracket [ Start character class (or set)
12. Opening curly brace { Start min/max quantifier

By now we have already discussed six of them; they carry a check mark.

Note that both } and ] are considered metacharacters only after an opening { or [. Without the opening counterpart they are taken literally; that's why they did not make it onto the list of metacharacters.

5.10. Search digits in a string

Let's suppose we want to match all digits in a string:

      '[0123456789]'⎕R '⍈' ⊣ 'It''s 23.45 plus 99.12.'
It's ⍈⍈.⍈⍈ plus ⍈⍈.⍈⍈.

Everything between the [ and the ] is treated as a simple character - with a few exceptions we'll soon discuss. That makes both [ and ] metacharacters.

The same but shorter:

      '[0-9]'⎕R '⍈' ⊣ 'It''s 23.45 plus 99.12.'
It's ⍈⍈.⍈⍈ plus ⍈⍈.⍈⍈.

The minus is treated as a metacharacter here: it means “all digits from 0 to 9”.

5.11. The escape character: \

Even shorter:

      '\d'⎕R '⍈' ⊣ 'It''s 23.45 plus 99.12.'
It's ⍈⍈.⍈⍈ plus ⍈⍈.⍈⍈.

Note that the metacharacter backslash (\ ) is used for two different purposes:

So \* takes away the special meaning from the * while \d gives the d the special meaning: all digits.

We take the opportunity to add the dot (.) and the minus (-) to the character class.

Note that the minus is not escaped; from the context the regular expression engine can work out that here the minus cannot mean from-to, so it is taken literally.

Note also that the . is not escaped either: inside the pair of [] the . is not a metacharacter.

Escaping several characters

Say you need to escape all the metacharacters because you want to search them. Escaping every single one of them with a backslash is laborious and decreases readability, but there is a better way:

      '\Q\^$.|?*+()[{\E'

This escapes all characters between the \Q and the \E.

      '[\d.-]'⎕R '⍈' ⊣ 'It''s 23.45 plus -99.12.'
It's ⍈⍈⍈⍈⍈ plus ⍈⍈⍈⍈⍈⍈⍈

Here we have another problem: we want the dot only to be a match when there is a digit to both the left and the right of the dot. Our search pattern does not deal with this, so the trailing . is a match. We will tackle this problem soon.

Character classes work for letters as well:

      '[a-zA-Z]'⎕R '⍈' ⊣'It''s 23.45 plus 99.12.'
⍈⍈'⍈ 23.45 ⍈⍈⍈⍈ 99.12.

We can negate with ^ right after the opening [:

    '[^a-zA-Z]'⎕R '⍈' ⊣'It''s 23.45 plus 99.12.'
It⍈s⍈⍈⍈⍈⍈⍈⍈plus⍈⍈⍈⍈⍈⍈⍈

Notes:

5.12. Negate with digits and dots

      '[^.0-9]'⎕R '⍈' ⊣'It''s 23.45 plus 99.12.'
⍈⍈⍈⍈⍈23.45⍈⍈⍈⍈⍈⍈99.12.

Want to search for gray and grey in a document?

      'gr[ae]y'⎕S 0⊣'Americans spell it "gray" and Brits "grey".'
20 37
      'gr[ae]y'⎕R '⍈' ⊣'Americans spell it "gray" and Brits "grey".'
Americans spell it "⍈" and Brits "⍈".

5.13. Metacharacters within a character class

Note that there are only a few metacharacters inside character classes:

Metacharacter Symbol Meaning
Closing bracket ]
Backslash \ Escape next character
Caret ^ Negate the character class but only after the opening [
Minus - From-to if there is something to both the left & right.

We already worked out the engine is smart enough to take a minus literally when it makes an appearance somewhere where it cannot mean from-to: the beginning and the end of a character class.

Similarly, the caret (^) character can only negate a character class as a whole when it follows the opening square bracket ([^). If the caret is specified elsewhere it is taken literally.

Therefore the expression [0-9^1] does not mean “all digits but 1”, it means “all digits and the caret character and a 1”.

5.14. Searching for whitespace characters

Finding 0 to 3 whitespace characters followed by an ASCII letter at the beginning of a line:

      {'^\s{0,3}[a-zA-Z]' ⎕R '⍈' ⊣ ⍵}¨'Zero' ' One' '  Two' '   Three' '    four'
⍈ero  ⍈ne  ⍈wo  ⍈hree      four

\s escapes the ASCII letter s, meaning that the s takes on a special meaning: \s stands for any whitespace character. That is at the very least the space character (⎕UCS 32) and the tab character (⎕UCS 9).

There are two options (set with the operator) that control which other characters qualify as white space:

Both will be discussed soon.

5.15. The options DotAll and Mode

We've learned that the . matched any character but not end of line.

      '".*"'⎕R'⍈'⊣ '"Foo' 'Goo"'
 "Foo  Goo"
Information

A vector of text vectors is internally processed like a file with multiple records.

Because the . does not match end-of-line it finds "Foo but then stops rather than carrying on trying to match the pattern with Goo", so no hit is found.

With the DotAll option — which defaults to 0 — we can tell the RegEx engine to let . even match the end of a line:

      '".*"'⎕R'⍈'⍠('DotAll' 1)('Mode' 'D')⊣'"Foo' 'Goo "'
 ⍈

Note that we had to specify the Mode option as well because DotAll←1 is invalid with Mode←L, which is the default.

It is important to understand the different modes and their influence on ^ (start of document or line), $ (end of document or line) and DotAll see Document mode for details.

It's not difficult to imagine a situation where you have a single search pattern but in one part you want the . to match end-of-line and in others you won't.

5.16. The metacharacter \N

\N has almost the same meaning as the . except that it never matches the end of a line. That means it's independent of the setting of DotAll:

      '"\N*"'⎕R'⍈'⍠('DotAll' 1)('Mode' 'D')⊣'"Foo' 'Goo "'
 "Foo  Goo "

Note that \N and \n are different: \n stands for a newline character.

5.17. Analyzing APL code: Replace

Let's suppose you want to investigate APL code for a variable name foo but you want text and comments ignored. This is our input string:

      is←'a←1 ⋄ foo←1 2 ⋄ txt←''The ⍝ marks a comment; foo'' ⍝ set up vars a, foo, txt'

We want foo←1 2 to be found or changed while the text and the comment remain ignored or unchanged. The problem is aggravated by the fact that the text contains a symbol.

A naïve approach does not work:

      'foo' ⎕R '⍈' ⊣ is
a←1 ⋄ ⍈←1 2 ⋄ txt←'The ⍝ marks a comment; ⍈' ⍝ set up vars a, ⍈, txt

Dyalog's implementation of regular expressions offers an elegant solution to the problem:

      '''\N*''' '⍝\N*$' 'foo'⎕R(,¨'&&⍈')⍠('Greedy' 0)⊣is
a←1 ⋄ ⍈←1 2 ⋄ txt←'The ⍝ marks a comment; foo' ⍝ set up vars a, foo, txt

This needs some explanation:

  1. \N is the same as a .: it matches all characters but the end-of-line character.

    Setting ('DotAll' 1) might make sense for the third search pattern (foo) but it would under certain circumstances prevent the first two search patterns from working, therefore we must use the \N syntax for those patterns.

  2. '''\N*''' catches all text — that is everything between quotes — and for that text & is specified as a replacement.

    Now & stands for the matching text, therefore nothing will change at all but the matching text won't participate in any further actions! In other words: everything between quotes is left alone.

  3. '⍝\N*$' catches everything from a lamp () to the end of the line ($) and replaces it by itself (&). Again nothing changes but the comment will not be affected by anything that follows.

    Since the first expression has already masked everything within (and including) quotes the first does not cause problems; it is ignored at this stage anyway.

  4. Finally foo catches the string foo in the remaining part, and that is what we are interested in.

As a result foo is found within the code but neither between the double quotes nor as part of the comment.

As far as we know, this powerful feature is specific to Dyalog, but we have only limited experience with other regular expression engines.

However, be aware that the third pattern must be very specific! To rephrase it: if the third pattern is matching anything between quotes etc. then it will change them anyway.

In this example this is illustrated:

      '''\N*''' '⍝\N*$' '^.*$'⎕R(,¨'&&⍈')⍠('Greedy' 1)('Mode' 'D')⊣is
⍈

The expression ^.*$ together with ('Greedy' 1) and ('Mode' 'D') means:

  1. The ^ matches the start of the document!
  2. The $ matches the end of the document!
  3. The expression .* matches everything including line breaks!

Therefore the expression changes the whole document into a single despite the first two patterns.

5.18. Regular expressions and scalar extension

Note that the in ,¨'&&⍈' is essential: without it the RegEx engine would use the pattern '&&⍈' thrice.

The reason is that ⎕R actually does not accept scalars, it only accepts vectors. So if you specify three search patterns on the left, then you need to specify not four and not two but three replace patterns as well.

If only a single vector is specified then this vector is taken thrice: a version of scalar extension.

5.19. Analysing APL code: Search

The extremely powerful syntax discussed above is also available with ⎕S:

      '''\N*''' '⍝\N*$' 'foo'⎕S 0 1 2⍠('Greedy' 0)⊣is
 6 3 2  20 28 0  49 25 1

We have already discussed the transformation codes 0 (offset) and 1 (length) but not 3: this returns the pattern number which matched the input document, origin zero.

In our case it is the third pattern (foo) we are interested in, so we can ignore those that are 0 and 1.

Greedy and lazy

Using the option ⍠('Greedy' 0) has a disadvantage: it makes all search patterns lazy.

There will be cases when you want only a part of your search pattern to be lazy and other parts greedy.

Luckily this can be achieved with the metacharacter question mark (?):

      '"\N*?"'⎕R '⍈' ⊣ is
He said ⍈ and ⍈

Since the engine is greedy by default, you need specify the ? only for those parts of your search pattern you want to be lazy.

5.20. Word boundaries

Our search pattern is still not perfect since it would work on boofoogoo as well:

      '''\N*''' '⍝\N*$' 'foo'⎕R(,¨'&&⍈')⍠('Greedy' 0)⊣'This boofoogoo is found as well'
This boo⍈goo is found as well

We can solve this problem with \b (word boundaries). \b does not attempt to match a particular character on a particular position. Instead it checks whether there is a word boundary either before are after the current position – but not both.

That means no matter whether they are successful or not they won't change the position the engine is currently investigating. They are also called anchors.

To put it simply, \b allows you to perform a ‘whole word only’ search as in \bword\b.

Prior to version 8 of PCRE (and 16.0 of Dyalog) this was true only for ASCII characters. So it worked only for the English language.

       ⍴'\bger\b'⎕S 0 ⊣'Kai Jägerßabc'
1

That's because both ä and ß are non-ANSI characters.

Now you can set the UCP option to 1 if you want Unicode characters to be taken into account as well:

      ⍴'\bger\b'⎕S 0 ⍠('UCP' 1)⊣'Kai Jägerßabc'
0

Now ä and ß no longer qualify as word boundaries.

5.21. Backreferences

Suppose you want to search for three digits:

      '[0-9]{3,}' ⎕R '⍈' ⊣ ' 3 digits: 123;'
 3 digits: ⍈;

But what if you want to ensure it's always the same digits?

For that you need back references:

      '([0-9])\1{2,}' ⎕R '⍈' ⊣ ' 3 digits: 333; 444; 123;'
 3 digits: ⍈; ⍈; 123;

The first two groups of digits are found while the last one is ignored — exactly what we want.

Notes:

Information

You can define and use up to 99 groups.

If you want to find only numbers that consist of exactly three digits which have to be the same then this would work:

      '([0-9])\1{2,}' ⎕R '⍈' ⊣ '333; 444; 123;'
⍈; ⍈; 123;

But be aware:

      '([0-9])\1{2,}' ⎕R '⍈' ⊣ '3333; 4444; 1234;'
⍈; ⍈; 1234;

To solve this problem you need to master look-arounds.

5.22. Look-ahead and look-behind (look-arounds)

We can use look-ahead and look-behind to solve a problem we ran into earlier with numbers. This did not really work because all dots got replaced when we wanted only those with digits to the right and the left being a match:

      '[\d.¯-]'⎕R'⍈'⊣'It''s 23.45 plus 99.12.'
It's ⍈⍈⍈⍈⍈ plus ⍈⍈⍈⍈⍈⍈

We don't want the last dot to be a match. Obviously we need to check the characters to the left and to the right of each dot.

Both look-ahead and look behind start with (?. A look-behind then needs a < while the look-ahead doesn't.

Both then need either a = for equal or a ! for not equal followed by the search token and finally a closing ). Hence (?<=\d) for the look-behind and (?=\d) for the look-ahead:

      '\d' '(?<=\d).(?=\d)'⎕R'⍈'⊣'It''s 23.45 plus 99.12.'
It's ⍈⍈⍈⍈⍈ plus ⍈⍈⍈⍈⍈.

That works! We use two expressions here: first we look for all digits and then we look for dots that have a digit to their right and their left.

Information

It’s important to realise that the current position does not change when a look-behind or a look-ahead is performed; that's why they are called zero-length assertions.

However, if you need ⎕S to return the start and the length of any matches then the result is unlikely to be what you are after:

      '\d' '(?<=\d).(?=\d)'⎕S 0 1 ⊣'It''s 23.45 plus 99.12.'
 5 1  6 1  7 1  8 1  9 1  16 1  17 1  18 1  19 1  20 1

We need an expression that identifies any vector of digits as one unit, no matter whether there is a dot between the digits or not:

      '\d+(?<=\d).(?=\d)\d+' ⎕R '⍈' ⊣ 'It''s 23.45 plus 99.12.'
It's ⍈ plus ⍈.
      '\d+(?<=\d).(?=\d)\d+' ⎕S 0 1 ⊣ 'It''s 23.45 plus 99.12.'
 5 5  16 5

That's better.

As mentioned earlier a look-ahead, as well as a look-behind, can be negated by using a ! rather than a =

Lets' try this. Assuming we look for x and y:

      'x(?<=y)' ⎕R'⍈' ⊣ 'abxycxd' ⍝ Exchange all "x" when followed by a "y"
ab⍈ycxd
      'x(?!y)' ⎕R'⍈' ⊣ 'abxycxd' ⍝ Exchange all "x" when NOT followed by a "y"
abxyc⍈d
      '(?<=x)y' ⎕R'⍈' ⊣ 'abxycyd' ⍝ Exchange all "y" when preceeded by an "x"
abx⍈cyd
      '(?<!x)y' ⎕R'⍈' ⊣ 'abxycyd' ⍝ Exchange all "y" when NOT preceeded by an "x"
abxyc⍈d

5.23. Transformation functions

Instead of providing a replace string one can also pass a function as operand to ⎕R (and ⎕S as well). This feature is particularly useful when not only pattern matching is required but also calculations based on the findings.

Look at this piece of CSS code:

p {
    /* font-size: 12px; */
    font-size: 12px;
    border: 1px solid black;
}

Though using “px” always was and still is quite common, these days you are advised to used “em” (and sometimes “rem”) instead. The reason is that “em” refers to the parent of the current object.

For example, this:

p {
    font-size: 0.9em;
}

means that the size is 90% oif the font size defined for <p>'s parent.

But there has to be at least one absolute definition, and that's usually to be found in the <body> tag:

body {
    font-size: 18px;
}
Information

If such a definition is missing then it falls back to the browser's default which these days usually is 16px.

The advantage is that in order to change the size of the fonts used you have to change one single value.

Now lets assume that we want to convert “px” into “em”. Just replacing “px” against “em” doesn't do the job, we also need to calculate the correct value (we ignore the inheritance problem here).

This is where a transformation functions come in handy:

     ∇ r←Change arg;v;em;baseFontSize
[1]    baseFontSize←18        ⍝ What's defined for body; that should be defined in "px".
[2]    :If 0=arg.PatternNum
[3]        r←arg.Match
[4]    :Else
[5]        v←⊃(//)⎕VFI¯2↓arg.Match
[6]        :If v∊1 2          ⍝ Those should be left alone.
[7]            r←arg.Match
[8]        :Else
[9]            em←0.01×⌊0.5+100×v÷baseFontSize
[10]           r←(⍕em),'em'
[11]       :EndIf
[12]   :EndIf
[13]  ⍝Done
     ∇

We assign the CSS code we've discussed earlier on a variable :

∆←''
∆,←⊂'p {'
∆,←⊂'    /* font-size: 12px; */'
∆,←⊂'    font-size: 12px;'
∆,←⊂'    border: 1px solid black;'
∆,←⊂'}'
∆←,¨∆

Note that the last statement is essential, otherwise you'll get a RANK ERROR: Invalid input source: the input vector must not contain scalars.

We want to investigate the right argument Change is going to get, therefore we set a stop vector on line 1 of Change:

      1 ⎕STOP 'Change'

Now we run this expression:

      ∆2←'/\*.*\*/' '\b[0-9]{1,3}px'⎕R Change⍠('DotAll' 1)('Mode' 'M')('Greedy' 0)⊣∆

The left operand of ⎕R is a two-element vector:

Because of the stop vector execution stops on line 1 of Change. That allows us to investigate the right argument:

      arg.(⊃{⍵ (⍎⍵)}¨↓⎕nl 2 9)
 Block       p {
                /* font-size: 12px; */
                font-size: 12px;
                border: 1px solid black;
             }
 BlockNum                                                                         0
 Lengths                                                                         22
 Match                                                       /* font-size: 12px; */
 Names
 Offsets                                                                          9
 Pattern                                                                   /\*.*\*/
 PatternNum                                                                       0
 ReplaceMode                                                                      1
 TextOnly                                                                         1

The right argument contains all the pieces of information that you might possibly need.

Now it's time to discuss the function Change in detail:

     ∇ r←Change arg;v;em;baseFontSize
[1]    baseFontSize←18        ⍝ What's defined for body; that should be defined in "px".
[2]    :If 0=arg.PatternNum
[3]        r←arg.Match
[4]    :Else
[5]        v←⊃(//)⎕VFI¯2↓arg.Match
[6]        :If v∊1 2          ⍝ Those should be left alone.
[7]            r←arg.Match
[8]        :Else
[9]            em←0.01×⌊0.5+100×v÷baseFontSize
[10]           r←(⍕em),'em'
[11]       :EndIf
[12]   :EndIf
[13]  ⍝Done
     ∇

“em” versus “rem”

There is not only “em” available, there is also “rem”. “em” are calculated in proportion to their parent. That's usually what you want.

Imagine a list (<ul> or <ol>). If you want to make the <li>'s font slightly smaller than the parent's font then specifying 0.85em; seems to be fine, right? Well, only as long as you do not have nested lists!

With deeply nested lists things get quickly out of hand because with every level the font gets smaller and smaller.

“rim” to the rescue! In opposite to “em” they refer to the base font size size; that's what was defined in the <body> tag. That way all lists use the same — slightly smaller — font size.

Transformation functions give you enormous power: you can do whatever you like.

5.24. Document mode

So far we have specified just a simple string as input. We can, however, pass a vector of strings as well. Look at this example:

      input←'He said: "Yes, that might' 'well be right." She answered: "So be it!"'

It's not a bad idea to think of the two elements of the input vector as ‘blocks’. Note that the first reported speech straddles the blocks.

By default the search engine operates in Line mode. That means each block is processed independently by the engine. You cannot search for \r (carriage return) in Line mode: the search engine will never see them.

In Mixed mode as well as Document mode you can search for \r because all blocks are passed at once. Naturally this also requires more memory than Line mode.

Let's do some tests:

      '".*"'⎕R'⍈'⍠('Greedy' 0)⊣input
 He said: "Yes, that might  well be right.⍈So be it!"
      '".*"'⎕R'⍈'⍠('Greedy' 0)('Mode' 'M')('DotAll' 1)⊣input
He said: ⍈ She answered: ⍈

Note that in order to specify ('DotAll' 1) it is necessary to set ('Mode' 'M'). ('Mode' 'D') would have worked in the same way. However, when it comes to ^ and $ then it makes a big difference:

5.25. Alternations

Suppose we want to match either the word cat or the word dog:

      'cat|dog' ⎕R '⍈'⊣ 'donkey, bird, cat, rat, dog'
donkey, bird, ⍈, rat, ⍈

The | has the lowest precedence of all RegEx operators. Therefore it first tries to match cat and only then dog.

However, be careful:

      'cat|catfish' ⎕R '⍈'⊣ 'donkey, bird, cat, rat, dog, catfish'
donkey, bird, ⍈, rat, dog, ⍈fish

This is not what we want. The reason for this is that once the string cat has matched the RegEx engine gives up because it was successful, therefore it does not see the need to check later options. It's said the engine is eager.

Sorting the alternatives by length gets us around this problem:

      'catfish|cat' ⎕R '⍈'⊣ 'donkey, bird, cat, rat, dog, catfish'
donkey, bird, ⍈, rat, dog, ⍈

However, in real life we would put word boundaries to good use, avoiding the problem altogether:

      '\bcat\b|\bcatfish\b' ⎕R '⍈'⊣ 'donkey, bird, cat, rat, dog, catfish'
donkey, bird, ⍈, rat, dog, ⍈

Or even shorter:

      '\b(cat|catfish)\b' ⎕R '⍈'⊣ 'donkey, bird, cat, rat, dog, catfish'
donkey, bird, ⍈, rat, dog, ⍈

Note the parentheses used here for grouping.

5.26. Optional items

The ? makes the preceeding token optional. If you want to find either November or its shortcut Nov:

      'Nov(ember)?' ⎕R '⍈'⊣ '...October, November, December; ... Oct, Nov, Dec'
...October, ⍈, December; ... Oct, ⍈, Dec

For plurals:

      'cars?' ⎕R '⍈'⊣ 'car, boat, plain, cars, boats, plains'
⍈, boat, plain, ⍈, boats, plains

5.27. Extract what's between HTML tags

Suppose we have a piece of HTML code and are interested in any text between anchor tags (<a>). Suppose also we know there is no other tag inside the <a>, just simple text.

Now an <a> tag has always either an href="..." or an id="..." because otherwise it has no purpose. <!– FIXME or name= –>

So it should be safe to say:

      '<a.*>.*<'⎕R'⍈'⊣'This <a href="http://aplwiki.com">is a link</a>, really!'
This ⍈, really!

Works, right?

Well, yes, but it also works on this:

      txt←'This <abbr title="FooGoo"><a href="#page">is a link</a></abbr>'
      '<a.*>.*</a>'⎕R'⍈'⊣txt
This ⍈</abbr>

That might come as a nasty surprise but when you think it through it's obvious why that is: the expression <a.*> does indeed catch not only <a but also <abbr>. Shows how important it is to be precise.

We can work around this quite easily: because any <a> tag must have at least one attribute in order to make sense there must be a space after the a; therefore we can rewrite the expression as follows:

      '<a .*>.*</a>'⎕R'⍈'⊣txt
This <abbr title="FooGoo">⍈</abbr>

What a difference a space can make!

But somebody might put in an <a> tag without any attribute at all, and then this would not work. So we are still in need not for a better but a perfect solution.

Here it is:

      '<a\b[^>]*>(.*?)</a>'⎕R'⍈'⊣'This <a>is a link</a>; more: <a id="_2">Foo</a>'
This ⍈; more: ⍈

Notes:

6. Warnings

6.1. The . character

Be very careful whith the . in RegEx: because it matches every character except newline (and with ('DotAll' 1) even newline) it can produce unwanted results, in particular with ('Greedy' 1) but not restricted to that.

Because it's so powerful it allows you to be lazy: you write a RegEx and it matches everything that you want it to match, but it might always match everything, including stuff it shouldn't!

To illustrate the point let's assume that we want to match a date in a text vector in the international date format (yyyy-mm-dd). The naive approach with a dot works fine:

      '\d\d\d\d.\d\d.\d\d'⎕S 0 ⊣'1988-02-03'
0

Not really:

      '\d\d\d\d.\d\d.\d\d'⎕S 0 ⊣'1988/02/03'
0

While this might be acceptable because it seems to give the user the freedom to use a different separator the following example is certainly not acceptable:

      '\d\d\d\d.\d\d.\d\d'⎕S 0 ⊣'1988020312'
0

Its's much better to specify what's excepted as separator explicitly:

      '\d\d\d\d[-./ ]\d\d[-./ ]\d\d'⎕S 0⊣'1988/02/03'
0

Even this has it's problems:

      '\d\d\d\d[-./ ]\d\d[-./ ]\d\d'⎕S 0⊣'1988 02/03'
0
      '\d\d\d\d[-./ ]\d\d[-./ ]\d\d'⎕S 0⊣'1988-99-99'
0

Whether that's acceptable or not depends on the application.

6.2. Assumptions

One of the greatest problems in programming is making assumptions and not document them. Or worse, not even being aware of your assumptions.

The above is an example. Imagine these two different scenarios:

  1. You want to extract everything from a log file that's a date. You know that every record, if it carries a date at all, will start with the date, and you can safely assume that the dates are correctly saved in international date format.
  2. You allow the user to enter her date of birth in a dialog box.

In the first case you can take a relaxed approach because you know all dates are valid and follow precise rules while in the second you have to be meticulous because otherwise you will accept and save rubbish sooner rather than later.

6.3. Empty vectors

Given this variable:

      v←''  'A paragraph.'  ''

This should not change anything because it just attempts to replace any <TAB> character by four spaces yet the trailing empty vector disappears:

      Display '\t'⎕R(4⍴' ')⍠('Mode' 'M')⊣v
┌→─────────────────────────┐
│ ┌⊖┐ ┌→─────────────────┐ │
│ │ │ │A paragraph.      │ │
│ └─┘ └──────────────────┘ │
└∊─────────────────────────┘

This is because the three-element vector is transformed into a single document <CRLF>A paragraph.<CRLF> which is then passed in its entirety to PCRE, the underlying RegEx engine. PCRE has only generated two lines of output from this so the result is a two-element vector.

If you want a stricter correspondence between input and output you need to process the elements separately, e.g.:

      ]display '\t'⎕R(4⍴' ')⍠('Mode' 'M')¨v
┌→─────────────────────────────┐
│ ┌⊖┐ ┌→─────────────────┐ ┌⊖┐ │
│ │ │ │A paragraph.      │ │ │ │
│ └─┘ └──────────────────┘ └─┘ │
└∊─────────────────────────────┘

That's what happens in version 16.0. Be aware that this might change in a later version of Dyalog.

7. Miscellaneous

7.1. Tests

Complex regular expressions are hard to read and maintain. Document them intensively and cover them with exhaustive test cases.

At first this might seem overkill, but as usual tests will prove to be useful when you need to…

7.2. Performance

Don't expect regular expressions to be faster than a tailored APL solution; expect them to be slightly slower.

However, many regular expressions, like finding a simple string in another simple string or uppercasing or lowercasing characters are converted by the interpreter into a native (faster) APL expression ( and ⌶ 819 respectively) anyway.

7.3. Helpful stuff

Online Tutorial

A website that explores Regular Expressions in detail:

https://www.regular-expressions.info/tutorial.html

From the author of RegExBuddy.

RegexBuddy

Software that helps interpret or build regular expressions:

https://www.regexbuddy.com/

Book reviews

The aforementioned website comes with detailed book reviews:

https://www.regular-expressions.info/hipowls.html

Chapter 18:

Managing source code with “acre-desktop”

1. Overview

Chapter 19:

Graphical user interface

1. Introduction

Modern graphical user interfaces (GUI) are a wonder. GUI conventions are so widely known it is now unremarkable for people to start using applications without prior training, expecting the software to make clear what they need to do.

This is a high standard to meet, and writing UIs is a deep art. The primary platforms for professional writers of UIs are currently a combination of HTML 5 and JavaScript (HTML/JS) and Windows Presentation Foundation (WPF). These are rich platforms, which enable effective and attractive UIs to be written.

The high quality of these UIs is particularly important for mass-market software, where users are unskilled and unsupported.

HTML/JS and WPF have a high learning threshold. There is much to be mastered before you can write good UIs on these platforms.

You have an alternative. The GUI tools native to Dyalog support perfectly workmanlike GUIs. They exploit and extend your existing knowledge of Dyalog. If you are producing high-value software for a few users, rather than software for casual use by millions, a native Dyalog GUI might be your best platform.

You can still run your application from within a browser if you wish to: Amazon offers the “AppStream” [1] service allowing exactly that.

1.1. ⎕WC versus ⎕NEW

⎕NEW came much later than ⎕WC. Is ⎕NEW replacing ⎕WC? Certainly not. It's just an alternative. Both have pros and cons, but after having tried them both in real-world projects we settle for ⎕WC. Here's why:

Pro ⎕NEW:

Pro ⎕WC:

We hope that Dyalog will eventually use ⎕DF for the Event Viewer for GUI controls created by ⎕NEW. However, for the time being the disadvantages on ⎕NEW are severe, therefore we settle on ⎕WC.

1.2. A simple example

Creating a GUI form in Dyalog could hardly be simpler:

      ∆Form←⍎'MyForm'⎕WC 'Form'
      ∆Form.Caption←'Hello world'

Hello world form

To the form we add controls, set callback functions to run when certain events occur, and invoke the form's Wait method or ⎕DQ to hand control over to the user. See the Dyalog for Microsoft Windows Interface Guide for details and tutorials.

Experience has shown that it is a good idea to keep references to all controls, as well as any variables that belong logically to those controls, within a namespace. Since this is a temporary namespace — it will exist only as long as the application is running — we use an unnamed namespace for this.

We create the controls with names but generate references for them which we assign to the very same names within that unnamed namespace. The concept will become clear when we create an example.

1.3. A simple GUI with native Dyalog forms

We are going to implement a sample form that looks like this:

Find and replace

Obviously this is a GUI that allows a programmer to search the current workspace.

We would like to emphasise that it is a very good idea to keep the GUI and its code separate from the application. Even if you think that you are absolutely sure that you will never go for a different — or additional — GUI, you should still keep it separate.

Over and over again in real life assumptions such as This app will be used only for a year or two or This app will never use a different type of GUI have proven to be wrong.

Besides, testing the business logic of an application is ar easier when it is separated from the GUI.

Better prepare for it from the start, in particular because it takes little effort to do this early, but becomes a major effort if you need to change or add a GUI later.

In this chapter we will construct the GUI shown above as an example, and we will keep everything unrelated to the GUI in a namespace BusinessLogic.

1.4. The goal

Our aim is simple: code that is easy to understand and easy to change.

For that we do the following:

While reading this you might think something along the lines of: I've never heard a programmer say she strives for unmaintainable code yet, most GUI code is ugly spaghetti. Stay with us!

2. The implementation

2.1. Prerequisites

We create a top-level namespace that will host our application:

      'MyApp' ⎕NS ''

We need three subspaces within MyApp:

      'GUI'#.MyApp.⎕ns''
      'BusinessLogic'#.MyApp.⎕ns''
      'GuiUtils'#.MyApp.⎕ns''

2.2. The function GUI.Run

We start with the function Run:

     ∇ {N}←Run testFlag
[1]    ⎕IO←1 ⋄ ⎕ML←1
[2]    'Invalid right argument'⎕SIGNAL 11/⍨~(⊂testFlag)∊0 1
[3]    N←Init ⍬
[4]    N.∆TestFlag←testFlag
[5]    N←CreateGUI N
[6]    :If 0=testFlag
[7]        N.SearchFor U.DQ N.∆Form
[8]        Shutdown N
[9]    :EndIf
[10]  ⍝Done
     ∇

What this function does:

Notes:

2.3. The function GUI.Init

Next we introduce the Init function:

     ∇ N←Init dummy
[1]    U←GuiUtils
[2]    N←U.CreateNamespace
[3]    N.∆Buttons←''
[4]    N.∆Labels←N.⎕NS''
[5]    N.(∆V_Gap ∆H_Gap)←5 10
[6]    N.∆Posn←80 30
[7]    N.∆Size←600 800
[8]    N.InputFont←'InputFont'⎕WC'Font'('PName' 'APL385 Unicode')('Size' 17)
[9]   ⍝Done
     ∇

GuiUtils is an ordinary namespace that contains some functions. We will discuss them when we need them.

2.4. The function GuiUtils.CreateNamespace

CreateNamespace is used to create the namespace N:

     ∇ r←CreateNamespace
[1]    r←⎕NS''
[2]    r.⎕FX'r←∆List' 'r←{⍵,[1.5]⍎¨⍵}'' ''~¨⍨↓⎕NL 2 9'
[3]   ⍝Done
     ∇

This function creates an unnamed namespace and populates it with a function ∆List which returns a matrix with two columns:

Column Contains
[;1] Name of a variable or reference
[;2] The value of that name

After Init runs the N namespace does not yet contain any GUI controls but it does contains some variables that will define certain properties of the GUI:

      N.∆List
 InputFont                         #._MyApp.MyApp.GUI.InputFont
 ∆Buttons
 ∆H_Gap                                                      10
 ∆Labels    #._MyApp.MyApp.GUI.GuiUtils.[Namespace].[Namespace]
 ∆Posn                                                    80 30
 ∆Size                                                  600 800
 ∆V_Gap                                                       5

2.5. The function GUI.CreateGUI

This function calls all the functions that create controls. They all start their names with Create.

     ∇ N←CreateGUI N
[1]    N←CreateMainForm N
[2]    N←CreateSearch N
[3]    N←CreateStartLookingHere N
[4]    N.∆Groups←⎕NS''
[5]    N←CreateOptionsGroup N
[6]    N←CreateObjectTypesGroup N
[7]    N←CreateScanGroup N
[8]    N←CreateRegExGroup N
[9]    {⍵.Size←(⌈/1⊃¨⍵.Size),¨1+2⊃¨⍵.Size}'Group'⎕WN N.∆Form
[10]   N←CreateList N
[11]   N←CreatePushButtons N
[12]   N←CreateHiddenButtons N
[13]   N.HitList.Size[1]-←(2×N.∆V_Gap)+N.∆Form.Size[1]-N.Find.Posn[1]
[14]   N.(⍎¨↓⎕NL 9).onKeyPress←⊂'OnKeyPress'
[15]   N.∆WriteToStatusbar←N∘{⍺.Statusbar.StatusField1.Text←⍵ ⋄ 1:r←⍬}
[16]   N.∆Form.onConfigure←'OnConfigure'(335,CalculateMinWidth N)
[17]  ⍝Done
     ∇

Notes:

That callback is very important: without it any Configure event would cause a VALUE ERROR, which in turn would make you lose your workspace because there are just too many of them. So we introduce it straight away; we cannot forget it.

2.6. The function GUI.OnConfigure'

∇ OnConfigure←{⍵[1 2 3 4],(⍺[1]⌈⍵[5]),(⍺[2]⌈⍵[6])}
∇

Since the Tracer cannot (currently) step into one line dfns, defining the OnConfigure callback as a one liner is exceedingly useful as it allows you to avoid stepping into the callback, which is itself likely to result in even more configure events being generated. As already mentioned it is absolutely essential that this function is a one-liner because that makes the Tracer ignore this function.

2.7. All the GUI.Create* functions

Although we list all functions here you might not necessarily follow us on each of them in detail, but you should at least keep reading until you reach GUI.AdjustGroupSize.

However, we suggest to scan through them rather than skipping them and carrying on with the callback functions.

2.7.1. The function CreateMainForm

     ∇ N←CreateMainForm N;∆
[1]    ∆←⊂'Form'
[2]    ∆,←⊂'Coord' 'Pixel'
[3]    ∆,←⊂'Caption' 'MyApp'
[4]    ∆,←⊂'Posn'N.∆Posn
[5]    ∆,←⊂'Size'N.∆Size
[6]    N.∆Form←⍎'Form'⎕WC ∆
[7]    N.∆Form.N←N
[8]    N←CreateMenubar N
[9]    N←CreateStatusbar N
[10]  ⍝Done
     ∇

One statement needs discussion: line [7] assigns N to N.∆Form.N – what for?

This allows us to find N with ease: we know that there is always a reference to N available inside the main form. When we introduce the callback functions we will need N is almost all of them, and this will make is easy for them to find it.

2.7.1.1. Collecting properties

Note that the function collects properties which are assigned to a local variable ; we don't attempt to give it a proper name because we use it just for collecting stuff.

Why are we not assigning the properties in one go? Something like this:

N.∆Form←⍎'Form'⎕WC 'Form'('Coord' 'Pixel')('Caption' 'MyApp')('Posn'∆Posn)('Size'∆Size)

Are shorter programs not better? They are, but there are exceptions.

Besides being more readable, having just one property on a line has the big advantage of allowing us to skip the line in the Tracer if we wish to. That is particularly pleasant when we don't want something to be executed like ('Visible' 0) or ('Active' 0). If they are part of a lengthy line, well, you get the idea.

2.7.1.2. Collecting controls

We are not using a visual designer to create the GUI; we use APL code. Wherever possible, we calculate position and size dynamically, or assign constants.

The name we use as left argument of ⎕WC is also used within N when we assign the reference that is created with the primitive on the (shy) result of ⎕WC. That's what a statement like this does:

      N.∆Form←⍎'Form'⎕WC ∆

CreateMainForm calls two functions which we introduce next.

2.7.2. The function GUI.CreateMenubar

     ∇ N←CreateMenubar N;TAB;∆
[1]    TAB←⎕UCS 9
[2]    N.∆Menubar←⍎'∆Menubar'N.∆Form.⎕WC⊂'Menubar'
[3]    N.∆Menubar.FileMenu←⍎'FileMenu'N.∆Menubar.⎕WC'Menu'('Caption' '&File')
[4]    ∆←⊂'MenuItem'
[5]    ∆,←⊂'Caption'('Quit',TAB,'Escape')
[6]    ∆,←⊂'Accelerator'(27 0)
[7]    N.∆Menubar.Quit←⍎'Quit'N.∆Menubar.FileMenu.⎕WC ∆
[8]    N.∆Menubar.Quit.onSelect←1
[9]   ⍝Done
     ∇

Note that we assign the result of ⍎'∆Menubar'N.∆Form.⎕WC⊂'Menubar' (which is actually the menubar on our form) to ∆Menubar rather than Menubar as you might have expected.

The reason is that we do not assign the “Menu” and “MenuItem” and “Separator” objects to N but to the menubar itself; because it's a static menu we don't want matters to be blurred, so we keep them separate, similar to the labels.

2.7.3. The function GUI.CreateStatusbar

     ∇ N←CreateStatusbar N;∆
[1]    N.Statusbar←⍎'Statusbar'N.∆Form.⎕WC⊂'Statusbar'
[2]    ∆←⊂'StatusField'
[3]    ∆,←⊂'Coord' 'Prop'
[4]    ∆,←⊂'Posn'(0 0)
[5]    ∆,←⊂'Size'(⍬ 100)
[6]    ∆,←⊂'Attach'('Bottom' 'Left' 'Bottom' 'Right')
[7]    N.StatusField1←⍎'StatusField1'N.Statusbar.⎕WC ∆
[8]   ⍝Done
     ∇

2.7.4. The function GUI.CreateSearch

     ∇ N←CreateSearch N;∆
[1]    ∆←⊂'Label'
[2]    ∆,←⊂'Posn'N.(∆V_Gap ∆H_Gap)
[3]    ∆,←⊂'Caption' '&Search for:'
[4]    ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Left')
[5]    N.∆Labels.SearchFor←⍎'SearchFor'N.∆Form.⎕WC ∆
[6]
[7]    ∆←⊂'Edit'
[8]    ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.∆Labels.SearchFor)N.∆H_Gap)
[9]    ∆,←⊂'Size'(⍬(N.∆Form.Size[2]-2×N.∆H_Gap))
[10]   ∆,←⊂'FontObj'N.InputFont
[11]   ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Right')
[12]   N.SearchFor←⍎'SearchFor'N.∆Form.⎕WC ∆
[13]  ⍝Done
     ∇

This function creates the label Search for and the associated edit field.

Notes:

2.7.5. The function GuiUtils.AddPosnAndSize

     ∇ AddPosnAndSize←{
[1]        +⌿↑⍵.(Posn Size)
[2]    }
     ∇

Not much code, but very helpful and used over and over again, so it makes sense to make it a function.

It just makes position and size a matrix and sums up the rows. That is exactly what we need for positioning the Edit control vertically. Its horizontal position is of course defined by N.∆H_Gap.

2.7.6. The function GUI.CreateStartLookingHere

 N←CreateStartLookingHere N;∆
 ∆←⊂'Label'
 ∆,←⊂'Posn'((N.∆V_Gap+⊃U.AddPosnAndSize N.SearchFor)N.∆H_Gap)
 ∆,←⊂'Caption' 'Start &looking here:'
 ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Left')
 N.∆Labels.StartLookingHere←⍎'StartLookingHere'N.∆Form.⎕WC ∆

 ∆←⊂'Edit'
 ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.∆Labels.StartLookingHere)N.∆H_Gap)
 ∆,←⊂'Size'(⍬(N.∆Form.Size[2]-2×N.∆H_Gap))
 ∆,←⊂'FontObj'N.InputFont
 ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Right')
 N.StartLookingHere←⍎'StartLookingHere'N.∆Form.⎕WC ∆
⍝Done

Note that this time the vertical position of the label is defined by the total of the Posn and Size of the Search for edit control plus N.∆V_Gap.

2.7.7. The function GUI.CreateOptionsGroup

     ∇ N←CreateOptionsGroup N;∆
[1]    ∆←⊂'Group'
[2]    ∆,←⊂'Caption' 'Options'
[3]    ∆,←⊂'Posn'((N.∆V_Gap+⊃U.AddPosnAndSize N.StartLookingHere),N.∆H_Gap)
[4]    ∆,←⊂'Size'(300 400)
[5]    ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Left')
[6]    N.∆Groups.OptionsGroup←⍎'OptionsGroup'N.∆Form.⎕WC ∆
[7]
[8]    ∆←⊂'Button'
[9]    ∆,←⊂'Style' 'Check'
[10]   ∆,←⊂'Posn'(3 1×N.(∆V_Gap ∆H_Gap))
[11]   ∆,←⊂'Caption' '&Match case'
[12]   N.MatchCase←⍎'MatchCase'N.∆Groups.OptionsGroup.⎕WC ∆
[13]
[14]   ∆←⊂'Button'
[15]   ∆,←⊂'Style' 'Check'
[16]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.MatchCase),N.∆H_Gap)
[17]   ∆,←⊂'Caption' 'Match &APL name'
[18]   N.MatchAPLname←⍎'MatchAPLname'N.∆Groups.OptionsGroup.⎕WC ∆
[19]
[20]   AdjustGroupSize N.∆Groups.OptionsGroup
[21]  ⍝Done
     ∇

The group as such is assigned to OptionsGroup inside N.∆Groups as discussed earlier.

The function calls AdjustGroupSize which we therefore need to introduce.

2.7.8. The function GUI.AdjustGroupSize

     ∇ AdjustGroupSize←{
[1]    ⍝ Ensures that the group is just big enough to host all its children
[2]        ⍵.Size←N.(∆H_Gap ∆V_Gap)+⊃⌈/{+⌿↑⍵.(Posn Size)}¨⎕WN ⍵
[3]        1:r←⍬
[4]    }
     ∇

The comment in line [1] tells it all.

Note that the system function ⎕WN gets a reference as right argument rather than a name; that's important because in that case ⎕WN returns references as well.

2.7.9. The function GUI.CreateObjectTypesGroup

     ∇ N←CreateObjectTypesGroup N;∆
[1]    ∆←⊂'Group'
[2]    ∆,←⊂'Caption' 'Object &types'
[3]    ∆,←⊂'Posn'({⍵.Posn[1],(2×N.∆V_Gap)+2⊃U.AddPosnAndSize ⍵}N.∆Groups.OptionsGroup)
[4]    ∆,←⊂'Size'(300 400)
[5]    ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Left')
[6]    N.∆Groups.ObjectTypes←⍎'ObjectTypes'N.∆Form.⎕WC ∆
[7]
[8]    ∆←⊂'Button'
[9]    ∆,←⊂'Style' 'Check'
[10]   ∆,←⊂'Posn'(3 1×N.(∆V_Gap ∆H_Gap))
[11]   ∆,←⊂'Caption' 'Fns, opr and scripts'
[12]   N.FnsOprScripts←⍎'FnsOprScripts'N.∆Groups.ObjectTypes.⎕WC ∆
[13]
[14]   ∆←⊂'Button'
[15]   ∆,←⊂'Style' 'Check'
[16]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.FnsOprScripts),N.∆H_Gap)
[17]   ∆,←⊂'Caption' 'Variables'
[18]   N.Variables←⍎'Variables'N.∆Groups.ObjectTypes.⎕WC ∆
[19]
[20]   ∆←⊂'Button'
[21]   ∆,←⊂'Style' 'Check'
[22]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.Variables),N.∆H_Gap)
[23]   ∆,←⊂'Caption' 'Name list &only (⎕NL)'
[24]   N.NameList←⍎'NameList'N.∆Groups.ObjectTypes.⎕WC ∆
[25]
[26]   AdjustGroupSize N.∆Groups.ObjectTypes
[27]  ⍝Done
     ∇

2.7.10. The function GUI.CreateScanGroup

      ∇ N←CreateScanGroup N;∆
[1]    ∆←⊂'Group'
[2]    ∆,←⊂'Caption' 'Scan... '
[3]    ∆,←⊂'Posn'({⍵.Posn[1],(2×N.∆V_Gap)+2⊃U.AddPosnAndSize ⍵}N.∆Groups.ObjectTypes)
[4]    ∆,←⊂'Size'(300 400)
[5]    ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Left')
[6]    N.∆Groups.ScanGroup←⍎'ScanGroup'N.∆Form.⎕WC ∆
[7]
[8]    ∆←⊂'Button'
[9]    ∆,←⊂'Style' 'Check'
[10]   ∆,←⊂'Posn'(3 1×N.(∆V_Gap ∆H_Gap))
[11]   ∆,←⊂'Caption' 'APL'
[12]   N.APL←⍎'APL'N.∆Groups.ScanGroup.⎕WC ∆
[13]
[14]   ∆←⊂'Button'
[15]   ∆,←⊂'Style' 'Check'
[16]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.APL),N.∆H_Gap)
[17]   ∆,←⊂'Caption' 'Comments'
[18]   N.Comments←⍎'Comments'N.∆Groups.ScanGroup.⎕WC ∆
[19]
[20]   ∆←⊂'Button'
[21]   ∆,←⊂'Style' 'Check'
[22]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.Comments),N.∆H_Gap)
[23]   ∆,←⊂'Caption' 'Text'
[24]   N.Text←⍎'Text'N.∆Groups.ScanGroup.⎕WC ∆
[25]
[26]   AdjustGroupSize N.∆Groups.ScanGroup
[27]  ⍝Done
     ∇

2.7.11. The function GUI.CreateRegExGroup

     ∇ N←CreateRegExGroup N;∆
[1]    ∆←⊂'Group'
[2]    ∆,←⊂'Caption' 'RegEx'
[3]    ∆,←⊂'Posn'({⍵.Posn[1],N.∆V_Gap+2⊃U.AddPosnAndSize ⍵}N.∆Groups.ScanGroup)
[4]    ∆,←⊂'Size'(300 400)
[5]    ∆,←⊂'Attach'('Top' 'Left' 'Top' 'Left')
[6]    N.∆RegEx←⍎'ObjectTypes'N.∆Form.⎕WC ∆
[7]
[8]    ∆←⊂'Button'
[9]    ∆,←⊂'Style' 'Check'
[10]   ∆,←⊂'Posn'(2 1×N.(∆V_Gap ∆H_Gap))
[11]   ∆,←⊂'Caption' 'Is RegE&x'
[12]   N.IsRegEx←⍎'IsRegEx'N.∆RegEx.⎕WC ∆
[13]   N.IsRegEx.onSelect←'OnToggleIsRegEx'
[14]
[15]   ∆←⊂'Button'
[16]   ∆,←⊂'Style' 'Check'
[17]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.IsRegEx),4×N.∆H_Gap)
[18]   ∆,←⊂'Caption' 'Dot&All'
[19]   N.DotAll←⍎'DotAll'N.∆RegEx.⎕WC ∆
[20]
[21]   ∆←⊂'Button'
[22]   ∆,←⊂'Style' 'Check'
[23]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.DotAll),4×N.∆H_Gap)
[24]   ∆,←⊂'Caption' '&Greedy'
[25]   N.Greedy←⍎'Greedy'N.∆RegEx.⎕WC ∆
[26]
[27]   AdjustGroupSize N.∆RegEx
[28]  ⍝Done
     ∇

This function ensures both DotAll and Greedy are indented, to emphasise they are available only when the Is RegEx check box is ticked.

The callback OnToggleRegEx will toggle the Active property of these two check boxes accordingly.

2.7.12. The function GUI.CreateList

     ∇ N←CreateList N;∆;h
[1]    ∆←⊂'ListView'
[2]    h←⊃N.∆V_Gap+U.AddPosnAndSize N.MatchCase.##
[3]    ∆,←⊂'Posn'(h,N.∆H_Gap)
[4]    ∆,←⊂'Size'((N.∆Form.Size[1]-h),N.∆Form.Size[2]-N.∆H_Gap×2)
[5]    ∆,←⊂'ColTitles'('Name' 'Location' 'Type' '⎕NS' 'Hits')
[6]    ∆,←⊂'Attach'('Top' 'Left' 'Bottom' 'Right')
[7]    N.HitList←⍎'HitList'N.∆Form.⎕WC ∆
[8]   ⍝Done
     ∇

2.7.13. The function GUI.CreatePushButtons

     ∇ N←CreatePushButtons N;∆
[1]    N.∆Buttons←''
[2]    ∆←⊂'Button'
[3]    ∆,←⊂'Caption' 'Find'
[4]    ∆,←⊂'Size'(⍬ 120)
[5]    ∆,←⊂'Default' 1
[6]    ∆,←⊂'Attach'(4⍴'Bottom' 'Left')
[7]    N.∆Buttons,←N.Find←⍎'Find'N.∆Form.⎕WC ∆
[8]    N.Find.Posn←(N.∆Form.Size[1]-N.Find.Size[1]+N.Statusbar.Size[1]+N.∆V_Gap),N.∆V_Gap
[9]    N.Find.onSelect←'OnFind'
[10]
[11]   ∆←⊂'Button'
[12]   ∆,←⊂'Caption' 'Replace'
[13]   ∆,←⊂'Size'(⍬ 120)
[14]   ∆,←⊂'Active' 0
[15]   ∆,←⊂'Attach'(4⍴'Bottom' 'Left')
[16]   N.∆Buttons,←N.Replace←⍎'Replace'N.∆Form.⎕WC ∆
[17]   N.Replace.Posn←(N.Find.Posn[1]),N.∆V_Gap+2⊃U.AddPosnAndSize N.Find
[18]  ⍝Done
     ∇

Note that the Find button gets a callback OnFind assigned to the Select event. That's the real work horse.

On callbacks

Rather than doing all the hard work in the callback we could have assigned a 1 to N.Find.onSelect (so that clicking the button quits ⎕DQ or Wait) and doing the hard work after that. At first glance there seems to be little difference between the two approaches.

However, if you want to test your GUI automatically then you must execute the 'business logic' in a callback and avoid calling ⎕DQ or Wait altogether.

That's the reason why our Run function expects a Boolean right argument, and that it's named testFlag. If it's a 1 then MyApp is running in test mode, and neither U.DQ nor Shutdown — which would close down the GUI — are executed.

That allows us in test mode to…

  1. call Run
  2. populate the Search for and Start looking here fields
  3. “click” the Find button programmatically
  4. populate the “Search for” and “Start looking here” fields
  5. “click” the “Find” button programmatically
  6. check the contents of N.HitList

2.7.14. The function GUI.CreateHiddenButtons

     ∇ N←CreateHiddenButtons N;∆
[1]    ∆←⊂'Button'
[2]    ∆,←⊂'Caption' 'Resize (F12)'
[3]    ∆,←⊂'Size'(0 0)
[4]    ∆,←⊂'Posn'(¯5 ¯5)
[5]    ∆,←⊂'Attach'(4⍴'Top' 'Left')
[6]    ∆,←⊂'Accelerator'(123 0)
[7]    N.Resize←⍎'Resize'N.∆Form.⎕WC ∆
[8]    N.Resize.onSelect←'OnResize'
[9]   ⍝Done
     ∇

Note that this button has no size ((0 0)) and is positioned outside the GUI. That means it is invisible to the user, and she cannot click it as a result of that. What's the purpose of such a button?

Well, it has an accelerator key attached to it which, according to the caption, is F12. This is an easy and straightforward way to implement a PF-key without overloading any “onKeyPress” callback.

It also makes it easy to disable F12: just execute N.Resize.Active←0.

2.8. The function GuiUtils.GetRef2n

We introduce this function here because almost all callbacks — which we will introduce next — will call GetRef2n.

Earlier on we saw that a reference to N was assigned to N.∆form.N. Now all callbacks, by definition, get a reference pointing to the control the callback is associated with as the first element of its right argument.

We also know that the control is owned by the main form, either directly, like Search for, or indirectly like the DotAll checkbox which is owned by the RegEx group, which in turn is owned by the main form.

That means that in order to find N we just need to check whether it exists at the current level. If not we go up one level (with .##) and try again.

GetRef2n is doing just that with a recursive call to itself until it finds N:

     ∇ GetRef2n←{
[1]        9=⍵.⎕NC'N':⍵.N
[2]        ⍵≡⍵.##:''           ⍝ Can happen in context menus, for example
[3]        ∇ ⍵.##
[4]    }
     ∇

Of course this means that you should not use the name N — or whatever name you prefer instead for this namespace — anywhere in the hierarchy.

Note that line [2] is an insurance against GetRef2n being called inside a callback that is associated with a control that is not owned by the main form. Of course that should not happen – because it makes no sense – but if you do it by accident then without that line the function would call itself recursively forever.

2.9. The callback functions

2.9.1. The function GUI.OnKeyPress

      ∇ OnKeyPress←{
[1]        (obj key)←⍵[1 3]
[2]        N←U.GetRef2n obj
[3]        _←N.∆WriteToStatusbar''
[4]        'EP'≢key:1                ⍝ Not Escape? Done!
[5]        _←2 ⎕NQ N.∆Form'Close'    ⍝ Close the main form...
[6]        0                         ⍝ ... and suppress the <esacape> key.
[7]    }
     ∇

This function just handles the Esc key.

2.9.2. The function GUI.OnToggleIsRegEx

     ∇ OnToggleIsRegEx←{
[1]        N←U.GetRef2n⊃⍵
[2]        N.(DotAll Greedy).Active←~N.IsRegEx.State
[3]        ⍬
[4]    }
     ∇

This callback toggles the Active property of both DotAll and Greedy so that they are active only when the content of Search for is to be interpreted as a regular expression.

2.9.3. The function GUI.OnResize

      ∇ OnResize←{
[1]        N←⎕NS''
[2]        list←CollectControls(⊃⍵).##
[3]        N.∆Form←(⊃⍵).##
[4]        width←CalculateMinWidth N.∆Form.N
[5]        ⎕NQ N.∆Form,(⊂'Configure'),N.∆Form.Posn,(N.∆Form.Size[1]),width
[6]    }
     ∇

This function makes sure that the width of the GUI is reduced to the minimum required to display all groups properly.

2.9.4. The function GUI.OnFind

     ∇ r←OnFind msg;N
[1]    r←0
[2]    N←U.GetRef2n⊃msg
[3]    N.∆WriteToStatusbar''
[4]    :If 0∊⍴N.SearchFor.Text
[5]        Dialogs.ShowMsg N'"Search for" is empty - nothing to look for...'
[6]    :ElseIf 0∊⍴N.StartLookingHere.Text
[7]        Dialogs.ShowMsg N'"Start looking here" is empty?!'
[8]    :ElseIf 9≠⎕NC N.StartLookingHere.Text
[9]    :AndIf (,'#')≢,N.StartLookingHere.Text
[10]       Dialogs.ShowMsg N'Contents of "Start looking here" is not a namespace'
[11]   :Else
[12]       Find N
[13]   :EndIf
[14]  ⍝Done
     ∇

The callback performs some checks and either puts an error message on display by calling a function Dialogs.ShowMsg or executes the Find function, providing N as the right argument.

Note that Dialog.ShowMsg follows exactly the same principles we have outlines in this chapter, so we do not discuss it in detail, but you can download the code and look at it if you want to.

One thing should be pointed out however: the form created by Dialog.ShowMsg is actually a child of our main form.

That gets us around a nasty problem: when it's not a child of the main form and you give the focus to another application which then hides completely the form created by Dialog.ShowMsg but not all of the main form, then a click on the main form should bring the application to the front, with the focus on Dialogs.ShowMsg, but that doesn’t happen by default.

By making it a child of the main form we enforce this behaviour. We don't want the user to believe the application has stopped working just because the form is hidden by another application window.

2.10. The function GUI.Find

This is the real work horse:

     ∇ Find←{
[1]        N←⍵
[2]        G←CollectData N
[3]        was←N.∆Buttons.Active
[4]        N.∆Buttons.Active←0
[5]        _←N.∆WriteToStatusbar'Searching...'
[6]        N.∆Result←(noOfHits noOfObjects cpuTime)←##.BusinessLogic.Find G
[7]        N.∆Buttons.Active←was
[8]        txt←(⍕noOfHits),' hits in ',(⍕noOfObjects),' objects. Search time ',(⍕cpuTime),' seconds.'
[9]        _←N.∆WriteToStatusbar txt
[10]       1:r←⍬
[11]   }
     ∇

An important thing to discuss is the function CollectData. We want our 'business logic' to be independent from the GUI. So we don't want anything in ##.BusinessLogic to access the N namespace.

But it needs access to the data entered and decisions made by the user on the GUI. So we collect all the data and assign them to variables inside a newly created anonymous namespace, which we assign to G.

2.11. The function GUI.CollectData

     ∇ CollectData←{
[1]        N←⍵
[2]        G←⎕NS''
[3]        _←G.⎕FX'r←∆List' 'r←{⍵,[1.5]⍎¨⍵}'' ''~¨⍨↓⎕NL 2'
[4]        G.APL←N.APL.State
[5]        G.Comments←N.Comments.State
[6]        G.DotAll←N.DotAll.State
[7]        G.FnsOprScripts←N.FnsOprScripts.State
[8]        G.Greedy←N.Greedy.State
[9]        G.IsRegEx←N.Greedy.State
[10]       G.MatchAPLname←N.MatchAPLname.State
[11]       G.MatchCase←N.MatchCase.State
[12]       G.NameList←N.NameList.State
[13]       G.SearchFor←N.SearchFor.Text
[14]       G.StartLookingHere←N.StartLookingHere.Text
[15]       G.Text←N.Text.State
[16]       G.Variables←N.Variables.State
[17]       G
[18]   }
     ∇

This namespace (G) is passed as the only argument to the Find function.

Suppose you replaces the Windows native GUI by an HTML5/JavaScript GUI. All you have to do here is ensure the G namespace is fed with all the data needed. ##.BusinessLogic will not be affected in any way.

2.12. The function ##.BusinessLogic.Find

Of course nothing is really happening in ##.BusinessLogic.Find, we just mock something up:

     ∇ (noOfHits noOfObjects cpuTime)←Find G;was
[1]   ⍝ Here's where all the searching takes place.
[2]   ⍝ `G` is a namespace that contains all the relevant GUI control settings (check boxes
[3]   ⍝ and text fields) as ordinary field. It's the interface between GUI and application.
[4]    ⎕DL 3
[5]    noOfHits←123
[6]    noOfObjects←645
[7]    cpuTime←2.3
     ∇

2.13. The function GUI.CalculateMinWidth

     ∇ CalculateMinWidth←{
[1]        N←⍵
[2]        ignore←'HitList' 'SearchFor' 'StartLookingHere' 'Statusbar' 'StatusField1' '∆Form'
[3]        list2←N.{⍎¨(' '~¨⍨↓⎕NL 9)~⍵}ignore
[4]        (2×N.∆V_Gap)+⌈/{0::0 0 ⋄ 2⊃+⌿↑⍵.(Posn Size)}¨list2
[5]    }
     ∇

The function calculates the minimum width needed by the form to be presentable. It takes all controls owned by the form into account except those listed on ignore.

2.14. The function GUI.CollectControls

     ∇ CollectControls←{
[1]        0∊⍴l←⎕WN ⍵:⍬
[2]        l,(⊃,/∇¨l)~⍬
[3]    }
     ∇

It collects all controls found in and then calls itself on them until nothing is found anymore.

2.15. The function GUI.DQ

     ∇ {r}←{focus}DQ ref
[1]    focus←{0<⎕NC ⍵:⍎⍵ ⋄ ref}'focus'
[2]    ⎕NQ focus'GotFocus' ⋄ r←⎕DQ ref
[3]   ⍝Done
     ∇

The function accepts an optional left argument which, when specified, must be a reference pointing to a control. The function then forces the focus onto that control before executing ⎕DQ on the right argument, usually the main form.

We put this into a separate function so that we can at any time interrupt the function, investigate variables or change functions and then carry on by executing →1.

2.16. The function GUI.Shutdown

     ∇ {r}←Shutdown N
[1]    r←⍬
[2]    :Trap 6 ⋄ 2 ⎕NQ N.∆Form'Close' ⋄ :EndTrap
     ∇

This function ensures the main form is closed in case it still exists.

Note that we need the trap. Even checking the main form with ⎕NS might fail if the user clicks the Close box right after the check has been performed but before the next line is executed.

This would produce one of these nasty crashes that occur only every odd year and are not reproducible. This is one of the rare cases where a trap is better than any check.

3. Changing the GUI

To demonstrate the power of the above approach we have to show it is easy to change.

Let's suppose the following:

  1. We have a user who is unhappy with the arrangement of the controls on the GUI. On her huge 5k monitor it looks a bit clumsy.
  2. She also wants the groups arranged in two rows, with Options and RegEx in the first row and Object types and Scan… in the second row.

How much work is required to make these changes?

First we double the vertical and horizontal distance between the controls on the main form:

     ∇ N←Init dummy
...
[4]    N.∆Labels←N.⎕NS''
[5]    N.(∆V_Gap ∆H_Gap)←10 20
[6]    N.∆Posn←80 30
...
     ∇

Then we change the sequence in which the groups are created:

leanpub-start-insert
     ∇ N←CreateGUI N;groups
[1]    N←CreateMainForm N
[2]    N←CreateSearch N
[3]    N←CreateStartLookingHere N
[4]    N.∆Groups←⎕NS''
[5]    N←CreateOptionsGroup N
[6]    N←CreateRegExGroup N
[7]    groups←'Group'⎕WN N.∆Form
[8]    {⍵.Size←(⌈/1⊃¨⍵.Size),¨1+2⊃¨⍵.Size}groups
[9]    N←CreateObjectTypesGroup N
[10]   N←CreateScanGroup N
[11]   {⍵.Size←(⌈/1⊃¨⍵.Size),¨1+2⊃¨⍵.Size}('Group'⎕WN N.∆Form)~groups
[12]   N←CreateList N
[13]   N←CreatePushButtons N
[14]   N←CreateHiddenButtons N
[15]   N.HitList.Size[1]-←(2×N.∆V_Gap)+N.∆Form.Size[1]-N.Find.Posn[1]
[16]   N.(⍎¨↓⎕NL 9).onKeyPress←⊂'OnKeyPress'
[17]   N.∆WriteToStatusbar←N∘{⍺.Statusbar.StatusField1.Text←⍵ ⋄ 1:r←⍬}
[18]   N.∆Form.onConfigure←'OnConfigure'(335,CalculateMinWidth N)
[19]  ⍝Done
     ∇

We need to calculate the height of the groups here twice: once in line [8], after having created the first two groups and then again in line [11] on all groups but the first two. For that we save the references of the first two on groups and exclude them in line 11.

We then tell the RegEx group where it should go, and we adjust the positions of the DotAll and the Greedy check boxes:

     ∇ N←CreateRegExGroup N;∆
[1]    ∆←⊂'Group'
[2]    ∆,←⊂'Caption' 'RegEx'
[3]   ⍝∆,←⊂'Posn'({⍵.Posn[1],(2×N.∆V_Gap)+2⊃U.AddPosnAndSize ⍵}N.∆Groups.ScanGroup)
[4]    ∆,←⊂'Posn'({⍵.Posn[1],(2×N.∆V_Gap)+2⊃U.AddPosnAndSize ⍵}N.∆Groups.OptionsGroup)
[5]    ∆,←⊂'Size'(300 400)
...
[16]   ∆←⊂'Button'
[17]   ∆,←⊂'Style' 'Check'
[18]  ⍝∆,←⊂'Posn'((⊃U.AddPosnAndSize N.IsRegEx),4×N.∆H_Gap)
[19]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.IsRegEx),2×N.∆H_Gap)
[20]   ∆,←⊂'Caption' 'Dot&All'
...
[24]   ∆,←⊂'Style' 'Check'
[25]  ⍝∆,←⊂'Posn'((⊃U.AddPosnAndSize N.DotAll),4×N.∆H_Gap)
[26]   ∆,←⊂'Posn'((⊃U.AddPosnAndSize N.DotAll),2×N.∆H_Gap)
[27]   ∆,←⊂'Caption' '&Greedy'
...
     ∇

We keep the old version to make a comparison easy.

Warning

Keeping old versions of lines

We do not generally advocate the technique used here. We do this only for demonstration.

In a real-world scenario, rather than cluttering the code, it should be left to a source-code management system and proper comparison tools to handle such issues.

For CreateObjectTypesGroup we just need to change the Posn property:

     ∇ N←CreateObjectTypesGroup N;∆
[1]    ∆←⊂'Group'
[2]    ∆,←⊂'Caption' 'Object &types'
[3]   ⍝∆,←⊂'Posn'({⍵.Posn[1],(2×N.∆V_Gap)+2⊃U.AddPosnAndSize ⍵}N.∆Groups.OptionsGroup)
[4]    ∆,←⊂'Posn'((N.∆V_Gap+1⊃U.AddPosnAndSize N.∆Groups.OptionsGroup),N.∆H_Gap)
[5]    ∆,←⊂'Size'(300 400)
...
     ∇

The last group-related function, CreateScanGroup, does not change at all because it still makes itself a neighbour of CreateObjectTypesGroup.

Since we now have significantly less space available for the HitList we need to change CreateList as well:

     ∇ N←CreateList N;∆;h
[1]    ∆←⊂'ListView'
[2]   ⍝h←⊃N.∆V_Gap+U.AddPosnAndSize N.MatchCase.##
[3]    h←⊃N.∆V_Gap+U.AddPosnAndSize N.FnsOprScripts.##
[4]    ∆,←⊂'Posn'(h,N.∆H_Gap)
...
     ∇

And that was it.

4. Testing

4.1. Prerequisites

For implementing tests execute the following steps:

  1. Create a namespace with
    'TestCases'#._Meddy.⎕ns''
  2. Load the script APLTreeUtils and the class Testers into # as discusses in the chapter regarding tests chapter.
  3. Execute these statements:
    )cs #._MyApp.TestCases
    #.Tester.EstablishHelpersIn ⍬

4.2. Testing “Find”

Edit #._MyApp.TestCases.Test_000 and make it look like this:

     ∇ R←Test_01(stopFlag batchFlag);⎕TRAP;N
[1]   ⍝ Test that the GUI comes up and the "Find" works well
[2]    ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
[3]    R←∆Failed
[4]
[5]    N←##.MyApp.GUI.Run 1
[6]    N.SearchFor.Text←'⎕IO'
[7]    N.StartLookingHere.Text←'#'
[8]    1 ⎕NQ N.Find'Select'
[9]    →GoToTidyUp 123 645 2.3≢N.∆Result
[10]   R←∆OK
[11]
[12]  ∆TidyUp:
[13]   1 ⎕NQ N.∆Form'Close'
     ∇

Of course this is not how a real test would look (line [9]) but it demonstrates the principles.

Note that N.∆Result was set in the GUI.Find function.

After having executed Init there is a global reference GUI.U around that points to the namespace GUI.GuiUtils; that's necessary because otherwise later calls might well fail with a VALUE ERROR when GUI.U is not available.

That's not a problem because we don't save the workspace, we recompile it from scratch whenever we start a development session.

4.3. Testing the business logic

Take a copy of the first test and make it look like this:

     ∇ R←Test_02(stopFlag batchFlag);⎕TRAP;N;G
[1]   ⍝ Test the business logic
[2]    ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
[3]    R←∆Failed
[4]
[5]    N←##.MyApp.GUI.Run 1
[6]    N.SearchFor.Text←'⎕IO'
[7]    N.StartLookingHere.Text←'#'
[8]    G←##.MyApp.GUI.CollectData N
[9]    →GoToTidyUp 123 645 2.3≢##.MyApp.BusinessLogic.Find G
[10]   R←∆OK
[11]
[12]  ∆TidyUp:
[13]   1 ⎕NQ N.∆Form'Close'
     ∇

4.4. Test missing “Search For”

Take a copy of the first test and make it look like this:

     ∇ R←Test_03(stopFlag batchFlag);⎕TRAP;N
[1]   ⍝ Test whether an empty "Search for" field leads to an error message.
[2]    ⎕TRAP←(999 'C' '. ⍝ Deliberate error')(0 'N')
[3]    R←∆Failed
[4]
[5]    N←##.MyApp.GUI.Run 1
[6]    N.StartLookingHere.Text←'#'
[7]    1 ⎕NQ N.Find'Select'
[8]    →GoToTidyUp 0=N.⎕NC'∆ErrorMsg'
[9]    →GoToTidyUp'"Search for" is empty - nothing to look for...'≢N.∆ErrorMsg
[10]   R←∆OK
[11]
[12]  ∆TidyUp:
[13]   1 ⎕NQ N.∆Form'Close'
     ∇

This works because when testFlag – the right argument of GUI.Run — is 1 then the function Dialog.ShowMsg does nothing but set a global variable N.∆ErrorMsg which we use in the test case in order to check whether our actions triggered the correct behaviour.

We have eariler stated that we won't discuss Dialogs.ShowMsg in details but we list it here because the function illustrates an important concept of how GUIs can be tested at all:

     ∇ {r}←{caption}ShowMsg(N msg);n2;U
[1]   ⍝ Takes `N` and a message to be displayed as mandatory right arguments.
[2]   ⍝ Takes an optional `caption` as left argument.
[3]   ⍝ Returns 1 when running under test conditions and 0 otherwise.
[4]   ⍝ In test mode no GUI is created at all but a global `N.∆ErrorMsg` is set to `msg`.
[5]    r←0
[6]    caption←{0<⎕NC ⍵:⍎⍵ ⋄ 'Attention!'}'caption'
[7]    :If N.∆TestFlag
[8]        N.∆ErrorMsg←msg
[9]        r←1
[10]   :Else
[11]       U←##.GuiUtils
[12]       n2←U.CreateNamespace
[13]       n2.(∆V_Gap ∆H_Gap)←N.(∆V_Gap ∆H_Gap)
[14]       n2←n2 CreateGUI N msg caption
[15]       {}n2.OK U.DQ n2.∆Form
[16]       Close n2.∆Form
[17]   :EndIf
     ∇

Now it becomes apparent why we assigned the right argument of GUI.RuntestFlag — to N.∆TestFlag: we need this potentially in order to avoid handing over control to the user; in test cases we don't want to do this.

4.5. Final steps

  1. Delete #._MyApp.TestCases.Test_0001
  2. Run #._MyApp.TestCases.RunDebug 0.

4.6. Pros and cons of GUI testing

GUIs tend to change quite a lot over time, and rewriting — or adding — plenty of test cases can take a significant amount of time, sometimes far more than needed to implement the GUI in the first place.

In such cases the time invested in exhaustive test cases might be too high.

On the other hand if the application's main purposes is to make complex tasks manageable, there will be a lot of dependencies and checking going on in the background. In that case exhaustive test cases might well be a good investment if only because you will find bugs for sure. Your call.

5. Conclusion

We believe we have demonstrated the combination of techniques outlined in this chapter leads not only to better code but makes it also far easier actually to write GUI applications in the first place.

Having said this, the approach outlined here is insufficient if the GUI of an application changes heavily depending on the user's actions.

However, the principal ideas can still be used but the list of controls were better compiled from scratch at the start of each callback function rather than having them as a static list in a namespace.

It was Paul Mansour came up with many of the ideas outlined here. We stole plenty from him and owe him a big thank you.


Footnotes

  1. https://aws.amazon.com/appstream2/

Chapter 20:

Git for APLers

1. Overview

Chapter 21:

Appendix 1 — Windows environment variables

1. Overview

Windows comes with quite a number of environment variables. Those variables are helpful in addressing, say, a particular path without actually using a physical path.

For example, on most PCs, Windows is installed in C:\Windows, but this is by no means guaranteed. It is therefore much better to address this particular folder as ⊣2 ⎕NQ # 'GetEnvironment' 'WINDIR'.

Below some of the environment variables found on a Windows 10 system are listed and explained.

Information

The Dyalog Cookbook is usually referring to Windows 10 which, in most cases, is identical with Windows 8 and 7. The Cookbook is not applicable to unsupported versions of Windows like Vista and earlier.

Notes:

2. Outdated?!

Some consider environment variables an outdated technology. We don't want to get involved in this argument here but enviroment variables will be round for a very long time, and Windows relies on them. (They are also standard under UNIX, including Linux and MacOS.)

3. The variables

AllUserProfile
Defaults to C:\ProgramData: see ProgramData
AppData

Defaults to C:\Users\{yourName}\AppData\Roaming

Use this to store data that is both application and user specific that is supposed to roam [1] with the user. An INI might be an example.

See also LocalAppData

CommonProgramFiles
Defaults to C:\Program Files\Common Files
CommonProgramFiles(x86)
Defaults to C:\Program Files (x86)\Common Files
CommonProgramW6432
Defaults to C:\Program Files\Common Files
ComputerName
The name of the computer
ComSpec
Defaults to C:\WINDOWS\system32\cmd.exe
ErrorLevel
This variable does not necessarily exist. If you execute ⎕OFF 123 in an APL application it will set ErrorLevel to 123.
HomePath
Defaults to \Users\{yourName}
LocalAppData

Defaults to C:\Users\{yourName}\AppData\Local

Use this to store data that is both application and user specific that is not supposed to roam [1] with the user.

A log file might be an example. The reason is that when a user logs in all the data stored in %APPDATA% is copied over. A large log file might take significant time to be copied with very little (or no) benefit.

See also AppData.

LogonServer
Defaults to the name of the computer your are logged on to. In case of your own desktop PC the values of LogonServer and ComputerName will be the same. In a Windows Server Domain however they will differ.
OS
Specifies the Operating System; under Windows 10, Windows_NT
Path
All the folders (separated by semicola) that the operating system should check if the user enters something like my.exe into a console window and my.exe is not found in the current directory.
PathExt
A list of the file extensions the operating system considers executable, for example: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC.
ProgramData
Defaults to C:\ProgramData. Use this for information that is application-specific and needs write access after installation. For Dyalog, this would be the right place to store the session file, workspaces and user commands.
ProgramFiles
Defaults to C:\Program Files. On a 64-bit version of Windows this is where 64-bit programs are installed. Note however that on a 32-bit version of Windows this points to ProgramFiles(x86).
ProgramFiles(x86)
Defaults to C:\Program Files (x86). This is where 32-bit programs are installed.
ProgramW6432
Defaults to C:\Program Files. On a 64-bit version of Windows this path points to ProgramFiles. On a 32-bit version of Windows it also points to ProgramFiles which in turn points to ProgramFiles(x86).

For details see WOW64 Implementation Detail [3].

PSModulePath
Defaults to C:\WINDOWS\system32\WindowsPowerShell\v1.0\Modules\. This path is used by Windows Power Shell [2] to locate modules when the user does not specify the full path to a module.
Public
Defaults to C:\Users\Public. It contains folders like Public Documents, Public Music, Public Pictures, Public Videos, … well, you get the picture.
SystemRoot
Specifies the folder in which Windows is installed. Defaults to C:\WINDOWS.
Temp
Points to the folder that holds temporary files and folders. Defaults to C:\Users\{username}\AppData\Local\Temp. See also TMP.
TMP
Points to the folder that holds temporary files and folders. Defaults to C:\Users\{username}\AppData\Local\Temp. Note that the GetTempFileName API function (which is available as FilesAndDirs.GetTempFilename) will first look for the TMP environment variable and only if that does not exist for the TEMP variable.
Username
The username of the user currently logged on. Same as ⎕AN in APL.
UserProfile
Defaults to C:\Users\{username}. That's where everything is saved that belongs to the user currently logged on. Note that this is kept apart from other user's eyes by the operating system.
WinDir
Defaults to the value of %SystemRoot%. Deprecated.

Footnotes

  1. https://en.wikipedia.org/wiki/Roaming_user_profile

  2. https://en.wikipedia.org/wiki/PowerShell

  3. https://msdn.microsoft.com/en-us/library/windows/desktop/aa384274(v=vs.85).aspx

Chapter 22:

Appendix 2 — User commands

1. Overview

User commands are a great way to make utilities available to the developer without cluttering the workspace. They also allow you to have one code base for several Dyalog installations. Since its introduction they have proven to be be indispensable.

Whether you want to write your own user commands or to make use of any third-party user commands like those available from the APL wiki for download [1], you need to consider your options for how to integrate non-Dyalog user commands into your development environment.

The default folder depends on your version of Dyalog of course, but you can always find out from a running instance of Dyalog APL:

      '"',(2⎕NQ # 'GetEnvironment' 'Dyalog'),'\SALT\spice\"'
"C:\Program Files\Dyalog\Dyalog APL-64 16.0 Unicode\SALT\spice\"

The above is the default folder for the 64-bit Unicode version of Dyalog 16.0 for all Dyalog user commands available within that version.

2. Using the default folder

If you want to keep life simple the obvious choice seems to be this folder: you just copy your user command into this folder and it becomes available straight away.

Simple may it be, but this is not recommended:

For these reasons you are advised to use a different folder.

3. Use your own dedicated folder

Let's assume that you have a folder C:\MyUserCommands that's supposed to hold all non-Dyalog user commands.

Via the Options > Configure command you can select the User Commands dialog and add that folder to the search path; remember to press the Add button once you have browsed to the right directory.

If you use several versions of Dyalog in parallel then you are advised not to add that folder via the configuration dialog box in each of those versions.

Instead we recommend writing an APL function that adds the folder to all versions of Dyalog currently installed. See the chapter The Windows Registry where this scenario is used as an example.

4. Name clashes

Where two user commands share the same name, the last definition wins. You can achieve this just by having a user command Foo in two different scripts in different folders with the same group name, or no group name at all!

In other words, the full name of a user command is compiled by the group name (say foo) and the user-command name (say goo): ]foo.goo. However, as long as there is only one user command goo this will do nicely:

      ]goo

5. Group name

Group names are not only useful to avoid name clashes, they also allow user commands to be, well, grouped in a sensible way. Whether you should add your own user commands to any of the groups Dyalog comes with is a diffult question to answer. There are pros and cons:

5.1. Pros

5.2. Cons

6. Updates

Note that once you've copied a new user command into that folder it is available straight away, even in instances of Dyalog that are already running.

However, auto-complete does not know about the new user command until it was called for the first time in an already running instance. You can at any time execute ]ureset to make sure that even auto-complete knows about it.

In case you change an existing user command, for example by modifying the parsing rules, you must execute ]ureset in order to get access to the changes from any instance of Dyalog that has already been running by then.

7. Writing your own user commands

It's not difficult to write your own user commands, and there is an example script available that makes that easy and straightforward. However, if your user command is not too simple consider developing it as an independent application, living in a particular namespace (let's assume Foo) in a particular workspace (let's assume Goo).

Then write a user command that creates a namespace local to the function in the user command script, copy the namespace Foo from the workspace Goo into that local namespace and finally run the required function. Make sure the workspace is a sibling of the user command script.

This approach has the advantage that you can develop and test your user commands independently from the user command framework.

This is particularly important because changing a user command script from the Tracer is a bit dangerous; you will see more aplcores than under normal circumstances. On the other hand it is difficult to execute the user command without the user command framework calling it: you need those arguments and sometimes even variables that live in the parent (##).

We therefore recommend you ensure no function in Foo relies on anything provided by the user-command framework. Instead, the calling function (Run in your user command) must pass such values as arguments to any functions in Foo called by Run.

That makes it easy to test all the public functions in Foo. Of course you should have proper test cases for them.

The following code is a simple example that assumes the following conditions:

:Namespace  Foo
      ⎕IO←1 ⋄ ⎕ML←1

    ∇ r←List
      r←⎕NS''
      r.Name←'Foo'
      r.Desc←'Does this and that'
      r.Group←'Cookbook'
      r.Parse←'1 -verbose'
    ∇

    ∇ r←Run(Cmd Args);verbose;ref;path;arg
      ref←⎕NS''
      verbose←Args.Switch'verbose'
      path←⊃1 ⎕NPARTS ##.SourceFile
      arg←⊃Args.Arguments
      :Trap 11
          'Foo'ref.⎕CY path,'\Goo'
      :Else
          'Copy operation for "Foo" in "Goo" failed' ⎕Signal 11
      :EndTrap
      r←ref.Foo.Run arg verbose
    ∇

    ∇ r←Help Cmd
      r←⊂'Help for "Foo".'
      r,←⊂'This user command ...'
      r←,[0.5]r
    ∇

:EndNamespace

Notes:

The workspace Goo can be tested independently from the user command framework, and the workspace Goo might well hold test cases for the functions in Foo.


Footnotes

  1. Dyalog user commands from the APL wiki http://aplwiki.com//CategoryDyalogUserCommands

Chapter 23:

Appendix 3 — aplcores and WS integrity

The workspace (WS) is where the APL interpreter manages all code and all data in memory.

The Dyalog tracer / debugger has extensive edit-and-continue capabilities; the downside is that these have been known occasionally to corrupt the workspace. However, there are many other ways the workspace may get corrupted:

The interpreter checks WS integrity every now and then; how often can be influenced by setting certain debug flags; see The APL Command Line in the documentation for details. Be warned that…

When the interpreter finds that the WS is damaged it will create a dump file aplcore and exit to prevent your application from producing (or storing) incorrect results.

Regularly rebuilding the workspace from source files removes the risk of accumulating damage to the binary workspace.

An aplcore is useful in two ways:

You can create an aplcore deliberately by executing:

      2 ⎕NQ '.' 'dumpws' 'C:\MyAplcore'

This might be a useful thing to do just before executing a line you already know will cause havoc in one way or another.

In order to create a real aplcore, in the sense of corrupting the workspace, this will do:

 ∇Crash;MEMCPY
 :Trap 102
     ⎕NA'dyalog32|MEMCPY u4 u4 u4'
 :Else
     ⎕NA'dyalog64|MEMCPY u4 u4 u4'
 :EndTrap
 MEMCPY 0 0 4
 ∇

By default, an aplcore is saved with the name aplcore in what is at that moment the current directory. This is not nice because it means that any aplcore might overwrite an earlier one. That can become particularly annoying when you try to copy from an aplcore with :

     )copy C:\MyAplcore.

but this might actually create another aplcore, overwriting the first one. Now it might well be too late to restrict the attempt to copy to what is most important to you: the object or objects you have worked on most recently.

If the aplcore is saved at all that is, because if the current directory is something like C:\Program files\ then you won't have the right to save into this directory anyway.

When a program asks Windows to save a file in a location where that program has no write permission (e.g. C:\Program Files, C:\Program Files (x86), C:\Windows) then Windows will tell the application that it has fulfilled the request, but the file will actually be saved in something like C:\Users\{username}\AppData\Local\VirtualStore\Program Files\Dyalog\Dyalog APL-64 16.0 Unicode\

For that reason it is highly recommended to set the value aplcorename in the Windows Registry:

Defining home and names of aplcores

This means that aplcores…

The same can be achieved by specifying APLCORENAME=... on the command line. That's particularly important for Windows Services.

Chapter 24:

Appendix 4 — The development environment

1. Configure your session

Most developers adapt the development environment in one way or another:

There are several ways to achieve this:

  1. Save a copy of the default session file somewhere outside the installation directory.

    By default this is def_{countryCode}.dse in the installation directory, for example def_uk.dse for the UK.

    Edit the configuration so that this copy is loaded.

  1. Save a copy of the build workspace (that is typically something like C:\Program Files\Dyalog\...\ws\buildse.dws) outside the installation directory. Then use it to create your own tailored version of a DSE.

    Then follow the advice in 1)

Warning

Leave the Dyalog installation directory alone

Do not alter any files in the Dyalog installation directory, and don't add your own files there either: Dyalog may update files in there in any patch.

Although this is not strictly true in 16.0 and earlier, Dyalog may start to issue .msp files which might change any file in the installation directory without warning!

Both approaches have their own problems, the most obvious being that with a new version of Dyalog you start from scratch. However, there is a better way: save a function Setup in either C:\Users\{UserName}\Documents\MyUCMDs\setup.dyalog or one of the SALT work directories and it will be executed when…

The function may be saved in that file either on its own or as part of a namespace.

Information

You might expect that saving a class script Setup.dyalog with a public shared function Setup would work as well but that's not the case.

SALT work directories

You can check which folders are currently considered SALT work directories by issuing ]settings workdir.

You can add a folder C:\Foo with ]settings workdir ,C:\Foo.

When called as part of the SALT boot process a right argument 'init' will be passed. When called via ]usetup then whatever is specified as argument to the user command will become the right argument of the Setup function.

The Dyalog manuals mention this feature only when discussing the user command ]usetup but not anywhere near how you can configure your environment; that's why we mention it here.

If you want to debug any Setup function then the best way to do this is to make ⎕TRAP a local variable of Setup and then add these lines at the top of the function:

[1] ⎕TRAP←0 'S'
[2] .

This will cause an error that stops execution because error trapping is switched off. This way you get around the trap that the SALT boot process uses to avoid Setup causing a hiccup. However, if you change the function from the Tracer don't expect those changes to be saved automatically: you have to take care of that yourself.

The following code is an example for how you can put this mechanism to good use:

:Namespace Setup
⍝ Up to - and including - version 15.0 this script needs to go into:
⍝ "C:\Users\[username]\Documents\MyUCMDs"
⍝ Under 16.0 that still works but the SALT workdir folders are scanned as well.
  ⎕IO←1 ⋄ ⎕ML←1

∇ {r}←Setup arg;myStuff
  r←⍬
  'MyStuff'⎕SE.⎕CY 'C:\MyStuff'
  ⎕SE.MyStuff.DefineMyFunctionKeys ⍬
  EstablishOnDropHandler ⍬
∇

∇ {r}←EstablishOnDropHandler dummy;events
  r←⍬
  events←''
  events,←⊂'Event' 'DropObjects' '⎕se.MyStuff.OnDrop'
  events,←⊂'Event' 'DropFiles' '⎕se.MyStuff.OnDrop'
  events,←⊂'AcceptFiles' 1
  events∘{⍵ ⎕WS ¨⊂⍺}'⎕se.cbbot.bandsb2.sb' '⎕se.cbbot.bandsb1.sb'
∇

:EndNamespace

Suppose in the workspace MyStuff there is a namespace MyStuff that contains at least two functions:

  1. DefineMyFunctionKeys; this defines the function keys.
  2. OnDrop; a handler that handles “DropObject” and “DropFiles” events on the session's status bar.

This is how the OnDrop function might look:

OnDrop msg;⎕IO;⎕ML;files;file;extension;i;target
⍝ Handles files dropped onto the status bar.
 ⎕IO←1 ⋄ ⎕ML←1
 files←3⊃msg
 :For file :In files
     extension←1(819⌶)3⊃1 ⎕NPARTS file
     :Select extension
     :Case '.DWS'
         ⎕←'     )XLOAD ',{b←' '∊⍵ ⋄ (b/'"'),⍵,(b/'"')}file
     :Case '.DYALOG'
         :If 9=⎕NC'⎕SE.SALT'
             target←((,'#')≢,1⊃⎕NSI)/' -Target=',(1⊃⎕NSI),''''
             ⎕←'      ⎕SE.SALT.Load ''',file,'',target
         :EndIf
     :Else
         :If 'APLCORE'{⍺≡1(819⌶)(⍴⍺)↑⍵}2⊃⎕NPARTS file
             ⎕←'      )COPY ',{b←' '∊⍵ ⋄ (b/'"'),⍵,(b/'"')}file,'.'
         :Else
             :If ⎕NEXISTS file
                 ⎕←{b←' '∊⍵ ⋄ (b/'"'),⍵,(b/'"')}file
             :Else
                 ⎕←file
             :EndIf
         :EndIf
     :EndSelect
 :EndFor

What this handler does depends on what extension the file has:

Information

When you start Dyalog with admin rights then it's not possible to drop files onto the status bar. That's because Microsoft considers drag'n drop too dangerous for admins. (One might think it better strategy to leave the dangerous stuff to the admins.)

How you configure your development environment is of course very much a matter of personal preferences.

However, you might consider loading a couple of scripts into ⎕SE from within Setup.dyalog; the obvious candidates for this are APLTreeUtils, FilesAndDirs, OS, WinSys, WinRegSimple and Events. That would allow you to write user commands that can reference them with, say, ⎕SE.APLTreeUtils.Split.

2. Define your function keys

Defining function keys is of course not exactly a challenge. Implementing it in a way that is actually easy to read and maintain is a challenge.

:Namespace FunctionKeyDefinition

    ∇ {r}←DefineFunctionKeys dummy;⎕IO;⎕ML
      ⎕IO←1 ⋄ ⎕ML←3
      r←⍬
      ⎕SHADOW⊃list←'LL' 'DB' 'DI' 'ER' 'LC' 'DC' 'UC' 'RD' 'RL' 'RC' 'Rl' 'Ll' 'CP' 'PT' 'BH'
      ⍎¨{⍵,'←⊂''',⍵,''''}¨list
      r⍪←'F01'('')('(Reserved for help)')
      r⍪←'F02'(')WSID',ER)(')wsid')
      r⍪←'F03'('')('Show next hit')                  ⍝ Reserved for NX
      r⍪←'F04'('⎕SE.Display ')('Call "Display"')
      r⍪←'F05'(LL,'→⎕LC+1 ⍝ ',ER)('→⎕LC+1')
      r⍪←'F06'(LL,'→⎕LC ⍝',ER)'→⎕LC'
      ...
:EndNamespace

This approach first defines all special shortcuts – like ER for Enter – as local variables; using ⎕SHADOW avoids the need for maintaining a long list of local variables. The statement ⍎¨{⍵,'←⊂''',⍵,''''}¨list assigns every name as an enclosed text string to itself like ER←⊂'ER'. Now we can use ER rather than (⊂'ER) which improves readability.

A definition like LL,'→⎕LC ⍝',ER reads as follows:

Information

If you don't know what LL and ER actually are read the page “Keyboard shortcuts” in the UI Guide.

3. Windows captions

If you always run just one instance of the interpreter you can safely ignore this.

If on the other hand you run occasionally (let alone often) more than one instance of Dyalog in parallel then you are familiar with how it feels when all of a sudden an unexpected dialog box pops up, be it an aplcore or a message box asking “Are you sure?” when you have no idea what you are expected to be sure about, or which instance has just crashed.

There is a way to get around this. With version 14.0 Windows captions became configurable. This is a screenshot from the online help:

Dyalog's help on Windows captions

Help — online versus offline

There are pros and cons:

We suggest you configure Windows captions in a particular way in order to overcome this problem. The following screen shot shows the definitions for all Windows captions in the Windows Registry for version 16 in case you follow our suggestions:

Windows Registry entries for 'Window captions'

Notes:

The other pieces of information are less important. For details refer to the page “Window captions” in the Installation and Configuration Guide. These definitions ensure most dialog boxes (there are a few exceptions) can easily be associated with a particular Dyalog session. This is just an example:

A typical dialog box

You can ask for the current settings with the user command ]caption:

      ]caption

You can also change the settings with this user command. For details enter:

      ]??Caption

Chapter 25:

Appendix 5 — Special characters

The following assumes ⎕IO←1.

1. Carriage return, new line and form feed

The following contains everything you need to know about those characters.

Character ⎕ML<3 ⎕ML≥3 Escaping ⎕UCS Abbr. Hex
Carriage return ⎕TC[3] ⎕TC[2] \r 13 CR 0x0D
New line [1] ⎕TC[2] ⎕TC[3] \n 10 LF,NL 0x0A
Form feed n/a n/a \f 12 FF 0x0D

2. Line-ending characters

The following table contains everything you need to know about how different operating systems make use of line-ending characters.

OS Lines end
Windows CR,LF
Linux & Unix NL
Old Macs[2] CR

Strictly speaking, a file should always end with such characters. However, for example under Windows even different software packages from Microsoft handle this differently.


Footnotes

  1. Was “linefeed” in the old days.

  2. Before OS X