In this second of two installments, we continue with the notification of events & errors and the Handle Event Script used in the previous post. We also propose techniques for (counter) threshold checks and restart-ability practices. These practices will ease maintenance tasks.
It is a best practice to differentiate generic event notification from those specific to the business process. The (earlier) Handle Event Script provides parameters for these details. Below is a particular business process data validation script for a field name “rating” where only certain values are allowed. Its purpose is to log the invalid value and the fact that the default (“Warm”) was assigned.

_fieldValue=_1; _fieldType=_2; _fieldFormat=_3; // _3 if available in field’s File Format (more common for Date types)
fieldName="rating"; fieldDefault="Warm";
If(!Validate("One Of",'Cold,Warm,Hot'),
  RunScript("../Handlers/Event",5,"Notice Type","Default: "+Quote(fieldDefault) +" used for "+Quote(_fieldValue),
    false,"NOTICE");
  false //return false to use File Format's Default value
,true //else return true; all is well
);

As more events are logged within data validations, one might be interested in counting these events. We provide a counter handler below when we discuss threshold checks. First, we consider the best practice for Operations, in general, using Setup and Wrap-up scripts. Soon after, the On Failure Operation for handling more critical events.
As you will learn in the Restart-ability topic below, initializing variables is a vital scripting task. For example, an Operation’s $LOG.level value used by the Handle Event Script may be re-initialized in each Operation’s setup Script. It’s good practice to begin each Operation with a setup script to initialize (or re-initialize) variables.
It’s also a good practice to summarize an Operation’s outcome in a final, wrap up Script in each Operation or a summary email at the end of an Operation chain. This summary will include statistics of good/processed versus bad/unprocessed records. (Including any data validation metrics.) This summary is useful for variance measurements over many executions. In an Email Target, one can attach the data validation’s failed records file as well.

On Failure Operations for Errors

Chained On Failure Operations get executed when a Jitterbit Operation Error or a Raise Error event occurs. The Raise Error event gets triggered within a Script with the RaiseError() function or with a field or row’s data validation.
Like On Success Operations, they are added using the Operation’s context menu. On Failure Operations are unique in that they can be executed via the Raise Error event or scripted after a failed RunOperation(). Within, they can do anything any other operation can do. Most are Scripts that call a Notification Method shown in our previous post. They can also have other Operations chained to them via the Operation On Failure/Success context menu items. Finally, when invoked via the RaiseError() function, take care whether the On Failure Operation aborts the Operation chain or, if desired, continues.

Counters and Threshold Checks

Along with potential log entries and termination with an On Failure Operation, other checks are possible. One might want similar logging during data validation if a threshold amount of errors is reached. The key here is using counter variables.
A dedicated counter handler script –that might call Handle Event– could be used. The Script might notify about the threshold once-only, rather than repeating with future checks. Below is an example of this dedicated script.

// Counter is a utility script to create and increment a thread-safe counter and log an event if any counter threshold is reached.
//   Important, this Script uses the Event Script to log threshold events.
//   Typically, it's used to increment a counter but, more importantly, log an event if a threshold value is provided and reached.
//   The first time the script is called (best in an Operations' Setup Script), the counter value is set to 0.
//   After initializing the counter, one can increment the counter within the Operation as needed by calling again.
//   The script always returns the new/current counter value.
// The only required parameter is the name of the global variable (a counter name) used to keep many counters unique.
//   This global variable holds the counter's "value".  
//   Future executions of Counter will increment and return the new (thread-safe) counter value.
//   Use the 3rd argument to return embedded commas in a String value (easier on the eyes -- this will not increment the counter!)
// Some other parameters are optional:
//   Argument 2 (_counter_threshold): an integer value (the threshold) used to provoke the log event indicating the threshold has been reached.
//     See 4th argument to continue any logging
//   Argument 3 (_thousand_commas): boolean (default: false) to have "commas" embedded in the returned counter value.  Note, this WILL NOT increment the counter.
//   Argument 4 (_always_notify): boolean to continue logging the "threshold has been reached/exceeded" event upon each future increment.
//     A global boolean variable (see optional argument 7) is used to show whether the threshold has been reached
//   Argument 5 (_increment): integer value providing the increment value (defaults to 1)
//   Argument 6 (_threshold_msg): string value providing the unique message to log. 
//     Rarely used; defaults to "$<_counter_name> reached threshold of <_counter_threshold>" Example/default: "$mycounter reached threshold of 500"
//   Argument 7 (_counter_warn_var): string value providing the variable name used to show whether the threshold has been reached.
//     Rarely used; defaults to "<_counter_name>__WARNED@ge<_counter_threshold" Example: "mycounter__WARNED@ge500". This will "hold" a boolean value.
//   Note arguments cannot be skipped; use "" for unused arguments
//Examples:
// RunScript("../Handlers/Counter","mycounter"); // initialize $mycounter to 0
// ...
// new_counter_value = RunScript("../Handlers/Counter","mycounter"); // increment counter by 1
// or
// RunScript("../Handlers/Counter","mycounter", 200); // increment counter by 1 and check/log if threshold of 200 has been reached
// or later
// counter_value_with_commas = RunScript("../Handlers/Counter","mycounter", "", true); //returns counter value as string with commas
    _counter_name = _1;
    _counter_threshold = IfEmpty(_2, "n/a"); // optional
    _thousand_commas = IfEmpty(_3,false); // optional, will not increment if true
    _always_notify = Bool(IfEmpty(_4, true)); // optional; log all increments beyond threshold
    _increment = Int(IfEmpty(_5, 1)); // optional
    _threshold_msg = IfEmpty(_6, "$" + _counter_name + " reached threshold of " + _counter_threshold); // optional
    _counter_warn_var = IfEmpty(_7, '?'+_counter_name + "__WARNED@ge" + _counter_threshold); // optional
    If(IsNull(Get(_counter_name)), // initialize the counter
      InitCounter(_counter_name,0);
      Set(_counter_warn_var,Bool(IfEmpty(Get(_counter_warn_var),false))); //set boolean to "false" as no threshold
    ,
      currently = Get(_counter_name);
      If(_thousand_commas, // return current counter value with embedded commas but don't increment the counter
        currently = RegExReplace(currently,"(\d)(?=(\d{3})+$)","\\1,")
      , // increment counter and check if any threshold value has been reached
        Set(_counter_name, Get(_counter_name) +_increment); // increment the counter
        If(IsInteger(_counter_threshold) && Get(_counter_name) >= Int(_counter_threshold) // if threshold reached and
          && (_always_notify || !Get(_counter_warn_var)), // (always log it or not reached earlier)
          result=RunScript("<TAG>Scripts/Handlers/Event</TAG>", 4, "Threshold Check", _threshold_msg, false, "WARNING"); // see Event script
          Set(_counter_warn_var,true); //threshold reached, set boolean variable to true
        )
      );
    );
    currently; // return counter value (with commas if requested) before any increment

The above script takes advantage of the Set() and Get() functions to define and reference customized global variables by string. (Rather than using the $-sign notation.) In particular, the counter name and the threshold value are used to define a global variable name showing whether a warning was issued.
For performance, Jitterbit will thread Operations depending upon available server CPU’s. (Threads are a way for an Operation to divide itself into two or more concurrently running tasks.) One possible side effect is miscounting counter values. To correct for threading, use the InitCounter() function in an Operation’s setup script. (The above script does this.)

Each call to the Script either initializes (the first call) the counter or increments the counter. In all cases, it returns the counter value prior to the call. The last call to the Counter Script might be used to log the counter value in a wrap-up script.

As an example of using the Handle Counter Script, you might run into an ill-mannered endpoint that may need many connection attempts. Use the Handle Count script within an On Failure Operation to repeat an Operation until some threshold is reached. For example, an On Failure Script Operation might use the following to attempt an Operation three times.

count=RunScript("../Handlers/Counter","myCounter",3,false); 
If($?myCounter__WARNED@ge3, // take care with this dynamic variable name 
  RaiseError("Operation Attempted "+count+" times, each resulting in failure") 
, 
  RunOperation(..) 
);

Restart-ability

Now that you’ve implemented the event and error capturing, you want your design restartable. That is, restartable from any point-of-failure (POF) during execution. After correcting the design or data error, a failed Operation chain can be restarted — if you’ve designed your Operations so they can be restarted.

You’ll notice the use of IfEmpty() in the above Scripts and, more importantly, the mentioning of Operation setup scripts. Setup scripts are the best method to enable restarting your operation chain(s) at various points in the data pipeline chain. Source parameters, counters, and other variables can be (re)initialized with desired changes. Using the IfEmpty() function, these variables defined and assigned values upstream in the Operation chain can be reset. That is, changes to the IfEmtpy() constructs won’t impact the end-to-end, complete Operation chain!
If Operation results are stored in initial, clean, and load-ready staging directories, one can restart from the POF. You can start from the next Operation’s setup script rather than starting the execution from its’ beginning. Directories in Jitterbit’s Temporary Storage make for good locations as these are purged (by default, every 24 hours).

Summary

The above and previous practices with Jitterbit empowers an integration design that is flexible and robust. Best practices include:

  • Differentiate generic from specific event notifications.
  • Use setup (initialize) and wrap up (summarize) Scripts.
  • Use InitCounter() for counters within threaded operations
  • Design for restart-ability

Try the above Handle Count Script with invocations like the following. After execution, view the Operation Log.
Increment a counter called “myCounter” by 1 (default) with a threshold check at a value of 2.

RunScript("../Handlers/Counter","myCounter",2,false); 
RunScript("../Handlers/Counter","myCounter",2,false); //warn once 
RunScript("../Handlers/Counter","myCounter",2,false);