When chaining Operations to execute, you might use Targets to stage data for the Source of the next Operation. At times for reading or appending many times during various chained Operations. Often, Operation executions only temporarily need these staged files. Also, temporary files for endpoint responses are the best practice for troubleshooting. This need is the purpose of the Jitterbit Source or Target Temporary Storage component. In this post, I share some best practices for and nuances of the Temporary Storage component.

Temporary Storage Sources and Targets

By default, “temp Storage” files (or temp files) exist for 24 hours on a Jitterbit Agent in a formal jitterbit/TemporaryStorage folder. Typically, they are within a Server’s temporary storage location (/tmp or C:\Windows\temp). (Each Agent configuration determines the lifetime and Server’s location of these files.) When using Agent Groups (or the Cloud Agent), these files will only exist on the Agent that executes the Operation chain.
You have a choice of allowing Jitterbit to declare an operation’s source as the previous operation’s target. See the chained Operation’s Source context menu item “Use Source from Previous Operation.” (This menu item is best read as “Use Previous Operation Target as Source”). Although this might simplify a final design, I like to use temp files and persist data so I can list, log and view content.
See the Source or Target context menu items “Copy to New Source” and “Copy to New Target” in the Project tree. You can create mirrored Source and Target components using these menu items. You will want to be sure to change “both” components when you change one component’s definition. (One exception might be a utility Source temp file — see below)
With the understanding of where and how long temp files exist, you can use them as you would computer Random Access Memory (RAM). That is after an Operation completes and you’ve given each a unique name; they are now referenceable. (If you configure their lifetime, this can be useful when troubleshooting production issues.)
I use a Script to generate a unique name for my temp files.

// NameGen: generates a unique name containing timestamp
purpose = IfEmpty(_1, "UNKNOWN_PURPOSE"); env=GetEnvironmentName();
// expect env+purpose+yyyy-mm-ddTHHMMSS_zzz or
// env+UKNOWN_PURPOSE+yyyy-mm-ddTHHMMSS_zzz if purpose is undefined
// where timestamp is the Jitterbit Agent's Time Zone
isoTime = RegExReplace(Now_(), //similar to ISO time: yyyy-mm-ddTHHMMSS_zzz (zzz is microseconds)
  "(\\d{4}-\\d{2}-\\d{2}) (\\d{2}):(\\d{2}):(\\d{2})\.(\\d{3})","\\1T\\2\\3\\4_\\5");
name = env+purpose+isoTime; // decorate and return name

Making a unique temp file name per Operation execution is a best practice. I frown on the use of the GUID() function for this. It defeats the name’s real value – quick recall & access to the content as needed. The use of a timestamp that includes microseconds can guarantee uniqueness but also makes the files sort-able. The 3-digit microsecond also makes the file name identifiable in a long file list. Note, if this file gets moved its file-system timestamp is not likely reliable.
I use something like the following in an Operation’s Setup Script (a leading Script of an Operation – a best-practice), so each temp file name is logged.

$%fileBasename = RunScript("Scripts/Utilities/NameGen","QueryPassThru");
WriteToOperationLog("Target filename: "+$%fileBasename+".xml ("+GetServerName()+")");

Note the use of GetServerName() above so we know which Agent is hosting the temp file. Another example is found in this post.
Within the Target definition, I’ll define the Target’s Filename form field value as [%fileBasename].xml. I might use the Path form field to park temp files in different directories if helpful.
Within Jitterbit Studio, you can display file lists and their content in a few ways. One way –and advantage of mirroring a Target– is to define the Source’s default filename using {}. (For example, [%fileBasename{*}].xml) Now I can use the Test Connection button on the Source component and see all the temp file names sorted by time. There are times I want a Source for utility purposes. For example, if I’m using this Source only for listing or displaying temp file content (and not within an Operation). I’ll rename the Source to QueryPassThru (utility) and the corresponding Target QueryPassThru. I’ll also put this utility Source in its own Utilities Project Tree Source folder.
I also like to use Jitterbit’s Script Pad (keyboard shortcut: F4) for the following.
1. To list files:

FileList("Sources/QueryPassThru (utility)")

or

FileList("Sources/QueryPassThru (utility)","","*.xml")

2. To display file content:

ReadFile("Sources/QueryPassThru (utility))","DEVQueryPassThru2018-01-02T030405_678.xml")

Note, the filename used above (“DEVQueryPassThru…”) can be made readily available if written in the Operation log. See this in the above NameGen use. In all cases, I use Ctrl-C and Ctrl-V for quick Cut & Paste.
There are a couple caveats to using Test Connection, FileList() and/or ReadFile() above in Script Pad. These caveats only apply to these file use cases and do not apply when Operations are executing.

  • If using a Private Agent Group with two or more Agents or the Jitterbit Cloud Agent Group, the TemporaryStorage directory may not be found. This failure is because the directory is on each group Agent and executions can occur on any Agent. You could receive an empty array for #1 (or empty Test Connection results) and a “File not found” error for #2 above. Re-testing may switch to another Agent.
  • When using Script Pad, you are using the Agent in a “test mode” so the Script output may be limited to some 50K bytes. (Later releases remove this restriction.) If you use ReadFile() on a file with a size greater than 50K bytes, ReadFile() may fail with an error indicating this limitation.

If one or both of these cases prevent effective use, an alternative is to define an Email Message and use this in a Target of the type of Email to send the file to you. See Create an Email Target for details. Here, you would have to change your Operation by replacing your temp file Target with the Email Target as the alternative to using Script Pad.
Another approach is to use WriteToOperationLog(ReadFile(…)) in an Operation’s wrapup script. (A script at the end of an Operation — another best practice — for concluding tasks.) The log may truncate after the same 50K byte limit.
Also, you may need to execute FlushFile() before the WriteToOperationLog(ReadFile()) above so the file is available for reading within the same Operation.
Swapping Targets, Sources, Scripts, and even Transformations are approaches used in unit and regression testing Operations and Operation chains. This testing approach is a topic for a later post.

Cloud Cached Variables

In later versions of Jitterbit Studio and Agents, one can persist data values in the cloud for use between Projects and Operations. The persistence lasts for a prescribed time up to 30 days. These are used to store simple values like “last execution time” for use in future executions, but the storage limit is actually up to one megabyte. When using data across projects in a multi-agent environment, a cloud-cached variable might be a viable option. This solution avoids the temp storage directory problem in multi-agent groups cited earlier. The creation and retrieval of values use specific Jitterbit cache functions so we will detail this in a future post.

Global Variable Source and Targets

Also in later versions of Jitterbit Studio and Agents, you can use a Global Variable for a Source or Target. This component type is useful if you want to display a variable assignment or global variable source being read in the Operation flow. The goal might be an entire Transformation target for reading as a Global Variable Source. If you choose to use a Global Variable Target (or Source), it’s important to remember the value is not persistent. Note, one can always use ReadFile() to assign the temp storage contents’ to a global variable. Global variable sources and targets offer design flexibility that can provide faster understanding.