Input/Output Interface - IOI - Documentation

Triggers

Triggers acts like "events" in IOI configuration. Detailed list of all triggers can be found in section Triggers. An example trigger configuration is listed below:

Trigger configuration template

                  <!-- Type attribute specifies the trigger type (without the "Trigger" suffix, e.g. FileAdded) -->
<IOInterfaceTrigger Type="">

    <!-- trigger specific properties should be here, if needed -->

    <!--If only one task is to be executed, Its name comes here-->
    <Task>NameOfTask</Task>

    <!-- For multiple tasks -->
    <Tasks>
        <Task Name="NameOfTask1">
            <!--there can also be some variables specified for task-->
            <Variables>
                <Variable Name="Variable1" Value="lorem ipsum" />
            </Variables>
        </Task>

        <!--second task without parameters-->
        <Task Name="NameOfTask2"></Task>
    </Tasks>
</IOInterfaceTrigger>
                

Example 1: Timer

Each 5 minutes run one task

                  <IOInterfaceTrigger Type="Timer">
  <Interval>300</Interval>
  <RunOnStart>true</RunOnStart>
  <Tasks>
    <Task Name="Invoice_SP2SAPTask" />
  </Tasks>
</IOInterfaceTrigger>
                

Example 2: FileAdded

Triggers two tasks with different variable value, when an file is added to specified folder

                  <IOInterfaceTrigger Type="FileAdded" MaxDop="1">
  <!-- folder where trigger watches for new files -->
  <Folder>D:\IOI_PROD\data\InvoicesInput</Folder>
  <Filter>*.xml</Filter>
  <Tasks>
    <Task Name="Invoice_Task">
      <Variables>
        <Variable Name="SpListName" Value="InboundInvoicesArchive" />
      </Variables>
    </Task>
    <Task Name="Invoice_Task">
      <Variables>
        <Variable Name="SpListName" Value="InboundInvoices" />
      </Variables>
    </Task>
  </Tasks>
</IOInterfaceTrigger>
                

Tasks

Tasks are defining the workload IOI is doing. It is composed by the three main elements:

• Input
• Processor(s)
• Output

1. Input retrieves data from a source (file/database/SharePoint/...)
2. Input data are broken down into rows and columns. Breakdown mechanism is either using a parser or comes out of input data logic (e.g. for SQL query it is query result set)
3. IOI engine processes each row and applies record processor(s).
4. Each row is passed into output which is then processing data depending on output type (writes data to file, sends data to SharePoint, executes SQL command etc.)
5. After processing of data, batch processors are applied (either success or error ones)

There are two types of processors used in task composition. Row processors are applied as described in point 2 on each data row. Batch processors are the other type, which might be applied before (or/and) after whole task processing. Depending on result of the task run, error or success processors are applied.

Following code snippet shows a template of IOI task with explaining comments. It contains all phases/elements for processors for different phases. The only ones required are Input and Output as they are the bare minimum for a runnable task.

                  <!-- Empty IOI task template (Full) -->
<IOInterfaceTask>
  <PreProcess>
    <!-- Optional. Before start of processing, batch processors to be fined here -->
  </PreProcess>
  
  <!-- Input for the task, required element, type of input specified in Type attribute -->
  <Input Type="">
  </Input>

  <Process>
    <!-- Optional. Record (row) processors to be defined here -->
  </Process>
  
  <Error>
    <!-- Optional. In case of failure, record processors to be fined here -->
  </Error>

  <Success>
    <!-- Optional. After record (row) was successfully processed, record processors to be fined here -->
  </Success>

  <!-- Output of the task, required element, type of input specified in Type attribute -->
  <Output Type="">
  </Output>
  
  <PostProcess>
    <!-- Optional. After successful processing of all records (rows), batch processors to be fined here -->
  </PostProcess>
  
  <TaskError>
    <!-- Optional. in case of failure, batch processors to be fined here -->
  </TaskError>  
</IOInterfaceTask>

                

Batches

There are two important terms used in IOI which needs to be explained:

• Batch - Batch is the overall name for data that is processed by a task. It is an entity that is determined by the execution of a trigger. E.g. for the FileAdded trigger, there will be one batch per file.
• Record - Record equals to set of data that represents one row of virtual table which comes out of Input. For example for CSV file, there will be literally one record per CSV file row (excluding header, if present.). For creating records out of an input stream (like a disk file is), Parsers (which are described later in detail) might be used. To merge records back to output stream, Formatters might be used, if appropriate.

Processors

There are two types of processors in IOI. Task processors which are executed for Batches and Record processors which are executed for each record. Some processors are available in both categories. For example, if a file should be moved after it has been processed, MoveFile task processor could be used. On the other hand, if it is required to send email per each row in input, the SendMail record processor could be used.

Replace tokens

There is a concept widely used in IOI which is called replace tokens. It is a specific string syntax that gets "replaced" during IOI processing. This means, it is possible to use specific syntax in IOI configuration files (and even in data) that is in runtime replaced by actual value. For example, there might be a variable that stores the file name of the file, which is the source of the data that is being processed (for a task that was triggered by the FileAdded trigger). This file name can be used in Task processing by using a replace token.

Replace tokens are wrapped into curly brakets and consists of two parts: prefix and the actual replace token. Syntactic template:

{(PREFIX):(REPLACE_TOKEN)}

{var:TriggerFileName}

Following table contains all available prefixes that might be used in replace tokens:

{func:Replace({rec:FieldA},'X','Y')}

{func:expr(rec_FieldA.replace(/X/g,'Y'))}

Replace functions

Functions provide specific functionality described for each function in the table below. Function can be without parameters, or could expect one or more parameters. Number of parameters is defined for each function separately depending on its purpose. Parameters are expected to be in parenthesis, each separated by colon. String parameters are not wrapped in quotation marks. An example of function is GetFileName function, which expects one parameter containing physical file path. Returns name of file (without path information). Example of usage of GetFileName function:

                  {func:GetFileName(C:\data\input\invoice010.pdf)}
                

Output of the replace token would be: invoice010.pdf

Replace tokens could be called recursively (means they can be nested one into another). An example of using replace function with parameter stored in variable:

                  {func:GetFileName({var:FileName})}
                

Variables

Variable is a key-value pair used widely in IOI. Variables can be defined at different levels:

• Global IOI variables defined in app config
• Variables defined in Triggers as Task arguments
• Predefined variables in Task / variables set in runtime in Task

Expressions

There is a number of entities, which are using expressions (e.g. If processor, expression replace token etc.). These expressions are based on JavaScript syntax (IOI evaluates expressions using a JavaScript interpret). So most of common JavaScript capabilities is available. Values can be passed to expressions in following ways:

• Replace Tokens - All replace tokens are processed before expression is passed to JavaScript evaluation engine. This means, that any variable can be used as string constant in expression, for example:

  '{var:MyVariable}'.length > 20

  Please note that string constants must be enveloped in quotes and that this is a potential vulnerable place for script injection.
• Variables - All variables present in current context are passed to JavaScript evaluation as JavaScript variables. They are named using var_ prefix to IOI variable lowercase name. For example, IOI variable named MyVariable will be available in javascript as var_myvariable. Example of usage in expression:

  var_myvariable.length > 20

• Record values - for record processor, record values are passed to expression evaluator as JavaScript variables with rec_ prefix, name of column lowercased. So for record column FileName, the JavaScript variable will be named rec_filename. Example of use:

  rec_filename == 'MyFile.xlsx'

Windows service

Installation as windows service is most common for production use. Windows service can be started on window startup and, in the event of failure, restarted. On windows service shutdown, process is not terminated immediately, but it waits until all the already running tasks are finished. In this stage, service shows "stopping" state in service manager and is not throwing any new task instances by triggers. Service manager might say that service is unavailable, but it might be waiting for batch to be finished. Information about running Batches can be found in log.

1. Unzip files from a zip archive to a local disk folder. For instance, as per the standard practice, it could be located at D:\Your_Folder\Your_Application\bin (e.g. D:\TreeINFO\IOI_PROD\bin).
2. Rename IOInterface.WinService.exe.config.sample to IOInterface.WinService.exe.config
3. Configure application configuration file options, as described in section App configuration.
4. Start powershell console as administrator
5. Register windows service using following command: New-Service -Name "IOInterface_PROD" -BinaryPathName "D:\TreeINFO\IOI_PROD\bin\IOInterface.WinService.exe" (change the binary path and service name according to your needs/physical binary location)
6. Set windows service executing account using windows service manager (can be started by pressing windows key and R, then typing "services.msc"). It is recommended the IOI service account to be local administrator and to create separated user only for IOI. Could also be an ActiveDirectory user.
7. Configure logging in nlog.config (if specific logging is required. Default config is supplied in package. Logging uses NLOG library.)
8. Start the service and check log content if there is not any error message

IIS web server

1. Unzip files from a zip archive to a local disk folder. For instance, as per the standard practice, it could be located at D:\Your_Folder\Your_Application\web (e.g. D:\TreeINFO\IOI_PROD\web).
2. Rename Web.config.sample to Web.config
3. Configure application configuration file options, as described in section App configuration.
4. Register IOI http handler in section system.webServer/handlers. Example (Change path attribute according to your needs):
5. Install IIS windows feature, if not already installed
6. Create IIS website in IIS management console. Set working directory to IOI binary path, create app pool with user with sufficient permissions to execute IOI operations (it can be local admin if security standards allows it)
7. Configure logging in nlog.config (if specific logging is required. Default config is supplied in package. Logging uses NLOG library.)

Note For each http handler, there must also be a trigger of type "WebService" referenced in Triggers config section with the same name in the attribute Name.

App configuration file

App configuration file is main configuration file for IOI. It is a XML file, standard .NET application config with a custom section for IOI configuration. For standard .NET configuration capabilities, please reffer MS documentation. An example of IOI Application configuration file:

<configuration>
  <configSections>
    <section name="IOInterface" type="IOInterface.Data.AppConfiguration.IOInterface,IOInterface.Data" />
  </configSections>
  <connectionStrings>
    <!-- Configurations database (optional)-->
    <add name="ConfigDB" connectionString="Data Source=(local);Initial Catalog=IOI_Config;Integrated Security=True" />
    <!-- Default data database (when not specified in task, this is used - optional) -->
    <add name="ConfigDefaultDataDB" connectionString="Data Source=(local);Initial Catalog=IOI_Data;Integrated Security=True" />
  </connectionStrings>

  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.7.2" />
  </startup>

  <IOInterface>
    <Management ConnectionType="Disabled" EnableRemoteManagement="False" Customer="Sabris Consulting" DisplayName="Production IOI instance" />
    <!--Paths for configuration file lookup-->
    <ConfigPaths>
      <ConfigPath Path="D:\Sabris\IOI_PROD\config" />
    </ConfigPaths>

    <!-- triggers to be enabled at service start-->
    <Triggers>
      <Trigger Name="FileToSP_Invoice"/>
      <Trigger Name="SPToXML_ExportSuppliers"/>
    </Triggers>

    <!-- Variables to be passed to IOI on start -->
    <Variables>
      <Variable Name="SPUrl" Value="http://devsp" />
      <Variable Name="TempFolder" Value="g:\Temp\ioi" />
    </Variables>

  </IOInterface>
</configuration>

IOInterface element sections