DOCUMENTATION

• 1: StreamZero FX
• 1.1: Architecture Overview
• 1.2: Developer Guide
• 1.2.1: Creating and Configuring Your First FX Service
• 1.2.2: FX Core Lib: Simplifying FX Service Development
• 1.2.3: Deploy a Service
• 1.2.4: Project and Code Structure
• 1.2.5: Secrets
• 1.2.6: State Management
• 1.2.7: Form Generator
• 1.2.8: Database Integration
• 1.2.9: Event Source Adapters
• 1.2.10: Extending the Platform
• 1.2.11: Git Integration
• 1.2.12: Event Manipulation Strategies
• 1.2.13: Logging and Monitoring
• 1.3: User Guide
• 1.3.1: Landing Page (Dashboard)
• 1.3.2: Projects
• 1.3.3: Taxonomy/Tagging
• 1.3.4: CronJob
• 1.3.5: Events
• 1.3.6: Executions - Packages
• 2: StreamZero K8X
• 2.1: K8X Developer Guide
• 2.2: User Guide
• 2.3: Architecture
• 3: StreamZero SX
• 3.1: Developer Guide
• 3.2: Integrations Guide
• 3.3: User Guide
• 3.4: Solutions snippets / explain problem solved / link to relevant use case
• 3.5: Containers + Purpose
• 3.6: Architecture
• 4: Integrations Guide
• 4.1: Database Guide
• 4.1.1: Supported Databases
• 4.1.1.1: Ascend.io
• 4.1.1.2: Amazon Athena
• 4.1.1.3: Amazon Redshift
• 4.1.1.4: Apache Drill
• 4.1.1.5: Apache Druid
• 4.1.1.6: Apache Hive
• 4.1.1.7: Apache Impala
• 4.1.1.8: Apache Kylin
• 4.1.1.9: Apache Pinot
• 4.1.1.10: Apache Solr
• 4.1.1.11: Apache Spark SQL
• 4.1.1.12: Clickhouse
• 4.1.1.13: CockroachDB
• 4.1.1.14: CrateDB
• 4.1.1.15: Databricks
• 4.1.1.16: Dremio
• 4.1.1.17: Elasticsearch
• 4.1.1.18: Exasol
• 4.1.1.19: Firebird
• 4.1.1.20: Firebolt
• 4.1.1.21: Google BigQuery
• 4.1.1.22: Google Sheets
• 4.1.1.23: Hana
• 4.1.1.24: Hologres
• 4.1.1.25: IBM DB2
• 4.1.1.26: IBM Netezza Performance Server
• 4.1.1.27: Microsoft SQL Server
• 4.1.1.28: MySQL
• 4.1.1.29: Oracle
• 4.1.1.30: Postgres
• 4.1.1.31: Presto
• 4.1.1.32: Rockset
• 4.1.1.33: Snowflake
• 4.1.1.34: Teradata
• 4.1.1.35: Trino
• 4.1.1.36: Vertica
• 4.1.1.37: YugabyteDB
• 4.2: Notifications and Messaging
• 5: Security
• 5.1: Permissions
• 5.2: Roles
• 5.3: Users
• 5.4: Statistics
• 6: Solutions
• 6.1:
• 6.2:
• 6.3: Automating Marketing Data Analysis
• 6.4: Churn Analysis and Alerting – Financial Services
• 6.5: Classification of products along different regulatory frameworks
• 6.6: Curn Analysis and Alerting – General
• 6.7: First Notice of Loss Automation
• 6.8: Idea to Trade / Next Best Product - Financial Services
• 6.9: Idea to Trade / Next Best Product - General
• 6.10: Intraday Liquidity Management Optimization
• 6.11: Metadata-controlled Data Quality & Data Lineage in Production
• 6.12: Onboarding and Fraud Remodelling - Financial Services
• 6.13: Onboarding and Fraud Remodelling - General
• 6.14: Prospecting 360 Degree
• 6.15: Prospecting 360 Degree - general
• 6.16: Regulatory Single Source of Truth
• 6.17: Sensor-based Monitoring of Sensitive Goods
• 6.18: Voice-based Trade Compliance
• 7: Release Notes
• 7.1: Release 1.0.2
• 7.2: Release 1.0.3
• 7.3: Release 1.0.4
• 7.4: Release 2.0.1
• 7.5: Release 2.0.2
• 8: Privacy Policy
• 9: Terms & Conditions

1 - StreamZero FX

StreamZero FX is a platform for building highly scalable, cross-network sync or async microservices and agents.

The unique low learning curve approach significantly reduces the cost of deploying enterprise wide process and integration pipelines across disparate systems at a speed. While at the same time creating a platform with practically unbound access and ease of integration.

FX is a ground-up rethink of how both sync and async microservices are built in multi-cloud, highly volatile and fragmented software environments.

On FX you are effectively writing large applications by connecting “blocks” of code (Services) through Events. An approach that is highly intuitive and in line with iterative agile practices.

The following is a brief review of some of the benefits and features of StreamZero FX. Upcoming features are shown in italics.

1.1 - Architecture Overview

Concepts

StreamZero FX is based on 2 simple concepts - Services and Events

On FX you are effectively writing large applications by connecting “blocks” of code through Events.

Each Service is a self contained piece of functionality such as loading a file, running a database view rebuild or launching a container. You can link and re-link the blocks of code at anytime you like. The source code can be as big or as tiny as you like.

Each Service is triggered by an Event. Each Service also emits Events thereby allowing other Services to be triggered following (or during) the execution of a Service.

A Service can respond to multiple Event types, and a single Event Type may trigger multiple Services - Thereby allowing you to also extend your Application(s) on the fly with ease.

You are not required to think in terms of pre-defined DAGS and can rapidly and iteratively build, test and deploy your applications.

Services

SERVICES are collections of scripts and modules which are executed in sequence by the FX Executor.

Services are triggered by EVENTS, which are JSON messages which carry a header and payload. A Service can be Linked to one or more events.

Each script is provided with the Payload of the Event that triggered it. It is the job of the FX Router to send Events to the appropriate Service.

The following is a basic Service which parses the event sent to it and prints the payload.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from fx_ef import context

# The context.params object carries the payload of the incoming event
param_value = context.params.get('param_name')

# And this is how a service sends an event
my_event_type = "com.test.my_event_type"
data = {"my attribute": "my_value"}

context.events.send(my_event_type, data)

Events

Events are messages passed through the platform which are generated either by Services or by the StreamZero Manager(in the case of manually triggered runs and scheduled runs).

Events are in the form of JSON formatted messages which adhere to the CloudEvents format.

Events carry a Header which indicates the event type and a Payload (or Data section) which contain information about the event.

The following is a sample Event.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
    "specversion" : "1.0",
    "type" : "com.example.someevent", // The Event Type
    "source" : "/mycontext",
    "subject": null,
    "id" : "C234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "datacontenttype" : "application/json",
    "data" : {                       // The event payload as JSON
        "appinfoA" : "abc",
        "appinfoB" : 123,
        "appinfoC" : true
    }
}

Service Triggering

Services can be triggered in the following ways:

• Manually: By clicking the ‘Run’ button on the StreamZero FX Management UI.
• On Schedule: As a cron job whereas the Cron expression is in the service manifest.
• On Event: Where a package is configured to be triggered by the FX Router when a specific type of event(s) is encountered on the platform - also configured in the service manifest.

Irrespective of how a Service is triggered it is always triggered by an Event. In the case of Manual and Scheduled triggering it is the FX platform that generates the trigger event.

Late Linking

One of the most important features of the FX Platform is that you are not required to link the Service to an Event during the course of development. And you can also change the Trigger Event(s) post-deployment.

This approach gives you a great flexibility to:

• not having to think of pre-defined flows but to build the Flow as well as the Services iteratively.
• maintain and test multiple versions of the same Service in parallel.

The StreamZero FX Flow

At the core of the FX Platform messages (Events) are passed through Apache Kafka. These ’events’ are JSON formatted messages which adhere to the CloudEvents format.

Each Event consists of what may be simplified as Headers and Payload. The headers indicate the type of event and other attributes. Whereas the payload are the attributes or parameters that are sent out by Services in order to either provide information about their state or for usage by downstream Services.

The FX Router(s) is listening on the stream of Events passing through Kafka. Based on the configuration of the platform which is managed in the StreamZero Management UI the Router decides if a Service requires to be executed based on the Event contents. On finding a configured Handler the gateway sends a message to the Executor and informs it of which packages or scripts are required to be run.

The FX Executor(s) executes the Service. The Service may use any Python module that is embedded in the Executor and also uses the platform internal configuration management database(at present Consul) for storing its configurations. The Executor sends a series of Events on Service execution. These are once again processed by the FX Router.

The FX Executor provides infrastructure which tracks logs, maintains record of service metrics and operational data. The Operational information is first sent to appropriate Kafka Topics from where they are picked up by Ops-Data Sinks whose role it is to store data within Elasticsearch and in some cases also filter the data for the purpose of alerting or anomaly tracking. All operational data may be viewed and queried through tools such as Kibana and is also viewable on the FX Management UI.

Required Infrastructure

The following are the infrastructure components required for a StreamZero FX installation

1.2 - Developer Guide

1.2.1 - Creating and Configuring Your First FX Service

Creating and Configuring Your First FX Service in a Local Environment

This guide provides a clear walkthrough of the process for creating and simulating services, all within the comfort of your desktop IDE. By following these steps, you’ll be able to seamlessly generate and define services, and then simulate their behavior before taking them live.

Step 1: Create a Virtual Environment

Before you start working on your FX service, it’s a good practice to create a virtual environment to isolate your project’s dependencies. A virtual environment ensures that the packages you install for this project won’t conflict with packages from other projects. You can create a virtual environment using a tool like virtualenv:

1
2

# Replace "my_fx_project" with your desired project name
virtualenv my_fx_project-env

Activate the virtual environment:

1

source my_fx_project-env/bin/activate

Step 2: Set Environment Variable

Set the EF_ENV environment variable to “local” to indicate that you’re working in a local development environment:

1

export EF_ENV=local

Step 3: Project Directory Setup

Create a directory that will serve as the main project directory. Inside this directory, you will organize multiple services. For example:

1
2

mkdir my_fx_project
cd my_fx_project

Step 4: Create Service Directory

Within the project directory, create a subdirectory for your specific service. This directory should have a name that consists of alphanumeric characters in lowercase, along with underscores (_) and hyphens (-) – no spaces allowed:

1
2

mkdir my-service-name
cd my-service-name

Step 5: Create app.py

Inside the service directory, create an app.py file. This file will serve as the entry point for your FX service. In this file, import the necessary context from the fx_ef (core library) for your service:

1
2
3
4

# app.py
from ferris_ef import context

# Your service code starts here

Step 6: Run app.py

Run the app.py file. This step generates two JSON files:

• ef_env.json: Simulates the parameters, secrets, and configurations of the service.
• ef_package_state.json: Holds the execution state of the service.

The above 2 files are used to simulate the service environment and are not used at runtime. They should not be checked in to git. A sample .gitignore for FX projects is provided here The GitIgnore File

1

python app.py

Step 7: Expand Your Service

With the initial setup done, you can now expand the app.py file with your actual service logic. Build and implement your FX service within this file.

Step 8: Module Placement

It’s important to note that any modules (additional Python files) your service relies on should be placed within the same directory as the app.py file. FX does not support nested directories for modules.

By following these steps, you’ll be able to create your first FX service in a local environment, set up the necessary configurations, and start building your service logic. Remember to activate your virtual environment whenever you work on this project and customize the app.py file to match the functionality you want your FX service to provide.

Adding a manifest.json file to describe your FX service to the platform is an essential step for proper integration and communication. Here’s how you can create and structure the manifest.json file:

Step 9: Create manifest.json

Inside your service directory, create a manifest.json file. This JSON file will contain metadata about your service, allowing the FX platform to understand and interact with it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "name": "My FX Service",
  "version": "1.0.0",
  "description": "A brief description of my FX service.",
  "author": "Your Name",
  "entry": "app.py",
  "configuration": "ef_env.json",
  "executionState": "ef_package_state.json",
  "modules": [
    "app.py"
  ]
}

• "name": Provide a name for your FX service.
• "version": Specify the version of your FX service (e.g., “1.0.0”).
• "description": Add a brief description of what your service does.
• "author": Add your name or the author’s name.
• "entry": Point to the entry point of your service (usually app.py).
• "configuration": Reference the ef_env.json file that holds service parameters, secrets, and configurations.
• "executionState": Reference the ef_package_state.json file that holds the execution state of the service.
• "modules": List the modules that your service relies on. In this case, it’s just "app.py".

Step 10: Manifest Structure

The manifest.json file provides vital information about your FX service, making it easier for the platform to understand and manage your service’s behavior and dependencies.

By including this file and its necessary attributes, your service can be properly registered, tracked, and executed within the FX platform. This manifest file essentially acts as a contract between your service and the platform, enabling seamless integration.

Understanding manifest.json: Defining Your Service

The manifest.json file plays a crucial role in describing your application to the DX Platform, as well as to fellow users and developers. Below is a sample manifest.json file along with an explanation of its parameters:

manifest.json Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
  "description": "Service with manifest file",
  "entrypoint": "app.py",
  "execution_order": ["app_1.py", "app.py"],
  "tags": ["devops"],
  "trigger_events": ["ferris.apps.minio.file_uploaded"],
  "schedule": "54 * * * *",
  "allow_manual_triggering": true,
  "active": true
}

Parameters and Descriptions:

This manifest.json file provides essential metadata about your service, making it easier for both the platform and other users to understand its purpose, behavior, and triggers. By customizing these parameters, you tailor the service’s behavior to your specific requirements.

Feel free to integrate this explanation into your documentation. Adapt the content to match your documentation’s style and format. This section aims to provide users with a comprehensive understanding of the manifest.json file and its significance in defining FX services.

Step 11: Expand ef_env.json

The ef_env.json file plays a crucial role in simulating your service’s environment during development. While on the FX platform, parameters, configs, and secrets are managed differently, in the local environment, you can define these elements within this JSON file for simulation purposes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  "parameters": {
    "param1": "value1",
    "param2": "value2"
  },
  "secrets": {
    "secret_key1": "secret_value1",
    "secret_key2": "secret_value2"
  },
  "configs": {
    "config_key1": "config_value1",
    "config_key2": "config_value2"
  }
}

• "parameters": In the local environment, you can define parameters directly within this dictionary. These parameters are typically accessed within your service code using the fx_ef library.

• "secrets": Similarly, you can define secret values in this section. While on the platform, secrets will be managed through the UI and loaded into your service securely. During local simulation, you can include sample secret values for testing.

• "configs": For configuration values, you can specify them in this dictionary. However, on the FX platform, configuration values are usually managed through an external config.json file. This is done to keep sensitive configuration data separate from your codebase.

Important Note: Keep in mind that the ef_env.json file is only for simulation purposes. On the FX platform, parameters are passed through trigger event payloads, configurations come from the config.json file, and secrets are managed through the platform’s UI.

By expanding your ef_env.json file with the appropriate parameters, secrets, and sample configuration values, you’ll be able to effectively simulate your service’s behavior in a local environment. This allows you to test and refine your service logic before deploying it on the FX platform, where parameters, secrets, and configurations are handled differently.

Step 12: Exploring the fx_ef Library

In the following section, we’ll delve into the capabilities of the ferris_ef library. This library serves as a bridge between your FX service and the platform, allowing you to seamlessly implement various platform features within your service’s logic.

The fx_ef library encapsulates essential functionalities that enable your service to interact with the FX platform, handling triggers, events, and more. By leveraging these features, you can create robust and responsive FX services that seamlessly integrate with the platform’s ecosystem.

Here’s a sneak peek at some of the functionalities offered by the fx_ef library:

1. Event Handling: The library facilitates event-driven architecture, allowing your service to react to various triggers from the platform. Whether it’s an incoming data event or an external signal, the library provides the tools to manage and respond to events effectively.

2. Parameter Access: While on the FX platform, parameters are passed through trigger event payloads. The library offers methods to access these parameters effortlessly, enabling your service to make decisions and take actions based on the provided inputs.

3. Configuration Management: Although configuration values are typically managed through a separate config.json file on the platform, the fx_ef library simplifies the process of accessing these configurations from within your service code.

4. Secrets Handling: On the platform, secrets are managed securely through the UI. The library ensures that your service can access these secrets securely when running on the platform.

5. Service State Tracking: The library also assists in managing your service’s execution state, tracking its progress and ensuring smooth operation.

By tapping into the capabilities of the fx_ef library, you can build powerful and versatile FX services that seamlessly integrate with the FX platform’s functionalities. In the next section, we’ll dive deeper into the specifics of how to utilize these features in your service logic.

Stay tuned as we explore the fx_ef library in depth, unraveling the tools at your disposal for creating impactful and responsive FX services.

Feel free to adapt this content to your documentation’s style and structure. It’s designed to introduce users to the significance of the fx_ef library and prepare them for a deeper dive into its features and usage.

1.2.2 - FX Core Lib: Simplifying FX Service Development

FX Core Lib: Simplifying FX Service Development

The FX Helper package, available through the fx_ef library, offers an array of convenient functions that streamline the development of FX services. This guide walks you through the different ways you can leverage this package to access service configurations, parameters, secrets, and state within your service logic.

Accessing Package Configuration

Retrieve configuration values that influence your service’s behavior by using the context.config.get() method:

1
2
3

from fx_ef import context

value = context.config.get('some_configuration_key')

Accessing Execution Parameters

Access parameters that affect your service’s execution using the context.params.get() method:

1
2
3

from fx_ef import context

param_value = context.params.get('param_name')

Accessing Secrets

Easily access secrets stored on platform, project, or package levels with the context.secrets.get() method:

1
2
3

from fx_ef import context

secret_value = context.secrets.get('secret_name')

Setting Secrets

Set secrets on project and platform levels using the context.secrets.set() method:

1
2
3

from fx_ef import context

context.secrets.set(name="platform_secret", value={"somekey": "someval"}, context="platform")

Accessing Package ID and Name

Retrieve your package’s ID and name using the context.package.id and context.package.name attributes:

1
2
3
4

from fx_ef import context

package_id = context.package.id
package_name = context.package.name

Accessing and Updating Package State

Manage your service’s execution state with context.state.get() and context.state.put():

1
2
3
4

from fx_ef import context

state_data = context.state.get()
context.state.put("some_key", "some_value")

Logging

Leverage logging capabilities at different levels - DEBUG, INFO (default), ERROR, WARNING, and CRITICAL:

1
2
3
4
5
6
7
8
9

from fx_ef import context

context.logging.setLevel('INFO')

context.logging.debug("debug msg")
context.logging.info("info msg")
context.logging.error("error msg")
context.logging.warning("warning msg")
context.logging.critical("critical msg")

Scheduling Retry of Service Execution

Use the context.scheduler.retry() method to schedule the next execution of your service from within your script:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

from fx_ef import context

# Retry in 3 minutes
job_id = context.scheduler.retry(minutes=3)

# Retry in 3 hours
job_id = context.scheduler.retry(hours=3)

# Retry in 3 days
job_id = context.scheduler.retry(days=3)

# Retry on the 56th minute of the next hour
job_id = context.scheduler.retry(cron_expression="56 * * * *")

This guide provides insight into the powerful functionalities offered by the fx_ef library, making FX service development more efficient and intuitive. These tools empower you to create responsive and feature-rich FX services with ease.

1.2.3 - Deploy a Service

Deploying Services: A Step-by-Step Overview

In this section, we provide you with a concise yet comprehensive overview of the steps required to deploy a service or a collection of services onto the FX platform. Following these steps ensures a smooth transition from development to deployment.

Step 1: Check Services into Git

Before anything else, ensure your collection of services is properly versioned and checked into a Git repository. This guarantees version control and a reliable source of truth for your services.

Step 2: Create a Project in the UI

In the FX platform UI, initiate the process by creating a project. Projects serve as containers for your services, aiding in organization and management.

Step 3: Add Git Repository to the Project

Once your project is in place, seamlessly integrate your Git repository with it. This connection allows the platform to access and manage your services’ source code.

Step 4: Sync the Repository to the Platform

The final step involves syncing the repository you’ve connected to your project with the FX platform. This synchronization imports the services’ code, configurations, and other relevant assets into the platform environment.

By following these four fundamental steps, you’re well on your way to deploying your services onto the FX platform. Each of these steps plays a vital role in ensuring that your services are seamlessly integrated, accessible, and ready for execution within the FX ecosystem.

Detailed Deployment Process: From Git to FX Platform

This section breaks down the steps outlined earlier for deploying services onto the FX platform in detail, starting with checking services into Git.

Check Services into Git

Since familiarity with Git is assumed, we’ll briefly touch on this step. Within the FX platform, each directory within a Git Repository represents a distinct service. Files placed directly in the root directory of a Repository are not considered part of any service.

Create a Project in the UI

Creating Projects and Linking with Git Repository:

1. Create a New Project:

   • Navigate to the “Projects” section on the left menu, then select “List Projects.”
   • Click “+Add” to create a new project.

   

2. Name the Project:

   • Provide a name for the project.
   • Save the project.

     

3. View Project Details:

   • Click the magnifying glass icon to access the project’s details page.

4. Add a GitHub Repository:
   • Access the “Git Repositories” tab.
   • Click “+Add” to add an SSH repository URL.

5. Copy GitHub Repo:
   • Generate a public SSH key (if not done previously).
   • Login to your GitHub account.
   • Go to the repository you want to link.
   • Click the green “Code” button to reveal repository URLs.
   • Copy the SSH URL.

6. Paste SSH URL:

   • Paste the copied SSH URL into the platform.
   • Save to set up the repository.
   • A pop-up will display a platform-generated public key. This key should be added to the GitHub Repo’s Deploy Keys to enable syncing.

   

7. Add Public Key to GitHub:

   • Return to GitHub.
   • Go to Settings > Deploy Keys.
   • Click “Add Deploy Key.”
   • Paste the generated public key, name it, and add the key.

8. Synchronize the Repository:
   • Return to the FX platform.
   • Click “Sync Now” to sync the platform with the Git Repository.
   • Check the synchronized details page; branches will be added, and status changes.

9. Check the Synced Packages:
   • Verify imported packages by clicking the “List Packages” tab.
   • Note that the main branch is automatically synchronized. As development continues and multiple branches are used, they can also be synced individually.

10. Change Git Branch on the Platform:
    • Users can choose a specific branch to work on or test.
    • Access the Edit Repository details page.
    • Select the desired branch from the dropdown (e.g., “dev”).
    • Save the selection and synchronize packages.

11. Verify Synced Packages on Dev Branch:
    • Check the “List Packages” tab to confirm successful synchronization from the dev branch.

Managing Public Keys for Security and Access

It’s important to understand the dynamics of managing public keys to ensure security and controlled access within the FX platform environment. Here’s a breakdown of key considerations:

1. Regenerating Public Keys:

   • You can regenerate a public key at any time if there’s a concern that unauthorized access might have occurred.
   • Regenerated keys must be added to GitHub again and synchronized on the platform afterward.

2. Ensuring Synchronization:

   • Whenever a new public key is generated, it must be added to the respective GitHub repository.
   • Failure to complete this step will result in synchronization issues on the platform.

3. Synchronization and Key Addition:

   • When generating a new key, add it to GitHub’s Deploy Keys.
   • Afterward, ensure the key is synchronized on the platform to maintain access.

4. Revoking Access:

   • If a situation arises where platform access should be revoked, keys can be deleted directly on GitHub.

The meticulous management of public keys is essential for maintaining the security and integrity of your FX services. By being proactive in regenerating keys, properly adding them to GitHub, and ensuring synchronization on the platform, you’re taking steps to uphold a secure development and deployment environment.

Integrate these insights into your documentation, adapting the content to match your documentation’s tone and style. This note aims to provide users with a clear understanding of how to manage public keys effectively within the FX platform ecosystem.

1.2.4 - Project and Code Structure

In this section, we’ll delve into the structure and components that make up an FX service. Understanding the organization of services, repositories, and the various artefacts involved is pivotal for efficient development within the FX platform.

Understanding Projects

Within the FX Platform, a Project serves as a container for multiple Services. Projects don’t play a functional role; they primarily aid in organizing services based on functional relationships, solution domains, or user access groups.

A project can be associated with multiple git repositories, each containing a collection of services.

Repository Structure

In the DX platform, every directory within a Repository represents a distinct service. Files located in the root directory of a Repository are disregarded.

Service Artefacts

A service encompasses an assortment of scripts, modules, and assets, including configuration files. The following are the supported types of artefacts along with their respective roles:

Understanding the components that constitute a service, repository, and project sets the foundation for effective FX service development. With this knowledge, you can seamlessly create, organize, and manage your services within the DX platform.

Sample Repository and Directory Structure

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

Project
│
├── Repository
│   ├── service_1
│   │   ├── app.py
│   │   ├── manifest.json
│   │   ├── config.json
│   │   ├── secrets.json
│   │   ├── asset.txt
│   │   └── ...
│   ├── service_2
│   │   ├── app.py
│   │   ├── manifest.json
│   │   ├── config.json
│   │   ├── secrets.json
│   │   ├── asset.txt
│   │   └── ...
│   └── ...
│
└── Repository ...

1.2.5 - Secrets

Secrets are sensitive configuration information which you wish to use within your service. These may be a single attribute(such as a password) structures with multiple attributes.

Secrets are the ability to deal with sensitive data through scripts (secrets.json) needed for package execution, such as:

• Database Passwords
• Secret Keys
• API Keys
• any other sensitive data

Secrets aren’t visible to any users and are passed encrypted to the actual script at the predefined package execution time. Once the script (secrets.json) is uploaded to the platform, the data is read and securely (double encryption) stored in the database.

Secret Scopes

The DX platform supports the following scopes for a secret.

When accessing secrets using fx_ef.context.secrets.get('secret_name') it will first lookup for secret_name within service secrets, then project and finally platform

The secrets.json File

To add service scope secrets you can upload a secrets.json file.

Those values are stored double encrypted in database and can be only accessed within the executing script. A sample secrets.json

1
2
3
4

{
    "DB_NAME": "test_db",
    "DB_PASS": "supersecretpas$"
}

Accessing secrets

With fx_ef.context.secrets you can access secrets stored on at the platform, project or service scope.

1
2

from fx_ef import context
context.secrets.get('secret_name')

This command will first lookup for secret named secret_name within package secrets (defined in secrets.json file of the package). If such key doesn’t exist it will lookup for it within project secrets, and finally within platform’s secrets. If secret with such name doesn’t exist None will be returned.

Can be accessed using fx_ef.context.secrets.get('secret_name'). Can be set using context.secrets.set(“secret_name”, {“somekey”:“someval”}, “project”)`

Can be accessed using fx_ef.context.secrets.get('secret_name'). Can be set using context.secrets.set("secret_name", {"somekey":"someval"}, "platform")

Setting secrets

Using fx_ef.context.secrets.set(name, value, context) method you can set secrets on project and platform level.

1
2
3

from fx_ef import context

context.secrets.set(name="platform_secret", value={"somekey":"someval"}, context="platform")

Create a new package

Note that the package creation was presented in another submenu of the User Guide, so only the needed parameters will be filled in the package to showcase the Secrets functionality.

1. Click on Executions in the left side menu and on Packages

2. Click on Add to create a new package

3. Name the package
4. Click on choose file and add the python scrypt (test_secrets.py)
5. Click on Add more scripts and click on choose file to add the JSON script (secrets.json)
6. Click on Save to save the package

test_secrets.py script

This is an example script that shows how secrets from the secrets.json file can be accessed from a script at execution time using the get_secret() helper function from the fx_ef package.

1
2
3
4
5
6

from fx_ef import context

print(f"DB NAME: {context.secrets.get('DB_NAME')}")
print(f"DB PASS: {context.secrets.get('DB_PASS')}")

print(f"PACKAGE NAME: {context.params.get('package_name')}")

1.2.6 - State Management

One key aspect in reactive applications is how to manage state between runs.

With StreamZero FX this is simple. Each Service has a state object available at run time. All you need to do is the following.

1
2
3
4
5

from ferris_cli import context

my_state = context.state.get() # returns a state previously set
some_value = my_last_state.get('key')
context.state.put('Key','Value')

The state is stored across Service runs. A state log is also maintained and stored for reference and reload.

How it works

When a Service is started the state is loaded from the consul key store.

When a state is stored it is placed in Consul as well as sent to Kafka. The Kafka stream maintains an audit log of the state. And also serves to retreive state after a system shut down.

1.2.7 - Form Generator

Occasionally you will come across use cases where you are required to provide a frontend for trigerring a service - usually by a non-technical person. FX and K8X both provide the ability to define Forms using a simple JSON structure.

The Forms are generated automatically by the StreamZero Management UI based on the ‘parameters.json’ file.

When a service directory contains a parameters.json file the ‘Run’ Button on th Management UI will automatically change the icon to a ‘Form’ icon.

The parameters.json file can be added to an existing service directory. When doing so you need to ensure that within the manifest.json file the ‘allow_manual_trigerring’ is set to ’true’

The following is a template for a parameters.json file.

The parameters.json file

The parameters.json file contains a JSON definition of fields that will be rendered and presented to user upon manually triggering a package execution in order to gather the parameter values for running the package. This way, same package can be easily adapted and reused by different scenarios or environments simply by sending different parameter values to the same package.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101

{
  "fields": [
    {
      "type": "text",
      "label": "Some Text",
      "name": "some_text",
      "required": true,
      "description": "This field is required"
    },
    {
      "type": "textarea",
      "label": "Some Textarea",
      "name": "some_textarea"
    },
    {
      "type": "file",
      "label": "Some File",
      "name": "some_file",
      "data": {
        "bucket": "testbucket",
        "async": true
      }
    },
    {
      "type": "int",
      "label": "Some Number",
      "name": "some_number",
      "default": 1,
      "min": 0,
      "max": 10
    },
    {
      "type": "float",
      "label": "Some Float",
      "name": "some_float",
      "placeholder": "0.01",
      "step": 0.01,
      "min": 0,
      "max": 10
    },
    {
      "type": "select",
      "label": "Some Select",
      "name": "some_select",
      "default": "value 2",
      "choices": [
        {
          "title": "Choice 1",
          "value": "value 1"
        },
        {
          "title": "Choice 2",
          "value": "value 2"
        },
        {
          "title": "Choice 3",
          "value": "value 3"
        }
      ]
    },
    {
      "type": "multiselect",
      "label": "Some MultiSelect",
      "name": "some_multiselect",
      "default": ["value 2", "value 3"],
      "choices": [
        {
          "title": "Choice 1",
          "value": "value 1"
        },
        {
          "title": "Choice 2",
          "value": "value 2"
        },
        {
          "title": "Choice 3",
          "value": "value 3"
        }
      ]
    },
    {
      "type": "radio",
      "label": "Some Radio",
      "name": "some_radio",
      "choices": [
        {
          "title": "Choice 1",
          "value": "value 1"
        },
        {
          "title": "Choice 2",
          "value": "value 2"
        },
        {
          "title": "Choice 3",
          "value": "value 3"
        }
      ]
    }
  ]
}

The above template will display a form which looks as below.

When the form values and entered and the ‘Run’ Button is clicked the form parameters and values will be sent to the service on trigger and these will be available to the service just as if it were trigerred by an event with the same payload as the form values.

The following is a sample script that extracts the parameters (you will notice it is no different from an event trigerred script). The only exception are the text areas which are dealt with as String data type and therefore should be converted using the relevant JSON library.

1
2
3
4
5
6
7
8
9

from fx_ef import context
import json

event_type = context.params.get("sample_event_type")
event_source = context.package.name
data = json.loads(context.params.get("sample_payload"))

context.events.send(event_type, event_source, data=data)

1.2.8 - Database Integration

Install Database Drivers

StreamZero DX requires a Python DB-API database driver and a SQLAlchemy dialect to be installed for each datastore you want to connect to within the executor image.

Configuring Database Connections

StreamZero can manage preset connection configurations. This enables a platform wide set up for both confidential as well as general access databases.

StreamZero uses the SQL Alchemy Engine along with the URL template based approach to connection management. The connection configurations are maintained as secrets within the platform and are therefore not publicly accessible i.e. access is provided for administrators only.

Retrieving DB Connections

The following is how to retrieve a named connection. The following sample assumes that the connection identifier key is uploaded to the package as a secrets.json.

1
2
3
4
5
6
7
8

from fx_ef import context
import sqlalchemy as db

db_url = context.secrets.get('my_connection')
engine = db.create_engine(db_url)

connection = engine.connect()
metadata = db.MetaData()

In the above example the db_url is set up as a secret with name 'my_connection'.

Depending on whether this is a service, project or platform level secret there are different approaches to set up the secret. For service level secret the following is a sample set up for a secrets.json file of the package.

1
2
3

{
  "my_connection" = "mysql://scott:tiger@localhost/test"
}

• For Project scope use the 'secrets' tab of the Project Management UI.
• For Platform scope secrets use the Vault UI in the DX Manager Application.

Database Drivers

The following table provides a guide on the python libs to be installed within the Executor docker image. For instructions on how to extend the Executor docker image please check this page: /docs/extending_executor_image

You can read more here about how to install new database drivers and libraries into your StreamZero DX executor image.

Note that many other databases are supported, the main criteria being the existence of a functional SQLAlchemy dialect and Python driver. Searching for the keyword “sqlalchemy + (database name)” should help get you to the right place.

If your database or data engine isn’t on the list but a SQL interface exists, please file an issue so we can work on documenting and supporting it.

A list of some of the recommended packages.

1.2.9 - Event Source Adapters

The Event Source Adapter enables easy integration of external event streams to Ferris.

The role of the Event Source Adapter is to receive events from external streams, convert them into Cloud Events and push them to the ferris.events Kafka Topic. The Cloud Events that are generated will contain an indicator of the source, one or more specific event types (depending on the type of source and the use case) and the content of the source event in the payload of the output Cloud Event.

Example Event Source Adapters

The following are a couple of examples of source adapters

Generic Webhook Adapter : Exposes a webhook end point outside the cluster which may be used to submit events as webhook requets. The generic adapter may source multiple event types and does not filter the content. It may be used for example to simultaneously accept AWS EventBrige CouldEvents and GitHub Webhooks. It is the role of a package to filter or split events as is suited for the use case.

Twitter Adapter: Streams twitter based on configured hash tags and converts them to cloud events.

IBM MQ Adapter

Kafka Adapter: Sources data from JSON streams within kafka and converts them to Cloud Events.

Azure MessageBus Adapter:

Amazon SQS Adapter

MQTT Adapter

Redis Queue Adapter

ActiveMQ Source

Amazon CloudWatch Logs Source

Amazon CloudWatch Metrics Sink

Amazon DynamoDB Sink

Amazon Kinesis Source

Amazon Redshift Sink

Amazon SQS Source

Amazon S3 Sink

AWS Lambda Sink

Azure Blob Storage Sink

Azure Cognitive Search Sink

Azure Cosmos DB Sink

Azure Data Lake Storage Gen2 Sink

Azure Event Hubs Source

Azure Functions Sink

Azure Service Bus Source

Azure Synapse Analytics Sink

Databricks Delta Lake Sink

Datadog Metrics Sink

Datagen Source (development and testing)

Elasticsearch Service Sink

GitHub Source

Google BigQuery Sink

Google Cloud BigTable Sink

Google Cloud Functions Sink

Google Cloud Spanner Sink

Google Cloud Storage Sink

Google Pub/Sub Source

HTTP Sink

IBM MQ Source

Microsoft SQL Server CDC Source (Debezium)

Microsoft SQL Server Sink (JDBC)

Microsoft SQL Server Source (JDBC)

MongoDB Atlas Sink

MongoDB Atlas Source

MQTT Sink

MQTT Source

MySQL CDC Source (Debezium)

MySQL Sink (JDBC)

MySQL Source (JDBC)

Oracle Database Sink

Oracle Database Source

PagerDuty Sink

PostgreSQL CDC Source (Debezium)

PostgreSQL Sink (JDBC)

PostgreSQL Source (JDBC)

RabbitMQ Sink

RabbitMQ Source Connector

Redis Sink

Salesforce Bulk API Source

Salesforce CDC Source

Salesforce Platform Event Sink

Salesforce Platform Event Source

Salesforce PushTopic Source

Salesforce SObject Sink

ServiceNow Sink

ServiceNow Source

SFTP Sink

SFTP Source

Snowflake Sink

Solace Sink

Splunk Sink

Zendesk Source

Generic Webhook Adapter

The Edge Adapter exposes a single endpoint for Webhooks. The webhook may be used for a large number of incoming integrations. Some examples are provided below.

To see the API please visit webhook.edge.YOURDOMAIN.COM/ui . For example webhook.edge.ferris.ai .

In order to use the end point you must first generate a token to be used when submitting to the endpoint. To generate a token please follow instructions here ….

How it Works

The StreamZero Edge Adapter is an edge service which is exposed to services outside the network for incoming integrations with external services. It exposes a single token protected endpoint which accepts a JSON payload within a POST request.

The payload encapsulated within the POST is forwarded to the ferris.events topic with the data encapsulated in the Cloud Events ‘data’ section. The event type is ‘ferris.events.webhook.incoming’ .

The platform may host any number of packages which then process the webhooks based on parsing the data section.

The StreamZero Edge Adapter is one of the few services exposed to the Internet.

Integrations

The following sections document details on some of the possible integrations.

AWS EventBridge

AWS S3 ( please switch images)

A pre-requisite is to ensure that EventBridge is sending Events to Ferris. Please see this section on how to set it up.

Create a bucket and switch to the Properties Tab of the UI

Scroll to the bottom and turn on Event Bridge Notfications by clicking on the Edit button below the section Amazon EventBridge

GitHub Integration

To be notified on changes to a Git Hub Repo please follow the steps below.

Click on the ‘Settings’ icon for the repo

Select the Webhooks menu on the left of the ‘Settings’ page. Then click on the ‘Add webhook’ button.

Add the URL of your edge adapter end point. And ensure the content type is application/json. Finally add the api token generated on the StreamZero Management UI. Further in the page you may select what event types should be sent. If unsure please maintain the default settings.

Test your integration by pushing an update to the repository.

1.2.10 - Extending the Platform

Extending the Platform

The platform may be extended at 3 logical points within the event life cycle.

• At Entry Point:
  • They are respnsible for injecting external event streams into the platform. Primarily they mediate betweeen the exrernal event stream and the internal CloudEvents based Kafka Topics. These run on separate containers within the platform. The following are the typical examples.
    • Event Gateway: are the primary mechanism. To build event gateways we provide templates. Please check this document on extension.
• At Processing
  • These are extensions that operate on internal event streams or are required by services that are created on the platform. The following are the types thereof.
    • Configuration Adapters and UIs. These are primarily used for connection setups and configurations which are applicable across the platform. Examples are the variety of connection set up UIs we provide. They are very easy to create. Use the the followinng guide to build your own.
    • Python Libraries and Modules: These are attached to the executor. It primarily involves extending the executor image with the required library. In order to add them to the platform use this guide.
    • Event Processing Packages: These are services that only modify event attributes normally convert one type of event to another. These can be implemented as services within the platform. Please see following guide to see how they are used and some typical scenarios.
    • No Code Generators: Generators combine UI based with templated code to allow No Code based approach to creating services. Please check this guide on how that works.
• At Exit Point
  • These are primarily modules that interact with external systems but operate across the example. The primarily operate on streams that originate from the platform and mediate with the outside. These run on separate containers within the platform. The following are typical implementaions. Examples are
    • Protocol Adapters They adapt between the internal kafka even streams and external protocols for example webhook adapter, Kafka to imbm mq adapter etc. Their primary purpose is to offload activity from the platform which may cause bottle necks or require long running services.
    • Splitters and Filters: These may operate on strams to split content or event infromation into derivative streams. Or feed data into supporting infrastructure. The elastissearch and Splunk adapters are typical examples. In order to build these use the following guide and related templates.

1.2.11 - Git Integration

The Git Integration is the capability to generate a connection from a git repository with projects and synchronise the Packages from the Executor with the git repository, with the goal to execute it through the StreamZero FX Platform. It provides another, more fluent way for connecting scripts with the StreamZero FX Platform without the necessity to upload files directly to the platform.

A new Project will be created to showcase the capabilty of the git integration:

Create a new project

1. Click on Projects in the left side menu to open drop-down and then on List Projects
2. Click on +Add to create a new project

3. Name the project
4. Save the new project

Check the created project

1. Click on the magnifying glass to open the details page of the project

Add a GitHub Repository to the created project

1. Click on the Git Repositories tab
2. Click on +Add to add a SSH repository URL

Copy GitHub Repo

Note that before adding your GitHub Repository to the platform, a public SSH key needs to be generated.

1. Login to your GitHub account
2. Go to the Repository you want to add to the project, in this use case “ferris-packages”
3. Click on the the green Code button to showcase the repository URLs
4. Copy the SSH URL

Paste SSH URL

1. Paste the copied SSH URL from your repo
2. Click save to create the repository on the platform

Note that a pair of public and private keys are generated for each repository which is safed on the StreamZero FX platform. The private key is encrypted and stored safely in the database and will never be presented to anyone, whereas the public key should be copied and added to the git repository in order to provide the StreamZero FX access to the repository and the possibility to clone packages.

Add the public key to GitHub

1. Return to your GitHub account
2. Click on Settings in the top menu bar
3. Click on deploy keys
4. Click on Add deploy key

5. Paste the generated public key
6. Name the public key
7. Click on Add key

8. Check the saved public key

Synchronise the repository

1. Return to the StreamZero FX platform
2. Click the Sync Now button to synchronise the platform with the GitHub

3. Check the synchronised details page

Note that the branches (main; dev) were added and the status has changed (synced).

4. Click on the List Packages tab to verify that the packages were imported

Change Git Branch on the platform

If a user wants to test or work on a specific branch, he can select the branch required to do so. The main branch is selected by default.

1. Click on the edit button to open the Edit Repository details page

2. Click in the drop-down to select the branch, in thise case “dev”
3. Click on Save to save the selected branch

4. Click on Sync to synchronise the packages from the dev branch

5. Click on the List Packages tab to verify the packages have been synced from the dev branch

Note that a public key can be regenerated at any moment if in doubt that someone has access to it. If a new key gets generated, it needs to get added to GitHub again and synced on the platform afterwards. If the step of adding the key is missed, the synchronisation will fail. Keys can also be deleted directly on GitHub if the platform access shouldn’t be granted anymore.

1.2.12 - Event Manipulation Strategies

Events are the powerful concept at the center of the DX Platform. There are a number of strategies for using event structures. The following are a few important related topics.

• Correlation IDs
• Event Mappings

Understand the Structure

The StreamZero FX events are based on CloudEvents …

Understanding Correlation IDs

Correlation IDs are a time tested approach within the Enterprise Software Landscape. A correlation ID allows one to correlate 2 steps in a flow with eachother and identify their location in the flow sequence.

When a package receives and event the platform also passes a CorrelationID. The correlation ID is usually generated by the platform at the start of the event or assigned by the event originator. If a correlation ID does not exist then a package may create a correlation id using the library provided. The correlation id consists of 2 parts

1

FLOWID_SEQUENCEID

The first part is the identifier of the unique originator id. The second part is a sequence ID which is incrementally assigned by subsequent processors. This allows the processor to indicate which the next stage of the processing is. It is left to packages to determine whether they wish to pass through the correlation ID or not. Usually it is preferable to apss the correlation ID with any event that is generated from within a package.

The following is a sample output

ABCDEF1234_01 -> ABCDEF1234_02 -> ABCDEF1234_03

Searching for a correlation ID will result in a time sorted list of runs which were triggered. By steppoing through the rsults of each stage you can easily identify the outgoing events and the results at each stage.

Leverage Event Mapping

Event mapping is the mechanism of converting from one event type to the other.

This is useful for converting from one type of even to another to trigger crossflows without altering the code of the target service.

Event mapping is done within the platform by using a configuration of event maps. Event maps describe the mapping of the attributes between the source and the target event. The also must chose between 1 of 2 strategies

• Map ONLY Mapped Fields
• Map ALL Fieds

Strategy Map only Mapped Fields

When this strategy is applied only the attributes present in the mapping file will be available in the output event.

Please note that you cannot map events to the same event type to avoid loopbacks.

Map

1
2
3
4
5

{
  "ferris.sample.event_a": "ferris.sample.event_b",
  "name":"first_name",
  "role": "designation"
}

Source Event

1
2
3
4
5
6

{
  "type": "ferris.sample.event_a",
  "name":"Bal",
  "role": "developer"
  "mobile": "1234567"
}

Output Event

When the above map is combined with the event it will result in the name and role attributes being available as first_name and designation in the output event. But the mobile number will be stripped.

1
2
3
4
5

{
  "type": "ferris.sample.event_b",
  "first_name":"Bal",
  "designation": "developer"
}

Strategy Map All Fields

When this strategy is applied only the attributes present in the mapping file will be available in the output event.

Please note that you cannot map events to the same event type to avoid loopbacks.

Map

1
2
3
4
5

{
  "ferris.sample.event_a": "ferris.sample.event_b",
  "name":"first_name",
  "role": "designation"
}

Source Event

1
2
3
4
5
6

{
  "type": "ferris.sample.event_a",
  "name":"Bal",
  "role": "developer"
  "mobile": "1234567"
}

Output Event

When the above map is combined with the event it will result in the name and role attributes being available as first_name and designation in the output event. But the mobile number will be stripped.

1
2
3
4
5
6

{
  "type": "ferris.sample.event_b",
  "first_name":"Bal",
  "designation": "developer",
  "mobile": "1234567"
}

1.2.13 - Logging and Monitoring

Logging and Monitoring

StreamZero FX aggregates all operational data into Elasticsearch. Most operational data and events are transported through Kafka from which it is placed in Elasticsearch by Elasticsearch Sink containers.

The following are the Kex Data and The matching Elasticsearch Indexes

Logs

Contains Logs from all applications. Elasticsearch index is XYZ

Events

All events that are transported through the ferris.events Topic are loaded into Elasticsearch Index.

Checking Logs

StreamZero Logs are in logstash format. The logs can be aggregated from the application by using the ferris_cli library.

The following is a sample log entry with extended descriptions below.

Logs are identified by the app_name attribute which provides you with an indication of the application from which it was generated.

To Filter Application Logs use the following

App_name:

Checking Events

Events are in form of cloud events. The data section of an event is schema less i.e. the data provided in the attributes may vara from event type to event type. If you require custom extractions for specific event types the best is to tap into the

Event Name Spaces

1.3 - User Guide

1.3.1 - Landing Page (Dashboard)

The StreamZero FX Landing page provides insights and analytics around typical platform related metrics mostly related to Data Ops and detailed event handling. It can be finetuned and tailored to customer specific needs.

In this specific use case the insights and analytics of the StreamZero FX Data Platform are highlighted as follows:

• In the first row, the last 18 executions and the last 18 executions with failed state

  • the last 18 executions showcase the following details:

    • Package (name)
    • Status
    • Execution time
    • Finished

  • the last 18 executions with failed state showcase the following details:

    • Package (name)
    • Status failed
    • Triggered time

    It allows users of the platform to verify why the triggered package has failed executing.

• In the second row, the executions statuses per day (last 7 days) and the executions by status (last 7 days)

  • Completed
  • Failed
  • Pending
  • In_progress

• In the third row, the exectuions trigger type per day (last 7 days) and the exectuions by trigger type (last 7 days)

  • triggered
  • scheduled
  • manual

• In the 4th row, the average execution time per day (last 7 days) and the most recently updated packages

  • the details of the most recently updated packages are divided as follows:
    • Package
    • Project
    • Updated on (date and time)

• In the 5th row, the most frequently executed packages in the last 7 days with the following details:

  • Package (name)
  • Number of exections

1.3.2 - Projects

This subcategory explains the creation of new projects and users withtin these projects.

Project

1. Click on Projects in the menu on the left side to open dropdown and then on List Projects
2. Click on "+Add"

1. Name the new project
2. Save

Project on the StreamZero Platform form the overarching organizational bracket for different types of objects. All users, packages, scripts, parameters, secrets and other elements are organized into projects to ease enterprise data management. The default owner for new projects is the Platform Admin (PA).

User creation within the project

Since the Platform Administrator created the Project, he is automatically assigned as the Project Owner. All user roles on the project level are defined in detail as follows:

• Project Owner - has all permissions on project and related entities (packages, users) including deletion of project(s) (and users) created by the PO. By default it is the user that created project
• Project Admin - has all permissions as Owner except deletion
• Project User - has only list / view permissions

Please note that users without Platform Administrator role, do not see any projects and packages they are not assigned to. All project role rights are translated to package level also, eg. a user with Project User role will not be able to edit packages of that project, only to list/view them and run a (manual) execution

1. Click on the magnifying glass to open Project details page

2. Click on "+Add" to add a new user to the project

3. Choose a user from the drowdown
4. Choose a user role from dropdown (Owner; Admin; User)
5. Click Save

• Check the user has been created

Since new users were created, no package(s) are yet assigned to the project. A project needs to be assigned to an existing package as follows:

1. Click on Executions to open dropdown and then Packages in the menu on the left
2. Click on edit

3. Choose a project from dropdown
4. Click Save

Once that a project was assigned to a package, the users created within that project will be able to perform their chores.

Project Owner

1. Log in as “Project Owner”
2. Click on Projects to open dropdown and then List Projects

As defined above, a Project Owner can add new projects, view, edit and delete the projects he belongs to.

3. Click on Executions to open dropdown and then on Packages

As Project Owner, one can add new packages, view, edit, delete and trigger manual runs and delete the packages within the projects the user belongs to.

Project Admin

1. Log in as “Project Admin”
2. Click on Projects to open dropdown and then List Projects

As defined above, a Project Admin can add new projects, view and edit the projects he belongs to but can not delete anything.

3. Click on Executions to open dropdown and then on Packages

As Project Admin, one can add new packages, view, edit and trigger manually runs within the projects the user belongs to. The Project Admin can not delete the packages, runs or related elements.

Project User

1. Log in as “Project User”
2. Click on Projects to open dropdwon and then List Projects

As defined above, a Project User can only view the projects he belongs to. He can neither edit, nore delete anything.

3. Click on Executions to open dropdown and then on Packages.

As Project User, one can view packages and trigger manually runs within the projects the user belongs to. The Project User can not delete or edit packages, runs or related elements.

1.3.3 - Taxonomy/Tagging

Taxonomies or Tags describe the ability to organize and structure types and classes of objects and their correlations within executions/packages, events (event types) and workflows across any given application, use case or project. Tags are searchable and makes it easy to group and relate objects across different components and lifecycle stages.

As a generic base module “taggability” can easily be included in any model, use case or application by the developers/users.

Note: As of the current release the Taxonomy is universal across all projects, use cases and cannot be segregated along different functional domains. It is thus essential to create a unified naming convention to be shared among the different projects & user groups.

Taxonomies / Tags

1. Click on Taxonomies in the left menu to open dropdown and then on Tags
2. Click Add to create a tag

1. Name Tag
2. Save

• Check created Tag(s)

1. Click on the magnifying glass to open details (show tag) page
2. This will automatically transfer you to the tag details page
3. Click on List Packages to see in which packages the same tag is used
4. Click on List Workflows to see in which workflows the same tag is used (in this example no workflow is associated with the tag just created)
5. Click on Event Types to see in which event type the same tag is uses (in this example no event type is associated with the tag just created)
6. Click on the Edit icon (List tags page) to edit/rename a tag

Search Tag

1. Click Search on top of the List Tags / Details Page
2. Click Add Filter to choose a filter (currently only the “Name” filter is supported)
3. From the dropdown list choose the tag to be searched for

• Starts with
• Ends with
• Contains
• Equal to
• Etc.

4. Insert tag “Name”
5. Hit the Search button

• Check search results

1.3.4 - CronJob

CronJobs are used to schedule regularly recurring actions such as backups, report generation and similar items. Each of those tasks should be configured to recur for an indefinite period into the future on a regular frequency (for example: once a day / week / month). The user also can define the point in time within that interval when the job should start.

Example:

This example CronJob manifest would execute and trigger an event every minute:

1

schedule: "*/1 * * * *"

Cron Schedule Syntax

1
2
3
4
5
6
7
8
9

# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * *

For example, the line below states that the task must be started every Friday at midnight, as well as on the 13th of each month at midnight:

0 0 13 * 5

To generate CronJob schedule expressions, you can also use web tools like crontab.guru.

Useful Cron Patterns

20 Useful Crontab Examples

Here is the list of examples for scheduling cron jobs in a Linux system using crontab.

1. Schedule a cron to execute at 2am daily.

This will be useful for scheduling database backup on a daily basis.

1

0 2 * * *

• Asterisk (*) is used for matching all the records.

2. Schedule a cron to execute twice a day.

Below example command will execute at 5 AM and 5 PM daily. You can specify multiple time stamps by comma-separated.

0 5,17 * * *

3. Schedule a cron to execute on every minutes.

Generally, we don’t require any script to execute on every minute but in some cases, you may need to configure it.

* * * * *

4. Schedule a cron to execute on every Sunday at 5 PM.

This type of cron is useful for doing weekly tasks, like log rotation, etc.

0 17 * * sun

5. Schedule a cron to execute on every 10 minutes.

If you want to run your script on 10 minutes interval, you can configure like below. These types of crons are useful for monitoring.

*/10 * * * *

*/10: means to run every 10 minutes. Same as if you want to execute on every 5 minutes use */5.

6. Schedule a cron to execute on selected months.

Sometimes we required scheduling a task to be executed for selected months only. Below example script will run in January, May and August months.

* * * jan,may,aug

7. Schedule a cron to execute on selected days.

If you required scheduling a task to be executed for selected days only. The below example will run on each Sunday and Friday at 5 PM.

0 17 * * sun,fri

8. Schedule a cron to execute on first sunday of every month.

To schedule a script to execute a script on the first Sunday only is not possible by time parameter, But we can use the condition in command fields to do it.

0 2 * * sun  [ $(date +%d) -le 07 ] && /script/script.sh

9. Schedule a cron to execute on every four hours.

If you want to run a script on 4 hours interval. It can be configured like below.

1

0 */4 * * *

10. Schedule a cron to execute twice on every Sunday and Monday.

To schedule a task to execute twice on Sunday and Monday only. Use the following settings to do it.

1

0 4,17 * * sun,mon

11. Schedule a cron to execute on every 30 Seconds.

To schedule a task to execute every 30 seconds is not possible by time parameters, But it can be done by schedule same cron twice as below.

1
2

* * * * * /scripts/script.sh
* * * * *  sleep 30; /scripts/script.sh

13. Schedule tasks to execute on yearly ( @yearly ).

@yearly timestamp is similar to “0 0 1 1 *“. It will execute a task on the first minute of every year, It may useful to send new year greetings 🙂

1

@yearly /scripts/script.sh

14. Schedule tasks to execute on monthly ( @monthly ).

@monthly timestamp is similar to “0 0 1 * *“. It will execute a task in the first minute of the month. It may useful to do monthly tasks like paying the bills and invoicing to customers.

1

@monthly /scripts/script.sh

15. Schedule tasks to execute on Weekly ( @weekly ).

@weekly timestamp is similar to “0 0 * * mon“. It will execute a task in the first minute of the week. It may useful to do weekly tasks like the cleanup of the system etc.

1

@weekly /bin/script.sh

16. Schedule tasks to execute on daily ( @daily ).

@daily timestamp is similar to “0 0 * * *“. It will execute a task in the first minute of every day, It may useful to do daily tasks.

1

@daily

17. Schedule tasks to execute on hourly ( @hourly ).

@hourly timestamp is similar to “0 * * * *“. It will execute a task in the first minute of every hour, It may useful to do hourly tasks.

1

@hourly

1.3.5 - Events

FX is an event driven platform wich means that each action generating an event can be reused for further triggering of executions. Also within an executing script, an event can be generated and sent as a message. Each event is defined at least by it’s source, type and payload (data). Event message format is following the cloudevents standard. A list of all event types is maintained so the user can bound package execution to certain event type, which means that each time such an event is received, the package execution will be triggered.

Events

Events are messages passed through the platform which are generated by Services.

Events are in the form of JSON formatted messages which adhere to the CloudEvents format. They carry a Header which indicates the event type and a Payload (or Data section) which contain information about the event.

To have a better detailed understanding of how Events are generated, please refer to the Architecture subcategory in the Overview category.

Events

This use case defines how to configure a package to be triggered bt the FX Router when a specific type of event is observed on the platform.

1. Click on Events on the left side of the dashboard menu to open drop-down
2. Click on Event Types
3. Check the predefined Event Types
   • ferris.apps.modules.approvals.step_approval_completed
   • ferris.apps.modules.minio.file_uploaded

Events can be created within scripts during package execution by sending a message to the Kafka Topic using the ferris_cli python package. For example, a package can be bound to a file_upload event that is triggered every time a file gets uploaded to MinIO using FX file storage module. New event types will be registered as they are sent to the Kafka Topic using the ferris_cli python package.

Further details regarding ferris_cli can be found in the subcategory Development Lifecycle in the Developer Guide.

Executions - Packages -> file upload trigger event

In this use case an existing package will be edited to define the file upload event type.

1. Click on Executions on the left side of the dashboard menu to open drop-down
2. Click on Packages
3. Click on the edit record button to edit the existing package Test Package with Scripts

1. Delete the CronJob Schedule to allow a Trigger Event Type
2. Select the Value of the event type (ferris.apps.modules.minio.file_uploaded)
3. Save the edited package.

File Storage

To finalize the process, a file needs to be uploaded to a MinIO bucket (file storage).

1. Click on File Storage on the left side of the dashboard menu to open drop-down
2. Click on List Files
3. Click on +Add to upload a file to the bucket

1. Choose file to upload
2. Choose File Type (CSV Table; Plain Text; JSON)
3. Select the Bucket Name
4. Click on Save to save the file

To verify if the package execution has been triggered, go back to the initial, edited package.

1. Click on Executions on the left side of the dashboard menu to open drop-down
2. Click on Packages
3. Click on the magnifying glass to open the details page of the package Test Package with Scripts

It will automatically open the List Package Executions tab.

1. Check the last Event, date and time to verify it corresponds to the time the file was uploaded
2. Click on the magnifying glass to open the details page of the triggered execution

• Check the details page of the event triggered run

Workflow -> approval completed trigger event

To finalize the second trigger event (ferris.apps.modules.approvals.step_approval_completed), an existing Workflow will be used to trigger a Case Management that will need to get approved.

1. Click on Workflows on the left side of the dashboard menu to open the drop-down
2. Click on List Workflows
3. Click on the magnifying glass to show the details page of the workflow

Note that before even getting a closer look at the Workflow details, the Entrypoint Event is displayed -> ferris.apps.modules.minio.file_uploaded

Check the details in the JSON snippet to understand what or which event types will trigger the second event type. The first event type shown in the JSON snippet is: ferris.apps.modules.minio.file_uploaded -> which means that a file will need to get uploaded for the event to get triggered. The second event type shown in the JSON snippet is: ferris.apps.modules.approvals.step_approval_completed -> meaning the uploaded file will need to get approved in the Case Management module before the wanted event gets triggered.

Case Management -> -> approval completed trigger event

1. Upload a file to a bucket (the process of uploading a file was described in detail on top of this page)
2. Click on Case Management on the left side of the dashboard menu to open the drop-down
3. Click on Approvals

1.3.6 - Executions - Packages

The Executions/Packages is an event oriented framework that allows enterprise organizations the automation of script processing which can be triggered by:

• Manually: By clicking the ‘Run’ button on the StreamZero FX Management Server.
• On Schedule: As a cron job whereas the Cron expression is added on the UI.
• On Event: Where a package is configured to be triggered bt the FX Router when a specific type of event is observed on the platform.

It allows users to deploy their locally tested scripts without DevOps specific changes or the need to learn complex DSL (description and configuration language). In tandem with Git integrated source code management FX allows distributed and fragmented tech teans to easily deploy and test new versions of code in an agile way with changes being applied immediately.

Contiuous Change Integration / Change Deployment becomes a component based and building block driven approach, where packages can be configurable and parametrised. All scripts and their parameters like secrets and environment variables form packages which makes them reusable for similar jobs or event chains. Event based package triggering allows users to run multiple packages in parallel as a reaction to the same event.

Executions - Packages

Primary entities for “Executions” are packages which are composed by scripts that are executed in a predefined order.

Executions -> Packages

This Use Case defines how to create and run a new package.

1. Click on Executions on the left side of the dashboard menu to open drop-down
2. Click on Packages
3. Click on +Add to create a package

Create Package

config.json script

The config.json file contains key/value configuration pairs that can be accessed in scripts at execution time.

1

{"somekey":"some value 2"}

test_scr.py script

This is an example script that shows how configuration from config.json file can be accessed from a script. package_name will be passed to the script as argument and then can be used for fetching configuration using ApplicationConfigurator from ferris_cli python package.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

import sys, json
from ferris_cli.v2 import ApplicationConfigurator

fa = json.loads(sys.argv[1])

package_name = fa['package_name']
config = ApplicationConfigurator.get(package_name)

for k, v in config.items():
    print(f"{k} -> {v}")
    print(v)

Check Created Package

The created package should be triggered every 20 minutes of every hour but can also be run manually.

• Click on the magnifying glass icon to open the package’s details page

1. Check details page
2. Click on “Show Trigger Event”

1. Check the triggered event details
2. Close

Package Executions / Runs

• Click on the “Run” button down the page to run the package manually

It will automatically transfer you to the “List Package Executions” tab

1. Check runs/package executions to see if you manually triggered execution was processed
2. Click on the magnifying glass icon of your latest manually triggered run to open details page of the exectuion

1. Check the details “Show Package Execution” of the run/exection
2. Click on “List Steps” tab to see the steps of the execution

1. Check the steps of the run and status (completed; pending; unprocessed; failed)
2. Click on “Show Results” to verify the script for failed executions

• Close window

Note that currently only python and sql handlers are available, files of different type will be unprocessed.

Save a Run/Execution

1. Go back to the “List Package Executions” tab
2. Click on the edit icon to open make the run/execution editable

1. Name the execution/run
2. Describe the execution/run
3. Click “Saved” check box
4. Save

1. Click on Executions to open dropdown
2. Click on Saved Executions to check the saved run

In the next section “UI Generator”, the importance of the saved run will be showcased.

2 - StreamZero K8X

What is StreamZero K8X?

StreamZero K8X brings event driven automation to Kubernetes.

With K8X you can create service flows which span multiple containers written in different programming languages. K8X takes over the responsibility of launching the right container when an event arrives that is mapped to the container. Further it provides the container with the incoming parameters, the service specific configurations and secrets injected into the container environment.

Since each service or container is invoked upon an event trigger, they (service, container) are dormant and require no compute resources.

The event driven nature of K8X makes it not only easy to use and fast to deploy, it brings unprecedented levels of resources efficiency as well as decreases resource contention to any Kubernetes Cluster.

Benefits of K8X

K8X shares the benefits provided by StreamZero FX in that it enables easy to build and operate event-driven microservices platform. In contrast to FX it is no more limited to the services built in the Python Programming language - i.e. the services (and containers) may be written in any lanuage. These can leverage the simple approach of FX to retreive event parameters, service configurations and secrets.

• K8X’s first and foremost benefit is that it significantly decreases developer time to develop event-driven microservices.
• K8X provides a very low learning curve.
• K8X significantly decreases time spent on deployments and CI/CD by offering a built in deployment mechanism.
• K8X improves observability by allowing easy viewing of the status as well as logs of the associated containers.

How it works

The following is a brief explanation of how K8X works.

• Edge Adapters are responsible for sourcing events from external systems, converting the incoming events into cloud events and forwarding them to the appropriate topic in Kafka.
• These events are consumed by the K8X Hub which looks up the mapping of the event to the target services.
• The K8X hub then deploys the appropriate service/container and injects the event parameters, service configs and secrets to the container environment.
• The container executes the service.
• The K8X hub collects the logs from the container for monitoring of the container status.

2.1 - K8X Developer Guide

StreamZero K8X aims to make it easy to build event-driven microservices in polyglot environments. As such it gives you complete freedom in selecting the language of your choice.

In order to ’event-enable’ a service K8X requires 3 artefacts to be created.

• The manifest.json file: Which describes your service to the platform.
• The deployment.yaml: A standard kubernetes deployment file which defines your Kubernetes deployment.

Optional Files

• The parameters.json file: Which can be used to define UI Forms attached to the service for manaully trigerred runs. Please read the section on parameters.json to understand the structure of this file.
• The configs.json file: Defines configurations of the service.
• The secrets.json file: Any secrets that are to be associated with the service. These will be injected to the container on launch.

The manifest.json

The following is a sample manifest.json file.

1
2
3
4
5
6
7
8
9

{
  "name": "k8s_test_job",
  "type": "k8s_job",
  "description": "Deploying k8 job",
  "allow_manual_triggering": true,
  "active": true,
  "trigger_events": ["ferris.apps.minio.file_uploaded"],
  "tags": ["k8s"]
}

The following table describes the attributes of the manifest.json file.

The deployment.yaml file

The following is a sample deployment.yaml file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

apiVersion: batch/v1
kind: Job
metadata:
  name: hello-world
spec:
  template:
    spec:
      containers:
        - name: hello-alpine
          image: frolvlad/alpine-bash
          command: [ "/bin/bash", "-c" ]
          args: [ "echo BEGIN; env; sleep 5; ls -la /usr/bin; echo DONE!" ]
          env:
            - name: SERVICE_PORT
              value: "80"
      restartPolicy: Never

The above is a standard kubernetes job deployment yaml file. As you will note there is nothing special about it. When the above file is processed by K8X it will add the incoming parameters, service secrets and configs into the environment.

2.2 - User Guide

StreamZero K8X is fully integrated with FX in the StreamZero Management UI. Hence K8X jobs will be visible in projects view along with the FX based jobs.

FX and K8X jobs can also share the same GIT Repos.

List Services

The list services view of projects will display both FX and K8X jobs. K8X jobs are differentiated by the job type.

2.3 - Architecture

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

3 - StreamZero SX

Overview

StreamZero SX is a streaming automation solution for the StreamZero Platform. It utilizes Apache Kafka, the distributed message broker used by thousands of companies, to enable data processing across the data mesh.

StreamZero SX drastically simplifies the creation of data pipelines and deployment of data streams, speeding up the time it takes to build stream processing applications.

It automates sourcing, streaming, and data management, and widely reduces the need for engineers’ involvement in topics management, DevOps, and DataOps.

What is Stream-Processing

Stream processing is a data management technique that involves ingesting a continuous data stream to quickly analyze, filter, transform or enhance the data in real time. Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale.

Most known for its excellent performance and fault tolerance, Kafka is designed to deliver high throughput and at the same time maintain low latency for real-time data feeds. It can handle thousands of messages per second with an average latency of 5–15ms.

Kafka serves as an ideal platform for building real-time streaming applications such as online streaming analytics or real-time message queue.

Apache Kafka has several advantages over other tools. Some notable benefits are:

• Building data pipelines.
• Leveraging real-time data streams.
• Enabling operational metrics.
• Data integration across countless sources.

Common Struggles For Companies Trying to Implement Kafka as an Integration Pattern

Now, while Kafka is great for building scalable and high-performance streaming applications, it’s actually hard to implement and maintain.

1. For one thing, the system is large and complex, which is why most companies fail to meet their goals.
2. On top of that, integrating client systems with Kafka brings additional challenges that can be difficult even for experienced teams, because there are many different technical complexities that could potentially cause hiccups in your integration strategy. -> Data schema, supported protocol and serialization are just some of the examples.
3. As a result, Kafka requires a dedicated team with advanced knowledge and varying skill sets to handle its adoption — engineers, DevOps specialists, DataOps engineers, and GitOps experts.
4. Moreover, due to the complexity of the applications, especially the concern of scalability, it can take a significant time to build each application.

There are many steps involved: from defining and writing business logic, setting up Kafka and integrating it with other services, to automating and deploying the applications.

How Does StreamZero SX Address And Solve These Issues?

StreamZero SX takes streaming automation to a whole new level. And the way it works is simple. It removes the complexity of Kafka connections, integrations, setups, automation, deployments and gives the end user the opportunity to focus on building client applications instead of losing time learning how to manage Kafka.

But how exactly does StreamZero SX solve the common issues and pitfalls mentioned above? By simplifying all processes:

• It is easy to adop and therefore has a low learning curve: Users can start working with StreamZero SX and experience first results within an hour.
• It removes the all complexities of Kafka: Engineers focus strictly on business logic for processing messages. The StreamZero SX python package takes care of configuration, Kafka connections, error handling, logging, and functions to interact with other services inside StreamZero.
• It is flexible. StreamZero SX allows using different underlying images and install additional components or pip modules.
• It enables connecting services code automatically to Streams and Topics.
• It helps you to quickly iterate on your service architecture. With StreamZero SX, once the images are deployed and the services are running, results are displayed right away.
• It takes care of all the underlying core processes. This means that you don’t need to worry about any technical or operational considerations.
• It is highly scalable and provides flexibility to up- or down-scale at any time, adjusted to the user’s needs and the number of topic partitions.

With the experience and knowledge gained over the past 7 years, the StreamZero Labs team has built an out-of-the-box module that lets developers concentrate on coding while taking care of all the complex processes that come with stream-processing data integrations.

3.1 - Developer Guide

Overview

StreamZero is a container level solution for building highly scalable, cross-network sync or async applications.

Using the StreamZero SX platform to run and manage stream processing containers utilizing StreamZero messaging infrastructure significantly reduces the cost of deploying enterprise application and offers standardized data streaming between workflow steps. This will simplify the development and as result create a platform with agile data processing and ease of integration.

Getting started with Stream Processors

Take a look at this library for creating Stream Processors on top of Kafka and running them inside StreamZero platform: StreamZero-SX

Example of a Stream Processor

Below you can find an example application that is using StreamZero-sx python library functions to count the number of words in incoming messages and then sending the result to twitter_feed_wc Kafka topic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import json
from StreamZero_sx.core import app
from StreamZero_sx.utils import sx_producer


def process(message):
    message_new = dict()
    message_new['text'] = message['text']
    message_new['word_count'] = len(message['text'].split(' '))
    message_new_json = json.dumps(message_new)
    print(message_new_json)
    sx_producer.send(topic="twitter_feed_wc", value=message_new_json.encode('utf-8'))


app.process = process

Creating Docker Container

Below is an example of a dockerfile to create a Docker image for the Twitter Word Count application shown in the previous section. The user is free to use whatever base python image and then add StreamZero module and other libraries.

FROM python:3.9-alpine
#RUN pip install -i https://test.pypi.org/simple/ StreamZero-sx==0.0.8 --extra-index-url https://pypi.org/simple/ StreamZero-sx
RUN pip install StreamZero-sx
COPY twitter_word_count.py app.py

After the user have built an image and pushed it to some Docker image regitry, he can run it in StreamZero SX UI.

3.2 - Integrations Guide

How does it Work?

There are two main approaches to implementing the external notifications support.

• Implementation within a StreamZero SX container
• Implementation in an Exit Gateway

The 2nd option is used in platforms which are behind a firewall and therefore require the gateway to be outside the firewall for accessing external services. In these cases the adapter runs as a separate container.

Irrespective of the infrastructure implementation the service internal API (as illustrated above) does not change.

3.3 - User Guide

StreamZero SX Management UI

Create a Stream Adapter

After a developer has built an image of a stream processing task and stored it to a container register, we can configure and launch it with StreamZero Management UI.

On left side menu, open Stream Adapters menu and select “Stream Adapter Definition”. Fill in the details.

Go to the “List Stream Adapters” page. You should find your the Stream Adapter that we created on the list. You can start the container by clicking the “Run”-button. The download and start-up of the image can take few minutes.

When the Stream Adapter is running you can find it in the list of running adapters.

StreamZero also has a list of all the Kafka topics that are currently attached to Stream Adapters or available to Stream Adapters.

3.4 - Solutions snippets / explain problem solved / link to relevant use case

Twitter message processing example

The first example application is using StreamZero-sx python library to implement stream processor to count the number of words in incoming messages. The messages are queried from Twitter API with specific filter condition and then fed to the processor. The results are sent to a Kafka topic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import json
from StreamZero_sx.core import app
from StreamZero_sx.utils import sx_producer


def process(message):
    message_new = dict()
    message_new['text'] = message['text']
    message_new['word_count'] = len(message['text'].split(' '))
    message_new_json = json.dumps(message_new)
    print(message_new_json)
    sx_producer.send(topic="twitter_feed_wc", value=message_new_json.encode('utf-8'))


app.process = process

3.5 - Containers + Purpose

Creating a Docker Container

Below is an example of a dockerfile to create a Docker image for some StreamZero SX application. The user is free to choose what base python image to use and then add StreamZero module and other libraries.

FROM python:3.9-alpine
#RUN pip install -i https://test.pypi.org/simple/ StreamZero-sx==0.0.8 --extra-index-url https://pypi.org/simple/ StreamZero-sx
RUN pip install StreamZero-sx
COPY app.py utils.py

After the user have built an image and pushed it to a Docker image regitsry, they can run it in StreamZero SX Management UI.

3.6 - Architecture

StreamZero SX Architecture Principles

Stream processing is a technique for processing large volumes of data in real-time as it is generated or received. One way to implement a stream processing architecture is to use Docker containers for individual workflow steps and Apache Kafka for the data pipeline.

Docker is a platform for creating, deploying, and running containers, which are lightweight and portable units of software that can be run on any system with a compatible container runtime. By using Docker containers for each step in the stream processing workflow, developers can easily package and deploy their code, along with any dependencies, in a consistent and reproducible way. This can help to improve the reliability and scalability of the stream processing system.

Apache Kafka is a distributed streaming platform that can be used to build real-time data pipelines and streaming applications. It provides a publish-subscribe model for sending and receiving messages, and can handle very high throughput and low latency. By using Kafka as the backbone of the data pipeline, developers can easily scale their stream processing system to handle large volumes of data and handle failover scenarios.

Overall, by using Docker containers for the individual workflow steps and Apache Kafka for the data pipeline, developers can create a stream processing architecture that is both scalable and reliable. This architecture can be used for a wide range of use cases, including real-time analytics, event-driven architectures, and data integration.

Below is the high-level architecture diagram of StreamZero SX:

Required Infrastructure

The following are the infrastructure components required for a StreamZero SX installation

4 - Integrations Guide

4.1 - Database Guide

Install Database Drivers

StreamZero DX requires a Python DB-API database driver and a SQLAlchemy dialect to be installed for each datastore you want to connect to within the executor image.

Configuring Database Connections

StreamZero can manage preset connection configurations. This enables a platform wide set up for both confidential as well as general access databases.

StreamZero uses the SQL Alchemy Engine along with the URL template based approach to connection management. The connection configurations are maintained as secrets within the platform and are therefore not publicly accessible i.e. access is provided for administrators only.

Retrieving DB Connections

The following is how to retrieve a named connection. The following sample assumes that the connection identifier key is uploaded to the package as a secrets.json.

1
2
3
4
5
6
7
8

from fx_ef import context
import sqlalchemy as db

db_url = context.secrets.get('my_connection')
engine = db.create_engine(db_url)

connection = engine.connect()
metadata = db.MetaData()

In the above example the db_url is set up as a secret with name 'my_connection'.

Depending on whether this is a service, project or platform level secret there are different approaches to set up the secret. For service level secret the following is a sample set up for a secrets.json file of the package.

1
2
3

{
  "my_connection" = "mysql://scott:tiger@localhost/test"
}

• For Project scope use the 'secrets' tab of the Project Management UI.
• For Platform scope secrets use the Vault UI in the DX Manager Application.

Database Drivers

The following table provides a guide on the python libs to be installed within the Executor docker image. For instructions on how to extend the Executor docker image please check this page: /docs/extending_executor_image

You can read more here about how to install new database drivers and libraries into your StreamZero FX executor image.

Note that many other databases are supported, the main criteria being the existence of a functional SQLAlchemy dialect and Python driver. Searching for the keyword “sqlalchemy + (database name)” should help get you to the right place.

If your database or data engine isn’t on the list but a SQL interface exists, please file an issue so we can work on documenting and supporting it.

A list of some of the recommended packages.

4.1.1 - Supported Databases

4.1.1.1 - Ascend.io

Ascend.io

The recommended connector library to Ascend.io is impyla.

The expected connection string is formatted as follows:

ascend://{username}:{password}@{hostname}:{port}/{database}?auth_mechanism=PLAIN;use_ssl=true

4.1.1.2 - Amazon Athena

AWS Athena

PyAthenaJDBC

PyAthenaJDBC is a Python DB 2.0 compliant wrapper for the Amazon Athena JDBC driver.

The connection string for Amazon Athena is as follows:

awsathena+jdbc://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...

Note that you’ll need to escape & encode when forming the connection string like so:

s3://... -> s3%3A//...

PyAthena

You can also use PyAthena library (no Java required) with the following connection string:

awsathena+rest://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com/{schema_name}?s3_staging_dir={s3_staging_dir}&...

4.1.1.3 - Amazon Redshift

AWS Redshift

The sqlalchemy-redshift library is the recommended way to connect to Redshift through SQLAlchemy.

You’ll need to the following setting values to form the connection string:

• User Name: userName
• Password: DBPassword
• Database Host: AWS Endpoint
• Database Name: Database Name
• Port: default 5439

Here’s what the connection string looks like:

redshift+psycopg2://<userName>:<DBPassword>@<AWS End Point>:5439/<Database Name>

4.1.1.4 - Apache Drill

Apache Drill

SQLAlchemy

The recommended way to connect to Apache Drill is through SQLAlchemy. You can use the sqlalchemy-drill package.

Once that is done, you can connect to Drill in two ways, either via the REST interface or by JDBC. If you are connecting via JDBC, you must have the Drill JDBC Driver installed.

The basic connection string for Drill looks like this:

drill+sadrill://<username>:<password>@<host>:<port>/<storage_plugin>?use_ssl=True

To connect to Drill running on a local machine running in embedded mode you can use the following connection string:

drill+sadrill://localhost:8047/dfs?use_ssl=False

JDBC

Connecting to Drill through JDBC is more complicated and we recommend following this tutorial.

The connection string looks like:

drill+jdbc://<username>:<passsword>@<host>:<port>

ODBC

We recommend reading the Apache Drill documentation and read the Github README to learn how to work with Drill through ODBC.

4.1.1.5 - Apache Druid

Apache Druid

Use the SQLAlchemy / DBAPI connector made available in the pydruid library.

The connection string looks like:

druid://<User>:<password>@<Host>:<Port-default-9088>/druid/v2/sql

Customizing Druid Connection

When adding a connection to Druid, you can customize the connection a few different ways in the Add Database form.

Custom Certificate

You can add certificates in the Root Certificate field when configuring the new database connection to Druid:

<img src={useBaseUrl("/img/root-cert-example.png")} />{" “}

When using a custom certificate, pydruid will automatically use https scheme.

Disable SSL Verification

To disable SSL verification, add the following to the Extras field:

engine_params:
{"connect_args":
	{"scheme": "https", "ssl_verify_cert": false}}

4.1.1.6 - Apache Hive

Apache Hive

The pyhive library is the recommended way to connect to Hive through SQLAlchemy.

The expected connection string is formatted as follows:

hive://hive@{hostname}:{port}/{database}

4.1.1.7 - Apache Impala

Apache Impala

The recommended connector library to Apache Impala is impyla.

The expected connection string is formatted as follows:

impala://{hostname}:{port}/{database}

4.1.1.8 - Apache Kylin

Apache Kylin

The recommended connector library for Apache Kylin is kylinpy.

The expected connection string is formatted as follows:

kylin://<username>:<password>@<hostname>:<port>/<project>?<param1>=<value1>&<param2>=<value2>

4.1.1.9 - Apache Pinot

Apache Pinot

The recommended connector library for Apache Pinot is pinotdb.

The expected connection string is formatted as follows:

pinot+http://<pinot-broker-host>:<pinot-broker-port>/query?controller=http://<pinot-controller-host>:<pinot-controller-port>/``

4.1.1.10 - Apache Solr

Apache Solr

The sqlalchemy-solr library provides a Python / SQLAlchemy interface to Apache Solr.

The connection string for Solr looks like this:

solr://{username}:{password}@{host}:{port}/{server_path}/{collection}[/?use_ssl=true|false]

4.1.1.11 - Apache Spark SQL

Apache Spark SQL

The recommended connector library for Apache Spark SQL pyhive.

The expected connection string is formatted as follows:

hive://hive@{hostname}:{port}/{database}

4.1.1.12 - Clickhouse

Clickhouse

To use Clickhouse with StreamZero you will need to add the following Python libraries:

clickhouse-driver==0.2.0
clickhouse-sqlalchemy==0.1.6

If running StreamZero using Docker Compose, add the following to your ./docker/requirements-local.txt file:

clickhouse-driver>=0.2.0
clickhouse-sqlalchemy>=0.1.6

The recommended connector library for Clickhouse is sqlalchemy-clickhouse.

The expected connection string is formatted as follows:

clickhouse+native://<user>:<password>@<host>:<port>/<database>[?options…]clickhouse://{username}:{password}@{hostname}:{port}/{database}

Here’s a concrete example of a real connection string:

clickhouse+native://demo:demo@github.demo.trial.altinity.cloud/default?secure=true

If you’re using Clickhouse locally on your computer, you can get away with using a native protocol URL that uses the default user without a password (and doesn’t encrypt the connection):

clickhouse+native://localhost/default

4.1.1.13 - CockroachDB

CockroachDB

The recommended connector library for CockroachDB is sqlalchemy-cockroachdb.

The expected connection string is formatted as follows:

cockroachdb://root@{hostname}:{port}/{database}?sslmode=disable

4.1.1.14 - CrateDB

CrateDB

The recommended connector library for CrateDB is crate. You need to install the extras as well for this library. We recommend adding something like the following text to your requirements file:

crate[sqlalchemy]==0.26.0

The expected connection string is formatted as follows:

crate://crate@127.0.0.1:4200

4.1.1.15 - Databricks

Databricks

To connect to Databricks, first install databricks-dbapi with the optional SQLAlchemy dependencies:

1

pip install databricks-dbapi[sqlalchemy]

There are two ways to connect to Databricks: using a Hive connector or an ODBC connector. Both ways work similarly, but only ODBC can be used to connect to SQL endpoints.

Hive

To use the Hive connector you need the following information from your cluster:

• Server hostname
• Port
• HTTP path

These can be found under “Configuration” -> “Advanced Options” -> “JDBC/ODBC”.

You also need an access token from “Settings” -> “User Settings” -> “Access Tokens”.

Once you have all this information, add a database of type “Databricks (Hive)” in StreamZero, and use the following SQLAlchemy URI:

databricks+pyhive://token:{access token}@{server hostname}:{port}/{database name}

You also need to add the following configuration to “Other” -> “Engine Parameters”, with your HTTP path:

{"connect_args": {"http_path": "sql/protocolv1/o/****"}}

ODBC

For ODBC you first need to install the ODBC drivers for your platform.

For a regular connection use this as the SQLAlchemy URI:

databricks+pyodbc://token:{access token}@{server hostname}:{port}/{database name}

And for the connection arguments:

{"connect_args": {"http_path": "sql/protocolv1/o/****", "driver_path": "/path/to/odbc/driver"}}

The driver path should be:

• /Library/simba/spark/lib/libsparkodbc_sbu.dylib (Mac OS)
• /opt/simba/spark/lib/64/libsparkodbc_sb64.so (Linux)

For a connection to a SQL endpoint you need to use the HTTP path from the endpoint:

{"connect_args": {"http_path": "/sql/1.0/endpoints/****", "driver_path": "/path/to/odbc/driver"}}

4.1.1.16 - Dremio

Dremio

The recommended connector library for Dremio is sqlalchemy_dremio.

The expected connection string for ODBC (Default port is 31010) is formatted as follows:

dremio://{username}:{password}@{host}:{port}/{database_name}/dremio?SSL=1

The expected connection string for Arrow Flight (Dremio 4.9.1+. Default port is 32010) is formatted as follows:

dremio+flight://{username}:{password}@{host}:{port}/dremio

This blog post by Dremio has some additional helpful instructions on connecting StreamZero to Dremio.

4.1.1.17 - Elasticsearch

Elasticsearch

The recommended connector library for Elasticsearch is elasticsearch-dbapi.

The connection string for Elasticsearch looks like this:

elasticsearch+http://{user}:{password}@{host}:9200/

Using HTTPS

elasticsearch+https://{user}:{password}@{host}:9200/

Elasticsearch as a default limit of 10000 rows, so you can increase this limit on your cluster or set Feris’s row limit on config

ROW_LIMIT = 10000

You can query multiple indices on SQL Lab for example

SELECT timestamp, agent FROM "logstash"

But, to use visualizations for multiple indices you need to create an alias index on your cluster

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "logstash-**", "alias" : "logstash_all" } }
    ]
}

Then register your table with the alias name logstasg_all

Time zone

By default, StreamZero uses UTC time zone for elasticsearch query. If you need to specify a time zone, please edit your Database and enter the settings of your specified time zone in the Other > ENGINE PARAMETERS:

{
    "connect_args": {
        "time_zone": "Asia/Shanghai"
    }
}

Another issue to note about the time zone problem is that before elasticsearch7.8, if you want to convert a string into a DATETIME object, you need to use the CAST function,but this function does not support our time_zone setting. So it is recommended to upgrade to the version after elasticsearch7.8. After elasticsearch7.8, you can use the DATETIME_PARSE function to solve this problem. The DATETIME_PARSE function is to support our time_zone setting, and here you need to fill in your elasticsearch version number in the Other > VERSION setting. the StreamZero will use the DATETIME_PARSE function for conversion.

4.1.1.18 - Exasol

Exasol

The recommended connector library for Exasol is sqlalchemy-exasol.

The connection string for Exasol looks like this:

exa+pyodbc://{username}:{password}@{hostname}:{port}/my_schema?CONNECTIONLCALL=en_US.UTF-8&driver=EXAODBC

4.1.1.19 - Firebird

Firebird

The recommended connector library for Firebird is sqlalchemy-firebird. StreamZero has been tested on sqlalchemy-firebird>=0.7.0, <0.8.

The recommended connection string is:

firebird+fdb://{username}:{password}@{host}:{port}//{path_to_db_file}

Here’s a connection string example of StreamZero connecting to a local Firebird database:

firebird+fdb://SYSDBA:masterkey@192.168.86.38:3050//Library/Frameworks/Firebird.framework/Versions/A/Resources/examples/empbuild/employee.fdb

4.1.1.20 - Firebolt

Firebolt

The recommended connector library for Firebolt is firebolt-sqlalchemy. StreamZero has been tested on firebolt-sqlalchemy>=0.0.1.

The recommended connection string is:

firebolt://{username}:{password}@{database}
or
firebolt://{username}:{password}@{database}/{engine_name}

Here’s a connection string example of StreamZero connecting to a Firebolt database:

firebolt://email@domain:password@sample_database
or
firebolt://email@domain:password@sample_database/sample_engine

4.1.1.21 - Google BigQuery

Google BigQuery

The recommended connector library for BigQuery is pybigquery.

Install BigQuery Driver

Follow the steps here about how to install new database drivers when setting up StreamZero locally via docker-compose.

echo "pybigquery" >> ./docker/requirements-local.txt

Connecting to BigQuery

When adding a new BigQuery connection in StreamZero, you’ll need to add the GCP Service Account credentials file (as a JSON).

1. Create your Service Account via the Google Cloud Platform control panel, provide it access to the appropriate BigQuery datasets, and download the JSON configuration file for the service account.
2. In StreamZero you can either upload that JSON or add the JSON blob in the following format (this should be the content of your credential JSON file):

{
        "type": "service_account",
        "project_id": "...",
        "private_key_id": "...",
        "private_key": "...",
        "client_email": "...",
        "client_id": "...",
        "auth_uri": "...",
        "token_uri": "...",
        "auth_provider_x509_cert_url": "...",
        "client_x509_cert_url": "..."
    }

3. Additionally, can connect via SQLAlchemy URI instead

   The connection string for BigQuery looks like:

   bigquery://{project_id}
   

   Go to the Advanced tab, Add a JSON blob to the Secure Extra field in the database configuration form with the following format:

   {
   "credentials_info": <contents of credentials JSON file>
   }
   

   The resulting file should have this structure:

   {
    "credentials_info": {
        "type": "service_account",
        "project_id": "...",
        "private_key_id": "...",
        "private_key": "...",
        "client_email": "...",
        "client_id": "...",
        "auth_uri": "...",
        "token_uri": "...",
        "auth_provider_x509_cert_url": "...",
        "client_x509_cert_url": "..."
        }
    }
   

You should then be able to connect to your BigQuery datasets.

To be able to upload CSV or Excel files to BigQuery in StreamZero, you’ll need to also add the pandas_gbq library.

4.1.1.22 - Google Sheets

Google Sheets

Google Sheets has a very limited SQL API. The recommended connector library for Google Sheets is shillelagh.

4.1.1.23 - Hana

Hana

The recommended connector library is sqlalchemy-hana.

The connection string is formatted as follows:

hana://{username}:{password}@{host}:{port}

4.1.1.24 - Hologres

Hologres

Hologres is a real-time interactive analytics service developed by Alibaba Cloud. It is fully compatible with PostgreSQL 11 and integrates seamlessly with the big data ecosystem.

Hologres sample connection parameters:

• User Name: The AccessKey ID of your Alibaba Cloud account.
• Password: The AccessKey secret of your Alibaba Cloud account.
• Database Host: The public endpoint of the Hologres instance.
• Database Name: The name of the Hologres database.
• Port: The port number of the Hologres instance.

The connection string looks like:

postgresql+psycopg2://{username}:{password}@{host}:{port}/{database}

4.1.1.25 - IBM DB2

IBM DB2

The IBM_DB_SA library provides a Python / SQLAlchemy interface to IBM Data Servers.

Here’s the recommended connection string:

db2+ibm_db://{username}:{passport}@{hostname}:{port}/{database}

There are two DB2 dialect versions implemented in SQLAlchemy. If you are connecting to a DB2 version without LIMIT [n] syntax, the recommended connection string to be able to use the SQL Lab is:

ibm_db_sa://{username}:{passport}@{hostname}:{port}/{database}

4.1.1.26 - IBM Netezza Performance Server

IBM Netezza Performance Server

The nzalchemy library provides a Python / SQLAlchemy interface to IBM Netezza Performance Server (aka Netezza).

Here’s the recommended connection string:

netezza+nzpy://{username}:{password}@{hostname}:{port}/{database}

4.1.1.27 - Microsoft SQL Server

SQL Server

The recommended connector library for SQL Server is pymssql.

The connection string for SQL Server looks like this:

mssql+pymssql://<Username>:<Password>@<Host>:<Port-default:1433>/<Database Name>/?Encrypt=yes

4.1.1.28 - MySQL

MySQL

The recommended connector library for MySQL is mysqlclient.

Here’s the connection string:

mysql://{username}:{password}@{host}/{database}

Host:

• For Localhost or Docker running Linux: localhost or 127.0.0.1
• For On Prem: IP address or Host name
• For Docker running in OSX: docker.for.mac.host.internal Port: 3306 by default

One problem with mysqlclient is that it will fail to connect to newer MySQL databases using caching_sha2_password for authentication, since the plugin is not included in the client. In this case, you should use [mysql-connector-python](https://pypi.org/project/mysql-connector-python/) instead:

mysql+mysqlconnector://{username}:{password}@{host}/{database}

4.1.1.29 - Oracle

Oracle

The recommended connector library is cx_Oracle.

The connection string is formatted as follows:

oracle://<username>:<password>@<hostname>:<port>

4.1.1.30 - Postgres

Postgres

Note that, if you’re using docker-compose, the Postgres connector library psycopg2 comes out of the box with Feris.

Postgres sample connection parameters:

• User Name: UserName
• Password: DBPassword
• Database Host:
  • For Localhost: localhost or 127.0.0.1
  • For On Prem: IP address or Host name
  • For AWS Endpoint
• Database Name: Database Name
• Port: default 5432

The connection string looks like:

postgresql://{username}:{password}@{host}:{port}/{database}

You can require SSL by adding ?sslmode=require at the end:

postgresql://{username}:{password}@{host}:{port}/{database}?sslmode=require

You can read about the other SSL modes that Postgres supports in Table 31-1 from this documentation.

More information about PostgreSQL connection options can be found in the SQLAlchemy docs and the PostgreSQL docs.

4.1.1.31 - Presto

Presto

The pyhive library is the recommended way to connect to Presto through SQLAlchemy.

The expected connection string is formatted as follows:

presto://{hostname}:{port}/{database}

You can pass in a username and password as well:

presto://{username}:{password}@{hostname}:{port}/{database}

Here is an example connection string with values:

presto://datascientist:securepassword@presto.example.com:8080/hive

By default StreamZero assumes the most recent version of Presto is being used when querying the datasource. If you’re using an older version of Presto, you can configure it in the extra parameter:

{
    "version": "0.123"
}

4.1.1.32 - Rockset

Rockset

The connection string for Rockset is:

rockset://apikey:{your-apikey}@api.rs2.usw2.rockset.com/

For more complete instructions, we recommend the Rockset documentation.

4.1.1.33 - Snowflake

Snowflake

The recommended connector library for Snowflake is snowflake-sqlalchemy<=1.2.4.

The connection string for Snowflake looks like this:

snowflake://{user}:{password}@{account}.{region}/{database}?role={role}&warehouse={warehouse}

The schema is not necessary in the connection string, as it is defined per table/query. The role and warehouse can be omitted if defaults are defined for the user, i.e.

snowflake://{user}:{password}@{account}.{region}/{database}

Make sure the user has privileges to access and use all required databases/schemas/tables/views/warehouses, as the Snowflake SQLAlchemy engine does not test for user/role rights during engine creation by default. However, when pressing the “Test Connection” button in the Create or Edit Database dialog, user/role credentials are validated by passing “validate_default_parameters”: True to the connect() method during engine creation. If the user/role is not authorized to access the database, an error is recorded in the StreamZero logs.

4.1.1.34 - Teradata

Teradata

The recommended connector library is teradatasqlalchemy.

The connection string for Teradata looks like this:

teradata://{user}:{password}@{host}

ODBC Driver

There’s also an older connector named sqlalchemy-teradata that requires the installation of ODBC drivers. The Teradata ODBC Drivers are available here: https://downloads.teradata.com/download/connectivity/odbc-driver/linux

Here are the required environment variables:

export ODBCINI=/.../teradata/client/ODBC_64/odbc.ini
export ODBCINST=/.../teradata/client/ODBC_64/odbcinst.ini

We recommend using the first library because of the lack of requirement around ODBC drivers and because it’s more regularly updated.

4.1.1.35 - Trino

Trino

Supported trino version 352 and higher

The sqlalchemy-trino library is the recommended way to connect to Trino through SQLAlchemy.

The expected connection string is formatted as follows:

trino://{username}:{password}@{hostname}:{port}/{catalog}

If you are running trino with docker on local machine please use the following connection URL

trino://trino@host.docker.internal:8080

Reference: Trino-Feris-Podcast

4.1.1.36 - Vertica

Vertica

The recommended connector library is sqlalchemy-vertica-python. The Vertica connection parameters are:

• User Name: UserName
• Password: DBPassword
• Database Host:
  • For Localhost : localhost or 127.0.0.1
  • For On Prem : IP address or Host name
  • For Cloud: IP Address or Host Name
• Database Name: Database Name
• Port: default 5433

The connection string is formatted as follows:

vertica+vertica_python://{username}:{password}@{host}/{database}

Other parameters:

• Load Balancer - Backup Host

4.1.1.37 - YugabyteDB

YugabyteDB

YugabyteDB is a distributed SQL database built on top of PostgreSQL.

Note that, if you’re using docker-compose, the Postgres connector library psycopg2 comes out of the box with StreamZero.

The connection string looks like:

postgresql://{username}:{password}@{host}:{port}/{database}

4.2 - Notifications and Messaging

StreamZero provides you access to over 40 notification services such as Slack, Email and Telegram.

StreamZero FX uses the Apprise Python Libs as an engine for notifiation dispatch. The power of Apprise gives you access to over 40 notification services. A complete list is provided in a table and the end of the document.

In order to send notifications from your package you need is to create and emit a pre-defined event type.

How to send notifications from your package

In order to send notifications from your package you need to send a ‘ferris.notifications.apprise.notification’ event from your package

You can do it like so.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

from fx_ef import context

# Please note that the value for the url_template is the name used within the config 
# For a specifc URL template in configurations.
# Please see configurations sample on how to configure

data = {
"url_template": "slack_1",
"body": "This is the content",
"title": "This is the subject"
}

event_type = "ferris.notifications.apprise.notification"
context.events.send(event_type, data)

How does it Work?

There are 2 approaches to implementing the notifications support.

• Implementation within a StreamZero Service
• Implementation in an Exit Gateway

The 2nd option is used in platforms which are behind a firewall and therefore require the gateway to be outside the firewall for accessing external services. In these cases the adapter runs as a separate container.

Irrespective of the infrastructure implementation the service internal API (as illustrated above) does not change.

The following documentation refers to Option 1.

Pre-Requisites

In order to send notifcations

• The Apprise Libs must be present in the Executor Image.

• You must have the Apprise Notifications Packages installed and running. You can find the code base further below in document.

• You must upload a secrets.json file for the Apprise Notifications Package. Please note that you should maintain a separate copy of the configs since this will not be displayed in your configuration manager since it contains credentials.

• A sample configuration file is provided below. Please use the table based on Apprise documentation to understand the URL Template structure.

• Once the Apprise Notifications Package is installed along with the configurations you must link the package to be triggered by the ‘ferris.notifications.apprise.notification’ event.

The StreamZero Apprise Package

The following is code for an StreamZero executor package to send apprise based notifications.

To send a notification from within your python application, just do the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import apprise
from fx_ef import context

# Getting the incoming parameters
url_template_name = context.params.get('url_template')

# Create an Apprise instance
apobj = apprise.Apprise()

# Setting up the Apprise object with URLs
# The URL is retreived from the uploaded secrets.json
apobj.add(context.secrets.get(url_template_name))

try: 
  apobj.notify(
    body=get_param('body'),
    title=get_param('title'),
  )
except Exception as ex:
  print(ex)

Configuration

The following is a sample configuration which is uploaded as a secrets.json file for the StreamZero Apprise Package.

The configuration consists of a set of named URL templates. With each url_template being based on the Apprise URL schema as shown in the sections further in document.

While you are free to name URL templates as you wish it is preferred to prefix them with an indication of the underlying service being used to send notifications.

1
2
3
4
5
6
7
8

{
  "slack_1": "slack://TokenA/TokenB/TokenC/",
  "slack_2": "slack://TokenA/TokenB/TokenC/Channel",
  "slack_3":"slack://botname@TokenA/TokenB/TokenC/Channel",
  "slack_4": "slack://user@TokenA/TokenB/TokenC/Channel1/Channel2/ChannelN",
  "telegram_1": "tgram://bottoken/ChatID",
  "telegram_2": "tgram://bottoken/ChatID1/ChatID2/ChatIDN"
}

The configurations must be added to a secrets.json file and uploaded as part of the apprise_package.

The apprise package must be configured to be triggered by the ‘ferris.notifications.apprise.notification’ event.

Popular Notification Services

The table below identifies the services this tool supports and some example service urls you need to use in order to take advantage of it.

Click on any of the services listed below to get more details on how you can configure Apprise to access them.

SMS Notification Support

Desktop Notification Support

Email Support

Apprise have some email services built right into it (such as yahoo, fastmail, hotmail, gmail, etc) that greatly simplify the mailto:// service. See more details here.

Custom Notifications

5 - Security

Concept

StreamZero is taking a multi-layered and integrative approach to security and access rights management, protecting systems, networks, users and data alike.

While the security architecture of StreamZero stands alone and operates well in isolation, it is built to integrate with enterprise security systems such as LDAP and Active Directory.

It supports Single Sign On (SSO) through open protocols such as Auth0 and SAML.

This user guide focuses on the application internal - user controlled - aspects of the seurity functions.

Approach

StreamZero applies the proven notion of #Users, #Roles and #Permissions and linking them to the application elements such as #Menues, Views and Pages.

This approach enables the breaking of the application into granular elements and organizing them into groups of like access control areas. The ultimate benefit is the implementation of user rights on a strict “need-to-know” basis.

Security Components

In the following sections you will learn how the security components work and how to set them up for your purpose.

If you want to follow the instructions and examples, you first need to connect to your StreamZero demo instance.

Navigation

The Security menu is found on left hand navigation of Ferris.

• Click on the Security menu to expand it and display all security relevant menu items.

StreamZero Security Navigation

• List Users: Setup individual users and assign one or more roles to them. If StreamZero is integrated with a company own Single Sign On, here is where all users can be viewed. Each user may be deactivated manually.
• List Roles: Setup and maintain individual roles and assign them viewing, editing, executing and other rights pertinent to the character and scope of the role. Roles can be integrated and inheritet with the company Active Directory.
• Users Statistics: Useful grafical statistic displaying the login behavior of individual users, such as login count and failed logins. For Security Admins only.
• User Registrations: Listing pending registration requests. For Security Admins only.
• Base Permissions: Listing the base permissions. For Security Admins only.
• Views/Menus: Listing of all Menu and View (aka Pages, UI) items. For Security Admins only.
• Permissions on Views/Menus: Listing of the assigned permissions of each Menu and View element. For Security Admins only.

NOTE that it is considered a good practice that security related tasks are provided to only a few dedicated Security Leads within the organization. For that purpose, setting up a dedicated Security Lead role is advised.

Authentication and Authorization

Add text

Data Level Security

Add text

Integrations

Add text

Audit Functions

Add viewing function (below)

Add logging and monitoring capabilities

5.1 - Permissions

Roles

This is us - humans - using StreamZero on a day to day basis. And in this section each user is listed with the most important attributes defininig name, e-mail, status and - most importantly - the associated roles.

To get the the Users page, navigate to: Security > List Roles

Example: List of Users

Besides displaying the list of all current users, the Users page offers a number of capabilities:

• Add User
• Show User
• Edit User
• Delete User

Note that each of these capabilities depends on the Permissions given to your Role. Some roles may be given full rights (e.g. add, show, edit, delete), where others may only be given viewing rights (e.g. show). As a result, some users may only be seeing the “Show User” magnifying glass icon.

5.2 - Roles

Roles

Roles represent a collection of permissions or rights that can be performed within Ferris. They can be directly associated with the job of job a #User performs.

To get the the Users page, navigate to: Security > List Roles

Example: List of StreamZero Roles

Besides displaying the list of all available Roles, the Roles page offers a number of capabilities:

• Add Role
• Actions
• Show Roles
• Edit Roles
• Delete Roles

Note that each of these capabilities depends on the Permissions given to your Role. Some roles may be given full rights (e.g. add, show, edit, delete), where others may only be given viewing rights (e.g. show). As a result, some users may only be seeing the “Show Roles” magnifying glass icon.

Add new Role

Adding a new Role can be done directly within StreamZero by any person who has the necessary permissions to do so. This is normally a designated Security Lead.

Often times, adding new Roles is done by integrating StreamZero with the corporate Identity Access Management (IAM) and Active Directory (AD).

Click on the plus sign to add a new user.

Example: Adding a new Role and Permissions

• Name: Use a meaningful name for the role, and consider applying naming standards for all Roles

• Permissions: Chose from the previously setup list of #Permissions

Note that it possible, and common practice, to apply multiple roles to a user. Therefore it is a good idea to keep the number of permissions within one role to a minimum. This enables the enforcement of the “need-to-know” basis for each role.

Actions

In order to apply an Action, first select the Role you want the action to apply to.

• Select a Role then click Actions > Copy Role > click Yes to confirm
• The Role is now copied and can be found in the list of Roles as "[original role name] copy"

5.3 - Users

Users

This is us - humans - using StreamZero on a day to day basis. And in this section each user is listed with the most important attributes defininig name, e-mail, status and - most importantly - the associated roles.

To get the the Users page, navigate to: Security > List Users

Example: List of StreamZero Users

Besides displaying the list of all current users, the Users page offers a number of capabilities:

• Add User
• Show User
• Edit User
• Delete User

Note that each of these capabilities depends on the Permissions given to your Role. Some roles may be given full rights (e.g. add, show, edit, delete), where others may only be given viewing rights (e.g. show). As a result, some users may only be seeing the “Show User” magnifying glass icon.

Add new User

Adding a new User can be done directly within StreamZero by any person who has the necessary permissions to do so. This is normally a designated Security Lead.

Most often though, adding new Users is done by integrating StreamZero with the corporate Identity Access Management (IAM) and Single Sign On (SSO) system.

Click on the plus sign to add a new user.

Example: Adding a new StreamZero User

• First Name, Last Name: Full first and last name of the user

• User Name: Apply any name or e-mail, but it is advised to adhere to a standard applied to all users.

  Possible options may include: Email, Company User ID, first name only or full name

• Is Active: Keep this boxed checked as long as the user is actively using Ferris.

  Uncheck the box once a user is no longer using Ferris. Delete the user after a prolonged period of inactivity.

• Email: The users email is the primary means of identification and communication.

• Role: This is where one or more roles - and therefore permissions - are associated with each user.

  Roles are setup in advance, and only already existing roles may be selected. A user may be associated to multiple roles.

• Save: Always save!

Show User

Click on the spyglass icon in front of the user you want to look at.

Example: View only List of StreamZero Users

User Info

• User Name: displays user name
• Is Active: shows whether the user is set to active or inactive. Note that an inactivated user may not be able to loginto Ferris
• Role: Associated roles
• Login Count: Count of how many times the user has logged in in total

Personal Info

• First Name: First name as given during setup. This name may also be provided from the corporate directory
• Last Name: Last name as given during setup. This name may also be provided from the corporate directory
• Email: Email as given during setup. This email may also be provided from the corporate directory

Audit Info

• Last login: Last successful login date and time
• Failed login count: Number of unsuccessful login attempts
• Created on: Exact date and time this user was created (manually or via corporate SSO)
• Created by: StreamZero User or corporate integration service
• Changed on: Exact date and time this user was edited (manually or via corporate SSO)
• Changed by: StreamZero User or corporate integration service

Edit User: Depending on your user rights (permission level) you may be able to enter the Edit function.

Back arrow: go back to the List Users page.

Edit User

Editing User details such as name, email or even Role(s) may be done here.

Note though that if the StreamZero Security is integrated with the corporate Identity Access Management (IAM), adding or removing Roles should be done there in order to ensure the change is permanent and not overwritten at the next syncronization.

Click on the edit icon in front of the user you want to apply changes to.

Delete User

Delete Users permanetly from Ferris.

Note though that if the StreamZero Security is integrated with the corporate Identity Access Management (IAM), deleting users should be done there in order to ensure the change is permanent and not overwritten at the next syncronization.

Also note that before permanetly deleting a user from Ferris, it is advised that the status of the user is first set to “inactive” for a period of time, and until it is sure that the user will not be reactivatet.

Click on the delete icon in front of the user you want to delete.

5.4 - Statistics

Users Statistics

Here a Security Lead finds useful information on any user’s successful as well as failed login attempts.

Navigate to: Security > User’s Statistics

Example: Users Statistics > Login count

6 - Solutions

6.1 -

Balance Sheet Quality Control

Executive summary

It is a routine task for the banks to assess the balance sheet quality for their corporate clients on an annual basis and to categorize them according to the risk ratings. The use case contributes to the optimization of this routine task by automating the balance sheet comparison process and setting up smart notificaton mechanism.

Problem statement

Target market / Industries

Solution

The solution

Balance Sheet Quality Control Model - different use case - for financial services -banks would have to identify the balance sheet quality for the corp clients - look on the balance sheet annually - analize and to give different risk ratings - use case automation of the risk assessment - skip the balance sheets that are very similar to previous years. If significant change - analyze manually - SEPARATE USE CASE - Notification Hierarchy! - depending on the event found in the balance sheet - different people need to be notified.

The solution included:

Slide

Stakeholders

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

Assets & Artefacts:

The deliverables included:

Impact and benefits

The use-case implementation resulted in:

Testimonials

“…” — Mr. XXX YYY, Title, Company ZZZ.

Tags / Keywords

6.2 -

Corporate Clients Quality Assessment

Executive summary

Implemented a fully automated data ingestion and orchestration system for the structural assessment of Corporate Client‘s balance sheets. Designed and implemented a two-level approach that includes both existing and publicly available information in a structured or semi-structured format. Extraction of relevant changes which are then compared to prior periods, explicit new data deliveries or internal policy baselines and thresholds.

Problem statement

Often banks are facing high staff turnover rate and general lack of advisory staff that is resulting in lack of client intelligence. Detailed client profiles either does not exist or outdated and does not reflect the latest situation. As a result it is hardly possible to increase the share of wallet of existing corporate clients.

Target market / Industries

Target industry for this use-case is Financial Services – Banks, Insurances, Asset Management Firms

Solution

The solution is aimed to understand which corporate clients are able to gain a bigger share of wallet. Using the publicly available information, such as news and social media, the selling proposition for those clients is outlined. At the next stage the segmentation of those clients is performed to understand how much efforts need to be put to increase share of wallet and to shortlist most prominent of them. This use case can be effectively combined with the Churn Analysis use-case.

Stakeholders

-Sales management

• Sales staff / Relationship Management
• Key Account Management

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Client base
• Transaction / product usage history

Assets & Artefacts:

The deliverables included:

Impact and benefits

The use-case implementation resulted in:

Testimonials

“…” — Mr. XXX YYY, Title, Company ZZZ.

Tags / Keywords

6.3 - Automating Marketing Data Analysis

Marketing Data Analysis

Executive summary

Implemented the first service- and on-demand based big data and data science infrastructure for the bank. Data pipelines are built and maintained leveraging two key infrastructure components: a custom-built aggregation tool and the marketing content & event platform. The aggregation tool builds the data lake for all analytics activities and enables the marketing platform to organically grow customer and campaign projects.

Problem statement

Relationship Managers spend too much valuable time researching talking points and themes that fit their different client profiles. A simple product recommender usually cannot grasp the complexity of private banking relationships and hence the product recommendations are usually without impact.

Target market / Industries

Private banking, wealth management, All relationship intense industries, i.e. insurance

Solution

Jointly with the client we developed a private banking marketing ontology (knowledge graph or rule book) that enabled various Machine Learning (ML) models to parse broad catalogue of unstructured data (financial research, company analysis, newsfeeds) to generate personalized investment themes and talking points.

The solution included:

• Private banking marketing ontology
• Thematic aggregator agents
• Personalized clustering

Slide

Stakeholders

• Head of marketing and campaigns
• Market heads
• Relationship Manager
• Chief Investment Officer

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Access to CRM details
• Client transaction history
• Research details

Assets & Artefacts:

• Financial Product Classification
• Product Risk Classification
• Event Lifecycle

The deliverables included:

• Private banking marketing ontology
• Thematic aggregator agents
• Personalized clustering
• End to end event cascade and workflow integration

Impact and benefits

Achieve a fully transparent Close the Loop on Campaigns and increased RoMI by 18%. Furthermore, this first mover program established the big data sandbox as a service capability to the entire bank. Also this project enabled marketing for the first time to close the loop between their digital client touchpoints and the events and campaigns run.

The use-case implementation resulted in:

• +18% increase in RoMI (return on marketing investments)
• -17% savings on campaign spend

Testimonials

“Using StreamZero we were able to digest a massive amount of text and extract personalized investment themes which allows our RMs to increase their face time with the clients and surprise them with the meaningful content.” — Mr. R. Giger, Head of Marketing and Campaigns, Swiss Private Bank

Tags / Keywords

#marketdataanalysis #bigdata #bigdatainfrastructure #datascience #datascienceinfrastructure #financialservices #bank

6.4 - Churn Analysis and Alerting – Financial Services

Churn Analysis and Alerting

Executive summary

Screening your existing client population for signs of dissatisfaction and pending attrition can involve a broad range of analysis. Usually the focus is given to transaction pattern analysis. And while this may prove helpful it can be misleading in smaller banks with limited comparative data. We thus integrate a broader variety of less obvious indicators and include an advisor based reinforcement loop to train the models for a bank‘s specific churn footprint.

Problem statement

When clients close their accounts or cancel their mandates, it usually does not come as a surprise to the Relationship Manager (RM). But for obvious reasons, the RM tries to work against the loss of a client with similar if not the same tools, processes and attitudes that have led to a client being dissatisfied. This is not to say that the RM manager is the sole reason for churn. But often clients do not or not sufficiently voice their issues and simply quit the relationship. To search, become aware and then listen for softer and indirect signs is at the heart of this use case.

Target market / Industries

The described use case can be efficiently applied to any industry that is providing services to a large number of clients and has a large number of transactions. Particularly following industries are benefiting most of this use-case:

• Financial services
• Insurance

Solution

Using historical data, client communication and structured interviews with client advisors, we create a bank-specific churn ontology, that is then used to screen existing clients on an ongoing basis. Creating an interactive reinforcement loop with new churn-cases, this classification, predictor and indicator approach is ever more fine tuned to specific segments, desks and service categories. As direct and ongoing Key Performance Indicators (KPIs) churn ratios and client satisfaction are measured alongside Assets under Management (AuM), profitability and trade volumes for the respective clients are classified as “endangered”. Usually a gradual improvement can be monitored within 3-6 months from the start of the use case.

The solution included:

• Initial typical churn cause analysis based on historical data (client positions, and transactions)
• Ideally inclusion of CRM notes and written advisor to client communication (prior to churn)
• Sales & Churn Ontology setup & subsequent ontology matching
• Identification of likely bank churn footprint & learning / improvement loops
• Aggregation of findings, reporting, alerting & action notifications

Stakeholders

• Chief Operations
• Client advisory, Relationship and Sales management

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Historic data about churned clients
• Client portfolios - positions / transactions
• Ideally pre-leave client-advisor communications

Assets & Artefacts:

• Client Behavioral Model
• Churn Prediction
• Action Monitoring

The deliverables included:

• Sales & Churn Indicator Ontology
• Use case specific orchestration flow

Impact and benefits

Lowered churn rates for distinct client segments by 16% after 6 months. Increased AuM / trades for clients „turned-around“ by about 25% within 6 months after “re-win”.

The use-case implementation resulted in:

+8% clients saved prior to loss of relationship

+24% reduction of customer asset outflows

Testimonials

“Changing the attitude we deal with churn from feeling like a failure to working a structured process, made all the difference. Turning around a dissatisfied client is now something transparent and achievable.” — Mr. Roland Giger, Head of Client Book Development, UBS

Tags / Keywords

#churn #churnanalysis #financialservices #insurance #bank #retention #clientretention #customerretention

6.5 - Classification of products along different regulatory frameworks

Classification of products along different regulatory frameworks

Executive summary

Multi-dimensional and ontology-based classification of products along different regulatory frameworks and systematic mapping of the company’s internal specialist expertise in a knowledge graph that can be used by both humans and algorithms or other systems.

Problem statement

Depending on the applicable regulation, the products and materials need to be classified for further usage. This requires an effective classification approach and also involves the challenge of knowledge management. Respective product / material information need to be extracted from various sources, cuch as core systems, product information systems, etc. This task usually requires lots of time-consuming manual work utilizing the domain knowledge.

Target market / Industries

The use case can be efficiently applied in the following industries:

• Financial services
• Manufacturing
• Retail

Solution

The solution is based on analyzing the domain knowledge and converting it into ontology. Domain knowledge is used as an input for the Machine Learning (ML) model: ontology-based annotator component is analyzing the available data, be it text, unstructured or semi-structured data and feed it into the ML model to perform the classification. The event-based workflow increases the efficiency and stability of the classification process.

Stakeholders

• Management
• Product management
• Domain experts
• Management

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Domain expertise in the ontology
• Detailed data that needs to be classified

Assets & Artefacts:

• Ontology-based annotator (DFA)
• Workflow engine

The deliverables included:

• Automated classification data or documents – final workflow

Impact and benefits

The classification process, that cannot be done manually with reasonable manpower and time resources is now automated.

Tags / Keywords

#classification #classificationofproduct #classificationofmaterial #classificationautomation #automation #production #financialservices #production #retail

6.6 - Curn Analysis and Alerting – General

Churn Analysis and Alerting

Executive summary

Screening your existing client population for signs of dissatisfaction and pending attrition can involve a broad range of analysis. Usually the focus is given to transaction pattern analysis. And while this may prove helpful it can be misleading in smaller companies with limited comparative data. We thus integrate a broader variety of less obvious indicators and include an advisor based reinforcement loop to train the models for a company’s specific churn footprint.

Problem statement

When clients close their accounts or cancel their subscriptions, it usually does not come as a surprise to the sales management. But for obvious reasons, the sales manager tries to work against the loss of a client with similar if not the same tools, processes and attitudes that have led to a client being dissatisfied. This is not to say that the sales manager can the sole reason for churn. But often clients do not or not sufficiently voice their issues and simply quit the relationship. To search, become aware and then listen for softer and indirect signs is at the heart of this use case.

Target market / Industries

The described use case can be efficiently applied to any industry that is providing services to a large number of clients and has a large number of transactions. Particularly following industries are benefiting most of this use-case:

• Retail
• Entertainment
• Mass media

Solution

Using historical data, client communication and structured interviews with sales people, we create a company-specific churn ontology, that is then used to screen existing clients on an ongoing basis. Creating an interactive reinforcement loop with new churn-cases, this classification, predictor and indicator approach is ever more fine tuned to specific segments and service categories. As direct and ongoing KPIs churn ratios and client satisfaction are measured alongside generated revenues, profitability for the respective clients are classified as “endangered”. Usually a gradual improvement can be monitored within 3-6 months from the start of the use case.

The solution included:

• Initial typical churn cause analysis based on historical data (client positions, and transactions)
• Ideally inclusion of CRM notes and written sales to client communication (prior to churn)
• Sales & Churn Ontology setup & subsequent ontology matching
• Identification of likely company churn footprint & learning / improvement loops
• Aggregation of findings, reporting, alerting & action notifications

Stakeholders

• Chief Executive
• Chief Operations
• Sales and Marketing

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Historic data about churned clients
• Client portfolios - positions / transactions
• Ideally pre-leave client-advisor coms

Assets & Artefacts:

• Client Behavioral Model
• Churn Prediction
• Action Monitoring

The deliverables included:

• Sales & Churn Indicator Ontology
• Use case specific orchestration flow

Tags / Keywords

#churn #churnanalysis #retail #entertainment #massmedia #retention #clientretention #customerretention

6.7 - First Notice of Loss Automation

Cross Jurisdictional First Notice of Loss Automation

Executive summary

The handling of cross-jurisdictional accident resolutions involving more than one country was automated for a pan-European insurance group.

Problem statement

The client was bound to a proprietary existing legacy core system which served all operational processes but did not lend itself to agile, digital use cases. Within the cross-jurisdictional context various third-party party core systems of partner network insurers also had to be integrated in the overall flow. In addition to already digital content, file-based and even handwritten forms of the European Accident Standard had to be taken into account. The growth of the customer did not allow for a continued manual processing.

Target market / Industries

Focused on Industry segments, but easily configured to work for similar case management related processes that involve expert knowledge combined with extensive manual fact checking.

Solution

The customer wanted to automate and streamline the handling and ideally straight-through-processing of new cases whenever the context allowed for such an option and involve the correct stakeholders when a human resolution was called for. An existing data warehouse provided historic resolution data that could be used to train various Machine Learning (ML) models. In addition, a knowledge graph contained the expertise on how the company wanted to deal with certain constellations in the future.

The solution included:

• Ingestion of all relevant base data into a use case message bus
• Automated plausibility check of the base claim (e.g. policy paid, client = driver, couterparty validity)
• ML model to assess “Fast Track” options (depending on likely cost footprint)
• Helper ML models to assess cost for vehicle, medical and legal cost
• Curation model to extend “fast track” rules within knowledge graph

Example Use Case Agent Cascade

Stakeholders

• Head of Operations / Claims Handling
• Domain Expert for Motor Vehicle Accidents Underwriting
• Domain Expert from Accounting & Controlling
• Tech Expert for mobile field agent application
• Tech Expert for core system

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Core data items on policies, clients, risk vs. claim details
• Core data from insurance partner network
• Historic claims & claim resolution data warehouse

Assets & Artefacts:

• Claims knowledge graph & ontology
• Vehicle, medical and legal cost assessment prediction models
• Fast track viability assessment model
• Ontology curation / extention model

The deliverables included:

• Automated decisioning on human vs. straight-through-processed case handling

Impact and benefits

The use-case implementation resulted in:

• the client was able to manage a +35% annual growth with fewer headcount (-3 FTE)
• turnaround times of automated cases could be reduced by >90%, from 8-10 working days to 1 day
• turnaround times of manual cases could be reduced by 30% due to elimination of manual triage
• the initial use case paved the way for additional AI based automation ideas

Testimonials

“We were sceptical about the limits of automation with rather difficult data quality we initially set out with. The learning loop for both the agents involved as well as the predicition models was a true surprise to me.” — Mr. Okatwiusz Ozimski, Inter Europe AG

Tags / Keywords

#insurance #firstnoticeofloss #FNOL

6.8 - Idea to Trade / Next Best Product - Financial Services

Idea to Trade / Next Best Product

Executive summary

To support advisors and clients with a “next best product” recommendation, a closed loop flow has been established from Research / Chief Investment Officer to Relationship Managers and eventually to the client. Evaluating which recommendations worked for RMs and Clients allowed for a learning loop informing Research & CIO to improve selection & tailoring of investment themes.

Problem statement

The information flow from research or strategic asset allocation (CIO) to client advisors and eventually to clients does rarely follow a structured path. Instead the bank‘s “house view” is communicated broadly to all front-office staff and portfolio managers. They then use their direct client relationship to assess risk appetite and extract specific investment themes or ideas from their client interaction. If these match, the resulting research / advice is forwarded to the client. It seems like a lucky punch if product information leads to a trade / product sale.

Target market / Industries

The challenge of customizing the offering to the customer profile is a common challenge across the industries. Financial services industry is benefiting most from this use case that can be efficiently applied e.g. in Industries, that are benefitting most of this use-case are:

• Banks
• Investment and finance firms
• Real estate brokers
• Tax and accounting firms
• Insurance companies

Solution

Starting with investment themes / product and occasion specific sales / investment opportunities, the existing client‘s portfolios and client to bank communication is screened for possible gap / fit. Research or asset allocation can then focus their efforts on topics suggested by front-office staff and / or clients themselves. Observing and identifying trade success the best practices are understood and can be multiplied across other (similar) client scenarios. Asset allocation and advisors work collaboratively as they both evaluate which information / proposals / investment ideas are forwarded to clients (and then accepted or not) and which ones are kept back by advisors (and for what reasons).

The solution included:

• Clustering & topic mapping of existing marketing material & client portfolio structures
• Optionally inclusion of CRM notes and written advisor to client communication
• Sales Ontology setup & learning loop inclusion & topic matching
• Identification of individual investment themes / topics of interest
• Aggregation of findings, reporting, alerting & action recommendation

Stakeholders

• Chief Investment Officers (CIO)
• Client Advisors / Relationship Managers
• Product Managers

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Sales organization setup (desks / books)
• Client to Client / Client to Company graphs

Capabilities utilized:

• Unstructured Data
• Semantic Harmonization
• Natural Language Processing
• Personalization

Assets & Artefacts:

• Financial Product Ontology
• Analytical CRM Models

The deliverables included:

• Sales & Onboarding Ontology
• Use case specific orchestration flow
• Integration with many info sources

Impact and benefits

Proposal / offer conversion rates were increased by 42% after an initial learning curve & algorithm calibration phase of 6 months resulting in additional Asset under Management growth of 8% from targeted clients.

The use-case implementation resulted in:

+18% increased targeted product sales

+8% share of wallet

Tags / Keywords

#ideatotrade #nextbestproduct #salesadvice #financialservices #bank #insurance #investment

6.9 - Idea to Trade / Next Best Product - General

Idea to Trade / Next Best Product recommendation

Executive summary

To support sales people and clients with a “next best product” recommendation, a closed loop flow has been established from Marketing / Research to Sales and eventually to the client. Evaluating which recommendations worked for Sales and Clients allowed for a learning loop informing Product Management to improve selection & tailoring of the offerings.

Problem statement

The information flow from Development and Product Management to Sales and Marketing and eventually to clients does rarely follow a structured path. Instead the company‘s “house view” is communicated broadly to all staff. Sales then use their direct client relationship to assess customer needs and extract specific requirements or ideas from their client interaction. If these match, the resulting research / advice is forwarded to the client. It seems like a lucky punch if product information leads to a trade / product sale.

Target market / Industries

Any industry which has enough data about the customer to make a recommendation for the next action / product will greatly benefit from this use-case.

Solution

Starting with product review and occasion specific sales opportunities, the existing client‘s portfolios and client communication are screened for possible gap / fit. Research / Product Management can then focus their efforts on topics suggested by Sales and Marketing staff and / or clients themselves. Observing and identifying trade success the best practices are understood and can be multiplied across other (similar) client scenarios. Product Management and Sales work collaboratively as they both evaluate which information / proposals / products are presented to clients (and then accepted or not) and which ones are kept back (and for what reasons).

The solution included:

• Clustering & topic mapping of existing marketing material & client portfolio structures
• Optionally inclusion of CRM notes and written Sales to client communication
• Sales Ontology setup & learning loop inclusion & topic matching
• Identification of individual products / topics of interest
• Aggregation of findings, reporting, alerting & action recommendation

Stakeholders

• Top Management
• Sales and Marketing
• Product Managers

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Sales organization setup (desks / books)
• Client to Client / Client to Company graphs

Capabilities utilized:

• Unstructured Data
• Semantic Harmonization
• NLP
• Personalization

Assets & Artefacts:

• Product Ontology
• Analytical CRM Models

The deliverables included:

• Sales & Onboarding Ontology
• Use case specific orchestration flow
• Integration with many info sources

Tags / Keywords

#ideatotrade #nextbestproduct #salesadvice #crossindustry

6.10 - Intraday Liquidity Management Optimization

Intraday Liquidity Management Optimization

Executive summary

In order to avoid long internal lead times and to cater to stringent time-to-market expectations, an end-to-end Analytics Design and streaming real time analytics environment for group wide BCBS (Basel III) Intraday Liquidity Management was implemented. The bank’s predictive liquidity and cash management models were rebuilt from scratch using real-time streams from 13 different SWIFT messaging gateways.

Problem statement

All financial institutions need to be on top of their liquidity levels throughout the entire day. Since every organization usually experiencing many cash inflows and outflows during the day, it is diffcult to understand what are the currenct liquidity levels. To be compliant with the regulations, the liquidity levels need to be monitored. Having too much cash is not commerically viable and too little cash is too risky. Knowing the current cash levels the bank can adjust accordingly. The entire cash balancing act is based on the cascade of different events. Cash flow events and also cash-related events need to be integrated from various transaction management systems.

Target market / Industries

The use case is applicable in all regualted and cash-intence industries, i.e.

• Financial service
• Treasury departments of large corporations

Solution

During the use case implementation 16 different cash-flow generating order systems were integrated using different schemas of how they handle transactions. StreamZero Data Platform was able to resolve the complexities of the event handling to absorb all different situations and rules that need to be applied depending on the different states that the system can take. Data sourcing patterns evolved quickly from single file batch to data streaming using Kafka and Flink. A global end-user enablement was achieved with a multi network environment for regional user and both logical and physical data segregation. Irreversible client data protection using SHA 256 hash algorithm allowed for globally integrated algorithms in spite of highly confidential raw input data. We were able to implement dynamic throttling and acceleration of cash flows depending on current market situations and liquidity levels.

The solution included:

• Adaptor agents to 16 cash-flow generating systems
• Throttling and acceleration logic
• Machine Learning (ML) models for liquidity projection
• Harmonized event architecture

Stakeholders

• Group treasury
• Group risk and compliance
• CFO

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Cash-flows

Assets & Artefacts:

• Harmonized transactional event model
• Throttling and acceleration rule book

The deliverables included:

• End to end solution for intraday liquidity

Impact and benefits

The use-case implementation resulted in:

• 10 MCHF annual savings on liquidity buffers
• 23% reduction of operations & treasury staff

Testimonials

“Moving from a batch to a real-time liquidity monitoring was a substantial task that had countless positive knock-on effects throughout the organization.”

— Mr. Juerg Schnyder, Liquidity expert, Global universal bank

Tags / Keywords

#liquidity #liquiditymanagement #intradayliquiditymanagement #cashmanagement #BCBS #Basel3 #financialservices

6.11 - Metadata-controlled Data Quality & Data Lineage in Production

Metadata-controlled Data Quality & Data Lineage in Production

Executive summary

Metadata-controlled data quality & data lineage along the production chain integrated with a laboratory information system for monitoring and quality documentation of various BoM & formulation variations in the biochemical production of active pharmaceutical ingredients (preliminary production of active ingredients in transplant medicine).

Problem statement

Neither technical, nor rule based approaches can adequately help in raising data quality without domain expertise. Using the domain expertise to create the rules is time consuming and often not feasible from the manpower prospective.

Target market / Industries

The use case is applicable to any industry dealing with large volumes of data if insufficient quality, e.g.:

• Serial manufacturing
• Mass production
• Retail
• Financial services
• Cross-industry applications

Solution

The approach is based on few shot manual learning, when the expert creates a few dozens of examples with real-life data. Later on from these examples the model learns strategies to identify and correct data quality errors.

Example Use Case Agent Cascade

Stakeholders

• Domain experts
• Product data quality
• Functional experts
• Risk managers

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Detailed data sets where the quality to be improved
• 20-30 examples of errors and their manual corrections
• Judgement on automated model performance

Assets & Artefacts:

• StreamZero error correction toolbox

The deliverables included:

• Customized data quality improvement workflow

Impact and benefits

The use case implementation allows to address data quality issues in an efficient manner with high quality of the process automation. If the data quality management process would remain manual, this would result in 5-6 Full Time Employees dedicated for this task. The Machine Learning model will over time accumulate respective knowledge and support domain expertise with relevant automated data quality improvement proposals.

Tags / Keywords

#dataquality #dataqualityimprovement #machinelearning #production #manufacturing #serialmanufacturing #massproduction #medicine #laboratory #pharma #financialservices #crossindustry

6.12 - Onboarding and Fraud Remodelling - Financial Services

Onboarding and Fraud Remodelling

Executive summary

Leverage an industry proven onboarding and Know Your Customer (KYC) ontology and fine-tune it to your corporate compliance policies. Ensure manual processing, existing triggers and information sources are tied together by a unified and harmonized process supporting front-office and compliance simultaneously.

Problem statement

Many financial service providers struggle with the complexity and inefficiency of their client onboarding and recurring KYC monitoring practice. And many have issues, when it comes to Anti-Money Laundering (AML), source of funds and transaction monitoring compliance. The front-office staff often tries to cut corners and the compliance staff is overwhelmed with the number of cases and follow-ups that are required from them. Disintegrated and high-maintenance systems and processes are the usual status quo, with little budget and energy to change from due to the inherent risk.

Target market / Industries

The use case is primarily applicable to the industries that are exposed to frauds and where fraud tracking and prevention is needed. The financial services industry benefiting most from this use-case.

Solution

We combine the domain knowledge of what is really required by law and regulation with the opportunity to automate many aspects of the background screening and adverse media monitoring. By integrating the robustness of process of global players and the lean and mean approach FinTech startups take, we usually are able to raise quality while reducing effort. Creating a centralized compliance officer workbench that is integrated with both front-office systems as well as risk-management and compliance tools, we are able to iteratively improve the situation by synchronizing the learning of the models and predictions with the feedback from compliance experts.

The solution included: -Integrated Compliance Officer Workbench

• Onboarding & Compliance Ontology configuration & subsequent ontology matching
• Integration of existing screening & trigger sources with learning / improvement loops
• Automation of standard cases and pre-filling of high-probability issues
• Aggregation of findings, reporting, alerting & compliance action notifications

Stakeholders

• Compliance
• Security

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Existing client onboarding / KYC policies
• KYC/AML/onboarding management cases
• Client‘s contracts, positions & transactions

Assets & Artefacts:

• Client risk attributes
• Historic client behavior information

The deliverables included:

• Compliance & KYC Ontology
• Compliance Officer Workbench
• Use case specific orchestration flow

Impact and benefits

Decreased compliance team by 30% from 18 down to 12 Fuull-Time Employees by automating standard case load. Increased compliance quality and decreased client case resolution time by eliminating aspects not required by current jurisdictional scope.

Testimonials

“This is a compliance expert‘s dream come true. Before I never had an oversight of where I or my team where standing. Now we can actually support our client facing colleagues.” — Mr. XXX YYY, Title, Company ZZZ.

Tags / Keywords

#fraud #fraudremodelling #compliance #kyc #financialservices #bank

6.13 - Onboarding and Fraud Remodelling - General

Onboarding and Fraud Remodelling

Executive summary

Leverage an industry proven onboarding and Know Your Customer (KYC) ontology and fine-tune it to your corporate compliance policies. Ensure manual processing, existing triggers and information sources are tied together by a unified and harmonized process supporting sales and compliance simultaneously.

Problem statement

Many service providers struggle with the complexity and inefficiency of their client onboarding and recurring KYC monitoring practice. And many have issues, when it comes to various types of monitoring of client’s activities. The front-office staff often tries to cut corners and the compliance staff is overwhelmed with the number of cases and follow-ups that are required from them. Disintegrated and high-maintenance systems and processes are the usual status quo, with little budget and energy to change from due to the inherent risk.

Target market / Industries

The use case is primarily applicable to the industries that are exposed to frauds and where fraud tracking and prevention is needed.

Solution

We combine the domain knowledge of what is really required by law and regulation with the opportunity to automate many aspects of the background screening and adverse media monitoring. By integrating the robustness of process of global players and the lean and mean approach, we usually are able to raise quality while reducing effort. Creating a centralized compliance officer workbench that is integrated with both front-office systems as well as risk-management and compliance tools, we are able to iteratively improve the situation by synchronizing the learning of the models and predictions with the feedback from compliance experts.

The solution included: -Integrated Compliance Officer Workbench

• Onboarding & Compliance Ontology configuration & subsequent ontology matching
• Integration of existing screening & trigger sources with learning / improvement loops
• Automation of standard cases and pre-filling of high-probability issues
• Aggregation of findings, reporting, alerting & compliance action notifications

Stakeholders

• Compliance
• Security

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Existing client onboarding / KYC policies
• KYC / onboarding management cases
• Client‘s contracts, positions & transactions

Assets & Artefacts:

• Client risk attributes
• Historic client behavior information

The deliverables included:

• Compliance & KYC Ontology
• Compliance Officer Workbench
• Use case specific orchestration flow

Tags / Keywords

#fraud #fraudremodelling #frauddetection #fraudtracking #compliance #kyc

6.14 - Prospecting 360 Degree

Prospect 360°

Executive summary

For many strategic prospects the preparation of possible offers and establishment of a real relationship either involves great effort or lacks structure and focus. The Prospect 360° use case augments traditional advisor intelligence with automation to improve this original dilemma.

Problem statement

Hunting for new important clients usually is driven by referrals and the search for an “ideal event“ to introduce a product or service. Existing client relationships are usually screened manually and approached directly to request an introduction, prior to offering any services. Monitoring the market and a prospect’s connections can be cumbersome and is error prone – either introductions are awkward or they do not focus on a specific and urgent need. Hence the success and conversion rates seem hard to plan.

Target market / Industries

This use case is traditionally applicable to such industries where the customer engagement and acquisition process is long and costs per customer are high:

• Financial Services
• Insurance
• General Business Services

Solution

We introduce the idea of “soft onboarding”. Instead of selling hard to a new prospect, we start to engage them with tailored and relevant pieces of information or advice free of charge. We do, however, tempt this prospect to embrace little initial pieces of an onboarding-like process, extending the period we are allowed to profile the needs and preferences of the client and the related social graph. Turning a prospect into an interested party and then increasing the levels of engagement of the period of up to six months allows for a more natural and client-driven advisory experience, that is shifting from a “product push” towards a “client pull”.

The solution included:

• Integration of disparate news & event information sources (licensed & public origins)
• Provisioning of select RM & client data points to understand social graphs
• Word parsing of text-based inputs (e.g. news articles and liquidity event streams)
• Onboarding & Sales Ontology matching
• Identification of possible liquidity events, new referral paths & sales topics of interest
• Aggregation of findings, reporting, notifications and organizational routing
• Ideally inclusion of reinforcement learning (via RM, client & assistant feedback loops)

Example Use Case Agent Cascade

Stakeholders

• Relationship Management
• Sales
• Marketing

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Sales organization setup (desks / books)

• Client to Client / Client to Company graphs

Capabilities utilized:

• Unstructured Data
• Semantic Harmonization
• NLP
• Personalization

Assets & Artefacts:

• Financial Product Ontology
• Analytical CRM Models

The deliverables included:

• Sales & Onboarding Ontology
• Use case specific orchestration flow
• Integration with many info sources

Impact and benefits

Strategic client team originally covered 200 prospects manually. Introducing Prospect 360° allowed to double that number while reducing the time-to-close by 35% — from more than 12 to an average of about 7 months.

The use-case implementation resulted in:

+8% growth of corporate loan book

+22% reduction on credit defaults

Testimonials

“I feel a lot more as a real advisor. I can be helpful and feel informed. And I still can make my own judgments of what is relevant for my personal relationship to existing clients and new referrals. I learn as the system learns.”

— Mr. Pius Brändli, Managing Director, Credit Suisse

Tags / Keywords

#kyc, #knowyourcustomer, #finance, #financialservices, #onboarding, #prospecting, #prospect360

6.15 - Prospecting 360 Degree - general

Prospect 360°

Executive summary

For many strategic prospects the preparation of possible offers and establishment of a real relationship either involves great effort or lacks structure and focus. The Prospect 360° use case augments traditional sales and marketing intelligence with automation to improve this original dilemma.

Problem statement

Hunting for new important clients usually is driven by advertisement, referrals and the search for an “ideal event“ to introduce a product or service. Existing client relationships are usually screened manually and approached directly to request an introduction, prior to offering any services. Monitoring the market and a prospect’s connections can be cumbersome and is error prone – either introductions are awkward or they do not focus on a specific and urgent need. Hence the success and conversion rates seem hard to plan.

Target market / Industries

This use case is suitable for the industries where the customer engagement and acquisition process is long and costs per customer are high.

Solution

We introduce the idea of “soft onboarding”. Instead of selling hard to a new prospect, we start to engage them with tailored and relevant pieces of information or advice free of charge. We do, however, tempt this prospect to embrace little initial pieces of an onboarding-like process, extending the period we are allowed to profile the needs and preferences of the client and the related social graph. Turning a prospect into an interested party and then increasing the levels of engagement of the period of up to six months allows for a more natural and client-driven sales experience, that is shifting from a “product push” towards a “client pull”.

The solution included:

• Integration of disparate news & event information sources (licensed & public origins)
• Provisioning of select sales & client data points to understand social graphs
• Word parsing of text-based inputs (e.g. news articles)
• Onboarding & Sales Ontology matching
• Identification of new referral paths & sales topics of interest
• Aggregation of findings, reporting, notifications and organizational routing
• Ideally inclusion of reinforcement learning (via sales, client & assistant feedback loops)

Example Use Case Agent Cascade

Stakeholders

• Relationship Management
• Sales
• Marketing

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Sales organization setup (desks / books)
• Client to Client / Client to Company graphs

Capabilities utilized:

• Unstructured Data
• Semantic Harmonization
• NLP
• Personalization

Assets & Artefacts:

• Product Ontology
• Analytical CRM Models

The deliverables included:

• Sales & Onboarding Ontology
• Use case specific orchestration flow
• Integration with many info sources

Tags / Keywords

#newclients #salesfunnel #sales #marketing #salesautomation #prospecting #prospect360

6.16 - Regulatory Single Source of Truth

Regulatory Single Source of Truth

Executive summary

Leveraging all existing data sources from core banking, risk- and trading systems to the Customer Relationship Management (CRM) and general ledger as granular input for your regulatory Single Source of Truth (SSoT).

Problem statement

Most regulatory solutions today require huge maintenance effort on both business and technology teams. Ever more granular and ever more near-time regulatory requirements further increase this pressure. Usually the various regulatory domains have created and continue to create silos for central bank, credit risk, liquidity, Anti-Money Laundering (ALM) / Know Your Customer (KYC) and transaction monitoring regulations. Further requirements from ePrivacy, Product Suitability and Sustainability regulations even further dilute these efforts.

Target market / Industries

• Financial services
• Insurance

Solution

Leveraging the semantic integration capabilities of StreamZero Data Platform, it allows you to reuse all the integration efforts you have previously started and yet converge on a common path towards an integrated (regulatory) enterprise view. The ability to eliminate high-maintenance Extraction Transformation Loading (ETL) coding or ETL tooling in favor of a transparent and business driven process will save you money during the initial implementation and during ongoing maintenance. Templates and a proven process were applied to use what exists and build what’s missing without long-term lock in.

The solution included:

• Semantic Integration leveraging all your prior integration investments
• Business driven data standardization and data quality improvements
• No Code implementation => business analysis is sufficient to generate the integration layer
• Implement data governance & data quality via reusable business checks
• Multiply your regulatory investments to be used for analytics, sales and risk

Stakeholders

• Compliance

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Full regulatory granular scope master & reference data (incl. UBO hierarchies)
• Client portfolio (positions / transactions)

Assets & Artefacts:

• Private Bank Data Model
• Optimization Algorithms
• Data Quality Business Rules

The deliverables included:

• E2E Models & Integration Schema
• Library of Business Checks

Impact and benefits

The semantic SSoT is now used by other functions across the bank leveraging regulatory investments for sales support, operations and risk management.

The use-case implementation resulted in:

9% reduction of risk weighted assets

9 FTE (50%) reduction of regulatory reporting team

In addition, recurring Cost of Capital savings of over 15m CHF p.a. were achieved.

Testimonials

“We have semantically integrated +220 different data sources at Switzerland largest independent Private Bank. The regulatory team was able to deliver better results faster and yet decreased the team size by 30%.” — Mr. XXX YYY, Title, Company ZZZ.

Tags / Keywords

#singlesourceoftruth #ssot #bank #privatebank #financialservices #insurance

6.17 - Sensor-based Monitoring of Sensitive Goods

Sensor-based Monitoring of Sensitive Goods

Executive summary

Sensor-based monitoring of sensitive goods along the transport chain (location, light, temperature, humidity & shocks in one sensor) and integration of these IoT components in a smart and decentralized cloud including the event-controlled connection to the relevant peripheral systems.

Problem statement

One of the biggest problems that exist in supply chain management is the lack of visibility and control once materials have left the site or warehouse. This leads to billions in losses due to missing or damaged products and leads to business inefficiency.

Target market / Industries

The use case can be applied for the following solutions:

• Intelligent packaging
• Intelligent logistics
• Industrial applications – Industry 4.0
• Consumer and luxury goods
• Home protection

Solution

A small-size energy-autonomous intelligent cells (microelectronic parts) are integrated into any object / package to enable remain in contact to it, identify it electronically, provide a location and sense temperature, pressure, movement, light, and more. They are intelligent, able to make basic decisions and save small information pieces. The cells communicate bidirectionally with the software through global IoT networks of our partners selecting the best energy-efficient technologies available where they are and building a neuronal backbone of objects in constant communication. Bidirectional capabilities allow transmitting data of the sensors and receiving instructions and updates remotely. Cells can also interact with the electronics of the objects they are attached to, they can read and transmit any parameters and providing remote control of them wherever they are. All data of the cells are transmitted to our own cloud applications, a learning brain using the power of data management and AI of Google Cloud and Microsoft Azure where we can combine IoT data of objects with any other data in intelligent and self-learning algorithms. The location or any other parameter of the objects can be displayed in any Browser or Smart device. The user interfaces can be customized and allow to interact at any moment changing frequency of connection, SMS or email alarm values and who receives them (human or machine) or acting as remote control of any object. All of this can be offered fully pay peruse, to ensure a very low entry point for the technology. You can turn the cell on and off remotely, and only pay cents per day when you use the cell capabilities or buy the solution.

Stakeholders

• Manufacturers
• Logistic services providers
• General management
• Supply chain
• Digitalization
• Quality control
• Risk management

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Container / goods data
• Transport routing
• Thresholds / boundary conditions
• Past Quality Control data for pattern recognition

Assets & Artefacts:

• Routing optimization model
• IoT cloud backbone

The deliverables included:

• Driver dispatcher and client apps
• Operational routing optimization

Tags / Keywords

#sensorbasedmonitoring #iot #sensitivegoods #intelligentpackaging #intelligentlogistics #industry40 #industry #packaging #logistics #homeprotection #consumergoods #luxurygoods

6.18 - Voice-based Trade Compliance

Voice-based Trade Compliance

Executive summary

Convert the voice-based advisor to client phone conversations into text. Analyze for possible breaches of regulatory and compliance policies. This multi-step analytical process involves voice-to-text transcription, a compliance ontology, text parsing & natural language understanding.

Problem statement

Many if not most client advisors to client communications still occur via phone. These conversations happen in a black box environment that is difficult to track and audit. Potential compliance breaches in areas such as insider trading or conflict of interest can only be identified and intercepted at great cost while only listening in on select phone calls. The vast majority of conversation remains unchecked, leaving the organization in the dark and at risk. Often compliance is at odds with sales – one controlling the business, the other pushing the boundaries of acceptable risk.

Target market / Industries

Described use case can be efficiently applied in the industries where the track / audit of the voice-based communication is required.

Solution

Leveraging your existing Public Branch eXchange (PBX) phone recording infrastructure & partnering with your choice of voice-to-text transcription service, the solution is to automatically screen every conversion. The transcribed text files are parsed against the Sales & Compliance Ontology. Using Natural Language Understanding (NLU) the use case identifies which call advice and trade decisions occurred and high-lights possible compliance breaches.

Once the predictions become more accurate a sales focused “topics-of-interest“ screening can be added.

The solution included:

• Voice to text transcription
• Word parsing of text-based inputs
• Compliance & Sales Ontology matching
• Identification of possible compliance breaches (and / or sales topics of interest)
• Aggregation of findings, reporting, alerting
• Action recommendation

Example Use Case Agent Cascade

Stakeholders

• Compliance
• Security

Data elements, Assets and Deliverables

As an Input from the client, the following items were used:

• Relationship Manager – Client Conversations (Voice or Text)
• Client portfolios - positions / transactions

Assets & Artefacts:

• Unstructured Data
• Semantic Harmonization
• Natural language processing
• Personalization

The deliverables included:

• Sales & Transaction Monitoring Ontology
• Use case specific orchestration flow

Impact and benefits

Sales Compliance headcount was reduced by 3 Full Time Employees while screening coverage increased to 90% (before only spot checks) and resolution quality focuses on sales and compliance.

Testimonials

“The overall sensitivity of advisors to client‘s sentiment and requirements has increased. Also, we improved the understanding how compliance and sales can work together to achieve client satisfaction.” — Ms. Milica L., Chief Risk Officer, Swiss Private Bank

Tags / Keywords

#voicetradecompliance #tradecompliance #compliance #bank #communicationscreening #financialservices

7 - Release Notes

7.1 - Release 1.0.2

New

• The new Services Overview is a useful page to monitor and manage all platform services.
• The Configuration Manager is a central point to manage configurations for all deployed services.
• The Executor Framework is a powerful new utility that enables creating, orchestrating, sequencing and running of pretty much any combination of jobs and services.
• With the Job Scheduler we have added a newly embedded console for scheduling, running and monitoring jobs, without the need of embedding third-party job scheduler.
• We have introduced a new Workflow Manager to support the building of processes, including four-eye reviews, approval chains and quality gates.
• We have integrated our Slack Support Channel into the Control Center menu so that you can contact us directly and easily with your questions.
• To get quicker access to the most frequently used services and components we’ve built a list of Platform Services links and placed them into the Control Center navigation.
• We have added a new File Storage capability for creating and uploading files and managing buckets.
• Introduced a new User Interface and Navigation.
• Made important improvements of Access Rights Management.

Changed

• no changes to report

Improved

• We have improved the Files Manager and in particular Files Onboarding Monitor to embed user credentials and therefore to make the files uploading and management more user specific.
• We have taken further strides to streamline and simplify the User Management. Creating users and assign their own secured container (working space) is now fully automated - albeit not yet self-service

Fixed

• no fixes to report

7.2 - Release 1.0.3

New

• We have introduced a Tagging capability, which enables users to tag almost any data element on the platform making it easy and user friendly to navigate through the application and find items.
• The Checks Framework greatly improves the data quality monitoring around any kind of data sources, data types and event types.
• The Topics & Consumer Monitor listens in on Kafka and provides an up-to-the-second view of data sourcing and streaming.
• New Service Logs can be used to track all services running within the network as well as their logs and events that occur between them. Each log or event have their own data stored within elasticsearch and can be filtered by numerous fields.
• Jupyter Auto Deploy makes the onboarding of new users even faster and self-serviced.
• Events are stored and managed in the new Event Registry.

Changed

• no changes to report

Improved

• We have improved elements of the Security and Access Rights Management in the areas of User and Access Management (Keycloak), Certificates Management as well as Authentication and Authorization (OpenID). Each role has its own set of permissions, which makes it possible to restrict users access to specific areas of the platform based on their role.
• Parametrised Workflows can be defined with by users with easy json files by following a specific set of rules. Complex use cases are supported such as chaining multiple actions one by another and having control over what happens in a different scenarios. Built in approval capability for each workflow action is supported and easy to implement by default.
• The new StreamZero Executor has become even better and more powerful. Add any Python, SQL or JSON to execute any type of jobs.
• The Scheduler is now fully embedded with the Executor.
• Files Upload has improved and is integrated with the Workflow and Approvals functionality. Files have type specific validation and are stored in different buckets. Bucket creation can be done within the same FAB module.
• The StreamZero Application Builder (FAB) is now capable of auto generating new pages (UI) on the fly.