StreamZero SX

• 1: Developer Guide
• 2: Integrations Guide
• 3: User Guide
• 4: Solutions snippets / explain problem solved / link to relevant use case
• 5: Containers + Purpose
• 6: Architecture

1 - Developer Guide

Overview

StreamZero is a container level solution for building highly scalable, cross-network sync or async applications.

Using the StreamZero SX platform to run and manage stream processing containers utilizing StreamZero messaging infrastructure significantly reduces the cost of deploying enterprise application and offers standardized data streaming between workflow steps. This will simplify the development and as result create a platform with agile data processing and ease of integration.

Getting started with Stream Processors

Take a look at this library for creating Stream Processors on top of Kafka and running them inside StreamZero platform: StreamZero-SX

Example of a Stream Processor

Below you can find an example application that is using StreamZero-sx python library functions to count the number of words in incoming messages and then sending the result to twitter_feed_wc Kafka topic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import json
from StreamZero_sx.core import app
from StreamZero_sx.utils import sx_producer


def process(message):
    message_new = dict()
    message_new['text'] = message['text']
    message_new['word_count'] = len(message['text'].split(' '))
    message_new_json = json.dumps(message_new)
    print(message_new_json)
    sx_producer.send(topic="twitter_feed_wc", value=message_new_json.encode('utf-8'))


app.process = process

Creating Docker Container

Below is an example of a dockerfile to create a Docker image for the Twitter Word Count application shown in the previous section. The user is free to use whatever base python image and then add StreamZero module and other libraries.

FROM python:3.9-alpine
#RUN pip install -i https://test.pypi.org/simple/ StreamZero-sx==0.0.8 --extra-index-url https://pypi.org/simple/ StreamZero-sx
RUN pip install StreamZero-sx
COPY twitter_word_count.py app.py

After the user have built an image and pushed it to some Docker image regitry, he can run it in StreamZero SX UI.

2 - Integrations Guide

How does it Work?

There are two main approaches to implementing the external notifications support.

• Implementation within a StreamZero SX container
• Implementation in an Exit Gateway

The 2nd option is used in platforms which are behind a firewall and therefore require the gateway to be outside the firewall for accessing external services. In these cases the adapter runs as a separate container.

Irrespective of the infrastructure implementation the service internal API (as illustrated above) does not change.

3 - User Guide

StreamZero SX Management UI

Create a Stream Adapter

After a developer has built an image of a stream processing task and stored it to a container register, we can configure and launch it with StreamZero Management UI.

On left side menu, open Stream Adapters menu and select “Stream Adapter Definition”. Fill in the details.

Go to the “List Stream Adapters” page. You should find your the Stream Adapter that we created on the list. You can start the container by clicking the “Run”-button. The download and start-up of the image can take few minutes.

When the Stream Adapter is running you can find it in the list of running adapters.

StreamZero also has a list of all the Kafka topics that are currently attached to Stream Adapters or available to Stream Adapters.

4 - Solutions snippets / explain problem solved / link to relevant use case

Twitter message processing example

The first example application is using StreamZero-sx python library to implement stream processor to count the number of words in incoming messages. The messages are queried from Twitter API with specific filter condition and then fed to the processor. The results are sent to a Kafka topic.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import json
from StreamZero_sx.core import app
from StreamZero_sx.utils import sx_producer


def process(message):
    message_new = dict()
    message_new['text'] = message['text']
    message_new['word_count'] = len(message['text'].split(' '))
    message_new_json = json.dumps(message_new)
    print(message_new_json)
    sx_producer.send(topic="twitter_feed_wc", value=message_new_json.encode('utf-8'))


app.process = process

5 - Containers + Purpose

Creating a Docker Container

Below is an example of a dockerfile to create a Docker image for some StreamZero SX application. The user is free to choose what base python image to use and then add StreamZero module and other libraries.

FROM python:3.9-alpine
#RUN pip install -i https://test.pypi.org/simple/ StreamZero-sx==0.0.8 --extra-index-url https://pypi.org/simple/ StreamZero-sx
RUN pip install StreamZero-sx
COPY app.py utils.py

After the user have built an image and pushed it to a Docker image regitsry, they can run it in StreamZero SX Management UI.

6 - Architecture

StreamZero SX Architecture Principles

Stream processing is a technique for processing large volumes of data in real-time as it is generated or received. One way to implement a stream processing architecture is to use Docker containers for individual workflow steps and Apache Kafka for the data pipeline.

Docker is a platform for creating, deploying, and running containers, which are lightweight and portable units of software that can be run on any system with a compatible container runtime. By using Docker containers for each step in the stream processing workflow, developers can easily package and deploy their code, along with any dependencies, in a consistent and reproducible way. This can help to improve the reliability and scalability of the stream processing system.

Apache Kafka is a distributed streaming platform that can be used to build real-time data pipelines and streaming applications. It provides a publish-subscribe model for sending and receiving messages, and can handle very high throughput and low latency. By using Kafka as the backbone of the data pipeline, developers can easily scale their stream processing system to handle large volumes of data and handle failover scenarios.

Overall, by using Docker containers for the individual workflow steps and Apache Kafka for the data pipeline, developers can create a stream processing architecture that is both scalable and reliable. This architecture can be used for a wide range of use cases, including real-time analytics, event-driven architectures, and data integration.

Below is the high-level architecture diagram of StreamZero SX:

Required Infrastructure

The following are the infrastructure components required for a StreamZero SX installation