Skip to content

Instantly share code, notes, and snippets.

@madhawav

madhawav/PySiddhi.GSOC.md

Last active August 24, 2017 07:02
Show Gist options
  • Save madhawav/195b7dc601d94c40958d88be1d56e705 to your computer and use it in GitHub Desktop.
Save madhawav/195b7dc601d94c40958d88be1d56e705 to your computer and use it in GitHub Desktop.
Download ZIP
Python API for Siddhi CEP
Raw
PySiddhi.GSOC.md

Python API for Siddhi CEP (PySiddhi)

Work Summary

During GSoC 2017, I worked on developing a Python Wrapper on Siddhi Complex Event Processor (Siddhi CEP) Java Library [GitHub, Release 3.1, Release 4.0, Wiki] developed by WSO2 Inc. The Python wrapper covered both versions 3.1.x and 4.x of Siddhi CEP by its two versions PySiddhi3 and PySiddhi4. Additionally, a REST API was developed for WSO2 Data Analytics Server (WSO2 DAS) 4.x as a feature of PySiddhi4.

The API PySiddhi4 on Siddhi CEP 4.x and WSO2 DAS 4.x may require future evolvements since Siddhi CEP 4.x and WSO2 DAS 4.x are still unreleased. However, the developed Python wrappers are in stable state as per the current development versions of Siddhi CEP 4.x (Version 4.0.0-M51) and WSO2 DAS 4.x (4.0.0-M6). The API PySiddhi3 on Siddhi CEP 3.x can be considered as final since Siddhi CEP 3.1 is a stable release.

Deliverables

  • PySiddhi4 - Python Wrapper on Siddhi 4.x and WSO2 DAS 4.x [GitHub, Wiki]
  • PySiddhi3 - Python Wrapper on Siddhi 3.1 [GitHub, Wiki]

Project Progress

The project consisted of 3 components Python wrapper on Siddhi CEP 4.x, Python wrapper on Siddhi CEP 3.1 and Python API (REST Client) on WSO2 DAS 4.x. All 3 components were developed from scratch. No previous coding work has been done on developing the relavant wrappers prior to GSOC 2017. The APIs dependended on existing opensource products requests and pyjnius as external libraries.

During the latter stage of the project, the components Python wrapper on Siddhi CEP 4.x and Python API (REST Client) on WSO2 DAS 4.x were merged to produce the product PySiddhi4. Python wrapper on Siddhi CEP 3.1 was evolved into product PySiddhi3.

Completed Work

*Both products PySiddhi3 and PySiddhi4 were initially developed by myself in a separate single repository. Later, they were divided into two branches of a single new repository PySiddhi hosted by WSO2. This operation has caused commit history to be lost from new repository.

*Python API for WSO2 DAS 4.0 was initially developed as a separate component in a separate repository by myself. Later, it was merged with the master branch of PySiddhi organisation repository hosted by WSO2. This operation has caused commit history to be lost from the new repository.

Not Completed

  • It was initially planned to develop a Python Client on WSO2 DAS 3.1 as well. However, due to time constraints and higher priority given to DAS 4.0 Python Client as a new requirement, this deliverable was not undertaken.

Contributions to other related projects

During the course of GSoC, I have contributed to the following Open Source Projects which are related to my project.

Major Challenges Faced

  1. Compatibility issue with Python 3.4+
  • In PySiddhi, pyjnius API was used internally for natively bridging JAVA with Python. The update 3.4 of Python has introduced a new issue to pyjnius API, which causes it to crash on callbacks that are triggered by threads spawned by JAVA. (since these new threads are unknown to Python Interpreter). This issue caused SiddhiDebuggerCallback of PySiddhi4 to fail.
  • The threadfix workaround mentioned here was used to fix this issue.
  1. Compatibility issue with Windows.
  • The pyjnius API used internally had an issue which causes JVM to crash when callbacks are triggered in Microsoft Windows 64 bit Operating Systems. This caused all callback events in PySiddhi such as QueryCallback, StreamCallback, SiddhiDebuggerCallback to fail in Microsoft Windows 64 bit Operating Systems.
  • The issue was fixed by myself using the following pull request.

Background

Siddhi is a Query Language and a Library for Realtime Streaming Complex Event Processing developed by WSO2 Inc. Siddhi CEP is currently used in WSO2 Data Analytics Server, an Enterprise Level Open Source Data Analytics Solution.

Further information on above products are available in the links below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment