We are really pleased to publish a guest contribution by Kai Wähner about Smart Distributed Applications.
Kai is Technology Evangelist and Community Director for TIBCO Software. His expertise lies within the fields of Big Data, Advanced Analytics, Machine Learning, Integration, SOA, Microservices, BPM, Cloud, Internet of Things, Blockchain and Programming Languages such as Java EE, Scala, Groovy, Go or R. He regularly writes about new technologies, articles and conference talks on his blog.
We are approaching Blockchain 3.0 these days. You ask “3.0”? Seriously? Blockchain is still in the early adoption phase! Yes, that is true. Nevertheless, while we are moving forward to real world use cases in various industries, we see that you need more than just a blockchain infrastructure. You need to build decentralized applications. These include blockchains, smart contracts plus other applications plus integration and analytics on top of these solutions.
Middleware is Key for Success in Blockchain Projects
Blockchain is the next big thing for middleware! There is no question around this. You need to interconnect other applications, microservices and cloud offerings with a blockchain infrastructure to get real value out of it. In addition, visual analytics and machine learning have to be leveraged to find insights and patterns in blockchain and non-blockchain data. Finally, streaming analytics is used to apply these insights and patterns to new events in a blockchain infrastructure. There is a variety of use cases like fraud detection, compliance issues, optimization of manufacturing or supply chain processes, or any kind of scenarios with the Internet of Things (IoT).
Reference Architecture for Blockchain and Middleware
Variety of Blockchain Platforms including Hyperledger and Ethereum
The blockchain market is growing significantly these days. You need to think about various blockchain characteristics for your next blockchain project.
- Who are the users of the blockchain? Is it public or private? Which partners do you need to work with?
- Do you want to build your own blockchain infrastructure (based on a framework such as Hyperledger and one if its implementations like IBM’s Hyperledger Fabric, Iroha, Intel’s Sawtooth Lake) or leverage an existing platform (like Ethereum)?
- Do you want to host the infrastructure yourself or leverage a cloud service like IBM’s Bluemix with focus on Hyperledger or Microsoft’ Azure with focus on Ethereum?
- What development environment and tooling do you want to use (like Truffle or BlockApps on top of Ethereum)
- Which characteristics are important for your scenario? What about speed, security, consensus algorithms, integration of non-blockchain services, and other important aspects? Maybe an add-on on top of a blockchain is needed, like the Raiden Network to leverage off-chain state networks to extend Ethereum with some nice properties like scalability or high performance for asset transfers?
- Or do you want to focus on a industry-specific blockchain solution like R3 Corda or Ripple for financial services?
- What middleware do you need? Do you need Application Integration or API Management to interconnect everything? Visual Analytics to find insights and patterns in historical blockchain data? Streaming Analytics to apply rules to action in (near) real time for new blockchain events?
The following shows how to leverage Streaming Analytics together with blockchain events. This example uses TIBCO StreamBase in conjunction with the public Ethereum test network. Note that similar scenarios can be build with any other blockchain infrastructure. A follow-up post about how to leverage middleware with Hyperledger will come soon, too.
Streaming Analytics for Correlation of Blockchain and Non-Blockchain Events
The scenario uses a Smart Contract to define a Coin system. You can mine coins and transfer them to other users (i.e. blockchain addresses). This example is similar to Bitcoin concepts to show how to leverage streaming analytics with any custom blockchain application and smart contracts. The goal is not to show the power of smart contracts (other articles are available for this). The programming language used to develop this Smart Contract is Solidity; more or less the de facto standard to write smart contracts for Ethereum.
Here is the Smart Contract built and deployed with Browser Solidity:
Smart Contract ‘Coin’ developed and deployed with Browser Solidity
MetaMask, a bridge to run Ethereum dApps in your Chrome browser, is running in the background to connect to the Ethereum network and commit the transactions developed with Browser Solidity. You could also use Streaming Analytics to deploy smart contracts, of course. However, in this example TIBCO StreamBase was only used for the following two parts:
- Receive new events from the blockchain network: You can filter, aggregate, analyse or transform any events like pending transactions, logs or blockchain blocks – and also combine this information with non-blockchain events, of course. For example, you could build a streaming analytics process to analyze just the logs relevant for your specific transaction IDs to spot issues and act proactively, let’s say if a pending transaction takes too long or fails.
- Execute transactions on the blockchain network via smart contracts: You can mine new coins, send coins to other blockchain addresses and also check the balance of an address. Anything what the smart contract allows can be included into the streaming analytics process.
The streaming analytics process monitors all Ethereum events continuously. This is not as trivial as you might know it from classical messaging systems. You cannot just listen to a topic or queue, but you have to pull information out of the blockchain. Depending on the use case, you have to implement some solution which solves your problem but also does not consume too many resources. This is always a trade-off, which has to be thought through when building your streaming analytics process. This also highly depends on the blockchain infrastructure you use and its feature set.
Please note that security considerations are not part of this example. In the real world, you would integrate encryption and other security requirements into the streaming process, of course. In this demo, we use “hardcoded” private keys for sending transactions. A no-go in a real world project.
Let’s now take a look at an implementation of this process.
TIBCO StreamBase + Ethereum Blockchain
Here is the demo setup:
Architecture with TIBCO StreamBase + Ethereum Blockchain
The Ethereum test network is a distributed peer to peer ledger. It runs on various Ethereum clients. We used one of the most mature ones on our local laptop: The geth client implement in Golang. This is synced and also part of the Ethereum test network.
TIBCO StreamBase is used to build the streaming analytics process:
TIBCO StreamBase Connectors for Ethereum
The web3j Java API is used to connect TIBCO StreamBase with the Ethereum network through our local geth client. You just need to write the connector once and can reuse it in all your streaming processes via drag&drop and configuration. These behave “just” like any other connector (such as messaging via MQTT or Apache Kafka) and components to build streaming logic (like filter, aggregate or transform).
For more details, please check out my live demo of combining streaming analytics and Ethereum blockchain
Building this process was actually a pretty easy task with TIBCO StreamBase. In the same way, you can build much more sophisticated blockchain processes in your real world project. Let’s also think about some other next steps.
Next Steps: Application Integration, API Management, Machine Learning, and more
A real world blockchain projects needs streaming analytics to correlate blockchain and non-blockchain events to fight fraud or compliance issues, to improve efficiency in manufacturing or supply chain processes, to combine Internet of Things with blockchain infrastructures, and for many other use cases.
Though, Streaming Analytics is just one piece of the puzzle. Here are some more thoughts about why you might combine blockchain with middlware and analytics:
- Live Visualization for Real Time Monitoring and Proactive Actions
- Cross-Integration with Ethereum and Hyperledger Blockchains
- Data Discovery for Historical Analysis to Find Insights and Patterns
- Machine Learning to Build of Analytic Models
- Application Integration with other Applications (Legacy, Cloud Services, …)
- API Management to expose blockchain services and handle caching / throttling challenges
- Native Hardware Integration with Internet of Things Devices
I will do more posts about these ideas and show more live demos in the next weeks and months. In the meantime, first customer projects also kicked off, already. Blockchain and middleware have a great and interesting future…
Keywords:
Blockchain, Ethereum, Hyperledger, Middleware, Integration, Streaming Analytics, TIBCO, StreamBase, Live Datamart, Smart Contracts, Cloud, web3j