Blockchain + Streaming Analytics = Smart Distributed Applications

We are really pleased to publish a guest contribution by Kai Wähner about Smart Distributed Applications.
Kai is Technology Evangelist and Community Director for TIBCO Software. His expertise lies within the fields of Big Data, Advanced Analytics, Machine Learning, Integration, SOA, Microservices, BPM, Cloud, Internet of Things, Blockchain and Programming Languages such as Java EE, Scala, Groovy, Go or R. He regularly writes about new technologies, articles and conference talks on his blog.

We are approaching Blockchain 3.0 these days. You ask “3.0”? Seriously? Blockchain is still in the early adoption phase! Yes, that is true. Nevertheless, while we are moving forward to real world use cases in various industries, we see that you need more than just a blockchain infrastructure. You need to build decentralized applications. These include blockchains, smart contracts plus other applications plus integration and analytics on top of these solutions.

Middleware is Key for Success in Blockchain Projects

Blockchain is the next big thing for middleware! There is no question around this. You need to interconnect other applications, microservices and cloud offerings with a blockchain infrastructure to get real value out of it. In addition, visual analytics and machine learning have to be leveraged to find insights and patterns in blockchain and non-blockchain data. Finally, streaming analytics is used to apply these insights and patterns to new events in a blockchain infrastructure. There is a variety of use cases like fraud detection, compliance issues, optimization of manufacturing or supply chain processes, or any kind of scenarios with the Internet of Things (IoT).

Reference Architecture for Blockchain and Middleware

Variety of Blockchain Platforms including Hyperledger and Ethereum

The blockchain market is growing significantly these days. You need to think about various blockchain characteristics for your next blockchain project.

  • Who are the users of the blockchain? Is it public or private? Which partners do you need to work with?
  • Do you want to build your own blockchain infrastructure (based on a framework such as Hyperledger and one if its implementations like IBM’s Hyperledger Fabric, Iroha, Intel’s Sawtooth Lake) or leverage an existing platform (like Ethereum)?
  • Do you want to host the infrastructure yourself or leverage a cloud service like IBM’s Bluemix with focus on Hyperledger or Microsoft’ Azure with focus on Ethereum?
  • What development environment and tooling do you want to use (like Truffle or BlockApps on top of Ethereum)
  • Which characteristics are important for your scenario? What about speed, security, consensus algorithms, integration of non-blockchain services, and other important aspects? Maybe an add-on on top of a blockchain is needed, like the Raiden Network to leverage off-chain state networks to extend Ethereum with some nice properties like scalability or high performance for asset transfers?
  • Or do you want to focus on a industry-specific blockchain solution like R3 Corda or Ripple for financial services?
  • What middleware do you need? Do you need Application Integration or API Management to interconnect everything? Visual Analytics to find insights and patterns in historical blockchain data? Streaming Analytics to apply rules to action in (near) real time for new blockchain events?

The following shows how to leverage Streaming Analytics together with blockchain events. This example uses TIBCO StreamBase in conjunction with the public Ethereum test network. Note that similar scenarios can be build with any other blockchain infrastructure. A follow-up post about how to leverage middleware with Hyperledger will come soon, too.

Streaming Analytics for Correlation of Blockchain and Non-Blockchain Events

The scenario uses a Smart Contract to define a Coin system. You can mine coins and transfer them to other users (i.e. blockchain addresses). This example is similar to Bitcoin concepts to show how to leverage streaming analytics with any custom blockchain application and smart contracts. The goal is not to show the power of smart contracts (other articles are available for this). The programming language used to develop this Smart Contract is Solidity; more or less the de facto standard to write smart contracts for Ethereum.

Here is the Smart Contract built and deployed with Browser Solidity:

Smart Contract ‘Coin’ developed and deployed with Browser Solidity

MetaMask, a bridge to run Ethereum dApps in your Chrome browser, is running in the background to connect to the Ethereum network and commit the transactions developed with Browser Solidity. You could also use Streaming Analytics to deploy smart contracts, of course. However, in this example TIBCO StreamBase was only used for the following two parts:

  • Receive new events from the blockchain network: You can filter, aggregate, analyse or transform any events like pending transactions, logs or blockchain blocks – and also combine this information with non-blockchain events, of course. For example, you could build a streaming analytics process to analyze just the logs relevant for your specific transaction IDs to spot issues and act proactively, let’s say if a pending transaction takes too long or fails.
  • Execute transactions on the blockchain network via smart contracts: You can mine new coins, send coins to other blockchain addresses and also check the balance of an address. Anything what the smart contract allows can be included into the streaming analytics process.

The streaming analytics process monitors all Ethereum events continuously. This is not as trivial as you might know it from classical messaging systems. You cannot just listen to a topic or queue, but you have to pull information out of the blockchain. Depending on the use case, you have to implement some solution which solves your problem but also does not consume too many resources. This is always a trade-off, which has to be thought through when building your streaming analytics process. This also highly depends on the blockchain infrastructure you use and its feature set.

Please note that security considerations are not part of this example. In the real world, you would integrate encryption and other security requirements into the streaming process, of course. In this demo, we use “hardcoded” private keys for sending transactions. A no-go in a real world project.

Let’s now take a look at an implementation of this process.

TIBCO StreamBase + Ethereum Blockchain

Here is the demo setup:

Architecture with TIBCO StreamBase + Ethereum Blockchain

The Ethereum test network is a distributed peer to peer ledger. It runs on various Ethereum clients. We used one of the most mature ones on our local laptop: The geth client implement in Golang. This is synced and also part of the Ethereum test network.

TIBCO StreamBase is used to build the streaming analytics process:

TIBCO StreamBase Connectors for Ethereum

The web3j Java API is used to connect TIBCO StreamBase with the Ethereum network through our local geth client. You just need to write the connector once and can reuse it in all your streaming processes via drag&drop and configuration. These behave “just” like any other connector (such as messaging via MQTT or Apache Kafka)  and components to build streaming logic (like filter, aggregate or transform).

For more details, please check out my live demo of combining streaming analytics and Ethereum blockchain

Building this process was actually a pretty easy task with TIBCO StreamBase. In the same way, you can build much more sophisticated blockchain processes in your real world project. Let’s also think about some other next steps.

Next Steps: Application Integration, API Management, Machine Learning, and more

A real world blockchain projects needs streaming analytics to correlate blockchain and non-blockchain events to fight fraud or compliance issues, to improve efficiency in manufacturing or supply chain processes, to combine Internet of Things with blockchain infrastructures, and for many other use cases.

Though, Streaming Analytics is just one piece of the puzzle. Here are some more thoughts about why you might combine blockchain with middlware and analytics:

  • Live Visualization for Real Time Monitoring and Proactive Actions
  • Cross-Integration with Ethereum and Hyperledger Blockchains
  • Data Discovery for Historical Analysis to Find Insights and Patterns
  • Machine Learning to Build of Analytic Models
  • Application Integration with other Applications (Legacy, Cloud Services, …)
  • API Management to expose blockchain services and handle caching / throttling challenges
  • Native Hardware Integration with Internet of Things Devices

I will do more posts about these ideas and show more live demos in the next weeks and months. In the meantime, first customer projects also kicked off, already. Blockchain and middleware have a great and interesting future…

Keywords:

Blockchain, Ethereum, Hyperledger, Middleware, Integration, Streaming Analytics, TIBCO, StreamBase, Live Datamart, Smart Contracts, Cloud, web3j

Static Type Safety for DApps without JavaScript

DApps, starting professionally…

You might not be aware, but despite its similarities to JavaScript, Solidity is actually a statically, strongly typed language, more similar to Java than to JavaScript.

solidity
static type check in browsersolidity

…and ending in frontend-chaos

Sadly, for a long time, there has only be one interface to Ethereum nodes, web3.js (besides JSON/RPC), which is, as its name implies, written in JavaScript.

Though providing this API in a web-native language is really a brilliant idea in terms of fast development, seperation of concerns and ease of use, it is a nightmare for professional, multi-developer, multi-year, enterprise products.

You may not agree with me here, but as there are currently no 10 year old 1.000.000 LoC enterprise projects in node.js/JavaScript out there, you should at least consider that such projects are nearly impossible to maintain with a dynamically, weakly typed language like JavaScript (JS).

So, we have this situation, where JS defines the lowest common denominator (dynamically, weakly typed

JavaScript_1 (2)

when we really would like to have this situation, where Java (C#, Haskell) defines the lowest common denominator (statically, strongly typed)

JavaScript_2 (2)

Removing chaos

The problem is was, that up to now only web3.js existed. However, today there is also a web3.py (which is Python and therefore at least strongly typed, but still dynamically) and, brandnew, web3j.

With the latter, we can easily model the call chain above, where we only use statically, strongly typed Java and omit JavaScript altogether. Welcome to hassle-free integration into existing Java/JEE-environments without workarounds. Finally: using the Ethereum Blockchain with Java.

If you want to actually get deeper and use Java with no RPC at all, you can also switch to EthereumJ, which is a Ethereum Node implemented in Java, like Eth (C++), Geth (Go), PyEthApp (Python) or Parity (Rust). It is crucial to understand the difference between web3j and EthereumJ. If you just want to use some Ethereum Node from a Java application, web3j is your choice, you are limited to the Web DApp API then, which should be enough for all “Ethereum user” use cases.

We will not explain in detail how to use web3j, it should be familiar to any Java developer how this library can be used just by adding Maven-dependencies to your project.

Fixing the front-end

We could stop here, since using JavaScript for the frontend is not really problematic and a common use today.

However, if you use JavaScript in your frontend, it might really make more sense to stay with web3.js. So, we want to go further: how are we going to create the GUI if we want to have no JavaScript at all?

This is just a PoC, but if you think of any other client to the Ethereum Blockchain other than a web site (let’s say: Batchjobs, Web Services, Message Queues, Databases, other proprietary software with Java adapters (there are some!)), this should make sense to you – you really wouldn’t want to use them from web3.js (hopefully).

Using templates: Thymeleaf and Spring Boot for slim enterprisy software

We will do a step-by-step guide for creating a No-JS-Dapp. Even without any Java experience, you will be able to follow without problems. Java is not that complicated anymore!

  • Get an infura.io account and key, so you don’t have to mess around with starting your own Ethereum node
  • Clone this repo: https://hellokoding.com/spring-boot-hello-world-example-with-thymeleaf/
  • Install Maven
  • Edit these files:

    pom.xml (add these dependency to section dependencies and add the repo, beware that web3j is a fast moving target, check for new versions)

    <dependency>
    <groupId>org.web3j</groupId>
    <artifactId>core</artifactId>
    <version>0.2.0</version>
    </dependency>

 

<repositories>
<repository>
<id>oss.jfrog.org</id>
<url>http://dl.bintray.com/web3j/maven</url>
</repository>
</repositories>

src/main/resources/templates/hello.html (change name to balance.html)

<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.thymeleaf.org">
<head>
<meta charset="UTF-8"/>
<title>Your Static Strongly Typed Wallet</title>
</head>
<body>
<p th:text="'The balance of account ' + ${address} + ' is ' + ${balance}" />
</body>
</html>

src/main/java/com/hellokoding/springboot/HelloController.java (change name to EthereumController.java)

@Controller
public class EthereumController {

@RequestMapping("/balance")
public String balance(Model model, @RequestParam(value="address", required=false, defaultValue="0xe1f0a3D696031E1F8ae7823581BB38C600aFF2BE") String address) throws IOException {
Web3j web3 = Web3j.build(new HttpService("https://consensysnet.infura.io/{YOUR_INFURA_KEY}"));
EthGetBalance web3ClientVersion = web3.ethGetBalance(address, DefaultBlockParameter.valueOf("latest")).send();
String balance = web3ClientVersion.getBalance().toString();
model.addAttribute("address", address);
model.addAttribute("balance", balance);
return "balance";
}

}

…that’s it. Start with mvn spring-boot:run

If you encounter an connection/handshake error, you may have to import the infura certificate into your local Java keystore (I didn’t have to)

$JAVA_HOME/Contents/Home/jre/bin/keytool -import -noprompt -trustcacerts -alias morden.infura.io -file ~/Downloads/morden.infura.io -keystore $JAVA_HOME/Contents/Home/jre/lib/security/cacerts -storepass changeit

Look Ma! Displaying the wallet balance with no JavaScript!

You can call the spring-boot web application with http://localhost:8080/balance (then the defined default argument is used) or with your address (in the consensys testnet) as parameter address= 

walletOf course, you can change the Ethereum net like you want in file EthereumController to morden or mainnet, just read the welcome mail from infura.io. Or you can just use a local Ethereum node like geth with RPC enabled (geth –rpc) and http://localhost:8545 as the constructor for HttpService of the Web3j-Factory in EthereumController.

Have fun, with or without JavaScript!

Distributing Business Processes using Finite State Machines in the Blockchain

Having disctinct, even competing organisations with a common goal and no shared technical infrastructure represent a perfect use case pattern for blockchain usage.

The quest for the holy business use case

As many blockchain enthusiasts we are constantly searching for use cases for establishing Distributed Ledger Technology (DLT) or even real blockchain technology in enterprises as solutions for common business problems.

21666276190_40647d8327_z
[Business *]

This turns out to be difficult, since each single aspect of this technology is already solved by several – established and well known – products. 

In this post we will present a use case, or even a use case pattern, which we think is ubiquitous and can best be solved by blockchain technology.

For these type of use cases the other technologies, even though being more mature, seem themselves like workarounds for the natural technological solution, which is blockchain-based.

Our definition is the use case pattern is:

Several distinct, competing organisations have a common goal and don't share technical infrastructure.

distinct, even competing organisations

Distinct organisations are not related to each other and therefore have no existing technical or organisational processes. To pursue a common goal, these processes would have to be established first, which is costly and time-consuming.

5882566863_ce0ccf745d_z
[Competing teams *]

The second aspect, competition, implies that there is no mutual trust. Obviously, this aspect is a crucial one. Blockchain technology might even make sense if just the other aspects of the pattern match, but it only is a perfect fit if this aspect is important, since the blockchain itself is inherently trust-less.

common goal

This is the fundamental requirement, if there is no common goal, there is no need to establish any collaboration. It is important to note, that the common goal is most likely not related to the core competence of the organisations, but to some crosscutting concerns, which have to be addressed by all organisations, but are just cost factors.

no existing shared technical infrastructure

To put it contrary: if there already exists a shared technical infrastructure, also organisational processes exist to establish new technology. Given this, there are several great technology products which implement each aspect of the blockchain technology (namely distributed data storage, immutability, security), and most likely even more performant, more mature, with less maintenance and evident responsibilities.

Ok, this is quite abstract and complex, let’s find an easy example.

…want a securities account with your new account?

There is one universal retail bank, FriendlyBank, which wants people to open accounts. And there are three deposit banks, FriendlyDeposit, EasyDeposit and NiceDeposit. There is also the Regulator, who must be able to audit all transactions between the parties.

Requirement

common goal: make money

The common goal of universal and deposit banks is to open accounts. The universal bank can be intermediary to deposit account opening, so you can open a securities account at one of the three deposit banks with your new account at FriendlyBank. This way, it is a win-win-situation for both parties, the intermediary gets commissions, the deposit banks get new customers.

FriendlyBank and FriendlyDeposit are related to each other, they have the same ownership structure, but are separated entities. Since the deposit banks are in competition with each other, the other deposit banks besides FriendlyBank want to be sure that not very valuable customers are transferred to FriendlyDeposit or the deposit account opening is “accidentally forgotten”, if not FriendlyDeposit is chosen.
Since the parties do not really trust each other, a trust-less (ie. no trust necessary) solution is needed.

the wonderful world of finance IT today

As we said, the depicted scenario exists quite often. Also, technical “solutions” for these scenarios exist.

They look similar to this one:

Flatfilesolution_crop

We wan’t go in detail what all this means, but to become trust-less, as well as assure that the data is in sync in each organisation, a lot has to be implemented and still, shortcomings of these solutions are quite common:

  • reconciliation processes are always necessary, but still, due to missing transaction support in flat file exchange, which is almost always chosen as the “simplest” integration pattern, errors occur.
  • adapters have to be built for each party, so there will soon exist a many-to-many problem.
  • special adapters have to be built for regulators.
  • crosscutting concerns have to be implemented: security, authentication, auditing, transaction support

Compared to a natural solution in the blockchain

Below is the blockchain solution, it fits naturally and implements all requirements.bcsample

Abstracting from the use case

Why is this solution matching so well, what is the pattern “behind”?

Different actors change the state of a business process on some event in a transactional way.

2645030_8e2b2269
[Old cigarette machine *]

You certainly know this pattern, it is describing a finite state machine, more exactly a distributed, secured by cryptoeconomics finite state machine, which events are transactions.

After all, the blockchain itself is a state transition system, it somes quite naturally to implement a simple state machine on the Ethereum EVM, it even is mentioned as a common pattern in the Solidity docs.

 

Approaching the finite state machine

How would the organisations collaborate? This could look like this informally:

 

process_v1

or like this (more) formally:

Blockchainsolution_FSM

so we actually can built this state machine and its transitions really easy using the blockchain, here in pseudo-Solidity, derived from the common pattern in the Solidity docs:

contract AccountDepositStateMachine {
enum Stages { Init, AccountOpened, DepositOpened, DepositConfirmed, AccountConfirmed } Stages public stage = Stages.Init; modifier atStage(Stages _stage) { if (stage != _stage) throw; _ } function nextStage() internal { stage = Stages(uint(stage) + 1); } // Order of the modifiers matters here! function openAccount() atStage(Stages.Init)
transitionNext
{ if (msg.sender != FRIENDLY_BANK) throw; } function openDeposit() atStage(Stages.AccountOpened)
transitionNext
{
if (msg.sender != FRIENDLY_BANK) throw;
}
function confirmDeposit()
atStage(Stages.DepositOpened)
transitionNext
{
if (msg.sender != DEPOSIT_BANK) throw;
}
modifier transitionNext() { _ nextStage(); } }

Great, so that’s it, we can build any business process using the blockchain, QED.

Leaving the ivory tower

As you might guess, it is not that simple. Let’s review the actual process:

  1. Account is opened 
  2. Deposit Account is opened
  3. Deposit Confirmed
  4. Account Confirmed

Ever saw an IT-process in real life? Exactly, it’s not like this, not even close. Let’s see a more real life example:

  1. Account is opened
  2. An Email is send to whoever feels responsible
  3. The Product Owner’s Excel has to be updated
  4. The new reporting engine must have its data updated, it is runnig with MongoDB, which has to be updated
  5. Revision wants to have it’s auditing data in their Oracle DB (must use 2PC)
  6. … (ok, you got it already…)

None of the tasks in bold red can be accomplished from the Ethereum blockchain, since calls from “inside” to “outside” are prohibited. Let’s hope this will never change, the separation of “inside” and “outside” is essential for the stability and security of the Ethereum blockchain

We could use Oracles for this, but it would be a kind of usage which is highly inappropriate for this type of external information retrieval.

Teaser: “Mastering the Flow: Blending Reality with the Blockchain”

This post is way too long already, so here comes a teaser for two posts next to come: “Mastering the Flow: Blending Reality with the Blockchain”

The idea is really simple: let’s just recentralize the business process (we are in eager anticipation for the comments to come…)

But we think this can make sense. Look at the sample above, illustrated as a flow:

nodered

The flow was created with Node-RED, an excellent and highly undervalued tool of IBM Emerging Technologies for flows in IoT. It could be very easily adapted to Ethereum with smart contract access by usage of web3.js, which itself can be integrated in node.js, which is the basis of Node-RED, hence the name.

We make the following assertion:

More than 80% of all use cases can be realized by a centralized business process engine as a first layer, eg. implemented in Node-RED, and a state machine implemented in the blockchain as a second layer

The business process engine is the glue between the transactional blockchain state transitions and the secondary business processes.

What do you think of it? Please let us know in the comments, we really appreciate feedback, positive or negative.

And stay tuned for the next episode, we will get to the nitty-gritty there.

images:
[Business] Business by Christophe BENOIT under CC
[Competing teams] Competing teams by Denis De Mesmaeker under CC
[Old cigarette machine] Old cigarette machine by Walter Baxter under CC