Excel2Blockchain with web3j

If you ever come to the conclusion, for whatever reason, that pushing Excel data to the Blockchain is what you need, we are happy to help!

We recently read about the Azure Blockchain Development Kit and had to cringe about the samples. But, who are we to judge those important enterprise business use cases requiring to get data from Excel to the Blockchain? 

Therefore, we took a look at the prerequisites, setup, initialisations and finally the execution of this with the new Microsoft Azure templates. And, to be clear, we love the idea of Microsoft getting involved into Blockchains and Ethereum and pushing Azure as a starter kit, that’s really great.

However, the whole procedure seemed quite long and exhaustive, that’s why we want to show how easy this is with standard development methods. Of course, these are lacking a lot of features and cool stuff you get with Azure then. But, they can help you to understand better what happens under the hood. So, let’s go for it.

No, wait, Excel to Blockchain, honestly? Why?

//TODO: Insert suitable reason here.

Ok then, but how?

Due to web3j it is really easy to connect to an arbitrary Ethereum Node, even Infura and Ganache, as simple as it is with web3.js or web3.py. All the other stuff is common Java dev tools, like Glueing with Maven, Excelling with Apache POI, etc. This is where Java shines, it’s really good at enterprisey integration stuff.

You can start directly and import the sources as a Maven project in your IDE or build it without an IDE and start it from the command line. See the details here.

Show me some code!

private void connectToBlockchain() throws IOException {
Web3j httpWeb3 = Web3j.build(new HttpService("http://localhost:8545"));
log.info("Connected to HTTP/JSON Ethereum client version: " + httpWeb3.web3ClientVersion().send().getWeb3ClientVersion());
// Create credentials from private key.
// Don't use with real (Mainnet) Ethereum accounts.
String privateKeyDeployer = "c87509a1c067bbde78beb793e6fa76530b6382a4c0241e5e4a9ec0a0f44dc0d3";
Credentials credIdentity = Credentials.create(pkDeployer);
BigInteger balance = httpWeb3.ethGetBalance(credIdentity.getAddress(), DefaultBlockParameterName.LATEST).send().getBalance();
log.info("Deployer address " + credIdentity.getAddress() + " has " + balance + " wei.");
}

private void deployExcelContractToBlockchain() throws Exception {
ExcelStorage excelStorageContractHttp = ExcelStorage.deploy(httpWeb3, credIdentity, ManagedTransaction.GAS_PRICE, Contract.GAS_LIMIT).send();
log.info("Deployed contract at " + excelStorageContractHttp.getContractAddress());
}

As you can see, ExcelStorage is a class which can used like any other Java class. It has been generated by a Maven plugin (web3j-maven-plugin) during build time. It is statically typed with methods resembling the Solidity functions of the smart contract and with all the nice aspects this has: compile time checks instead of runtime checks and brilliant IDE support (code completion, type checking, static code analysis).

With great IDE and build tool support, you can try this out yourself, you will be surprised how lightweight and easy Ethereum smart contract usage is from Java. Even more, integration with your existing (“legacy”) software and infrastructure is straightforward. 

Have fun!

Addendum: This is a Testsetup only! What about the real Blockchain?

Where does the private key in the current Java sources come from? 
If you start Ganache with the mnemonic mentioned, everytime the same accounts are generated. The first one being 0x627306090abaB3A6e1400e9345bC60c78a8BEf57. This address has one private key which is set into the Java sources to create the Credentials for contract deployment and transactions.

Setup for other test chains

  • Create an account with infura.io
  • Use the Endpoint URL from the dashboard and copy in into the Java sources

  • Create an account with MetaMask
  • Copy the address of your account

  • Copy the private key to the Java source code to use the account for contract deployment and transactions

Identity is the New Money

This post is inspired by the book with the same title, which we found in in this nice summary post by Alex PreukschatIn that blog post and especially in the book you will find all information about Self-Sovereign Identity. We will only refer to the concepts if they are feasible for our samples.

This is not a Blockchain-bullshit-only post. Head to https://github.com/ice09/identity for a sample implementation of the “federated” claim issuer sample.

The Story of Alice, Booze and her local Bank

Alice just turned 18 years old, being in good ol’ Germany she wants to celebrate this with a lot of booze, but her parents refused to buy it for her, since she can do this on her own now.

Currently, she can do this easily by showing her identity card when buying the drinks. The dealer would check her birthdate and verify her age.

This obviously has major shortcomings:

  • The ID could easily be altered, stolen or otherwise not belonging to the person showing it
  • The dealer gets a lot more information than necessary for this transaction, including a lot of sensitive personal data: name, address, height, etc.

So, the dealer would just have to verify that the person is older than 18 years, how could this work with Alice and eg. her local bank?

The book differentiates between identity and statements about this identity (claims), which entitle the person to do things. In our example, the fact that Alice turned 18 years old entitles her to buy and drink alcohol (in certain legislations). However, showing the identity card to the dealer mixes up two aspects very badly: the dealer has to check the identity information to derive the entitlement. This he has to do by himself, if for whatever reason (eg. the dealer not being able to calculate the age) he cannot derive the entitlement, Alice’s intention would be rejected. Also, only information which can be derived from the identity information can be used, if eg. Alice would be incapacitated and therefore not allowed to buy alcohol, the dealer wouldn’t know, since this information is not stated on her identity card.

In the world of cryptography and decentralized data storage (read: DLT), Alice’s bank could easily help her with an identity or better: entitlement verification service. Since Alice participated at the bank’s KYC process, the bank has a lot of information about Alice, one of these being her age. Now, if the local bank is trusted by the shop owner, this information would be enough for the owner: “As your local trusted bank, we verify that this customer is older than 18 years.”

Setup for Ethereum

In Ethereum, the easiest setup would be to use ERC 725/735. These standards (requests) provide all necessary equipment for identity creation and claims/endorse mechanism. Other standards and products are provided by uPort and standard bodies like the W3C DID.

The initial setup Alice has to execute, together with the claim issuers like the bank, a utility provider, an insurance company, etc. consists mainly of three steps which could easily be embedded into a mobile app:

  • Alice creates an identity ERC 725/735 (which might be one of multiple identities she has)
  • As a claim issuer, the bank adds a claim (using ERC 735 addClaim(…)) about Alice being older than 18 years. This claim is signed by the private key of the bank. The claim is stored with a returned ID and must be approved by Alice (1, 2).
  • Alice approves (using ERC 725 approve(claimId)) the bank’s claim on her identity.
  • Now, anyone can verify the claim about the topic she is interested in, eg. the fact that the identity is older than 18 years (4).

After the setup the process of verifying a claim is depicted below:

  • Alice signs a challenge of the shop owner with her private key (3).
  • With the derived public key of Alice, the shop owner can identify the claim in ERC 735 and check the signature of the trusted party, which is Alice’s bank (4), against the publicly available public key of the bank.

This would make a great use case for banks desperately looking for their next Blockchain use case: forget about value transfer, asset creation, settlement and whatnot, you are fighting against proven and established systems which might be embedded in complex processes but have been running stable, reliable and performant for years.

Identity verification on the contrary could be a completely new service offering and banks have the great advantage of having identified thousands of customers in strongly regulated processes, they have valuable data at their hand. The same is true for insurances, mobile providers, utility providers, governmental companies, etc.

But let’s say we want to decentralize even more, how could we go on?

Incentivizing Identity Verification

Couldn’t we incentivize people for doing something only they are able to do right: prove the identities of their social network?

There are some assumptions:

  1. People know their social environment best, especially their direct environment like family, best friends and neighbors.
  2. People want to earn (even few) money with something they – and only they – can do easily following a simple procedure

The process is almost equal to the process above, except that notaries are slashed if they defraud and earn fees if they behave according to the protocol. The slashing is complex and would have to be implemented by some governance mechanism. This step adds complexity, but is solvable and will be very similar to PoS. Most likely there will be a incentivization for detecting and proving fraud included into the protocol.

Incentivization is a great means for getting anonymous parties to agree on a fact. This works great for Blockchains and has also been proven theoretically by eg. game theory and the Nash equilibrium.

Therefore, with incentivization of notaries, their advantage of doing fraud is minimized.

Build a Social Network based on Trust

Trustlines is a decentralized, permissionless, open platform to host currency networks in which money is represented as IOUs issued by its participants. Trustlines implements money as bilateral peer-to-peer issued blockchain-based credit.

In the example above Bob gave Alice a creditline of 20 EUR and Alice Bob of 100 EUR, so they have bilateral creditlines which is a trustline. Bob gave Charly 30 EUR credit and Charly Bob 50 EUR. Now if Alice wants to pay Charly, whom she doesn’t know personally, the system finds a path between Alice and Charly (via Bob) and the creditlines will be used. If Alice pays Charly 10 EUR, Bob afterwards owes Charly 10 EUR and Alice owes Bob 10 EUR. The creditlines have been used and the remaining credit is lower or higher, depending on the direction of the payment.

There is no settlement in the system, if it is necessary or wanted by some party to settle, this is done outside of the system and then reflected in the system by an opposite transaction.

More information about the project is available on trustlines.network, we encourage you to inform yourself about its concepts of people powered money and complementary currencies and its potential to boost cryptocurrencies adoption and enhance accessibility for cryptocurrencies for all people.

Extra Trustlines Benefit: Socially proven Identity

However, for this blog post, a secondary aspect of Trustlines is more important: if you have a social network of trust, you have a social network of identities. This is immediately obvious by the simple fact that if you trust someone with a certain amount of money, you certainly know and trust this person, because you are in risk of losing money if your creditline is used. Depending on the social context this can be family members, good friends, co-workers and the amount of credit you give will mirror your amount of trust in these people.

Given a network with creditlines, people will most likely have creditlines with different people and a identity (and reputation) algorithm could value the amount of credit given and the number of creditlines for a probabilistic identity verification.

This might even be pseudonymous, as someone offering a service for money, I am most interested in a receiver to pay me money, which is related to his trustworthyness and therefore to the amount of creditlines given, not so much who this person actually is. If 10 people who have medium sized networks of creditlines have creditlines with the receiver, I am tending to assume that:

  1. This person actually exists
  2. This person is trustworthy for some amount to other people

For now, this is the only attribute of identity which Trustlines helps solving, but with very high accuracy: if a person actually exists and that this person is trustworthy to some degree.

We won’t have information about other identity attributes like gender, age, wealth, educational degrees and so on.

Let’s fix Identity now!

If you didn’t know a good use case for Blockchain technology, now you have it. This is one of the most natural use cases for a decentral, immutable database accessible by everyone without central control and the possibility to incentivize people.

This is a win for all involved parties, enhancing privacy and security and enabling service providers to make use of their already available data. As there is no “disruption” of any existing business model, but a creation of a completely new one, public identity management can only get better.

Token ERC Comparison for Fungible Tokens

“The good thing about standards is that there are so many to choose from.” Andrew S. Tanenbaum

Current State of Token Standards

The current state of Token standards on the Ethereum platform is surprisingly simple: ERC-20 Token Standard is the only accepted and adopted (as EIP-20) standard for a Token interface.

Proposed in 2015, it has finally been accepted at the end of 2017.

In the meantime, many Ethereum Requests for Comments (ERC) have been proposed which address shortcomings of the ERC-20, which partly were caused by changes in the Ethereum platform itself, eg. the fix for the re-entrancy bug with EIP-150. Other ERC propose enhancements to the ERC-20 Token model. These enhancements were identified by experiences gathered due to the broad adoption of the Ethereum blockchain and the ERC-20 Token standard. The actual usage of the ERC-20 Token interface resulted in new demands and requirements to address non-functional requirements like permissioning and operations.

This blogpost should give a superficial, but complete, overview of all proposals for Token(-like) standards on the Ethereum platform. This comparison tries to be objective but most certainly will fail in doing so.

The Mother of all Token Standards: ERC-20

There are dozens of very good and detailed description of the ERC-20, which will not be repeated here. Just the core concepts relevant for comparing the proposals are mentioned in this post.

The Withdraw Pattern

Users trying to understand the ERC-20 interface and especially the usage pattern for transfering Tokens from one externally owned account (EOA), ie. an end-user (“Alice”), to a smart contract, have a hard time getting the approve/transferFrom pattern right.

From a software engineering perspective, this withdraw pattern is very similar to the Hollywood principle (“Don’t call us, we’ll call you!”). The idea is that the call chain is reversed:  during the ERC-20 Token transfer, the Token doesn’t call the contract, but the contract does the call transferFrom on the Token.

While the Hollywood Principle is often used to implement Separation-of-Concerns (SoC), in Ethereum it is a security pattern to avoid having the Token contract to call an unknown function on an external contract. This behaviour was necessary due to the Call Depth Attack until EIP-150 was activated. After this hard fork, the re-entrancy bug was not possible anymore and the withdraw pattern did not provide any more security than calling the Token directly.

But why should it be a problem now, the usage might be somehow clumsy, but we can fix this in the DApp frontend, right?

So, let’s see what happens if a user used transfer to send Tokens to a smart contract. Alice calls transfer on the Token contract with the contract address 

….aaaaand it’s gone!

That’s right, the Tokens are gone. Most likely, nobody will ever get the Tokens back. But Alice is not alone, as Dexaran, inventor of ERC-223, found out, about $400.000 in tokens (let’s just say a lot due to the high volatility of ETH) are irretrievably lost for all of us due to users accidentally sending Tokens to smart contracts.

Even if the contract developer was extremely user friendly and altruistic, he couldn’t create the contract so that it could react to getting Tokens transferred to it and eg. return them, as the contract will never be notified of this transfer and the event is only emitted on the Token contract.

From a software engineering perspective that’s a severe shortcoming of ERC-20. If an event occurs (and for the sake of simplicity, we are now assuming Ethereum transactions are actually events), there should be a notification to the parties involved. However, there is an event, but it’s triggered in the Token smart contract which the receiving contract cannot know.

Currently, it’s not possible to prevent users sending Tokens to smart contracts and losing them forever using the unintuitive transfer on the ERC-20 Token contract.

The Empire Strikes Back: ERC-223

The first attempt at fixing the problems of ERC-20 was proposed by Dexaran. The main issue solved by this proposal is the different handling of EOA and smart contract accounts.

The compelling strategy is to reverse the calling chain (and with EIP-150 solved this is now possible) and use a pre-defined callback (tokenFallback) on the receiving smart contract. If this callback is not implemented, the transfer will fail (costing all gas for the sender, a common criticism for ERC-223).

 

Pros:

  • Establishes a new interface, intentionally being not compliant to ERC-20 with respect to the deprecated functions
  • Allows contract developers to handle incoming tokens (eg. accept/reject) since event pattern is followed
  • Uses one transaction instead of two (transfer vs. approve/transferFrom) and thus saves gas and Blockchain storage

Cons:

  • If tokenFallback doesn’t exist then the contract fallback function is executed, this might have unintended side-effects
  • If contracts assume that transfer works with Tokens, eg. for sending Tokens to specific contracts like multi-sig wallets, this would fail with ERC-223 Tokens, making it impossible to move them (ie. they are lost)

The Pragmatic Programmer: ERC-677

The ERC-667 transferAndCall Token Standard tries to marriage the ERC-20 and ERC-223. The idea is to introduce a transferAndCall function to the ERC-20, but keep the standard as is. ERC-223 intentionally is not completely backwards compatible, since the approve/allowance pattern is not needed anymore and was therefore removed.

The main goal of ERC-667 is backward compatibility, providing a safe way for new contracts to transfer tokens to external contracts.

Pros:

  • Easy to adapt for new Tokens
  • Compatible to ERC-20
  • Adapter for ERC-20 to use ERC-20 safely

Cons:

  • No real innovations. A compromise of ERC-20 and ERC-223
  • Current implementation is not finished

The Reunion: ERC-777

ERC-777 A New Advanced Token Standard was introduced to establish an evolved Token standard which learned from misconceptions like approve() with a value and the aforementioned send-tokens-to-contract-issue.

Additionally, the ERC-777 uses the new standard ERC-820: Pseudo-introspection using a registry contract which allows for registering meta-data for contracts to provide a simple type of introspection. This allows for backwards compatibility and other functionality extensions, depending on the ITokenRecipient returned by a EIP-820 lookup on the to address, and the functions implemented by the target contract.

ERC-777 adds a lot of learnings from using ERC-20 Tokens, eg. white-listed operators, providing Ether-compliant interfaces with send(…), using the ERC-820 to override and adapt functionality for backwards compatibility.

Pros:

  • Well thought and evolved interface for tokens, learnings from ERC-20 usage
  • Uses the new standard request ERC-820 for introspection, allowing for added functionality
  • White-listed operators are very useful and are more necessary than approve/allowance, which was often left infinite

Cons:

  • Is just starting, complex construction with dependent contract calls
  • Dependencies raise the probability of security issues: first security issues have been identified (and solved) not in the ERC-777, but in the even newer ERC-820

(Pure Subjective) Conclusion

For now, if you want to go with the “industry standard” you have to choose ERC-20. It is widely supported and well understood. However, it has its flaws, the biggest one being the risk of non-professional users actually losing money due to design and specification issues. ERC-223 is a very good and theoretically founded answer for the issues in ERC-20 and should be considered a good alternative standard. Implementing both interfaces in a new token is not complicated and allows for reduced gas usage.

A pragmatic solution to the event and money loss problem is ERC-677, however it doesn’t offer enough innovation to establish itself as a standard. It could however be a good candidate for an ERC-20 2.0.

ERC-777 is an advanced token standard which should be the legitimate successor to ERC-20, it offers great concepts which are needed on the matured Ethereum platform, like white-listed operators, and allows for extension in an elegant way. Due to its complexity and dependency on other new standards, it will take time till the first ERC-777 tokens will be on the Mainnet.

Links

[1] Security Issues with approve/transferFrom-Pattern in ERC-20: https://drive.google.com/file/d/0ByMtMw2hul0EN3NCaVFHSFdxRzA/view

[2] No Event Handling in ERC-20: https://docs.google.com/document/d/1Feh5sP6oQL1-1NHi-X1dbgT3ch2WdhbXRevDN681Jv4

[3] Statement for ERC-20 failures and history: https://github.com/ethereum/EIPs/issues/223#issuecomment-317979258

[4] List of differences ERC-20/223: https://ethereum.stackexchange.com/questions/17054/erc20-vs-erc223-list-of-differences

 

 

Blockchain + Streaming Analytics = Smart Distributed Applications

We are really pleased to publish a guest contribution by Kai Wähner about Smart Distributed Applications.
Kai is Technology Evangelist and Community Director for TIBCO Software. His expertise lies within the fields of Big Data, Advanced Analytics, Machine Learning, Integration, SOA, Microservices, BPM, Cloud, Internet of Things, Blockchain and Programming Languages such as Java EE, Scala, Groovy, Go or R. He regularly writes about new technologies, articles and conference talks on his blog.

We are approaching Blockchain 3.0 these days. You ask “3.0”? Seriously? Blockchain is still in the early adoption phase! Yes, that is true. Nevertheless, while we are moving forward to real world use cases in various industries, we see that you need more than just a blockchain infrastructure. You need to build decentralized applications. These include blockchains, smart contracts plus other applications plus integration and analytics on top of these solutions.

Middleware is Key for Success in Blockchain Projects

Blockchain is the next big thing for middleware! There is no question around this. You need to interconnect other applications, microservices and cloud offerings with a blockchain infrastructure to get real value out of it. In addition, visual analytics and machine learning have to be leveraged to find insights and patterns in blockchain and non-blockchain data. Finally, streaming analytics is used to apply these insights and patterns to new events in a blockchain infrastructure. There is a variety of use cases like fraud detection, compliance issues, optimization of manufacturing or supply chain processes, or any kind of scenarios with the Internet of Things (IoT).

Reference Architecture for Blockchain and Middleware

Variety of Blockchain Platforms including Hyperledger and Ethereum

The blockchain market is growing significantly these days. You need to think about various blockchain characteristics for your next blockchain project.

  • Who are the users of the blockchain? Is it public or private? Which partners do you need to work with?
  • Do you want to build your own blockchain infrastructure (based on a framework such as Hyperledger and one if its implementations like IBM’s Hyperledger Fabric, Iroha, Intel’s Sawtooth Lake) or leverage an existing platform (like Ethereum)?
  • Do you want to host the infrastructure yourself or leverage a cloud service like IBM’s Bluemix with focus on Hyperledger or Microsoft’ Azure with focus on Ethereum?
  • What development environment and tooling do you want to use (like Truffle or BlockApps on top of Ethereum)
  • Which characteristics are important for your scenario? What about speed, security, consensus algorithms, integration of non-blockchain services, and other important aspects? Maybe an add-on on top of a blockchain is needed, like the Raiden Network to leverage off-chain state networks to extend Ethereum with some nice properties like scalability or high performance for asset transfers?
  • Or do you want to focus on a industry-specific blockchain solution like R3 Corda or Ripple for financial services?
  • What middleware do you need? Do you need Application Integration or API Management to interconnect everything? Visual Analytics to find insights and patterns in historical blockchain data? Streaming Analytics to apply rules to action in (near) real time for new blockchain events?

The following shows how to leverage Streaming Analytics together with blockchain events. This example uses TIBCO StreamBase in conjunction with the public Ethereum test network. Note that similar scenarios can be build with any other blockchain infrastructure. A follow-up post about how to leverage middleware with Hyperledger will come soon, too.

Streaming Analytics for Correlation of Blockchain and Non-Blockchain Events

The scenario uses a Smart Contract to define a Coin system. You can mine coins and transfer them to other users (i.e. blockchain addresses). This example is similar to Bitcoin concepts to show how to leverage streaming analytics with any custom blockchain application and smart contracts. The goal is not to show the power of smart contracts (other articles are available for this). The programming language used to develop this Smart Contract is Solidity; more or less the de facto standard to write smart contracts for Ethereum.

Here is the Smart Contract built and deployed with Browser Solidity:

Smart Contract ‘Coin’ developed and deployed with Browser Solidity

MetaMask, a bridge to run Ethereum dApps in your Chrome browser, is running in the background to connect to the Ethereum network and commit the transactions developed with Browser Solidity. You could also use Streaming Analytics to deploy smart contracts, of course. However, in this example TIBCO StreamBase was only used for the following two parts:

  • Receive new events from the blockchain network: You can filter, aggregate, analyse or transform any events like pending transactions, logs or blockchain blocks – and also combine this information with non-blockchain events, of course. For example, you could build a streaming analytics process to analyze just the logs relevant for your specific transaction IDs to spot issues and act proactively, let’s say if a pending transaction takes too long or fails.
  • Execute transactions on the blockchain network via smart contracts: You can mine new coins, send coins to other blockchain addresses and also check the balance of an address. Anything what the smart contract allows can be included into the streaming analytics process.

The streaming analytics process monitors all Ethereum events continuously. This is not as trivial as you might know it from classical messaging systems. You cannot just listen to a topic or queue, but you have to pull information out of the blockchain. Depending on the use case, you have to implement some solution which solves your problem but also does not consume too many resources. This is always a trade-off, which has to be thought through when building your streaming analytics process. This also highly depends on the blockchain infrastructure you use and its feature set.

Please note that security considerations are not part of this example. In the real world, you would integrate encryption and other security requirements into the streaming process, of course. In this demo, we use “hardcoded” private keys for sending transactions. A no-go in a real world project.

Let’s now take a look at an implementation of this process.

TIBCO StreamBase + Ethereum Blockchain

Here is the demo setup:

Architecture with TIBCO StreamBase + Ethereum Blockchain

The Ethereum test network is a distributed peer to peer ledger. It runs on various Ethereum clients. We used one of the most mature ones on our local laptop: The geth client implement in Golang. This is synced and also part of the Ethereum test network.

TIBCO StreamBase is used to build the streaming analytics process:

TIBCO StreamBase Connectors for Ethereum

The web3j Java API is used to connect TIBCO StreamBase with the Ethereum network through our local geth client. You just need to write the connector once and can reuse it in all your streaming processes via drag&drop and configuration. These behave “just” like any other connector (such as messaging via MQTT or Apache Kafka)  and components to build streaming logic (like filter, aggregate or transform).

For more details, please check out my live demo of combining streaming analytics and Ethereum blockchain

Building this process was actually a pretty easy task with TIBCO StreamBase. In the same way, you can build much more sophisticated blockchain processes in your real world project. Let’s also think about some other next steps.

Next Steps: Application Integration, API Management, Machine Learning, and more

A real world blockchain projects needs streaming analytics to correlate blockchain and non-blockchain events to fight fraud or compliance issues, to improve efficiency in manufacturing or supply chain processes, to combine Internet of Things with blockchain infrastructures, and for many other use cases.

Though, Streaming Analytics is just one piece of the puzzle. Here are some more thoughts about why you might combine blockchain with middlware and analytics:

  • Live Visualization for Real Time Monitoring and Proactive Actions
  • Cross-Integration with Ethereum and Hyperledger Blockchains
  • Data Discovery for Historical Analysis to Find Insights and Patterns
  • Machine Learning to Build of Analytic Models
  • Application Integration with other Applications (Legacy, Cloud Services, …)
  • API Management to expose blockchain services and handle caching / throttling challenges
  • Native Hardware Integration with Internet of Things Devices

I will do more posts about these ideas and show more live demos in the next weeks and months. In the meantime, first customer projects also kicked off, already. Blockchain and middleware have a great and interesting future…

Keywords:

Blockchain, Ethereum, Hyperledger, Middleware, Integration, Streaming Analytics, TIBCO, StreamBase, Live Datamart, Smart Contracts, Cloud, web3j

Static Type Safety for DApps without JavaScript

DApps, starting professionally…

You might not be aware, but despite its similarities to JavaScript, Solidity is actually a statically, strongly typed language, more similar to Java than to JavaScript.

solidity
static type check in browsersolidity

…and ending in frontend-chaos

Sadly, for a long time, there has only be one interface to Ethereum nodes, web3.js (besides JSON/RPC), which is, as its name implies, written in JavaScript.

Though providing this API in a web-native language is really a brilliant idea in terms of fast development, seperation of concerns and ease of use, it is a nightmare for professional, multi-developer, multi-year, enterprise products.

You may not agree with me here, but as there are currently no 10 year old 1.000.000 LoC enterprise projects in node.js/JavaScript out there, you should at least consider that such projects are nearly impossible to maintain with a dynamically, weakly typed language like JavaScript (JS).

So, we have this situation, where JS defines the lowest common denominator (dynamically, weakly typed

JavaScript_1 (2)

when we really would like to have this situation, where Java (C#, Haskell) defines the lowest common denominator (statically, strongly typed)

JavaScript_2 (2)

Removing chaos

The problem is was, that up to now only web3.js existed. However, today there is also a web3.py (which is Python and therefore at least strongly typed, but still dynamically) and, brandnew, web3j.

With the latter, we can easily model the call chain above, where we only use statically, strongly typed Java and omit JavaScript altogether. Welcome to hassle-free integration into existing Java/JEE-environments without workarounds. Finally: using the Ethereum Blockchain with Java.

If you want to actually get deeper and use Java with no RPC at all, you can also switch to EthereumJ, which is a Ethereum Node implemented in Java, like Eth (C++), Geth (Go), PyEthApp (Python) or Parity (Rust). It is crucial to understand the difference between web3j and EthereumJ. If you just want to use some Ethereum Node from a Java application, web3j is your choice, you are limited to the Web DApp API then, which should be enough for all “Ethereum user” use cases.

We will not explain in detail how to use web3j, it should be familiar to any Java developer how this library can be used just by adding Maven-dependencies to your project.

Fixing the front-end

We could stop here, since using JavaScript for the frontend is not really problematic and a common use today.

However, if you use JavaScript in your frontend, it might really make more sense to stay with web3.js. So, we want to go further: how are we going to create the GUI if we want to have no JavaScript at all?

This is just a PoC, but if you think of any other client to the Ethereum Blockchain other than a web site (let’s say: Batchjobs, Web Services, Message Queues, Databases, other proprietary software with Java adapters (there are some!)), this should make sense to you – you really wouldn’t want to use them from web3.js (hopefully).

Using templates: Thymeleaf and Spring Boot for slim enterprisy software

We will do a step-by-step guide for creating a No-JS-Dapp. Even without any Java experience, you will be able to follow without problems. Java is not that complicated anymore!

  • Get an infura.io account and key, so you don’t have to mess around with starting your own Ethereum node
  • Clone this repo: https://hellokoding.com/spring-boot-hello-world-example-with-thymeleaf/
  • Install Maven
  • Edit these files:

    pom.xml (add these dependency to section dependencies and add the repo, beware that web3j is a fast moving target, check for new versions)

    <dependency>
    <groupId>org.web3j</groupId>
    <artifactId>core</artifactId>
    <version>0.2.0</version>
    </dependency>

 

<repositories>
<repository>
<id>oss.jfrog.org</id>
<url>http://dl.bintray.com/web3j/maven</url>
</repository>
</repositories>

src/main/resources/templates/hello.html (change name to balance.html)

<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.thymeleaf.org">
<head>
<meta charset="UTF-8"/>
<title>Your Static Strongly Typed Wallet</title>
</head>
<body>
<p th:text="'The balance of account ' + ${address} + ' is ' + ${balance}" />
</body>
</html>

src/main/java/com/hellokoding/springboot/HelloController.java (change name to EthereumController.java)

@Controller
public class EthereumController {

@RequestMapping("/balance")
public String balance(Model model, @RequestParam(value="address", required=false, defaultValue="0xe1f0a3D696031E1F8ae7823581BB38C600aFF2BE") String address) throws IOException {
Web3j web3 = Web3j.build(new HttpService("https://consensysnet.infura.io/{YOUR_INFURA_KEY}"));
EthGetBalance web3ClientVersion = web3.ethGetBalance(address, DefaultBlockParameter.valueOf("latest")).send();
String balance = web3ClientVersion.getBalance().toString();
model.addAttribute("address", address);
model.addAttribute("balance", balance);
return "balance";
}

}

…that’s it. Start with mvn spring-boot:run

If you encounter an connection/handshake error, you may have to import the infura certificate into your local Java keystore (I didn’t have to)

$JAVA_HOME/Contents/Home/jre/bin/keytool -import -noprompt -trustcacerts -alias morden.infura.io -file ~/Downloads/morden.infura.io -keystore $JAVA_HOME/Contents/Home/jre/lib/security/cacerts -storepass changeit

Look Ma! Displaying the wallet balance with no JavaScript!

You can call the spring-boot web application with http://localhost:8080/balance (then the defined default argument is used) or with your address (in the consensys testnet) as parameter address= 

walletOf course, you can change the Ethereum net like you want in file EthereumController to morden or mainnet, just read the welcome mail from infura.io. Or you can just use a local Ethereum node like geth with RPC enabled (geth –rpc) and http://localhost:8545 as the constructor for HttpService of the Web3j-Factory in EthereumController.

Have fun, with or without JavaScript!

Create your own Oracle with Ethereum Studio and oraclize.it

This is a follow-up post to Gambling with Oracles and describes the technical part for the implementation of an Oracle with oraclize.it

People told us last time that we had not really created an binary option. This might be true and to not confuse the reader and also to introduce just another cool use case and link to current events, we are going to bet on the winner of the EURO 2016 soccer championship.

As you see from the posting date, the championship is already finished and the winner is well known, it is Portugal.

So we can just create a static Oracle to make things easier, we will define the Oracle call in our betting contract and on calling update(), the winner will be determined, which is the static json {“winner”: “portugal”}.

You can call bet(winner) with exactly 1 ETH and on calling claim() you will get nothing or (loseBets + winBets) / (numberOfParticipantsWhoBetOnWinner), eg. 3+2/2 = 2.5

BettingOracle1

Prerequisites

You will need to sign up for the ether.camp Studio and check out the corresponding github project in the terminal, follow the instructions in the README.

Starting the Oracle

After starting the web server, you can access the “Oracle” with https://yourid.by.ether.camp:8080/web/euro2016winner.json which redirects to the file web/eu2016winner.json in your project.

You can test the data retrieval in the Studio, just choose the Oraclize tab, Test query, “URL” and put json(…).winner around the URL.

oracle1

That’s it, by clicking “Test” you should get the result “portugal”. So now the Oracle Smart Contract can call your Oracle datasource and submit the result via callback to your contract.

oraclizeObviously, a real Oracle should be a little more dynamic, but for testing purposes this Oracle is enough. You can change the answer by stopping the web server, changing the file manually and restarting the server.

To avoid duplicating information here, please consult the documentation for the ether.camp Studio IDE for actual usage of the sandbox. Remember to give enough value (at least 0.1 Ether in Wei: 100000000000000000) for the update() call, but also 1 Ether in Wei (1000000000000000000) for the bet()-calls, otherwise it will not work.

Deploying the betting contract to Morden

The contract.sol contains the complete contract, it imports some helper classes and the oraclize-API, which, depending on the deployed environment, is instantiated with the corresponding contract address.

In the README you will see the local environment sample, but also the Morden example. To deploy the contract to the testnet Morden, you will have to get Ether first, which is really simple in Morden

Note: you have to set the networkID to Morden in the constructor of the contract before deploying:

 function FinalWinner() {
oraclize_setNetwork(networkID_morden);
}

send2morden

After deploying, you can click on the Transaction-Hash in the Transaction-Tab, which automatically opens the deployed contract in morden.ether.camp.

To be able to invoke the contract in ether.camp, you will have to upload the sources of the contract, however, there is a problem with inline assembly at the time of writing this post. Therefore we added a special file (for-upload-to-ether-camp-morden-as-source-only.sol-not.executable) for uploading in the root of the github-project, this contract is just for this purpose and is not executable.

In Morden, each call should be done with value 1 Ether, the bet-calls will throw; otherwise and the update call needs some for the call of Oraclize, which costs Ether for each call (consult pricing on oraclize.it).

Distributing Business Processes using Finite State Machines in the Blockchain

Having disctinct, even competing organisations with a common goal and no shared technical infrastructure represent a perfect use case pattern for blockchain usage.

The quest for the holy business use case

As many blockchain enthusiasts we are constantly searching for use cases for establishing Distributed Ledger Technology (DLT) or even real blockchain technology in enterprises as solutions for common business problems.

21666276190_40647d8327_z
[Business *]

This turns out to be difficult, since each single aspect of this technology is already solved by several – established and well known – products. 

In this post we will present a use case, or even a use case pattern, which we think is ubiquitous and can best be solved by blockchain technology.

For these type of use cases the other technologies, even though being more mature, seem themselves like workarounds for the natural technological solution, which is blockchain-based.

Our definition is the use case pattern is:

Several distinct, competing organisations have a common goal and don't share technical infrastructure.

distinct, even competing organisations

Distinct organisations are not related to each other and therefore have no existing technical or organisational processes. To pursue a common goal, these processes would have to be established first, which is costly and time-consuming.

5882566863_ce0ccf745d_z
[Competing teams *]

The second aspect, competition, implies that there is no mutual trust. Obviously, this aspect is a crucial one. Blockchain technology might even make sense if just the other aspects of the pattern match, but it only is a perfect fit if this aspect is important, since the blockchain itself is inherently trust-less.

common goal

This is the fundamental requirement, if there is no common goal, there is no need to establish any collaboration. It is important to note, that the common goal is most likely not related to the core competence of the organisations, but to some crosscutting concerns, which have to be addressed by all organisations, but are just cost factors.

no existing shared technical infrastructure

To put it contrary: if there already exists a shared technical infrastructure, also organisational processes exist to establish new technology. Given this, there are several great technology products which implement each aspect of the blockchain technology (namely distributed data storage, immutability, security), and most likely even more performant, more mature, with less maintenance and evident responsibilities.

Ok, this is quite abstract and complex, let’s find an easy example.

…want a securities account with your new account?

There is one universal retail bank, FriendlyBank, which wants people to open accounts. And there are three deposit banks, FriendlyDeposit, EasyDeposit and NiceDeposit. There is also the Regulator, who must be able to audit all transactions between the parties.

Requirement

common goal: make money

The common goal of universal and deposit banks is to open accounts. The universal bank can be intermediary to deposit account opening, so you can open a securities account at one of the three deposit banks with your new account at FriendlyBank. This way, it is a win-win-situation for both parties, the intermediary gets commissions, the deposit banks get new customers.

FriendlyBank and FriendlyDeposit are related to each other, they have the same ownership structure, but are separated entities. Since the deposit banks are in competition with each other, the other deposit banks besides FriendlyBank want to be sure that not very valuable customers are transferred to FriendlyDeposit or the deposit account opening is “accidentally forgotten”, if not FriendlyDeposit is chosen.
Since the parties do not really trust each other, a trust-less (ie. no trust necessary) solution is needed.

the wonderful world of finance IT today

As we said, the depicted scenario exists quite often. Also, technical “solutions” for these scenarios exist.

They look similar to this one:

Flatfilesolution_crop

We wan’t go in detail what all this means, but to become trust-less, as well as assure that the data is in sync in each organisation, a lot has to be implemented and still, shortcomings of these solutions are quite common:

  • reconciliation processes are always necessary, but still, due to missing transaction support in flat file exchange, which is almost always chosen as the “simplest” integration pattern, errors occur.
  • adapters have to be built for each party, so there will soon exist a many-to-many problem.
  • special adapters have to be built for regulators.
  • crosscutting concerns have to be implemented: security, authentication, auditing, transaction support

Compared to a natural solution in the blockchain

Below is the blockchain solution, it fits naturally and implements all requirements.bcsample

Abstracting from the use case

Why is this solution matching so well, what is the pattern “behind”?

Different actors change the state of a business process on some event in a transactional way.

2645030_8e2b2269
[Old cigarette machine *]

You certainly know this pattern, it is describing a finite state machine, more exactly a distributed, secured by cryptoeconomics finite state machine, which events are transactions.

After all, the blockchain itself is a state transition system, it somes quite naturally to implement a simple state machine on the Ethereum EVM, it even is mentioned as a common pattern in the Solidity docs.

 

Approaching the finite state machine

How would the organisations collaborate? This could look like this informally:

 

process_v1

or like this (more) formally:

Blockchainsolution_FSM

so we actually can built this state machine and its transitions really easy using the blockchain, here in pseudo-Solidity, derived from the common pattern in the Solidity docs:

contract AccountDepositStateMachine {
enum Stages { Init, AccountOpened, DepositOpened, DepositConfirmed, AccountConfirmed } Stages public stage = Stages.Init; modifier atStage(Stages _stage) { if (stage != _stage) throw; _ } function nextStage() internal { stage = Stages(uint(stage) + 1); } // Order of the modifiers matters here! function openAccount() atStage(Stages.Init)
transitionNext
{ if (msg.sender != FRIENDLY_BANK) throw; } function openDeposit() atStage(Stages.AccountOpened)
transitionNext
{
if (msg.sender != FRIENDLY_BANK) throw;
}
function confirmDeposit()
atStage(Stages.DepositOpened)
transitionNext
{
if (msg.sender != DEPOSIT_BANK) throw;
}
modifier transitionNext() { _ nextStage(); } }

Great, so that’s it, we can build any business process using the blockchain, QED.

Leaving the ivory tower

As you might guess, it is not that simple. Let’s review the actual process:

  1. Account is opened 
  2. Deposit Account is opened
  3. Deposit Confirmed
  4. Account Confirmed

Ever saw an IT-process in real life? Exactly, it’s not like this, not even close. Let’s see a more real life example:

  1. Account is opened
  2. An Email is send to whoever feels responsible
  3. The Product Owner’s Excel has to be updated
  4. The new reporting engine must have its data updated, it is runnig with MongoDB, which has to be updated
  5. Revision wants to have it’s auditing data in their Oracle DB (must use 2PC)
  6. … (ok, you got it already…)

None of the tasks in bold red can be accomplished from the Ethereum blockchain, since calls from “inside” to “outside” are prohibited. Let’s hope this will never change, the separation of “inside” and “outside” is essential for the stability and security of the Ethereum blockchain

We could use Oracles for this, but it would be a kind of usage which is highly inappropriate for this type of external information retrieval.

Teaser: “Mastering the Flow: Blending Reality with the Blockchain”

This post is way too long already, so here comes a teaser for two posts next to come: “Mastering the Flow: Blending Reality with the Blockchain”

The idea is really simple: let’s just recentralize the business process (we are in eager anticipation for the comments to come…)

But we think this can make sense. Look at the sample above, illustrated as a flow:

nodered

The flow was created with Node-RED, an excellent and highly undervalued tool of IBM Emerging Technologies for flows in IoT. It could be very easily adapted to Ethereum with smart contract access by usage of web3.js, which itself can be integrated in node.js, which is the basis of Node-RED, hence the name.

We make the following assertion:

More than 80% of all use cases can be realized by a centralized business process engine as a first layer, eg. implemented in Node-RED, and a state machine implemented in the blockchain as a second layer

The business process engine is the glue between the transactional blockchain state transitions and the secondary business processes.

What do you think of it? Please let us know in the comments, we really appreciate feedback, positive or negative.

And stay tuned for the next episode, we will get to the nitty-gritty there.

images:
[Business] Business by Christophe BENOIT under CC
[Competing teams] Competing teams by Denis De Mesmaeker under CC
[Old cigarette machine] Old cigarette machine by Walter Baxter under CC

Gambling with Oracles

[casino kurhaus*]
[casino kurhaus*]

 

How to create a provably-fair high risk financial product with live data feeds provided by oracles.

Introducing Binary Options

This time, we gonna gamble. We will construct a financial product, more specifcally: a binary option, European style, cash-or-nothing.

[binary*]

Binary options are “a type of option in which the payoff is structured to be either a fixed amount of compensation if the option expires in the money, or nothing at all if the option expires out of the money” [1], therefore cash-or-nothing. European style means the option “can only be exercised at the end of its life, at its maturity” [2].

This is a high risk product which is “most likely traded over the Internet on platforms existing outside of regulations” [1], so “the customer is betting against the broker, who is acting as a bucket shop.” [3]

In general, this means that “because these platforms operate outside of regulations, investors are at greater risk of fraud” [1] and “manipulation of price data to cause customers to lose is common” [1].

BinaryOption (3)So, in contrast to common regulated financial products we have a really comprehensible option, which is mostly valued as high-risk, because there is a real clash of interests if the broker also evaluates the stock against the strike price and thus is highly motivated to manipulate this evaluation.

Smart Contracts to the Rescue

So, can smart contracts help here? You bet! The blockchain is trust-less, so you just don’t have to trust anyone, be it broker or bucket shop or both, but you can just prove if everything’s ok.

If a smart contract guarantees you that you will get 195% if you predict correctly and 0% if not, this rule has to be implemented in the contract.

Meeting the Oracle

Given that we can trust the contract, it should be easy to implement a binary option like this:

After deploying the contract, you have 1 hour to invest Ether on calls or puts on the contracts conditions (eg. DAX is above the strike price in 1 hour). After the 1 hour offering period, it's rien ne va plus, no purchase of calls or puts are possible. At this point of time, the current value of the DAX is stored. After exactly 1 hour the DAX is evaluated against the stored DAX value.
  • if the value is above or equal to the stored value, all calls are returned 195% of their investment, all puts lose all invested money
  • if the value is below the stored value, all puts are returned 195% of their investment, all calls lose all invested money

Sounds simple? It is!

[oracle*]
[oracle*]

You can easily prove that the contract actually implements the described rules, but there are two problems: how does the contract know the DAX spot? Contracts in Ethereum intentionally cannot connect to the outside world, therefore the outside world has to call the contract and provide the necessary data. Who tells the contract the exact time it has to retrieve the DAX spot? There must be some callback mechanism, since Ethereum smart contracts are not able to call themselves, have a Daemon, Threading etc.

Let’s gonna oraclize.it

There is a startup solving this problems in a elegant way: oraclize.it

You should look up and understand how oraclize.it works and helps us to solve our problems. In short, callbacks are used by oraclize.it and you can determine the exact data feed of the data which oraclize.it will provide at a certain point of time to our contract.BinaryOptionOrale (1)Here the two problems are addressed:

  1. The contract has to provide a callback for the oracle to call. The data feed is configured when the callback is provided, as well as the time interval.
  2. The oracle facilitates a scheduling to call the provided smart contract callback at the configured time interval.

Both solutions require the user and the contract owner to actually trust the “trusted datasource” (the contract’s user can decide if he trusts this datasource) and the oracle itself to have a working scheduling and not to manipulate the data. In contrast to the “broker” above, the oracle has no interest in manipulating the data.

However, oraclize.it not only offers a service, but even more defines a protocol, so there will most likely be more than one oracle to choose from to offer redundancy in the future.

Building Binary Options in Ethereum Studio

…in the next post.

Realizing how long this post already is, we decided to split the tutorial and the theoretical part. The tutorial will follow soon! Please let us know in the comments if you’d like to have this tutorial and what you expect of it.

Meanwhile, try out this ether.camp tutorial which describes the integration of oraclize.it into Ethereum Studio.

images:
[casino kurhaus] Kurhaus Casino by fr4dd under CC
[binary] binary by christine592 under CC
[
oracle] Oracle by Bob Doran under CC

icons:
icons made by Freepik from www.flaticon.com 

KW 21.16 – Weekly Blockchainers Recap

From Blockchain to Distributed Ledger Technology (DLT) If you follow the news, you might have realized that the term DLT is currently more and more used to describe what we usually describe with “the Blockchain”. Even though the definitions are more or less subjective, a good comparison between these technologies is here. In really easy terms, the DLT is the Blockchain without the “chain of blocks”, which also means without the security and immutability of the blockchain. However, as with private chains vs. public chains, this might exactly be what you need for your use case. So, keep in mind that Bitcoins are not the Blockchain and that the Blockchain is more than DLR and all of these technologies fit perfectly to their use cases, which are certainly not the same. It helps to know about the definitions to get more information about the complete topic, since DLT is not as spammed (aka SEOed) as Blockchain, a Google searchs gets you more valuable results like the BaFin evaluation and the really well made UK Government Science report. Even more about the Blockchain Fed up with all the small bits and pieces about Blockchain technology on all those news sites? Then read a book, we recommend “The Blockchain” from leanpub and hope that it will be completed soon. More into videos? Then watch primavera in her talk about Skynet or Freenet or her more current talks.  

Ethereum Development Tools – an Evaluation Matrix

Developing for the Ethereum World Computer – Revisited

When comparing development methods for and in the Ethereum space, it becomes obvious how brilliant the people involved actually are: as a developer, you have several choices of great development tools and environments, even though the development process for Dapps is quite difficult: it involves different “layers” (frontend with HTML/CSS/JavaScript and backend with the Ethereum blockchain), different languages (JavaScript for frontend dev, Solidity for backend dev), different contexts (public/private/proprietary blockchains).

Taking these different contexts into account, evaluating the different development environments is only possible by identifying the different aspects and value them according to the own preferred usage.

Aspects of Dapp Development

Ethereum-docs-intro (1)

  • Solidity Development / Solidity Environment
    User can edit Solidity content in a text editor, there should be at least syntax highlighting.
  • JavaScript/Web Environment
    User can edit JavaScript, HTML and CSS in a text editor and gets different levels of support like syntax highlighting, code completion, etc.
  • IDE
    User can not only edit code (Solidity and JavaScript), but can also compile, check into a versioning system, debug and deploy to some operating environment.
  • Versioning
    Versioning is supported, eg. by enabling the user to check edited content into a versioning system, show diffs, apply patches, etc.
  • Collaboration
    Modification of code by multiple users is supported, at minimum support of different versions of the code is possible to avoid conflicts.
  • Deployment
    Working code can be deployed to some environment which enables the user and other users to use the working program (a sandbox or real blockchain and a web application server for JavaScript like node.js and static content like HTML and CSS files)

A first Categorization and a Teaser

We will summarize in short the different aspects which are supported in these development environment: Solidity Browser / Ethereum Wallet, Truffle and Ethereum Studio.

This is also a teaser for the upcoming blog posts, which will explain the environments in detail.

Solidity Browser / Ethereum Wallet

In terms of architectural styles, both environments represent minimalism. They are slim and handy, easy to use and fast to learn, but lack some functionality if real development “in the large” is necessary.

Ethereum-docs-browser
Solidity Browser, only Solidity Development, no Environment, ie. not even save of file is possible

Ethereum-docs-wallet
Ethereum Wallet (Mist), only Solidity Environment, no save of file possible, but integrated compiler

Truffle (also: Embark, Dapple)

Truffle satisfies all aspects which a usual client application can offer, therefore you can think of Gothik, it is mighty, almost lavish, and well structured. If you come from web development in JavaScript with node.js, who should look no further, this environment is for you (at least if you don’t need support for online development, versioning and multi user directly in your environment, but use external tools like Git and testrpc for these aspects).

Ethereum-docs-truffle
Truffle, also Embark and Dapple: web development environments, usefully extended with Solidity support

Ethereum Studio, the all-in-one-solution

The Ethereum Studio, in our opinion, resembles Deconstructivism. Why’s that? Because here all aforementioned aspects of software development are taken apart and are reconstructed to fit perfectly to Dapp development. This is an all-in-one-solution which we can really recommend, with two limitation: you have to agree to a uncertain pricing model (it just doesn’t exist right now, you can still test the product) and a “closed” environment, which works seemless and smooth, but expects you to let in to this tool and the development process.

But no other tool lets you test your code this easy with manual and even unit testing built into the environment.

Ethereum-docs-studio
The all-in-one-solution, steep learning curve, but all you will ever need in multi user, versionized, unit tested Dapp development

Stay tuned for the detailed explanations of these great development tools.