Amazon Internet of Things platform Just in time

Screen Shot 2015-10-09 at 5.15.48 PM

Yesterday Amazon introduced its IoT services for AWS. Bravo! Kodus to Andy Jassy and AWS leadership to identify the opportunity and responding timely with a decent product. Other cloud services may start catching up in a couple of years when they realized they missed the boat.

Four years ago, I worked for a company which was a world leader in control systems. For many of us IoT is a new concept, but it is just commoditization of control systems. There is no new concept in IoT. At the time I predicted that IoT will disrupt the control system market (I was laughed at). A market full of PLCs and Industrial computers with ridiculous programming languages and terrible data analysis products. It was obvious to me that Arduino’s and IoT will one day consume this lucrative market. I even wrote a report for my Surviving Disruptive Technology course I took at Coursera. This hasn’t happened yet but it will, sooner or later. My prediction is that it takes a few years for Amazons of the world to provide sensor network tools that are orders of magnitude nicer, and more powerful than their old sluggish enterprise counterparts. The only thing that keeps the old enterprise in place is the ability to talk to industrial motors and high voltage, three phase systems plus the complex sales structure of enterprise, yet it won’t be long before someone cracks this code too and the open IoT floods the enterprise and with it takes out a lot of legacy businesses.

In 2012, I created an IoT cloud service prototype, with similar features like the Amazon’s IoT service launched yesterday and a grand vision on analytics, I started to pitch the idea all the way from CTO to my direct management. I gave a demo to the CTO (and everybody else down the leadership ladder) and showed how these distributed sensors can communicate in real-time securely and how such platform unleashes the potential for machine learning and analytics. It was well received on the top but at the bottom, it got stuck with the mid-level management. They decided that we should pursue the idea but there were endless meetings with sparse timing and eventually no decision was made until I left there to work at Microsoft. Maybe if I was a lot more patient we would have built an IoT platform before Amazon. Same old story about large traditional corporates (I encourage you to read about the Kodak downfall and take the Coursera course, it is a very interesting contemporary tale). I wanted to build this platform which I called Wombat, as an open-source project but at the time my afternoons and weekends were busy with writing my thesis and journal papers.

I am glad that that Amazon is doing this eventually and I am sure they will kill it! Go Amazon!

P.S. Here is the link to my presentation deck I used those days to pitch my idea. If it helped you in anyway let me know.


How not to fail like Ashley Madison! or a guide to hackproof application (part 1)

Ashley madison failure


The irony of a company so built around secrecy and gigabytes of profile informations roaming around internet is hilarious!

Even Ashley Madison’s advertisement is inferring: “We are here to keep your secret”

So how did such a scandalous failure happen? I don’t know but in this post I will try to describe a (near) hack-proof architecture with the hope that it does not happen to your application.

Whatever your company does, if there is customer data somewhere, there is a risk it could leak. Security should not be taken lightly. A data breach can crush a successful company to dust in one day! And in the cloud era, you should make sure not only your application is secure, but also a breach in cloud provider does not jeopardize your data.

On the other hand, security stays in the way of productivity. Making it hard for hackers to access your data will make it hard for your own developers and analysts to do the same. The valuable resources that need to go to building the product has to go to the security.

So, “What is the minimum we should do to make our application hack-proof?”

In these series of blogs I will address this question, and we will discuss an architecture to particularly prevent these hacks:

  1. Separating data that uniquely identifies the customer from the rest of data and securing this data separately.
  2. Encrypting data such that even the breach of data and some encryption keys does not cause a problem.
  3. Hot to make sure only authenticated services from specific hosts can access your sensitive data.
  4. In the worst case scenario that a hacker has access to your application and your main credentials, and can imitate your service to get data, how to prevent large scale data breach and catch them early?

Yet another Amazonian response to NYT article, except this one has a solution

Disclaimer: This short text is not about Amazon, it is about the whole tech industry in united states.

My grandpa used to say: If you want treasure, you need work for it! Hard.

This quote, is as true for companies as is for individuals. If a company wants to do extra-ordinary things that others deem impossible, they need to work really, really, really hard and smart. Delivering a gummy bear in the same day for free and still being profitable, having the best customer service in the planet, or slashing the price on cloud and still make profit while competitors can’t even get close to that price, are incredible achievements and require a lot of work.

And people enjoy working hard when they are solving hard problems. Any person who has hiked for a peak on a weekend knows this.

So, is there a way to work mediocre and still get great results? I don’t think so.

BUT, there remains the problem of empathy. Putting a cancer survival on ‘performance improvement plan’ is (… fill up the blank), even as an isolated case.

We don’t know how to formalize empathy? When it comes to soft values, we can only hope our managers will do the right thing.

Hoping that managers who are already under pressure to deliver will do the right thing in the most complex human situation is not the right approach.

Solution is simple:
Have the policy to let employees with hardship (families with new-born child, people struggling with disease, or other issues) have long paid time offs, and give them the chance to get their acts together and help them come back to a team that is fit for their new reality.

But is this going to make the company bankrupt?
No, it is only going to improve the brand, retain talent, and improve employee moral, if “I know company is there for me in hard times, I would be there for the company in hard times too.”

This is not a new concept. There is even successful, insanely profitable business model around empathy (can you guess, yes it is insurance). A large company can easily afford to guarantee empathy in workplace, and even make profit from it.

Will this ever happen:
Yes, many companies already do such practices. Amazon’s Career Choice Program is a good example of how empathy can become a policy. As the demand for a healthier workplace grows among high skilled workers, companies will adopt better practices.

Billion dollar idea:
I can see (and hope) that there would be a billion dollar venture in future that empowers every small and large company to give their new moms long vacations. They can makes money from charging a monthly premium (from employer or as salary sacrifice) like insurance companies.

Club W! The new business model of “Scam the customers, Scam the venture capitalist”

After reading an article that mentioned “Club W“, a (so far) successful startup that sells wine “personalized” to the customer’s taste with great! price, I decided to give them a shot!

There is something great with the idea of personalized wine! With so many variety of wines and the sheer amount of wine being sold everyday which produces valuable data, this idea makes total sense. I would love to get affordable high quality wine, personalized to my taste!

However, there is a problem with my “Club W” experience. A simple one; their wine is neither personalized, nor high quality or good price! Basically they look more like a scam than a real business. They have built a website that asks you some basic questions, which at best can help them segment customers as red/white dry/sweet drinkers – not a real personalization, and get you to accept a monthly subscription by giving up a one-time discount on first purchase. A monthly subscription that you can only cancel if you call their customer service in business hours. Instead of coming up with great products with good prices, they focus on physiological tricks to get customers money. An approach that is “guaranteed” to fail on low-margin highly competitive retail market.

I think “Scam the customers, scam the investors” is a business model commonly seen in U.S. due to the large pool of potential first (and only) customers. I personally see this over, and over with companies that have terrible products and terrible service, but big marketing budget. Another example is “ADT Home Secutiry” with a two star ranking on Yelp – to get you thinking.

I developed a recipe for a successful “Scam the customer, scam the investor” business model. Don’t hold me accountable on this, but it has worked for many companies.

1 – Come up with a great idea that everybody likes, but you can not deliver on (such as affordable home security or cheap quality wine tuned to customers taste)

2 – Somehow! raise a bunch of money and build a very good marketing team.

3 – While others are trying hard to win the customers with great products, focus on getting new customers, forget about customer retention and referral, just rely on traditional / online marketing channels. You will have a good go until your marketing budget finishes.

4 – Use the sales figure growth to convince the venture capitalist that you have a great growing business.

5 – Hope that you will figure out a real sustainable business model, once you raised several hundred million dollars.

Quick Javascript encryption

Don’t use this for production:

Why interviews are failing at large and small companies?

Brian Acton, founder of WhatsApp, which was acquired by Facebook for $16B (B really B), stated in tweeter that he was rejected by Facebook in 2009!

So, you don’t hire someone for $1X0K range, then you buy him for $16B? 160,000 times more! Of course Acton would not be as valuable as an engineers, as he is as a successful entrepreneur, but my point is: If this isn’t a sign that interview processes are a failure – even at Facebook – then what is it?

How to generate human readable IDs?

All of us have been given a record number or an ID of some sort for further reference. For example, the payment receipt, or a reservation number.

It is great to have a way to uniquely identify something, but don’t you hate spelling those numbers over the phone or writing meaningless numbers on paper stickers?

What if the IDs where not so boring? Actually, they don’t have to. Feeling tired of this disconnection between databases and humanity, I suggest to translate unique ids, uniquely to funny interesting, (well yet meaningless) sentences.

I have created a shell script on my mac that generates 16, 24, 32, 40, 48, and 64 bit unique IDs.

Please note that there is no need to store these IDs as long strings in databse, basically every number can easily be translated to a sentence with a discrete and simple mapping:

Assume we have 2 maps of “adj” for Adjectives, and “non” for Nouns. Use the following rule to generate the human readable IDs.

For example a 32 bit ID: adj  non “with” adj non

In above every adj, or non translates to 8 bit. We just need to keep two list of 256 words. One for adj and one for non.

Below is some examples for generating random IDs:

Read the rest of this entry »


Autodart library introduced at dartwatch.

Sweet spot for buying a used Honda Civic!

Sweet spot for buying a used Honda Civic!

Sweet spot for buying a used Honda Civic! Buy 2009, sell in 2 years! Would depreciate only $30/month.

Hidden Markov Models, Bayesian Statistics, and Quantum Computing together can estimate inventories

In production systems, organisations that produce things, like mines, factories, refineries, etc. a whole system exists dubbed inventory. The inventory is a graph of materials and objects moving through the systems and changing face. For example, in a coal mine, dirt is digged out of pits in units called lots. The lot moves around the mine, gets merged, crushed, dried, and cleaned. The lot moves from the ground to trucks and process points, ports, railways, etc. where most likely it changes shape and characteristics. Same thing happens in manufacturing, chemicals, water procurements, and many other industries.

It is crucial and critical for the business to know their inventory at each stage. They usually have manual or automatic processes to measure characteristics of every part of the inventory. For example weightometers are fitted all over the system which generate data, and they may manually keep record of what happens on the ground. The data generated is almost always limited, of low quality and is not directly a reflection of the reality. In fact, computer science should come in and start filling the gaps, improve data quality and estimate inventories much better than the most skilled human can do by looking at the inventories and excel sheets. In this post I want to reflect on how three techniques, two from AI, and one from Quantum Computing can provide a set of tools to just do that.

Hidden Markov Models (HMM) are used to extract state machine in temporal data. For example, if we track Joe movements, the fact that Joe  usually gets out of the kitchen, go to the Mary’s desk for five minutes and come back to his own desk, is a Markov chain of states hidden in the data. In inventories HMM can be very well used to extract sate machines. I refer to this paper for a good tutorial on HMMs. In the inventory settings, HMMs can be used to extract movements of materials and objects and the change in their characteristics. For example coal goes to the process point A and becomes dry and the dry coal goes to the crusher and becomes grained, and then goes to another process points and becomes pure by loosing ash. Or again coal goes into a process point and ash comes out. Such MMs can be extracted by looking at the data.

Bayesian statistics is simply the implications of the Bayes theorem: P(A|B) = P(B|A)P(A)/P(B). In computer science a mixture of Bayesian statistics and HMM is used for improving data quality in time series (read a good paper here). It is mainly done by forming correlation tables. For example, Joe always goes from Mary’s office to his own, so the correlation of Joe location between Mary’s office and Joe’s office is high, but he never goes from Mary’s office to the Jack’s office, so the correlation of Joe’s location between Mary’s and Jack’s office is zero. Using this knowledge, if we are in doubt if Joe is currently in his own office or in Jack’s office, we can check to see where has he been before. If he has been in mary’s office beforehand, we can estimate that Joe is probably in his own office. In literature this is called probabilistic easing. The probabilistic easing, however, does not produce an exact value of the location and characteristics of objects, instead it creates a set of probable location with their probability. For example in a coal mine, we may estimate that the lot X is out of the crusher with 20% probability and it is merged with lot Y with 80% probability. This dramatically extends our knowledge of the inventory, however, it is probabilistic, which is ok, because life and reality are probabilistic.

Quantum mechanics may seem irrelevant to a coal mine. However, Quantum mechanics is extremely beautiful, and I love to find a use for some of the formulas and methods coming out of Quantum computing in unrelated fields. There is one specific method in Quantum computing that might be relevant to our settings. It can become very useful to improve our estimations for eased data generated from the previous steps.

The problem of the eased data is of course that it is probabilistic (its both curse and blessing). In reality, inventories get measured occasionally and this measurements infer valuable information that are ignored in the easing approaches. For example, the easing method reports a specific inventory of coal to be with 60% probability high in Ash. The engineers on the ground sample some of the coal in this inventory and send it to lab, and the result comes that the coal has exactly 20% ash. The easing approach uses this fact merely to improve the easing from this point forward. However, quantum algorithms tell us that this fact contains much more information.

In quantum mechanics there is a state which is called entanglement. Entanglement is a very unintuitive phenomenon but we know that it happens. It means once two atoms are entangled, their electron are in the same superpositions, no matter how far apart those atoms are from each other. By the way, superposition is nothing new in this post. Remember we said that easing method says that Joe is with 10% probability in Jack’s office and 90% probability in his own office, this is called super position in quantum mechanics. Just replace Joe with your favorite electron. Now that we figure out the similarity with eased state of inventory and quantum state of matter, we can utilize the work in quantum computing to improve our knowledge of the state of inventory.

In quantum computing, once entangled atoms pass through quantum gates, their state changes, and we can compute their superpositions, which is something similar to Bayesian easing. However, in quantum computing, once we measure the sate of one of the atoms, we break the engagement, and the atom collapses to the new state which we measured. This is very similar to when we send the coal samples to lab and realize the exact state of our coal. In fact, the eased probabilistic state collapses to one value that has come out of lab. In quantum computing however, we use this data to calculate new superposition for all other atoms. For example, if the measurement of the output of the first quantum gate comes out as 001, we know that other gates can not be in an state which is inconsistent with the first gate being in 001. We can re-estimate the superposition of other atoms with some linear algebra. We can do exactly the same in inventory. For example, we can apply this new knowledge to back track and re-estimate the eased sate of all other bits of coal in the system but limit easing only to states that are consistent with the new knowledge out of lab.

This may sound a bit superficial and hard to implement, but remember that the Bayesian easing based on HMM is not my invention and people have been using it successfully. Adding the quantum flavour is not hard even because it is also implemented and used by many others in the computer science field. I don’t see a feasibility problem here. There might be a marketing problem but if you are interesting in implementing above, and you are research student, feel free to drop me an email.