Watt's What

About

Sat, 20 Aug 2016 00:00:00 +0000

Strive for simple when you can, be pragmatic when you can’t.

Welcome to my (Nicki Watt’s) personal blog and website where all views, thoughts and ideas expressed here are my own.

I am a techie who enjoys problem solving, making stuff work, and exploring technologies and approaches which can enable this. I relish engaging in a good architectural design or challenge, and statements like “it can’t be done” simply spur me on rather than deter me! Less is often more, and am thus also partial to a good proverb- they tend to convey the essence of what you want to say in a language everyone can understand. My own proverbial motto is “Strive for simple when you can, be pragmatic when you can’t”

Professional info

I currently work as a lead consultant at OpenCredo helping companies embrace, build and deliver their solutions, with a current focus on Cloud, infrastructure as code, continuous integration and delivery and various NoSQL technologies. Along with my colleagues at OpenCredo I have also authored the book Neo4j In Action.

Have questions or suggestions? Feel free to ask me on Twitter.

Thanks for reading!

Kubernetes on Mesos on Google Cloud – Version Alignment

Mon, 02 Feb 2015 00:00:00 +0000

I was recently trying to follow the instructions in the tutorial by Mesosphere detailing how to run Kubernetes on Mesos on the Google Cloud Platform. The tutorial (At time of writing) did not quite work for me out the box and I had to make a few adjustments to get it working. This simply details a few of the extra steps I needed to take to get this setup working for me. This should hopefully mean that those who find themselves in a similar situation, can avoid running into my same issue upfront, and thus get to a working version faster! The Mesosphere team do update the tutorials periodically, usually when new releases occur, so the official tutorial may well be updated in due course in which case this entry will probably be redundant. But until then, for those who want to try this out now, I hope this can help!

Underlying issue

The underlying cause of all my problems was centred around the fact that I was mixing Mesos versions in various places. In the pre-requisite list in the tutorial, you need to have a Mesosphere cluster on Google Compute Engine. (BTW Google provides you with $300 worth of free trial Google compute resources to play with). You however, are not in control of what version of Mesos is deployed for you by Mesosphere, and this is where I came unstuck. I was blindly following the instructions (centred around 0.21.0 at the time of writing) whilst Mesosphere had provisioned me a 0.21.1 cluster. Apparently the kubernetes-mesos framework is currently using the native Mesos binding compiled against a specific version of Mesos. So if the versions are out of sync, as mine were then issues may arise. My main issue was that my pods were simply not able to run on my minions. They appeared to stay in the “Pending” state forever.

Steps to resolve

The basic resolution strategy is to: Adjust everything in the tutorial to align with whatever Mesos version Mesosphere ultimately provisions for you in the Google cloud.

Practically this is what you need to do:

Follow the instructions for setting up your Mesosphere cluster on Google Compute Engine as per normal.
Build the Kubernetes-Mesos executables- When you reach this step then:
- After executing the first (dpkg) command, take note of the Mesos version which has been provisioned for you as you will use this in subsequent steps. For example running the step below I can see the main Mesos version is 0.21.1
```
$ dpkg -l | grep -e mesos
ii mesos 0.21.1-1.2.debian77 amd64 Cluster resource manager ...
```
- In addition to simply cloning the git repo in the next command, you need to ensure you are using a version of the kubernetes-mesos code compatible with the Mesos version you are actually running against. There is actually a note about this tutorial in the readme in the kubernetes-mesos Github repo stating that this tutorial is not necessarily always kept in sync with the repo, and it is suggested that tutorial users stick with the 0.3.x releases to ensure everything still works. In my case v0.3.2 was the latest kubernetes-mesos release compatible with Mesos 0.21.1
```
git clone https://github.com/mesosphere/kubernetes-mesos k8sm
cd k8sm
git checkout v0.3.2
```
- The final step under this section is to run the kubernetes-mesos docker image. Once again, modify the image name to use a docker image compatible with the version of Mesos you are running. Note the use of build-mesos-0.21.1-compat docker image, instead of build-mesos-0.21.0-compat
```
mkdir -p bin && sudo docker run --rm \ 
-v $(pwd)/bin:/target -v $(pwd)/k8sm:/snapshot \ 
jdef/kubernetes-mesos:build-mesos-0.21.1-compat
```
Run the Example Guestbook App- When you get here:
- Provided you have checked out the correct git tag in the previous step, the example code should be compatible with your Mesos version. Note however that on the tutorial itself, the link to the example instructions is to an older v0.2.2 version, so if you follow the link, you may be following incorrect instructions.

Thats it! Everything else should work just fine!

The SQL of trust in a NoSQL world

Tue, 20 Jan 2015 00:00:00 +0000

From the movie Meet the Parents, the “circle of trust” quote is amongst other things, used to infer that you can only really be part of the family once you are inside “the circle of trust”. NoSQL vendors are desperate to attain a greater de facto acceptance as heavyweights and household names to be relied on in the broader software development community, and especially in the lucrative enterprise space. To this end, there appears to be a courting of a few of the battle hardened supporters from the RDBMS world with the lure of “some sort of familiarity” to their current known world in order to help try and break into this circle of trust. One such area of familiarity is SQL, and it is interesting to observe the different degrees to which some NoSQL vendors have gone in adapting their respective querying languages to cater for this fact. Now let me be clear that there is absolutely nothing wrong with this approach – provided of course it does not completely skew and dumb down the original benefits of the underlying datastore in question.

It should come as no surprise that when it comes to data, the vast majority of developers today (yes even in 2015) still think in structured RDBMS terms, and when it comes to querying data – SQL is probably the most prominent query language still out there. This simply stems from the fact that NoSQL databases are still relatively young compared to their RDBMS counterparts which have been around for a lot longer.

N*SQL (No/NewSQL) databases are by definition, going to be very different to RDBMS and querying the data from them will more than likely be different as well. So is there any common ground here, specifically is it possible for a structured query language like SQL to easily translate or morph to being used in an unstructured context as well? Should we even trust this approach? SQL vs NoSQL? Time for some examples …

Setting the scene – an RDBMS example

Suppose we are modelling an ordering system where we have ORDERS, consisting of ORDERLINES linked to PRODUCTS as shown in the diagram below.

Lets also assume we want to ask two questions of our data relating to the purchases of Raspberry Pi’s (product ID P4). We could write some SQL queries as shown below to query this data:

Now not every NoSQL database is going to have some form of SQL sympathy built into its query language, and there are simply too many of them out there for us to compare them all. I shall therefore pick two that I have actively worked with to use as examples for my purposes. Lets see how Cassandra and Neo4j deal with the modelling and querying of this data with their respective query languages.

Cassandra and CQL

Cassandra falls into the column family NoSQL camp with its “semi structured” format and is presently heavily pushing CQL (The Cassandra Query Language) as the recommended way of interacting with it’s underlying datastore.

“Cassandra Query Language (CQL) is the default and primary interface into the Cassandra DBMS. Using CQL is similar to using SQL (Structured Query Language). CQL and SQL share the same abstract idea of a table constructed of columns and rows. The main difference from SQL is that Cassandra does not support joins or subqueries” - www.datastax.com/documentation/ cassandra/2.1/cassandra/cql.html

For this example I am going to work backwards, i.e. first show you the CQL query statements which result in the same answers to the SQL queries used in the RDBMS example above, and then pick up on a few related discussion points.

-- Num Raspberry Pi's bought by each customer:
SELECT * FROM CustPurchByProduct
WHERE productId = 'P4'

-- Total number of distinct users who bought Raspberry Pi's  
SELECT COUNT(*) FROM CustPurchByProduct
WHERE productId = 'P4'

Syntax looks quite similar to SQL, albeit the query seems scarily simple. So where are those SQL joins, what’s going on? As per the official philosophy, Cassandra does not really do joins, rather you are actively encouraged to denormalize your data (i.e. don’t be afraid to write data multiple times if necessary) to answer different questions. In this case, you might decide to model your domain something as follows:

Each table is an island

OK, Cassandra does not do joins, and in this case we can see that a whole separate denormalized table (aka column-family in Cassandra) CustPurchByProduct was created to answer the two core questions posed.

So in this case, the comparison is simple, the concept of SQL joins simply does not exist. You should never really expect to see any SQL join like syntax in CQL statements, and by implication, no foreign key references either – it simply does not compute in Cassandra.

The 3 legged table

Leaving the joins and foreign keys aside, via the CQL shell (similar to SQL shell type programs), we gain some tabular insight into our data. The CQL CREATE and SELECT statements relating to Orders and Products column families seem to map very similarly to their RDBMS counterpart – tables, so far so good.

Lets however take a closer look at OrderLines. Firstly, what is interesting to note is that even though the CQL SELECT queries appear to give the impression of bringing back four traditional RDBMS style rows with three columns (orderId, productId and qty), under the covers the story is quite different. The internal representation of the OrderLines column family actually has two Cassandra partitions (think rows) with a variable number of columns. You can think of a partition (and in fact this is how the underlying storage engine would represent it) as a very wide row in an RDBMS; one that has no fixed set of columns, but rather the columns are dynamically created based on the table definition, and the actual data stored. In our example P1:qty, P2:qty and P4:qty are the effective column names, with the only value stored as a table cell being the qty itself. This is an example of Cassandra ensuring that CQL can use the full potential of the underlying storage engine – one of Cassandra’s key strengths, whilst at the same time, giving users an easier way of defining and visualising the data, closer to the way in which a more traditional RDBMS developer might expect to see/reason about it.

This RDBMS column mismatch phenomenon is also highlighted when looking at the CQL CREATE statement for OrderLines and CustPurchByProduct. Although they look very similar to SQL CREATE statements, they do not translate into a 1-to-1 mapping of physical column names in the traditional RDBMS. In this case a pivot like transformation is performed on the fly, both when the data is initially presented/inserted, as well as queried via CQL. This way CQL ensures that the transformation between what is logically defined and presented to the user, and the physical representation, is kept in sync for both the CREATE and SELECT statements, so ultimately, the two move together.

SideNote: The OrderLines column family is actually a little thin. In reality, this column family would more than likely have a few extra columns, something along the lines of what is shown in the diagram below. Again, this is because there are no joins and Cassandra actively encourages you to denormalize your data. For the purposes of this blog entry however, it has deliberately been kept thin to allow for an easier comparison with the alternative options. CustPurchByProduct probably falls into the same category too.

To be, or not to be - that is the question

Another thing which tends to confuse newcomers to Cassandra, is that a PRIMARY KEY definition in CQL, is not treated in the same way as a that for an RDBMS.

Firstly lets examine this from the perspective of the data model and storage perspective. If you look closely at the CQL CREATE statement for the CustPurchByProduct column family, and what this physically translates to under the covers, you will see that the first part of the SQL looking compound primary key (productId) was used as the row key (also referred to as a partition key in Cassandra). The remaining column customerId (in Cassandra terminology also referred to as a clustering key) actually got its value‘s absorbed into the various column headings itself (C1:cnt, C2:cnt)! Not quite what you might have been expecting.

Secondly, you would expect a primary key to ensure that you only have one unique record in the database for a given key, can we at least rely on this? The answer is Yes but… Cassandra will indeed ensure that there is only one reference to a combination of a primary key, however Cassandra tends to treat both its INSERT and UPDATE statements as ‘Add or update’. In other words, having already inserted a record with primary key X, you could call that exact same statement again and it would not throw a “Primary Key Already Exists” type exception, rather, it would simply update the existing row. Likewise you could call UPDATE with data which does not exist and it would merely insert it the first time and update it thereafter.

The point here is that a PRIMARY KEY as used in CQL is not an exact 1-to-1 logical mapping to its counterpart used in SQL.

Cassandra wrap up …

There is much more which could be said about the specifics of Cassandra data modelling, however as the point of this particular blog post is more to highlight how NoSQL query languages translate/compare to SQL, and not what the merits and demerits of particular stores and data modelling approaches are, we shall leave this example here. Perhaps a dedicated blog on specific DB comparisons could be next on the cards. In any case, we now move on to look at Neo4j.

Neo4j and Cypher

Neo4j is a graph database, where its sweet spot is getting answers about highly connected data. With a graph, you tend not so much to query vast swathes of its total structure, as you do start somewhere and then traverse it, gathering info along the way. Cypher has evolved as Neo4j’s humane graph query language.

“Cypher is inspired by a number of different approaches and builds upon established practices for expressive querying. Most of the keywords like WHERE and ORDER BY are inspired by SQL. Pattern matching borrows expression approaches from SPARQL…. Cypher borrows it structure from SQL — queries are built up using various clauses.” – http://neo4j.com/docs/stable/cypher-introduction.html

Once again lets work backwards from the Cypher query and creation statements which roughly equate to the RDBMS counterparts originally shown and then discuss a few points.

// Num Raspberry Pi's bought by each customer
MATCH  (o:Order)-[r:HAS_ORDER_ITEM]-(p:Product)
WHERE   p.prodId = 'P4'
RETURN  o.custId, sum(r.qty)

// Total number of distinct users who bought Raspberry Pi's  
MATCH  (o:Order)-[:HAS_ORDER_ITEM]-(p:Product)
WHERE   p.prodId = 'P4'
RETURN  count( distinct o.custId )

Hhhm, some aspects looks similar to SQL, and others very much not. The MATCH statement is obviously quite unique. The WHERE clause gives us a slightly warmer feeling of familiarity and RETURN looks (ish) like a SQL SELECT but slapped onto the end. There also appears to be some familiar looking count and distinct functions. So how does Neo4j model and create data for this scenario, and what does it look like under the covers?

As with CQL, Neo4j also provides a shell whereby we can try out our queries and statements to gain some tabular insight into our data. We shall also be using this as shown in the diagrams below.

I should at this stage point out that Neo4j was a wonderful graph visualisation tool via the Neo4j console available for seeing visual “connections” however as we are comparing SQL, so we shall also use its shell program.

Eat on your lap – no table required

At its heart, Neo4j is a property based graph DB, consisting simply of nodes and relationships. Nodes of the same type can be grouped together using something called a “Label” – roughly equated to the way a table groups the same types of records in an RDBMS, except that there is no traditional schema per so – i.e. no restrictions on which properties (columns) the node (table ish) can contain. So in our example, although at present all of the Order nodes have three properties, this does not restrict us from adding another property, say a poNumber to new Order nodes created in the future – without affecting any of the previous ones. Neo4j is officially schema-optional as there are some forms of constraints (e.g. creating a genuinely unique key) and indexes which can be applied to your model. However, as far as defining for example that a User node will always have a name and an age attribute – that level of schema definition cannot be enforced out of the box at present.

Join the dots to make the connection …

Unlike Cassandra which goes for a denormalized approach, graphs are at the other end of the NoSQL spectrum. Being all about connected data, nodes are indeed able to be joined to each other and this is done via the first class Neo4j relationship concept. When it comes to querying or traversing the graph however, this joining is expressed rather as a pattern match. For example the generic “find me all orders with at least one product (orderItem)” is expressed in the ascii art syntax of MATCH (:Order)-[:HAS_ORDER_ITEM]-(:Product). In general with a graph you would typically look to constrain your query a bit more i.e. “find me all orders which contain the P4 product ” is expressed as:

MATCH (o:Order)-[:HAS_ORDER_ITEM]-(p:Product)
WHERE p.prodId = 'P4' return o.

Now this does look fundamentally different to any SQL you’ve seen. However, if you can get your head around the fact that the MATCH bits are “roughly” the join part, then the remaining syntax is more easily digested with things like WHERE clauses, count and distinct functions etc.

In an effort to make the Cypher language more “whiteboard friendly”, as well as try and expose and convey the underlying native graph structure inherent in the queries been undertaken, the Neo4j Cypher language designers explicitly opted for making pattern matching central, rather than opting for the more familiar, yet potentially more verbose standard SQL join like syntax. In order to help SQL thinkers jump this divide however, Neo4j provides a dedicated From SQL to Cypher section on the official Neo4j website.

Interestingly enough, OrientDB, which also operates in the graph (and document) database space have taken a slightly different stance. Disclaimer: I do not have commercial experience with OrientDB. From their website:

“… Instead of inventing “Yet Another Query Language”, we started from the widely used and well understood SQL. We then extended it to support more complex graph concepts like Trees and Graphs. Why SQL? SQL ubiquitous in the database developer world, it is familiar …” - [http://www.orientechnologies.com/docs/2.0/orientdb.wiki/Tutorial-SQL.html]http://www.orientechnologies.com/docs/2.0/orientdb.wiki/Tutorial-SQL.html

However in OrientDB’s version of SQL, as with Cypher, the choice was taken not to make use of the traditional SQL join syntax, instead opting for using a dot (.) notation[1] to navigate its relationships (referred to as links) instead. So even though its starting point is SQL, it too has chosen to deviate from using the standard SQL join syntax in order to better exploit the underlying data structure.

Conclusion

Via an analogy …

So you are a ‘box drawing expert’. You have however been using standard pens all your life for drawing boxes, and now you have been asked to draw a box using a new pen, which you are told, is specifically aimed at drawing boxes in a ‘new and shiny way’. So armed with your previous box drawing knowledge and experience, you set out on your task. Well how hard can it be you say. You know how to use a pen, you even have a trusted old favourite which you bring out for special occasions. This new pen probably just does some fancy highlighting or something, but at the end of the day if it looks like a pen, and writes like a pen, it’s probably still just a pen. You finish your box drawing exercise, and well, it does sort of look like the boxes you usually draw, maybe a little bit more messy than you were expecting, but hey, this must be how all the new kids are doing it, and it’s probably just what ‘new and shiny box world’ looks like.

Problem is, the pen you were given was actually a 3D doodling pen[2]. Yes it’s a sort of a pen, however it is not really meant for drawing in a 2D world, but rather a 3D one. Because however you were comfortable in applying your 2D way of thinking to the task at hand, you simply started by drawing your box in this mindset. If however, you understand that actually this 3D pen is capable of more than simply mimicking what can be done in a 2D world, your box may instead have turned out like this:

Via conventional words …

So yes, it appears some NoSQL vendors have “somewhat” adapted their querying languages, to a larger or lesser degree, to be more sympathetic towards some SQL like syntax and concepts. The fundamental differences in underlying data structures however, mean that this is never going to be a perfect fit and it is worth bearing this in mind when trying to use them.

Query languages and approaches to try and help people adapt/transition into new ways of thinking by comparing or making aspects of the query language seem familiar can be very helpful in some instances. However, as has been demonstrated in this blog, this can sometimes be confusing, especially if your background means you expect the semantics behind statements, and not merely the usage of common terminology, to operate in the same way as SQL. I would therefore urge caution in simply skimming the surface when engaging in these technologies and only thinking in your current (SQL) mindset, as you may fundamentally miss out on leveraging the best parts of your underlying datastore if you do, or even worse, design your model incorrectly in the first place as a result.

Can a language or database be all things to all people at the same time? Probably not. When it comes to trying to get to grips with a new kind of datastore, I feel that trying to gain some mechanical sympathy for your new datastore can go a long way to helping you understand it, as well as effectively being able to use it, regardless of what query language is being used; be that SQL like, a hybrid, or something completely different.

[1] OrientDB uses the “dot (.) notation” to navigate LINKS.” – https://github.com/orientechnologies/orientdb/wiki/SQL

[2] The 3D doodling pen is actually a real thing, and well it looks pretty impressive to be honest. Here is a link in case you are interested: http://www.makerstechnology.com/3doodler-3d-creativity-pen

Life’s a journey, learn from the ride

Mon, 12 Jan 2015 00:00:00 +0000

Over my years of consulting and working with people, building, and developing stuff, I have inevitably built up some views over what I feel works, what doesn’t, and, more often than not, come to understand that things don’t quite work in the way I thought they did!

Initially centred around tech and development, but evolving to include general life lessons as well, I decided to share some of my realisations in the form of some short and sweet, value-based resolution statements for this inaugural post. If you are reading this and you are not from a techie background, then some of the statements may not make much sense – apologies for this but this did actually start as a technical post first! Feel free to skip to the end if you simply want to see the statements, or carry on if you are interested in the background on how I arrived at them.

One of my 2015 new years resolutions was to stop procrastinating and actually get my blog site up and running. Better late than never, and tick for me – this one is now done :) However, I sometimes find that new years resolutions tend to focus on a single narrow particular outcome. One of the lessons I have learned is that more often than not, the value is in the journey itself, where hidden gems (not Ruby) and underlying principles are there to be mined and extracted if you look carefully – often yielding more fruit than the single end outcome itself.

So, below is a short and sweet, A-Z listing of some of my value resolution statements laid out in a manner not dissimilar to that of the original agile manifesto. As per the spirit (but not necessarily the implementations) of the agile manifesto, it is not to say that things on the right are not important, but rather, through time and experience, I see evidence of those on the left bearing better fruit and value in the long run. And of course in the spirit of agile, I reserve the right to change my mind as I learn and discover more! So without further ado:

Life’s a journey, and thus far on my ride, I have come to value:

Automated Testing over Ad-hoc testing
Building Bridges over Building empires
Challenging over Conforming
- There is a reason why square pegs do not fit in round holes
Data Consistency (eventually) over Data Cleansing
- Think before you drink your data, as it may have bad consequences for you later down the line otherwise
Engaging over Emailing
Face-to-face friends over Facebook friends
Guiding over Guarding
- Being taught how to navigate a situation is immensly more beneficial than being prevented from ever making a mistake and not being able to learn from it
Holistic teams over Hand-offs
Integrity over Intellect
Jumping in over Jumping out
- When the going gets tough, the tough will jump in, rather than out, to help
Knowing how something works under the covers over Knowing how to merely operate it
Load Balancing over Load shedding
- Inspired by a recent trip to South Africa, but the general principle remains – do something before its too late and drastic action needs to be taken!
Mentoring over Managing
Nurture over Nature
- Relying on raw talent alone will only take you so far, but hard graft and access to a good mentor can often take you further
Offer over Obligation
- Offer yourself or your services before it is asked or demanded of you
Proposed ideas over (waiting for) Perfectly thought out ones
- Ideas are seldom right when you first start out, but its generally only by sharing and bouncing them off others that they become well formed
Questioning over Quietly trying to work out what’s going on by yourself for hours
Re-use over Re-invent
Simplicity over Smart and super cool, but complex
- My own personal motto is “Strive for simple when you can, be pragmatic when you can’t”
Teaching over Telling
Uniqueness over Uniformity (when innovating) BUT
Uniformity over Uniqueness (when maintaining)
Viral over Virus
- Make an impact, but just ensure it’s a positive one!
Wisdom over Wealth
X-Factor over X
- Because sometimes it takes something uniquely different, or someone with that unique perspective, to be able to shift you from your default position
You as-you-are over Yet-another-clone
ZX Spectrum over all other 1980’s personal home computers
- Z is not an easy letter to find comparisons for, and well, as the ZX Spectrum was my first computer which helped usher in my tech beginnings, and, valuing challenging over conforming to the status quo, I break from my own self imposed format and give it a special mention!