Enjoyed reading your book, the concepts are well explained. Thanks for sharing it to the open community.

Chinmay

That said, I believe that any application where partition tolerance is more important than consistency is a good fit for NoSQL. Partition tolerance seems to be poorly understood in the field. I took the Git example because many developers know what “distributed” means in context of Git. I could have taken any other distributed, partition tolerant revision control system for that purpose. Source code repositories are a particularly good fit because they hardly every reach consistency anyway.

The question—what is more important, partition tolerance or consistency—depends on the data. Huge social networks have to cope with tons of data that has very little value. Strict consistency doesn’t pay off for that. The damage caused by conflicts is little compared to the costs to establish strict consistency. That argument is, however, nonexistent for small sites because consistency is easy to achieve there.

I also mentioned BLOBs in the article. As a software architect, I have been involved in many discussions where to store low value binary data like user uploads. I remember a meeting where the DBA smashed the proposal to use BLOBs for vast amounts of user data by proclaiming that “BLOBs have no business in my database”. BLOBs have, quite often, little value—even if connected to high value relational data. From that angle, I believe that some NoSQL systems make a great distributed BLOB store which can coexist with a relational database.

Scaling writes is subject to Brewer’s CAP Theorem—take two out of three. Sharding and similar methods bypass it by not distributing the data at large—that is, only a small subset of the nodes is responsible to maintain a particular data sub-set.

I feel, however, that the need to scale out is constantly decreasing for most applications. I have observed a multi-national banking system over the past decade. It was initially running on a huge two node active-active cluster to distribute load. A few years later, it was moved to a hot-standby cluster—just one node active. Today, it’s being migrated to a virtualized server running other databases on the same hardware. “Scale-out” is not the trend—virtualization is, at least in enterprise environments. Huge social sites being the obvious exception.

Other than performance, could you provide examples of applications that are more suitable to NoSQL semantics? SQL has all other advantages, like persistence frameworks, consistency, and other tools. NoSQL solutions like cassandra are target for scaling writes. Other than sharding how do you think to scale writes on a RDMS?

Well, I made another observation. In fact, everybody knows that database design is important and must be done carefully. There are many books covering that in more or less detail. I do not say that database design is not important, but I say it’s usually done carefully anyway because everybody knows it’s important.

What I find at client sites is that developers are not aware how to index properly and how to write queries that can benefit from indexing. The DBAs, on the other hand, know about indexing but don’t have the deep domain knowledge to know how the data is queried.

Nobody ignores schema design but indexing is almost always ignored until it’s too late. Adding some more or less random indexes might improve the situation, but it is exactly what I refer to as “improper” indexing. Indexing without a plan. In fact, my position is that indexing must be designed with the same care as the schema.

I pretty much agree with your statements about NoSQL.

I’m sorry, but that is simply not true. The main reason you run into performance problems with relational databases is because the data model has problems. Indices are more like “inherent opportunities” to speed things up here and there, but the main performance comes from understanding the data, understanding the access patterns and understanding how the SQL engine will calculate a plan. And then designing a relational model that will balance these things out.

Designing a relational model that performs well is hard. Which is why there are so few database professionals available who seem able to do this well. It requires deep understanding of both the domain that needs to be modeled as well as the technology it will run on.

Just tweaking indices is going to work for a small subset of problems, but the real performance gains are made during the design of the relational model you will use. And unfortunately, they do not teach this in school.

Also, the whole NoSQL vs SQL debate is artificial. There is no real debate: they address entirely different classes of problems. It can be boiled down to this: SQL is about consistency and flexibility during querying at the expense of scalability and performance.

NoSQL is about “performance at scale”and flexibility when writing data at the expense of consistency and querying flexibility. Note that I said “performance at scale”, because many of the NoSQL databases do not have particularly impressive performance for small datasets.

Sacrificing absolute consistency is hard, but as it turns out, for a lot of “new” problems lack of consistency is less of a problem than long response times. The “new” problems here are online systems with massive numbers of users. For some classes of companies you can feel this directly. Most online banks are still pretty slow. Sites like Amazon, on the other hand are quite snappy given that they have a lot more web traffic than any bank. (And when it comes to online commerce, every millisecond counts).

A system can be said to be scalable when the cost of increasing its size is sublinear with respect to the dimension you need to scale. Since the relational model is inherently expensive to apply in a distributed manner, it is relatively easy to show that you cannot get sublinear cost for arbitrary scale along any dimension.

However, with certain sacrifices you can get sublinear cost. For instance by breaking the relational model somewhat and partitioning the data into independent instances that have no dependencies on other instances.

Note that when we say “cost” we mostly talk in terms of latency and processing power. Not dollars. Although it will end up costing dollars.

SQL has its place and NoSQL has its place, but it is important to understand that they address different types of problems. I have worked at companies that have naively used SQL databases for NoSQL type problems and vice versa. It is unhelpful that people keep comparing them directly instead of trying to develop and disseminate the kind of knowledge needed to reason about this.

Also it doesn’t help that Stonebreaker et al, to draw attention to themselves, muddy the waters and confuse the issues by planting the idea that NoSQL is somehow the antithesis of SQL. In fact the label “NoSQL” has been incredibly unhelpful because it suggests that there is a problem with SQL and that NoSQL is the magic solution. This is, at best, naive. And unfortunately leads people to get hung up on the wrong ideas.

final static String seqsql_new = “SELECT DISTINCT av.* FROM ab4p.alloc_review_vw av WHERE alloc_id = 2267294″;

static int
executeTest(CallableStatement ps, PreparedStatement ps_new)
throws SQLException
{
int cnt = 0;
//ps.setInt(1, 100000);
// ps.execute(); — this is run from outside
//Get Resultset from SP
//ResultSet rs = (ResultSet) ps.getObject(1);

//Get Result set from qury to Temp Table produced by SP.
ResultSet rs = ps_new.executeQuery();

while (rs.next()) {
++cnt;
}
rs.close();
return cnt;
}

Results using PreparedStatement for Fetching data after Proc run
—————————————————————-

Time for ResultSet.next() loop
FetchSize first run second run
1 42784 42951
10 11673 11863
5000 8037 7645
10000 8373 7985

Note: I had to run the test with different data item this time.
Again thanks a lot for all your comments.

I like to respond fast, if I can.

As I mentioned in my previous comment, I have never tested that thing with REFCURSORS—however, might be an interesting topic. Unfortunately I do not have the timeto look into that in much detail.

Regardless of that, the figures you sent made me doubt that your problem is related to the network latency—at least not within a LAN. A typical network latency in a LAN is 0.xx (zero-dot-something) milliseconds. Let’s assume it is 1 ms, that would sum up to about 3.2 seconds for all the server round trips—about 15% of the overall execution time. Unless you are doing this on a WAN or you have a really slow LAN, I’d guess that the performance bottleneck is the procedure itself, not the data transfer to the client.

However, your observation is interesting. I hope I can have a closer look at it some other time.

static int
executeTest(CallableStatement ps)
throws SQLException
{
int cnt = 0;
//ps.setInt(1, 100000);
ps.execute();
ResultSet rs = (ResultSet) ps.getObject(1);
while (rs.next()) {
++cnt;
}
rs.close();
return cnt;
}
**********************
Result1-

getFetchSize(): 10
getFetchSize(): 1
31850 rows fetched in 3187 server round trips and 21446 ms
*********************
Result 2-

getFetchSize(): 10
getFetchSize(): 1000
31850 rows fetched in 3187 server round trips and 20927 ms