My unrepresentative observation is that NoSQL is often taken for performance reasons. Probably because SQL performance problems are an everyday experience. NoSQL, on the other hand, is known to “scale well”. However, performance is often a bad reason to choose NoSQL—especially if the side effects, like eventual consistency, are poorly understood.

Most SQL performance problems result out of improper indexing. Again, my unrepresentative observation. But I believe it so strongly that I am writing a book about SQL indexing. But indexing is not only a SQL topic, it applies to NoSQL as well. MongoDB, for example, claims to support “Index[es] on any attribute, just like you’re used to“. Seems like there is no way around proper indexing—no matter if you use SQL or NoSQL. The latest release of my book, “Response Time, Throughput and Horizontal Scalability“, describes that in more detail.

Performance is—almost always—the wrong reason for NoSQL. Still there are cases where NoSQL is a better fit than SQL. As an example, I’ll describe a NoSQL system that I use almost every day. It is the distributed revision control system Git. Wait! Git is not NoSQL? Well, let’s have a closer look.

Git doesn’t have an SQL front end

Git has specialized interfaces to interact with the repository. Either on the command line or integrated into an IDE. There isn’t anything that remotely compares to SQL or a relational model. I never missed it.

Git doesn’t use an SQL back-end

Honestly, if I would have to develop a revision control system, I wouldn’t take an SQL database as back-end. There is no benefit in putting BLOBs into a relational model and handling BLOBs all the time is just too awkward.

Git is distributed

That’s my favourite Git feature. Working offline is exactly what is meant by ‘partition tolerance’ in Brewer’s CAP Theorem. I can use all Git features without Internet connection. Others can, of course, still use the server if they can connect to it. Full functionality on either end. It is partition tolerant.

Conflicts happen anyway

If there is one thing we learned in the 25 years since Larry Wall introduced patch, it is that conflicts happen. No matter what. Software development has a very long “transaction time” and we are mostly using optimistic locking—conflicts are inevitable. But here comes the famous CAP Theorem again. If we cannot have consistency anyway, let’s focus on the other two CAP properties: availability and partition tolerance.

Acknowledging inconsistencies means to take care of methods and tools to find and resolve them. That involves the software (e.g., Git) as well as the user. But here comes one last unrepresentative observation from my side: most NoSQL users just ignore that. They assume that the system magically resolves contradicting writes automatically. It’s like using a CVS work flow with Git—it works for a while, but you’ll end up in trouble soon.

I’m not aware of a minimum feature set for NoSQL datastores—it’s therefore hard to tell if Git fulfils them or not. However, Git feels to me like using NoSQL for the right reason.

It’s about choosing the right tool for the job. But I can’t get rid of the feeling that NoSQL is too often taken for the wrong reasons—query response time, in particular. No doubt, NoSQL is a better fit for some applications. However, an index review would often solve the performance problems within a few days. SQL is no better than NoSQL, nor vice-versa. Because the question is not what’s better. The question is what is a better fit for a particular problem.

I believe my solution is worth a few extra words to explain it in more detail. Even though it’s a perfect fit for Use The Index, Luke it’s a little early to put it as an exercise there. It is, however, a very good complement to my previous article Analytic Top-N queries—so I put it here.

Although the problem was raised for a MySQL database, my solution applies to all databases that can properly optimize Top-N queries.

The original SQL statement in the question was like that:

select 1
  from T1
 where 'fxg87698x84' like concat (C1, '%')

T1.C1 is the column that holds the prefix patterns—one per row. Although a prefixed LIKE filter can use an index range scan, the problem is that it is the wrong way around: it’s not searching for a string that matches the pattern, it’s searching for a pattern that matches the string.

The query, as written, must check all the patterns against the string. E.g., by a full table scan or (fast) full index scan. However, it’s always a full scan. Can that be improved?

Let’s start step-by-step. The simplest case is that the exact input string is a pattern in the table. A SQL statement to check for the exact pattern is very simple:

select C1
  from T1
 where C1 = 'fxg87698x84'

The next case that the exact pattern doesn’t exist in the table, but a prefix pattern, that matches the input string, exists. That pattern must be shorter than the input string—otherwise it cannot match. Because we aim to solve the problem with an index, let’s imagine the patterns, as they would be stored in an index:

axt3
fxg
      <- place where 'fxg87698x84' would be
tru56

If the exact pattern doesn’t exist, the preceding index entry is the best possible match (precondition: no overlapping patterns exist). That’s because shorter strings are considered “smaller” when sorted. So, let’s extend the select to find the preceding record if the exact pattern is not in the table:

select C1
  from T1
 where C1 <= 'fxg87698x84'
 order by c1 desc
 limit 1

The less than or equals condition will match the exact pattern, if it exists, and all that precede it. The reverse ORDER BY clause makes sure that the index is traversed upwards. In conjunction with the where clause, it means that the tree traversal is done to find the input string, and the leaf node scan continues upwards from there. The LIMIT 1 clause is the MySQL way to make a Top-N query so that the leaf node scan aborts after the first record. Voilà, this statement will return the best candidate pattern (or none at all) by performing a very small index range scan.

The final case we need to take care of is that no pattern matches the input string. There are two sub-variants that can happen: (a) a potentially matching pattern would be the very first entry in the index. In that case the Top-N query will not return any row and we are done; (b) the Top-N query returns a pattern that is not a prefix for the input string. That can be handled by wrapping the Top-N query to filter the result through the original LIKE expression:

select 1
  from (
        select C1
          from T1
         where C1 <= 'fxg87698x84'
         order by C1 desc
         limit 1
       ) tmp
 where 'fxg87698x84' like concat (C1, '%')

Done.

Simple? With a good understanding of index fundamentals, it is simple! That’s why I am writing a Web-Book about indexing basics: Use The Index, Luke!. Funny enough, the basics are the same for all databases—we all put our pants on one leg at a time.

Closing Note

The precondition for all that is that there are no overlapping patterns in the table. E.g., the statement doesn’t work with the following patterns:

axt3
fxg
fxg1
      <- place where 'fxg87698x84' would be
tru56

In that case, the closest entry doesn’t match although there is a matching entry. However, the FXG entry matches everything that FXG1 can possibly match—the two patterns are overlapping.

Second Closing Note

The original problem posted on Stack Overflow mentioned that this lookup must be performed 1 million times—within half an hour. The author did not mention if that target was reached, nor if the process is single-threaded.

However, considering the overall problem, the most computing resource efficient solution would probably be to sort both sets—the patterns and the input strings—and implement a manual merge. But that’s probably much more effort to implement. The index solution is very efficient on human resources. Whatever is the best solution for the business is up to the company to decide.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

 

I started the book because I noticed that almost every existing book or online document on that topic is stuffed with plenty of information that is not relevant to developers.

Although a certain know-how of the database’s internals is required to get the best performance, Use The Index, Luke keeps that information at a minimum and presents it from a developers perspective.

The book is supplemented by the Ask Use The Index, Luke community where further questions of that matter are asked, discussed and answered.

The first chapters of Use The Index, Luke are already online. The remaining parts will be published on a bi- or tri-weekly basis. Follow the RSS feed not to miss the new chapters.

I hope you enjoy reading the existing chapters in the meanwhile.

The article starts with a general introduction into Top-N queries, applies that technique to analytic queries and explains the case where I miss an optimization. But is is really worth all that efforts? The article concludes with my answer to that question.

Please find the CREATE and INSERT statements at the end of the article.

Top-N Queries

Top-N queries are queries for the first N rows according to a specific sort order—e.g., the first three rows like that:

select * from (
  select start_key, group_key, junk
    from demo
   where start_key = 'St'
   order by group_key
)
 where rownum <= 3;

That’s well known and very straight. However, the interesting part is performance—as usual. A naïve implementation executes the inner SQL first—that is, fetch and sort all the matching records—before limiting the result set to the first three rows. In absence of a useful index, that is really happening:

START_KEY  GROUP_KEY JUNK
---------- --------- ----------
St                 1 junk
St                 3 junk
St                10 junk

3 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 142682949

--------------------------------------------------------------
| Id | Operation               | Name | Rows  | Bytes | Cost |
--------------------------------------------------------------
|  0 | SELECT STATEMENT        |      |     3 |  1032 | 8240 |
|* 1 |  COUNT STOPKEY          |      |       |       |      |
|  2 |   VIEW                  |      |   370 |   124K| 8240 |
|* 3 |    SORT ORDER BY STOPKEY|      |   370 | 76960 | 8240 |
|* 4 |     TABLE ACCESS FULL   | DEMO |   370 | 76960 | 8239 |
--------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter(ROWNUM<=3)
   3 - filter(ROWNUM<=3)
   4 - filter("START_KEY"='St')

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
      30370  consistent gets
      30365  physical reads
          0  redo size
        998  bytes sent via SQL*Net to client
        419  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
          3  rows processed

A full table scan is performed—more on that in a few seconds—to retrieve all the rows that match the where clause; about 370 according to the optimizers estimate. The next sorts the entire result set. Finally the limit is applied—the COUNT STOPKEY step—and the number of rows is reduced to three.

The performance problem of this query is obviously the full table scan. Let’s create an index to make it go away:

create index demo_idx on demo (start_key);
exec dbms_stats.gather_index_stats(null, 'DEMO_IDX');

That’s much better:

START_KEY  GROUP_KEY JUNK
---------- --------- ----------
St                 1 junk
St                 3 junk
St                10 junk

3 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1129354520

------------------------------------------------------------------------
| Id | Operation                      | Name     | Rows | Bytes | Cost |
------------------------------------------------------------------------
|  0 | SELECT STATEMENT               |          |    3 |  1032 |  372 |
|* 1 |  COUNT STOPKEY                 |          |      |       |      |
|  2 |   VIEW                         |          |  370 |   124K|  372 |
|* 3 |    SORT ORDER BY STOPKEY       |          |  370 | 76960 |  372 |
|  4 |     TABLE ACCESS BY INDEX ROWID| DEMO     |  370 | 76960 |  371 |
|* 5 |      INDEX RANGE SCAN          | DEMO_IDX |  370 |       |    3 |
------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter(ROWNUM<=3)
   3 - filter(ROWNUM<=3)
   5 - access("START_KEY"='St')

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
        360  consistent gets
        201  physical reads
          0  redo size
        998  bytes sent via SQL*Net to client
        419  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
          3  rows processed

You can see that the full table scan was replaced by an index lookup and the corresponding table access. The other steps remain unchanged.

However, this is still a bad execution plan because all matching records are fetched and sorted just to throw most of them away. The following index allows a much better execution plan:

drop index demo_idx;
create index demo_idx on demo (start_key, group_key);
exec dbms_stats.gather_index_stats(null, 'DEMO_IDX');

The new execution plan looks like this:

        ID START_KEY  GROUP_KEY
---------- ---------- ---------
    936196 St                 1
    232303 St                 3
    759212 St                10

3 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1891928015

-----------------------------------------------------------------------
| Id | Operation                     | Name     | Rows | Bytes | Cost |
-----------------------------------------------------------------------
|  0 | SELECT STATEMENT              |          |    3 |   465 |    7 |
|* 1 |  COUNT STOPKEY                |          |      |       |      |
|  2 |   VIEW                        |          |    3 |   465 |    7 |
|  3 |    TABLE ACCESS BY INDEX ROWID| DEMO     |    3 |    36 |    7 |
|* 4 |     INDEX RANGE SCAN          | DEMO_IDX |  370 |       |    3 |
-----------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter(ROWNUM<=3)
   4 - access("START_KEY"='St')

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          7  consistent gets
          0  physical reads
          0  redo size
        609  bytes sent via SQL*Net to client
        419  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          3  rows processed

Well, that is efficient. The sort operation has vanished at all because the index definition supports the ORDER BY clause. But even more powerful, the STOPKEY takes effect down to the index range scan. You can see the reduced number of table accesses in the plan. Although not visible in the execution plan, the index range scan is also aborted after fetching the first three records.

Well, that optimization is in the Oracle database for quite a while—at least since 8i I guess. After that preparation, I can demonstrate what 10R2 has to offer on top of that.

Analytic Top-N Queries

It is actually the very same story with a small extension: I don’t want to retrieve the first N rows, but all the rows where the group_key value is at it’s minimum for the respective start_key. A very straight solution is that:

select id, start_key, group_key
  from demo
 where start_key = 'St'
   and group_key = (select min(group_key)
                      from demo
                     where start_key = 'St'
                   );

That statement is perfectly legal—even performance wise:

        ID START_KEY  GROUP_KEY
---------- ---------- ---------
    936196 St                 1

1 row selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1142136980

------------------------------------------------------------------------
| Id | Operation                      | Name     | Rows | Bytes | Cost |
------------------------------------------------------------------------
|  0 | SELECT STATEMENT               |          |    1 |    12 |    8 |
|  1 |  TABLE ACCESS BY INDEX ROWID   | DEMO     |    1 |    12 |    5 |
|* 2 |   INDEX RANGE SCAN             | DEMO_IDX |    1 |       |    3 |
|  3 |    SORT AGGREGATE              |          |    1 |     7 |      |
|  4 |     FIRST ROW                  |          |    1 |     7 |    3 |
|* 5 |      INDEX RANGE SCAN (MIN/MAX)| DEMO_IDX |    1 |     7 |    3 |
------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("START_KEY"='St'
               AND "GROUP_KEY"= (SELECT MIN("GROUP_KEY") FROM
              "DEMO" "DEMO" WHERE "START_KEY"='St'))
   5 - access("START_KEY"='St')

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          8  consistent gets
          0  physical reads
          0  redo size
        550  bytes sent via SQL*Net to client
        419  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

The fist step is to fetch the smallest group_key. Because of the min/max optimization in combination with a well supporting index, the database doesn’t need to sort the data—it just picks the first record from the index which must be the smallest anyway. The second step is to perform a regular index lookup for the start_key and the group_key that was just retrieved from the sub-query.

Another possible implementation for that is to use an analytic function:

select * from (
  select id, start_key, group_key,
         dense_rank() OVER (order by group_key) rnk
    from demo
   where start_key = 'St'
)
 where rnk <= 1;

Do you recognize the pattern? It is very similar to the traditional Top-N query that was described at the beginning of this article. Instead of limiting on the rownum pseudocolumn we use an analytic function. The execution plan reveals the performance characteristic of that statement:

        ID START_KEY  GROUP_KEY        RNK
---------- ---------- --------- ----------
    936196 St                 1          1

1 row selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 3221234897

-----------------------------------------------------------------------
| Id | Operation                     | Name     | Rows | Bytes | Cost |
-----------------------------------------------------------------------
|  0 | SELECT STATEMENT              |          |  370 | 62160 |  374 |
|* 1 |  VIEW                         |          |  370 | 62160 |  374 |
|* 2 |   WINDOW NOSORT STOPKEY       |          |  370 |  4440 |  374 |
|  3 |    TABLE ACCESS BY INDEX ROWID| DEMO     |  370 |  4440 |  373 |
|* 4 |     INDEX RANGE SCAN          | DEMO_IDX |  370 |       |    3 |
-----------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("RNK"<=1)
   2 - filter(DENSE_RANK() OVER ( ORDER BY "GROUP_KEY")<=1)
   4 - access("START_KEY"='St')

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          6  consistent gets
          0  physical reads
          0  redo size
        610  bytes sent via SQL*Net to client
        419  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

What was the COUNT STOPKEY operation for the traditional Top-N query has become the WINDOW NOSORT STOPKEY operation for the analytical function. However, the expected number of rows is not known to the optimizer—any number of rows could have the lowest group_key value. Still the index range scan is aborted once the required rows have been fetched. On the one hand, the consistent gets are even better as with the sub-query statement. On the other hand, the cost value is higher. Whenever you use analytic functions, go for a benchmark to know the actual performance.

Let’s have some thoughts about this optimization. The database knows that the index order corresponds to the OVER (ORDER BY) clause and avoids the the sort operation. But even more impressive is that that it can abort the range scan when the first value that doesn’t match the rnk <= 1 expression is fetched. That is only possible because the dense_rank() function can not decrease if the rows are fetched in order of the OVER(ORDER BY) clause. That’s impressive, isn’t it?

Mass Top-N Queries

The next step towards the issue that made me writing this article is to make a mass Top-N query. With the previous statement as basis, it is actually quite simple; just remove the inner where clause to get the result for all start_key values and add a partition clause to make sure the rank is built individually for each start_key:

select * from (
  select start_key, group_key, junk,
         dense_rank() OVER (partition by start_key
                                order by group_key) rnk
    from demo
)
 where rnk <= 1;

Declaring the partition is required to make sure those start_keys that don’t have a group_key of one will still show up, with their lowest group_key value.

With that query, we have reached the end of the optimizers smartness—as of release 11r2. On the first sight, the plan is not surprising:

3260 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1766530486

--------------------------------------------------------------
| Id | Operation                | Name | Rows | Bytes | Cost |
--------------------------------------------------------------
|  0 | SELECT STATEMENT         |      | 1000K|   340M| 8239 |
|* 1 |  VIEW                    |      | 1000K|   340M| 8239 |
|* 2 |   WINDOW SORT PUSHED RANK|      | 1000K|   198M| 8239 |
|  3 |    TABLE ACCESS FULL     | DEMO | 1000K|   198M| 8239 |
--------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("RNK"<=1)
   2 - filter(DENSE_RANK() OVER ( PARTITION BY "START_KEY" ORDER BY
              "GROUP_KEY")<=1)

Statistics
----------------------------------------------------------
         22  recursive calls
         20  db block gets
      30370  consistent gets
      33163  physical reads
          0  redo size
      59215  bytes sent via SQL*Net to client
       2806  bytes received via SQL*Net from client
        219  SQL*Net roundtrips to/from client
          0  sorts (memory)
          1  sorts (disk)
       3260  rows processed

It’s a full table scan. However, a “mass” query performs a full table scan on good purpose—that did not call my attention. What did call my attention is the following:

select * from (
select * from (
   select start_key, group_key, junk,
          dense_rank() OVER (partition by start_key
                                 order by group_key) rnk
     from demo
) where rnk <= 1
) where start_key = 'St';

It is actually the individual Top-N query again. This time it is built on the basis of the mass Top-N query—that was set up as view. That way, a single database view can be used for any mass query as well as for individual Top-N queries—that’s a maintainability benefit. If the advanced magic to abort the index range scan is still working it would be extremely efficient as well. The execution plan proves the opposite:

START_KEY  GROUP_KEY JUNK              RNK
---------- --------- ---------- ----------
St                 1 junk                1

1 row selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1309566133

-----------------------------------------------------------------------
| Id | Operation                     | Name     | Rows | Bytes | Cost |
-----------------------------------------------------------------------
|  0 | SELECT STATEMENT              |          |  370 |   128K|  373 |
|* 1 |  VIEW                         |          |  370 |   128K|  373 |
|* 2 |   WINDOW NOSORT               |          |  370 | 76960 |  373 |
|  3 |    TABLE ACCESS BY INDEX ROWID| DEMO     |  370 | 76960 |  373 |
|* 4 |     INDEX RANGE SCAN          | DEMO_IDX |  370 |       |    3 |
-----------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("RNK"<=1)
   2 - filter(DENSE_RANK() OVER ( PARTITION BY "START_KEY" ORDER BY
              "GROUP_KEY")<=1)
   4 - access("START_KEY"='St')

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
        363  consistent gets
          0  physical reads
          0  redo size
        808  bytes sent via SQL*Net to client
        419  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

Although no sort is required, the STOPKEY has disappeared from the WINDOW NOSORT operation. That means that the full index range scan will be performed; for all 359 rows where start_key='St'. On top of that, the number of consistent gets is rather high. A closer look into the execution plan reveals that the entire row is fetched from the table before the filter on the analytic expression is applied. The junk column that is fetched from the table is not required for the evaluation of this predicate; it would be possible to fetch that column only for those rows that pass the filter.

The “premature table access” is the reason why the full table scan is more efficient for the mass query than a index full scan. Have a look into the (hinted) full index scan execution plan for the mass query:

3260 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 1402975529

------------------------------------------------------------------------
| Id | Operation                     | Name     | Rows | Bytes |  Cost |
------------------------------------------------------------------------
|  0 | SELECT STATEMENT              |          | 1000K|   340M| 1002K |
|* 1 |  VIEW                         |          | 1000K|   340M| 1002K |
|* 2 |   WINDOW NOSORT               |          | 1000K|   198M| 1002K |
|  3 |    TABLE ACCESS BY INDEX ROWID| DEMO     | 1000K|   198M| 1002K |
|  4 |     INDEX FULL SCAN           | DEMO_IDX | 1000K|       | 2504  |
------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("RNK"<=1)
   2 - filter(DENSE_RANK() OVER ( PARTITION BY "START_KEY" ORDER BY
              "GROUP_KEY")<=1)

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
    1002692  consistent gets
     817172  physical reads
          0  redo size
      59215  bytes sent via SQL*Net to client
       2806  bytes received via SQL*Net from client
        219  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
       3260  rows processed

The expensive step in this execution plan is the table access. If the table access would be moved up to take place after the window filter, the cost for this step would be 3260 (one for each fetched row). The total cost for the plan would probably stay below 7000; that is, lower then the cost for the full table scan plan.

Conclusion

Just to re-emphasize the motivation behind the view that can serve both needs; it is about multidimensional optimization—and that has nothing to do with OLAP!

Typically, performance optimization takes only one dimension into account; that is, performance. So far so good, but what about long term maintenance? Very often, performance optimization reduces the maintainability of the software. That’s not a coincidence, it’s because maintainability is the only degree of freedom during optimization. Unfortunately, reduced maintainability is very hard to notice. If it is noticed at all, it is probably years later.

I have been in both worlds for some years—operations and development—and try to optimize for all dimensions whenever possible because all of them are important for the business.

Create and Insert Statements

To try it yourself:

create table demo (
       id          number        not null,
       start_key   varchar2(255) not null,
       group_key   number        not null,
       junk        char(200),
       primary key (id)
);

insert into demo (
       select level,
              dbms_random.string('A', 2) start_key,
              trunc(dbms_random.value(0,1000)),
              'junk'
         from dual
      connect by level <= 1000000
);

commit;

exec DBMS_STATS.GATHER_TABLE_STATS(null, 'DEMO');

My tests were conducted on 11R2.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

First of all, I’d like to introduce the two most frequently considered software quality aspects:

Correctness

The software must produce correct results. There is no point in using software, which produces results that you need to verify before you can trust them.

Usability

If you know the task you want to perform, the software should make it easy to do so. Frequently used functions must be easily accessible and allow to complete the task in an efficient manner.

These two aspects belong to the functional cluster of software quality. Today’s software industry has a rather high awareness for these issues—I guess because users complain immediately otherwise.

The industrialization of those aspects becomes evident when looking at the supporting industries. As of writing, Google shows 8 sponsored links for “requirements engineering tools” and countless software testing tools. There are even education programs to certify software testers.

On the other hand there are also aspects that belong to the technical cluster of software quality.

Performance

On my business card, I use the following definitions:

Short response time for a given piece of work.

Low utilization of computing resources.

Please note that testers can observe the effects of the first definition. Sometime performance issues become that bad that they become a usability issue. There is a certain amount of industrial support for performance available.

Reliability

On my business card, I use the following definition:

The probability of failure free operation of a computer program in a specified environment for a specified time.

The majority of reliability issues are system crashes and system hangs. They are typically caused by memory leaks, deadlocks and race conditions.

At this point, industrial support becomes sparse. Reliability issues are usually very hard to reproduce and they often remain unsolved for a long time. The first signs are often regarded as one-time events that are not taken seriously. However, these are the opportunities to handle the problem before it becomes unmanageable. Unfortunately reliability issues are easier to debug once they occur more often.

Maintainability

On my business card, I use the following definition:

The ease with which a software product can be modified in order to correct defects and meet new requirements.

The users of a software product can not directly observe maintainability issues. There is, however, one sign that can be observed from the outside and might indicate maintainability issues: the time to market. If a software vendor can quickly adopt the product for new requirements, it is usually a good sign. The opposite conclusion doesn’t hold true because the vendor’s priorities might favor other issues at that time.

Although there are tools to automatically calculate maintainability metrics, I believe the most efficient way to improve maintainability is a review by knowing eyes.

Scalability

On my business card, I use the following definition:

The property of a system which indicates its ability to handle growing amounts of work in a graceful manner.

Scalability is related to performance. Many performance testing tools can be used to analyze the application’s scalability. The key difference—from my perspective—is that performance questions tend to focus on current load scenarios while scalability questions focus on future business development.

There are some more software quality topics. You probably noticed that there was not a single word about security. The reason I didn’t list security is that this area is way to wide to be properly covered by a one-man business like mine.

So, the four technical software quality aspects performance, reliability, maintainability and scalability are my primary area of interest and work—and therefore also the topic of this blog.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

For easy C&P I the complete new script to try:

CREATE TABLE row_mig1 (
  a CHAR(2000),
  b CHAR(2000),
  c CHAR(2000),
  x NUMBER      NOT NULL,
  filter NUMBER NOT NULL,
  CONSTRAINT row_mig1_pk PRIMARY KEY (x)
) ENABLE ROW MOVEMENT;

BEGIN
   FOR i IN 1..100000 LOOP
      INSERT INTO row_mig1
                    (x, filter)
             VALUES (i, trunc(dbms_random.value(0, 10)));

      UPDATE row_mig1
         SET a ='a', b = 'b', c = 'c'
       WHERE x=i;
   END LOOP;
END;
/
COMMIT;

CREATE INDEX row_mig1_idx ON row_mig1(filter);

ANALYZE TABLE row_mig1 COMPUTE STATISTICS;
ANALYZE INDEX row_mig1_pk COMPUTE STATISTICS;
ANALYZE INDEX row_mig1_idx COMPUTE STATISTICS;

SELECT i.index_name         idx_name
     , t.blocks             table_blocks
     , i.clustering_factor  idx_clust_factor
     , i.num_rows           idx_rows
     , t.chain_cnt          table_chain_cnt
  FROM user_indexes i
  JOIN user_tables t USING (table_name)
 WHERE table_name = 'ROW_MIG1';

This is the very same table as in my previous article. I just replaced the DBMS_STATS with ANALYZE. As expected, the CHAIN_CNT is set correctly:

IDX_NAME       TABLE_BLOCKS IDX_CLUST_FACTOR IDX_ROWS TABLE_CHAIN_CNT
-------------- ------------ ---------------- -------- ---------------
ROW_MIG1_PK          100103              919   100000           99999
ROW_MIG1_IDX         100103             9187   100000           99999

So far, so good. Now let’s create the second table—without row migration:

CREATE TABLE row_mig2 (
  a CHAR(2000),
  b CHAR(2000),
  c CHAR(2000),
  x NUMBER      NOT NULL,
  filter NUMBER NOT NULL,
  CONSTRAINT row_mig2_pk PRIMARY KEY (x)
) ENABLE ROW MOVEMENT;

INSERT INTO row_mig2 (x, filter, a, b, c)
     SELECT level, trunc(dbms_random.value(0,17)), 'a', 'b', 'c'
       FROM dual CONNECT BY level <= 100000;
COMMIT;

CREATE INDEX row_mig2_idx ON row_mig2(filter);

ANALYZE TABLE row_mig2 COMPUTE STATISTICS;
ANALYZE INDEX row_mig2_pk COMPUTE STATISTICS;
ANALYZE INDEX row_mig2_idx COMPUTE STATISTICS;

SELECT i.index_name         idx_name
     , t.blocks             table_blocks
     , i.clustering_factor  idx_clust_factor
     , i.num_rows           idx_rows
     , t.chain_cnt          table_chain_cnt
  FROM user_indexes i
  JOIN user_tables t USING (table_name)
 WHERE table_name = 'ROW_MIG2';

Just to verify that there is no chaining when a plain insert approach is followed:

IDX_NAME       TABLE_BLOCKS IDX_CLUST_FACTOR IDX_ROWS TABLE_CHAIN_CNT
-------------- ------------ ---------------- -------- ---------------
ROW_MIG2_IDX         100877           100000   100000               0
ROW_MIG2_PK          100877           100000   100000               0

So, let’s see how my “trapQL” works:

SET AUTOTRACE TRACEONLY;
SET TIMING ON;

SELECT /* analyze that! */ *
  FROM row_mig1 d1
  JOIN row_mig2 d2 ON (d1.x = d2.x)
 WHERE d1.filter = 0
   AND d2.filter = 0;

SET TIMING OFF
SET AUTOTRACE OFF

And the result is:

602 rows selected.

Elapsed: 00:01:00.73

Execution Plan
----------------------------------------------------------
Plan hash value: 3163423506

----------------------------------------------------------------------
| Id  | Operation                     | Name         | Rows  |  Cost |
----------------------------------------------------------------------
|   0 | SELECT STATEMENT              |              |  5882 | 17663 |
|   1 |  NESTED LOOPS                 |              |       |       |
|   2 |   NESTED LOOPS                |              |  5882 | 17663 |
|   3 |    TABLE ACCESS BY INDEX ROWID| ROW_MIG2     |  5882 |  5896 |
|*  4 |     INDEX RANGE SCAN          | ROW_MIG2_IDX |  5882 |    12 |
|*  5 |    INDEX UNIQUE SCAN          | ROW_MIG1_PK  |     1 |     0 |
|*  6 |   TABLE ACCESS BY INDEX ROWID | ROW_MIG1     |     1 |     2 |
----------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("D2"."FILTER"=0)
   5 - access("D1"."X"="D2"."X")
   6 - filter("D1"."FILTER"=0)

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
      20929  consistent gets
      12994  physical reads
          0  redo size
      39658  bytes sent via SQL*Net to client
        859  bytes received via SQL*Net from client
         42  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
        602  rows processed

Surprising, at least to me. The optimizer considers the CHAIN_CNT and produces the best possible execution plan. The cost for the TABLE ACCESS BY INDEX ROW ID has grown to two. That’s a very concise reflection of the additional block that needs to be fetched.

To verify that the difference is the CHAIN_CNT let’s verify the plan after DBMS_STATS collection:

SQL> BEGIN
       DBMS_STATS.GATHER_TABLE_STATS(null, 'ROW_MIG1',
                                     CASCADE => TRUE);
       DBMS_STATS.GATHER_TABLE_STATS(null, 'ROW_MIG12',
                                     CASCADE => TRUE);
     END;
     /

PL/SQL procedure successfully completed.

SQL> SELECT i.index_name         idx_name
          , t.blocks             table_blocks
          , i.clustering_factor  idx_clust_factor
          , i.num_rows           idx_rows
          , t.chain_cnt          table_chain_cnt
       FROM user_indexes i
       JOIN user_tables t USING (table_name)
      WHERE table_name IN ('ROW_MIG1', 'ROW_MIG2')
      ORDER BY i.index_name;

IDX_NAME       TABLE_BLOCKS IDX_CLUST_FACTOR IDX_ROWS TABLE_CHAIN_CNT
-------------- ------------ ---------------- -------- ---------------
ROW_MIG1_IDX         100622             9180   100000           99999
ROW_MIG1_PK          100622              918   100000           99999
ROW_MIG2_IDX         100239           100000   100000               0
ROW_MIG2_PK          100239           100000   100000               0

Well, as a matter of fact, DBMS_STATS just doesn’t care about the CHAIN_CNT. So, the old values remain there. On the other hand, there are some minor differences in the other stats like the table blocks and the index clustering factor of ROW_MIG1_IDX. However, the execution plan doesn’t change:

602 rows selected.

Elapsed: 00:01:03.71

Execution Plan
----------------------------------------------------------
Plan hash value: 3163423506

----------------------------------------------------------------------
| Id  | Operation                     | Name         | Rows  |  Cost |
----------------------------------------------------------------------
|   0 | SELECT STATEMENT              |              |  6084 | 18270 |
|   1 |  NESTED LOOPS                 |              |       |       |
|   2 |   NESTED LOOPS                |              |  6084 | 18270 |
|   3 |    TABLE ACCESS BY INDEX ROWID| ROW_MIG2     |  6084 |  6098 |
|*  4 |     INDEX RANGE SCAN          | ROW_MIG2_IDX |  6084 |    12 |
|*  5 |    INDEX UNIQUE SCAN          | ROW_MIG1_PK  |     1 |     0 |
|*  6 |   TABLE ACCESS BY INDEX ROWID | ROW_MIG1     |     1 |     2 |
----------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("D2"."FILTER"=0)
   5 - access("D1"."X"="D2"."X")
   6 - filter("D1"."FILTER"=0)

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
      21131  consistent gets
      12911  physical reads
          0  redo size
      39658  bytes sent via SQL*Net to client
        859  bytes received via SQL*Net from client
         42  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
        602  rows processed

Well, it seems that the CHAIN_CNT is really making all the difference since it is the only value that was not calculated by DMBS_STATS…or is it? The ultimate proof would be to update the CHAIN_CNT manually to zero in the same way as I manually update the CLUSTERING_FACTOR using DBMS_STATS in my previous post. However, I already mentioned DBMS_STATS doesn’t care about the CHAIN_CNT, consequently there is no parameter in SET_TABLE_STATS to manually update it.

For the sake of science, I have done what nobody should ever do. And I will definitely not post what I’ve done. However, I did it, and I can tell you; the CHAIN_CNT is making all the difference.

Summary

The story from the previous posts is a little bit depressing. Although ANALYZE is deprecated, it provides more information to the CBO than DBMS_STATS. Using the recommended DBMS_STATS alone opens the problem that I have demonstrated in my previous post. The depressing factor is that this is actually a regression. A useful functionality of the CBO was “removed” by the recommendation to use DBMS_STATS.

Once more I must emphasize that all of that trouble was caused by the excessive use of the “insert empty, update everything” anti-pattern. Obviously everybody should try very hard not to follow that pattern. That solves all the problems.

Thanks

Thanks to Jonathan Lewis for pointing me onto the question how my “trapQL” works with ANALYZE.

Special thanks to Gerhard Kircher.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

Scope

My scope for this release was rather limited: implement -v switches to hatimerun and halockrun to make them more communicative.

The reason for that is quite simple; hatools will be easier to learn and use if they talk more. I also noticed that there are two “new” tools in the ever growing locking tool zoo—not to mention flock(1)—that talk more than hatools:

• Steve Friedl’s lockrun.c
• Scylla and Charybdis lockrun

My original design goal of hatools was to allow easy integration into shell scripts, thus every error is reported via the exitcode. I still believe in that goal because error handling should not parse error messages. UNIX commands should communicate errors to the caller in a way that allows easy handling in scripts.

However, I must admit that my implementation is a little bit harsh. Except of fatal errors, hatools don’t write anything to STDOUT or STDERR. Especially “designed” errors—such as “lock busy”—don’t cause a message to the user. Today—almost 9 years later—I wonder about the missing verbosity of hatools for two reasons:

• Just because the exitcode contains all the information doesn’t mean that a message isn’t handy.
• Steve Friedl’s lockrun.c has an option that causes a warning if the program takes longer than a specified timeout. He mentions that this is very handy in cron jobs, because cron e-mails that message.

I also believe that hatools have become a very powerful and a little complex in the last few years—most notably: multiple occurrences of -t, -k and -e in hatimerun. A verbose mode will make debugging much easier.

So, here comes the story why it took more than 1 hour do to it.

More Community

First of all, I moved the source code repository to GitHub—after listening to Tim Pritlove’s (german) podcast on “Verteilte Versionskontrollsysteme”.

The Implementation

Although halockrun was quickly done, hatimerun challenged me a little bit.

Finally hatimerun got two verbose modes:

-v

Will write a message if a timeout has passed by:

$ ./hatimerun -v -t 1 sleep 2
./hatimerun: process 9494 terminated on signal SIGKILL after 1s (sleep 2)

-vv

Writes a message on every timeout:

$ ./hatimerun -vv -t 1 -k hup -t 1 nohup sleep 3
nohup: appending output to `nohup.out'
./hatimerun: Timout #1 after 1s: sending signal SIGHUP to process group -9711 (nohup sleep 3)
./hatimerun: Timout #2 after 2s: sending signal SIGKILL to process group -9711 (nohup sleep 3)
./hatimerun: process 9711 terminated on signal SIGKILL after 2s (nohup sleep 3)

After years of silence, quite a lot of verbosity.

The “hard” part was to map the signal number to the signal name. I have already put a lot of effort in previous releases to make halockrun -k accept symbolic signal names—in a portable manner. That’s there since many years and seems to work quite well. So, it would be rather inappropriate to write numbers in the messages. The mapping took me quite a while and caused a lot of testing because I touched the “portability layer“ that has three different variants.

Special thanks go to Vallo Kallaste and the guys at 25th-floor for testing. After all, the release was tested on the following platforms:

• Linux 2.6.22-15-server #1 SMP Wed Aug 20 19:08:24 UTC 2008 i686 GNU/Linux—with gcc and icc
• FreeBSD 4.11-STABLE FreeBSD 4.11-STABLE #0: Thu Feb 12 08:04:00 GMT 2009
• HP-UX B.11.11 U 9000/800 9000/800 1 HP-UX
• SunOS 5.10 Generic_127111-02 sun4u sparc SUNW,UltraSPARC-IIi-cEngine
• Darwin 8.11.1 Darwin Kernel Version 8.11.1: Wed Oct 10 18:23:28 PDT 2007; root:xnu-792.25.20~1/RELEASE_I386 i386 i386
• aix 5300-09— with xlc (C for AIX version 5.0.2.0) and gcc.

Compatibility

Because the verbose mode was inspired by Steve Friedl’s lockrun, I checked again if hatools can do what lockrun.c does. Although halockrun provides a very flexible timeout mechanism, it doesn’t support the same feature as --max-time in Steve’s lockrun. The focus of hatimerun is to kill the process after a while, the --max-time switch in lockrun.c is just a error reporting feature. Well, I believe it is perfectly reasonable to have a warning if the program takes too long, but not kill it automatically.

halockrun can not be used for that purpose because it doesn’t fork() and can therefore not do anything after the child program has been started. hatimerun is the tool for timeouts in hatools. As it turned out, hatimerun could “not send a signal“ ever since the first release:

$ ./hatimerun -v -k 0 -t 1 sleep 2
./hatimerun: process 11957 terminated with status 0 after 2s (sleep 2)

The trick is to use “signal” zero; that is, not a real signal! However, -k 0 is rather awkward and most people are not aware of it’s meaning. So I introduced the symbolic name NONE for that purpose. This allows you to implement a warning level:

$ ./hatimerun -v -t 1:00 -k NONE -t 1:00 -k KILL sleep 130

This will wait for a minute (first -t 1:00), then do nothing (-k NONE) but write a warning in the end (-v). After another minute (second -t 1:00) kill the process (-k KILL).

Portability

Because I have already downloaded and tried Steve’s lockrun.c, I tried it together with halockrun. Bad enough, they don’t work together at all. That means, if a lock is occupied by lockrun, that doesn’t affect halockrun. The reason is that both tools use different advisory locking mechanisms. While halockrun uses POSIX fcntl(2), lockrun takes BSD flock(2) or POSIX lockf(3), depending on the platform. No surprise, the BSD flock() doesn’t care about POSIX locks. The Linux manpage is quite clear about that:

Since kernel 2.0, flock() is implemented as a system call in its own right rather than being emulated in the GNU C library as a call to fcntl(2). This yields true BSD semantics: there is no interaction between the types of lock placed by flock() and fcntl(2), and flock() does not detect deadlock.

However, POSIX isn’t much better, as it doesn’t define the interaction of fcntl() and lockf():

The interaction between fcntl() and lockf() locks is unspecified.

AFAIK, most systems implement lockf() in terms of fcntl(). Still there is no guarantee for that and the worst case is that a particular operating system has three different locking mechanisms. Special thanks to the “Portable Operating System Interface [for Unix]” that explicitly pushes two incompatible variants. I suppose there was a good reason for that decision, but I am not aware of it.

However, halockrun will continue to use fcntl() because it can be queried about the PID that currently holds the lock. halockrun -t hands this feature on to you.

Poor man’s fix is that I added a note about the incompatibility into the man-page.

An Advertisement: It’s All About Details

You might wonder why I write all of that? The point is that I aim to make hatools a piece of quality software. That takes quite a lot of time because quality is about details.

The advertisement is that I am an independent Software Quality Consultant for non-functional issues like performance, reliability, maintainability, scalability and so on. Let me know if I can help you.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

In my previous article—Row Migration and Row Movement—I have demonstrated that the “insert empty, update everything” anti-pattern can lead to 100% row migration. This article continues the research on row migration and unveils surprising effects on the clustering factor. To be precise, the clustering factor can become completely bogus in presence of a very high row migration rate. Once the clustering factor is “wrong”, it’s just a finger exercise to construct an optimizer trap and proof that row migration can affect the query plan.

To start off, I create a table similar to the one in the previous article. However, I add one more column and populate it with random values from 0 to 9. I will need this column to build my optimizer trap later on.

CREATE TABLE row_mig1 (
  a CHAR(2000),
  b CHAR(2000),
  c CHAR(2000),
  x NUMBER      NOT NULL,
  filter NUMBER NOT NULL,
  CONSTRAINT row_mig1_pk PRIMARY KEY (x)
) ENABLE ROW MOVEMENT;

BEGIN
   FOR i IN 1..100000 LOOP
      INSERT INTO row_mig1
                    (x, filter)
             VALUES (i, trunc(dbms_random.value(0, 10)));

      UPDATE row_mig1
         SET a ='a', b = 'b', c = 'c'
       WHERE x=i;
   END LOOP;
END;
/
COMMIT;

EXEC DBMS_STATS.GATHER_TABLE_STATS(null, 'ROW_MIG1', CASCADE=>TRUE);

Up till now everything is very similar to the last article, except that I used DBMS_STATS instead of ANALYZE TABLE. Although we know exactly that almost every row was migrated, DBMS_STATS doesn’t care about that:

SQL> SELECT num_rows, chain_cnt
       FROM user_tables
      WHERE table_name='ROW_MIG1';

  NUM_ROWS  CHAIN_CNT
---------- ----------
    100000          0

SQL> 

However, let’s have a look at the index:

SQL> SELECT num_rows, leaf_blocks, clustering_factor
       FROM user_indexes
      WHERE index_name = 'ROW_MIG1_PK';

  NUM_ROWS LEAF_BLOCKS CLUSTERING_FACTOR
---------- ----------- -----------------
    100000         187               918

SQL> 

Well, that’s an excellent clustering factor. A low clustering factor indicates that the index is in the same sequence as the table. That’s true in that case because the index column corresponds to the order in which the rows were inserted to the table. However, 918, how can this be? The clustering factor is supposed to be between the number of table blocks—which indicates a good clustering factor—and the number of index rows—which is the worst case. So, let’s look at the table size:

SQL> SELECT t.blocks             table_blocks
          , i.clustering_factor  index_clustering_factor
          , i.num_rows           index_rows
       FROM user_indexes i
       JOIN user_tables t USING (table_name)
      WHERE index_name = 'ROW_MIG1_PK';

TABLE_BLOCKS INDEX_CLUSTERING_FACTOR INDEX_ROWS
------------ ----------------------- ----------
      100877                     918     100000

SQL> 

According to that, the lower bound for the clustering factor is higher than the upper bound. Hmm, let’s investigate the clustering factor manually and verify the distribution of the rows across the table blocks:

SQL> SELECT * FROM (
       SELECT DBMS_ROWID.ROWID_BLOCK_NUMBER(rowid) block_number
            , COUNT(*) rows_in_block
         FROM row_mig1
        GROUP BY DBMS_ROWID.ROWID_BLOCK_NUMBER(rowid)
     ) WHERE ROWNUM <=10;

BLOCK_NUMBER ROWS_IN_BLOCK
------------ -------------
         523           109
         524           113
         525           109
         526           109
         527           110
         536           109
         537           110
         538           109
         539           109
         540           110

10 rows selected.

SQL> 

There are 109 different ROWIDs that refer to the block number 523. But we know that each record has about 6k. It is impossible to fit 109 rows into a single block of 8k. However, the “insert empty, update everything” anti-pattern makes it possible. The game goes like this:

1. The very first row is inserted into a new block.

2. The first row is updated, and fits into the same block. The free space in that particular block is still more than a kilobyte.

3. The second row is inserted into the very same block, because there is enough free space available in that block.

4. The update of the second row triggers the migration of that row to a different block.

   The row migration changes neither the ROWID nor the index entry. That means that the forwarding address—that is, somehow, the new ROWID—is stored in the original block so that the index can find the row.

5. The third row is inserted into the very first table block, again.

   Because the second row was moved into a different block, the very first block has still free space. There is only the first row—as a whole—and one forwarding address stored in that block. So, the insert of the third row can take place there.

6. And so on. Until the forwarding addresses fill the block—up to PCTFREE.

That’s a good one, hmm?

We can even verify that:

SQL> SELECT * FROM (
       SELECT MIN(X)                               min_x
            , MAX(x)                               max_x
            , MAX(x) - MIN(x) + 1                  diff_x
            , DBMS_ROWID.ROWID_BLOCK_NUMBER(rowid) block_number
            , COUNT(*)            rows_in_block
         FROM row_mig1
        GROUP BY DBMS_ROWID.ROWID_BLOCK_NUMBER(rowid)
        ORDER BY min_x
     ) WHERE ROWNUM <=10;

     MIN_X      MAX_X     DIFF_X BLOCK_NUMBER ROWS_IN_BLOCK
---------- ---------- ---------- ------------ -------------
         1        113        113          524           113
       114        222        109          523           109
       223        332        110          537           110
       333        441        109          538           109
       442        550        109          539           109
       551        660        110          540           110
       661        769        109          541           109
       770        878        109          542           109
       879        987        109          543           109
       988       1096        109          525           109

10 rows selected.

SQL>

You see that the first row was inserted into block number 524. All subsequent rows up to the 113^th were put into the same block. When that block was finally filled up—with one row and 112 forwarding addresses—the game starts over in the next block. All the INSERT statements took place in just 918 distinct blocks. Because the ROWID is assigned during the INSERT, the subsequent migration of the row due to the UPDATE is not reflected in the ROWID.

Neither DBMS_STATS nor ANALYSE TABLE look into the table to check if the row is really there or if the particular block it is just an accumulation of forwarding addresses. From their perspective, all the rows are in the same block—this is how the clustering factor is calculated. Although correctly calculated—technically—and up to date, the clustering factor of this index does not reflect the real situation.

The “correct” value—in that sense that it reflects the data distribution correctly—for the clustering factor would be 100.000; that is, the number of rows. If the clustering factor equals the number of rows in the index—which is the worst possible case—it means that there are no two adjacent index entries that refer to the same table block. This is actually the case because no 8k block can contain two complete 6k rows.

The Clustering Factor is Wrong, So What?

So, what’s the problem if the clustering factor is wrong?

The problem is that the Cost Based Optimizer (CBO) uses the clustering factor in its cost calculation for an INDEX RANGE SCAN (see Fallacies of the Cost Based Optimizer [pdf]). That means, the cost of the INDEX RANGE SCAN will be too low, because the clustering factor is way too low. Luckily there is an effect that makes all that less problematic; all indexes on that table are affected.

Let’s make a second index to verify that:

SQL> CREATE INDEX row_mig1_idx ON row_mig1(filter);

Index created.

SQL> BEGIN
       DBMS_STATS.GATHER_TABLE_STATS(null, 'ROW_MIG1',
                                     CASCADE => TRUE);
     END;
     /

PL/SQL procedure successfully completed.

SQL> SELECT i.index_name         index_name
          , t.blocks             table_blocks
          , i.clustering_factor  index_clustering_factor
          , i.num_rows           index_rows
       FROM user_indexes i
       JOIN user_tables t USING (table_name)
      WHERE table_name = 'ROW_MIG1';

INDEX_NAME   TABLE_BLOCKS INDEX_CLUSTERING_FACTOR INDEX_ROWS
------------ ------------ ----------------------- ----------
ROW_MIG1_PK        100877                     918     100000
ROW_MIG1_IDX       100877                    9180     100000

SQL> 

The clustering factor of the new index is bigger than the one of the original index, but still way off its real value. The “correct” clustering factor for the new index would be 100.000 because there are no two adjacent index entries that refer to the table block. Well, they actually do, but there is nothing more than the forwarding address in those blocks.

Although the clustering factor does not correctly reflect the efforts to fetch the table rows, it correctly reflects the relation between the two indexes. The primary key index has a lower clustering factor because it has the same sequence as the table itself. On the other side, the index on the filter column has a different order, thus the value is higher. The side note explains why it is Ten Times as High.

If all indexes are affected, there is hardly any real problem I can see for a single table access. Even queries that can be executed with two different indexes, the CBO will most likely not do wrong because both clustering factors are misleading.

The Join Trap

If it’s not possible to confuse the optimizer with a single table, let’s use more of them. So, I try to build a trap where the optimizer’s decision of the join order is influenced by the phony clustering factor so that the optimizer takes the less efficient execution plan.

For that purpose, I build a second table very similar to the first one. There are only two differences:

1. I don’t follow the “insert empty, update everything” anti-pattern—there will be no row migration.
2. The selectivity of the filter columns is slightly increased (about 6% instead of 10%).

Here is the overall script:

CREATE TABLE row_mig2 (
  a CHAR(2000),
  b CHAR(2000),
  c CHAR(2000),
  x NUMBER      NOT NULL,
  filter NUMBER NOT NULL,
  CONSTRAINT row_mig2_pk PRIMARY KEY (x)
) ENABLE ROW MOVEMENT;

INSERT INTO row_mig2 (x, filter, a, b, c)
     SELECT level, trunc(dbms_random.value(0,17)), 'a', 'b', 'c'
       FROM dual CONNECT BY level <= 100000;
COMMIT;

CREATE INDEX row_mig2_idx ON row_mig2(filter);

EXEC DBMS_STATS.GATHER_TABLE_STATS(null, 'ROW_MIG2', CASCADE=>TRUE);

Now let’s check the statistics:

SQL> SELECT i.index_name         index_name
          , t.blocks             table_blocks
          , i.clustering_factor  index_clustering_factor
          , i.num_rows           index_rows
       FROM user_indexes i
       JOIN user_tables t USING (table_name)
      WHERE table_name = 'ROW_MIG2';

INDEX_NAME   TABLE_BLOCKS INDEX_CLUSTERING_FACTOR INDEX_ROWS
------------ ------------ ----------------------- ----------
ROW_MIG2_PK        100749                  100000     100000
ROW_MIG2_IDX       100749                  100000     100000

SQL> 

Please note that the clustering factor is at the upper bound of the expected range; that means, it indicates that there are no two adjacent index entries referring to the same table block. That’s somehow logical, if we consider that every row is in its own table block.

After that preparation, I can present my “trapQL”:

SELECT *
  FROM row_mig1 d1
  JOIN row_mig2 d2 ON (d1.x = d2.x)
 WHERE d1.filter = 0
   AND d2.filter = 0;

It is a rather trivial join on the primary keys. Additionally each table is filtered on the filter column for one particular value. The trap works because a NESTED LOOPS join is possible in both ways. Either by filtering the first table by an INDEX RANGE SCAN on filter and then fetch the corresponding entry from the second by a primary key lookup, or vice versa. However, because we—as well as the optimizer—know that the second table is more selective than the first one, the more efficient way to execute that query is to first perform the INDEX RANGE SCAN on the second table and then join in the first one. In that way, the number of primary key lookups is reduced and the overall performance will be better. Considering that the first table suffers from heavy row migration, that effect becomes even more relevant. However, it is of course the purpose of the discussion to proof that the optimizer is doing “wrong”:

SET AUTOTRACE TRACEONLY;
SET TIMING ON;

SELECT /* original clustering factor */ *
  FROM row_mig1 d1
  JOIN row_mig2 d2 ON (d1.x = d2.x)
 WHERE d1.filter = 0
   AND d2.filter = 0;

SET TIMING OFF
SET AUTOTRACE OFF

And the result is:

577 rows selected.

Elapsed: 00:01:08.35

Execution Plan
----------------------------------------------------------
Plan hash value: 4162018446

---------------------------------------------------------------------
| Id | Operation                     | Name         | Rows  | Cost  |
---------------------------------------------------------------------
|  0 | SELECT STATEMENT              |              |  5882 | 10941 |
|  1 |  NESTED LOOPS                 |              |       |       |
|  2 |   NESTED LOOPS                |              |  5882 | 10941 |
|  3 |    TABLE ACCESS BY INDEX ROWID| ROW_MIG1     | 10000 |   938 |
|* 4 |     INDEX RANGE SCAN          | ROW_MIG1_IDX | 10000 |    20 |
|* 5 |    INDEX UNIQUE SCAN          | ROW_MIG2_PK  |     1 |     0 |
|* 6 |   TABLE ACCESS BY INDEX ROWID | ROW_MIG2     |     1 |     1 |
---------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("D1"."FILTER"=0)
   5 - access("D1"."X"="D2"."X")
   6 - filter("D2"."FILTER"=0)

Statistics
----------------------------------------------------------
        913  recursive calls
          0  db block gets
      35584  consistent gets
      21027  physical reads
          0  redo size
      39012  bytes sent via SQL*Net to client
        837  bytes received via SQL*Net from client
         40  SQL*Net roundtrips to/from client
         12  sorts (memory)
          0  sorts (disk)
        577  rows processed

The optimizer has chosen to perform the INDEX RANGE SCAN on ROW_MIG1_IDX first. The optimizer is well aware of the fact that the INDEX RANGE SCAN will return about 10000 rows; still it was preferred over the alternative execution plan.

So let’s check what happens if we tell the optimizer the truth about that table’s indexes?

BEGIN
  DBMS_STATS.SET_INDEX_STATS(null, 'ROW_MIG1_PK', clstfct=>100000);
  DBMS_STATS.SET_INDEX_STATS(null, 'ROW_MIG1_IDX',clstfct=>100000);
END;
/

SET AUTOTRACE TRACEONLY;
SET TIMING ON;

SELECT /* updated clustering factor */ *
  FROM row_mig1 d1
  JOIN row_mig2 d2 ON (d1.x = d2.x)
 WHERE d1.filter = 0
   AND d2.filter = 0;

SET TIMING OFF
SET AUTOTRACE OFF

The only change is that the statistics have been manually updated to the “more correct” clustering factor of 100.000. Unfortunately neither DBMS_STATS nor ANALYZE TABLE can be used for that purpose, so I did it manually. Please note that the table itself was not changed; most of the rows are still migrated.

And the result is:

577 rows selected.

Elapsed: 00:00:59.91

Execution Plan
----------------------------------------------------------
Plan hash value: 3004301745

---------------------------------------------------------------------
| Id | Operation                     | Name         | Rows  | Cost  |
---------------------------------------------------------------------
|  0 | SELECT STATEMENT              |              |  5882 | 11780 |
|  1 |  NESTED LOOPS                 |              |       |       |
|  2 |   NESTED LOOPS                |              |  5882 | 11780 |
|  3 |    TABLE ACCESS BY INDEX ROWID| ROW_MIG2     |  5882 |  5896 |
|* 4 |     INDEX RANGE SCAN          | ROW_MIG2_IDX |  5882 |    12 |
|* 5 |    INDEX UNIQUE SCAN          | ROW_MIG1_PK  |     1 |     0 |
|* 6 |   TABLE ACCESS BY INDEX ROWID | ROW_MIG1     |     1 |     1 |
---------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("D2"."FILTER"=0)
   5 - access("D1"."X"="D2"."X")
   6 - filter("D1"."FILTER"=0)

Statistics
----------------------------------------------------------
        913  recursive calls
          0  db block gets
      20695  consistent gets
      12817  physical reads
          0  redo size
      39012  bytes sent via SQL*Net to client
        837  bytes received via SQL*Net from client
         40  SQL*Net roundtrips to/from client
         12  sorts (memory)
          0  sorts (disk)
        577  rows processed

The more representative clustering factor makes the optimizer take the expected plan. The filtering takes place on the more selective table first, which matches about 5900 rows. The other table is joined in later. The execution is about 13% faster, the logical and physical gets dropped by about 40%. That makes quite a difference.

The cost of the TABLE ACCESS BY INDEX ROWID that follows the INDEX RANGE SCAN reflects the clustering factor’s impact. The second query plan has a cost of about 5900, that actually means that each fetched row will need a block read. The original execution plan had a cost value of 938 for that step, so that the overall cost value was lower.

After that I must remind the reader that the rows in table ROW_MIG1 are still migrated. The performance difference is not cause by the row migration per se, but by the misleading statistics that result out of the row migration.

Correcting the Row Migration

To complete the exercise, I will correct the row migration, run the statement again, and compare the performance improvement:

SQL> ALTER TABLE row_mig1 MOVE;

Table altered.

SQL> ALTER INDEX row_mig1_pk REBUILD;

Index altered.

SQL> ALTER INDEX row_mig1_idx REBUILD;

Index altered.

SQL> BEGIN
       DBMS_STATS.GATHER_TABLE_STATS(null, 'ROW_MIG1',CASCADE=>TRUE);
     END;
     /

PL/SQL procedure successfully completed.

SQL> SELECT i.index_name         index_name
          , t.blocks             table_blocks
          , i.clustering_factor  index_clustering_factor
          , i.num_rows           index_rows
       FROM user_indexes i
       JOIN user_tables t USING (table_name)
      WHERE table_name = 'ROW_MIG1';

INDEX_NAME     TABLE_BLOCKS INDEX_CLUSTERING_FACTOR INDEX_ROWS
-------------- ------------ ----------------------- ----------
ROW_MIG1_PK          100506                  100000     100000
ROW_MIG1_IDX         100506                  100000     100000

SQL> 

Then execute the statement again:

SET AUTOTRACE TRACEONLY;
SET TIMING ON;

SELECT /* no row migration */ *
  FROM row_mig1 d1
  JOIN row_mig2 d2 ON (d1.x = d2.x)
 WHERE d1.filter = 0
   AND d2.filter = 0;

SET TIMING OFF
SET AUTOTRACE OFF

Because the statistics are almost identical, the plan doesn’t change, nor does the cost. What does change is the execution time as well as the number of logical gets:

577 rows selected.

Elapsed: 00:00:30.90

Execution Plan
----------------------------------------------------------
Plan hash value: 3004301745

---------------------------------------------------------------------
| Id | Operation                     | Name         | Rows  | Cost  |
---------------------------------------------------------------------
|  0 | SELECT STATEMENT              |              |  5882 | 11780 |
|  1 |  NESTED LOOPS                 |              |       |       |
|  2 |   NESTED LOOPS                |              |  5882 | 11780 |
|  3 |    TABLE ACCESS BY INDEX ROWID| ROW_MIG2     |  5882 |  5896 |
|* 4 |     INDEX RANGE SCAN          | ROW_MIG2_IDX |  5882 |    12 |
|* 5 |    INDEX UNIQUE SCAN          | ROW_MIG1_PK  |     1 |     0 |
|* 6 |   TABLE ACCESS BY INDEX ROWID | ROW_MIG1     |     1 |     1 |
---------------------------------------------------------------------            

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("D2"."FILTER"=0)
   5 - access("D1"."X"="D2"."X")
   6 - filter("D1"."FILTER"=0)

Statistics
----------------------------------------------------------
        925  recursive calls
          0  db block gets
      14271  consistent gets
      11952  physical reads
          0  redo size
      39012  bytes sent via SQL*Net to client
        837  bytes received via SQL*Net from client
         40  SQL*Net roundtrips to/from client
         13  sorts (memory)
          0  sorts (disk)
        577  rows processed

Summary

The article investigates the effects of row migration on the clustering factor and the optimizer. A “proof of concept” SQL demonstrates that row migration can affect the optimizer. The lessons from this article are:

• The “insert empty, update everything” pattern can lead to a very high row migration rate.
• DBMS_STATS doesn’t populate the CHAIN_CNT.
• A high row migration rate can cut the clustering factor, even below its theoretic minimum.
• A wrong clustering factor can affect the optimizer and result in a suboptimal plan.

Update 2010-04-30: As commented by Jonathan Lewis, I also checked the “trapQL” against tables that were analyzed with ANALYZE—surprising results.

Disclaimer

Please note that this trap was intentionally built, just to prove that row migration can potentially influence the optimizer. There are better ways to tune that particular SQL, foremost a better indexing approach.

I have put many adjectives in quotation marks because they are not in line with the technical definition of the respective noun.

The “insert empty, update everything“ anti-pattern was used to create a very high row migration rate. Although that anti-pattern can lead to 100% migration rate, it does not always.

I have successfully verified my results on Oracle 10gR1, 11gR1 and 11gR2.

Thanks

Thanks to the guys at 25th-floor who verified my results on Oracle 11gR1.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

Luckily all three are well described in excellent articles: The Secrets of Oracle Row Chaining and Migration and Row Movement in Oracle.

For the impatient, I provide some very short definitions:

Row Chaining

Distribution of a single table row across multiple data blocks.

Row Migration

Relocation of an entire table row to a new place, without updating the indexes.

Row Movement

Relocation of an entire table row to a new place and updating the indexes.

This article was inspired by the question if Oracle 11r2 performs Row Movement instead of Row Migration for ordinary UPDATE statements—that is, in absence of partitions. The short answer is: no, it doesn’t. The long answer is the rest of this article.

The difference between Row Chaining and Row Migration is somehow understandable: If the row doesn’t fit into a single data block, it must be chained; otherwise it can be migrated as a whole. The limitation of the Row Migration is that it does not update the indexes on the table. That means, the ROWID that is stored in the index still refers to the old location of the row. An additional block, the new location of the row, must be read to fetch the required data.

The more modern Row Movement is different as it updates the corresponding indexes—the ROWID actually changes. This has benefits on the long run, because the additional block read can be avoided in the TABLE ACCESS BY INDEX ROWID operation. On the short run, the actual UPDATE operation is much more complex. I suppose this is the reason why a regular update does not (yet) trigger a Row Movement.

The Row Movement was introduced to support an UPDATE on a partition key—so that a table row is moved from one table partition to another one. In the meanwhile it is also used for flashback and space management—as described in the above-mentioned article.

As of Oracle Release 11g2, Row Movement is optional and disabled per default. It can be enabled per table with a very trivial ALTER TABLE statement. The usual reason to enable it is that one of the features which require Row Movement is used; partition key update, table shrink and flashback. There is hardly any reason not to enable row movement—the only side effect is that ROWID‘s might change; that should not affect well designed applications.

Row Migration has some considerable problems, as pointed out in the later in this article. On the other hand, Row Movement has also a drawback; that is, the update of all the affected indexes can be very expensive. It’s a trade off between read and write performance. While Row Movement is more expensive for UPDATES, it maintains best index performance. In contrast, the Row Migration favors UPDATE performance over index speed. Although not true for all cases, I believe that most data is read more often than it is written—especially in our modern society where nobody every deletes data—so that Row Movement is generally the better choice.

The article examines how to get 100% migrated rows with the “insert empty, update everything” anti-pattern, why DBMS_STATS isn’t a perfect substitute for ANALYZE TABLE, the immunity of migrated rows to alter table ... shrink space and why PCTFREE is still the only rescue from DBAs perspective.

Just to show what I mean, I repeat a modified version of the test originally created by Tom Kyte and reused by Martin Zahn:

CREATE TABLE row_migration_demo (
  a CHAR(2000),
  b CHAR(2000),
  c CHAR(2000),
  d CHAR(2000),
  e CHAR(2000),
  x int,
  constraint row_migration_pk primary key (x)
) enable row movement
/

I have changed the size of the CHAR field to 2k because my block size is 8k whereas it was 4k in the original example. I have also put the x column to the end for a better verification of Row Migration versus Row Chaining. Finally I enable Row Movement, which does actually not change anything but allows me to perform an alter table ... shrink space.

The next step is a classical insert empty, update everything sequence:

INSERT INTO row_migration_demo (x) values (1);
UPDATE row_migration_demo set a = 'a', b='b', c='c' where x=1;
COMMIT; 

Up till now everything is perfectly fine and the entire row is in a single block:

SQL> select * from row_migration_demo where x=1;

         X
----------
         1

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            0

So, lets do it again and insert a second row:

INSERT INTO row_migration_demo (x) values (2);
UPDATE row_migration_demo set a = 'a', b='b', c='c' where x=2;
COMMIT;

And select it again:

SQL> select * from row_migration_demo where x=2;

         X
----------
         1

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            1

Hooray, the row is migrated:

SQL> ANALYZE TABLE row_migration_demo COMPUTE STATISTICS;

PL/SQL procedure successfully completed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
         2          1

Note: ANALYZE is deprecated. Yup, I know, but DMBS_STATS.GATHER_TABLE_STATS does not propagate the CHAIN_CNT column. Ask Tom!

It’s hard to figure out if the row is chained or migrated—I am actually not sure is there any difference in the data structure. Because the additional table fetch continued row occurs for the very first column, I believe the row is migrated:

SQL>  select length(a) from row_migration_demo where x=2;

 LENGTH(A)
----------
      2000

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            2

To be on the safe side, I will double verify:

INSERT INTO row_migration_demo (x, a) values (3, 'a');
UPDATE row_migration_demo
   SET b = 'b', c = 'c', d = 'd', e = 'e'
 WHERE x=3; 

Indeed, the a column is at the place where the row was inserted while the e column is somewhere else:

SQL>  select length(a) from row_migration_demo where x=3;

 LENGTH(A)
----------
      2000

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            2

SQL>  select length(e) from row_migration_demo where x=3;

 LENGTH(A)
----------
      2000

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            3

To make things even worse, single rows can be migrated and chained. If we insert two more records, it becomes visible:

INSERT INTO row_migration_demo (x) values (4);
UPDATE row_migration_demo
   SET a='a', b='b', c='c', d='d', e='e'
 WHERE x=4;

INSERT INTO row_migration_demo (x) values (5);
UPDATE row_migration_demo
   SET a='a', b='b', c='c', d='d', e='e'
 WHERE x=5;

COMMIT;

Each of those rows must be chained, because they need more space than a single block has. However, due to migration, the row is actually distributed to three blocks:

SQL>  select length(a) from row_migration_demo where x=5;

 LENGTH(A)
----------
      2000

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            4

SQL>  select length(b) from row_migration_demo where x=5;

 LENGTH(B)
----------
      2000

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            5

SQL>  select length(c) from row_migration_demo where x=5;

 LENGTH(C)
----------
      2000

SQL> SELECT a.name, b.value
       FROM v$statname a, v$mystat b
      WHERE a.statistic# = b.statistic#
        AND lower(a.name) = 'table fetch continued row';

NAME                             VALUE
--------------------------- ----------
table fetch continued row            7

Please note that the “continued fetch” increased by two for the select on column c . The statistics show that all but one row is chained (the very first row):

SQL> ANALYZE TABLE row_migration_demo COMPUTE STATISTICS;

PL/SQL procedure successfully completed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
         5          4

If we continue the game and continue to insert data in this way, we get an astonishing chain count:

begin
   for i in 10..1000 loop
      INSERT INTO row_migration_demo (x) values (i);
      UPDATE row_migration_demo set a ='a', b = 'b', c = 'c' where x=i;
   end loop;
end;
/

SQL> ANALYZE TABLE row_migration_demo COMPUTE STATISTICS;

PL/SQL procedure successfully completed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
       996        995

All the inserted rows are chained.

The only way to clean up the mess is to move the table. Shrinking doesn’t help to much, especially not if there was no DELETE or UPDATE that has freed some space:

SQL> alter table row_migration_demo shrink space;

Table altered.

SQL> analyze table row_migration_demo compute statistics;

Table analyzed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
       996        995

Even if some space is freed and rows are moved, the chaining is not notably reduced:

SQL> delete from row_migration_demo where x < 200;

195 rows deleted.

SQL> analyze table row_migration_demo compute statistics;

Table analyzed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
       801        801

SQL> alter table row_migration_demo shrink space;

Table altered.

SQL> analyze table row_migration_demo compute statistics;

Table analyzed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
       801        800

Please note that the shrink has corrected one chained row. That proofs that a Row Movement assigns a new ROWID and updates the index. However, one “unchained” row isn’t really the correction I would like to see. The only way to correct all chains, is to mote the table:

SQL> alter table row_migration_demo move;

Table altered.

SQL> alter index row_migration_pk rebuild;

Index altered.

SQL> analyze table row_migration_demo compute statistics;

Table analyzed.

SQL> select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
       801          0

No chaining anymore.

There is actually another way to correct row migration. The Oracle documentation suggest to delete and re-insert all affected rows. Watch out for your triggers.

From administrators‛ perspective, the only way to prevent row migration in the first place is to increase PCTFREE. The complete re-execution of this test with PCTFREE increased to 50% “solves” the problem and no chained rows occur anymore:

drop table row_migration_demo;
CREATE TABLE row_migration_demo (
  a CHAR(2000),
  b CHAR(2000),
  c CHAR(2000),
  d CHAR(2000),
  e CHAR(2000),
  x int,
  constraint row_migration_pk primary key (x)
) enable row movement pctfree 50
/
begin
   for i in 1..1000 loop
      INSERT INTO row_migration_demo (x) values (i);
      UPDATE row_migration_demo set a ='a', b = 'b', c = 'c' where x=i;
   end loop;
end;
/
commit;
analyze table row_migration_demo compute statistics;
select num_rows, chain_cnt
       from user_tables
      where table_name='ROW_MIGRATION_DEMO';

  NUM_ROWS  CHAIN_CNT
---------- ----------
      1000          0

If this is not acceptable, there is one very last option: change the application.

However, it would be very nice if an operation—similar to shrink—would move migrated rows. It would be even more compelling if a table option would allow to disable Row Migration in favor of Row Movement.

UPDATE 2010-03-09: A follow up article describes the impact of Row Migration on the Clustering Factor.

Appendix

After some years of professional experience, I always wonder why a particular idea of mine should be unique. So I suspected that this feature might be there, somewhere hidden in Oracle. So I checked for hidden parameters (thanks to a little documentation on that topic):

SQL> select a.ksppinm parameter, b.kspftctxvl
       from x$ksppi a join x$ksppcv2 b on (a.INDX+1= b.kspftctxpn)
      where a.KSPPDESC like '%movement%';

PARAMETER                                KSPFTCTXVL
---------------------------------------- ----------
_disable_implicit_row_movement           FALSE

SQL> alter session set "_disable_implicit_row_movement" = true;

Session altered.

SQL> select a.ksppinm parameter, b.kspftctxvl
       from x$ksppi a join x$ksppcv2 b on (a.INDX+1= b.kspftctxpn)
      where a.KSPPDESC like '%movement%';

PARAMETER                                KSPFTCTXVL
---------------------------------------- ----------
_disable_implicit_row_movement           TRUE

SQL>

As suggested by the name of the parameter, this doesn’t change anything in the direction I would like.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/

My procedure uses the close_trace call of oradebug. This call closes the currently written trace file for a session. Alex Gorbachev has used this to delete big trace files that are still open. My procedure goes a little bit further and exposes one more side effect of oradebug close_trace.

Warning: The technique described here uses the undocumented and unsupported oradebug facility. Although I am not aware of any negative side effects of this procedure, the use of this method takes place at your own risk.

The oradebug utility can be used from the SQL*Plus prompt. If you have an account with sufficient rights (sigh) you can attach to any session using the oradebug setorapid command. To obtain the required Oracle PID, just query the v$process view:

SQL> SELECT s.sid, s.serial#, p.pid 
       FROM v$session s, v$process p
      WHERE s.paddr=p.addr;

       SID    SERIAL#        PID
---------- ---------- ----------
         1          1          2
[... skipped ...]
        20       3126         21

26 rows selected.

SQL> oradebug setorapid 21
Oracle pid: 21, Unix process pid: 145, image: oracle@test.fatalmind.com
SQL>

Once attached to a specific process, every oradebug operation is applied to that particular process. The trick to rotate the trace file is very simple and exploits a “feature” of UNIX file systems; that is, that the name of the file is used only at the moment the file is opened. Once the file is open, all access is managed with the so called inode and the name is not relevant for the read and write operations anymore. The inode system has some more implications, one of them is frequently used to rotate log files without data loss. Another implications makes it hard to delete open files, as Alex Gorbachev explained.

The log rotation trick is to rename the open file and then cause the writing process to re-open that file. All write operations to the renamed file will be written properly to new file name—more correctly expressed: to the file which can be accessed under the new name. If the writing process re-opens the file under its original name, it will create a new file because there is no file with that name anymore. Every subsequent write operation goes to the new file. The original file will not grow anymore and can be taken away safely.

Although the oradebug command name close_trace doesn’t suggest that the trace file is reopened, it is—if SQL tracing is enabled. So we have everything to rotate a trace file. Consider the following example:

$ ls -lrt TEST_ora_14457.trc*
-rw-r----- 1 oracle oinstall 14758723 2010-02-01 TEST_ora_14457.trc
$ mv  TEST_ora_14457.trc  TEST_ora_14457.trc.old

After renaming the file, the file still grows as the SQL trace is written. You can now use oradebug close_trace to actually rotate the file:

SQL> oradebug close_trace
Statement processed.
SQL> 

The verification in the file system shows that the ‘old’ file became bigger, and the new file is also filling up:

$ ls -lrt TEST_ora_14457.trc*
-rw-r----- 1 oracle oinstall 18878704 2010-02-01 TEST_ora_14457.trc.old
-rw-r----- 1 oracle oinstall  1555972 2010-02-01 TEST_ora_14457.trc
$ 

Because I often need to trace many sessions, I wrote a very tiny script (switch_traces.sql) that creates another script to re-open all trace files:

set echo off
set feedback off
set pages 0
set heading off
set define off

spool oradebug_close_traces.sql

select cmd from (
select pid, 1 o, 'oradebug setorapid ' || pid cmd from v$process
UNION ALL
select pid, 2 o, 'oradebug flush' from v$process
UNION ALL
select pid, 3 o, 'oradebug close_trace' from v$process) order by pid,o;

spool off

prompt
prompt c&p >>> @oradebug_close_traces.sql <<< to switch traces

set feedback on
set pages 1000
set heading on
set echo on

This script will close all traces in all sessions and continue tracing to new files if SQL tracing is enabled. You might want to add additional where clauses to limit your scope—e.g., to skip background processes. The script will also issue a flush before the close_trace. I don’t know if this is required or not, it doesn’t seem to make any harm. The use of the script is very simple:

SQL> @switch_traces.sql
SQL> set echo off
oradebug setorapid 1
oradebug flush
oradebug close_trace

[... skipped ...]

oradebug setorapid 29
oradebug flush
oradebug close_trace

c&p >>> @oradebug_close_traces.sql <<< to switch traces
SQL> @oradebug_close_traces.sql
SQL> oradebug setorapid 1
ORA-00072: process "Unix process pid: 0, image: PSEUDO" is not active
SQL> oradebug flush
Statement processed.
SQL> oradebug close_trace
Statement processed.
SQL> oradebug setorapid 2
Oracle pid: 2, Unix process pid: 137, image: oracle@test.fatalmind.com
SQL> oradebug flush
Statement processed.
SQL> oradebug close_trace
Statement processed.

[... skipped ...]

Oracle pid: 29, Unix process pid: 138, image: oracle@test.fatalmind.com
SQL> oradebug flush
Statement processed.
SQL> oradebug close_trace
Statement processed.
SQL>

In case you have not forgotten to rename your trace files you will see the new files growing now.

Read also my previous article To Trace or Not to Trace for information about fine grained SQL tracing.

Instant-Consulting

Häppchenweise Online Consulting für Entwickler. Jetzt gratis testen!
winand.at/consulting/