A hint is an instruction to the optimizer

This is what’s written in Oracle documentation. Instruction is defined as

a code that tells a computer to perform a particular operation

Which means Oracle CBO must obey the hints and must perform particular operation. The latter is hard to define correctly and explain precisely because it involves the logic of the block-box (what Cost Based Optimizer is). Some of the operations are mentioned in the standard Oracle documentation, some of them scattered across different places, and there are exceptions as usual. I think I’ll list here these cases which could lead to “ignoring hints” with the links to documentation/blogs.

Unsurprisingly, most of the cases are covered by the documentation. Good to know.
PS. Apart from documentation, an excellent source of information about hinting is presentation and paper Hint on Hints by Jonathan Lewis.

Here’s a simple table T1 holding 1M rows with the indexed column X having 10 distinct values:

drop table t1 cascade constraints purge;
create table t1 (id, x, pad, constraint t1_pk primary key(id))
as
select rownum
     , mod(rownum, 10)
     , lpad('x', 200, 'x')
  from all_source s1, all_source s2
 where rownum <= 1e6;
 
create index t1_indx on t1(x);
exec dbms_stats.gather_table_stats(user, 't1', method_opt=>'for all columns size 1', cascade=>true, no_invalidate=>false)

I run multiple EXPLAIN PLANs for a query accessing T1 by a single X value using T1_INDX range scan. I know it’s a very inefficient way to access the data in this case, but it fits perfectly well for my purposes. I’m using different OICA/OIC settings with the help of OPT_PARAM hint and print the resulting IO cost for the query based on the PLAN_TABLE data:

delete plan_table;
commit;

set serveroutput on
declare
  l_query varchar2(4000);
begin
  for oica in 0..10 loop
    for oic in 0..10 loop
      l_query := 'explain plan set statement_id = ''' || oica*10 || ',' || oic*10 || ''' for ' ||
                  'select /*+ index_rs(t1 t1_indx) ' || 
                            ' opt_param(''optimizer_index_cost_adj'' ' || oica*10 || ') ' ||
                            ' opt_param(''optimizer_index_caching'' ' || oic*10 || ') */ * ' ||
                  '  from t1 ' ||
                  ' where x = :1';
--      dbms_output.put_line(l_query);
      execute immediate l_query;
    end loop;
  end loop;
end;
/

set numwidth 7

select *
  from (select to_number(substr(statement_id, 1, instr(statement_id, ',') - 1)) oica
              ,to_number(substr(statement_id, instr(statement_id, ',') + 1)) oic
              ,io_cost
          from plan_table
         where id = 0) 
  pivot(max(io_cost) for oic in (0  as oic_0
                                ,10 as oic_10
                                ,20 as oic_20
                                ,30 as oic_30
                                ,40 as oic_40
                                ,50 as oic_50
                                ,60 as oic_60
                                ,70 as oic_70
                                ,80 as oic_80
                                ,90 as oic_90
                                ,100 as oic_100))
 order by oica desc;

The results are expected:

   OICA   OIC_0  OIC_10  OIC_20  OIC_30  OIC_40  OIC_50  OIC_60  OIC_70  OIC_80  OIC_90 OIC_100
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
    100   30499   30499   30499   30499   30499   30499   30499   30499   30499   30499   30499
     90   27449   27449   27449   27449   27449   27449   27449   27449   27449   27449   27449
     80   24399   24399   24399   24399   24399   24399   24399   24399   24399   24399   24399
     70   21349   21349   21349   21349   21349   21349   21349   21349   21349   21349   21349
     60   18299   18299   18299   18299   18299   18299   18299   18299   18299   18299   18299
     50   15250   15250   15250   15250   15250   15250   15250   15250   15250   15250   15250
     40   12200   12200   12200   12200   12200   12200   12200   12200   12200   12200   12200
     30    9150    9150    9150    9150    9150    9150    9150    9150    9150    9150    9150
     20    6100    6100    6100    6100    6100    6100    6100    6100    6100    6100    6100
     10    3050    3050    3050    3050    3050    3050    3050    3050    3050    3050    3050
      0       1       1       1       1       1       1       1       1       1       1       1

Since the table is around 30K blocks and the data associated with any given value X is spread across all of the table blocks and the data is poorly clustered, I’m expecting the query cost to be around the number of table blocks, with table access being the major contributor, and it’s true by default:

---------------------------------------------------------------------------------------
| Id  | Operation                   | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |         |   100K|    19M| 30511   (1)| 00:06:07 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T1      |   100K|    19M| 30511   (1)| 00:06:07 |
|*  2 |   INDEX RANGE SCAN          | T1_INDX |   100K|       |   197   (1)| 00:00:03 |
---------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("X"=TO_NUMBER(:1))

With the changes of OICA, the query cost changes almost linearly, except for some rounding error. Changes to the OIC do not impact query cost, as the documentation suggests:

OPTIMIZER_INDEX_CACHING lets you adjust the behavior of cost-based optimization to favor nested loops joins and IN-list iterators.

And I have nor NLJ, nor IN-list iterator here.

Now let’s change the table layout: make it index-organized and run the same PL/SQL test & query for the PLAN_TABLE results

drop table t1 cascade constraints purge;
create table t1 (id, x, pad, constraint t1_pk primary key(id))
organization index
as
select rownum
     , mod(rownum, 10)
     , lpad('x', 200, 'x')
  from all_source s1, all_source s2
 where rownum <= 1e6;
 
create index t1_indx on t1(x);
exec dbms_stats.gather_table_stats(user, 't1', method_opt=>'for all columns size 1', cascade=>true, no_invalidate=>false)

   OICA   OIC_0  OIC_10  OIC_20  OIC_30  OIC_40  OIC_50  OIC_60  OIC_70  OIC_80  OIC_90 OIC_100
------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -------
    100   30582     255     226     198     170     142     113      85      57      29   30304
     90   27524     230     203     178     153     128     102      77      51      26   27274
     80   24466     204     181     158     136     114      90      68      46      23   24243
     70   21407     179     158     139     119      99      79      60      40      20   21213
     60   18349     153     136     119     102      85      68      51      34      17   18182
     50   15291     128     113      99      85      71      57      43      29      15   15152
     40   12233     102      90      79      68      57      45      34      23      12   12122
     30    9175      77      68      59      51      43      34      26      17       9    9091
     20    6116      51      45      40      34      28      23      17      11       6    6061
     10    3058      26      23      20      17      14      11       9       6       3    3030
      0       1       1       1       1       1       1       1       1       1       1       1

This time the results are amusing. As long as the OIC doesn’t change, everything goes pretty much similar to the heap-organized table, with the query cost being a little bit higher – that’s OK as secondary indexes, based on logical rowids, may be slightly less efficient than indexes on heap tables.
When the OIC changes, everything gets too crazy. By modifying the CBO index caching assumption to a mere 10%, the query cost drops by two orders of magnitude! This is tremendous shift in the costing which poses terrific consequences for legacy applications trying to utilize IOTs advantages.
It looks like the issue was introduced by the bug fix 5236908, which is described in the V$SESSION_FIX_CONTROL as “Allow _optimizer_index_caching to affect IOT primary keys” and was introduced in the 10.2.0.4. Probably they did it wrong: they’ve discounted UQ scan completely, thus dropping major part of the cost in some cases:

SQL> explain plan for select /*+ index(t1 t1_indx) opt_param('optimizer_index_caching' 10) */ * from t1 where x = :1;

Explained.

SQL> @xp

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------------------
Plan hash value: 728231638

-----------------------------------------------------------------------------
| Id  | Operation         | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |         |   100K|    19M|   257   (1)| 00:00:04 |
|*  1 |  INDEX UNIQUE SCAN| T1_PK   |   100K|    19M|   257   (1)| 00:00:04 |
|*  2 |   INDEX RANGE SCAN| T1_INDX |   100K|       |   256   (1)| 00:00:04 |
-----------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - access("X"=TO_NUMBER(:1))
   2 - access("X"=TO_NUMBER(:1))

SQL> explain plan for select /*+ index(t1 t1_indx) opt_param('optimizer_index_caching' 10) */ * from t1 where id in (:1);

Explained.

SQL> @xp

PLAN_TABLE_OUTPUT
-------------------------------------------------------------------------------------------------------------------------
Plan hash value: 426392120

---------------------------------------------------------------------------
| Id  | Operation         | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |       |     1 |   209 |     0   (0)| 00:00:01 |
|*  1 |  INDEX UNIQUE SCAN| T1_PK |     1 |   209 |     0   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - access("ID"=TO_NUMBER(:1))

SQL> explain plan for select /*+ index(t1 t1_indx) opt_param('optimizer_index_caching' 10) */ * from t1 where id in (:1, :2);

Explained.

SQL> @xp

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------------------------------------------
Plan hash value: 1529266855

----------------------------------------------------------------------------
| Id  | Operation          | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |       |     2 |   418 |     2   (0)| 00:00:01 |
|   1 |  INLIST ITERATOR   |       |       |       |            |          |
|*  2 |   INDEX UNIQUE SCAN| T1_PK |     2 |   418 |     2   (0)| 00:00:01 |
----------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("ID"=TO_NUMBER(:1) OR "ID"=TO_NUMBER(:2))

The bug fix can’t be turned off with _fix_control for some reason and I don’t have an old Oracle database to run the test so I can’t confirm if the bug-fix is the culprit or not. Anyway it looks stupid. I’ll try to submit bug report for this issue.

• how to gather fixed object stats for a given X$
• should I gather fixed objects stats on both RAC nodes
• why I can’t gather stats on some X$-tables

At the time of asking these question I’ve read CBO development team blog post on the topic, but still I was unable to answer them distinctly. Well, it appears the questions are simple and could probably be deduced after careful reading (first two questions for sure). The core phrase is:

The DBMS_STATS.GATHER_FIXED_OBJECTS_STATS procedure gathers the same statistics as DBMS_STATS.GATHER_TABLE_STATS except for the number of blocks

That’s the key, the statistics are the same, except for the number of blocks. Which effectively means, you can gather the statistics on the specific X$-table in the same way as you do with a normal table, by using DBMS_STATS.GATHER_TABLE_STATS procedure. The only difference would be you will not populate the number of blocks (but you can do this manually). Now, since these statistics are essentially the same as statistics for normal tables, it makes no sense to run the GATHER_FIXED_OBJECTS_STATS procedure on all RAC nodes. After all, the database is still the same and it doesn’t care about the table statistics in relation to RAC node (at least for now in 11g).

I wish I was that smart back then I’ve found the answers recently via MOS notes updates:
In RAC, Should We Execute GATHER_FIXED_OBJECTS_STATS And Gather_dictionary_stats In Both Nodes? [ID 1479804.1] which gives the definitive answer “No” and
Why do some Fixed Tables not have Optimizer Statistics [ID 1355608.1] which explains that some fixed tables cannot have optimizer statistics by design decisions made by Oracle.
What I don’t understand after getting better idea of fixed stats is why Oracle made the choice to introduce a specific new procedure, without any parameters/configuration options if they could have better done it inside the standard procedure for statistics gathering? To me it looks very strange and I would prefer not to introduce one more complication to the already over-complicated statistics package.

The start

A glass for questions. It was full in the end of the day

Tom started with a bit of history of how Oracle developed its database. Old stuff, C compilers, 37kg disk drives followed by the modern, upcoming features in 12c. Pluggable Databases mostly – the biggest change in the Oracle architecture since version 6, he asserted.

Also mentioned: automatic data tiering, aka “data heat maps”, data redaction, adaptive query optimization (scary 12c feature of), parallel execution and partitioning improvements.
One thing about adaptive query optimization that is not mentioned is you can’t change any given execution in run time. I think it’s only possible if the client (or some buffered row source before giving data back to the client) didn’t fetch any row. Otherwise I believe it’s not possible to guarantee the same row order output.

Next in agenda was Security. Key points: everybody afraid to delete data (not only for historic reason, inserting data could be much faster – see Tim Gorman’s article in the latest NoCOUG journal), it’s hard to detect data theft, SQL Injection is dangerous; Database Firewall could help in protecting the database (for example, no DDL in production). Audit trails in the 12c will be someway different, “unified”. Advanced Security will be free in the next release. Different ways to encrypt: tablespace, column, backup… Interesting way of getting a part (like 10%) of the database masked and copied to QA/dev environment. New feature to detect which privileges are actually needed to application, to be able to easily turn off unnecessary grants. More separation of duties: new SYSBACKUP, SYSDG, SYSKM roles.

New features in the development was after a coffee break. A lot of new features expected in 12c: transaction guard, application continuity (very interesting client-side analog of TAF). With SQL Translator Oracle will be able to run code created for different databases (but will it be correct?).
APEX can do a lot of work even on a low-end machine. With 12c there’s option to install it either on CDB or PDB level.
IDENTITY columns – finally!
SQL Developer is evolving towards developer and DBA toolkit. I need to start using it.
The number of new features in 12c is astonishing. A lot of new stuff I don’t remember already! Good thing I was able to record the whole event, so I can hear Tom again when I would like to.
You can do this too, here are 99MB (5h30min) of Tom Kyte speaking. Enjoy!

What I do remember clearly: Tom answered my questions and gave me a signed copy of his book. I made sure that he will get the questions: I first asked Tom in twitter and then repeated questions on paper:

As for the first question, the lack of statistical data in V$SQL for redo/undo, Tom said he understands why somebody would want this information be available; to actually get this feature in Oracle means the need to open an enhancement request. I’ll see if I’m allowed to submit ER and if I’m allowed I’ll definitely submit it.
The second question, about flashback cursors, had a mistake. Tom has built a test case to show that flashback query builds new child cursor on every parse call, not execution as I thought it was. Still Tom thinks that this behavior looks like a bug and he promised to file a bug on this matter. Great! My questions deserved a book which I’ll try to read next year.

It was again a very well-organized, day of Oracle presentations of excellent quality by one of the best presenters in the world. Big thanks to Oracle CIS and Tom for the event!

Here is a case to demonstrate:

declare
    n number := 0;
begin
    dbms_session.session_trace_enable(true, true, 'all_executions');
   
    for x in 1..10 loop
        for y in (select * from all_source) loop
            n := n + length(y.text);
        end loop;
    end loop;

    dbms_session.session_trace_disable;
end;
/

And if I search through the resulting trace file, I’ll find 10 occurrences of the STAT lines for runtime execution statistics of the SQL, as desired:

C:\Users\Tim>grep "VIEW  ALL_SOURCE" d:\oracle\diag\rdbms\test\test\trace/test_ora_1660.trc
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2369570 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2329706 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2351289 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2334011 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2350335 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2337449 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2340285 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2329053 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2362534 us cost=1039 size=569297377 card=276761)'
STAT #191268904 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2326067 us cost=1039 size=569297377 card=276761)'

Now I’ll change the way I’m enabling trace and repeat the case in another session with 30 iterations:

alter session set events 'sql_trace wait=true, bind=true, plan_stat=adaptive';

declare
    n number := 0;
begin
    for x in 1..30 loop
        for y in (select * from all_source) loop
            n := n + length(y.text);
        end loop;
    end loop;
end;
/

alter session set events 'sql_trace off';

Using new 11g syntax for the EVENTS I’ve set PLAN_STAT option to ADAPTIVE and the plan appears two times in the trace:

C:\Users\Tim>grep "VIEW  ALL_SOURCE" d:\oracle\diag\rdbms\test\test\trace/test_ora_1788.trc
STAT #191264936 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2337943 us cost=1039 size=569297377 card=276761)'
STAT #191264936 id=1 cnt=633054 pid=0 pos=1 obj=4813 op='VIEW  ALL_SOURCE (cr=39988 pr=0 pw=0 time=2352998 us cost=1039 size=569297377 card=276761)'

One for the first execution and the second after 18th execution of the cursor. Why 18th? It’s because per 1 execution the DB spends 2.3 seconds of time on average and after 10 + 18 executions for this child cursor the time spent by the DB has gone beyond 60 seconds and this triggered dumping execution statistics again. It should probably happened after 27th execution one might think. Most likely it relates to the time when the statistics in the V$ view is refreshed and the check is performed.
Anyway this is nice improvement to the SQL trace. Here is the bug for this improvement: Bug 8328200 – Misleading or excessive STAT# lines for SQL_TRACE / 10046. One minor issue is the new option is not available with DBMS_SESSION/DBMS_MONITOR packages and works only with EVENTS syntax – I hope it will be fixed some day, as well as documentation.

2 or 3 weeks before the event I’ve got the list of topics to work on. It included several Java and Oracle subjects. I’ve offered couple of topics myself too. The preparation went well although it took a lot of time to make some slides (which I don’t really like doing hence as a result slides look clumsy. Just as my blog ) and to put together meaningful examples. In fact I didn’t prepare everything in the volume I thought about initially because of the time – and as it turned out it was right decision. I didn’t practice presentations even though I tried. Each time I’ve tried I took different ways to describe same things so I’ve decided to go unprepared and see how it will go. Since I have clear understanding of what I’m talking about and had a plan in slides it wasn’t so hard. What was a bit of a problem I didn’t know what to expect from the audience and how to keep the balance between interesting and boring slides. Both problems didn’t materialize and were somehow avoided Must be because of simplicity I think. I tried hard to make it simple and keep what’s important in my opinion.

So I took a day off and went to present. It started badly. My notebook is a tiny Aspire S3 with only HDMI output available. When I tried to connect it to the projector, it just didn’t work. I spent half an hour trying different things until when I did something wrong and the screen went black with nothing on the projector screen. After a reboot laptop didn’t boot up properly, and I had no other choice than to use safe mode and apply System Restore which added another 10 minutes. Luckily restore finished well. It must have been Friday 13th impact but no. Turned out the plug I tried was not for the projector, but for the speakers!! The right plug was hided behind the screen wall. Fortunately there was another laptop available with DVI port, which I used to show slides. Mine notebook was behind the wall connected via HDMI, so each time I needed to show a demo I had to go up, switch the projector’s input and hide myself from the audience (the only thing visible was my back) appearing from time to time. It was funny.

First part was about JVM performance. I talked about general HotSpot JVM features such as just in time compiler and garbage collection. The latter was the main topic of this part. I covered JVM memory layout, GC types, their high-level algorithms and options to tune them based on my experience. In the end of the section there was a demo for out-of-the-box monitoring/debugging/analyzing and third-party tools such as Memory Analyzer. With only 16 slides it took almost 2 hours to talk and answer questions. I didn’t anticipate it that long, and expected to fit into an hour or so. Apparently I can talk for a long time

Next topic was Oracle JDBC. I’ve started with an overview of what is end-to-end workflow of an Oracle client, i.e. basic Oracle instance architecture, connection methods and cursors life cycle. Then I showed demos of several important JDBC features: fetch size; update batching; working with Oracle collections; client- and server-side cursors cache. Examples were focused on the performance of the work with database and its consequences for the client. I’ve finished this section with an explanation how important it is to instrument applications (which is a must for all Oracle-based apps) and a short list of common JDBC driver exceptions. Most of the things I’ve explained are pretty well described in the FAQ, documentation and white papers.

Then I went to database topics, beginning with a general topic of “Performance”. Although I’ve been working in this area for about five years, it was hard to find the right words to begin. And I have to admit it sucked a little bit. Two opening slides & explanation were not the same quality as the rest. I know that, because I didn’t sound convincing to myself. Good thing it was only introduction and it didn’t take too much time to go through. After introduction I’ve reviewed most common Oracle tools for performance diagnostics: out-of-the-box, such as SQL Trace and SQL Monitor, and also some third-party tools, such as MOATS and Snapper. I also showed what scripts and queries I use most of the time in my work. They are very simple BTW; I plan to write about them some day in the blog.
As part of this section I thought I should talk about reading execution plans. But after some thinking I’ve decided to skip this very important and necessary topic and just point to the excellent Christian Antognini’s paper. Instead I showed how to get run-time execution statistics for a plan with an example and where to look at to understand why the query runs slow. Initially I thought I’ll create just one complex example based on a query I’ve seen recently, and then use it throughout the event to also demonstrate several CBO issues such as blind guesses, arithmetic bugs, transformations and something “unknown” which I’ve never seen before. Unfortunately I wasn’t able to build a test case as I wanted, and it served only as an example of a bad performing query.
Then I moved to interpreting Statspack reports. I had sample reports from their production system which experienced some issues. Based on these reports I’ve shown how to approach analysis of the information, what (not) to expect from reports, and where (not) to look for and how to make balanced conclusions about the amount of work done by the database. Also I’ve explained some questionable configuration parameters; where to look next for more data and tried to explain what a potential reason of the problem could be. It is in fact the impossible reason (which I’ve picked up in the OakTable mailing list), but fits perfectly well and explains all issues at once. It is unfortunate system’s DBA missed the event so I couldn’t talk about all of the “why”s. Maybe next time I also thought about showing some more reports but once again it took me almost an hour to overview just a single report even without looking at the SQL execution statistics sections. I think I did this part very well and it was enough of analyzing reports for one day.

And the final topic of the day was statistics. This part was a basic explanation of how Oracle deals with execution plans, what kind of features influences its decisions, how they impact building an execution plan and fundamental understanding of what is statistics and what are the things to consider when it should (not) be gathered. I didn’t try to explain the term “cost” in this section and I only talked about cardinality estimates instead. It is enough for beginning, I think. For demos I’ve shown two very common examples when CBO estimates may go off: linear decay for out of range values (which can be “fixed” by manual adjustment to low/high column values) and correlation between columns (which is nicely covered by 11g’s extended statistics).
I’ve also shown in action my script stat.sql which I use all the time to get the basic understanding of table and index statistics.
I have to admit that in this section I only touched this complicated topic. It’s relatively easy to talk about it for the whole day with tons of examples and gotchas. Nevertheless I’m sure that I’ve covered the basics.
And the last part was supposed to be Q&A session (with questions given to me before the event), but I didn’t have time to try and write some code so it went without examples.

All in all it went smoothly; not everything went very well, but some parts were pretty darn good and some were normal. It sucked little bit for 10 minutes or so, which is acceptable. Presentations looked not too sexy – plain text with minimal drawings. Well, it wasn’t about presentations after all. I know there’s plenty of room for improvement in this area. The feedback from the audience was quite positive which is excellent. To me it was a very good experience and I’m happy I did it.

Lessons learned:

• HDMI may be a problem
• Don’t try to cover too much the same day
• Preparation takes longer than expected due to various reasons
• Preparation brings something new
• It’s better to know who will attend
• It would be better not only talk, but also “do something” useful with a real system
• VM is better choice for demos
• Google Docs and Dropbox offer very good services

PS. If you think you may need something similar in (preferably) Moscow region, drop me an e-mail. It’s doable.

Basically OP has a query with disjuncted (OR-ed) predicate which started to fail after 11.2.0.3 upgrade with ORA-01790: expression must have same datatype as corresponding expression. Here is a test case (I’ve renamed column and table names cause I’ve used to such naming):

drop table t1 cascade constraints purge;
drop table t2 cascade constraints purge;

create table t1 (
  x int not null,
  y varchar2(13)
);
                 
insert into t1 values(0,100);
insert into t1 values(0,100);
insert into t1 values(0,100);
                 
create table t2 (
 x number,
 y number(15)
);

insert into t2 values(1,100);
insert into t2 values(2,100);
insert into t2 values(1,100);
insert into t2 values(1,100);

commit;

exec dbms_stats.gather_table_stats(user, 't1', no_invalidate=>false)
exec dbms_stats.gather_table_stats(user, 't2', no_invalidate=>false)

Now let’s try to run following query:

SQL> select *
  2    from t2
  3   where not exists
  4          (select 1
  5            from t1
  6           where (t1.y = t2.y)
  7              or (t1.x = t2.y));
            or (t1.x = t2.y))
                *
ERROR at line 7:
ORA-01790: expression must have same datatype as corresponding expression

Oops. But if you change second condition with swapping left and right parts it is executed OK:

SQL> ed
Wrote file afiedt.buf

  1  select *
  2    from t2
  3   where not exists
  4          (select 1
  5            from t1
  6           where (t1.y = t2.y)
  7*             or (t2.y = t1.x))
SQL> /

no rows selected

And the plan is FILTER-based

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     6 |     7   (0)| 00:00:01 |
|*  1 |  FILTER            |      |       |       |            |          |
|   2 |   TABLE ACCESS FULL| T2   |     4 |    24 |     3   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| T1   |     1 |     6 |     2   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter( NOT EXISTS (SELECT 0 FROM "T1" "T1" WHERE "T1"."X"=:B1
              OR TO_NUMBER("T1"."Y")=:B2))
   3 - filter("T1"."X"=:B1 OR TO_NUMBER("T1"."Y")=:B2)

So why first query error-ed? Let’s look at the critical lines of 10053 trace file (which is incomplete due to error):

SU: Unnesting query blocks in query block SEL$1 (#1) that are valid to unnest.
Subquery Unnesting on query block SEL$1 (#1)SU: Performing unnesting that does not require costing.
SU: Considering subquery unnest on query block SEL$1 (#1).
SU:   Checking validity of unnesting subquery SEL$2 (#2)
SU:   Passed validity checks.
Branch query block:******* UNPARSED QUERY IS *******
SELECT 1 "1" FROM "TIM"."T1" "T1" WHERE "T1"."Y"="T2"."Y" OR "T1"."X"="T2"."Y"
Branch query block:******* UNPARSED QUERY IS *******
SELECT 1 "1" FROM "TIM"."T1" "T1" WHERE "T1"."Y"="T2"."Y" OR "T1"."X"="T2"."Y"
Set query block:******* UNPARSED QUERY IS *******
 (SELECT 1 "1" FROM "TIM"."T1" "T1" WHERE "T1"."Y"="T2"."Y") UNION ALL  (SELECT 1 "1" FROM "TIM"."T1" "T1" WHERE "T1"."X"="T2"."Y")
Registered qb: SET$E74BECDC 0xd0e9a0c8 (COMPLEX SUBQUERY UNNEST SEL$2)
---------------------
QUERY BLOCK SIGNATURE
---------------------
  signature (): qb_name=SET$E74BECDC nbfros=1 flg=0
    fro(0): flg=0 objn=0 hint_alias="NULL_HALIAS"@"SET$E74BECDC"

Query block after disjuncts converted to set:******* UNPARSED QUERY IS *******
SELECT "T2"."X" "X","T2"."Y" "Y" FROM "TIM"."T2" "T2" WHERE  NOT EXISTS ( (SELECT 1 "1" FROM "TIM"."T1" "T1" WHERE "T1"."Y"="T2"."Y") UNION ALL  (SELECT 1 "1" FROM "TIM"."T1" "T1" WHERE "T1"."X"="T2"."Y"))
SU:   Unnesting subquery query block SET$E74BECDC (#0)Registered qb: SET$7FD77EFD 0xd0e9a0c8 (SUBQ INTO VIEW FOR COMPLEX UNNEST SET$E74BECDC)
...
SELECT "T2"."X" "X","T2"."Y" "Y" FROM  ( (SELECT "T1"."Y" "ITEM_1" FROM "TIM"."T1" "T1") UNION ALL  (SELECT "T1"."X" "ITEM_2" FROM "TIM"."T1" "T1")) "VW_SQ_1","TIM"."T2" "T2" WHERE "VW_SQ_1"."VW_COL_1"="T2"."Y"

Basically Oracle tries to transform the query to replace disjunct predicate with a set (UNION ALL) of two subqueries, which allows it to unnest the resulting subquery and then join it. It’s not merged though due to UNION ALL presence.
And since for reasons unknown to me Oracle throws an exception for a query block:

select 'x' from dual
union all
select 2 from dual

original query fails. It can be avoided with an explicit type conversion which will still allow unnesting to happen:

SQL> explain plan for
  2  select *
  3    from t2
  4   where not exists
  5          (select 1
  6            from t1
  7           where (to_number(t1.y) = t2.y)
  8              or (t1.x = t2.y));

Explained.

SQL> @xp

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
Plan hash value: 1288684780

--------------------------------------------------------------------------------
| Id  | Operation            | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |         |     1 |    19 |    10  (10)| 00:00:01 |
|*  1 |  HASH JOIN ANTI      |         |     1 |    19 |    10  (10)| 00:00:01 |
|   2 |   TABLE ACCESS FULL  | T2      |     4 |    24 |     3   (0)| 00:00:01 |
|   3 |   VIEW               | VW_SQ_1 |     6 |    78 |     6   (0)| 00:00:01 |
|   4 |    UNION-ALL         |         |       |       |            |          |
|   5 |     TABLE ACCESS FULL| T1      |     3 |    12 |     3   (0)| 00:00:01 |
|   6 |     TABLE ACCESS FULL| T1      |     3 |     6 |     3   (0)| 00:00:01 |
--------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - access("VW_COL_1"="T2"."Y")

But if you don’t need unnesting, then by using hidden _optimizer_unnest_disjunctive_subq parameter you can turn off the transformation (for me it does so, but be careful with the hidden stuff):

explain plan for
select /*+ opt_param('_optimizer_unnest_disjunctive_subq', 'false') */ *
  from t2
 where not exists 
        (select 1
          from t1
         where (t1.y = t2.y)
            or (t1.x = t2.y));
@xp

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |     1 |     6 |     7   (0)| 00:00:01 |
|*  1 |  FILTER            |      |       |       |            |          |
|   2 |   TABLE ACCESS FULL| T2   |     4 |    24 |     3   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| T1   |     1 |     6 |     2   (0)| 00:00:01 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter( NOT EXISTS (SELECT 0 FROM "T1" "T1" WHERE "T1"."X"=:B1
              OR TO_NUMBER("T1"."Y")=:B2))
   3 - filter("T1"."X"=:B1 OR TO_NUMBER("T1"."Y")=:B2)

And in Jonathan’s case it’s also possible to invoke such optimization. All that is needed is a simple modification of a predicate to move 1000 to the left, which will allow Oracle to hash join the resulting view without a problem:

SQL> explain plan for
  2  select
  3      *
  4  from
  5      t2
  6  where
  7      not exists (
  8          select  /*+ unnest */
  9              null
 10          from    t3
 11          where   n3a = n2a
 12          and n3b = n2b
 13          and (id3 = id2 or id3 - 1000 = id2)
 14      )
 15  ;

Explained.

SQL> @xp

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------
Plan hash value: 3482347739

-----------------------------------------------------------------------------------
| Id  | Operation               | Name    | Rows  | Bytes | Cost (%CPU)| Time     |
-----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |         |  5000 |   776K|    33   (4)| 00:00:01 |
|*  1 |  HASH JOIN RIGHT ANTI   |         |  5000 |   776K|    33   (4)| 00:00:01 |
|   2 |   VIEW                  | VW_SQ_1 | 10000 |   380K|    10   (0)| 00:00:01 |
|   3 |    UNION-ALL            |         |       |       |            |          |
|   4 |     INDEX FAST FULL SCAN| T3_PK   |  5000 | 60000 |     5   (0)| 00:00:01 |
|   5 |     INDEX FAST FULL SCAN| T3_PK   |  5000 | 60000 |     5   (0)| 00:00:01 |
|   6 |   TABLE ACCESS FULL     | T2      |  5000 |   585K|    22   (0)| 00:00:01 |
-----------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - access("VW_COL_1"="ID2" AND "VW_COL_2"="N2A" AND "VW_COL_3"="N2B")

What’s in total:

1. Predicates order matters sometimes
2. The way a predicate is written matters sometimes

In the OakTable mailing list there was an idea that it could be due to some additional data stored alongside with the child cursor which may be used by the query, such as index validness or object statistics as of different time. This is just an idea, which doesn’t seem correct to me since with normal queries it’s very easy to get a re-parse on execute if something changes between the parse and execute calls. Like this:

drop table t cascade constraints purge;
create table t as select * from all_objects;
create index t_indx on t(object_id);
exec dbms_stats.gather_table_stats(user, 't', cascade=>true, no_invalidate=>false)

exec dbms_session.session_trace_enable

var x number
var n varchar2(100)
begin
      := dbms_sql.open_cursor;
    dbms_sql.parse(:x, 'select object_name from t where object_id = 42', dbms_sql.native);
    dbms_sql.define_column(:x, 1, :n, 100);
end;
/

drop index t_indx;

declare
    l_tmp number := dbms_sql.execute(:x);
begin
    if dbms_sql.fetch_rows(:x) > 0 then
        dbms_sql.column_value(:x, 1, :n);
    end if;
    dbms_sql.close_cursor(:x);
end;
/

exec dbms_session.session_trace_disable

I parse a cursor at one point in time, then drop an index on the accessed table and execute the query. And necessary part the trace file is:

PARSING IN CURSOR #47420142723192 len=177 dep=0 uid=62 oct=47 lid=62 tim=1338964506334918 hv=1969760827 ad='70b47a5d8' sqlid='3mmfzntuqhajv'
begin
      := dbms_sql.open_cursor;
    dbms_sql.parse(:x, 'select object_name from t where object_id = 42', dbms_sql.native);
    dbms_sql.define_column(:x, 1, :n, 100);
end;
END OF STMT
PARSE #47420142723192:c=0,e=27,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=0,tim=1338964506334917
=====================
PARSING IN CURSOR #47420144475512 len=46 dep=1 uid=62 oct=3 lid=62 tim=1338964506335822 hv=3206175636 ad='6a7f9ef50' sqlid='a75z50azjnqwn'
select object_name from t where object_id = 42
END OF STMT
PARSE #47420144475512:c=1000,e=709,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=1,plh=4155885868,tim=1338964506335821
WAIT #47420142723192: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=7035526 tim=1338964506335882
EXEC #47420142723192:c=1000,e=922,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=1338964506335899
WAIT #47420142723192: nam='SQL*Net message from client' ela= 8405 driver id=1413697536 #bytes=1 p3=0 obj#=7035526 tim=1338964506344329
...
EXEC #47420144475512:c=13998,e=13591,p=0,cr=79,cu=0,mis=1,r=0,dep=1,og=1,plh=1601196873,tim=1338964506406680
FETCH #47420144475512:c=4999,e=4864,p=0,cr=1042,cu=0,mis=0,r=0,dep=1,og=1,plh=1601196873,tim=1338964506411587
STAT #47420144475512 id=1 cnt=0 pid=0 pos=1 obj=7035525 op='TABLE ACCESS FULL T (cr=1042 pr=0 pw=0 time=4864 us cost=235 size=32 card=1)'
CLOSE #47420144475512:c=0,e=4,dep=1,type=0,tim=1338964506411685

Notice that the cursor was parsed twice – first on the PARSE and then on the EXEC call. And the plan at those two calls are different: they have different PLAN_HASH_VALUE (plh).
So this doesn’t look like a the cause to me. Oracle can do all the associated work on the execute if it needs to.
What else could it be? I’ve checked how normal and flashback cursors are different in the shared memory chunks using a simple case:

col curr_date new_value curr_date
select to_char(sysdate, 'HH24MISS') curr_date from dual;
set define on serveroutput off

var x number
exec   := dbms_flashback.get_system_change_number
select /* &curr_date */ * from t as of scn   where object_id = 42;

col prev_sql_id new_value sql_id_1
select prev_sql_id from v$session where sid = userenv('sid');

exec   := 42
select /* &curr_date */ * from t where object_id =  ;

col prev_sql_id  new_value sql_id_2
select prev_sql_id from v$session where sid = userenv('sid');

select function, chunk_com from v$sql_shared_memory where sql_id = '&sql_id_1'
minus
select function, chunk_com from v$sql_shared_memory where sql_id = '&sql_id_2';

With the help of V$SQL_SHARED_MEMORY it’s easy to check what’s inside shared cursors. And it looks like a simple flashback cursor has only 3 more chunks allocated:

FUNCTION                                                         CHUNK_COM
---------------------------------------------------------------- ----------------
kkfdParal                                                        pqctx:kkfdParal
ktfxfsnp                                                         qeSnp: ktfxfsnp
qkexrXfo                                                         ub1[]: qkexrXfo

On a 10.2.0.4 the final query produced no output at all, which makes me think that there’s nothing significant inside a flashback cursor. So why it works this way? I guess it’s to make things easier in implementation. With a new child for each flashback query execution there’s no need to check that executions plan is valid. With a DDL (drop index in my example) it’s a different story – it invalidates dependent cursors and on the execution attempt Oracle knows that the plan is no longer valid. Kind of hard making the same for flashback query. How to implement objects dependency for any given point in time? The simplest approach in my opinion is “just don’t” and try using an existent flashback child cursor and in case of an error – like missing index at this point in time – build a new cursor. I have no idea if it’s doable or not, but I’d like to see it implemented. We use flashback query; it causes hundreds of child cursors to appear in Oracle versions before 11.2.0.3 (this version limits number of child cursors to 100 by default). Of course they should be purged from shared pool relatively quickly, but still this could be done better.

• 42% use the defaults (if you truncate the value. I can’t put something other than 42 here, sorry guys )
• 16% don’t like unexpected histograms and prefer to take situation under control
• OMG another 42% (since it’s not Russian elections it has to be 100% in total, you know) use custom statistics management

Nice! Now we know that the defaults are suited to less than a half of Oracle installations. And my opinion is there would be more than a half using the defaults, should Oracle decided to switch the default METHOD_OPT value to ‘FOR ALL COLUMNS SIZE REPEAT’. Consider Tim Hall’s reply “On low priority systems the defaults are fine in many cases”, and think about why defaults are fine for such systems? My humble opinion is: nobody cares about low priority systems. So it doesn’t really matter if it’s reliable or not, whether there are histograms present or not, whether a system is stable or not – it’s low priority anyway. It will survive somehow. For me it means it’s not too much of a problem to go back to the future and modify default METHOD_OPT so that no histograms are gathered initially by default – as all should do manually now /me thinks.
PS. I wouldn’t mind switching GATHER_TEMP back to the FALSE and returning it to the documentation too, as it was in 9i.

I was in a performance testing project on a customer’s environment. It was RAC and included two DBs plus two standbys. In the process of testing it was necessary to rollback changes made to the databases during one test run. Flashback database was chosen as a way to do that. The approach was tested by DBAs and we’ve started working according to our plan. Unfortunately Flashback didn’t work out very well. Minor issues with available space in fast recovery area were solved, but more importantly the procedure took valuable DBA’s time and made our project highly contended on shared resource That was very bad because we were out of the plan by a week (or two) already. So after pushing hard we were given direct access to the databases to not disturb DBAs and to do everything we need by ourselves. That “we” was me of course.
Now little background how do I feel about mine DBA skills. Obviously I can do everything in some time – and depending on the topic the time it will take me to do something will vary from seconds to days or maybe weeks. With the backup and recovery I was at the level of absolute zero: I’ve never done PITR; I’ve never done flashback DB; I’ve never used RMAN. So you can imagine that for me doing all that wasn’t easy and it wasn’t fun either due to time pressure.
Anyway after I’ve tried it and it worked pretty easy I’ve decided that doing it manually is so boring and time consuming that I need a script. Especially since I knew I will need to repeat the same operation several dozen times in the next weeks.
Now little background how do I feel about my scripting abilities. Obviously I can do everything but my scripting skill is on a level similar to DBA. So I spent several hours to come up with something that will do the right thing in reasonable time. It was actually several scripts with one main entry point that called other scripts concurrently. The problem was: nor did I have time to test, nor did I have a test environment. So I was very nervous when I launched it like this:

./flashback.sh "20120405 18:00"

And I was so happy it finished without any error! It actually flashed back all 4 databases. The only problem was, as it turned out, the date was not correct. The time portion was ignored. I didn’t forget to include double apostrophes on the command line, but I forgot to add them inside the main script, which passed the parameter to the next scripts. And then, in turn, the date became two parameters instead of one. So when actually flashback database to timestamp command was issued the date was without time!! OMFG that was 1 day of work lost…
Now I know that it was too stupid to try it this way. Anyway after fixing the error the script was successfully used many times and usually worked about 20 minutes – that was good. Well, sometimes it didn’t work due to ORA-600, but that’s a different story.

Top 5 Timed Events                                                    Avg %Total
~~~~~~~~~~~~~~~~~~                                                   wait   Call
Event                                            Waits    Time (s)   (ms)   Time
----------------------------------------- ------------ ----------- ------ ------
log buffer space                                    28       1,396  49870   41.0
buffer busy waits                                   51       1,049  20567   30.8
CPU time                                                       320           9.4
log file parallel write                          4,011         159     40    4.7
log file sync                                    3,275         153     47    4.5

The complete report (without SQL and several other sections) is here. Can you explain what & why is wrong in this Top?
I must admit I don’t have the definitive answer (probably yet), but I have partial explanation of the issue as well as several ideas.

• Method and apparatus for reapplying changes to a database – gives you an algorithm how Oracle optimizes recovery process
• Method and mechanism for efficient implementation of ordered records – another redo-related patent with the title hiding concept of multiple redo log buffers
• Method and mechanism for relational access of recovery logs in a database system – basically it’s describing LogMiner
• METHOD AND SYSTEM FOR IMPLEMENTING EFFICIENT CURSOR PRUNING – a very good-looking patent that describes how Oracle handles sharing of the child cursors nicely. Actually after reading this I realized that V$SQL_SHARED_CURSOR isn’t just an informational data supplied for diagnostic purpose, but rather it’s used and supported by Oracle to minimize number of comparisons while searching for a child with the required properties.
• EXTENDED CURSOR SHARING – this one is also on the topic of cursor sharing features related to adaptive optimization in run-time
• Compilation and processing a parallel single cursor model – describes the way in which a database can build and use parallel cursor without additional slave SQL. This is what Oracle does since 10g.

The list can go on. Just use “assignee:oracle” in the search box if you want to see Oracle’s patents only. What I like the most in these documents is they are unique. You will rarely find such a precisely described algorithms with block diagrams describing inner workings of a potential database system. Of course the information in patents is not the exact features description, nevertheless I find it quite useful.

Here it is. I’m starting to execute a simple query against DUAL while constantly changing optimizer environment, forcing Oracle to build a new child cursor for each execution:

alter session set workarea_size_policy=manual;
col curr_date new_value curr_date
select to_char(sysdate, 'HH24MISS') curr_date from dual;
set define on verify off

begin
    for i in 1..100 loop
        execute immediate 'alter session set sort_area_size=' || to_char(100000 + i*1024);
        execute immediate 'select /* &&curr_date */ count(*) from dual';
    end loop;
end;
/

col sql_id new_value sql_id
select sql_id, address from v$sqlarea where sql_text = 'select /* &&curr_date */ count(*) from dual';
clear columns
select count(*) from v$sql_shared_cursor where sql_id = '&&sql_id';
@unshared &&sql_id

As expected, I’ve got 100 child cursors and 99 of them were created due to OPTIMIZER_MISMATCH:

SQL> select sql_id, address from v$sqlarea where sql_text = 'select /* &&curr_date */ count(*) from dual';

SQL_ID        ADDRESS
------------- ----------------
8ugq0vhq0z7pb 000007FF5D45C0B0

SQL> clear columns
columns cleared
SQL> select count(*) from v$sql_shared_cursor where sql_id = '&&sql_id';

            COUNT(*)
--------------------
                 100

SQL> @unshared &&sql_id

SQL_ID        NONSHARED_REASON                          COUNT(*)
------------- ----------------------------- --------------------
8ugq0vhq0z7pb OPTIMIZER_MISMATCH                              99

Now let’s continue and see what will happen:

begin
    for i in 101..250 loop
        execute immediate 'alter session set sort_area_size=' || to_char(100000 + i*1024);
        execute immediate 'select /* &&curr_date */ count(*) from dual';
    end loop;
end;
/

select count(*) from v$sql_shared_cursor where sql_id = '&&sql_id';

select sum(cnt), avg(cnt), min(child_number), max(child_number)
  from (select child_number, count(*) cnt
          from v$sql
         where sql_id = '&&sql_id'
         group by child_number);

select address, count(*) from v$sql where sql_id='&&sql_id' group by address;
select sql_id, address from v$sqlarea where sql_text = 'select /* &&curr_date */ count(*) from dual';
@unshared &&sql_id
select * from table(dbms_xplan.display_cursor('&&sql_id', 1));

SQL> select count(*) from v$sql_shared_cursor where sql_id = '&&sql_id';

            COUNT(*)
--------------------
                 250

SQL>
SQL> select sum(cnt), avg(cnt), min(child_number), max(child_number)
  2    from (select child_number, count(*) cnt
  3            from v$sql
  4           where sql_id = '&&sql_id'
  5           group by child_number);

            SUM(CNT)             AVG(CNT)    MIN(CHILD_NUMBER)    MAX(CHILD_NUMBER)
-------------------- -------------------- -------------------- --------------------
                 250                  2.5                    0                   99

SQL>
SQL> select address, count(*) from v$sql where sql_id='&&sql_id' group by address;

ADDRESS                      COUNT(*)
---------------- --------------------
000007FF5D1E18A8                   50
000007FF5D45C0B0                  100
000007FF5D5BA268                  100

SQL> select sql_id, address from v$sqlarea where sql_text = 'select /* &&curr_date */ count(*) from dual';

SQL_ID        ADDRESS
------------- ----------------
8ugq0vhq0z7pb 000007FF5D1E18A8

SQL> @unshared &&sql_id

SQL_ID        NONSHARED_REASON                          COUNT(*)
------------- ----------------------------- --------------------
8ugq0vhq0z7pb OPTIMIZER_MISMATCH                             249

SQL> select * from table(dbms_xplan.display_cursor('&&sql_id', 1));

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------------------------------------------
SQL_ID  8ugq0vhq0z7pb, child number 1
-------------------------------------
An uncaught error happened in prepare_sql_statement : ORA-01422: exact fetch returns more than requested number of rows

Plan hash value: 3910148636

-----------------------------------------------------------------
| Id  | Operation        | Name | Rows  | Cost (%CPU)| Time     |
-----------------------------------------------------------------
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
-----------------------------------------------------------------

SQL_ID  8ugq0vhq0z7pb, child number 1
-------------------------------------
An uncaught error happened in prepare_sql_statement : ORA-01422: exact fetch returns more than requested number of rows

Plan hash value: 3910148636

-----------------------------------------------------------------
| Id  | Operation        | Name | Rows  | Cost (%CPU)| Time     |
-----------------------------------------------------------------
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
-----------------------------------------------------------------

SQL_ID  8ugq0vhq0z7pb, child number 1
-------------------------------------
An uncaught error happened in prepare_sql_statement : ORA-01422: exact fetch returns more than requested number of rows

Plan hash value: 3910148636

-----------------------------------------------------------------
| Id  | Operation        | Name | Rows  | Cost (%CPU)| Time     |
-----------------------------------------------------------------
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   0 | SELECT STATEMENT |      |       |     2 (100)|          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   1 |  SORT AGGREGATE  |      |     1 |            |          |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
|   2 |   FAST DUAL      |      |     1 |     2   (0)| 00:00:01 |
-----------------------------------------------------------------


114 rows selected.

Bingo. After the point of 100 child cursors per parent, Oracle builds a new parent (V$SQL.ADDRESS holds its address) cursor, marking old parent and child cursors as obsolete. V$SQLAREA handles the situation well, but V$SQL doesn’t and that’s clearly a bug. The threshold point of when to build a new parent cursor is 100 by default and is controlled with a new hidden parameter _cursor_obsolete_threshold. I’ve tested setting this parameter to 150 and it worked as expected. Note that this bug-fix-improvement-again-bugs 10187168 claimed to be included into Patch Set Update 11.2.0.2.2 (and higher). I’ve tested 11.2.0.2.3 & 11.2.0.2.5 and it seems the improvement is not enabled by default and they act the old, pre-11.2.0.3 way; most likely the bug-fix have to be enabled explicitly with event 106001 in the 11.2.0.2 PSUs. I haven’t tested it though.
PS. If you have time and desire, please file a bug to Oracle.

Starting with 10g oracle introduced SQL_ID for simplicity; it is used in combination with CHILD_NUMBER to locate a particular row in the V$SQL. The documentation is clear (bold is mine):

V$SQL lists statistics on shared SQL areas without the GROUP BY clause and contains one row for each child of the original SQL text entered

Today I’ve seen a query that was running too long on a 11.2.0.3 instance. I’ve tried to get its runtime execution plan with DBMS_XPLA.DISPLAY_CURSOR, as I usually do, and it took literally a minute to get it although I’ve supplied CHILD_NUMBER and there was no much load on the DB. It was strange. When I got the result I was astonished. DISPLAY_CURSOR selected more than 80K rows and the plan was a total mess composed of numerous identical plans merged together. WTF?! I’ve started looking around and noticed that parent cursor has 1800+ child cursors – absolutely no good situation. Since there’s no v$sql_unshared_cursor (yet ) to show why there are so many child cursors in easy way, I wrote a query to return something like this with what I have right now – V$SQL_SHARED_CURSOR + DBA_TAB_COLS:

select * from
(select sql_id, nonshared_reason, count(*) from v$sql_shared_cursor
unpivot
(nonshared_value for nonshared_reason in (
UNBOUND_CURSOR as 'UNBOUND_CURSOR',
SQL_TYPE_MISMATCH as 'SQL_TYPE_MISMATCH',
OPTIMIZER_MISMATCH as 'OPTIMIZER_MISMATCH',
OUTLINE_MISMATCH as 'OUTLINE_MISMATCH',
STATS_ROW_MISMATCH as 'STATS_ROW_MISMATCH',
LITERAL_MISMATCH as 'LITERAL_MISMATCH',
FORCE_HARD_PARSE as 'FORCE_HARD_PARSE',
EXPLAIN_PLAN_CURSOR as 'EXPLAIN_PLAN_CURSOR',
BUFFERED_DML_MISMATCH as 'BUFFERED_DML_MISMATCH',
PDML_ENV_MISMATCH as 'PDML_ENV_MISMATCH',
INST_DRTLD_MISMATCH as 'INST_DRTLD_MISMATCH',
SLAVE_QC_MISMATCH as 'SLAVE_QC_MISMATCH',
TYPECHECK_MISMATCH as 'TYPECHECK_MISMATCH',
AUTH_CHECK_MISMATCH as 'AUTH_CHECK_MISMATCH',
BIND_MISMATCH as 'BIND_MISMATCH',
DESCRIBE_MISMATCH as 'DESCRIBE_MISMATCH',
LANGUAGE_MISMATCH as 'LANGUAGE_MISMATCH',
TRANSLATION_MISMATCH as 'TRANSLATION_MISMATCH',
BIND_EQUIV_FAILURE as 'BIND_EQUIV_FAILURE',
INSUFF_PRIVS as 'INSUFF_PRIVS',
INSUFF_PRIVS_REM as 'INSUFF_PRIVS_REM',
REMOTE_TRANS_MISMATCH as 'REMOTE_TRANS_MISMATCH',
LOGMINER_SESSION_MISMATCH as 'LOGMINER_SESSION_MISMATCH',
INCOMP_LTRL_MISMATCH as 'INCOMP_LTRL_MISMATCH',
OVERLAP_TIME_MISMATCH as 'OVERLAP_TIME_MISMATCH',
EDITION_MISMATCH as 'EDITION_MISMATCH',
MV_QUERY_GEN_MISMATCH as 'MV_QUERY_GEN_MISMATCH',
USER_BIND_PEEK_MISMATCH as 'USER_BIND_PEEK_MISMATCH',
TYPCHK_DEP_MISMATCH as 'TYPCHK_DEP_MISMATCH',
NO_TRIGGER_MISMATCH as 'NO_TRIGGER_MISMATCH',
FLASHBACK_CURSOR as 'FLASHBACK_CURSOR',
ANYDATA_TRANSFORMATION as 'ANYDATA_TRANSFORMATION',
PDDL_ENV_MISMATCH as 'PDDL_ENV_MISMATCH',
TOP_LEVEL_RPI_CURSOR as 'TOP_LEVEL_RPI_CURSOR',
DIFFERENT_LONG_LENGTH as 'DIFFERENT_LONG_LENGTH',
LOGICAL_STANDBY_APPLY as 'LOGICAL_STANDBY_APPLY',
DIFF_CALL_DURN as 'DIFF_CALL_DURN',
BIND_UACS_DIFF as 'BIND_UACS_DIFF',
PLSQL_CMP_SWITCHS_DIFF as 'PLSQL_CMP_SWITCHS_DIFF',
CURSOR_PARTS_MISMATCH as 'CURSOR_PARTS_MISMATCH',
STB_OBJECT_MISMATCH as 'STB_OBJECT_MISMATCH',
CROSSEDITION_TRIGGER_MISMATCH as 'CROSSEDITION_TRIGGER_MISMATCH',
PQ_SLAVE_MISMATCH as 'PQ_SLAVE_MISMATCH',
TOP_LEVEL_DDL_MISMATCH as 'TOP_LEVEL_DDL_MISMATCH',
MULTI_PX_MISMATCH as 'MULTI_PX_MISMATCH',
BIND_PEEKED_PQ_MISMATCH as 'BIND_PEEKED_PQ_MISMATCH',
MV_REWRITE_MISMATCH as 'MV_REWRITE_MISMATCH',
ROLL_INVALID_MISMATCH as 'ROLL_INVALID_MISMATCH',
OPTIMIZER_MODE_MISMATCH as 'OPTIMIZER_MODE_MISMATCH',
PX_MISMATCH as 'PX_MISMATCH',
MV_STALEOBJ_MISMATCH as 'MV_STALEOBJ_MISMATCH',
FLASHBACK_TABLE_MISMATCH as 'FLASHBACK_TABLE_MISMATCH',
LITREP_COMP_MISMATCH as 'LITREP_COMP_MISMATCH',
PLSQL_DEBUG as 'PLSQL_DEBUG',
LOAD_OPTIMIZER_STATS as 'LOAD_OPTIMIZER_STATS',
ACL_MISMATCH as 'ACL_MISMATCH',
FLASHBACK_ARCHIVE_MISMATCH as 'FLASHBACK_ARCHIVE_MISMATCH',
LOCK_USER_SCHEMA_FAILED as 'LOCK_USER_SCHEMA_FAILED',
REMOTE_MAPPING_MISMATCH as 'REMOTE_MAPPING_MISMATCH',
LOAD_RUNTIME_HEAP_FAILED as 'LOAD_RUNTIME_HEAP_FAILED',
HASH_MATCH_FAILED as 'HASH_MATCH_FAILED',
PURGED_CURSOR as 'PURGED_CURSOR',
BIND_LENGTH_UPGRADEABLE as 'BIND_LENGTH_UPGRADEABLE',
USE_FEEDBACK_STATS as 'USE_FEEDBACK_STATS'
))
where nonshared_value = 'Y'
group by sql_id, nonshared_reason
)
where sql_id = :sql_id

And the result was

NONSHARED_REASON                          COUNT(*)
----------------------------- --------------------
INSUFF_PRIVS_REM                              1815
AUTH_CHECK_MISMATCH                           1815

The query uses DB links, so I was not surprised to see the reason of non-shared cursors as “Insufficient privileges on remote objects referenced by the existing child cursor”. But why so many of them? I don’t know. Maybe it is supposed to work this way – if an error is caused by insufficient privileges on a remote system, Oracle marks child as “wrong” and tries again next time with a new child. It sounds like an easy implementation option but looks very clumsy.
I looked at a particular child cursor and…

SQL> select sql_id, child_address from v$sql where sql_id = '22d06uj1ut8d4' and child_number = 63;

SQL_ID        CHILD_ADDRESS
------------- ----------------
22d06uj1ut8d4 00000002C76328D0
22d06uj1ut8d4 00000002CF528C90
22d06uj1ut8d4 00000002D45C6E58
22d06uj1ut8d4 00000002AF4C1470
22d06uj1ut8d4 00000002AF3D8C10
22d06uj1ut8d4 00000002C78AD830
22d06uj1ut8d4 00000002971BF538
22d06uj1ut8d4 00000002864ABFF0
22d06uj1ut8d4 00000002B7FAC550
22d06uj1ut8d4 00000002BFC95E50
22d06uj1ut8d4 00000002A7AB1B98
22d06uj1ut8d4 000000029E76EAA8
22d06uj1ut8d4 00000002B67A7C38
22d06uj1ut8d4 000000027F23AFF8
22d06uj1ut8d4 0000000276EF9040
22d06uj1ut8d4 00000002969C4A68
22d06uj1ut8d4 00000002BEC691C8
22d06uj1ut8d4 00000002869F2AF0

18 rows selected.

Wow. This just can’t be right. Next I run a query to see if this behavior is system-wide or cursor-specific and it turned out it’s cursor-specific. No other cursors have more than 1 row with the same SQL_ID and CHILD_NUMBER in V$SQL. After some more experiments it turned out that the statement from documentation for this particular cursor is incorrect; and a correct version would have been written as

V$SQL lists statistics on shared SQL areas without the GROUP BY clause and contains one row for each not obsolete child of the original SQL text entered

See:

SQL> select sql_id, child_number, is_obsolete, max(cnt)
  2    from (select sql_id
  3                ,child_number
  4                ,is_obsolete
  5                ,count(*) over(partition by sql_id, child_number, is_obsolete) cnt
  6                ,count(*) over(partition by sql_id, child_number) total_cnt
  7            from v$sql)
  8   where total_cnt > 1
  9   group by sql_id, child_number, is_obsolete
 10   order by sql_id, child_number;


               Child
SQL_ID        number I             MAX(CNT)
------------- ------ - --------------------
22d06uj1ut8d4      0 N                    1
22d06uj1ut8d4      0 Y                   14
22d06uj1ut8d4      1 N                    1
22d06uj1ut8d4      1 Y                   14
22d06uj1ut8d4      2 N                    1
...
22d06uj1ut8d4     95 Y                   15
22d06uj1ut8d4     96 Y                   15
22d06uj1ut8d4     97 Y                   15
22d06uj1ut8d4     98 Y                   15
22d06uj1ut8d4     99 Y                   15

So the pattern is: either there is just 1 non obsolete child cursor for a given child number and some obsolete children or there’re obsolete children only. I don’t have an explanation of what is going on, but clearly, it looks like a bug. Even Oracle’s code to show an execution plan can’t handle this situation right. I also think it may have something to do with bug 10187168.8.
Please leave a comment here if you see something similar in your 11.2.0.3 environment. You can use a query to find such evil cursors with a query (but please be careful – it’s not safe to run it on a production system)

select sql_id, avg(cnt), sum(cnt), count(*)
  from (select sql_id, child_number, count(*) cnt from v$sql 
         group by sql_id, child_number having count(*) > 1)
 group by sql_id

In short if you are using DATEs and TIMESTAMPs without time zone in your application, you are almost “safe” and there’s most likely no need to do anything in particular on the database software level. In our application we use DATE and TIMESTAMP without time zone, so I thought it all should be fine. Not until today when I found out that I’ve a difference of 1 hour between stored TIMESTAMP and current date. Turns out it’s client’s issue (not a rare situation). The client is a WebLogic running Oracle JDK version 6u26 (1.6.0_26). And, unlike Oracle RDBMS, Oracle JRE is not consistent with OS time, so when you create a new java.sql.Timestamp from a current time, it will take DST into account and will return value correspondingly – this was news to me. In my case without time zone patch it means 1 hour behind actual time. The time zone data is fixed in the JDK update 31 (release date 14 Feb 2012); it also can be patched easily with the help of Timezone Updater Tool:

$ ./bin/java -jar tzupdater.jar -V
tzupdater version 1.3.45-b01
JRE time zone data version: tzdata2011g
Embedded time zone data version: tzdata2011n
$ ./bin/java -jar tzupdater.jar -u
$ ./bin/java -jar tzupdater.jar -V
tzupdater version 1.3.45-b01
JRE time zone data version: tzdata2011n
Embedded time zone data version: tzdata2011n

Oracle’s internal JVM (aka Aurora) in Oracle 11.2.0.3 looks to be unaffected and works correctly. It reports itself as version 1.5.0_10:

SQL> create or replace and compile java source named "test" as
  2  import java.util.Date;
  3  public class test {
  4      public static String getCurrentDate() {
  5          return "" + new Date();
  6      }
  7  };
  8  /

Java created.

SQL> sho err
No errors.
SQL>
SQL> create or replace function fnc_get_date return varchar2 as
  2  language java name 'test.getCurrentDate() return java.lang.String';
  3  /

Function created.

SQL> sho err
No errors.

SQL> select fnc_get_date, sysdate from dual;

FNC_GET_DATE                             SYSDATE
---------------------------------------- -----------------
Fri Feb 24 17:59:25 GMT+04:00 2012       20120224 17:59:25

And one more thing on the subject. I’d be happy if somebody would explain an anomaly with V$ACTIVE_SESSION_HISTORY.SAMPLE_TIME:

SQL> select * from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE    11.2.0.2.0      Production
TNS for Linux: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production

SQL> select startup_time from v$instance;

STARTUP_TIME
-----------------
20111020 16:38:21

SQL> sho parameter nls

NAME_COL_PLUS_SHOW_PARAM                 TYPE        VALUE
---------------------------------------- ----------- ----------------------------
nls_calendar                             string      GREGORIAN
nls_comp                                 string      BINARY
nls_currency                             string      $
nls_date_format                          string      YYYYMMDD HH24:MI:SS
nls_date_language                        string      AMERICAN
nls_dual_currency                        string      $
nls_iso_currency                         string      AMERICA
nls_language                             string      AMERICAN
nls_length_semantics                     string      BYTE
nls_nchar_conv_excp                      string      FALSE
nls_numeric_characters                   string      .,
nls_sort                                 string      BINARY
nls_territory                            string      AMERICA
nls_time_format                          string      HH.MI.SSXFF AM
nls_time_tz_format                       string      HH.MI.SSXFF AM TZR
nls_timestamp_format                     string      DD-MON-RR HH.MI.SSXFF AM
nls_timestamp_tz_format                  string      DD-MON-RR HH.MI.SSXFF AM TZR

-- this is the question: v$active_session_history.sample_time lags 1 hour behind
SQL> select systimestamp, sysdate, max(sample_time) from v$active_session_history;

SYSTIMESTAMP                                                                SYSDATE           MAX(SAMPLE_TIME)
--------------------------------------------------------------------------- ----------------- -------------------------
24-FEB-12 06.27.25.685962 PM +04:00                                         20120224 18:27:25 24-FEB-12 05.27.22.510 PM

col short_name  format a20              heading 'Load Profile'
col per_sec     format 999,999,999.9    heading 'Per Second'
col per_tx      format 999,999,999.9    heading 'Per Transaction'
set colsep '   '

select lpad(short_name, 20, ' ') short_name
     , per_sec
     , per_tx from
    (select short_name
          , max(decode(typ, 1, value)) per_sec
          , max(decode(typ, 2, value)) per_tx
          , max(m_rank) m_rank 
       from
        (select /*+ use_hash(s) */ 
                m.short_name
              , s.value * coeff value
              , typ
              , m_rank
           from v$sysmetric s,
               (select 'Database Time Per Sec'                      metric_name, 'DB Time' short_name, .01 coeff, 1 typ, 1 m_rank from dual union all
                select 'CPU Usage Per Sec'                          metric_name, 'DB CPU' short_name, .01 coeff, 1 typ, 2 m_rank from dual union all
                select 'Redo Generated Per Sec'                     metric_name, 'Redo size' short_name, 1 coeff, 1 typ, 3 m_rank from dual union all
                select 'Logical Reads Per Sec'                      metric_name, 'Logical reads' short_name, 1 coeff, 1 typ, 4 m_rank from dual union all
                select 'DB Block Changes Per Sec'                   metric_name, 'Block changes' short_name, 1 coeff, 1 typ, 5 m_rank from dual union all
                select 'Physical Reads Per Sec'                     metric_name, 'Physical reads' short_name, 1 coeff, 1 typ, 6 m_rank from dual union all
                select 'Physical Writes Per Sec'                    metric_name, 'Physical writes' short_name, 1 coeff, 1 typ, 7 m_rank from dual union all
                select 'User Calls Per Sec'                         metric_name, 'User calls' short_name, 1 coeff, 1 typ, 8 m_rank from dual union all
                select 'Total Parse Count Per Sec'                  metric_name, 'Parses' short_name, 1 coeff, 1 typ, 9 m_rank from dual union all
                select 'Hard Parse Count Per Sec'                   metric_name, 'Hard Parses' short_name, 1 coeff, 1 typ, 10 m_rank from dual union all
                select 'Logons Per Sec'                             metric_name, 'Logons' short_name, 1 coeff, 1 typ, 11 m_rank from dual union all
                select 'Executions Per Sec'                         metric_name, 'Executes' short_name, 1 coeff, 1 typ, 12 m_rank from dual union all
                select 'User Rollbacks Per Sec'                     metric_name, 'Rollbacks' short_name, 1 coeff, 1 typ, 13 m_rank from dual union all
                select 'User Transaction Per Sec'                   metric_name, 'Transactions' short_name, 1 coeff, 1 typ, 14 m_rank from dual union all
                select 'User Rollback UndoRec Applied Per Sec'      metric_name, 'Applied urec' short_name, 1 coeff, 1 typ, 15 m_rank from dual union all
                select 'Redo Generated Per Txn'                     metric_name, 'Redo size' short_name, 1 coeff, 2 typ, 3 m_rank from dual union all
                select 'Logical Reads Per Txn'                      metric_name, 'Logical reads' short_name, 1 coeff, 2 typ, 4 m_rank from dual union all
                select 'DB Block Changes Per Txn'                   metric_name, 'Block changes' short_name, 1 coeff, 2 typ, 5 m_rank from dual union all
                select 'Physical Reads Per Txn'                     metric_name, 'Physical reads' short_name, 1 coeff, 2 typ, 6 m_rank from dual union all
                select 'Physical Writes Per Txn'                    metric_name, 'Physical writes' short_name, 1 coeff, 2 typ, 7 m_rank from dual union all
                select 'User Calls Per Txn'                         metric_name, 'User calls' short_name, 1 coeff, 2 typ, 8 m_rank from dual union all
                select 'Total Parse Count Per Txn'                  metric_name, 'Parses' short_name, 1 coeff, 2 typ, 9 m_rank from dual union all
                select 'Hard Parse Count Per Txn'                   metric_name, 'Hard Parses' short_name, 1 coeff, 2 typ, 10 m_rank from dual union all
                select 'Logons Per Txn'                             metric_name, 'Logons' short_name, 1 coeff, 2 typ, 11 m_rank from dual union all
                select 'Executions Per Txn'                         metric_name, 'Executes' short_name, 1 coeff, 2 typ, 12 m_rank from dual union all
                select 'User Rollbacks Per Txn'                     metric_name, 'Rollbacks' short_name, 1 coeff, 2 typ, 13 m_rank from dual union all
                select 'User Transaction Per Txn'                   metric_name, 'Transactions' short_name, 1 coeff, 2 typ, 14 m_rank from dual union all
                select 'User Rollback Undo Records Applied Per Txn' metric_name, 'Applied urec' short_name, 1 coeff, 2 typ, 15 m_rank from dual) m
          where m.metric_name = s.metric_name
            and s.intsize_csec > 5000
            and s.intsize_csec < 7000)
      group by short_name)
 order by m_rank;

Couple of notes:

• It’s a simple piece of code with most of it being copy-paste to select data that is only required and to sort it the same way as AWR does
• I’ve deliberately supplied a USE_HASH hint to the query because on one DB without a fixed objects statistics (or wrong – not sure) Oracle decided to do a NESTED LOOP with a FULL SCAN of a huge in-memory structure so the query took around a second to execute. Don’t wanna that happen
• It’s possible to utilize PIVOT in the query, but it’ll limit applicability to 11g only – and I want this SQL to be able to run on 10g too

And here is a sample output. Looks familiar

Load Profile               Per Second   Per Transaction
--------------------   --------------   ---------------
             DB Time               .7
              DB CPU               .7
           Redo size        217,570.3       4,283,958.7
       Logical reads          4,177.1          82,248.0
       Block changes          1,475.5          29,053.0
      Physical reads               .1               2.0
     Physical writes               .4               8.3
          User calls               .3               6.0
              Parses              8.2             161.3
         Hard Parses              7.3             144.0
              Logons               .1               1.3
            Executes            722.3          14,222.3
           Rollbacks               .0
        Transactions               .1
        Applied urec               .1               1.0