Wednesday, 21 August, 2013 22:00 Written by Brian B
Collier IT is attending Oracle OpenWorld again.
Our President, Sales Director, Senior Educational Services Representative and myself are attending.
Here is my schedule –
Saturday
Arrow – Oracle Partner OpenWorld Dinner
Sunday
UGF9757 — An Exploration of Oracle Database 12c
UGF9760 — Oracle Database 12c Best New Features
UGF9794 — Managing Oracle Enterprise Manager Cloud Control 12c with Oracle Clusterware
KEY10161 — Oracle PartnerNetwork Exchange @ OpenWorld Keynote
GEN10244 — General Session: Engineered Systems and Hardware for Partners
KEY11026 — Oracle Welcome Keynote
Monday
SBH11107 — How Server-Side Flash in Oracle Database Instances Improves Performance and Lowers Latency
CON4717 — Building an Ultrafast, Scalable, Multithreaded, Multiprocess Server
CON8710 — Oracle Database 12c for Data Warehousing and Big Data
GEN8954 — General Session: SPARC Systems Update and Roadmap
CON6967 — Getting Unbeatable App Performance on SPARC Enterprise M-Series and SPARC T-Series Servers
CON8932 — Oracle WebLogic Server Strategy and Roadmap
BOF11055 — Best Practices for Oracle TimesTen In-Memory Database for Oracle Exalytics
BOF8958 — Extreme Availability: Oracle WebLogic/Application Continuity with Oracle Database 12c
Tuesday
CON10326 — Oracle Exalogic, an Optimized Solution for Cloud Computing and Oracle Fusion Middleware
CON7278 — Why Oracle Fusion Middleware Runs Best on Oracle Solaris
KEY11049 — Oracle OpenWorld Tuesday Afternoon Keynote
ESS10663 — The Best Platform for Oracle Database 12c and Big Data
CON8632 — Deep Dive into Oracle SuperCluster
Wednesday
CON8880 — Oracle WebLogic Server Management: Simplified, Centralized, and Automated
CON5844 — Be a Hero with Your DBA: Database Performance Tuning for Systems Administrators and Architects
CON9191 — The World’s Fastest Microprocessor: Today and Tomorrow
CON8581 — Top 10 Tips and Tricks for Getting the Best Database Performance from Oracle Solaris System
CON7393 — SPARC + Oracle Solaris: Enterprise-Class Performance and Scalability…to the Cloud
Wednesday, 07 August, 2013 11:02 Written by Brian B
The Solaris Crash Analysis Tool is a fantastic solution that is available in “My Oracle Support” (MOS) that can assist those that don’t have a strong background in Solaris internals in looking at potential issues with a system that is in a panic condition.
The built-in modular debugger (mdb) can also augment or at times work faster than SCAT
Here is a very basic walkthrough that I provide to our Collier IT engineers to assist them in initial diagnostics.
There’s much more, and I’ll add some additional walk-throughs later.
1. Useful information can be found in the stack backtrace to search keywords against MOS. Sometimes you get lucky here.
> $c vpanic(127def0, 2a100ed40c0, 0, 0, 3effffff8000000, 1869c00) cpu_deferred_error+0x568(ecc1ecc100000000, 2, 1000060000003a, 600000000, 0, 30001622360) ktl0+0x48(29fff982000, 2a100ed4d78, 30000, 16, 60, 30) pp_load_tlb+0x1e4(29fff980000, 29fff9822c0, 1d00, 29fff980300, 1822f00, 2) ppcopy_common+0x12c(70001d32500, 700030b2500, 1, 1, 29fff982000, 29fff980000) ppcopy+0xc(70001d32500, 700030b2500, 0, 0, 1822348, 70001d32500) do_page_relocate+0x228(2a100ed5120, 2a100ed5128, 700030b2500, 2a100ed53e0, 0, 2a100ed4fb0) page_relocate+0x14(2a100ed5120, 2a100ed5128, 1, 1, 2a100ed53e0, 0) page_lookup_create+0x244(60017811400, 6007c570000, 70001d32500, 0, 2a100ed53e0, 0) swap_getconpage+0xb4(60017811400, 6007c570000, 2000, 0, 2a100ed53c8, 2000) anon_map_getpages+0x474(60010c02008, 0, 200, 109a420, 2a100ed53e0, 1) segvn_fault_anonpages+0x32c(0, 800000, 0, 1, 6001753c2a8, 3) segvn_fault+0x530(300034bc3c0, 300012abc20, 1, 1, 892000, ffffffffff76e000) as_fault+0x4c8(300012abc20, 6001766b9d0, 890000, 60016881390, 186c0b0, 0) pagefault+0xac(890000, 0, 1, 0, 60016881318, 1) trap+0xd50(2a100ed5b90, 8903bb, 0, 1, fea0ad6c, 0) utl0+0x4c(1e, fe8f8104, 9e58, fe8fee34, 7aebd8, fe8fa524) >
2. Status can also give you things like the hostname and the kernel revision they’re running:
> ::status debugging crash dump vmcore.0 (64-bit) from sunbkpsrv5 operating system: 5.10 Generic_142900-13 (sun4u) panic message: UE CE Error(s) dump content: kernel pages only >
3. cpuinfo also shows some good info on what was running when the system panicked
> ::cpuinfo -v ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 0 0000183a620 1b 7 0 60 no no t-0 3000371fb20 java | | RUNNING <--+ +--> PRI THREAD PROC READY 60 2a1000c7ca0 sched EXISTS 59 30001e121e0 java ENABLE 59 30001d293e0 in.mpathd 59 3000371d480 java 59 3000371ce00 java 59 3000371c440 java 59 3000371f4a0 java ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 1 0000180c000 1d 6 0 59 yes no t-0 30001dc01c0 syslogd | | RUNNING <--+ +--> PRI THREAD PROC QUIESCED 99 2a100237ca0 sched EXISTS 60 2a100a83ca0 sched ENABLE 53 3000371c100 java 53 3000371c780 java 51 3000371aaa0 java 50 300032a9940 savecore >
4. ::ps gives good info on everything running at the time of the crash
> ::ps S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 0000000001838150 sched R 3 0 0 0 0 0x00020001 0000060012dab848 fsflush R 2 0 0 0 0 0x00020001 0000060012dac468 pageout R 1 0 0 0 0 0x4a004000 0000060012dad088 init R 808 1 807 807 0 0x42000000 0000060016acf890 nbevtmgr R 805 1 7 7 60002 0x4a304102 0000060016746038 java R 764 1 764 764 0 0x42000000 0000060016acec70 dbsrv11 R 712 1 711 711 0 0x42000000 0000060016ad04b0 bpcd R 709 1 708 708 0 0x42000000 00000600167fa040 vnetd R 386 1 385 385 0 0x42000000 0000060016ad10d0 snmpd R 382 1 382 382 25 0x52010000 00000600169a2048 sendmail R 381 1 381 381 0 0x52010000 00000600169a2c68 sendmail R 334 1 334 334 0 0x42000000 0000060016747878 syslogd R 327 1 327 327 0 0x42000000 00000600161c0490 sshd R 324 1 323 323 0 0x42010000 00000600167fb880 smcboot R 326 324 323 323 0 0x42010000 0000060013fba018 smcboot R 325 324 323 323 0 0x42010000 00000600167fac60 smcboot R 275 1 275 275 0 0x42000000 0000060016748498 utmpd R 267 1 266 266 0 0x42000000 00000600159bb860 pbx_exchange R 263 1 263 263 0 0x42000000 00000600159bac40 inetd R 257 1 257 257 0 0x42000000 0000060013e26c30 automountd R 259 257 257 257 0 0x42000000 0000060015d02488 automountd R 251 1 251 251 1 0x42000000 0000060013fbc478 rpcbind R 234 1 234 234 0 0x42010000 00000600161c10b0 cron R 208 1 208 208 0 0x42000000 0000060015d00c48 xntpd R 185 1 7 7 0 0x42000000 0000060013fbd098 iscsid R 155 1 154 154 0 0x42000000 0000060013e28470 in.mpathd R 144 1 144 144 0 0x42000000 00000600159ba020 picld R 139 1 139 139 1 0x42000000 00000600159bd0a0 kcfd R 136 1 136 136 0 0x42000000 0000060012daac28 nscd R 120 1 120 120 0 0x42000000 0000060015d030a8 syseventd R 80 1 79 79 0 0x42020000 0000060013e26010 dhcpagent R 61 1 61 61 0 0x42000000 0000060013fbb858 devfsadm R 9 1 9 9 0 0x42000000 0000060013e29090 svc.configd R 7 1 7 7 0 0x42000000 0000060012daa008 svc.startd R 357 7 7 7 0 0x4a004000 0000060016746c58 rc2 R 702 357 7 7 0 0x4a004000 00000600167490b8 lsvcrun R 703 702 7 7 0 0x4a004000 0000060013e27850 sh R 809 703 7 7 0 0x4a004000 00000600169a3888 pdde R 812 809 7 7 0 0x4a004000 0000060016ace050 pdde R 813 812 7 7 0 0x4a004000 00000600169a44a8 sleep R 342 7 7 7 0 0x4a004000 0000060015d00028 svc-webconsole R 717 342 7 7 0 0x4a004000 00000600169a50c8 sjwcx R 720 717 7 7 0 0x4a004000 00000600167fc4a0 java R 304 7 304 304 0 0x4a004000 0000060013fbac38 ttymon R 290 7 7 7 0 0x4a004000 00000600167fd0c0 svc-dumpadm R 293 290 7 7 0 0x4a004000 00000600161bf870 savecore R 269 7 269 269 0 0x4a014000 00000600161be030 sac R 278 269 269 269 0 0x4a014000 0000060015d01868 ttymon
5. ::panicinfo shows more info on the panic itself
> ::panicinfo cpu 0 thread 3000371fb20 message UE CE Error(s) tstate 80001606 g1 1270ce4 g2 127dc00 g3 3effffff8000000 g4 fbfffffe g5 1 g6 0 g7 3000371fb20 o0 127def0 o1 2a100ed4098 o2 0 o3 0 o4 fc30ffffffffffff o5 3cf000000000000 o6 2a100ed3761 o7 11020dc pc 104982c npc 1049830 y 0 >
6. Find the address of the thread that was executing when the system panicked.
> panic_thread/K panic_thread: panic_thread: 3003acf7020 gt;
7. Run the thread macro against the pointer value from above. Search for the t_procp structure.
> 3003acf7020$<$thread t_link = 0 t_stk = 0x2a108333ae0 t_startpc = 0 t_bound_cpu = 0x30004b42000 t_affinitycnt = 0 t_bind_cpu = 0xffff t_flag = 0x1800 t_proc_flag = 0x104 ... t_procp = 0x3005a6713e0 <== use the value here ... >
8. run the proc2u macro against the pointer from the t_procp structure. Look for the value stored in p_user.u_psargs. This is the full path to the command that was running on the CPU at the time of the system panic.
> 0x3005a6713e0$<proc2u p_user.u_execsw = execsw+0x28 p_user.u_auxv = [ { a_type = 0x7d8 a_un = { a_val = 0xffffffff7fffff90 a_ptr = 0xffffffff7fffff90 a_fcn = 0xffffffff7fffff90 } ... p_user.u_start = { tv_sec = 2007 Jun 11 00:00:00 tv_nsec = 0xcf77e0 } p_user.u_ticks = 0x191b148 p_user.u_comm = [ "bgscollect" ] p_user.u_psargs = [ "bgscollect -I noInstance -B /usr/adm/best1_7.3.00" ] <== use the value here p_user.u_argc = 0x5 p_user.u_argv = 0xffffffff7ffffc98 ... >
Monday, 29 July, 2013 07:47 Written by Brian B
Gary Koopman, Group Vice President of NAS Commercial Channels hosts Steve Collier, Founder and President at Collier IT to discuss selling Oracle hardware and software.
President and Founder of Collier IT Steve Collier speaks with Gary Koopman, GVP Oracle NAS Commercial Channels about decisions and positioning Collier IT took to make a successful transition from Sun Microsystems to Oracle.
Interesting listen
Sunday, 28 July, 2013 17:00 Written by Brian B
At times, I’m asked for my opinion as it relates to certification exams and the necessity in completing them.
I’m a proponent of them. They validate your expertise and they provide proof that you carry an advertised skill set.
They aren’t an end all, be all, method of displaying your skill set, but they are another tool to evaluate an employee, a potential employee, or the ability of a company to demonstrate requirements to complete a contract.
I’ve not hired an employee because they may or may not carry a certification, but they are used in my grading criteria. As a CTO at an Oracle Platinum Partner (Collier IT) we have many requirements placed upon us by Oracle to prove to them that we carry specific skill sets. One of Oracle’s many grading criteria include certifications and specializations that are assigned to both the company and its employees.
So yes, if the skill sets between candidates are close, certifications will assist in my hiring one over another.
We have a number of Engineers and Senior Sales Account Managers that carry a similar certification Résumé as do company management and ownership.
Saturday, 20 July, 2013 16:13 Written by Brian B
As a means of introduction, I thought it best to introduce my employer and the services that it provides.
Collier IT is an Oracle Platinum Partner that is heavily invested in the entire Oracle Stack.
Collier IT is also an Oracle Approved Education Center and a Oracle Approved Education Reseller. The facility is located in Roseville, MN.
As of this post, Collier IT carries the following Oracle Specializations:
Exadata Database Machine Models X2-2 and X2-8
Oracle Database 11g
Oracle Database 11g Performance Tuning
Oracle Database 11g Data Warehousing
Oracle Linux 5
Oracle Real Application Clusters 11g
Oracle Solaris 10
Oracle Solaris 11
Collier IT partners with other industry leaders to bring an entire IT solution to your IT business needs.
The Collier IT Engineering Staff carry the following certifications / specializations and are able to assist in the deployment, management, or troubleshooting of Enterprise IT solutions.
• Oracle Exadata 11g Certified Implementation Specialist
• Oracle Database 10g Real Applications Clusters Administrator Certified Expert
• Oracle Exalogic Elastic Cloud X2-2: (Install and Maintenance)
• Oracle Data Warehouse 10g RAC Administrator Certified Expert
• Oracle 11g Performance Tuning Implementation Specialist
• Oracle Database 11g Data Warehouse Certified Implementation Specialist
• Oracle VM2 for x86 Certified Implementation Specialist
• Enterprise Linux: Fundamentals
• Enterprise Linux: System Administration
• Oracle Virtual Machine
• VCP5
• Oracle Database 10g Real Applications Clusters Administrator Certified Expert
• Solaris Core
• Workgroup
• Enterprise
• Datacenter
• E10K
• Midframe
• Cluster 3
• LSC Storage Engineer (SAM-FS / QFS)
• Highground (Storage Resource Manager)
• E25K
• StorageElite
• SE3000 Storage Engineer
• SE6000 Storage Engineer
• SE9900 Storage Engineer
• Sun Certified Data Management Engineer with VERITAS Volume Manager
• Oracle Sun Certified Network Administrator for Solaris 10 OS
• Oracle Certified System Administrator for Sun Cluster 3.2
• Oracle Sun Certified Security Administrator for the Solaris 10 OS
• Oracle Sun Certified System Administrator for Solaris 10 OS
• Sun Certified System Administrator for Solaris 8
• Sun Certified Network Administrator for Solaris 8
• Oracle Solaris Implementation Specialist
• Sun Chip Multithreading (CMT) Servers Implementation Specialist
• Sun SPARC Enterprise Entry-Level and Midrange Servers Implementation Specialist
• Sun SPARC Enterprise High-End Servers Implementation Specialist
• Hitachi Data Systems Certified Professional – Enterprise
• Hitachi Data Systems Certified Professional – Modular
• Operating Systems: HP-UX, Linux, MS Windows Server 2008,MPE and Solaris
• HP Storageworks Data Protector
• Command View EVA & TL
• Symantec Backup and Exec & NetBackup
• McKesson Archival, Storage Software
• FileNet, CommVault
• Lefthand Networks
• SAN Switches: Brocade, Cisco, McData, Nishan, QLogic
• Routers/Hubs: Cisco, HP ProCurve, Nortel, 3 Com
• HP Proliant, HP P Class & C Class Blades
• HP Netserver, HP 9000, HP 3000
• HP Storageworks XP P9500/24,000/12000/1024/128/512/48/256
• EVA 8000/6000/4000/8100/6100/4100/4400/5000/3000
• VLS (Virtual Library System), 3Par, D2D
• HP Storageworks MSL, ESL, ESL-E, & EML series Tape Libraries, & SSL
Please contact Collier IT at info@collier-it.com if you want any additional information regarding their services.
And please remember, the blog posts are mine and do not necessarily reflect the opinions of my employer nor any of their business partners.