Nürnberg and München are just 175 km away from each other, a bit more than 1 hour by train. Still I felt a post-conference blues already when I boarded the ICE train yesterday to head back to the south. It was an awesome conference – again. Except for one thing which I will write about below. So here’s my report on DOAG 2019 – Recap and a personal complaint.

Photo by Philippe Mignot on Unsplash
DOAG 2019 – What an agenda
I knew it before already. But my VP, Roy Swonger, visited DOAG Conference in Nürnberg for the first time. And a few days before he sent me his schedule with a comment saying “Wow, what an agenda. It was sometimes really hard to pick between several interesting talks in parallel.” And Roy selected only from the talks which were supposed to be given in English or get simultaneous translation in the big rooms. I can choose from even more – and I can confess: The agenda was awesome. And the talks I went to were excellent, too.
Of course, thanks to the fine DOAG team and everybody who was involved in making this such a great conference. Call me patriotic as DOAG Conference happens in my hometown, but it’s a pleasure every year for so many years now. Well done, folks! Thank you!
Our presentations
Honestly … THANK YOU for coming to our talks. I’ve had the last slot on the first day at 17:00h, and I was a bit anxious before whether most people will be tired and just enjoying a drink at one of the exhibition booths instead. But you really surprised me. Wow! “Tokio” tends to feel a bit empty but you made it a cozy place on Tuesday evening. And thanks to Daniel Overby Hansen from SimCorp for inviting me to be with him on stage on Thursday morning. We didn’t expect so many people being up so early after the great party on Wednesday night.
In case you visited one of our 3 talks, please find the slides here:
- AutoUpgrade to Oracle Database 19c – Tips and Secrets
Tuesday, Nov 19, 17:00h - Diagnosing Oracle Database Upgrade Issues – Roy Swonger
Wednesday, Nov 20, 13:00h - Supercharging Database Upgrades – Daniel Overby Hansen, SimCorp A/S
Thursday, Nov 21, 9:00h
And here’s the video (no audio!) for the AutoUpgrade talk on Tuesday showing the “resume”, “abort” and “restore” command options of AutoUpgrade:
Finally … a complaint
Actually this year something has happened which made me angry and frustrated. Usually Tim Hall is the grumpy but always fair complainer, this time it is me.
Since June 2019 people from our team and I exchanged 40 (!! – I counted them) emails with a customer regarding AutoUpgrade. Actually it is not our job to solve SRs. But in this, as in many other cases, we supported a customer directly. The customer found issues with AutoUpgrade. And it is in our own interest to fix things as quickly as possible. Which we do – what many others can confirm. My team mates, in this particular case Hector and Byron but also all the others, do an awesome and tough job. And I’m really grateful to be part of such a fantastic team (I really mean it).
On Wednesday, I had one of the many “upgrade/migration” discussions at DOAG with a customer I know now for a decade. We’ve been at OOW on stage together a while ago. We know each other very well. And he told me that the customer we helped a lot with AutoUpgrade complained on stage about our tool, the support coverage and bugs per se.
Of course, this is a tech conference. And it’s the place to name things which are not working well (yet). That’s the reason why we all like these conferences. If you’d be interested in marketing-only, you could watch a youtube video instead of paying for a conference pass and be away for days from family and work.
Yesterday night I downloaded the slides of this talk. And I realized that my long-year customer’s impression may have been right. It made me really sad.
Let me clarify a few things.
At first, as I wrote above, we are “Development“, not “Support“. If you need to complain about Support, complain to Support or your sales contact. There were Oracle Support people at DOAG at the Oracle booth. But if our team gives you a several-month-long coverage assisting you directly, personally, reply overnight, send you intermediate version of AutoUpgrade with fixes others don’t have access to yet, then PLEASE go on stage and tell everybody about it. We don’t expect you saying “Thank You” necessarily – but at least tell others.
The SRs we ask you usually to open are mainly used to exchange files. We are not allowed to share files via the typical file sharing services. Hence, an SR is the right way. And this way, things for other customers can be documented. It’s not fair to complain about slow SR response when we supported you in a 1:1 coverage directly.
Furthermore, it is not – really not! – OK to complain about functionality of the tool which is in beta stage. Which we clearly explained and described as beta. Beta functionality obviously is not in the documentation. Never! And yes, it’s on my blog. It will be even in the next version of the lab for testing purposes. We love when you test. And we’ll fix things when you find them. But beta is beta is beta. Period. We made this VERY clear.
I deal a lot with customers in reference projects. Which means usually that I deal a lot with SRs, and of course with bugs, too. Not every team at Oracle which will give you such direct and immediate support. It is tough for us because it would be much easier to hide behind the wall of MOS and the bug database. Of course, I understand that it is hard for you, too, when you find issues. But there wasn’t a single one where there wasn’t a fix provided at the speed of development-light, or at least a workaround given.
We are thankful when you find things because we would like this tool to fly perfectly well and ease your daily work. We never promoted it as a flying cow. In case you wonder what I mean, watch this video. Then you’ll understand. We are honest with you. We talk about the things not working perfectly well. But we tell you as well that we are working to improve the tool. And we give you workarounds and deliver fixes more rapidly than for any other product I can think about. We even support customers directly who haven’t signed a reference agreement with us. As we did in this particular case, with plenty of extra effort.
What I would have expected is respect and fairness.
I can promise that a future version of the tool will fix the issues you found. But we can’t release a new version every night as we want to test it, too. System testing does not complete just overnight with the massive numbers of tests the tool undergoes.
I would like to say “Thank You” for an outstanding 1:1 coverage to my team mates from the Database Upgrade Development team.
And something very positive at the end:
Thank you so much to the people who came by at DOAG and told me that they used AutoUpgrade a week ago to upgrade 50 databases unattended. And to those who said, it’s such a tiny little tool but gives such a great improvement in the work with the database. Thank you for coming to our talks and giving us constant feedback. This is really important to us – and I don’t write this just because it reads nicely.
A big thank you from the Database Upgrade Development team to you!
–Mike
So sad and unfair.
But let me tell you that you are my primary resource for Oracle knowledge and that I’m very thankful for the effort you make for us DBA’s to make life easier 🙂
Best regards,
Jan Willem Ellerman
Thanks Jan Willem!
Cheers,
Mike
Hi.
I don’t know the specific issue here, but more generally it’s a difficult balancing act at times. There is limited time to speak and you sometimes feel the need to express an opinion (product X is good/bad) and after the fact you sometimes cringe thinking, I wonder how that case across? I remember saying, “Tom Kyte is always banging on about…” in one talk. After the talk I thought, that sounded really disrespectful and hated the thought of how some people may have taken that statement.
The “beta” thing is also easy to misinterpret these days. Some companies (Google, Epic Games etc.) keep products listed as beta long after they are in general use and really should be considered GA. Depending on when someone has come to this tech stack, they might be thinking of beta in that sense, and so expect more from it than they get. That’s no excuse, but I can understand a difference of opinion about what beta means to different people.
The “support” issue is a tough one. I recently asked an Oracle PM if I should stop bugging them (DMs on Twitter) and just raise SRs instead, and the response was, “No. Keep sending them to us. If you raise them as an SR we will never see them.” I’m sure that’s an exaggeration, but it doesn’t fill you with a lot of confidence.
One last thing. The judgement of good/bad is really dependent on where you sit in the value chain. I have some really good interactions with people in the business, but overall they will say, “IT Services are crap!” Even if some bits of the chain are good, it doesn’t stop the whole experience being crap. It can be a bitter pill to swallow at times, but sometimes the customer is correct. It might not be your fault, and you might not like it, but it doesn’t stop it from being the truth. Like I say, I can’t comment on this specific case, so please take this in context. 🙂
Cheers
Tim…
PS. Thanks for the shout out… I think… 🙂
Thanks for your thoughts, Tim!
I don’t want to make this bigger as it is – I just feel a bit sad for my team mates who did an awesome job of helping directly, and not playing the “please update the SR at first” game. As you know (and wrote above already) many of us PMs are fine if you mail us directly. That’s great. And if any of us asks you to open an SR “in addition” to keep track of things, you wouldn’t deny – and of course not complain afterwards about slow support response times in this matter when you get direct coverage from development. There are way more things to mention – but as I said, I don’t want make it bigger as it doesn’t solve anything.
Well, and we all know that IT services are not valued enough. I just guess, the accountant in my company would agree that her or his job isn’t valued enough either.
Cheers,
Mike
You’re bound to find a few ungrateful b*****s Mike.
You know you do a fantastic job. We know you do a fantastic job. Just keep going.
Regards,
Stuart Barkley
Thanks Stuart!
Cheers,
Mike
People which attended the session have heard:
– the support is incredible good – with you and with the developers
– the tools evolution is very fast and the fixes are released incredible fast
– it’s very reliable to also administrators faults (like not using admin command prompt on Windows)
– logs are best in class
– there are things that still can happen, e.g. you can hit database bugs, small things not working at the tool (so check official documents also) if it could be a bug or if it is beta
– wished this kind of tool was released years ago.
So don’t trust some screenshots only…
Hi Joerg,
people who haven’t been at the talk may download slides only – which for instance I do for talks I couldn’t attend. DOAG conference has approx 2000 attendees every year. I can only sense how many people where in the talk I’m talking about – so potentially 1900 others may only see the slides. Hence, slides matter for many reasons. And slides speak for themselves.
Thanks for your feedback – cheers,
Mike
Hi Mike
I totally understand you but i can only say that you and your team doing an excellent job! I know that in all projects that we did in the last years you did always more than we can except as a customer. I’m happy to have you on board for the next upgrade project and i hope there will follow a lot more
Thanks a lot, Alain!
And you know, I look very much forward to work together with you and your fantastic team in the future 🙂
Cheers,
Mike
Hi Mike,
I can’t comment on the specific talk because I don’t know about it. Personally, I’m always going first to discuss with the involved parties before making issues public. So, concerning the way, the complaints were done, I completely agree with you.
And please don’t get me wrong, you and your team is doing a great job by developing AutoUpgrade. I like the approach of this tool very much and testing it myself a lot.
But for my opinion, one of the main reasons why people take the AutoUpgrade tool as “GA” and not as “still in development” is, that Oracle is massively pushing it at the basically “most suggest way to go” for upgrades. Just check the Upgrade talk made by Roy Swonger on the Quest Experience Week, for example.
If I missed something or if my understanding is wrong, please correct me.
Looking forward for the next update on AutoUpgrade tools.
Regards, Silvio
Hi Silvio,
autoupgrade is GA – we improve specific topics such as the RAC upgrade or Data Guard handling or – as in this case – the migration from non-CDB to PDB.
These are either documented with specific extra tasks (such as for RAC and Data Guard) you have to do manually. Or it isn’t documented – which is the case here as “target_cdb” is neither documented nor explained. It is only on my blog. Everybody is free to use and test it – even thought the autoupgrade from october 24, 2019 will give you an GRP error. That’s why I provided in this specific case a version which doesn’t stop due to the GRP issue but handles it correctly.
Cheers,
Mike
Hi Mike, Glad with all your work on upgrade issues! Can you tell me how I can report a
bug I just found in the latest autoupgrade.jar version:
$ $ORACLE_HOME/jdk/bin/java -jar autoupgrade.jar -version
build.hash 04a58df
build.version 20191125
build.date 2019/11/25 17:49:18
build.max_target_version 19
build.type production-19.7.2
Thanks
Peter Jansen
Hi Peter,
please open an SR, upload the entire log tree and the alert.log.
Optionally you can mail me the SR number and I will share it with the team.
Cheers,
Mike