.NET Diagnostics Expert

Last week we published the final module of the .NET Diagnostics Expert Course:

I’m excited and happy that it’s finally available. But I’m also relieved as there were times when I thought it would never happen In this post, I want to share some details about the course and why I decided to make it.

How it all started

It was my plan for a long time to publish a course on .NET diagnostics. However, I have never found enough time to focus on it, and apart from handling a few free training sessions for the devWarsztaty initiative, I hadn’t made much progress in realizing it. So what changed last year? Firstly, in late summer, I decided to leave Turbo.net and focus on learning and my private projects, most importantly wtrace. I finally published https://wtrace.net and started working on a new wtrace release and other tracing tools. Three weeks passed, and while talking with Szymon, the idea of a .NET diagnostics course emerged. I thought that the timing would never be better, so I drafted the course plan, and Konrad published the https://diagnosticsexpert.com site. Then I prepared two webinars, and Dotnetos started the presale of the course at the beginning of December. I was under stress that the interest would be low and not many people will find the subject interesting. The presale, however, went very well (thank you all who put trust in me!), giving me some relief.

Making-of

And then the recordings started. At first, I tried recording the lessons as I was presenting the slides. It did not work very well, and every lesson took me hours to complete. My wife took pity on me and browsed tutorials for starting YouTubers to find that they often use transcripts. And thus, I began writing transcripts That made the lesson preparation longer, but the recording time was much more manageable. Unfortunately, transcripts did not work for DEMOs, so they still required hours in front of the microphone. Another problem with DEMOs is that they are not always entirely predictable. Sometimes, during the DEMO, I received an output I haven’t thought of or, even worse, discovered a bug in the diagnostics tool. Once I had raw materials ready, Konrad and Paulina reviewed them, and Andrzej processed the videos. Konrad then watched the final recordings, making sure they do not contain any repetitions or other issues.

There were various problems I hit along the way. In the first month, my camera broke. And, as I didn’t want to lose any time, I created a temporary solution:

The positive side of it is that you may see that the course has solid fundaments! I also spilled tea on my laptop (thankfully, it survived after drying) and corrupted my drive when testing diskspd (entirely my fault – I shouldn’t be doing it after midnight). Fortunately, the backup worked.

What’s in the course

Having covered the course making, let me describe to you the course content. When preparing the lessons, I focused on practicality, presenting techniques and tools which you may employ to diagnose various .NET (and not only) problems. There are 11 modules in the course:

• Module 1, “Debugging,” describes the building blocks of a debugger, symbol files management, and features of Debuggers, both managed (VS, VS Code) and native (WinDbg, LLDB)
• Module 2, “Tracing,” focuses on Event Tracing for Windows, Linux tracing APIs (perf, LTTng, eBPF), and .NET Event Pipes
• Module 3, “Windows and Linux diagnostic sources,” covers various applications to monitor processes on Windows and Linux (including ps, top, htop, Process Hacker)
• Module 4, “High CPU usage,” describes ways of troubleshooting CPU-related issues, including CPU sampling, thread time (clock time) profiling, and .NET Profiling API
• Module 5, “Deadlocks,” covers techniques for troubleshooting deadlocks and waits using memory dumps and trace-based wait analysis
• Module 6, “Network issues – TCP, UDP,” concentrates on monitoring TCP and UDP connections and troubleshooting various connectivity issues (including slow server responses, dropped connections, or intermittent network errors). Apart from typical network tracing tools such as Wireshark or tcpdump, we also use .NET network traces.
• Module 7, “Network issues – DNS, HTTP(S),” is about troubleshooting issues in higher layer protocols: DNS, HTTP, and TLS. Apart from system tools, I also cover ASP.NET Core and Kestrel logs. In this module, we also implement and use various network proxies to intercept and modify the traffic.
• Module 8, “Application execution issues,” targets assembly loading issues, exceptions, and production debugging techniques (including system image preparations, automatic dump collection, and remote debugging)
• Module 9, “Miscellaneous issues,” describes diagnosing memory, File I/O, and some other issues. It also lists final tips and tricks for troubleshooters.
• Module 10, “Diagnostics logs in the application,” shows ways of how we can interact with the debugger from within the application and how we may publish custom performance traces.
• Module 11, “Writing custom diagnostics tools,” covers usage of ClrMD, Diagnostics Client, and TraceEvent libraries to write our custom diagnostics tools.

As you maybe noticed, the first three modules present general concepts and tools, while the subsequent six modules focus on various diagnostics cases. Finally, the last two modules are about implementing code for diagnosing purposes. Each module ends with homework exercises. I wanted them to be challenging and resemble problems I observed in the production. And, as it’s a practical course, I spent a lot of time in DEMOs.

Although the course concentrates on .NET, many presented techniques could be employed to troubleshoot native applications or even system problems. I also believe that debugging, tracing, and reading source code (if it’s available, of course) are the best ways to learn how libraries and applications function.

Final words

In the end, I would like to thank Dotnetos, in particular, Konrad, for the endless hours he spent reviewing the videos and slides. It was a huge and challenging project that occupied me for the last six months (two months more than initially anticipated :)). But I’m happy with the final result, and I hope that those who decide to participate in it will enjoy the prepared materials.