I wrote about Windows RT some years ago without really thinking about the underlying type system design in a multi language environment. Never occurred to me how Microsoft defines the underlying structures developers will use to define their own APIs. Microsoft Interface Definition Language (MIDL) 3.0 is a simplified syntax for declaring those Windows Runtime types inside Interface Definition Language files (.idl). The chief concern then is how you end up projecting the fundamental building blocks that each language needs, these would include:-
- Classes
- Interfaces
- Base Types (32, 64 BIT float; 8,16, 32,64 BIT INT; strings; GUID, etc)
- Structures
- Enumerations
- Delegates
These engineers started by supporting the usual language suspects (C#, Visual Basic, JavaScript) but we now also have consistent projects for things like Objective-C and a pure standards based C++ (avoiding platform specific extensions).
Anyway, this is a place holder for me to quickly access this brilliant interview of Larry Osterman on MIDL 3 by Nikola Metulev. Give it a watch, totally worth your time!
I was reviewing a hang dump for really important service with WinDbg recently, externally it was obvious that our responses were extremely slow and just a casual look at the CPU showed it was hovering between 70 and 90%.
Whenever I see sporadically slow running applications where the data layer is not causing issues I tend to first look at the Thread pool. The command is as follows:
!threadpool
Thread pool command shows the CPU and the threads that are currently in use:
CPU utilization: 81%
Worker Thread: Total: 1 Running: 0 Idle: 1 MaxLimit: 32767 MinLimit: 4
Work Request in Queue: 0
--------------------------------------
Number of Timers: 1
--------------------------------------
Completion Port Thread:Total: 2 Free: 2 MaxFree: 8 CurrentLimit: 2 MaxLimit: 1000 MinLimit: 4
Now the machine.config is where MaxLimit of 32767 threads and we are currently using none of them, which means our thread pools are not exhausted or over taxed (that max number is weird by the way).
The CPU is infinitely more interesting in this case, because as a rule if the CPU is over 80% new threads do not get allocated. It was also really interesting that we happened to catch this hang dump at exactly 81% because the garbage collector artificially also raises the CPU to 81% to prevent thread allocation, obviously if we cannot allocate threads we cannot get work done.
We can run the following command and look at the stack of each thread:
~* kb
0cbeeae0 7c822124 77e6baa8 000003ec 00000000 ntdll!KiFastSystemCallRet
0cbeeae4 77e6baa8 000003ec 00000000 00000000 ntdll!NtWaitForSingleObject+0xc.
0cbeeb54 77e6ba12 000003ec ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac
0cbeeb68 791fed5c 000003ec ffffffff 791fe901 kernel32!WaitForSingleObject+0x12
0cbeeb74 791fe901 00000001 025372e8 00000001 mscorsvr!GCHeap::WaitUntilGCComplete+0x4f
Ah see! This particular issue is being caused by a GC (WaitUntilGCComplete) so the problem may be that we are being forced to perform too many garbage collections, maybe because of some poorly written code. To confirm this theory over time I started looking at performance counters, specifically ".NET CLR Memory->% Time in GC" which happened to be way over 60%, not good.
So I wanted to document a couple of interesting points that had not occurred to me about hosting ASP.NET Core application. The reason for the disconnect, I believe, is because I have only ever known a web world that was housed within a modern IIS context.
web.config
Your web.config file was the primary mechanism for your app to interact with IIS, with it we do things like define Forms Authentication, URL Rewriting, define Http Modules and so much more. However almost all of this has gone, your typical configuration now ends up in Startup.cs configured with Middleware. With this still being IIS though if you deploy from Visual Studio you will get a stripped down web.config for free and it typically looks like this:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="dotnet" arguments=".\DasBlog.Web.dll" />
</system.webServer>
</configuration>
So what are the important parts? The handler called AspNetCoreModule is the connection between your application and IIS, in fact I have noted before that if you look at task manager you will notice two process (w3wp.exe and dotnet.exe) where once there was one (w3wp.exe). Running outside of the IIS process is a significant architectural change and has significant repercussions that I am sure I will talk about in the future. Finally the aspNetCore element defines your assembly that will be run by dotnet.exe.
app_offline.htm
FTP deploying my asp.net core application was a genuine hassle for a long time because all you core dlls are in use and it does not provide a shadow copy. In order to push new files out to your server you really need deploy the app_offline.htm file first. I have been using IIS for a long time and I barely remember this, however, this file shuts your app down gracefully and once shutdown you get the chance to copy files. Even performing soft resets was not helpful. WebDeploy in Visual Studio essentially does this process for you automatically and then deletes the file when it is finished.
It took me a long time to realize what was happening, I hope this helps someone.
I got asked an interesting question about that would potentially apply to lots of products and thought it would be an interesting question to talk about here. It went like this:
Most sites have a sign in mechanism that takes passwords from end users via an ASP.NET textbox. We go onto to store that password in a string and, by definition then , strings are stored in managed memory on the heap, and if one happens to take a hang dump the password will be in clear text. It would also be true to say that we do not control exactly when the clear text password gets garbage collected. The question then is would immediately storing user entered passwords in a System.Net.NetworkCredential and/or System.Security.SecureString in ASP.NET provide any worthwhile security benefit.
My initial answer started as a "maybe" and finally morphed into a fairly strong "no".
To be clear there are lots of situations where managing sensitive information with SecureString is critical, let's look at a canonical command line example:
public static void Main()
{
using (var securePwd = new SecureString())
{
ConsoleKeyInfo key;
Console.Write("Enter password: ");
do
{
key = Console.ReadKey(true);
if (((int)key.Key) >= 65 && ((int)key.Key <= 90))
{
securePwd.AppendChar(key.KeyChar);
Console.Write("*");
}
}
while (key.Key != ConsoleKey.Enter);
Console.WriteLine();
try
{
Process.Start("Notepad.exe", "MyUser", securePwd, "MYDOMAIN");
}
catch (Win32Exception e)
{
Console.WriteLine(e.Message);
}
}
}
You can see in this console app I am appending each character directly SecureString so there are no strings floating about. I am also taking advantage of the using pattern and ensure that SecureString object will be disposed of quickly or I could have explicitly used the Dispose method to remove the memory more directly. Would such a pattern help in ASP.NET? I am inclined to think not. What is important to remember here is that strings are immutable (unchangeable) so after you create one for any reason anything you do to it ends up creating a new string for you to manipulate. With that in mind consider then how you get from w3wp process to your specific app domain in the following image.
If the information you are getting off the wire is all about strings to begin with whatever you deserialize that string into after the fact becomes the equivalent of closing the barn door after the horse has already bolted.
This is my loosely held opinion I would love for someone to present a contrary opinion, even if it covers a small use case where SecureString might be useful when handling passwords in this way.
Most of us are unprepared for catastrophic failure in our application, that is, the kind of failure that is so pernicious that even our precious log files have nothing useful to share. In fact to get adequate data during these moments we usually need the conditions to be replicated exactly and then hope we are adequately prepared to monitor the details we missed.
For the last couple of years I have been asking our clients to prepare hang dumps for me under a variety of circumstances so that I can ensure I have an accurate portrait of a given situation. In years past we would normally do this using ADPlus, however, there are few clients who are willing to install a subset of the the Windows SDK during or after production outages.
More recently then I have been pushing the more straightforward copy paste installation of ProcDump. It is a popular command-line utility that serves as a general purpose dump utility. There is a whole list of ProcDump parameters to become familiar with but I wanted to share the combinations that I find myself using repeatedly when collecting hang or crash dumps under a variety of circumstances.
Immediately take a full hang dump:
C:\Dev\Procdump>procdump -ma w3wp
Take a full hang dump when the application crashes:
C:\Dev\Procdump>procdump -e -ma w3wp
Write three full hang dumps 10 seconds apart:
C:\Dev\Procdump>procdump -ma -s 10 -n 3 w3wp
Take a full hang dump when the w3wp process exceeds 40%:
C:\Dev\Procdump>procdump -c 40 -s 5 -n 3 w3wp
Take a full hang dump when the overall CPU exceeds 70%:
C:\Dev\Procdump>procdump - ma w3wp -p "\Processor(_Total)\% Processor Time" 70
Write a full hang dump when an exception contains the word "IO" e.g. "IOException":
procdump - ma -e 1 -f IO w3wp
All the previous hang/crash dump strategies are concerned with w3wp, the process associated with the application pool in IIS, however, you can update this for any process name or number you like.
Much to my disappointment PBS does not have a Xbox One app and they have a lot of British programming that I would miss out on. It feels like an awful oversight, and considering how much emphasis Microsoft has pushed fixing app gaps across their platforms this is especially vexing. Your alternatives are to log in to PBS via the edge browser on Xbox One or Cast to your Xbox from a device you own with the app.
For some reason I thought that the Xbox sat on my network waiting to be projected to from every Windows device on my network. I have found that this is simply not the case.
To cast video or images to the Xbox One from Windows (or any device) you need an app on the Xbox that will allow you to make the connection. If you are a part of the Xbox Insider Program then you probably have access to the Wireless Display (preview) app. For most of us the insider programs represents a bit too much instability and risk if this is the case you should consider using AirServer for Xbox One.
- Install AirServer
- Launch the application and click on Play trial
- On your Windows device go to your Notification Center and click on Connect
- Your Xbox name should be listed in the available devices click on it and you are connected!
I have been tasked with sharing some of things I have learned about .NET applications and how they manage memory and sharing it with the team with intent of ensuring we are better prepared to investigate issues in production and that we are not creating new ones. Concepts like Garbage Collection (GC) exist to shield us from unnecessary complexity unfortunately in doing so it can mask ideas that are important.
For me, working with a high availability custom app server and IIS, healthy memory management is incredibly important, so delving below even good and robust abstractions is critical. I have spent the last few weeks looking for the right combination of videos, articles and topics to kick of this learning exercise. The first one is Everything you need to know about .NET memory by Ben Emmett.
I am encouraging my colleagues to start here, and after watching this video to consider and describe the following concepts:
General information about memory (below the .NET GC abstraction)
Each process has its own and separated virtual address space (we never manipulate physical memory directly), all processes on the machine share the same physical memory (plus the page file). The amount each process can access depends on the whether you are using a 32 or 64 bit machine. 4 GB (2GB user mode) for 32-bit Windows, 16 terabytes (8 terabytes user mode) for 64-bit Windows.
If you are writing managed code the GC is the one who allocates/frees virtual memory on your behalf. Even though this is handled for you, for the sake of speed, virtual address space can and will get fragmented, so it is possible and likely that you get free blocks between used address spaces, otherwise known as fragmentation.
Virtual memory can also be in different states:- free, reserved and committed.
- Free is … well free.
- Reserved is saying “I want to make this region of memory available for use later”. After reserving the space that block may not be used for other requests.
- You still cannot use reserved memory until it is committed. Committed memory has been paired up to the available physical memory so you can actually put data in it.
You can get out of memory if you are running out space to reserve, or space to commit. In the age of 64bit processors out of memory is most likely to mean unable to commit (out of physical memory rather virtual memory).
Programming for long periods is like walking a high tight rope for a really, really long period of time, or at least I imagine it is. As you go further and further out on the rope any break in concentration or just simply stopping becomes more and more difficult. I have described this in the past as tumbling down the rabbit hole but even that does not quite capture the sheer force of will it takes to successfully take an idea from concept to useful working application.
Programmers, I am sure, are not special in this regard there are I assume many occupations that require hours of long, intense concentration. This is especially true when the fundamental concepts are so amorphous that you are forced to construct a kind of memory palace (method of loci) in order to successfully navigate ideas, consider redesigns or fix complex problems.
For the developer this complexity can extend through several abstraction layers (OS, network, or memory), across multiple devices and then simultaneously within multiple threads. All these theoretical considerations can get quite overwhelming which is why so many developers insist on silence, headphones or music of a particular tempo and type. Personally I cannot listen to technical podcasts while I develop software, it's like I have single threaded access to the technical portion of my brain.
Unwinding after programming
My advice to anyone who is forced to do perform this kind of mental gymnastics over long periods of time is to engage in countervailing exercises that allow you to almost wholly inhabit the moment. For many people I work with going for a walk away from your workspace is the simplest and most accessible way to clear your mind of programming fog.
Here are a few other suggestions:-
- Playing video games
- Cooking
- Watching TV
- Listening to music
- Talking to non-technical friends
- Closing my eyes and slow your breathing
- Exercise
The common theme for any anti-programming exercise is to simply be present. Actions that force you to consider things in parallel tend to play on the same exhausted brain cells, I find that some video games do this regardless of my best intentions.
For me personally playing a musical instrument does more to settle my restlessness than any of the previous suggestions. Especially when I am playing a piece of music I enjoy and know well enough to play without additional practice, and more importantly, without doubt or fear of failure, remember the whole point is to unwind. When I play music in this context it is usually error prone and experimental and leaves glaring mistakes hanging in the air for all to hear, and you know what that it is fine, because the whole point is to enjoy the music and the moment in real time.
I know that a lot of people do not have musical inclination but I would encourage you to pick up an instrument and pluck around. Getting good might require time but having fun can happen immediately. Just think of someone playing air guitar the enthusiasm and the joy is exactly the point!
So this is me, what do you do to relax after marathon programming sessions?