It has nearly been 12 months of what might be called the moustache growing competition of 2020. Also known as work from home life. I have begun to see how people can easily become accustomed to working for them selves. Being their own boss, and working whenever, or wherever they want. I’m still up in the air on the whole conundrum on deciding if work from home is for me. But the associated “downtime” has allowed me to take more time in poking around and sorting out my systems. Luckily I don’t have a huge amount of systems, but under my domain of pretty little flashing lights a new system had peaked my intrigue. It had been there for a while but I wanted to know more. I wanted it to tell me it’s secrets. What did this system do? Why was it created? I wonder how much of a scream test it will produce if I turn it off? You know, the usual thoughts one has in their head. What follows is a little retrospective mainly for myself, with the absence of any time frames.

Backstory

What you need to know is that we have a very stringent rules that need to be adhered too, also known as following some sort of standard. I love a naming convention that you can easily tell what a system is at a glace. At least internally. Sure if you have external systems call them what ever you want, like PRODLBUS-01 (production, load balancer, United States, 01). What? I’m not an animal! So when I stumbled across this system with a tag: 'name': 'vm-app-adsc'. So as you can see from this name, this server is a VM, and runs an application called “adsc”. Simple, concise, and also slightly redundant. But this isn’t a tale about a server I provisioned, this is an inherited server. The best kinds.

If you are privy enough to be ensnared by the temptress called Active Directory, you’ll likely be thinking it is related to Active Directory Services Connector. I sadly, am not well versed in this world as I’ve been lucky enough to enjoy an MS-AD free tech life so far, and lived on the planet of oAuth instead. This was where my mind lead me too as well, because why would someone follow our naming conventions if it wasn’t true. Thus a quick duckduckgo search later proved that AWS has an Active Directory Connector service to expand your AD into AWS. (FYI, it’s a hosted service, not an app you can run on a box. Just so you don’t get the wrong idea). Thinking I was on the right track, I sent off the required forms to get approval for RDP access. It was time to connect to this windows box and figure out how this connector service is setup.

What information follows is probably incorrect, might be made up to protect the innocent, and highly edited to induce comedic effect.

Investigation

Approval has been sought, and now to connect! Like all of the internet, the answer of “How shall I connect” is answered by using an application and the hostname of the server. When computers are connecting to active directory, the magic of active directory ensure they have an associated DNS entry in the DNS service. This server was not an exception to that rule, or the rule of active directory’s “just works” methodology. The dns entry: vm-app-adsc.ccc.example.com existed and had an associated time stamp of within the last month, so it was pretty accurate. Thus providing me with a quick an easy reason to not look into our aws account, and finding the associated IP address[1].

A quick remote desktop in confirmed my original suspicions in that Azure Active Directory Connect Sync app was running. So, why did this box need to be run inside AWS? At some point, everything is setup for a reason. So if we needed the Azure connector here, it was possible that we needed to link Azure and AWS together. Why would that be the case, when AD forests exists?

Forests are beautiful things. All year round that show us beauty in different forms. AD forests, do not deserve to use the same noun. For those of you not in-the-know, AD forests are a way to separate a myriad of objects. Each forest can be completely independent of each other (just like a real forest). Birds and animals (computers and people) can migrate from one forest to another and still live, work and play without any issues. So I learnt about how forests were formed, how the DNS forwarding works, and increased my general understanding of AD two fold. What I did not find was an explanation of why a service needed to be extended cross cloud provider.

It seems from my research, that Azure AD needs the application when you are hosting your own AD server. This creates the bridge needed so you can handle the authentication of Azure AD services, against your directory. Frustrated with my lack of progress I go full diagnostic mode on the server. Mentally ticking off a checklist and confirming the use-case of this server. I end up confirming I am connecting to the Azure AD Connect server, you know, the one that is actually in azure. :facepalm: Checking the local IP confirmed it was in the Azure subnet. It’s time to celebrate! A problem has been found. That problem? The user is connecting to the wrong server. :( But that does not make sense, unless there is a server in Azure and AWS called the same thing. Having two servers with the same hostname is not possible under Active Directory (as far as I know). :spooky: Did I just break AD?

Relief

So after feeling foolish, I close the connection to the Azure ADSC server and evaluate where I went wrong. I go back to the drawing board as it were, and login to AWS to find out the IP address of this phantom server. A quick RDP via an IP instead of using the hostname resulted in a certificate error, and lo and behold, access to a different server. A server that does not have ADSC installed. Now it’s time to party. It certainly was joined to Active Directory, it was just under a completely different computer name. Who would have thought! First order of business, renaming. 'name': 'vm-app-HostName'. So what was it doing you might ask? When you choose to host your own directory instead of buying into using a completely hosted service such as Azure AD or AWS AD. You still want to expand your directory into them as it will allow Azure & AWS services to authenticate against your directory. To facilitate this, you need to create extra servers and services that act as the glue to make that happen. For Azure AD that is the ADSC application. For AWS you can straight up create an AWS forest. But by default, that AWS forest cannot be accessed by a server not inside AWS. That is what this servers purpose was. It was the initial server setup to help expand our services into AWS.

Retrospective

Luckily for me in the cloud computing era, I didn’t have to drive to separate data centres go through racks of servers, or follow ethernet cables through ceiling tiles into a janitors closet with a server still running. But I was still tripped up at certain points and went down the rabbit hole of a certain task unable to see the reasons of why things were not making sense. I do not think this is a bad thing and each rabbit hole forces us to take a step back and re-evaluate our assumptions and knowledge.

So many people seem to fall into failing to write documentation and keep it up-to-date. The logic and complexity of systems and services, especially when in development, seem to turn people off writing one version of documentation and then realising they have to go back and completely re-write it. Sure it’s a hassle, but that is what increases the value of documentation. If there was any documentation before I started it would have been a step in the right direction, but at least there is very thorough documentation now.

My “new” rules:

  • Rule 1. Never assume
  • Rule 2. New Service? New Documentation!

[1] I failed my first rule of diagnosing IT problems. You cannot assume.