How I killed Exchange.

Reading Time: 7 minutes

Introduction

So, you might be confused by the title, and you’d be right to be; I am too.

Backstory

As many of you may know, I run a somewhat large homelab, with pretty much anything I can run myself running. Now, this is where Exchange comes into play.

I chose to use Exchange as my primary mail server since I already had an existing Active Directory and it fit best with my requirements. Usually I would just use something like Postfix and Dovecot on a Linux or BSD server, however it doesn’t run on anything but Windows 😭.

In the end, I chose to install it on Windows Server 2022 Core, since it’s fairly lightweight as there is no graphical environment to interact with.

This server has been running for the last couple of years non-stop, and without failure, and last year I upgraded the setup to multiple servers for High Availability so I can update one node without any downtime; I do use it as my primary personal mail server, after all!

Now, up until recently, this setup has been running perfectly fine without any issues, even surviving a good half-dozen or so updates since installing, however around Friday last week something happened!

The “Something”

Oh so mysterious! Last Friday, I went to change some settings and add a user to the server, since I like to have individual users for each service, but something happened! Upon adding the user, I get an ERROR!!!! Now, this isn’t too unusual, it is a Microsoft product after all, but this error is different.

WARNING: An unexpected error has occurred and a Watson dump is being generated:
minLength > maxLength
minLength > maxLength
    + CategoryInfo          : NotSpecified: (:) [Enable-Mailbox], ArgumentException
    + FullyQualifiedErrorId : System.ArgumentException,Microsoft.Exchange.Management.
    RecipientTasks.EnableMailbox
    + PSComputerName        : GLTX-EXSRV-01.galactix.local

Usually I would just spend 20 minutes googling and reading through documentation, but I ran into a problem… Apparently nobody has had the same issue, ever. After posting on Spiceworks for any information (because who knows what those geniuses might know!), I soon came to a conclusion that they were equally as confused with the error; with 2 of them even stating their surprise.

We are all trying to find any hints that might indicate an issue, ranging everywhere from DNS to those pesky little computer gremlins.

Why can’t I just reinstall it?

Short answer: I can, and probably will.

Long answer: While I could just reinstall it, I would rather come up with a fix for said issue, and document it for anyone else in the future if they somehow fuck it up as badly as me! Do I know if it can be fixed? Nope, not at all, but I would rather try than give up as I do enjoy fixing things like this but don’t get to do it that often. If it can’t be resolved, I probably will reinstall everything; Exchange is fairly piss-easy to install anyway, it just takes an age to do so.

What do I think it might be?

Well, initially I thought it might be, and it did in fact indicate that it was, a corrupt installation of Exchange, but running an install of the latest CU should fix this with ease, and the fact it was the same error across all Exchange servers in this setup rules that out somewhat, though it could always be a corrupt database (a quick database repair ruled this out too); but all in truth, I really have no idea what it could be or what could cause it.

The good, bad, and the confusing.

Yes, you read that right, there is some good things here. On the plus side, with all these issues, mail transport still works perfectly fine, so I can just leave it up and never touch anything.

The downsides? New mailboxes can’t be created, which is somewhat of a pain in the ass, but I can deal with it for now.

The confusing? This MinLength > MaxLength error pops up on pretty much any setting I modify, so nothing can be changed on the server-side in regards to that.

What am I going to do?

Honestly, I’m probably just going to rip everything out and start over from scratch if I can’t fix it by the end of next week, and it would probably be best anyway. I do have backups of my mailboxes, and I am more than happy to re-import them all to a new install. I’ve learnt a lot since my last install, so it would likely work in my favour, it just takes for-fucking-ever to do.

I also plan on making it multi-site with my off-site servers, and might possibly get rent a couple of dedicated servers over in the USA for extended availability and speed.

The plan.

My updated install would be fairly overly complex, to be honest, with a lot of resources in use, but it is a somewhat “critical” service for me to run, and I’m not a fan of gmail nor Microsoft 365 (plus, a true geek like me should be running his own services where possible, right?)

It consists of serveral load balancers, a minimum of 2 Edge Transport nodes per site, a minimum of 2 mailbox servers per site, a witness node per site for High Availability, and a Domain Controller for each site. All of this will be running on Windows Server 2022 Core, with an access server running Windows Server 2022 with a desktop experience for remotely accessing sites in the event I can’t be arsed to use PowerShell.

Side rant: My thoughts on Windows.

To be honest, I do quite like Windows at times. “Why, it’s terrible! You should use GNU/Linux for everything instead like me!!!”, I hear a lot of you cry.

In my opinion, every major operating system has it’s place in the world, that’s why they’re popular and not just another hobby OS.

Windows does a lot of things well, and can honestly be quite nice to work with at times. PowerShell, although buggy at times, integrates with pretty much ALL Windows and other misc. Microsoft services to the point where I would agree it to be nae-on essential for those dealing with Windows systems often to learn. Sure, it can all be a huge pain in the bollocks at times, as can anything, but if you’re doing everything with good practice, it’s extremely robust. For those that don’t want a desktop environment and just need a bare-bones hypervisor, domain controller, email server, etc that can be setup with relative ease, there are Core editions of server that can be fairly lightweight at times; and for those that need to run GUI applications or run a terminal services server, there is Windows Server with Desktop Experience. If you’re in a company that uses any Windows machines, and even MacOS and Linux machines, Windows Server usually appears somewhere in the topology, even if you’re running everything in Azure!

Are there things I would change, add or remove from it? ABSOLUTELY! Updating without a reboot every few days for one, although if you’re running everything correctly, clustering and HA in software should mitigate this. It would also be nice if we didn’t get all the added crap Microsoft likes to add, like Copilot 🤢 that should NOT be installed by default on a server operating system; it just makes everything need more resources, and can create security risks. If I’m paying £6,500 for a license, I should be able to choose what software I do and don’t want, I shouldn’t have to spend a day removing useless crap just because Microsoft wants you to use it!

What’s next?

Well, I do plan on keeping you guys up to date on the situation between me and this Exchange chaos, but it might take some time. If I don’t get any further with resolving the bug, I might just make a post teaching you all how to install Exchange server correctly, and hopefully it shouldn’t fuck itself over anymore.

Update (as of 29th August 2024)

So it’s been a while since I updated this last, so here it is: I eventually managed to get exchange working today! Here’s the solution:

You need to rebuild the IIS virtual directories!

Here’s a little script I prepared for the future, just replace GLTX-EXSRV-01 with your server’s name, and mail.galactix.xyz with your relevant details:

# Recreate ActiveSyncVirtualDirectory
Get-ActiveSyncVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, Identity, InternalUrl, ExternalUrl
Remove-ActiveSyncVirtualDirectory -Identity "GLTX-EXSRV-01\Microsoft-Server-ActiveSync (Default Web Site)" -Confirm:$false
New-ActiveSyncVirtualDirectory -Server "GLTX-EXSRV-01" -InternalUrl "https://mail.galactix.xyz/Microsoft-Server-ActiveSync" -ExternalUrl "https://mail.galactix.xyz/Microsoft-Server-ActiveSync"

Get-AutodiscoverVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, Identity
Remove-AutodiscoverVirtualDirectory -Identity "GLTX-EXSRV-01\Autodiscover (Default Web Site)" -Confirm:$false
New-AutodiscoverVirtualDirectory -Server "GLTX-EXSRV-01" -BasicAuthentication $true -WindowsAuthentication $true
Set-ClientAccessServer -Identity "GLTX-EXSRV-01" -AutodiscoverServiceInternalUri "https://autodiscover.galactix.xyz/Autodiscover/Autodiscover.xml"

Get-EcpVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, InternalUrl, ExternalUrl, Identity
Remove-EcpVirtualDirectory -Identity "GLTX-EXSRV-01\ecp (Default Web Site)" -Confirm:$false
New-EcpVirtualDirectory -Server "GLTX-EXSRV-01" -InternalUrl "https://mail.galactix.xyz/ecp" -ExternalUrl "https://mail.galactix.xyz/ecp"

Get-MapiVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, InternalUrl, ExternalUrl, Identity
Remove-MapiVirtualDirectory -Identity "GLTX-EXSRV-01\mapi (Default Web Site)" -Confirm:$false
New-MapiVirtualDirectory -Server "GLTX-EXSRV-01" -InternalUrl "https://mail.galactix.xyz/mapi" -ExternalUrl "https://mail.galactix.xyz/mapi" -IISAuthenticationMethods Ntlm, OAuth, Negotiate

Get-OabVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, InternalUrl, ExternalUrl, Identity
Remove-OabVirtualDirectory -Identity "GLTX-EXSRV-01\OAB (Default Web Site)" -Confirm:$false -Force
New-OabVirtualDirectory -Server "GLTX-EXSRV-01" -InternalUrl "https://mail.galactix.xyz/OAB" -ExternalUrl "https://mail.galactix.xyz/OAB"

Get-OwaVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, InternalUrl, ExternalUrl, Identity
Remove-OwaVirtualDirectory -Identity "GLTX-EXSRV-01\owa (Default Web Site)" -Confirm:$false
New-OwaVirtualDirectory -Server "GLTX-EXSRV-01" -InternalUrl "https://mail.galactix.xyz/owa" -ExternalUrl "https://mail.galactix.xyz/owa"

# Create new PS session for this :)
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.SnapIn
Get-PowerShellVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, InternalUrl, Externalurl, Identity
Remove-PowerShellVirtualDirectory -Identity "GLTX-EXSRV-01\PowerShell (Default Web Site)" -Confirm:$false
New-PowerShellVirtualDirectory -Server "GLTX-EXSRV-01" -Name Powershell -InternalUrl "https://mail.galactix.xyz/PowerShell" -ExternalUrl "https://mail.galactix.xyz/PowerShell" -RequireSSL:$false

# Continue on new session, the old one won't work :)
Get-WebServicesVirtualDirectory -Server "GLTX-EXSRV-01" | Format-List Server, Name, InternalUrl, ExternalUrl, Identity
Remove-WebServicesVirtualDirectory -Identity "GLTX-EXSRV-01\EWS (Default Web Site)" -Confirm:$false -Force
New-WebServicesVirtualDirectory -Server "GLTX-EXSRV-01" -InternalUrl "https://mail.galactix.xyz/EWS/Exchange.asmx" -ExternalUrl "https://mail.galactix.xyz/EWS/Exchange.asmx"

After that, it should work.

Note: There may be some corrupt mailboxes, unfortunately I have not found a way to recover these yet, so just run the following and revert the mailbox from a backup:

Set-Mailbox -Identity "user@galactix.xyz" -Database PrimaryDB -Force

Update (as of 18th October 2024)

So far, so good, the server is now stable and running properly eithout issues. I have also recently migrated my backup solution to Veeam for better Exchange backups as it supports Exch natively, while also allowing me to monitor my infrastructure with Veeam ONE (even my SunOS/Solaris VMs!!!).

I have had a few people asking me “Why use Exchange Server, why not use Microsoft 365 or even Postfix?” Well, that’s a complicated one and I do have good reason to, in my opinion. Here we go: For one, I like to host my own infrastructure and have complete control over everything. I use M365 for other projects and for work, and I like it, but for personal stuff, Exchange does what I want it to do and how I want to do it. This is where the “why not Postfix or <insert another OpenSource solution here>?” argument comes into play. Well, I do actually run Postfix on my Inbound relays. This is due to more regular updates, mainly for security, however it doesn’t quite meet my use-case for my primary mail server of choice.

My choosing methods for a mail server include easy management, APIs for plugins and automation, and the ability to configure redundant connections between servers.

Exchange does all of this for me and that is exactly why I chose it.

Footnote

Thank you for taking your time to read this post, I understand it is fairly long-winded and may be boring, but this is just how I like to vent and document my technical problems and solutions, so it means a lot!

If you did, however, enjoy this post or find it interesting, I promise I have more to come soon as I am working on a few as I write this.

GLTX.ORG