Storage Replica in Windows Server 2016

What is Storage Replica?

Block level synchronous or asynchronous, volume based data replication.

What does it do?

Replicates storage data from a disk attached to one instance of Windows Server to another disk attached to a different instance of Windows Server. Because it works at the block level, it doesn’t care about open files. It operates way down the driver stack:

It allows you to create stretch clusters (clusters that don’t have shared storage), or you can just use it to replicate data from one place to another.

How do you get it?

Storage Replica is new in Windows Server 2016. You need the Datacenter edition, but your Hyper-V/vSphere hosts are probably licensed for that anyway, right? It’a available in the “full” editions (i.e. with desktop or core) and also nano. It’s just a Windows Feature, so enable it with Server Manager or PowerShell:

Install-WindowsFeature -Name Storage-Replica

Synchronous vs Asynchronous

It’ll do both, you just need to pick the appropriate one. Firstly, is your network latency between the two servers greater than 5ms?  In which case you shouldn’t use Synchronous replication, the performance impact will be too great. If it’s 5ms or less, you can choose either, based on your needs.

Synchronous replicationstorage-replica-synchronous-replication

This works pretty much the same as oldskool SAN replication, kit like the EMC Clariion’s MirrorView feature does exactly the same thing (but for somewhat more than the cost of a Windows Server licence and some cheap server hardware – which is entertaining as certain models of Clariion actually ran Windows Server internally!). The key point with Synchronous replication is that the write is not confirmed on the source storage system until it has been written to the log disk on both the source and destination storage systems. This gives you peace of mind that your data is safe, at (usually) the expense of performance.

Asynchronous replication

See diagram for Synchronous replication, but confirm the write after step 2 – in other words, once the data has been written to the source log disk. The data is written to the destination log disk separately.

How Storage Replica changes your disk IO

On a volume that is being replicated via Storage Replica, all write activity happens to the Log disk. This data is then “destaged” to the Data disk, which thus only ever handles Because the Log disk is a log, the writes to this are sequential. Despite this, it’s recommended that you make the log disk an SSD – writes tend to slower than reads, and Windows (and applications, e.g. SQL Server, Exchange) can cache data for read operations in RAM. Write operations have to be securely written to disk, and so if the write speed of the log disk isn’t fast enough it’ll become the point of contention.

Is it safe to use now?

Probably – Microsoft has had customers running this technology on production systems since 2014. I plan on using it right now.

Want to know more?

Watch Ned Pyle’s excellent, informative, hilarious Ignite session video (where I stole the diagrams above from!). Hipsters beware…!

Posted in Storage, Uncategorized, Windows | Tagged , , , , , , , , | 1 Comment

Fix PowerShell WinRM remote connection errors

I’ve had two annoying PowerShell errors today, both for the same server:

Enter-PSSession : Connecting to remote server failed with the
following error message : The client cannot connect to the destination specified in
the request. Verify that the service on the destination is running and is accepting
requests. Consult the logs and documentation for the WS-Management service running
on the destination, most commonly IIS or WinRM. If the destination is the WinRM
service, run the following command on the destination to analyze and configure the
WinRM service: "winrm quickconfig". For more information, see the
about_Remote_Troubleshooting Help topic.

The standard “internet” response to this is to open an Administrator command prompt and run

winrm qc

(or winrm quickconfig). Or open an Administrator PowerShell prompt and run:


But these are quite annoying solutions if you know that it should be working, and indeed is working on all your other servers, because you’ve configured it via Group Policy! There’s no harm in running the commands anyway, but most likely they’ll just come back and say “already configured” or words to that effect.

So the fix (for me) for the above error was to check what IP addresses the listener was listening on using:

netstat -aon | find "5985"

where 5985 is the default port used by WinRM. You should see the system process (ID 4) listening on like this:


However on my problem server it was listening on This can be confirmed using:

netsh http show iplisten

and you’ll only see in the list.
On the problem server itself you’ll find that using:

Enter-PSSession -Computername localhost

works, whereas:

Enter-PSSession -Computername

does not.
localhost is the name for the loopback address whereas the fully qualified server name will give you the IP address of the server as seen from your network.
The fix for this is to delete the loopback address from the http listener, which then makes it listen on all valid addresses:

netsh http delete iplisten

Check what addresses it is now listening on (plus the port) by using:

winrm e winrm/config/listener

Problem solved. Or not, I then got this error:

Enter-PSSession : Connecting to remote server failed with
the following error message : The WinRM client sent a request to an HTTP server and 
got a response saying the requested HTTP URL was not available. This is usually
returned by a HTTP server that does not support the WS-Management protocol. For
more information, see the about_Remote_Troubleshooting Help topic.

Which it turns out can be caused by having IPv6 enabled on the server.
In the netstat output (see above) I also had some IPv6 addresses showing as listening on port 5985.
I don’t use IPv6 but it’s on by default in Windows and will auto-assign itself a link local address (starts with fe80:). So I disabled IPv6 and the error went away. There’s also a workaround using the hosts file if you don’t want to disable IPv6.

Posted in PowerShell, Windows | Tagged , , , , , , , , , , , | Leave a comment

Extract private key from Microsoft CA-issued certificate

I wanted to use my internal Active Directory Certificate Services server to create a certificate for a Synology NAS. The Synology needs the private key and the certificate to be in separate files.

I created the certificate by using the CA web interface https://my-ad-cs/certsrv and then choosing the following options:

  • Request a certificate
  • advanced certificate request
  • Create and submit a request to this CA
  • Template: Web Server (5 years) – note that this is a custom certificate template that I created that allows the private key to be exported. This enables the Mark keys as exportable option, which needs to be ticked.
  • Fill in the identifying information, and leave all other options as default

Once the certificate was created, I installed it. I then opened certmgr.msc, found the certificate (under Personal – Certificates).

Next, I right-clicked the certificate, and chose All tasks – Export. I chose to export the private key, and under the .pfx option ticked Include all certificates in the certification path if possible. I set a password and saved the file.

Then I downloaded the latest version of OpenSSL and extracted the zip file.

I copied the pfx file into the folder where I’d extracted OpenSSL to, and opened a command prompt in that folder. I used the following two command lines to extract the private key and certificate from the pfx file.

openssl pkcs12 -in extracted.pfx -nocerts -out privatekey.pem -nodes

openssl pkcs12 -in extracted.pfx -nokeys -out cert.pem

These two files were then uploaded to the Synology from Control Panel, Security, Certificate.

Once the certificate was installed, I selected it and clicked Edit, then ticked Set as default certificate. Finally, I clicked Configure and changed each of the services to use the new certificate. Upon clicking OK, the web services restarted and are now using my CA certificate.


Posted in Security, Storage | Tagged , , , , , , , , , , , , | Leave a comment

Windows Server 2016

Lots of good stuff. This video about Storage Replica from the guy who heads the Storage team up, Ned Pyle, is definitely worth watching. Informative and funny.

There’s also a free eBook: Introducing Windows Server 2016

Posted in Free training, Storage | Tagged , , , , , , | Leave a comment

Create a PFX file from separate private and public key certificate files

Whilst installing ADFS I needed to import my wildcard certificate. This proved fiddly as Microsoft need it to be in pfx format, and I only had separate private .key and public .crt files.

PFX is a Microsoft certificate format that combines the public and private keys into one file, but is different from the combined format used by (e.g.) OpenSSL. I found that I had to go through several steps to achieve this.

Your CRT public key file should start with the following (open it in notepad to check):


Convert the public key into a PVK file

If you open your public key file (e.g. in notepad) and it says


then you’ll need to convert it into a PVK file. a PVK file is a binary file, so it’ll look have lots of non-alphanumeric characters in it. To do the conversion, I used a utility called pvk written by Dr Stephen N Henson (thank you!). The command line for pvk.exe is:

pvk.exe -in rcmtech_private.key -out rcmtech_private.pvk -topvk

You’ll be prompted for a password, and you do need to specify one or the next step in the process will fail.

Download the Windows SDK

Now you need a utility called pvk2pfx. Annoyingly this only comes as part of the Windows SDK, so you need to download that first – you just need to tick the Windows Software Development Kit option when selecting the features to install. Once you’ve pulled it down (it is several GB), you’ll find pvk2pfx in the following folder (you are using a 64-bit OS, right?):

C:\Program Files (x86)\Windows Kits\10\bin\x64\pvk2pfx.exe

Create the PFX file

The command line for pvk2pfx.exe is:

pvk2pfx.exe /pvk rcmtech_private.pvk /spc rcmtech_public.crt /pfx rcmtech.pfx

You’ll get a pop up asking for your pvk file password, which you specified when creating it using pvk.exe. (If you didn’t specify a password, and just hit enter on this popup you’ll just get an error: ERROR: Password incorrect. (Error Code = 0x80070056).)

You should now find yourself with a PFX file, which you can use to import into Windows (e.g. ADFS config wizard).

Posted in Security, Windows | Tagged , , , , , , , , , , , , , | 2 Comments

Send email via Yahoo with PowerShell

Quick reference for sending email via a Yahoo! mail account and their SMTP servers from within a PowerShell script.

You’ll need the Yahoo SMTP server details, which require you to use a specific port (587 – I couldn’t get 465 to work) and an encrypted connection.

I also set up a Yahoo “app password” for PowerShell, which means I don’t have to use my regular password in plain text in a script, which is a good thing. To do this click the settings cog from within Yahoo Mail, go to Account info, then when that opens go to Account Security. From there you’ll find Manage app passwords and can add a new one called (e.g.) PowerShell. Copy and paste the password generated into the PowerShell script. Note that the generated password is shown in four blocks of four letters with spaces in between but when you paste it it’ll come out with no spaces – and this is correct – you do not need the spaces displayed on the Yahoo web page.

Here’s the sample script:

$Username = ""
$Password = "abcdefghijklmnop"

$SecurePassword = $Password | ConvertTo-SecureString -AsPlainText -Force
$Credentials = New-Object System.Management.Automation.PSCredential -ArgumentList $Username, $SecurePassword

$RcptTo = ""
$Subject = "Yahoo Test"
$Body = "This is a test message"
Send-MailMessage -From $Username -To $RcptTo -Subject $Subject -Body $Body -SmtpServer -Port 587 -UseSsl -Credential $Credentials

And here’s the same thing as a function:

function Send-YahooMail ($Username, $Password, $RcptTo, $Subject, $Body){
    $SecurePassword = $Password | ConvertTo-SecureString -AsPlainText -Force
    $Credentials = New-Object System.Management.Automation.PSCredential -ArgumentList $Username, $SecurePassword
    Send-MailMessage -From $Username -To $RcptTo -Subject $Subject -Body $Body -SmtpServer -Port 587 -UseSsl -Credential $Credentials

If you don’t want any kind of password in plain text in a script then you need to also look at this post of mine too.

Posted in PowerShell, Security | Tagged , , , , , , , , , , , , | 1 Comment

Connection State Announcer

I’ve been having some very annoying problems with my home broadband where the connection keeps dropping every few minutes. I wrote this quick script to monitor a site on the internet and tell me when it can’t be reached anymore, but it has plenty of other uses. I’m using PowerShell’s ability to speak because I wanted to leave this running whilst I had a web browser open over the top or was away from the computer.
It was also a fun coding exercise to write something that would detect a state change and only report when the connection came up or down, not the entire time it was down. I did this using a two position array and comparing the contents between subsequent tests of the internet site. If the test results were the same, the connection hadn’t changed state, if they were different it had.

# Configure the test
$TestSubject = ""
$TestInterval = 2
$TriesPerTest = 2

# Initialise speech
Add-Type -AssemblyName System.Speech
$Speak = New-Object -TypeName System.Speech.Synthesis.SpeechSynthesizer

# Initialise an array to hold state information
$Connected = New-Object System.Collections.ArrayList
# Assume the connection is down, write this to the first two positions in the array
$Connected.Add($false) | Out-Null
$Connected.Add($false) | Out-Null

# Loop forever
    # Test the connection, store the result in the second position in the array
    if(Test-Connection -ComputerName $TestSubject -Count $TriesPerTest -ErrorAction SilentlyContinue){
        $Connected[1] = $true
        $Connected[1] = $false
    # Compare the first and second positions to give us a state change trigger
    if($Connected[0] -ne $Connected[1]){
        # State has changed, write timestamp to screen
        Write-Host ("`n"+(Get-Date -Format s)+" ") -ForegroundColor Gray
        if($Connected[1] -eq $true){
            # Announce that the connection is now up
            $Speak.Speak($TestSubject+" is up!")
            # Announce that the connection is now down
            $Speak.Speak($TestSubject+" is down.")
        # Copy the test result into the first position in the array, ready for comparison with the next test result
        $Connected[0] = $Connected[1]
    # Print something colourful to the screen to show relative up/down times
    if($Connected[0] -eq $true){
        Write-Host "#" -ForegroundColor Green -NoNewline
        Write-Host "#" -ForegroundColor Red -NoNewline
    # Pause before next test
    Start-Sleep -Seconds $TestInterval
Posted in Networking, PowerShell, Windows | Tagged , , , , | Leave a comment

VeeamZip Hyper-V Backup

Veeam Backup & Replication drops down into VeeamZip mode if you don’t have a licence key. But it’ll still provide some handy functionality – I’m using it to take a basic backup of all the VMs on a Hyper-V host at a remote site, just in case the (rather crusty old) hardware dies.

There’s no scheduling in the free Veeam backup product, but you can write a PowerShell script and use Windows Task Scheduler to run it, which is what I’ve done. I’m backing up all powered on VMs on a particular host to a UNC path on a physical server.

Add-PSSnapin VeeamPSSnapin
$VMs = Find-VBRHvEntity -Name * -Server HVHost01
foreach($VM in $VMs){
    if($VM.PowerState -eq "PoweredOn"){
        Write-Host ("Backing up "+$VM.Name+"...") -ForegroundColor Gray -NoNewline
            $Result = Start-VBRZip -Entity $VM -Folder "\\FileServer\Veeam\Backups" -Compression 5 -AutoDelete In1Week
            if($Result.Result -ne "Failed"){
                Write-Host "Done" -ForegroundColor Green
                # You could use Send-MailMessage here to get email notification of a successful backup
                Write-Host "Failed" -ForegroundColor Red
            Write-Host "Error" -ForegroundColor Red

I created a domain admin account in Active Directory to use as a service account for Veeam, and used this to run the scheduled task too. The task needs to be “Run with highest privileges”.

If you get the error:

06/07/2016 10:13:10 :: Error: Access is denied.
--tr:Error code: 0x00000005
Cannot create folder. Folder path: [\\FileServer\Veeam\Backups].
--tr:FC: Failed to create directory. Directory path: [\\FileServer\Veeam\Backups].
--tr:Failed to call DoRpc. CmdName: [FcCreateDir].
Access is denied.
Cannot create folder. Folder path: [\\FileServer\Veeam\Backups].

It’s probably because the Veeam B&R server authenticates to the UNC path using its local system account, so you need to grant that computer account access to the share and NTFS permissions on the server holding the shared folder, e.g. VBRServer$

Posted in PowerShell, veeam | Tagged , , , , , , , | Leave a comment

SMTP email diagnostics and info – troubleshoot email not received

Every now and then somebody will contact you via a method that they wouldn’t normally use and say something like “did you get the email I sent to you a few days ago”. If the answer to this is “no, I did not receive your email” then the following might be useful.

SMTP (Simple Mail Transport Protocol) servers, classically, should never blackhole email. To blackhole an email message, years ago, was a bad thing and meant that your mail server, or the recipient’s mail server was probably malfunctioning. A receiving SMTP server should either accept and deliver an email, or return a message to the sending server explaining why it was unable to deliver the email. Likewise, a sending SMTP server should either successfully transmit an email to a destination mail server, or deliver an NDR (non-delivery report) back to the user who tried to send the email.

Of course that was before spam became a problem. Now it is standard practice to silently drop emails in certain circumstances. However this does occasionally lead to the problems described at the start of this article. Spam is simply such a huge volume of email now that it would take significant extra server resource to deal with it according to the original “proper” SMTP methodology. You also don’t necessarily want a spammer’s mail server to know if an address they’ve tried to send to is legitimate or not – if they know it exists, or just that a particular internet server is an SMTP server, they’ll likely just send more junk to it.

Diagnosis and troubleshooting

So how to diagnose the “never received” email problem?

Firstly, ask the sender if they received an NDR. If they did, their mail system will usually have provided a log of the conversation when it was trying to talk to your mail system, and this will often be quite helpful.

If they never received an NDR, check your mail system logs to see if you can see any trace of their mail system trying to connect to your mail system. Many companies route all their email though a third party spam/malware filter (e.g. Symantec Email, Microsoft Exchange Online Protection), so this is a good place to start – if you can’t see any emails being received from your sender’s email address there then you can rule out any problems with your internal mail servers.

You can also use online tools such a the Microsoft Remote Connectivity Analyser or MX Toolbox to check that your mail servers (or a third party’s) are configured correctly. From a command prompt (on Windows) you can also get your MX records:

nslookup -type=mx

where is a Google public DNS server, you can change this to use any DNS server you like.

Has the sender been blacklisted?

If a sender’s domain sends spam, or too much spam, it might be blacklisted. You can check this by using MX Toolbox and/or talking to your third party spam filter vendor. If a domain is blacklisted, spam filtering systems will frequently silently drop all (or most) mail send out from it.

Not all failures are bad

When using these tools, it helps to know a bit about how MX record preference values work, and tricks that mail filtering companies use to try and cut down on the amount of spam they have to process. Your MX (mail exchanger) DNS record will usually have more than one entry, because you’ll (ideally) have a primary mail server (or cluster of servers) and one or more lower priority servers in case your primary server is unavailable.

A sending SMTP server will use the lowest preference number server first. It will only use a server with a higher preference number if all lower preference number servers are unavailable or reject the message/communication attempt.

In the case of MessageLabs, the server addresses are actually server clusters. Also, the clusterna names are all spam traps, and will never accept any email. This is because spammers apparently often deliberately send to higher preference number servers because historically they might have no or less spam filtering applied to them, and they would normally only be used on the rare occasions when the primary server (lowest/lower preference number) was unavailable. No correctly configured/normally functioning sending mail server would ever pick one of the “a” servers. This is important to know because some SMTP testing tools will test all servers in the MX records for a domain, and so now we know that in the case of servers, we can expect a failure from the clusterna server, but not from the clustern one!

Posted in Exchange | Tagged , , , , , , , , , , , , , , | Leave a comment

PowerShell: Change Hyper-V VM VLAN after Live Migration

One of my old Hyper-V hosts has its virtual switch uplink NICs connected to access ports rather than trunk ports. My other hosts all have trunked uplinks, which means that the VMs need to have a VLAN ID specified to connect them onto the correct VLAN.

I’m gradually moving VMs off the old host, but want to minimise downtime and down’t want to do it out of hours. The problem is that as soon as the VM has finished its Live Migration onto the new host, it stops being able to talk to the network, due to the VLAN ID not being set.

So I wrote this script to monitor the new host for the presence of the VM that’s being migrated, and as soon as it sees it, to set the VLAN ID to the correct value.

$HVHost = "RCMHV01"
$VMName = "OldVM01"
$SwitchName = "LAN Switch"
$VlanID = 250

$Finished = $false

Write-Host "Waiting for $VMName to move onto $HVHost"
while($Finished -ne $true){
    $VMs = Get-VM -ComputerName $HVHost
    if($VMs.Name -contains $VMName){
        Write-Host "Migrated, waiting for Status to be OK"
        while((Get-VM -ComputerName $HVHost -Name $VMName).PrimaryOperationalStatus -ne "Ok"){
            Start-Sleep -Seconds 1
        $VM = Get-VM -ComputerName $HVHost -Name $VMName
        $VMNICs = Get-VMNetworkAdapter -VM $VM
        foreach($VMNIC in $VMNICs){
            if($VMNIC.SwitchName -eq $SwitchName){
                Write-Host "Configure VMNIC"
                Set-VMNetworkAdapterVlan -VMNetworkAdapter $VMNIC -VlanId $VlanID -Access
                $Finished = $true

Posted in Hyper-V, PowerShell | Tagged , , , , , , | 1 Comment