Bad request timestamp () [40105]

by Vince
in Blog
Hits: 33796

I've been doing this job for far too long and something like a power outage creates a certain level of panic at first discovery.  While we take numerous steps to protect from disasters the one true test is pulling the plug to see what happens.  That is essentially what a power outage does -- it brings a certain sense of randomness with it. 

When disaster strikes, we reach for the documentation which includes pictures of the entire server to guide those on site through the rack.  "Two down from the Cisco switch, that's the firewall.  Check to see if it's on."  Etc.

Once we were able to see the network, back to the documentation to retrieve the IP addresses for the hypervisors. 

In the event the UPS's shuts down the hypervisors, the virtual machines stay in the off state.  Once connected to the hypervisors, we spin up the virtual machines.  The servers come up and I feel like our disaster recovery procedures, and our documentation, are in order.  Albeit tiny, I feel like we've survived a disaster without incident.

With a feeling of confidence, I login to a server and it doesn't like the password.  Another go, perhaps I'm fat fingering the keys.  Again, it doesn't like the password.  On my third attempt, I get the following error:  "Bad request timestamp () [40105]"




  
I've yet to see this message until today but a hunch tells me it's something related to the Duo two-factor authentication we're using.  A quick search confirms my guess -- the time is out of sync.  Forget about the WHY for a moment -- the WHY doesn't matter.  What matters is that the time is out of sync on ALL of the servers and we can't login at the desktop to fix the issue.  

After some head scratching, I realize that we just need:

1.  A machine in the domain to sync against.
2.  A way of getting the servers to sync to that machine.  

Each situation is unique and the way I solved this problem probably isn't relevant to you but I was able to execute:  net time \\system-with-correct-time /set /y 

When the clock was back in sync, we were able to login.

In your situation, you have to think about what you can and can't do.  You can't login at the desktop.  But you can access the file system, you can access the task scheduler, and you can pretty much do everything except login at the desktop.  If you have a patch management system, can you execute commands from that system?  If so, that's your easiest solution.  

A method that might work for you -- using PsExec which can be found in the PsTools download on Microsoft's website:





From a remote machine, we launch PsExec to spawn a command prompt for us on the remote machine.  Once we're on the remote machine, we can use net time to sync with the remote machine.  We could probably use the time command as well.  

I can think of a few other ways to accomplish this task.  Enabling Telnet for example -- which is native and could be a better option if introducing outside tools is prohibited.