How to reset or recover NSX-T root password?

How to reset/recover NSX-T root, admin or audit password.

GRUB MUST BE CONFIGURED TO RECOVER YOUR ROOT PASSWORD

VMware documentation states:

“Important If the configuration (GRUB) is not performed after deploying the appliance and you forget the root password, resetting the root password is not possible”

NSX-T DATA CENTER INSTALLATION GUIDE (PG. 39)
If you have access to the NSX-T appliances with a root login you can enable grub or verify grub is enabled by referring to the (How-To) Configure GRUB – A Password Recovery Insurance Policy article.

Before we start walking through the steps, we need to have the Ubuntu 16.04 Server iso downloaded and placed on a datastore accessible to the ESXi host where the NSX-T manager resides. The iso can be downloaded here.

Note: Pre-staging the Ubuntu image is important because after you complete step #9, you MUST complete the remaining steps in less than 15 minutes to prevent the password from reverting back.

1. First, we will need to power our NSX-T manager down.

2. Power the NSX-T manager up and quickly open a console screen and press the SHIFT key to stop the boot sequence and enter the GRUB loader.

3. Select the ‘Advanced options for Ubuntu‘ and press Enter.


4. You will be presented with a login prompt. This prompt is used to login using the predefined GRUB credentials.

Here are the predefined GRUB credentials:

Enter username: root
Enter password: VMware1
5. On the next screen, highlight the ‘Ubuntu, with Linux‘ option without (upstart) at the end and press the e key to edit.


6. Now that we have the editor open, we need to scroll to the bottom of the text box and locate the line with the word ‘linux‘. Follow that line to the end where you see root=UUID=. Delete everything on that line after the root=UUID=.

if [ x$feature_platform_search_hint = xy ]; then
search –no-floppy –fs-uuid –set=root –hint-bios=hd\
0,msdos2 –hint-efi=hd0,msdos2 –hint-baremetal=ahci0,msdos2 4b76f7c8-7\
84c-4741-950e-c8d2dca486c5
else
search –no-floppy –fs-uuid –set=root 4b76fc8-784c-\
4741-950e-c8d2dca486c5
fi
echo ‘Loading Linux 4.14.74-nn3-server …’
linux /vmlinuz-4.14.74-nn3-server root=UUID=4b76f\
7c8-784c-4741-950e-c8d2dca486c5 ro audit=1 quiet splash rootdelay=90 net\
.ifnames=0 biosdevname=0 $vt_handoff
echo ‘Loading initial ramdisk …’
initrd /initrd.img-4.14.74-nn3-server
In this example we will be partially deleting line 11 and all of line 12 as shown here in red:

7. Now, we need to replace the text we deleted with our own to so we can load our bash shell to reset the NSX-T password.

We need to append after the root=UUID= with the following:

rw single init=/bin/bash
The text box should now look similar to this with the addition in the red box:

if [ x$feature_platform_search_hint = xy ]; then
search –no-floppy –fs-uuid –set=root –hint-bios=hd\
0,msdos2 –hint-efi=hd0,msdos2 –hint-baremetal=ahci0,msdos2 4b76f7c8-7\
84c-4741-950e-c8d2dca486c5
else
search –no-floppy –fs-uuid –set=root 4b76fc8-784c-\
4741-950e-c8d2dca486c5
fi
echo ‘Loading Linux 4.14.74-nn3-server …’
linux /vmlinuz-4.14.74-nn3-server root=UUID=4b76f\
7c8-784c-4741-950e-c8d2dca486c5 rw single init=/bin/bash
echo ‘Loading initial ramdisk …’
initrd /initrd.img-4.14.74-nn3-server
8. After making the change press ‘Ctrl-x‘ or the ‘F10‘ key to save our changes and reboot the manager into single user mode.

Once the NSX-T manager reboots, it will boot into single user mode. If your console session disconnects, simply open a new console window.

Single User Mode Shell
**After Step #9, You Will Have 15 Minutes to Complete the Remaining Steps**

9. If you are resetting the password for the ‘root’ account, run the following command:

passwd
If you are resetting the password for the ‘admin’ or ‘audit’ accounts, run the following command:

passwd
**Note: You can run the passwd command multiple times**

After running the passwd command above, you will be prompted to enter a password, press enter and then enter the same password again. Press enter again to complete the password change.

10. After you have changed the password on the accounts you want, we will then synchronize the passwords by running:

sync
11. Once complete, shut the NSX-T manager down for the final phase of the password reset process.


12. Next, we need to download ubuntu-16.04.6-server-amd64.iso

13. After the Ubuntu iso is on your desktop, we need to place the iso onto a datastore accessible to the ESXi host where the NSX-T manager virtual machine resides and attach the iso to the NSX manager virtual machine. To do this, we need to login to the vCenter server or ESXi web UI and import/upload the Ubuntu iso file onto the datastore.

Once the Ubuntu server iso has been uploaded to a datastore, we need to attach the iso to the NSX Manager virtual machine by right clicking on the NSX-T manager virtual machine and clicking ‘Edit Settings’. If needed, add a CD ROM drive if one does not already exist.


In the CD ROM Drive configuration, check the Connect at power on checkbox.

In CD/DVD Media, select Browse then select the Ubuntu image from the applicable datastore.


14. Next, we need to change the boot order from within the BIOS of the NSX manager virtual machine so the CD-ROM drive is used before the primary hard disk. To do this, we need to force the NSX manager to boot to its BIOS so we can make the change.

Click on the ‘VM Options‘ tab at the top of the ‘Edit Settings‘ page.

Select the radio button to ‘Force BIOS setup‘ on the next boot. Then click, OK.


15. Now we can boot the NSX-T manager into its BIOS so we can make the necessary boot order changes.

Power on the NSX-T manager and open a console. Once the BIOS page loads, navigate to the ‘Boot‘ tab and move the ‘CD-ROM Drive‘ to the top of the list.

Then move to the ‘Exit‘ tab and highlight ‘Exit Saving Changes‘ and press Enter.

The NSX-T manager will now reboot into the Ubuntu Server CD we mounted earlier.

16. Once the NSX-T manager reboots, you will navigate through the Ubuntu setup pages.

Select Your Preferred Language
Select ‘Rescue a broken system‘
Select Your Preferred Language (again)
Select Your Location
Select Your Keyboard Configuration
Select ‘Do not configure the network at this time‘
Leave Host Name as ‘ubuntu‘ (not used for recovery)
Select Your Timezone
Select ‘Do not use a root file system‘
Select ‘Execute a shell in the installer environment‘
Select ‘‘ to Execute a Shell

17. After you press the Enter key on , a shell prompt will appear on the next screen. Here you will run the following commands to ensure the password change(s) made in the previous steps do not revert and are permanent.

The following command will mount the /dev/sda2 partition of the NSX-T manager to the /mnt directory.

mount /dev/sda2 /mnt
Next, we need to mount the /dev directory. However, we need to use the bind mount option instead so we take the existing directory tree of /dev and replicate it under /mnt. The directories and files in the bind mount are the same as the original.

mount –bind /dev /mnt/dev
Next, we need to change the root directory for our recovery purposes to /mnt. To do this, we will be using the chroot command. The chroot command will allow us to move back into the NSX-T partition from our ‘alternate’ root file system that we just loaded from the rescue CD.

1
chroot /mnt
Next, move to the section for your version of NSX-T. It is divided up into two sections, 2.4.x and 2.5.x.

Steps for NSX-T 2.4.x Versions
Now, we need to mount the /config directory and touch the ‘reset_cluster_credentials’ file to update its timestamp. Then we will un-mount the /config directory when done.

mount /config
touch /config/vmware/nsx-node-api/reset_cluster_credentials
umount /config
Steps for NSX-T 2.5.x Versions

touch /var/vmware/nsx/reset_cluster_credentials
Now that we have completed making our password changes permanent, we need to exit our chroot instance and clean up.

exit
umount /mnt/dev
umount /mnt
sync
exit

Once we have performed all the above steps and exited the shell, we can reboot the system by selecting ‘Reboot the system’.

Reference: https://cloudbytesecurity.com/2019/10/14/recover-nsx-t-passwords/

How to update SSH known_hosts fingerprints? (WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!)

You are connecting to a previously known host over ssh, however the following message pops up.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!  @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
8a:ff:a3:b6:32:06:4e:fd:2e:2f:67:80:9e:ba:8e:gf.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending key in /home/username/.ssh/known_hosts:8
RSA host key for 192.168.1.100 has changed and you have requested strict checking.
Host key verification failed.

Follow the below steps to fix this issue.

1. Open the known_hosts file in vi as follows

vi /home/username/.ssh/known_hosts +8 (NOTE: Replace username with your user id)

(Note: Line number 8 shown in this example would be most likely different for you. Just match it with the number you see in the warning message)

2. vi editor would open and show you the offending key on line 8. Simply type dd on your keyboard to delete the offending key. Save and exit vi by typing :wq

3. Try connecting again and accept the new key.

4. You have now made a successful ssh connection.

How to list running Hadoop job names and id from command line?

Sometimes hadoop jobs submitted via UI may get stuck and you need to stop them from CLI. Below command will list the running jobs and then you can issue the kill command to stop the job.

To list hadoop job names and id

hadoop job -list | egrep ‘^job’ | awk ‘{print $1}’ | xargs -n 1 -I {} sh -c “hadoop job -status {} | egrep ‘^tracking’ | awk ‘{print \$3}'” | xargs -n 1 -I{} sh -c “echo -n {} | sed ‘s/.*jobid=//’; echo -n ‘ ‘;curl -s -XGET {} | grep ‘Job Name’ | sed ‘s/.* //’ | sed ‘s/<br>//‘“

Sample output:
job_201412111202_0320 …rec_type(Stage-2)
job_201412111202_0321 year,
job_201412111202_0322 distcp
job_201412111202_0323 year,
job_201412111202_0324 year,

Now lets assume you want to kill job distcp, issue the below command

To kill a hadoop job

Hadoop job –kill job_201412111202_0322

Run the list job command again and verify your intended job is killed and not listed as running any more.

How to fix networking after copying/cloning a CentOS virtual machine?

You created a clone of a CentOS linux virtual machine and now the new clone is not coming up on the network. What steps are needed to resolve this issue?

1. Change the host name
Host name is kept in 2 files on linux
/etc/hosts
/etc/sysconfig/network
Open the above files with vi and update the host name

2. Change IP address and MAC address
First obtain the new MAC adddress from the network properties of cloned virtual machine and write it down.

IP and MAC address is kept in these files so we’ll update them.
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/udev/rules.d/70-persistent-net.rules

3. If you are not comfortable with the above file changes, make a backup of the files first.

cp /etc/hosts /etc/hosts.orig
cp /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth0.orig
cp /etc/udev/rules.d/70-persistent-net.rules /etc/udev/rules.d/70-persistent-net.rules.orig

How to load Operating System remotely to HP Blade server in c7000 chassis?

First have a web server where you are hosting your .iso image. It could be any webserver apache, IIS or other. We’ll assume apache in this environment

Copy the .iso file to /var/www/html folder.

Connect to the blades ILO interface over SSH

ssh Administrator@X.X.X.X (where X is the IP address of ILO)

once connected issue the following commands

POWER OFF (make sure the blade server is powered off)

vm cdrom insert http://X.X.X.X/centos.iso (Where X is the IP address of the web server)

vm cdrom set boot_once

POWER ON

Once the blade server is powered on, you can connect to the console. Your .iso image should be loading.

If there are any issues, please verify you have a ILO licence installed. HP offers a 30 day free eval license. Check http://www.hp.com for more information.