NetApp NFS Mount Access Denied By Server

mount.nfs: access denied by server while mounting

Just a quick tip today. While setting up a lab I had the need to mount a cDOT (8.3.0) export from behind a NAT gateway. When attempting the mount operation I got a relatively unhelpful error:

mount.nfs: access denied by server while mounting

After some digging, I found that the cause of this is a setting on the storage virtual machine (a.k.a. SVM, formerly vserver). The problem is that by default cDOT expects that a port <= 1024 will be used for the mount operation. When NAT happens between you and the export, you are at the mercy of the gateway device for the port to be used. By setting the SVM NFS option mount-rootonly to disabled, this requirement is lifted.

To fix the problem from the cluster shell:

To fix the problem using the NetApp PowerShell toolkit:

Linux Console Scrolling

A simple, but extremely useful, tip…I didn’t know about it for a long time, but now that I do it’s quite helpful.

In most linux consoles, including RHEL and it’s derivatives, SUSE, and Ubuntu (these are the ones I’ve tried) you can scroll up and down to view the console history by holding the Shift key and using Page Up/Page Down.

Unfortunately, it does not work with the cDOT console.

Stupid Bash Tricks for SSH

My last post explained how to set up SSH key based authentication for connecting to a NetApp. If you have multiple/many systems to administer this makes it easy to quickly connect to and execute commands against your systems.

However, I’m lazy. I don’t want to type ssh some_system_name or ssh some.ip.add.ress for every system. Also, on some of my systems I have to specify the private key and username to use for connecting, which further lengthens the amount of typing I have to do: ssh -i ~/.ssh/some_special_id my_account@some.netapp.lan.

I have found it to be convenient and easy to create bash aliases for these systems. It’s simple to do:

Now, whenever I type na01 version it will automatically expand the “na01” to be the full command.

To make the alias permanent, add it to .bashrc file in your home directory…

If you are feeling particularly fancy, you can configure SSH for autocomplete of the hostnames also.

NetApp: Quick and dirty way to start the simulator at system startup

Being a primarily NetApp shop I do a fair amount of testing against their simulator before using any of the perl (and slowly PoSH) scripts against production systems. One of the things that I did a while ago was create a simple way of having the simulator(s) start when my virtual machine starts so that I don’t have to worry about logging in to start it.

NetApp’s documentation for the simulator states two ways of having it start when the server does: using screen to start it in the background, and the more “brute force” method of simply backgrounding the process when it’s started (by appending an ampersand to the end of the command). While both of these methods work, I wanted a way that I didn’t have to login to the system first in order to access the console of the simulator.

Read moreNetApp: Quick and dirty way to start the simulator at system startup

PXE Server Configuration Tutorial

Configuring a PXE server to present the files and information needed for kickstarting your ESX hosts isn’t too difficult a task. It does require some basic unix/linux knowledge, but aside from that, not too bad. I use a CentOS virtual machine with just 256 MB of RAM (you’ll need at least 512 for a GUI, but one isn’t necessary) to act as the PXE server for my ESX hosts. This same virtual machine also serves as a management point, as it has access to the management lan and with the perl toolkit and rCLI installed I can automate much of the work I need to accomplish with the hosts.

I happen to segregate the different types of traffic on the ESX hosts onto different VLANs. This means management (COS/PXE), VMotion, IP Storage, and virtual machine traffic (usually several VLANs by itself) are all separate. It is important that the server (or virtual machine) that you are using is configured with at least one interface on the same VLAN/network that the ESX management network is on. That interface will also need to have a static IP address.

It is also important that DHCP is able to function on this network when the host is in a totally unconfigured state. This means if you are trunking to your ESX hosts you must have the native VLAN set to the same as your management VLAN and port channeling (802.11q / LACP) can not be turned on during the PXE process.

Read morePXE Server Configuration Tutorial

Authentication weirdness…

For some reason, some of my servers have been having trouble with AD authentication. The symptoms were: when logging into the console using an AD enabled account (i.e. an account that should be authenticated by AD), despite providing the correct password the system will return password incorrect. Upon providing the username again, the user is immediately granted access without having to provide a password.

I’m not sure about the provide-once-denied-provide-name-authorized behavior, but after some testing I discovered that the reason that it’s not letting the user in the first time is because the lines for pam_unix and pam_kerberos (pam_krb5 to be exact) in /etc/pam.d/system-auth are reversed. It should have pam_unix before pam_kerberos. I have no clue what is causing these lines to be reversed in the configuration file. I’m configuring all of the authentication and security measures using the esxcfg-... commands so it seems weird that starting with the blade servers this behavior has been exhibited.

Anyway, the fix files: one bash, one awk…

Read moreAuthentication weirdness…

Does anyone know the password for this database?

Those that I work with know that my first, and primary, job is as a MySQL DBA. Unfortunately, cause I love MySQL, I haven’t been doing as much with it lately because of all the virtualization work going on.

Today I’m going to post about MySQL. Occasionally you may encounter a MySQL server that has been around for a while, and no one knows who set it up, where it came from, or who owns it. Those wonderfully inaccessible databases are still someone’s responsibility. So, what do you do if you don’t know the root password? Well, it’s actually not all that difficult, assuming you can start and stop the instance a few times.

Read moreDoes anyone know the password for this database?

Fedora 8 Suspension

I’ve previously mentioned that I use Fedora 8 on my laptop at home. It is a Core 2 Duo Dell with a GeForceGo 7300. Originally, it had Vista Home Premium, and I really did give Vista a chance (for almost 8 months!!), but I just like linux more. I do still have to go back to Vista on the (extremely) rare occasion I need bluetooth support. For some reason I can’t get the integrated bluetooth modem to work with Fedora. The GeForce Go has caused me nothing but problems. Nvidia’s normal drivers won’t work with the Go series from Dell, I have to get the drivers directly from Dell…and they are flaky.

Anyway, I recently reloaded my laptop and let it update everything to the newest available. Unfortunately, at some point, suspend stopped working. I’m not sure when it was (it applied ~ 300 updates), but it stopped. Well, it didn’t exactly stop working…it still suspends, once, after which the monitor refuses to work. I can still ssh in, and everything seems to be functioning normally, but the monitor doesn’t work. Which makes a laptop very useless.

So, since I’ve reloaded linux a number of times, and it seems each time I forget what I did to fix it, I’m documenting it for myself, and posterity.

Read moreFedora 8 Suspension

No wireless networks detected…

I use VMware server on my computers at home. Both of my current systems run Fedora 8, kernel I say this because on my laptop I couldn’t configure a VM to use bridged network mode when wlan0 was the only active interface.

After a lot of googling, I came across this post. The post is almost entirely in german, however there is an abbreviated version somewhere in the middle in english.

Normally, I wouldn’t go any further than posting a link, however while I was reviewing some links on my account, I clicked the above, and discovered that the site has a tremendous number of errors. This is bad. It usually means that the site is not well maintained and not long for the internet.

Additionally, the patch that’s posted is slightly out of date. So, I’ve created an updated patch, and I’m going to post some instructions in english here.

Read moreNo wireless networks detected…

sudo, let me log you doing something stupid

Allow me to step on my security soap box for a moment. I’ve seen in many places around the internet where bloggers will recommend, and explain how, to enable root to login to the console via ssh. I can not tell you enough how bad this is. An attacker no longer needs to guess two passwords to gain root access to the system, but, rather, only one. It is much, much more secure to disallow root access.

Access to the console operating system of ESX should be limited to the absolute minimum. Only users who absolutely need it, and know what they’re doing, should be able to login. From the console, the user has access to all of the configuration and datafiles for virtual machines. With the built-in tools provided by VMware, administrators can mount vmdk files and gain read/write access to a virtual machine’s hard drive. Additionally, because nearly all aspects of the virtual networking configuration can be changed from the console operating system, anyone with access can gain the ability to see all network traffic traveling to and from virtual machines.

Ok, less words, more action…

Read moresudo, let me log you doing something stupid