Jared Mauch's Comments

So, we often reboot machines with little to no consequences. We reboot our phones, cars, laptops, desktops and even servers. This uneventful thing isn’t what happened to me on Monday.

So, many years ago I moved my machine out of my home and decided it would be a good idea to pool resources with several other people for whom I was either hosting or sharing space with. Being a technology person, I had a T1 at my home from 1997-2010. Friends, and other would share resources with me and I returned the favor in-kind.

I have used a variety of technology over the years from the FreeBSD jail support in 4.8 (with a patch) up to the FreeBSD 7-8 series. Due to personal preference and my desire to spend less time compiling things (Plus the fact that I disagree with FreeBSD packaging, development and have had problems with modern hardware support…) I undertook building a replacement host in 2011.

FreeBSD jail can be quite elegant. You could run multiple servers on one physical hardware, share the pool of disk space, cpu and memory all without being limited to #cpu or memory footprint within a virtual machine as you are with vmware and other systems. Having used vmware in some form since my original 1.0.x license that expired in 1999, I wanted to provide a reasonable service to those I shared with.

I went and moved the system to Linux and the closest thing I could find at the time that wasn’t going to limit the CPU/memory/disk usage was Linux-Vserver.org. This required a small kernel package and was distributed as part of Fedora in the base OS without trouble. There were a few limitations to management, but I was willing to live with them at the time and proceeded to move over ~7 machines to the new hardware. Sometimes I would stand up something for a friend then tear it down, but on Monday there were a total of 8. (One I have left down until that the owner contacts me ..).

So during the Monday reboot, the goal was to upgrade the IPMI interface on the motherboard (SuperMicro X9SCA-F) as well as various firmware on the SAS controller.

What happened next was something that would consume me for the next 48 hours.

Upon rebooting the system, the virtual machines would not start properly. I went and tried to upgrade/downgrade the related packages. Rebuild with the latest kernels and modules… I waited through a very long BIOS and SAS boot up and initalization process (it takes ~45 seconds for the mpt2sas driver to probe my 4 disks) each time I rebooted the machine. When I typed “shutdown -r now” the IPMI interface would show the system actually powered off instead of rebooting. When you are sleep deprived and feeling a small bit of pressure, these small things worse.

At some point approaching 24 hours into the process the decision was made to just move all the systems into VirtualBox. You can judge and whatnot, but it was easy. It was free, and I found documentation online about using qemu-nbd to be able to mount and rsync/move the files from the ~1.8TB /home partition that had puck.nether.net and the other hosts over.

Well, in theory. When I built the system, it was the height of the hard drive shortage. I was also “cheap” and just got 4x1T 7200RPM SATA disks. The case for the chassis is 2U and only has 8 bays. Turns out interesting things happen that slow you down, such as the I/O performance of the RAID 1+0 setup isn’t what you would like. As usual, linear reads can run fast, but the lots of random files that people collect on their systems take a long time to stat() as part of that rsync process. The disk cache never seems like enough, and most filesystems don’t perform well under this load.

After trying to rsync the data over with qemu-nbd, it turned out this was corrupting the new VM vdi file filesystem. One system took 3-4 tries to get it recovered right and I finally had to destroy the file and redo everything. Trying to run 7 parallel rsyncs as well? Will cause some really high numbers with iostat -x … you will see read/write wait times approaching 10+ seconds. I’ve seen some mean numbers this week, and those felt like they were slowing me down. Turns out doing them one-at-a-time may have worked out better, but I was hoping the OS disk cache would work better than it did… Also, when you see these long iowait times, it’s enough to cause an OS in VirtualBox (at least) to time out the emulated disk and reset the internal disk controller(!). This was not expected.

After many hours in the process I decided to take a nap Tuesday morning and got in about 3 hours of sleep. Tuesday night, I got more as I waited for the syncs to happen. Sometimes it’s just OK to leave something down and broken for a bit longer. Nobody was “really” screaming about things, but I felt obligated to fix it ASAP.

Of course, once I started to get the machines turned up there were the inevitable problems. Mailman bounced a lot of mail as it wasn’t permitted by smrsh, but the user email worked ok. The load average on the new VM went very high during the mail processing and would periodically reject the messages.

There’s a lot more that could be included but I wanted to highlight a few last things.. having more spindles good. Having friends that will look at something when you are sleep deprived is good. Perhaps using a VM isn’t as evil as I had originally thought, but still isn’t my first choice. Taking a nap and leaving things broken? Good.

Having a wife that is understanding and didn’t shoot me? Very good. I don’t think she often realizes how much she is appreciated, but she is more than I will share in public here.

Hope everyone is having a better week.. I promise to not upgrade anything else for the next 15 minutes.

I’ve been doing a lot of weird projects around the house, either my indoor rock wall, snow making or other things. As a result, the usual locations of Lowes and Home Depot can be very-expensive or don’t even stock the parts you need. I wanted to make a short list of the things I’ve purchased in recent years, including how and where you can save some major money as a result.

You may think you can’t buy from Grainger because they only sell to a business, but there are a few exceptions to that.

1) If your employer has an account there, you can buy things as an “Employee Purchase”. Typically all you need is the main phone number, but also having the account number can be helpful.
2) You can sign-up for a business account. This may require doing something like setting up a LLC. In Michigan where I live, this is around $50.
3) Find someone else that has an account and use theirs but do an employee purchase as well.

It shouldn’t be hard to do one of these. Here’s the reasons why you might want to.

For my rock-wall, I needed about 200 3/8-16 T-nuts. These are typically very expensive at a place like Lowes. At Grainger (Part# 1XGJ1) they are $16 for a box of 100 instead of almost 50c-$1 each. The same is true for the Socket-Cap screws (bolts) which are about $1-1.50 each at Lowes, you can get a box of 50 (4XE65) for $18. Keep in mind this size works for *most* holds, but some require longer bolts of 2 1/2 inches or more.

For my snow maker, I needed a liquid filled pressure gauge to measure the water PSI. This was not something that was in-stock at Lowes/Home Depot and only cost $20 to arrive next-day at Will-Call.

I’ve also used them to replace the blower motor in my furnace when the local furnace repair shop did not call me back. (Nothing like waking up to the smell of a burning electric motor)!

I wanted to take a moment and explain my claim on the term “Culture of Crisis”. I had a chance several years ago to do some consulting work. Through this I came to have a new found understanding of what can easily happen at any organization if they don’t keep their operational culture in check.

Every company and group faces a challenge when something goes wrong. Some have a detailed process, including the need to write postmortem reports, assign blame elsewhere or to punish those that were responsible.

When one is hit with a crisis, this distracts from whatever else was being done. While some employees may sit idle waiting for the panic button to be pressed, usually the most valuable employees get involved quickly in a crisis as they have experience fixing things.

One has to be careful to avoid an all-hands-on-deck strategy to responding to problems. This can be useful if your team or company is just 5 people. The problem may actually require everyone to solve it. Generally larger organizations do not require this.

Be mindful of how many people you involve in solving your problem. Don’t have a crisis conference call where the business and technical people are together discussing the impact. Split these, but maintain communication. Have a leader willing to ask questions and direct the response. Ask questions. Communicate with the experts. Engage only those necessary. Having everyone join the 911 call can lead to a situation where everyone is there, but nobody is willing to speak up.

The shotgun approach to problem solving is good if you need a large team to solve the problem, such as cleaning up a disaster site. Responding to a technical issue works better having the right people engaged. Too many people and it becomes any other large meeting with people worried about the “internal or external optics” of the event.

Please take 5 minutes today to call both your senators and your representative. If you don’t know who they are, look that up here: https://writerep.house.gov/writerep/welcome.shtml and http://www.senate.gov/general/contact_information/senators_cfm.cfm

Protect IP and SOPA are important for the technical underpinnings of the internet that you depend upon daily. They are easily bypassed by using an IP address to reach sites such as using typing 74.125.225.81 to reach google instead of that name.

A simple script for you to follow if you are not sure what to say:

Hello, I live in (Location, eg: Ann Arbor, Michigan/City/Township) and was wondering what the Senator/Representatives position is on the proposed legislation of SOPA (House – HR 3261) and/or Protect IP (Senate – SB 968).

Be respectful when talking on the phone, and convey any feelings you have on the topic.

I sometimes travel for work and pleasure, but also have packages shipped to my home without some sort of pre-notice of what they are or when they will arrive. For several years I have been subscribed to FedEx Insight to get notices of inbound packages. This is very nice as we could enter in many variations of our home address and people who would receive and get notices of this. It is often incomplete as shippers can block this visibility or it may come to yet another variation that we are unaware of.

UPS has finally launched a consumer-oriented version of this service they dub “My Choice”. One can sign-up online here and get notices when packages are inbound, usually the night before. What’s even better here is you can ask them to hold/delay delivery, and see what items may be signature required. They also give an estimated time of delivery based on the schedule and expected load of the drivers.

I’ve already come to love this capability as it can easily identify when a package is coming to our home. If you miss your driver, or have people ship items to you unexpectedly I strongly suggest you sign up for this service, as well as the FexEx variant. FedEx will call you to verify each addition, even if it’s just an address variant (eg: LN vs LANE).

Gizmodo has been banned from WWDC 2010, and it sure seems like they’re going down the “Please-Hurt-Me” path. They are looking to violate the “hot news doctrine” which has been decided case law since 1918 (248 U.S. 215).

They are going to aggregate and re-publish other “hot news” from their competitor blogs. This was ruled illegal then, and surely applies in this case where they were BANNED and can not blog live legitimately.

Should be interesting to see what happens tomorrow.

It’s been reported recently that part of the FCC broadband plan is the desire to move some broadcast TV stations out of the 500Mhz frequency band so this can be used for cellular companies to have more bandwidth for their userbases.

This is entirely the WRONG move. Much more can be attained by using smaller cellular sites, and the deployment of picocell/microcell technology. If each home had even just 5Mb/s internet access, and a microcell device, the need for this would be minimized and cellular customers would be happier with increased coverage at their homes/offices.