real quick, wanted to document this for others out there. here’s the steps you need to run dnsdist + dnstap to log/process your dns queries easily
sudo apt-get install -y golang
go get -u github.com/dnstap/golang-dnstap/dnstap
sudo vi /etc/dnsdist/dnsdist.conf
# Add these lines
rl = newFrameStreamTcpLogger("127.0.0.1:8000")
addResponseAction(AllRule(), DnstapLogResponseAction("rdns", rl))
:wq
sudo systemctl restart dnsdist.service
go/bin/dnstap -l 127.0.0.1:8000
And you’re all set.
It’s halfway through 2019 and we still have some major backbones that are not implementing operational best practices. Those operating large networks know the risk of BGP hijacks and other malfeasance. We had a major incident in 2018 that was used to take down parts of Amazon that was tied to crypto currency theft. Real money is lost when these events occur, despite the value that we may individually see as part of this.
Today was the most recent event impacting many providers, directing traffic via a previously unknown provider using a BGP optimizer product from Noction. Many people use solutions like this, but the risks posed by this are regularly seen.
In 2007 I gave a talk at NANOG about some extremely simple mitigations that could be performed to protect one from accepting invalid routes using AS_PATH based filtering. I figure it’s time to link to it again – https://www.youtube.com/watch?v=W9WBBZOfWcA to allow people to see how regularly these occur. The system is still up and running 12 years later here https://puck.nether.net/bgp/leakinfo.cgi showing the problem is ongoing. Today just search for a contact ASN of 396531 to see the problems.
We must put pressure on our providers and operators of backbones to implement things like peer locking and sanity filters to prevent backbone routes to be learned from customers. There is no reason for a provider like Cogent (174) to accept Sprint (1239) or level3 (3356) routes from Verizon Business (701).
120.209.192.0/19 3277 39710 20632 31133 174 701 396531 33154 1239 9808
104.31.88.0/21 3277 39710 20632 31133 174 701 396531 33154 3356 13335
It’s time to end this madness.
I’ve been helping a few people optimize their ADS-B setups recently and wanted to provide a simple aggregated location for people to purchase their parts and see my setup.
– Outdoor Case
– Mounting Plate inside case
– PoE injector 48V
– Raspberry PI 4
– 48V PoE HAT for Raspberry PI 4
– RTL-SDR and Filter (this is really critical!)
– Filter to Antenna cable
– 5dB ADS-B 1090 Antenna or GO BIG, 9dB Antenna and see 300miles when properly mounted
Once you install Raspian you will want to follow the instructions at Flightaware to update to the latest piaware. Previously I had build instructions here but they are no longer necessary as the changes are merged upstream these days.
Once that’s in there, go ahead and edit your /etc/default/dump1090-fa file and make the options look like this:
RECEIVER_OPTIONS="--gain -10 --ppm 0 --net-bo-port 30005 --oversample --phase-enhance"
DECODER_OPTIONS="--max-range 450 --lat x.x --lon -y.y --fix --modeac --enable-agc"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1000 --net-ro-interval 1 --net-ri-port 0 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --forward-mlat"
This should result in a nice setup where you can see 200-300 miles away. You will still need to register with Flightaware, eg:
sudo piaware-config -autoUpdate 1 -manualUpdate 1
sudo piaware-config -mlatResultsFormat beast,connect,localhost:30004
sudo piaware-config -user username -password
Hope this helps you!
I often am standing outside wondering what that plane is flying overhead. Services like Flightaware or even Siri where you can say “Wolfram Alpha Planes Overhead” can help you with this. But most have a delay in the data you receive of 5-10 minutes.
ADS-B (Automatic dependent surveillance) is an automated system for delivering data from planes to surrounding aircraft and ground listeners. All aircraft are required to be retrofitted by 2020 in the US/FAA region.
After spending some time tinkering, I have an optimal setup for ADS-B established at my home which allows me to see 150 planes up to 200 miles away. I wanted to document the parts list for what I did. While Flightaware has a list, here: http://flightaware.com/adsb/piaware/build that list is imperfect and slowly becoming out of date. Most items are available via Amazon Prime.
Required Parts:
* Raspberry Pi Model B+ (B Plus) 512MB
or Raspberry Pi 2 Model B
* ADS-B USB Adapter with antenna $24
* Power for Raspberry PI (2 Amp USB) *or*
* WS-POE-USB-Kit for Raspberry Pi
* 16GB MicroSD card w/ Adapter
Recommended:
* STRONGLY RECOMMENDED: 1090Mhz Filter + Preamp – £41.99 + Shipping (may take 2+ weeks due to customs)
* ADS-B Antenna – $150
* ADS-B Antenna to Amplifier cable – $14
* Amplifier to USB Dongle cable $6
* Weatherproof Enclosure $45
* Fittings to attach box to building/chimney
I’ve been starting to play around with the LB4M as a cheap switching platform. These can be had easily on eBay and other sites for around $100-105, including 2x10G-SR optics as part of the deal. The downside is the switches are perhaps a bit noisy and a bit hard to work with as the CLI and software are a bit difficult to operate with. It’s also not well supported by the manufacturer, and the software.
I’ve decided to create a small archive of the images and data related to this platform. Those can be found here: http://puck.nether.net/~jared/lb4m/
I am hoping to document some of the efforts I’m undertaking with these and any progress I have on getting more modern software, or even something linux based running on the box.
If you know how to do a factory firmware restore on these, please do contact me, even if it requires XMODEM or JTAG. I managed to load the improper firmware on the box such that the Boot Menu does not even appear.
Turns out, the saga may not yet be over. There is a defect in the current version of VirtualBox. I’m using VirtualBox-4.2-4.2.6_82870_fedora18-1.x86_64 right now and there seems to be an issue where IPv6 performance is only ~22Kb/s or so in most of my experiences.
https://www.virtualbox.org/ticket/9380
Hope this helps someone else.
UPDATE:
Disabling GRO seemed to work around the problem for me.
I spent a bunch of time doing “iperf -V -s” between both a VM and a host on the same machine/lan/network interface. The performance one-way would be fast but the other way would be slow with GRO on. Hope this helps you.
So, we often reboot machines with little to no consequences. We reboot our phones, cars, laptops, desktops and even servers. This uneventful thing isn’t what happened to me on Monday.
So, many years ago I moved my machine out of my home and decided it would be a good idea to pool resources with several other people for whom I was either hosting or sharing space with. Being a technology person, I had a T1 at my home from 1997-2010. Friends, and other would share resources with me and I returned the favor in-kind.
I have used a variety of technology over the years from the FreeBSD jail support in 4.8 (with a patch) up to the FreeBSD 7-8 series. Due to personal preference and my desire to spend less time compiling things (Plus the fact that I disagree with FreeBSD packaging, development and have had problems with modern hardware support…) I undertook building a replacement host in 2011.
FreeBSD jail can be quite elegant. You could run multiple servers on one physical hardware, share the pool of disk space, cpu and memory all without being limited to #cpu or memory footprint within a virtual machine as you are with vmware and other systems. Having used vmware in some form since my original 1.0.x license that expired in 1999, I wanted to provide a reasonable service to those I shared with.
I went and moved the system to Linux and the closest thing I could find at the time that wasn’t going to limit the CPU/memory/disk usage was Linux-Vserver.org. This required a small kernel package and was distributed as part of Fedora in the base OS without trouble. There were a few limitations to management, but I was willing to live with them at the time and proceeded to move over ~7 machines to the new hardware. Sometimes I would stand up something for a friend then tear it down, but on Monday there were a total of 8. (One I have left down until that the owner contacts me ..).
So during the Monday reboot, the goal was to upgrade the IPMI interface on the motherboard (SuperMicro X9SCA-F) as well as various firmware on the SAS controller.
What happened next was something that would consume me for the next 48 hours.
Upon rebooting the system, the virtual machines would not start properly. I went and tried to upgrade/downgrade the related packages. Rebuild with the latest kernels and modules… I waited through a very long BIOS and SAS boot up and initalization process (it takes ~45 seconds for the mpt2sas driver to probe my 4 disks) each time I rebooted the machine. When I typed “shutdown -r now” the IPMI interface would show the system actually powered off instead of rebooting. When you are sleep deprived and feeling a small bit of pressure, these small things worse.
At some point approaching 24 hours into the process the decision was made to just move all the systems into VirtualBox. You can judge and whatnot, but it was easy. It was free, and I found documentation online about using qemu-nbd to be able to mount and rsync/move the files from the ~1.8TB /home partition that had puck.nether.net and the other hosts over.
Well, in theory. When I built the system, it was the height of the hard drive shortage. I was also “cheap” and just got 4x1T 7200RPM SATA disks. The case for the chassis is 2U and only has 8 bays. Turns out interesting things happen that slow you down, such as the I/O performance of the RAID 1+0 setup isn’t what you would like. As usual, linear reads can run fast, but the lots of random files that people collect on their systems take a long time to stat() as part of that rsync process. The disk cache never seems like enough, and most filesystems don’t perform well under this load.
After trying to rsync the data over with qemu-nbd, it turned out this was corrupting the new VM vdi file filesystem. One system took 3-4 tries to get it recovered right and I finally had to destroy the file and redo everything. Trying to run 7 parallel rsyncs as well? Will cause some really high numbers with iostat -x … you will see read/write wait times approaching 10+ seconds. I’ve seen some mean numbers this week, and those felt like they were slowing me down. Turns out doing them one-at-a-time may have worked out better, but I was hoping the OS disk cache would work better than it did… Also, when you see these long iowait times, it’s enough to cause an OS in VirtualBox (at least) to time out the emulated disk and reset the internal disk controller(!). This was not expected.
After many hours in the process I decided to take a nap Tuesday morning and got in about 3 hours of sleep. Tuesday night, I got more as I waited for the syncs to happen. Sometimes it’s just OK to leave something down and broken for a bit longer. Nobody was “really” screaming about things, but I felt obligated to fix it ASAP.
Of course, once I started to get the machines turned up there were the inevitable problems. Mailman bounced a lot of mail as it wasn’t permitted by smrsh, but the user email worked ok. The load average on the new VM went very high during the mail processing and would periodically reject the messages.
There’s a lot more that could be included but I wanted to highlight a few last things.. having more spindles good. Having friends that will look at something when you are sleep deprived is good. Perhaps using a VM isn’t as evil as I had originally thought, but still isn’t my first choice. Taking a nap and leaving things broken? Good.
Having a wife that is understanding and didn’t shoot me? Very good. I don’t think she often realizes how much she is appreciated, but she is more than I will share in public here.
Hope everyone is having a better week.. I promise to not upgrade anything else for the next 15 minutes.
I’ve been doing a lot of weird projects around the house, either my indoor rock wall, snow making or other things. As a result, the usual locations of Lowes and Home Depot can be very-expensive or don’t even stock the parts you need. I wanted to make a short list of the things I’ve purchased in recent years, including how and where you can save some major money as a result.
You may think you can’t buy from Grainger because they only sell to a business, but there are a few exceptions to that.
1) If your employer has an account there, you can buy things as an “Employee Purchase”. Typically all you need is the main phone number, but also having the account number can be helpful.
2) You can sign-up for a business account. This may require doing something like setting up a LLC. In Michigan where I live, this is around $50.
3) Find someone else that has an account and use theirs but do an employee purchase as well.
It shouldn’t be hard to do one of these. Here’s the reasons why you might want to.
For my rock-wall, I needed about 200 3/8-16 T-nuts. These are typically very expensive at a place like Lowes. At Grainger (Part# 1XGJ1) they are $16 for a box of 100 instead of almost 50c-$1 each. The same is true for the Socket-Cap screws (bolts) which are about $1-1.50 each at Lowes, you can get a box of 50 (4XE65) for $18. Keep in mind this size works for *most* holds, but some require longer bolts of 2 1/2 inches or more.
For my snow maker, I needed a liquid filled pressure gauge to measure the water PSI. This was not something that was in-stock at Lowes/Home Depot and only cost $20 to arrive next-day at Will-Call.
I’ve also used them to replace the blower motor in my furnace when the local furnace repair shop did not call me back. (Nothing like waking up to the smell of a burning electric motor)!
I wanted to take a moment and explain my claim on the term “Culture of Crisis”. I had a chance several years ago to do some consulting work. Through this I came to have a new found understanding of what can easily happen at any organization if they don’t keep their operational culture in check.
Every company and group faces a challenge when something goes wrong. Some have a detailed process, including the need to write postmortem reports, assign blame elsewhere or to punish those that were responsible.
When one is hit with a crisis, this distracts from whatever else was being done. While some employees may sit idle waiting for the panic button to be pressed, usually the most valuable employees get involved quickly in a crisis as they have experience fixing things.
One has to be careful to avoid an all-hands-on-deck strategy to responding to problems. This can be useful if your team or company is just 5 people. The problem may actually require everyone to solve it. Generally larger organizations do not require this.
Be mindful of how many people you involve in solving your problem. Don’t have a crisis conference call where the business and technical people are together discussing the impact. Split these, but maintain communication. Have a leader willing to ask questions and direct the response. Ask questions. Communicate with the experts. Engage only those necessary. Having everyone join the 911 call can lead to a situation where everyone is there, but nobody is willing to speak up.
The shotgun approach to problem solving is good if you need a large team to solve the problem, such as cleaning up a disaster site. Responding to a technical issue works better having the right people engaged. Too many people and it becomes any other large meeting with people worried about the “internal or external optics” of the event.
Please take 5 minutes today to call both your senators and your representative. If you don’t know who they are, look that up here: https://writerep.house.gov/writerep/welcome.shtml and http://www.senate.gov/general/contact_information/senators_cfm.cfm
Protect IP and SOPA are important for the technical underpinnings of the internet that you depend upon daily. They are easily bypassed by using an IP address to reach sites such as using typing 74.125.225.81 to reach google instead of that name.
A simple script for you to follow if you are not sure what to say:
Hello, I live in (Location, eg: Ann Arbor, Michigan/City/Township) and was wondering what the Senator/Representatives position is on the proposed legislation of SOPA (House – HR 3261) and/or Protect IP (Senate – SB 968).
Be respectful when talking on the phone, and convey any feelings you have on the topic.