automated nmap scans

published on 2009-03-13 in computing

Whipped this up for work, figured I'd share with the world, since it's decently useful. Stick it in cron nightly, needs to run as root. It will run a diff on what it sees and email you if there are new ports/hosts that pop up on your networks. If you find errors or mods, use this: http://pastebin.com/f635a7517 to modify it and post in the comments.

#! /bin/sh

DIR="/opt/nmap/scans"  
NETWORKS="192.168.1.0-255"  
TODAY=`date +%Y%m%d`  
YESTERDAY=`date -d yesterday +%Y%m%d`

for network in $NETWORKS  
do  
nmap -n -sS $network -oG $DIR/$network.$TODAY.nmap  
done

for network in $NETWORKS  
do  
diff -I "^#" $DIR/$network.$TODAY.nmap
$DIR/$network.$YESTERDAY.nmap > $DIR/$network.$TODAY.diff  
done

for network in $NETWORKS  
do  
SIZE=`find $DIR/$network.$TODAY.diff -size +0b`
if [ "$SIZE" = "$DIR/$network.$TODAY.diff" ]  
then  
cat $DIR/$network.$TODAY.diff | mail -s "Change Detected for
$network" user@host.com  
fi  
done

automated disk partitioning with sfdisk

published on 2009-01-04 in computing

I discovered sfdisk a few years ago (part of util-linux) and have been using it in automation scripts ever since.  sfdisk is like fdisk, but is scriptable.  So for example, to list the partitions on a disk:

[root@host]# sfdisk -l /dev/sdc

Disk /dev/sdc: 121601 cylinders, 255 heads, 63 sectors/track  
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from
0

Device Boot Start End #cyls #blocks Id System  
/dev/sdc1 0+ 121600 121601- 976760001 83 Linux  
/dev/sdc2 0 - 0 0 0 Empty  
/dev/sdc3 0 - 0 0 0 Empty  
/dev/sdc4 0 - 0 0 0 Empty

To list them in a dump format, suitable as input to sfdisk (for cloning, saving or for some wacky awesome script):

[root@host ]# sfdisk -d /dev/sdc# partition table of /dev/sdc  
unit: sectors

/dev/sdc1 : start= 63, size=1953520002, Id=83  
/dev/sdc2 : start= 0, size= 0, Id= 0  
/dev/sdc3 : start= 0, size= 0, Id= 0  
/dev/sdc4 : start= 0, size= 0, Id= 0

You can use that dump in a fashion like this to clone a disks's partition map:

sfdisk -d /dev/sdc | sfdisk /dev/sdd

Or for saving it and using it later:

sfdisk -d /dev/sdc > partition.sfdisk  
...  
sfdisk /dev/sdc < partition.sfdisk

End of 2008

published on 2009-01-01 in motorcycles , photography , video

I have done a few things in the last few months that are worthy of mention. I haven't had much of a chance to blog about them or write them down, what with them all being back to back and then holidays, being sick, vacation, more holidays, more being sick. But here are some links to the media I've produced. Enjoy.

2 Adventurers

San Diego CA USA to Cabo San Lucas, Baja California Sur, MX (in 6 days, 2200+ miles, on my new BMW adventure motorcycle)

One month of beard growth in 5 seconds (an experiment in time lapse)

Deleting that same beard at high speed (an experiment with tracy's camera)

The mechanical wonderment behind a bowling alley machine

ssh tab completion on known_hosts

published on 2008-11-05 in computing

It's silly I've waited this may years to go figure this out. Many of you may already know that modern installs of OpenSSH will tab complete hostnames based on what's in the /etc/hosts file. But there is a neat little addition to your .bashrc that will tack on the ability to tab complete hostnames based on what's in ~/.ssh/known_hosts. Add this to your .bashrc:

SSH_COMPLETE=( $(cat ~/.ssh/known_hosts | \\  
cut -f 1 -d " " | \\sed -e s/,.*//g | \\  
uniq ) )complete -o default -W "${SSH_COMPLETE[*]}" ssh

All your new shells will auto complete based on what hosts you've connected to once (and therefore have entries in the known_hosts file). Any host you've never visited, well it won't be there. If you want to filter it based on certain hosts (for example, hosts in a certain domain name), just add a | grep domain.com after the uniq. If you're like me, this will save a lot of keystrokes over the next few years.

Tip: If you cut and paste my text above and it gives errors, make sure your cut-n-paste didn't change the quotes. If you want to see what it's going to use (or troubleshoot/modify), you can run this on the command line:

cat ~/.ssh/known_hosts | \\  
cut -f 1 -d " " | \\  
sed -e s/,.*//g | \\  
uniq

New Midpipe

published on 2008-09-28 in motorcycles


Out for a hyperspin

Installed a new LeoVince midpipe on the Hypermotard earlier in the week (2am in the garage with a rubber hammer the night before an early meeting). Finally got a moment away from the keyboard to take it out for a spin tonight. Total awesomeness. It starts better, sounds better and runs better. I think it's a bit faster too. How awesome is that for the effort and price? I should have done this a long time ago.

Pics: Old and Busted. New Hotness.

I had also adjusted the bars a bit to be a tad higher, but I didn't like it. The turn signals are pointed at the ground 5 feet in front of me, which means my effective road use brightness went from 'are those lasers?' to 'dead lightning bug'. I may pick a position half way between this position and neutral. Will have to test it again later in the week when I've got some more time.

HD Video Test

published on 2008-08-13 in video

Finally got myself a video camera. Been looking at them critically for about 8 years now...ever since my Hi8 cam broke in 2000. I loved making movies back then, and even lately have been doing various time-lapse type videos...not really 'movies' since they were shot with a SLR. A few years ago I got interested in video again with the advent of consumer level HD cameras. Canon came out with the HF10 and HF100, soda-can sized cameras that record 1080p HD to SDHC flash cards...no moving parts, great for strapping to cars/motorcycles, etc. When the price finally fell to a range I could deal with, I pulled the trigger. It's so cool...and little! This is my first test. Video quality is great! I hope to be posting more mischief and fun events in the coming months.

Is Vimeo the best place to host HD video? Anyone? Bueller? Anyway, enjoy some test shots from the backyard...

HD Test - Backyard Oddities

changing your server over to GMT

published on 2008-08-05 in computing

Funny how it's kind of hard to find documentation on this fairly simple task. The host I'm working with is CentOS 5, a variant of Redhat Enterprise 5. I use the following commands:

cat /usr/share/zoneinfo/GMT > /etc/localtime

That is actually what sets the time. It's a binary file and is what our system tools and libraries look at.

vi /etc/sysconfig/clock

Edit the ZONE to be GMT. Mine looks like this:

ZONE="GMT"
UTC=true
ARC=false

Verify it looks right with these commands.

date ; date -u ; /sbin/hwclock --show

Reboot, done!

Statistics, Charts and R

published on 2008-06-02 in computing

R is an open source environment for statistical computing. It can do some pretty neat breakdowns of your data and has a lot of built in functions for doing so. One of it's great strengths is generating production quality graphics and charts. This is what I needed it for and what I'll be explaining here in a moment. I learned R by watching a video introduction to R created by Decision Science News. There were 2 actually. But not very long and it got me to a base level. I then installed R on my Mac, it was cake. Go to the R site, download the DMG, run the R executable and you're ready to go. That got me up and running so I could start playing around on my own and using other examples from the web. YMMV on other platforms.

Now for the example. Let's set the stage. Say you have some data in a table, for example, a race my girlfriend competed in, the 2006 San Diego 10K race. I copied, pasted that data into a file, scrubbed it down, did some math with perl to get me the # of seconds, and ended up with a CSV file. Download the file, save it locally, read that file in with R:

race<-read.csv("race.csv")

You are reading that CSV file in as a table into a variable called 'race'. Because that CSV has a header as the first line, it automatically assigns variables based on those column names. To reference those columns, use race$CITY, to check out the 'CITY' column. So to check out what you've just done, type "race</span>" on the console. Typing the variable name will spit it all back out. To see a breakdown of what that variable contains, type:

summary(race)

To see stats on the racers ages, type in:

summary(race$AGE)

Which spits out:

Min. 1st Qu. Median Mean 3rd Qu. Max.  
10.00 28.00 35.00 37.06 44.00 81.00

Minimum age of a runner was 10, oldest was 81. Average age was 37.06 years old. Doing this for race$SEX shows us there were 411 women and 474 men. Neat! Now for the visuals:
Age vs
Time
Below is a script I used to generate the graph above. You can see how I am plotting the dots, and drawing both lines:

race<-read.csv("race.csv")

Main Plot.

plot(race$SECONDS/60,race$AGE,  
col="#5fae27",  
main="",  
xlab="Minutes",  
ylab="Age",  
cex=0.5,  
type="p")

Set the Title

title(main="Age vs Time")

Draw the Red Line

lines(stats::lowess(race$SECONDS/60,race$AGE,f=0.1),  
col="red",  
lwd=2)

Draw the Blue Line

lines(stats::lowess(race$SECONDS/60,race$AGE,f=0.3),  
col="blue",  
lwd=3)

Not too hard, not too much code...pretty easy in fact! One of the great things about R is the built in help. Any of those functions, just type: ?function ..and you'll have immediate help. I encourage you to do that for the example above, to better understand it. It will describe far better than I can how each one of those functions works.

Let's generate another one, a histogram. That's easy:

hist(race$AGE,col="RED",xlab="Age",breaks=100,main="Histogram of
Racers Age"

To generate this:
2006 San Diego Super Run
10K
So, what did I learn from the creation of this plot? My initial suspicion was that younger people would do better in the race...the data shows that is's almost average across the board. The average age is in the late 30's, but the histogram shows the biggest group was mid-late 20's. Hardly anyone in their early 20's even entered the race...too busy drinking? Also, there is a neat little cluster at the bottom left of the plot that shows a group of young kids in their teens that did well.

I have been making more of these, mostly around sysadmin type stuff. I'll post those as I get more time.

I got Joe Jobbed!

published on 2008-04-16 in computing

In case you've never heard of this, getting 'Joe Jobbed' is when a spammer uses your e-mail address as the 'From:' in their spam and you get all the billion bounces. It rendered my Blackberry useless all afternoon, and I thought I'd finally be screwed on this one. Alas, SpamAssassin 3.2.0 and above (I'm running 3.2.4) comes with rules to block this. Unknown to me, most of the bounces were already going to my spam folder, but any that didn't have the spam body were getting to me. I followed the instructions on this page: http://wiki.apache.org/spamassassin/VBounceRuleset to make sure it was all set up and wouldn't classify bounces from my own server as bad. Then I set up the following procmail rule to put bounces in their own folder that I can review and run stats on:

:0:
* ^X-Spam-Status:.*ANY_BOUNCE_MESSAGE.*
bounces

Done!

More good reading on this: http://www.sitepoint.com/print/sabotage-coping-joe-job

UNIX paste, sed and nl commands

published on 2008-02-27 in computing

3 highly useful commands you will find as part of your standard UNIX toolbox. I'll give an example situation for relevance:

I quite often find myself needing to merge 2 files together for some reason or another. My latest awesomeness consists of ripping/encoding favorite seasons of DVD's I own so my MediaCenter can have an easily accessed library (that I can also stream to my iPod Touch). When the encoding is done, I get files based on the name of the DVD Media and the track number. Like this:

Blah Season 1 Disc 1-1.mp4  
Blah Season 1 Disc 1-2.mp4  
Blah Season 1 Disc 1-3.mp4  
Blah Season 1 Disc 1-4.mp4  
Blah Season 1 Disc 2-1.mp4  
Blah Season 1 Disc 2-2.mp4  
Blah Season 1 Disc 2-3.mp4  
Blah Season 1 Disc 2-4.mp4

I can go to a place like Wikipedia or Amazon and find a list of the track names, that should look like this:

Pilot  
The Fat Man  
Little John  
Howard  
The Reconing  
Half Way  
Blah  
Blah Pt 2

What I want to end up with is files named something like this:

Blah - S01E01 - Pilot.mp4  
Blah - S01E02 - The Fat Man.mp4

It'd be so much easier to rename these on the command line if I could at least partly automate it. Re-typing is a PITA. So, here is how I do it to save a lot of time...

If I assume that they were ripped in order, I can get a listing of the order of the episode files based on the time stamps (ls) and add a " to the front and end of the line (sed commands):

# ls -tr *.mp4 \\  
| sed 's/^/"/g' \\  
| sed 's/$/"/g' > tracklist.txt

Next I go to Amazon, Wikipedia, whatever. Find a list of episodes that is represented with tables (which cut-n-pastes as individual lines with tabs as the delimiter) and paste it into a vi edit session:

# vi episodes.txt

In general, edit this file down to a point that the only thing that exists are the track names, one per line. Let's pretend the first column contained the name of the track, 2nd column the Writer, etc. We only care about the first column, so you can execute this command in vi:

:%s/\\t.*//g

You can do other various cleanup like removing the " character:

:%s/"//g

Clean up other stuff like invalid shell characters, extra spaces, etc. This is the least automated part, but a hell of a lot faster/easier than re-typing. Especially if you are a vi whiz. If you use some other text editor, I'm sure this can be accomplished in a similar fashion.

Now we have 1 file that is the list of mp4 files, in order, and another that is the episode names, in order. What we now need is to make a file with what we actually want the files to be named. I accomplish this with the following command-line awesomeness:

nl -n rz -w 2 -s " - " episodes.txt \\  
| sed "s/^/\\"Blah - S01E/g" \\  
| sed "s/$/.mp4\\"/g" \\  
> newnames.txt

To break that down, this is what is happening:

The nl command adds numbered lines...the -n means "right justified, padded zeros" and the -w means "padded with 2 characters" and the -s means "separate the columns with what's in the quotes".
The 2 sed commands add a " to the beginning and end of the lines.
The > sends the output to a file.

Output looks like this:

"Blah - S01E01 - Pilot.mp4"  
"Blah - S01E02 - The Fat Man.mp4"  
"Blah - S01E03 - Little John.mp4"  
"Blah - S01E04 - Howard.mp4"  
"Blah - S01E05 - The Reconing.mp4"  
"Blah - S01E06 - Half Way.mp4"  
"Blah - S01E07 - Blah.mp4"  
"Blah - S01E08 - Blah Pt 2.mp4"

Now we merge the 2 files and prepend the 'mv' command to get a script we can run:

paste tracklist.txt newname.txt \\  
| sed 's/^/mv /g' \\  
> script.sh

Which looks like this:

mv "Blah Season 1 Disc 1-1.mp4" "Blah - S01E01 - Pilot.mp4"  
mv "Blah Season 1 Disc 1-2.mp4" "Blah - S01E02 - The Fat Man.mp4"  
mv "Blah Season 1 Disc 1-3.mp4" "Blah - S01E03 - Little John.mp4"  
mv "Blah Season 1 Disc 1-4.mp4" "Blah - S01E04 - Howard.mp4"  
mv "Blah Season 1 Disc 2-1.mp4" "Blah - S01E05 - The Reconing.mp4"  
mv "Blah Season 1 Disc 2-2.mp4" "Blah - S01E06 - Half Way.mp4"  
mv "Blah Season 1 Disc 2-3.mp4" "Blah - S01E07 - Blah.mp4"  
mv "Blah Season 1 Disc 2-4.mp4" "Blah - S01E08 - Blah Pt 2.mp4"

Check the script for sanity, then run it!

bash -x script.sh

w00t!

If you want the shell script I use to automate this somewhat:

#! /bin/sh

ls -tr *.mp4 \\  
| sed 's/^/"/g' \\  
| sed 's/$/"/g' \\  
> tracklist.txt  
nl -n rz -w 2 -s " - " episodes.txt \\  
| sed "s/^/\\"$1 - $2E/g" \\  
| sed "s/$/.mp4\\"/g" \\  
> newnames.txt  
paste tracklist.txt newnames.txt \\  
| sed 's/^/mv /g' \\  
> script.sh

The arguments are the name of the series and the season, like this:

bash ./rename.sh Blah S01

P.S. If you're trying to guess the show by the track names, I made them up. :)