This post explains the easy way to get “fuzzy” search results when using sunspot with ruby on rails. This is probably obvious to solr experts out there, but I found the information to be lacking in the rails community. I originally had compared the results of solr vs sphinx for fuzzy searching. In the original comparison I was using the Levenshtein distance for solr, which turns out not to scale well and doesn’t always return the best results when there are exact matches.

For this reason, we switched to using the “Double-Metaphone” algorithm for fuzzy searching in solr. It provides a simple way to get fuzzy results for solr searches while still being able to scale well since most of the work is done at the time of indexing.

Here is how to make it work using Sunspot.

Edit solr/conf/schema.xml and add the following line to the <analyzer> section.

So it looks something like this.

Note: order is important and you will want this towards the bottom of the config block. See the Solr Wiki for more detailed information

Last step, restart solr and reindex. Then test your searches with misspellings etc.

I lot has changed since my original post on automating my backups to s3 using s3sync. There are more mature and easier to use solutions now. I am switching because using s3fs gives you much more options for using s3, it is easier to set up and it is faster.

I now use a combination of s3fs to mount a S3 bucket to local directory and then use rsync to keep up to date with my files. The following directions are geared towards Ubuntu linux, but could be modified for any linux distribution and Mac OSX.

STEP 1: Install s3fs

The first step is to install s3fs dependencies. (Assuming Ubuntu)

sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev libfuse-dev

Next, install the most recent version of s3fs. As of now the most recent is r177, but a quick check of s3fs downloads will show the most recent.

wget http://s3fs.googlecode.com/files/s3fs-r177-source.tar.gz
tar -xzf s3fs*
cd s3fs
make
sudo make install
sudo mkdir /mnt/s3
sudo chown yourusername:yourusername /mnt/s3

STEP 2: Create script to mount your Amazon s3 bucket using s3fs and sync files.

The following assumes you already have a bucket created on Amazon S3. If this is not the case, you can use a tool like s3Fox to create one.

Choose a text editor of your choice and make a shell script to mount your bucket, perform rsync, then unmount. It is not necessary to unmount your S3 directory after each rsync, but I prefer to be safe. One mistake like an ‘rm’ on your root directory could wipe all of your files on your machine and your S3 mount. You should probably start with a test directory to be safe.

Make the file s3fs.sh

#!/bin/bash
/usr/bin/s3fs yourbucket -o accessKeyId=yourS3key -o secretAccessKey=yourS3secretkey /mnt/s3
/usr/bin/rsync -avz --delete /home/username/dir/you/want/to/backup /mnt/s3
/bin/umount /mnt/s3

Note, the –delete option. This will delete any files that have been removed on the ‘source’.
Change permissions to make executable

chmod 700 s3fs.sh

Before you run the entire script, you might want to run each line separately to make sure everything is working properly. The paths to rsync, umount might be different on your system. (Use ‘which rsync’ to check) Just for fun, I did a ‘df -h’, which showed I now have 256 Terabytes available on the s3 mount!

Next, run the script and let it do its work. This could take a long time depending on how much data you are uploading initially. Your internet upload speed will be the bottleneck.

sudo ./s3fs.sh

That’s it! You are backing up to Amazon S3. You probably want to automate this using cron after you are sure everything is running o.k. Just for simplicity of this tutorial, lets assume you are setting up the cron job as root so we don’t need to worry about editing permissions for mount/umounting directory.

STEP 3: Automate it with cron

sudo su
crontab -e
0 0 * * * /path/to/s3fs.sh # this runs it everyday at midnight

p.s. I use this in combination with hourly backups to a second local machine using git to have revision history. I only backup nightly to s3 without revision history in case my house burns down etc. If you would like to know how I set up my git backups locally, just leave a comment and I can make a follow up post.

Download the Google Docs & Spreadsheets toolbar button here.

Enjoy!

A jpeg is good for a lot of things, but a live data feed is not one of them. In fact it might be violation of the Americans with Disabilities Act.

The Seattle Fire Department has been hosting a live html feed of the latest 911 calls to the Seattle Fire Department. Many people have found this information useful for various reasons, eg. avoid areas of major accidents/activity. I have been using this data as a feed for seattle911.com, which is a classic google maps mashup that displays the 911 call data on a map of the Seattle Area. I host the site at my own expense, without any advertising.

Last night, I noticed www.seattle911.com was suddenly broken. After 30 seconds of investigation, I found out that they swithced their data feed from text to a jpeg.

See it in action for yourself.

Here are their reasons for switching the feed to a jpeg, I found via a message board at psrg.org

“The following message has been posted to our web site: “PLEASE NOTE: To address security concerns raised by the public safety community, Seattle Fire Department now displays current incident data as an image within your browser rather than as plain text. The information displayed in the image is updated every minute and then automatically refreshed to the screen. Historical incident information will continue to be displayed as text, just as it has been in the past. We are sorry for any inconvenience this action has caused. Our intent is to enhance the safety of personnel and the public but still provide information about current emergencies in our community. I’ve posted the above on the real-time page.”I know this does not address your specific concerns but it is the reason for the change.”

When I questioned the safety concerns, he responded:

I cant go into details but putting the information in text (data) format allows people to pipe the data into other computer programs to instantly analyze patterns. An image makes that very difficult (although not impossible). We don’t want to make it easy for the “bad guys”.

Rob Bruce Miller wrote: Starting today, SFD implemented a change, which When I questioned the safety concerns, he responded: “I can’t go into details but putting the information in text (data) format allows people to pipe the data into other computer programs to instantly analyze patterns. An image makes that very difficult (although not impossible). We don’t want to make it easy for the “bad guys”.”

I think this might be a case of “we try to make it harder for the bad guys, who will still be able to do what they want, but in reality we just hurt everybody else’s fair use.”

If you want to show your support or opposition to this change, here are few email addresses:

Nick Licata, City Council Person who oversees Public Safety: nick.licata {at] seattle.gov
Gregory Dean, Seattle Fire Chief (through his assistant): debbie.brooks [at} seattle.gov

Here is a screenshot of seattle911.com before the lost data feed.

I am not really concerned about losing the feed for my site (it is less work for me anyway), but this change does nothing but hurt those who were using the feed for legitimate purposes, and does nothing for preventing the so-called “bad guys” from using the information.

UPDATE 1: Just to prove how silly this idea is, I used this one line to extract all data from the new “jpeg” method. If I can do this, I am sure the “bad guys” could as well.djpeg -pnm -gray text.jpg | gocr -

UPDATE 2: I like reddit.com. Here is a link to the comments over there. This page was the number one link at reddit for awhile last night which sent over 20,000 unique visitors here over a 24 hour period.

UPDATE 3: This topic was covered by the Seattle PI, Seattlest and many more.

UPDATE 4: This post has been picked up by Slashdot. Also, the jpeg thing can be thwarted by one line curl "www2.cityofseattle.net/fire/realTime911/sfdIncidentList.jpg" | djpeg -pnm -gray | gocr - Credit probablycorey from reddit for piping it in via curl. I probably would have made it 2 lines. Silly me.

UPDATE 5: Article at Programmableweb.com

UPDATE 6: A link to a Washington State law regarding this topic. Credit amemily (462019) from Slashdot.

UPDATE 7: A lawsuit related to this issue.

UPDATE 8: An interesting perspective on Open Source Intelligence.

UPDATE 9: An article in my home state newspaper – Lincoln Journal-Star

Jeremy Zawodny has an excellent article/discussion about the different tools currently available to take advantage of Amazon simple storage service (S3). After testing many tools available for S3 currently, I decided to use the ruby program s3sync to backup my data to S3.
As I explained an earlier post, I wanted a simple low level tool to perform automatic backups S3. I decided to use s3sync to do the heavy lifting and use the jets3t Cockpit GUI to monitor my S3 account. The following explains how I successfully started automating my backups to S3 using s3sync and cockpit.

My server is running Ubuntu Dapper with samba server. All the machines in my house use a “Public” drive on the samba server to store all files from Windows and Linux. All of our important files like photos, home movies, and documents are stored on this “public” drive. This simplifies the backup procedure, since I don’t have to backup multiple sources.

The following steps describe how I backup my “public drive” to Amazon’s awesome S3 storage service. I decided to post this, because I haven’t found a fairly “simple” guide to actually automate backups to S3 that functions similar to rsync on Linux. This is a follow-up post to my original post on choosing a backup solution.

STEP 1: Activate an Amazon s3 account.

Go http://www.amazon.com/s3 and sign up for a s3 web service account

Have your Access Key ID and your Secret Access Key handy.

STEP 2: Install a management tool

(update, I no longer use cockpit, now I use the command line tools that come with s3sync that were not available at the time I wrote this original article, see Option 1.)

Option 1 use the command line shell tools that are included with s3sync (my new preferred method)

Here is a sampling of the commands from the readme file for command line tool, s3cmd.rb that can be used to create buckets and verify upload success or failure. If you use, this option, make sure you have the correct version of ruby installed on your system and you have downloaded the s3sync package (See step 3)

List all the buckets your account owns:

s3cmd.rb listbuckets

Create a new bucket:

s3cmd.rb createbucket BucketName

Delete an old bucket you don’t want any more:

s3cmd.rb deletebucket BucketName

Find out what’s in a bucket, 10 lines at a time:

s3cmd.rb list BucketName 10

Only look in a particular prefix:

s3cmd.rb list BucketName:startsWithThis

I plan to write a shell script to verify success of backup and run via cron job each night, but I haven’t done it yet. I will update here when I do.

Option 2 (original option that I used before s3sync command line shell tools were available)
UPDATE: I have had trouble getting this (or any other GUI) to work for folders containing large amounts of files. If you plan to have thousands of files stored at Amazon, then I suggest option 1.

Download a GUI tool and make sure you can log into your S3 account, create a bucket, add files, and delete them.

I have tried a lot of them, but I prefer jets3t Cockpit. It is java and open source, plus it is able to read objects uploaded to S3 by other tools. Some tools like Jungle Disk create buckets and objects in a propietary format. This means you would not be able to see your files uploaded to S3 by other tools using JD.
Here is a screenshot of Cockpit.

Create a bucket that you will store your backups in. Make sure to give your Bucket a unique name, because bucket names have to be unique for all users of S3. Many recommend to use your Access Key ID from S3 as a prefix. For example, fakeaccesskey1234.backups. For the rest of this article, I will assume our bucket name is “mybucket”.

Cockpit will be a handy tool for you to monitor your backups in S3, but the actual file uploading/downloading will be done with a shell script using s3sync.

STEP 3: Install s3sync (ruby)

s3sync is an open source ruby script that acts similar to rsync, the linux file sync program. Remember to read the README file from s3sync. Also, all the normal warnings apply. Test this on a couple folders and files you don’t care about and make sure you understand what you are doing. Put the source/destination in the wrong order while using the –delete option and you could blow away all of your precious data.

Lets move on.

The following apply to a Debian/Ubuntu based distribution, but could easily be adapted to your own distro.

First, make sure you have ruby 1.8.4 or greater and the ssl lib for ruby or higher

$ sudo apt-get install ruby libopenssl-ruby

check ruby version

$ ruby -v
ruby 1.8.4 (2005-12-24) [i486-linux]

change into the directory where you want to install s3sync, like /home/john/s3sync

download and unpack s3sync

$ wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
$ tar xvzf s3sync.tar.gz

clean up

$ rm s3sync.tar.gz

make directory for ssl certificates and download some (important, read README for info about these SSL certs)

$ mkdir certs
$ cd certs
$ wget http://mirbsd.mirsolutions.de/cvs.cgi/~checkout~/src/etc/ssl.certs.shar

run this shell archive

$ sh ssl.certs.shar

get back into main s3sync dir

$ cd ..

create two files with your favorite editor, upload.sh and download.sh with the following contents and update to suit your needs. (Important, like rsync, slashes matter, see README for examples)

upload.sh —————————————-

#!/bin/bash
# script to upload local directory upto s3
cd /path/to/yourshellscript/
export AWS_ACCESS_KEY_ID=yourS3accesskey
export AWS_SECRET_ACCESS_KEY=yourS3secretkey
export SSL_CERT_DIR=/your/path/to/s3sync/certs
ruby s3sync.rb -r --ssl --delete /home/john/localuploadfolder/ mybucket:/remotefolder
# copy and modify line above for each additional folder to be synced

download.sh —————————————-

#!/bin/bash
# script to download local directory upto s3
cd /path/to/yourshellscript/
export AWS_ACCESS_KEY_ID=yourS3accesskey
export AWS_SECRET_ACCESS_KEY=yourS3secretkey
export SSL_CERT_DIR=/your/path/to/s3sync/certs
ruby s3sync.rb -r --ssl --delete mybucket:/remotefolder/ /home/john/localdownloadfolder
# copy and modify line above for each additional folder to be synced

NOTICE: These scripts use the –delete option. This means it will delete any file on the destination not on source. Also, these shell scripts contain your Amazon secret info, so you will want to make sure they are only readable by you (chmod 700, credit Kelvin below). You can also add the “-v” option, so you get a verbose about of the changes. I did this this after my initial upload, so I can monitor activity via cron job emails.

Create the local upload and download directories and put some test files in the upload folder

$ mkdir localuploadfolder
$ mkdir localdownloadfolder

change the permissions on the files

$ chmod 700 upload.sh
$ chmod 700 download.sh

Test upload.sh

$./upload.sh

Use s3cmd.rb or Cockpit to make sure you can see the files made it to Amazon.

Test download.sh

$ ./download.sh

The files you uploaded to S3 should now be in your localdownloadfolder.

Once you are confident everything is working fine and your understand what you are doing. Change the shell scripts to backup your actual folders. Run the scripts manually first to ensure everything is working properly. Remember, the upload script will be limited to the upload speed of your ISP, which can be very slow. If you have a typical Cable internet connection upload speed of 384 k it will take approx. 6 hours to upload 1GB. Download speeds are usually much faster, approx 1GB/20 min, but hopefully you never need it.

STEP 4: set up cronjob to run backup script once a week/month etc.

Once you are sure the script is working for your uploads, you can automate the task by creating a cron job to run once a week, day or month. I have it run once a week, because I do nightly backups locally to my Desktop machine using rsync.

$ crontab -e

add the following line.

30 2 * * sun /path/to/upload.sh

save and exit.

Obviously, monitor to make sure everything is working.

STEP 5: kick back and relax

Now you can relax, if your laptop battery explodes and burns down your house, you know your data is safe sitting on Amazon’s geo-redundant servers right between some bits describing a new book from Oprah and a bad review on latest Ben Affleck movie!

Feel free to leave a comment if you find this useful, incorrect, or just plain uninteresting.

UPDATE 1: One additional step I did, was to create one additional bucket where I uploaded all the necessary code/scripts to restore my files using s3sync (minus my s3 information).

UPDATE 2: I have changed the chmod 755 to chmod 700 to make script not readable to all. (Credit Kelvin below). Also, updated the information about the tools I use. I no longer use cockpit to verify success, but I mostly rely on the s3sync command line tools there were not present at the time I wrote the original article.

UPDATE 3: I never gave enough credit to the actual author of s3sync. Without him, this entire process would not be possible, thanks again.

O.k. I am a classic digital packrat. My actual garage is very clean and organized, but my digital garage is full of every byte I have saved since 1998, old Job emails, photos, music, websites, some videos, etc. I never, ever want to lose this stuff, even if my house burns down because my laptop battery explodes. So I need to backup my stuff, simple huh? Well, I already store all of our files on a Linux file server via Samba. This machine is backed up nightly to my Linux Desktop machine with cron and rsync, but this doesn’t solve the fire/flood issue. I used to copy data to DVD and store off-site, but burning DVDs is a pain and my data is out growing DVDs. I currently have approx. 10GB of data, but expect it to grow, especially since I want to store our digital camcorder footage as well. So I began the quest for remote storage. While these are not all of the options I considered (I looked/tried many) they came out at the top of the heap for me.
My dream remote storage.

1. Cheap, of course ($10 or less per month)
2. Secure, some of my files have info I don’t want in the wrong hands.
3. Relatively fast, but I am sure my upload speed (768 k) from my ISP, Comcast will be the bottleneck.
4. Some sort of Linux server of with shell access (so I can automate with cron and rsync)
5. Not have to set up another full fledged energy hog of a server
6. Bonus, unlimited scalability.

How to store files offsite? Let go searchin’…….

Option 1: Amazon S3
This was actually my first choice, but I haven’t been able to find a good tool to use the service. I tried JungleDisk (on linux and windows), but JD had so many crashes I decided to never use it again, but I am still interested in S3. I have heard people express concerns of whether Amazon S3 storage will be around forever and I cannot answer that. While I do think Amazon S3 will be successful, I do trust Amazon over most to provide adequate time to move off of their service if they ever decide to discontinue it.

Pros:

• From Amazon, a big name company with a very good reputation
• Relatively cheap at $0.15/GB stored and $0.20/GB transferred
• Infinetly scalable and stored by Amazon in geographically distributed data centers (huge plus)
• Reliability… I assume this would be the most reliable of all my options

Cons:

• No simple way to use rsync to upload data

Summary: I just wish there was a reliable tool to get my data uploaded like rsync/ssh. s3sync looks promising but I haven’t tried it yet.

Option 2: Dreamhost.com
After researching S3 for awhile, I thought maybe I should just get another hosting account. Dreamhost had everything I wanted, shell access, plenty of storage (200GB) and only $40/year after googling for coupon code. Sounds to good to be true. 1and1 had similar offerings, but I have heard to many horror stories to try 1and1.

Pros:

• Cheap
• Relatively good reputation
• Shell access
• Ability to host websites there as well

Cons:

• Security concerns

Summary: Well they obviously oversell, they even admit it. But that didn’t bother me, if I could use the 10-50GB I would need. So I signed up and started to rsync some of my data, but then I started to worry a little. Anybody with $40 could sign up for an account with shell access. What would happen if someone gained access to my files? After a simple ls of the home directory, the first site I looked at that was hosted on my server was a blog from an ethical hacker, I decided to jump ship entirely. Not that I think Dreamhost doesn’t secure their servers, I am sure they do, but it just didn’t feel right to leave my somewhat sensitive data there. I would trust it for my photos and videos, but not my documents. So now what.

Option 3: Modified Linksys WRTSL54GS router with openwrt or dd-wrt firmware and external USB drive.
I thought about taking a Linksys WRTSL54GS and putting the much improved firmware options available from openwrt or dd-wrt and slapping a USB drive on top and then putting it at my mom’s house. This would cost around $200 for about 250 GB and a router. This one appeals to the DIY part of me. I would set it all up and make initial data sync locally, then mail it to her to save a lot of time with the initial upload.

Pros:

• No monthly fees
• I assume this setup would consume less energy than a traditional PC, but I have not calculated anything yet.
• All hardware is under my control
• fun to set up

Cons:

• One more piece of hardware to buy and maintain, secure etc.
• Limited to upload/download speed of my mom’s DSL
• consumes her electricity

Option 4: Carbonite.com
Pros:

• Easy install on Windows, “unlimited” for $5/month. Thought I could use with Samba share, I was wrong, shares not supported.

Cons:

• Doesn’t work with Network shares

Summary: Since it doesn’t work with network shares, this is not going to work as a solution for me, but would be a very good solution for the average Windows user with photos, music etc in his My Document folder.

Option 5: Use my extra space on my webhost ~ 10GB
Although this is a tempting option, I don’t want to mix my hosting with my backups. If I accidentally filled up my webhost quota, I could risk downtime. I ruled this option out right away, but I thought I should mention it.
Conclusion

Here are my cost guesstimates for two scenarios….

Assuming 2 years and 20GB of storage

Assuming 2 years and 100GB of storage

So what am I going to use? Good question. First of all, Carbonite will not work for me at all. No way to backup from Linux or even Samba share. The Linksys/USB solution would be kind of cool, but is one more device for me to have to maintain and it will be offsite. Plus, it will be limited to my Mom’s DSL upload speed, if I need to restore. It would be faster, literally, to overnite the drive to me, than to download 20-100GB back.

I am left with Dreamhost or Amazon S3. While Dreamhost seems like a good fit for me, I am very nervous at the thought of letting my data sit on a server that 500 people have shell access to plus not very scalable. So I am left with Amazon S3. I only have a couple of reservations about this. One, privacy of my data, I am trusting that completely to Amazon, for better or worse. Secondly, I am worried I might start to backup much more data which would push the monthly price over my $10 monthly limit.

My next post will be on the success or failure of my Amazon S3 via s3sync option. Stayed tuned…..

Leave a comment if you have other ideas on how best to solve the growing challenge of offsite storage.

While I have been researching this for awhile now, I was inspired to write my findings down after reading a good post on this topic from Jeremy Zawodny and realizing many people are interested in this topic.