Launch of scrible’s new bookmarklet toolbar

May 4, 2011

I’m the cofounder of scrible.com, and we’ve just launched our new bookmarklet toolbar to help you do your online research.  Previously we’d done a limited launch with only an IE plugin, but the bookmarklet works in pretty much all the major web browsers and OS’s (finally – I’m a linux guy, so no more having to jump into the windows VM for this).  Our tool allows you to highlight and take notes on live webpages, and then save them into a gmail like library where you can tag, full text search.  Our service caches the page so you don’t have to worry about link rot in your research.

Check it out and click the feedback button to let us know what you think!

XFS versus JFS…

August 18, 2010

I recently built a new linux dev box and was faced with a choice I’d made several times before.  What file system do I want to use.  A little background on the hardware I’m using – it’s a quad core machine with plenty of ram and for storage I’m using 3 7200 RPM 1TB Hitachi drives (Deskstar 7K1000.C – SATA 3Gb/s) in a software RAID.  I have the boot partition mirrored on all 3 drives, I have a swap partition on each drive as well (all mounted with equal priority) and then the rest of the drive is a RAID 5 LVM partition.  Within the LVM partition, I’ve created the root and /home filesystems.

I’d heard that XFS offered some interesting features compared to JFS as far as taking snapshots of the filesystem state for backups, etc so I decided to give it a try.  I’d seen some benchmarks that showed that XFS was slower on metadata writes, but hadn’t fully appreciated what that might mean in my day to day usage and how drastic the difference could be.

Well here is one benchmark that I found as I was building the system that I couldn’t believe, and has made me since switch back to JFS.

When I was extracting the linux kernel source and the Gentoo stage 3 tar it seemed like it was taking a longer time that usual to extract.  Afterwards, I decided to do a test to compare XFS to JFS (which is what I’ve used on my other machines), so I created another LVM partition, formatted it JFS and did a quick and dirty comparison.

Copying the the linux kernel source tree (with the compiled object files in it – 716MB) on JFS (copying to/from the same partition)

time (cp -a linux-2.6.34-gentoo-r1 test && sync)
real    2m28.284s
user    0m0.377s
sys     0m4.525s

Versus copying it on XFS (again to/from the same partition)

time (cp -a linux-2.6.34-gentoo-r1 test && sync)
real    6m56.095s
user    0m0.372s
sys     0m5.076s

And the deletes show even worse performance

JFS

time (rm -rf test && sync)
real    0m27.679s
user    0m0.043s
sys     0m0.818s

XFS

time (rm -rf test && sync)
real    4m47.683s
user    0m0.054s
sys     0m1.842s

I even tried some XFS mount options that are supposed to optimize it for RAID5 reads and writes (the sunit and swidth) options

Since I’m doing software development on this box, I’m dealing with lots of small source file operations, so I’m guessing that XFS would have been considerably slower for my work load.  I use the IntelliJ Idea IDE for my Java development, and I’ve read blog posts saying that it’s performance is sensitive to disk performance (I’m guessing for all the caches, etc they create, plus scanning for changes in the sources every time you switch in and out of the IDE – I’ve never tried it on an SSD to see how that compares).

When comparing filesystems benchmarks I’d found previously, I’d always assumed that the difference in performance in actual day to day usage wouldn’t be very dramatic between the different systems.  Sure maybe one was 30% faster – not a huge penalty if you’re getting handy features, but 2-3x difference in actual (fairly common) usage floored me.  Your mileage may vary, and it could be that I have something configured incorrectly with the RAID that causes XFS to perform this poorly compared to JFS.

X509Certificate2 Verify method fails on valid certificate…

April 8, 2010

I’m using a the .NET X509Certificate2 Verify method to verify that a file was validly signed with my code signing certificate.  I ran into one machine that this check was failing on (while it worked on all the other machines on the same network) and beat my head against it for 2 hours trying to figure out why.

Turns out that if the date/time on a machine is off by a certain amount (in this case it was about a day) it causes the certificate validation to fail (apparently relates to the revocation check).  Hopefully this will save someone some time in the future.

For reference here is how I’m preforming the validation of the signature on the file (as an example args[0]) – I additionally check that the subject matches my certificate using the cert2.Subject property.

X509Certificate cert = X509Certificate.CreateFromSignedFile(args[0]);
X509Certificate2 cert2 = new X509Certificate2(cert);
Console.Out.WriteLine("Is Signed File Valid? Verify=" + cert2.Verify());

If you need more information about the chain or failures of the check you can use code outlined here on MSDN for the X509Chain.  For this particular error, it didn’t help – just said “The revocation function was unable to check revocation for the certificate.” and “The revocation function was unable to check revocation because the revocation server was offline.”

Supposedly you can configure how strict the checks are (whether it checks for revocations, whether it errors out if certificates are expired, etc) with the X509Chain.ChainPolicy, but I wasn’t having much luck with it.  I couldn’t get it to validate iexplore.exe (IE8 for win7) presumably because the code signing cert used has now expired.

Internet Explorer Plugins and Vista

October 20, 2008

So I’ve recently got a new machine that came pre installed with Windows Vista.  And now I’ve got the fun task of trying to make everything work again with my plugin :)

 

There are a number of places where I’ve found some good info on this, which I will enumerate here for your and my benefit.

http://www.codeproject.com/KB/vista-security/PMSurvivalGuide.aspx – A Developer’s Survival Guide to IE Protected Mode

http://technet.microsoft.com/en-us/magazine/cc138019.aspx – Inside Windows Vista User Account Control

http://msdn.microsoft.com/en-us/library/bb250462.aspx – Understanding and Working in Protected Mode Internet Explorer

http://blogs.msdn.com/ie/archive/2006/02/09/528963.aspx – Protected Mode in Vista IE7

A couple of key things if you’re relying on the compatibility shims: (described in a little more detail in the 4th link)
Registry Accesses (to HKCU software) are moved to
HKEY_CURRENT_USER\Software\Microsoft\Internet Explorer\InternetRegistry\REGISTRY\USER\[User security identifier – SID]\Software\[my key]
File accesses (to acceptable locations) are moved to 
 %userprofile%\AppData\Local\Microsoft\Windows\Temporary Internet Files\Virtualized\{system drive letter}\Users\{username}
I hope to update more this as I dive further into updating my plugin and making it work in both XP and Vista.

Improving AJAX performance of large tables of results…

November 6, 2007

I’ve been writing an AJAX site. I’ve got listings of results (40 per page) that I dynamically pull in from the server and allow the user to index through. Everything worked beautifully in firefox (rendered the new recordset in the page in less than 500ms). IE 6 however CHOKED bigtime – it took around 3-4 seconds to render the first “page” of results. And as each successive page of results was rendered (replacing the previous one) it progressively slowed, to the point that after the 4th page, it was taking 10+ seconds to render. I was just removing and dynamically adding rows to a table through the DOM.

First thing I found was this. That helped some. The next problem I looked into was using the prototype js libraries .each iterator. It’s really handy, but IE 6 performs very badly with it. So I replaced all of my loops over lists that could be long with standard for loops.

The next problem I found was that I was extending all of the elements I was using with the prototype js library to set styles and classes on them, and observing events. (Each entry in the recordset has several onclick events to deal with labels, etc). It turns out that IE majorly chokes on this. One of IE 6’s big problems is garbage collecting events. Prototype has done some work to help it with this. On page unload, they’ll automatically remove all event handlers (otherwise they permanently leak). The problem with my site was that I’m creating a bunch of events, and my page never unloads because I’m dynamically changing the content. So these event handlers just kept piling up. I can’t just call the prototype Event.unloadCache because I have other events on the page I want to keep – I’m just changing the main content in a table.

To work around this I ended up creating my own analogies to several of the prototype event functions. The Event.observe equivalent takes an extra (optional) parameter which specifies the name of an event group. Rather than just keeping a list of all events registered, I categorize them by this event group name so I can call the unloadCache equivalent method with this and ONLY unload the events from a given group. If no event group is specified to the unloadCache method, I remove all events (for instance, on the page unload event).

With these changes I was able to get get IE 6 consistently rendering the content in the page in around 500ms. Haven’t had a chance to note the performance numbers in IE 7 yet.

Also of interest if you’re looking into javascript performance is this simple javascript performance instrumentation helper. Nothing fancy, just a quick way to time various aspects of your app crossbrowser.

IE BHO: Including local images in a remote page…

October 1, 2007

I’ve been writing an IE plugin (or BHO) and ran across a problem when I tested it under IE 7.  When I included images off of the local harddrive, most didn’t show up under IE7 (but worked fine under IE 6).  However, I noticed that one image I was using did work.

The difference was that this image was a CSS style background.  So I was able to work around this problem by simply converting all of my images to be spans/divs with the style.backgroundImage=”url(‘path to local image file‘)”.

Of course the caveat is that this behavior is probably not something they intended, and may fix it at some point.  But until then, it’s an easy change (compared to using the res protocol or a custom protocol – mentioned in the BhoWiki ) to get the job done.  Also, I’ve heard that the IE7 is slightly different under Vista as compared to XP.  I have been testing IE 7 under XP using the Microsoft Virtual PC testing images they have freely available for download (Info about it here and download page actually has XP with IE6 (if you have upgraded to IE7 already) and XP with IE7 (in case you still are running IE6 or Vista).  The article links to an older version of Microsoft Virtual PC, but you can use the latest one here (I’ve noticed it fixed some issues I saw under the old version).

UPDATE: 4/7/2010 Forgot to post this back here till now, but IE8 seems to have closed this hole.  I’ve given up on locally serving them and now host them on my website.  There was something I had read about possibly creating the images as resources in a DLL (like the images in the about:tabs page – something like res://ieframe.dll/tab_icon.png .)   I’ve not tried it, so this approach may not work due to security zone restrictions.

Running Ruby on Rails with mod_fcgid and Apache

June 15, 2007

I had a particularly painful experience trying to get Rails and FCGID to play nicely. It seemed most of what I found was for older versions of rails working with FastCGI.

(UPDATE: See bottom of post – Phusion Passenger makes running Rails under Apache so much easier than configuring mod_fcgid )

So here’s my pass at getting it setup under a Linux environment

First off, I’m using

  • Apache 2.0.52
  • mod_fcgid version 1.10
  • Ruby 1.8.4
  • Rails 1.2.3

Assuming that this is more of a production type setup, put your entire rails project in a directory (assuming you want this to be separate from your development location). I’ll refer to this directory as [rails app]. [rails app] should be the directory that contains the public, app, config, etc directories.

Now, edit your config/database.yml file and make sure that the production database configuration is correct.

I don’t believe that any extra libraries are needed to make rails work with FCGI – I believe they are all built-in now.

Next, in the public directory, you’ll need to edit the .htaccess file. By default rails starts this file with

# General Apache options
AddHandler fastcgi-script .fcgi

AddHandler cgi-script .cgiOptions +FollowSymLinks +ExecCGI

You’ll want to change this part to read

# General Apache options

AddHandler fcgid-script .fcgi

Options +FollowSymLinks +ExecCGI

so that .fcgi is handled by fcgid instead of fastcgi. I removed the .cgi entry just because I wanted to ensure I was getting it working via the mechanism I intended. Also note the 500 error message at the bottom of the .htaccess file – if you’ve gotten Apache configured to get to this directory correctly and something else is going wrong, this is the error message you’ll see. If you’re trying to debug, it can be useful to change this to ensure you’re really working with what you think you are.

Next, and VERY important (no – I didn’t lose 5 hours of my life on this…). You must change the ownership of ALL the files (or at least some critical number of them, I never figured out which magical files mattered) to the user that Apache will try to run them as. This may be apache, or some other user if you have, for instance ,an SuexecUserGroup directive. Ensure that dispatch.fcgi has read/execute permissions Also, the rails apps log and tmp directories must be writable by the user rails is being run as. If the user is incorrect, fcgid won’t run the files, and you’ll get the error message from your .htaccess file, and very unhelpful error_log entries.

One other important thing to note. If any of your Ruby code writes to the console with puts or equivalent, this breaks the rails integration with mod_fcgid. If you wanted to be really sneaky, you could probably do something clever in the dispatch.fcgi file and override the common console output functions to write to a log file or something, but I leave that up to you.

Now on the Apache end:

First make sure mod_fcgid is loaded. In your httpd.conf (or in a conf.d/… config file if you prefer) you’ll need to have the following (with the correct path to your modules directory)

LoadModule fcgid_module /usr/lib/httpd/modules/mod_fcgid.so
<IfModule mod_fcgid.c>
    AddHandler fcgid-script .fcgi .fcg .fpl
</IfModule>
<IfModule mod_fcgid.c>
  SocketPath /var/lib/apache2/fcgid/sock
  DefaultInitEnv  HTTP_RAILS_ENV production
  IdleTimeout 600
  ProcessLifeTime 3600
  MaxProcessCount 8
  DefaultMinClassProcessCount 3
  DefaultMaxClassProcessCount 4
  IPCConnectTimeout 80
  IPCCommTimeout 48
</IfModule>

Note the HTTP_RAILS_ENV setting in there. If you don’t want to run in production, you can change that here. YOU MUST ALSO ADD the following line NEAR THE TOP of your config/environment.rb file – I put it right under the 3 commented lines already in the file.   Apparently based on this article the mod_fcgid has limitations on the environment variables it can set (looks like they have to start with HTTP_??). The following line simply sets the RAILS_ENV to the value of HTTP_RAILS_ENV unless it’s already been set.

# Uncomment below to force Rails into production mode when
# you don't control web/app server and can't set it the proper way
# ENV['RAILS_ENV'] ||= 'production'

ENV['RAILS_ENV'] ||= ENV['HTTP_RAILS_ENV']

This line must be before any of the rest of the rails environment gets loaded or else the environment will initialize in development mode

To actually map in your rails application into Apache, I saw two ways, but I’ve only tried one. I had success with creating a symbolic link in my web servers root directory to the [rails app]/public directory (this of course requires that you have Options FollowSymLinks set). This puts the rails app at http://your site.com/[linkname] The other would be to have inside your VirtualHost directive an Alias directive such as Alias /myRailsApp [rails app]/public .

For either method I believe you’ll need an entry in the VirtualHost directive of your Apache configuration with something like the following

  <Directory [rails app]/public>
    Options Indexes FollowSymLinks ExecCGI
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>

Now, if I haven’t forgotten anything, if you restart Apache, you should be able to access your rails app. Note that this puts your rails application under a subdirectory on the webserver. I haven’t tried to place a rails app as the root of the server, but hopefully this will help you get started.

Note that once you actually get into the rails world, errors, etc will be logged in [rails app]/log. If you didn’t make it into rails, check the apache error_log (note that your sites error log may not be in /var/log/httpd/error_log or the like if there a specific CustomLog directive for the VirtualHost).

Good luck!

Updates

8/16/2007: Discovered a problem with using DefaultInitEnv of RAILS_ENV and the need to use the HTTP_RAILS_ENV setting

1/26/2010: Should have posted this a while back, but somewhere around July/Aug 2009 I heard about Phusion Passenger and it made all of this pain go away.  The installation is very straightforward.  They have a gem which builds the required apache module, etc (see instructions).  I switched my test and production environments at the time over to this and didn’t have any problems (after I removed all the crap I had put in my configs to make rails run under mod_fcgid).  I’ve since switched hosting providers and it also worked well there in FreeBSD.

Creating a cookie within a C# plugin in Internet Explorer

June 5, 2007

I had the need to create a cookie from within my IE plug-in. I create a session cookie when the browser is loaded so that when a user visits my site, I know what version of the plug-in is installed. There appears to be no direct C# API to do this (based on a few google searches). I did find from these searches that there is a method called InternetSetCookie in wininet.dll

For those who don’t regularly have to jump out into the COM world, the code to do this can be a bit of a pain to track down, so here’s the C# import of the method as well as a nice wrapper that will throw an exception if the call is not successful.

        [DllImport("wininet.dll",EntryPoint="InternetSetCookie",ExactSpelling=false,CharSet=CharSet.Unicode,SetLastError=true)]
        static extern bool InternetSetCookie(string url, string cookieName, string cookieData);

        public static void SetCookie(string siteUrl, string cookieName, string cookieData)
        {
            if (!InternetSetCookie(siteUrl, cookieName, cookieData))
            {
                throw new Exception("Exception setting cookie: Win32 Error code="+Marshal.GetLastWin32Error());
            }
        }

Note that siteUrl must be in the form of “http://www.abc.com&#8221; and cannot include a port specification.

For reference the MSDN entry for InternetSetCookie is here.

More details on how to specify the expiration dates of the cookies is here.

And in case you want even more info on cookies in IE, this page describes each of the possible cookie fields in detail.

Intro

June 5, 2007

Occasionally when coding, I run across something I think is particularly spiffy, or spectacularly painful. Especially when I run across the painful things that I haven’t had much luck searching for solutions, I thought it would be a grand idea to create a blog of my own to detail these in for future use by myself and others, as many times when I do find a good solution, it is in this type of forum.

Happy coding!


Follow

Get every new post delivered to your Inbox.