FEED Validator

Congratulations!

This is a valid RSS feed.

Recommendations

This feed is valid, but interoperability with the widest range of feed readers could be improved by implementing the following recommendations.

• line 8, column 116: Self reference doesn't match document location [help]

  ... com/roller/spidaman/feed/entries/rss" />
                                               ^

• line 578, column 0: description should not contain object tag [help]

   <div style="width:425px;text-align:left" id="__ss_1344181 ...
  

• line 622, column 0: description should not contain relative URL references: /roller/spidaman/entry/hub_a_dub_bub_feed [help]

  </description>          </item>
  

• line 853, column 0: style attribute contains potentially dangerous content: bottom (2 occurrences) [help]

  <img src="http://www.arachna.com/images/blowjob_180_day_20081103.png ...
  

• line 1225, column 0: description should not contain script tag [help]

              <description>&lt;SCRIPT TYPE=&quot;text/javascript&quot;&gt;
  

• line 1241, column 0: Invalid HTML: malformed start tag, at line 17, column 123 [help]

  <a href="http://www.arachna.com/images/solar_the_graph.gif" onC ...
  

Source: http://www.arachna.com/roller/rss/spidaman

1. <?xml version="1.0" encoding="utf-8"?>
2. <?xml-stylesheet type="text/xsl" href="http://arachna.com/roller/roller-ui/styles/rss.xsl" media="screen"?><rss version="2.0"
3.  xmlns:dc="http://purl.org/dc/elements/1.1/"
4.  xmlns:atom="http://www.w3.org/2005/Atom" >
5. <channel>
6.  <title>What&apos;s That Noise?! [Ian Kallen&apos;s Weblog]</title>
7.  <link>http://arachna.com/roller/spidaman/</link>
8.      <atom:link rel="self" type="application/rss+xml" href="http://arachna.com/roller/spidaman/feed/entries/rss" />
9.    <description>percolations | ruminations | illustrations | inspirations</description>
10.  <language>en-us</language>
11.  <copyright>Copyright 2011</copyright>
12.  <lastBuildDate>Sat, 30 Apr 2011 09:38:29 -0700</lastBuildDate>
13.  <generator>Apache Roller (incubating) 4.0 (20071120033321:dave)</generator>
14.        <item>
15.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/the_mobile_web_bookmarks_and</guid>
16.    <title>The mobile web, bookmarks and the apps they should be able to play with</title>
17.    <dc:creator>spidaman</dc:creator>
18.    <link>http://arachna.com/roller/spidaman/entry/the_mobile_web_bookmarks_and</link>
19.        <pubDate>Sat, 30 Apr 2011 09:22:48 -0700</pubDate>
20.    <category>Musings</category>
21.            <description>&lt;p&gt;
22. <a href="http://www.macrumors.com/2011/04/29/apples-upcoming-reading-list-for-safari-to-compete-with-instapaper-and-readitlater/" title="MacRumors">
23. <img src="http://www.arachna.com/images/reading_list.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="492" height="283" /></a>
24.  
25. Last year Marco Arment of Instapaper and Tumblr fame posted a plea to Apple with <a href="http://www.marco.org/2010/10/10/an-open-enhancement-request-to-the-mobile-safari-team">an open enhancement request to the Mobile Safari team for sane bookmarklet installation or alternatives</a>. In it he detailed how mobile browsers such as mobile Safari on iPhones and iPads are second class citizens on the web. You can't build plug-ins for them or easily add bookmarklets (javascript you can invoke from the browser's bookmarks). <a href="http://www.instapaper.com/">Instapaper</a> on iOS devices would greatly benefit from easier integration into with the device's browser. I went through the bookmarklet copy/paste exercise that Instapaper walks you through, it's definitely a world of pain. Well, I read this morning that it looks like Apple is answering him, perhaps. Maybe backhanding him.
26. </p>
27. <p>
28. Apple is apparently implementing their own "Reading List" feature that current speculation says is squarely aimed at Instapaper and <a href="http://readitlaterlist.com/">Read It Later</a>. This came out of a <a href="http://www.macrumors.com/2011/04/29/apples-upcoming-reading-list-for-safari-to-compete-with-instapaper-and-readitlater/">MacRumors</a> report on early builds of Lion, the next version of Mac OS X. <a href="http://venturebeat.com/2011/04/29/apple-safari-reading-list/">VentureBeat</a> rightly supposes there will be complimentary features on iOS devices.
29. </p>
30. <p>
31. Like Marco, I'd like to be able to build apps that run natively on devices that can play more smoothly with the device's browser. While mobile SDKs may have support for running an embedded browser (AKA "web views") they're still crippled by having the main browser in it's own silo (and has been previously circulated, web view javascript engines <a href="http://www.tuaw.com/2011/03/18/apple-confirms-some-webkit-optimizations-unavailable-to-ios-apps/">are crippled in their own</a>). Ideally what we're seeing isn't an assault on existing bookmark services but the harbinger of API's that will allow developers to integrate more of the browser's functionality. Certainly some things should continue to be closely sandboxed (e.g. browsing history visibility) or requires full disclosure and permission (e.g. access to the cookie jar). Privacy leaks, phishing attacks and other sorts of miscreant behavior should obviously be guarded against but bookmark additions seem like an easy thing to provide an API and safe user experience around.
32. </p>
33. <p>
34. As <a href="http://www.delicious.com/">Delicious</a> (and before them, <a href="http://www.backflip.com/">Backflip</a> RIP and good riddance) showed, bookmarks are significant social objects. Bookmarks are also a significant personal utility, much like an address book or calendar. I've often considered building an Instapaper or <em>Read It Later</em> type of app, I can't count the number of times I've found something I've wanted to read but didn't have time to read it that instance. Numerous efforts to build interesting things with bookmarks have risen and fallen (ma.gnolia.com) or stalled (sigh... xmarks, simpy). I'm convinced that the emergence of the mobile web and the native app ecosystems that play with them will create an abundance of new opportunities. But developers need to be enabled. Apple, please give us API's, not more silos!
35. </p>
36.  
37. </description>          </item>
38.    <item>
39.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/kusf_a_village_on_the</guid>
40.    <title>KUSF: A village on the airwaves burned down</title>
41.    <dc:creator>spidaman</dc:creator>
42.    <link>http://arachna.com/roller/spidaman/entry/kusf_a_village_on_the</link>
43.        <pubDate>Thu, 20 Jan 2011 10:27:58 -0800</pubDate>
44.    <category>Music</category>
45.            <description>&lt;p&gt;
46. <a href="http://kusf.org/" title="KUSF">
47. <img src="http://www.arachna.com/images/kusf.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="150" height="150" /></a>
48.  
49. As some readers may know, I founded <a href="http://rampageradio.blogspot.com/">Rampage Radio</a> with the guidance and support of  <a href="http://downwithtyranny.blogspot.com/2011/01/systematic-destruction-of-independent.html">Howie Klein</a> back in 1982. I only stuck around for a few years and thereafter left it in Ron Quintana's able hands. But those were years with impact, I look back at them fondly and the show has been running on the air ever since, the last broadcast was in it's usual time slot last Saturday night. As someone who grew up in San Francisco, I always felt that KUSF's presence at 90.3 was a comforting constant. Apparently a deal to sell off KUSF's frequency was consummated last week and <a href="http://www.radiosurvivor.com/2011/01/18/university-of-san-francisco-takes-kusf-off-fm-and-plans-to-sell-license-to-public-radio-group/">the signal was abruptly shutoff Tuesday morning</a>. A <a href="http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2011/01/19/MNSQ1HBP9D.DTL">rally and a dialog took place last night</a> at Presentation Theater with USF President Father Stephen Privett. I commend Father Privett for coming out to face the music, all 500 or so of us in the packed theater were upset by these events and I think it took a degree of courage to show up.  However, after the <a href="http://www.ktvu.com/news/26552893/detail.html">two hour question and answer sessions</a>, it became clear to me that Father Privett has suffered a third degree failure.
50. </p>
51. <p>
52. First, the outcome was poor; the students who he claimed to be acting on behalf of will have reduced volunteer support, the revenue (purported to benefit students) wasn't subject to a competitive bid (it was the first and only deal under discussion); just an NDA-cloaked back-room agreement. Aside from poorly serving the students, his notion of the University as an island, that serving the broader community is detrimental to serving the students, is fundamentally flawed. Serving the community and accepting the efforts of volunteers benefits both the students <em>and</em> the broader community.
53. </p>
54. <p>
55. Second, the process was terrible; instead of backing up and reaching out to the array of interested parties that a deal discussion might commence, he signed the non-disclosure agreement and completely shut out the faculty, students and community. Instead of embracing the stakeholders and providing some transparency, he went straight to the NDA and ambushed them.
56. </p>
57. <p>
58. And the third degree failure was the cowardly absence of recognition of the first two failures.
59. </p>
60. <p>
61. Father Privett claimed full responsibility, explained his rationale for what he did and the process he followed but his rationale for the process was weak. Before going under the cover of NDA, he should have reached out to the students, faculty and volunteers to say: before this goes away, give me some alternatives that will serve you better. Father Privett's gross incompetence was saddening, he should just resign. In the meantime, <a href="http://blogs.law.harvard.edu/doc/2011/01/20/suggestion-for-kusf-head-for-87-7/">using another frequency</a> as a fall back for a rejected FCC petition makes sense but there'll always be this sense of a vacated place in our hearts for 90.3 as San Francisco's cultural oasis.
62. </p>
63. <p>
64. I'm certainly hoping that KUSF can reemerge from the ashes. Please join the effort on Facebook to <a href="http://www.facebook.com/SaveKUSF">Save KUSF</a>!
65. &lt;/p&gt;</description>          </item>
66.    <item>
67.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/managing_granular_access_controls_for</guid>
68.    <title>Managing Granular Access Controls for S3 Buckets with Boto</title>
69.    <dc:creator>spidaman</dc:creator>
70.    <link>http://arachna.com/roller/spidaman/entry/managing_granular_access_controls_for</link>
71.        <pubDate>Sun, 16 Jan 2011 15:09:15 -0800</pubDate>
72.    <category>Musings</category>
73.            <description>&lt;p&gt;
74.  
75. <a href="http://aws.amazon.com/" title="AWS">
76. <img src="http://www.arachna.com/images/aws_logo.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="164" height="60" /></a>
77.  
78. Storing backups is a long-standing <a href="http://aws.amazon.com/s3/">S3</a> use case but, until the release of <a href="http://aws.amazon.com/iam/">IAM</a> the only way to use S3 for backups was to use the same credentials you use for everything else (launching <a href="http://aws.amazon.com/ec2/">EC2</a> instances, deploying artifacts from S3, etc). Now with IAM, we can create users  with individual credentials, create user groups, create access policies and assign policies to users and groups. Here's how we can use <a href="http://boto.cloudhackers.com/">boto</a> to create granular access controls for S3 backups.
79. </p>
80.  
81. <p>
82. So let's create a user and a group within our AWS account to handle backups. Start by pulling and installing the latest boto release from <a href="https://github.com/boto/boto">github</a>. Let's say you have an S3 bucket called "backup-bucket" and you want to have a user whose rights within your AWS infrastructure is confined to putting, getting and deleting backup files from that bucket. This is what you need to do:
83. <ol>
84. <li>Create a connection to the AWS IAM service:
85. <pre>
86. import boto
87. iam = boto.connect_iam()
88. </pre>
89. </li>
90. <li>Create a user that will be responsible for the backup storage. When the credentials are created, the <em>access_key_id</em> and <em>access_key_id</em> components of the response will be necessary for the user to use them, save those values:
91. <pre>
92. backup_user = iam.create_user('backup_user')
93. backup_credentials = iam.create_access_key('backup_user')
94. print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['access_key_id']
95. print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['secret_access_key']
96. </pre>
97. </li>
98. <li>Create a group that will be assigned permissions and put the user in that group:
99. <pre>
100. iam.create_group('backup_group')
101. iam.add_user_to_group('backup_group', 'backup_user')
102. </pre>
103. </li>
104. <li>Define a backup policy and assign it to the group:
105. <pre>
106. backup_policy_json="""{
107.  "Statement":[{
108.      "Action":["s3:DeleteObject",
109.        "s3:GetObject",
110.        "s3:PutObject"
111.      ],
112.      "Effect":"Allow",
113.      "Resource":"arn:aws:s3:::backup-bucket/*"
114.    }
115.  ]
116. }"""
117. created_backup_policy_resp = iam.put_group_policy('backup_group', 'S3BackupPolicy', backup_policy_json)
118. </pre>
119. </li>
120. </ol>
121. Permissions can be applied to individual users but my preference is to put users in a permitted group.
122. </p>
123.  
124. <p>
125. <em>Note: the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> environment variables need to be set or they need to be provided as arguments to the boto.connect_iam() call (which wraps the boto.iam.IAMConnection ctor)</em>.
126. </p>
127. <p>
128. Altogether Now:
129. <pre>
130. import boto
131. iam = boto.connect_iam()
132. backup_user = iam.create_user('backup_user')
133. backup_credentials = iam.create_access_key('backup_user')
134. print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['access_key_id']
135. print backup_credentials['create_access_key_response']['create_access_key_result']['access_key']['secret_access_key']
136. iam.create_group('backup_group')
137. iam.add_user_to_group('backup_group', 'backup_user')
138. backup_policy_json="""{
139.  "Statement":[{
140.      "Action":["s3:DeleteObject",
141.        "s3:GetObject",
142.        "s3:PutObject"
143.      ],
144.      "Effect":"Allow",
145.      "Resource":"arn:aws:s3:::backup-bucket/*"
146.    }
147.  ]
148. }"""
149. created_backup_policy_resp = iam.put_group_policy('backup_group', 'S3BackupPolicy', backup_policy_json)
150. </pre>
151. </p>
152.  
153. <p>
154. While the command line tools for IAM are OK (Eric Hammond wrote a good post about them in <a href="http://alestic.com/2010/09/aws-iam">Improving Security on EC2 With AWS Identity and Access Management (IAM)</a>), I really prefer interacting with AWS programmatically over remembering all of the command line options for the tools that AWS distributes. I thought using boto and IAM was blog-worthy because I had a hard time discerning the correct policy JSON from the AWS docs. There's a <a href="http://docs.amazonwebservices.com/IAM/latest/UserGuide/index.html?PoliciesOverview.html">vocabulary</a> for the "Action" component and a <a href="http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?setpolicyonbucket.html">format</a> for the "Resource" component that wasn't readily apparent but after some digging around and trying some things, I arrived at the above policy incantation. Aside from my production server uses, I think IAM will make using 3rd party services that require AWS credentials to do desktop backups much more attractive; creating multiple AWS accounts and setting up cross-account permissions is a pain but handing over keys for all of your AWS resources to a third party is just untenable.
155. </p>
156. <p>
157. In the meantime, if you're using your master credentials for everything on AWS, stop. Adopt IAM instead! To read more about boto and IAM, check out
158. <ul>
159. <li><a href="http://aws.amazon.com/documentation/iam/">IAM documentation</a></li>
160. <li><a href="http://boto.cloudhackers.com/ref/iam.html">boto.iam. IAMConnection </a></li>
161. <li><a href="http://www.elastician.com/2010/09/using-identity-access-management-iam.html">Using Identity & Access Management (IAM) Service in boto</a></li>
162. </ul>
163. </p>
164.  
165. </description>          </item>
166.    <item>
167.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/scala_regular_expressions</guid>
168.    <title>Scala Regular Expressions</title>
169.    <dc:creator>spidaman</dc:creator>
170.    <link>http://arachna.com/roller/spidaman/entry/scala_regular_expressions</link>
171.        <pubDate>Sun, 29 Aug 2010 18:40:32 -0700</pubDate>
172.    <category>Musings</category>
173.            <description>&lt;p&gt;
174. <a href="http://www.scala-lang.org/" title="Scala">
175. <img src="http://www.arachna.com/images/scala_logo.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="323" height="46" /></a>
176. I've been using <a href="http://www.scala-lang.org/">Scala</a> for the past several weeks, so far it's been a win. My use of Scala to date has been as a "better Java." I'm peeling the onion layers of functional programming and Scala's APIs, gradually improving the code with richer Scala constructs. But in the meantime, I'm mostly employing plain old imperative programming. At this point, I think that's the best way to learn it. Trying to dive into the functional programming deep end can be a bit of a struggle. If you don't know Java, that may not mean much but for myself, I'd used Java extensively in years past and the project that I'm working on has a legacy code base in Java already. One of the Java annoyances that has plagued my work with in the past was the amount code required to work with regular expressions. I go back a long way with regular expressions, Perl (a good friend, long ago) supports it natively and the code I've written in recent years, mostly Python and Ruby, benefitted from the regular expression support in those languages.
177. </p>
178. <p>
179. By annoyance, let's take an example that's simple in Perl since regexps are most succint in the language of the camel (and historically the state of Perl is given in a <a href="http://www.perlfoundation.org/perl6/index.cgi?state_of_the_onion">State of the Onion</a> speech):
180. <pre>
181. #!/usr/bin/env perl
182. # re.pl
183. $input = "camelCaseShouldBeUnderBar";
184. $input=~ s/([a-z])([A-Z])/$1 . "_" . lc($2)/ge;
185. print "$input\n";
186. # outputs: camel_case_should_be_under_bar
187. # now go the other way
188. $input = "under_bar_should_be_camel_case";
189. $input=~ s/([a-z])_([a-z])/$1 . uc($2)/ge;
190. print "$input\n";
191. # outputs underBarShouldCamelCase
192. </pre>
193. Wanna do the same thing in Java? Well, for simple stuff Java's <a href="http://download-llnw.oracle.com/javase/6/docs/api/java/util/regex/Matcher.html">Matcher </a> has a <em>replaceAll</em> method that is, well, dumb as a door knob. If you want the replacement to be based on characters captured from the input and processed in some way, you'd pretty much have to do something like this:
194. <pre>
195. import java.util.regex.Pattern;
196. import java.util.regex.Matcher;
197.  
198. public class Re {
199.    
200.    Pattern underBarPattern;
201.    Pattern camelCasePattern;
202.    
203.    public Re() {
204.      underBarPattern = Pattern.compile("([a-z])_([a-z])");
205.      camelCasePattern = Pattern.compile("([a-z])([A-Z])");
206.    }
207.  
208.    
209.    public String camel(String input) {
210.        StringBuffer result = new StringBuffer();
211.        Matcher m = underBarPattern.matcher(input);
212.        while (m.find()) {
213.            m.appendReplacement(result, m.group(1) +  m.group(2).toUpperCase());
214.        }
215.        m.appendTail(result);
216.        return result.toString();        
217.    }
218.  
219.    public String underBar(String input) {
220.        StringBuffer result = new StringBuffer();
221.        Matcher m = camelCasePattern.matcher(input);
222.        while (m.find()) {
223.            m.appendReplacement(result, m.group(1) + "_" + m.group(2).toLowerCase());
224.        }
225.        m.appendTail(result);
226.        return result.toString();        
227.    }
228.    
229.    public static void main(String[] args) throws Exception {
230.        Re re = new Re();
231.        System.out.println("camelCaseShouldBeUnderBar => " + re.underBar("camelCaseShouldBeUnderBar"));
232.        System.out.println("under_bar_should_be_camel_case => " + re.camel("under_bar_should_be_camel_case"));
233.  
234.    }
235. }
236. </pre>
237. OK, that's way too much code. The reason why this is such a PITA in Java is that the replacement part can't be an anonymous function, or a function at all, due to the fact that... Java doesn't have them. Perhaps that'll change in Java 7. But it's not here today.
238. </p>
239. <p>
240. <a href="http://www.scala-lang.org/node/133">Anonymous functions</a> (in the camel-speak of olde, we might've said "coderef") is one area where Scala is just plain better than Java. <a href="http://www.scala-lang.org/api/current/scala/util/matching/Regex.html">scala.util.matching.Regex</a> has a replaceAllIn method that takes one as it's second argument. Furthermore, you can <em>name</em> the captured matches in the constructor. The anonymous function passed in can do stuff with the Match object passed in. So here's my Scala equivalent:
241. <pre>
242. import scala.util.matching.Regex
243. val re = new Regex("([a-z])([A-Z])", "lc", "uc")
244. var output = re.replaceAllIn("camelCaseShouldBeUnderBar", m =>
245.  m.group("lc") + "_" + m.group("uc").toLowerCase)
246. println(output)
247.  
248. val re = new Regex("([a-z])_([a-z])", "first", "second")
249. output = re.replaceAllIn("under_bar_should_be_camel_case", m =>
250.  m.group("first") +  m.group("second").toUpperCase)
251. println(output)
252. </pre>
253. In both cases, we associate names to the capture groups in the Regex constructor. When the input matches, the resulting Match object makes the match data available to work on. In the first case
254. <pre>
255. m.group("lc") + "_" + m.group("uc").toLowerCase
256. </pre>
257. and in the second
258. <pre>
259. m.group("first") +  m.group("second").toUpperCase)
260. </pre>
261. That's fairly succinct and certainly <em>so</em> much better than Java. By the way, if regular expressions are a mystery to you, get the <a href="http://oreilly.com/catalog/9780596528126/index.html">Mastering Regular Expressions</a> book. In the meantime, keep peeling the onion.
262. </p>
263. <p>
264. <div class="tags">
265. <a href="http://technorati.com/tag/scala" rel="tag">scala</a>  
266. <a href="http://technorati.com/tag/regular+expressions" rel="tag">regular expressions</a>  
267. <a href="http://technorati.com/tag/perl" rel="tag">perl</a>  
268. <a href="http://technorati.com/tag/java" rel="tag">java</a>  
269. </div>
270. &lt;/p&gt;</description>          </item>
271.    <item>
272.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/getting_started_with_a_scala</guid>
273.    <title>Getting Started with a Scala 2.8 Toolchain</title>
274.    <dc:creator>spidaman</dc:creator>
275.    <link>http://arachna.com/roller/spidaman/entry/getting_started_with_a_scala</link>
276.        <pubDate>Sat, 24 Jul 2010 11:51:30 -0700</pubDate>
277.    <category>Musings</category>
278.            <description>&lt;p&gt;
279. <a href="http://www.scala-lang.org/" title="Scala">
280. <img src="http://www.arachna.com/images/scala_logo.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="323" height="46" /></a>
281.  
282. With the release of version 2.8 and enthusiasm amongst my current project's collaborators, I've fired up my latent interest in scala. There's a lot to like about scala: function <em>and</em> object orientation, message based concurrency and JVM garbage collection, JITing, etc; it's a really interesting language. The initial creature comforts I've been looking for are a development, build and test environment I can be productive in. At present, it looks like the best tools for me to get rolling are <a href="http://www.jetbrains.com/idea/">IntelliJ</a>, <a href="http://code.google.com/p/simple-build-tool/">sbt</a> and <a href="http://www.artima.com/scalatest/">ScalaTest</a>. In this post, I'll recount the setup I've arrived at on a MacBook Pro (Snow Leopard), so far so good.
283. </p>
284. <p>
285. <strong>Development</strong>:
286. </p>
287. <p>
288. I've used <a href="http://eclipse.org/">Eclipse</a> for years and there's a lot I like about it but I've also had stability problems, particularly as plugins are added/removed from the installation (stay away from the Aptana ruby plugin, every attempt to use it results in an unstable Eclipse for me). So I got a new version of IntelliJ ("Community Edition"); the Scala plugin for v2.8 doesn't work with the IntelliJ v9.0.2 release so the next task was to grab and install a pre-release of v9.0.3 (see <a href="http://confluence.jetbrains.net/display/IDEADEV/Maia+EAP">ideaIC-95.413.dmg</a>). I downloaded recent build of the scala plugin (see <a href="http://confluence.jetbrains.net/display/SCA/Scala+Plugin+Nightly+Builds">scala-intellij-bin-0.3.1866.zip</a>) and unzipped it in IntelliJ's plugins directory (in my case, "/Applications/IntelliJ IDEA 9.0.3 CE.app/plugins/"). I launched IntelliJ, Scala v2.8 support was enabled.
289. </p>
290. <p>
291. <strong>Build</strong>:
292. </p>
293. <p>
294. I've had love/hate relationships in the past with <a href="http://ant.apache.org/">ant</a> and <a href="http://maven.apache.org/">maven</a>; there's a lot to love for flexibility with the former and consistency with the latter (hat tip at "convention over configuration"). There's a lot to hate with the tedious XML maintenance that comes with both. So this go around, I'm kicking the tires on <a href="http://code.google.com/p/simple-build-tool/">simple build tool</a>. One of the really nice bits about sbt is that it has a shell you can run build commands in; instead of waiting for the (slow) JVM launch time everytime you build/test you can incur the launch penalty once to launch the shell and re-run those tasks from within the sbt shell. Once sbt is <a href="http://code.google.com/p/simple-build-tool/wiki/Setup">setup</a>, a project can be started from the console like this:
295.  
296.  $ <code>sbt update</code><br/>
297. You'll go into a dialog like this:
298. <pre>
299. Project does not exist, create new project? (y/N/s) y
300. Name: scratch
301. Organization: ohai
302. Version [1.0]: 0.1
303. Scala version [2.7.7]: 2.8.0
304. sbt version [0.7.4]:
305. </pre>
306.  
307. Now I want sbt to not only bootstrap the build system but also bootstrap the IntelliJ project. There's an <a href="http://github.com/mpeltonen/sbt-idea-plugin">sbt plugin</a> for that. Ironically, bringing up the develop project requires creating folder structures and code artifacts... which is what your development environment should be helping you with. While we're in there, we'll declare our dependency on the test framework we want (why ScalaTest isn't in the scala stdlib is mysterious to me; c'mon scala, it's 2010, python and ruby have both shipped with test support for years). So, fallback to console commands and vi (I don't do emacs) or lean on Textmate. <br/>
308.  
309.  $ <code>mkdir project/build</code><br/>
310.  $ <code>vi project/build/Project.scala</code><br/>
311. <pre>
312. import sbt._
313. class Project(info: ProjectInfo) extends DefaultProject(info) with IdeaProject {
314.    // v1.2 is the current version compatible with scala 2.8
315.    // see http://www.scalatest.org/download
316.    val scalatest = "org.scalatest" % "scalatest" % "1.2" % "test->default"
317. }
318. </pre>
319.  $ <code>mkdir -p project/plugins/src</code><br/>
320.  $ <code>vi project/plugins/src/Plugins.scala</code><br/>
321. <pre>
322. import sbt._
323. class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
324.  val repo = "GH-pages repo" at "http://mpeltonen.github.com/maven/"
325.  val idea = "com.github.mpeltonen" % "sbt-idea-plugin" % "0.1-SNAPSHOT"
326. }
327. </pre>
328.  $ <code>sbt update</code><br/>
329.  $ <code>sbt idea</code><br/>
330.  
331. Back in IntelliJ, do "File" > "New Module" > select "Import existing module" and specify the path to the "scratch.iml" file that that last console command produced.
332. </p>
333. <p>
334. Note how we declared our dependency on the test library and the repository to get it from with <em>two lines</em> of code, not the several lines of XML that would be used for each in maven.
335. </p>
336. <p>
337. <strong>Test</strong>:
338. </p>
339. <p>
340. Back in IntelliJ, right click on src/main/scala and select "New" > "Package"
341. and specify "ohai.scratch". Right click on that package and select "New" > "Scala Class", we'll create a class ohai.scratch.Bicycle - in the editor put something like
342. <pre>
343. package ohai.scratch
344. class Bicycle {
345.  var turns = 0
346.  def pedal { turns +=1 }
347. }
348. </pre>
349. Now to test our bicycle, do similar package and class creation steps in the src/test/scala folder to create ohai.scratch.test.BicycleTest:
350. <pre>
351. package ohai.scratch.test
353. import org.scalatest.Spec
354. import ohai.scratch.Bicycle
356. class BicycleSpec extends Spec {
358.  describe("A bicycle, prior to any pedaling -") {
359.    val bike = new Bicycle()
360.  
361.    it("pedal should not have turned") {
362.      expect(0) { bike.turns }
363.    }
364.  }
365.  
366.  describe("A bicycle") {
367.    val bike = new Bicycle()
368.    
369.    bike.pedal
370.  
371.    it("after pedaling once, it should have one turn") {
372.      expect(1) { bike.turns }
373.    }
374.  }
375. }
376. </pre>
377.  
378. Back at your console, go into the sbt shell by typing "sbt" again. In the sbt shell, type "test". In the editor, make your bicycle do more things, write tests for those things (or reverse the order, TDD style) and type "test" again in the sbt shell. Lather, rinse, repeat, have fun.
379. </p>
380. <p>
381. <div class="tags">
382. <a href="http://technorati.com/tag/programming" rel="tag">programming</a>  
383. <a href="http://technorati.com/tag/scala" rel="tag">scala</a>  
384. <a href="http://technorati.com/tag/intellij" rel="tag">intellij</a>  
385. <a href="http://technorati.com/tag/sbt" rel="tag">sbt</a>  
386. <a href="http://technorati.com/tag/scalatest" rel="tag">scalatest</a>  
387. </div>
388. &lt;/p&gt;</description>          </item>
389.    <item>
390.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/programmatic_elastic_mapreduce_with_boto</guid>
391.    <title>Programmatic Elastic MapReduce with boto</title>
392.    <dc:creator>spidaman</dc:creator>
393.    <link>http://arachna.com/roller/spidaman/entry/programmatic_elastic_mapreduce_with_boto</link>
394.        <pubDate>Fri, 19 Mar 2010 11:48:51 -0700</pubDate>
395.    <category>LAMP</category>
396.            <description>&lt;p&gt;
397. <a href="http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html" title="Elastic MapReduce">
398. <img src="http://www.arachna.com/images/hadoop_processing.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="284" height="173" /></a>
399. I'm working on some cloud-homed data analysis infrastructure. I may focus in the future on using the <a href="http://www.cloudera.com/hadoop/">Cloudera distribution</a> on <a href="http://aws.amazon.com/ec2/">EC2</a> but for now, I've been experimenting with <a href="http://aws.amazon.com/elasticmapreduce/">Elastic MapReduce</a> (EMR). I think the main advantages of using EMR are:
400. <ul>
401. <li>Configuring the namenode, tasktracker and jobtracker is tedious, EMR relieves you of those duties</li>
402. <li>Instance pool setup/teardown is tightly integrated</li>
403. <li>Automated pool member replacement if an instance goes down</li>
404. <li>Built in verbs like the "aggregate" reducer</li>
405. <li>Programmatic and GUI operation</li>
406. </ul>
407. On the downside, Amazon is using an old version of Hadoop (0.18.3, not 0.20.x such as Cloudera's package) but I think until I've got clusters running 24/7 and can justify the administrative tool investment, using EMR is just fine.
408. </p>
409. <p>
410. While there's a slick <a href="http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2264&categoryID=262">EMR client tool implemented in ruby</a>, I've got a workflow of data coming in/out of <a href="http://aws.amazon.com/s3/">S3</a>, I'm otherwise working in <a href="http://www.python.org/">Python</a> (using an old friend <a href="http://code.google.com/p/boto/">boto</a>) and so I'd prefer to keep my toolchain in that orbit. The last release of boto (v1.9b) doesn't support EMR but lo-and-behold it's in HEAD in the source tree. So if you check it out the Google Code svn repo as well as set your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, you can programmatically run the <a href="http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2273">EMR wordcount sample</a> like this:
411. <pre>
412. from time import sleep
413. from datetime import datetime
414. from boto.emr.step import StreamingStep
415. from boto.emr.connection import EmrConnection
416.  
417. job_ts = datetime.now().strftime("%Y%m%d%H%M%S")
418. emr = EmrConnection()
419. wc_step = StreamingStep('wc text', \
420.  's3://elasticmapreduce/samples/wordcount/wordSplitter.py', \
421.  'aggregate', input='s3://elasticmapreduce/samples/wordcount/input', \
422.  output='s3://wc-test-bucket/output/%s' % job_ts)
423. jf_id = emr.run_jobflow('wc jobflow', 's3n://emr-debug/%s' % job_ts, \
424.  steps=[wc_step])
425.  
426. while True:
427.  jf = emr.describe_jobflow(jf_id)
428.  print "[%s] %s" % (datetime.now().strftime("%Y-%m-%d %T"), jf.state)
429.  if jf.state == 'COMPLETED':
430.    break
431.  sleep(10)
432. </pre>
433. Have fun hadooping!
434. </p>
435. <p><em>
436. <a href="http://technorati.com/tag/hadoop" rel="tag">hadoop</a>  
437. <a href="http://technorati.com/tag/boto" rel="tag">boto</a>  
438. <a href="http://technorati.com/tag/python" rel="tag">python</a>  
439. <a href="http://technorati.com/tag/aws" rel="tag">aws</a>  
440. <a href="http://technorati.com/tag/elasticmapreduce" rel="tag">elasticmapreduce</a>  
441. <a href="http://technorati.com/tag/s3" rel="tag">s3</a>  
442. <a href="http://technorati.com/tag/ec2" rel="tag">ec2</a>  
443.  
444. </em></p>
445.  
446. </description>          </item>
447.    <item>
448.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/odysseys</guid>
449.    <title>2010: An Odyssey Continues</title>
450.    <dc:creator>spidaman</dc:creator>
451.    <link>http://arachna.com/roller/spidaman/entry/odysseys</link>
452.        <pubDate>Sat, 23 Jan 2010 22:02:26 -0800</pubDate>
453.    <category>Music</category>
454.            <description>&lt;p&gt;
455. <a href="http://www.odysseyofthemind.com/" title="Odyssey of the Mind">
456. <img src="http://www.arachna.com/images/omer.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="225" height="198" /></a>
457. Hello, 2010. 2009 was a strange year, indeed. I quit my job after <a href="http://technorati.com/">5 years</a> last spring, since then I've been working as a software/infrastructure consultant and playing with a lot of new technologies (of course also leveraging expertise I have with some more established ones). I've also done a lot of thinking. Thinking about <a href="http://www.nicholasgcarr.com/bigswitch/">cloud computing and compute marketplaces</a>, about energy, mobile and transportation technologies, about life and the meaning of the whole thing. Consulting has had its ups and downs. Looking back at 2009 though, I can safely say I'm glad it's over. Not that it's all bad news, I have my share of things to be pleased about too. My daughter's Bat Mitzvah was beautiful. I've been spending more time with the kids, switched to decaf (I really enjoy the taste of coffee but excess caffeine isn't healthy) and was invited by my old friends in <a href="http://www.arachna.com/roller/spidaman/entry/going_to_metallica_s_rock">Metallica</a> to some of their seminal events.
458. </p>
459. <p>
460. Nonetheless, 2010 will no doubt be an upgrade over 2009.
461. <p>
462. One of the things I'm spending more domestic time around is <a href="http://www.odysseyofthemind.com/">Odyssey of the Mind</a>, a creative problem solving competition that my son is participating in. My daughter did it previously and went to the world finals with her team twice, my son went last year, and (warning: proud papa bragging alert) placed 4th! This year the coaching torch is passed to me and I'm working with the team facilitating their solution for the long term problem which will be presented at the regional competition next month. We're hoping for a set of repeat victories that will send us to the world finals again this year. I'm new to coaching OotM but my co-coach has coached before and I've helped the teams in supporting roles in years past, so the Odyssey regime is not completely new to us.
463. </p>
464. <p>
465. This week will present some interesting challenges. My co-coach is physician and will be flying into Haiti to join the relief effort; I'm really happy for her to have this opportunity to be of service. I'll do my best to keep the OotM team moving forward until she returns.
466. </p>
467. <p>
468. Work will pick up too. Consulting has given me an opportunity to learn Ruby, Rails and a lot of stuff in that technology orbit. I've also been putting <a href="http://aws.amazon.com/">Amazon Web Services</a> to heavy use and playing around with Twitter's APIs. I've been looking for opportunities to scratch some Hadoop itches and lately my interests have turned towards programming in <a href="http://www.scala-lang.org/">Scala</a>, I expect some consulting gigs to shake loose to sate those interests. If the opportunity is right, my entrepreneurial impulses will get  the best of me and I'll stop (or scale back) consulting to jump a on new start-up. My desire to set the world on fire will never be satisfied.
469. </p>
470. <p>
471. Yea, 2010 is gonna be good. Check it out, the <a href="http://sfgiants.com/">Giants</a> have made some changes that look like a credible offense. Already, it's gotta be better than 2009.
472. </p>
473. </description>          </item>
474.    <item>
475.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/scaling_rails_with_mysql_table</guid>
476.    <title>Scaling Rails with MySQL table partitioning</title>
477.    <dc:creator>spidaman</dc:creator>
478.    <link>http://arachna.com/roller/spidaman/entry/scaling_rails_with_mysql_table</link>
479.        <pubDate>Thu, 3 Dec 2009 00:08:37 -0800</pubDate>
480.    <category>LAMP</category>
481.            <description>&lt;p&gt;
482. <a href="http://www.mysql.com/" title="MySQL">
483. <img src="http://www.arachna.com/images/mysql_logo.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="114" height="68" /></a>
484. Often the first step in scaling MySQL back-ended web applications is integrating a caching layer to reduce the frequency of expensive database queries. Another common pattern is to denormalize the data and index to optimize read response times. Denormalized data sets solve the problem to a point, when the indexes exceed reasonable RAM capacities (which is easy to do with high data volumes) these solutions degrade. A general pattern for real time web applications is the emphasis on <em>recent</em> data, so keeping the hot recent data separate from the colder, larger data set is fairly common. However, purging denormalized data records that have aged beyond usefulness can be expensive. Databases will typically lock records up, slow or block queries and fragment the on-disk data images. To make matters worse, MySQL will block queries while the data is de-fragmented.
485. </p>
486. <p>
487. MySQL 5.1 introduced table partitioning as a technique to cleanly prune data sets. Instead of purging old data and the service interruptions that a DELETE operation entails, you can break up the data into partitions and drop the old partitions as their usefulness expires. Furthermore, your query response times can benefit from knowing about how the partitions are organized; by qualifying your queries correctly you can limit which partitions get accessed.
488. </p>
489. <p>
490. But MySQL's partition has some limitations that may be of concern:
491. <ul>
492. <li>The partitioning criteria must be an integer value. It's easy to express <a href="http://dev.mysql.com/doc/refman/5.1/en/partitioning-management.html">date information as integer values for the purposes of partitioning</a> and in fact you can (albeit intrusively) <a href="http://datacharmer.blogspot.com/2009/09/partitioning-with-non-integer-values.html">partition with non integer values using triggers</a>. But the underlying partitioning that MySQL implemented mandates integer criteria.</li>
493. <li>All columns used in the partitioning expression for a partitioned table must be part of every unique key that the table may have. This will impact your design as to your use of <a href="http://dev.mysql.com/doc/refman/5.1/en/partitioning-limitations-partitioning-keys-unique-keys.html">primary keys and unique keys</a>.</li>
494. </ul>
495. So you want to have an auto_increment primary key but partition on the date? Too bad. MySQL's doc on the matter closes woefully, "We are working to remove this limitation in a future MySQL release series." I don't think PostgreSQL's table partitioning has this limitation but I haven't verified that.
496. </p>
497. <p>
498. When we first looked at MySQL's table partitioning, this seemed like a deal breaker. The platform we've been working with was already on MySQL 5.1 but it was also a <a href="http://rubyonrails.org/">rails</a> app. Rails has its own ORM semantics that uses integer primary keys. What we wanted were partitions like this: <br />
499. <img src="http://www.arachna.com/images/daily_partitions_1.png" border="0" /><br />
500. Here, data is segmented into daily buckets and as time advances, fresh buckets are created while old ones are discarded. Yes, with some social web applications, a week is the boundary of usefulness that's why there are only 8 buckets in this illustration. New data would be written to the "Now" partition, yesterday's data would be in yesterday's bucket and last Monday's data would be in the "Monday" bucket. I initially thought that the constraints on MySQL's partitioning criteria might get in the way; I really wanted the data partitioned by time but needed to partition by the primary keys (sequential integers) that rails used like this: <br />
501. <img src="http://www.arachna.com/images/pk_partitions_1.png" border="0" /><br />
502. In this case, data is segmented by a ceiling on the values allowed in each bucket. The data with ID values less than 100 go in the first bucket, less than 200 in the second and so on.
503. </p>
504. <p>
505. Consider an example, suppose you had a table like this:
506. <pre>
507. CREATE TABLE tweets (
508.  id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
509.  status_id bigint(20) unsigned NOT NULL,
510.  user_id int(11) NOT NULL,
511.  status varchar(140) NOT NULL,
512.  source varchar(24) NOT NULL,
513.  in_reply_to_status_id bigint(20) unsigned DEFAULT NULL,
514.  in_reply_to_user_id int(11) DEFAULT NULL,
515.  created_at datetime NOT NULL,
516.  PRIMARY KEY (id),
517.  KEY created_at (created_at),
518.  KEY user_id (user_id)
519. );
520. </pre>
521. You might want to partition on the "created_at" time but that would require having the "created_at" field in the primary key, which doesn't make any sense. I worked around this by leveraging the common characteristic of rails' "created_at" timestamp and integer primary key "id" fields: they're both monotonically ascending. What I needed was to map ID ranges to time frames. Suppose we know, per the illustration above, that we're only going to have 100 tweets a day (yes, an inconsequential data volume that wouldn't require partitioning, but for illustrative purposes), we'd change my table like this:
522. <pre>
523. ALTER TABLE tweets PARTITION BY RANGE (id) (
524.  PARTITION tweets_0 VALUES LESS THAN (100),
525.  PARTITION tweets_1 VALUES LESS THAN (200),
526.  PARTITION tweets_2 VALUES LESS THAN (300),
527.  PARTITION tweets_3 VALUES LESS THAN (400),
528.  PARTITION tweets_4 VALUES LESS THAN (500),
529.  PARTITION tweets_5 VALUES LESS THAN (600),
530.  PARTITION tweets_6 VALUES LESS THAN (700),
531.  PARTITION tweets_maxvalue VALUES LESS THAN MAXVALUE
532. );
533. </pre>
534. When the day advances, it's time create a new partition and drop the old one, so we do this:
535. <pre>
536. ALTER TABLE tweets REORGANIZE PARTITION tweets_maxvalue into (
537.  PARTITION tweets_7 VALUES LESS THAN (800),
538.  PARTITION tweets_maxvalue VALUES LESS THAN MAXVALUE
539. );
540. ALTER TABLE tweets DROP PARTITION tweets_0;
541. </pre>
542. Programmatically determining what the new partition name should be (tweets_7) requires introspecting on the table itself. Fortunately, MySQL has metadata about tables and partitions in its "information_schema" database. We can query it like this
543. <pre>
544. SELECT partition_ordinal_position AS seq, partition_name AS part_name, partition_description AS max_id
545. FROM information_schema.partitions
546. WHERE table_name='tweets' and table_schema='twitterdb' ORDER BY seq
547. </pre>
548. </p>
549. <p>
550. So that's the essence of the table partition life-cycle. One issue here is that when we reorganize the partition, it completely rewrites it; it doesn't atomically rename "tweets_maxvalue" to "tweets_7" and allocate a new empty "tweets_maxvalue" (that would be nice). If the partition is populated with data, the rewrite is resource consumptive; the more data, the more so. The availability of a table that has a moderate throughput of data (let's say 1 million records daily, not 100) would suffer, locking the table for a long duration waiting for the rewrite is unacceptable for time sensitive applications. Given that issue, what we really want is to anticipate the ranges and reorganize the MAXVALUE partition in advance. When we allocate the new partition, we set an upper boundary on the ID values for but we need to make a prediction as to what the upper boundary will be. Fortunately, we put an index on created_at so we can introspect on the data to determine the recent ID consumption rate like this:
551. <pre>
552. SELECT UNIX_TIMESTAMP(created_at),MAX(id) FROM tweets WHERE DATE(created_at)=DATE( now() - interval 1 day );
553. SELECT UNIX_TIMESTAMP(created_at),MAX(id) FROM tweets WHERE DATE(created_at)=DATE( now() - interval 2 day );
554. SELECT UNIX_TIMESTAMP(created_at),MAX(id) FROM tweets WHERE DATE(created_at)=DATE( now() - interval 3 day );
555. SELECT UNIX_TIMESTAMP(created_at),MAX(id) FROM tweets WHERE DATE(created_at)=DATE( now() - interval 4 day );
556. </pre>
557. The timestamps (epoch seconds) and ID deltas give us what we need to determine the ID consumption rate N; the new range to allocate is simply <em>max(id) + N</em>.
558. </p>
559. <p>
560. OK, so where's the rails? All of this so far has been transparent to rails. To take advantage of the partition access optimization opportunities mentioned above, it's useful to have application level knowledge of the time frame to ID range mappings. Let's say we just want recent tweets from a user M and we know that the partition boundary from a day ago is N, we can say
561. <pre>
562. Tweet.find(:all, :conditions => ["user_id = M AND id > N"])
563. </pre>
564. This will perform better than the conventional (even if we created an index on user_id and created_at).
565. <pre>
566. Tweet.find(:all, :conditions => ["user_id = M AND created_at > now() - interval 1 day"])
567. </pre>
568. Scoping the query to a partition in this way isn't unique to rails (or even to MySQL, with Oracle we can explicitly name the partitions we want a query to hit in SQL). Putting an id range qualifier in the WHERE clause of our SQL statements will work with any application environment. I chose to focus on rails here because of the integer primary keys that rails requires and the challenge that poses. To really integrate this into your rails app, you might create a model that has the partition metadata.
569. </p>
570. <p>
571. I hope this is helpful to others who are solving real time data challenges with MySQL (with or without rails), I didn't turn up much about how folks manage table partitions when I searched for it. There's an interesting article about using the <a href="http://dev.mysql.com/tech-resources/articles/partitioning-event_scheduler.html">MySQL scheduler and stored procedures to manage partitions</a> but I found the complexity of developing, testing and deploying code inside MySQL more of a burden than I wanted to carry, so I opted to do it all in ruby and integrate it with the rails app. If readers have any better techniques for managing MySQL table partitions, please post about it!
572. </p>
573. <p>
574. Table partitioning handles a particular type of data management problem but it won't answer all of our high volume write capacity challenges. Scaling write capacity requires distributing the writes across independent indexes, sharding is the common technique for that. I'm currently investigating HBase, which transparently distributes writes to Hadoop data nodes, possibly in conjunction with an external index (solr or lucene), as an alternative to sharding MySQL. Hadoop is sufficiently scale free for very large workloads but real time data systems has not been its forte. Perhaps that will be a follow up post.
575. </p>
576. <p>
577. Further Reading:<br />
578. <div style="width:425px;text-align:left" id="__ss_1344181"><a style="font:14px Helvetica,Arial,Sans-serif;display:block;margin:12px 0 3px 0;text-decoration:underline;" href="http://www.slideshare.net/datacharmer/mysql-partitions-tutorial" title="MySQL partitions tutorial">MySQL partitions tutorial</a><object style="margin:0px" width="425" height="355"><param name="movie" value="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=partitionstutorial-090426130303-phpapp02&stripped_title=mysql-partitions-tutorial" /><param name="allowFullScreen" value="true"/><param name="allowScriptAccess" value="always"/><embed src="http://static.slidesharecdn.com/swf/ssplayer2.swf?doc=partitionstutorial-090426130303-phpapp02&stripped_title=mysql-partitions-tutorial" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="425" height="355"></embed></object><div style="font-size:11px;font-family:tahoma,arial;height:26px;padding-top:2px;">View more <a style="text-decoration:underline;" href="http://www.slideshare.net/">documents</a> from <a style="text-decoration:underline;" href="http://www.slideshare.net/datacharmer">Giuseppe Maxia</a>.</div></div>
579. <ul>
580. <li><a href="http://dev.mysql.com/doc/refman/5.1/en/partitioning-management-range-list.html">Management of RANGE and LIST  Partitions</a></li>
581. <li><a href="http://www.mysqlperformanceblog.com/2008/03/14/sharding-and-time-base-partitioning/">Sharding and Time Base Partitioning</a></li>
582. </ul>
583. </p>
584. <p><em>
585. <a href="http://technorati.com/tag/mysql" rel="tag">mysql</a>  
586. <a href="http://technorati.com/tag/scalability" rel="tag">scalability</a>  
587. </em></p>
588. </description>          </item>
589.    <item>
590.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/hub_a_dub_bub_feed</guid>
591.    <title>Hub-a-dub-bub, feed clouds in a tub</title>
592.    <dc:creator>spidaman</dc:creator>
593.    <link>http://arachna.com/roller/spidaman/entry/hub_a_dub_bub_feed</link>
594.        <pubDate>Thu, 10 Sep 2009 12:35:37 -0700</pubDate>
595.    <category>Musings</category>
596.            <description>&lt;p&gt;
597.  
598. <a href="/roller/spidaman/entry/hub_a_dub_bub_feed" title="Pubsubhubbub and rssCloud">
599. <img src="http://www.arachna.com/images/push_rsscloud.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="171" height="52" /></a>
600. While at Technorati, I observed a distinct shift around summer of 2005 in the flow of pings from largely worthwhile pings to increasingly worthless ones. A ping is a simple message with a URL. Invented by Dave Winer, it was originally implemented with XML-RPC but RESTful variants also emerged. The intention of the message is "this URL updated." For a blog, the URL is the main page of the blog; it's not the feed URL, a post URL ("permalink") or any other resource on the site - it's just the main page. Immediately, that narrow scope has a problem; as a service that should <em>do something</em> with the ping, the notification has no more information about <em>what</em> changed on the site. A new post? Multiple new posts? Um, what are the new post URL(s)? Did the blogroll change? And so on. Content fetching and analysis is cheap; network latencies, parsing content and doing interesting stuff with it is easy at low volumes but is difficult to scale.
601. </p>
602. <p>
603. In essence, a ping is a cheap message to prompt an expensive analysis to detect change. It's an imbalanced market of costs and benefits. Furthermore, even if the ping had a richer payload of information, pings lack another very important component: authenticity. These cheap messages can be produced by anybody for any URL and the net result I observed was that lots of people produced pings for lots of things, most of which weren't representative of real blogs (other types of sites or just spam) or real changes on blogs, just worthless events that were resource intensive to operate on. When I left Technorati in March (2009), we were getting around 10 million pings a day, roughly 95% of them of no value. Pings are the SMTP of the blogosphere; weak identity systems, spammy and difficult to manage at scale.
604. </p>
605. <p>
606. Besides passively waiting for pings, the other method to find things that have changed is to poll sites. But <a href="http://code.google.com/p/pubsubhubbub/wiki/WhyPollingSucks">polling is terribly inefficient</a>. To address those inefficiencies, FriendFeed implemented <a href="http://code.google.com/p/simpleupdateprotocol/">Simple Update Protocol</a>. SUP is reminiscent of another Dave Winer invention, <a href="http://oldweblogscomblog.scripting.com/changesXml">changes.xml</a> however SUP accounts for discovery and offers a more compact format.
607. </p>
608. <p>
609. But SUP wasn't the first effort to address the aforementioned deficiencies, 2005 saw a lot of activity around something called "feedmesh." Unfortunately, the activity degenerated into a lot of babble; noise I attribute to unclear objectives and ego battles but I'm sure others have their own interpretation. The feedmesh discussion petered out and little value beyond ping-o-matic and other ping relayers emerged from it. Shortly thereafter SixApart created their <a href="http://www.sixapart.com/labs/update/">atom stream</a>, I think spearheaded by Brad Fitzpatrick. The atom stream is essentially a long lived HTTP connection that streams atom elements to the client. The content flow was limited to SixApart's hosted publishing platforms (TypePad, LiveJournal and Vox) and the reliability wasn't that great, at least in the early days, but it was a big step in the right direction for the blog ecosystem. The atom stream was by far the most efficient way to propogate content from the publishing platform to interested parties such as Technorati operating search, aggregation and analytic systems. It eliminates the heavyweight chatter of cheap ping messages and the heavyweight process that follows: fetch page, discover feed, fetch feed, inspect contents and do stuff with it.
610. </p>
611. <p>
612. So here we are in 2009 and it feels like deja-vu all over again. This time Dave is promoting <a href="http://rsscloud.org/walkthrough.html">rssCloud</a>. rssCloud does a waltz of change notification; the publisher notifies a hub, the hub broadcasts notifications to event subscribers and the subscriber does the same old content fetch and analysis cycle above. rssCloud seems fundamentally flawed in that it is dependent on IP addresses as identifiers. Notification receivers who must use new a IP address must re-subscribe. I'm not sure how an aggregator should distribute notificaton handling load across a pool of IP addresses. The assumption is that notification receivers will have stable, singular IP addresses; rssCloud appears scale limited and a support burden.  The focus on specification simplicity has its merits, I think we all hate gratuitous complexity (observe the success of JSON over SOAP). However, Dave doesn't regard system operations as an important concern; he'll readily evangelize a message format and protocol but the real world operability is Other Peoples Problem.
613. </p>
614. <p>
615. <a href="http://code.google.com/p/pubsubhubbub/">Pubsubhubub</a> (hey, Brad again) has a similar waltz but eliminates the content fetch and analysis cycle ergo it's fundamentally more efficient than rssCloud's waltz. Roughly, Pubsubhubub is to the SixApart atom stream what rssCloud is to old school XML-RPC ping. If I were still at Technorati (or working on event clients like Seesmic, Tweetdeck, etc), I would be implementing Pubsubhubbub and taking a pass on rssCloud.  With both systems, I'd be concerned with a few issues. How can the authenticity of the event be trusted? Yes, we all like simplicity but looking at SMTP is apt; now mail systems must be aware of SPF, DKIM, SIDF, blahty blah blah. Its common for mail clients to access MTA's over cryptographically secure connections. Mail is a mess but it works, albeit with a bunch of junk overlaid. I guess this is why <a href="http://www.waveprotocol.org/">Wave Protocol</a> has gathered interest. Anyway, Pubsubhubbub has a handshake during the subscription processes, though that looks like a malicious party could still spin up endpoints with bogus POSTs. I'd like to see an OAuth or digest authentication layer in the ecosystem. Yea, it's a little more complicated, but nothing onerous... suck it up. Whatever. Brad knows the authenticity rap, I mean he also invented OpenID (BTW, we adopted <a href="http://technorati.com/weblog/2006/10/144.html">OpenID at Technorati back in 2007</a>). At Technorati we had to implement ping throttling to combat extraneous ping bots; either daemons or cron jobs that just ping continuously, whether the URL has changed or not. You can't just blacklist that URL, we don't know it was the author generating those pings, there's no identity authenticity there. We resorted to blocking IP addresses but that scales poorly and creates other support problems. We had 1000's of domain names blocked and whole class C networks blocked but it was always a wackamole exercise; so much for an open blogosphere, a whitelist is the only thing that scales within those constraints. Meanwhile, back to rssCloud and Pubsubhubbub, what event delivery guarantees can be made? If a data center issue or something else interrupts event flow, are events spooled so that event consumption can be caught up? How can subscribers track the authenticity of the event originators? How can publishers keep track of who their subscribers are? Well, I'm not at Technorati anymore so I'm no longer losing any sleep over these kinds of concerns but do I care about the ecosystem. For more, <a href="http://grack.com/blog/">Matt Mastracci</a> is posting interesting stuff about Pubsubhubbub and a sound comparison was posted to <a href="http://www.techcrunch.com/2009/09/09/rsscloud-vs-pubsubhubbub-why-the-fat-pings-win/" title="RSSCloud Vs. PubSubHubbub: Why The Fat Pings Win">Techcrunch</a> yesterday.
616. </p>
617. <p>
618. <a href="http://technorati.com/tag/rsscloud" rel="tag">rsscloud</a>  
619. <a href="http://technorati.com/tag/pubsubhubbub" rel="tag">pubsubhubbub</a>  
620. <a href="http://technorati.com/tag/ping" rel="tag">ping</a>  
621. </p>
622. </description>          </item>
623.    <item>
624.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/business_models_in_consulting_contracting</guid>
625.    <title>Business Models in Consulting, Contracting and Training</title>
626.    <dc:creator>spidaman</dc:creator>
627.    <link>http://arachna.com/roller/spidaman/entry/business_models_in_consulting_contracting</link>
628.        <pubDate>Sun, 2 Aug 2009 20:57:08 -0700</pubDate>
629.    <category>Musings</category>
630.            <description>&lt;p&gt;
631. Since leaving <a href="http://technorati.com/">Technorati</a> last spring, I've been working independently with a few entrepreneurs on their technical platforms. This has mostly entailed working with tools around Infrastructure-As-A-Service (<a href="http://aws.amazon.com/">AWS</a>), configuration management (<a href="http://wiki.opscode.com/display/chef/Home">Chef</a> and <a href="http://www.rightscale.com/">RightScale</a>), search (<a href="http://lucene.apache.org/solr/">Solr</a>) and learning a lot about programming and going into production with ruby and rails. I've spoken to a number of friends and acquaintances who are working as consultants, contractors, technical authors and trainers. Some are working as lone-wolves and others working within or have founded larger organizations. I'm always sniffing for where the upside opportunities are and the question that comes to my mind is: how do such businesses scale?
632. </p>
633. <p>
634. A number of technology services companies that I've taken notice of have been funded in the last year or so including <a href="http://www.opscode.com/">OpsCode</a>, <a href="http://reductivelabs.com/">Reductive Labs</a>, <a href="http://cloudera.com/">Cloudera</a> and <a href="http://www.lucidimagination.com/">Lucid Imagination</a>. I think all of these guys are in great positions; virtual infrastructure (which is peanut butter to the chocolate of IaaS), big data and information retrieval technologies provide the primordial goo that will support new mobile, real time and social software applications. They are all working in rapid innovation spaces that hold high value potentials but also new learning and implementation challenges that rarefy specialized knowledge.
635. </p>
636. <p>
637. Years ago when I was working with Covalent Technologies, we tried to build a business around "enterprise open source" with Apache, Tomcat, etc as the basis. Packaging and selling free software is difficult. On the one hand, offering a proven technology stack configuration to overcome the usual integration and deployment challenges as well as providing a support resource is really valuable to Fortune 500's and such. However, my experience there and observations of what's happened with similar businesses (such as Linuxcare and SpikeSource) has left me skeptical how big the opportunity is. After all, while you're competing with the closed-source proprietary software vendors, you're also competing with Free.
638. </p>
639. <p>
640. The trend I'm noticing is the branching out away from the packaging and phone support and into curriculum. Considering that most institutional software technology education, CS degrees, extended programs, etc have curricula that are perpetually behind the times, it makes sense that the people who possess specialized knowledge on the bleeding edge lead the educational charge. <a href="http://www.lucidimagination.com/How-We-Can-Help/Training/Online-Training-Schedule">Lucid Imagination</a>, <a href="http://www.cloudera.com/hadoop-training">Cloudera</a> and <a href="http://www.scaleunlimited.com/courses">Scale Unlimited</a> are illustrating this point. While on-premise training can be lucrative, I think online courseware may provide a good answer to the business scale question.
641. </p>
642. <p>
643. For myself, I'm working with and acquiring knowledge in these areas tactically. Whatever my next startup will be, it should be world-changing and lucrative. And I'll likely be using all of these technologies. Thank goodness these guys are training the workforce of tomorrow!
644. </p>
645. <p>
646. <a href="http://technorati.com/tag/startups" rel="tag">startups</a>  
647. <a href="http://technorati.com/tag/scalability" rel="tag">scalability</a>  
648. <a href="http://technorati.com/tag/entrepreneuring" rel="tag">entrepreneuring</a>  
649. &lt;/p&gt;</description>          </item>
650.    <item>
651.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/configuring_web_clouds_with_chef</guid>
652.    <title>Configuring Web Clouds with Chef</title>
653.    <dc:creator>spidaman</dc:creator>
654.    <link>http://arachna.com/roller/spidaman/entry/configuring_web_clouds_with_chef</link>
655.        <pubDate>Thu, 21 May 2009 12:30:07 -0700</pubDate>
656.    <category>Musings</category>
657.            <description>&lt;p&gt;
658. <a href="http://code.fotonauts.com/blog/2009/05/introducing-casserole.html" title="Casserole">
659. <img src="http://www.arachna.com/images/casserole-icon-small-725869.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="128" height="128" /></a>
660. I'm not generally passionate about network and system operations, I prefer to focus my attention and creativity on system and software architectures. However, infrastructure provisioning, application deployment, monitoring and maintenance are facts of life for online services. When those basic functions aren't functioning well, then I get passionate about them. When service continuity is impacted and operations staff are overworked, it really bothers me; it tells me that I or other developers I'm working with are doing a poor job of delivering resilient software. I've had many conversations with folks who've accepted as a given that development teams and operations teams have friction between them; some even suggest that they <em>should</em>. After all, so goes that line of thinking, the developers are graded on how rapidly they implement features and fix bugs whereas the operators are graded on service availability and performance. Well, you can sell that all you want but I won't buy it.
661. </p>
662. <p>
663. In my view, developers need to deliver software that can be operated smoothly and operators need to provide feedback on how smoothly the software is operating; dev and ops must collaborate. I accept as a given that developers
664. <ul>
665. <li>Use source control</li>
666. <li>Write unit tests (after the fact or before/during <a href="http://en.wikipedia.org/wiki/Test-driven_development">TDD</a> style)</li>
667. <li>Write functional and integration tests</li>
668. <li>Maintain a build system for running test harnesses and packaging code</li>
669. <li>Document internal architecture and operating interfaces</li>
670. <li>Plan for change with respect to scale charactistics and functionality</li>
671. </ul>
672. Conversely, I accept as a given that operators
673. <ul>
674. <li>Use configuration management</li>
675. <li>Automate infrastructure provisioning, code deployment and rollback</li>
676. <li>Monitor infrastructure and application metrics</li>
677. </ul>
678. I don't want to oversimplify, there's more to the obligations that dev and ops have to each other in order to collaborate effectively. What I've noticed though is that a lot of operators might be skilled with configuring specific server infrastructure or performing OS analysis but configuration management and automation requires using really good tools that they lack. I've seen situations where the available tools are perceived as too complicated and so tools are developed that usually consist of a lot of specialized shell scripts (or perhaps it's just plain old NIH). <a href="http://www.cfengine.org/">Cfengine</a> is a good start but the reports I have is that it's difficult to work with and, if you're not very careful, may automatically manage to <em>mis</em>configure your systems. <a href="http://reductivelabs.com/products/puppet/">Puppet</a> was developed to be a more powerful system for configuration management but the feedback I've seen on it is adding new functionality is hard; it has its own configuration language and when you want to extend it you have to deal with a lot of complicated mechanisms. <a href="http://wiki.opscode.com/display/chef/Home">Chef</a> was developed to answer that frustration; by making the configuration language a <a href="http://en.wikipedia.org/wiki/Domain-specific_programming_language">DSL</a> on top of an already widely used scripting language (ruby), the chef code base provides an easier way to extend it and the chef codebase itself is reportedly an order of magnitude smaller and simpler than puppet's (cavaet: I generally distrust <a href="http://en.wikipedia.org/wiki/Source_lines_of_code">SLoC</a> metrics but just sayin').
679. </p>
680. <p>
681. So I've been giving Chef a test-drive for this infrastructure-on-EC2 management project that's been cooking. The system implemented the following use cases:
682. <ol>
683. <li>Launch web app servers on EC2 with Apache, <a href="http://www.modrails.com/">Passenger</a>, RoR (+other gems) and overlay a set of rails apps out of git</li>
684. <li>Launch a pair of reverse proxies (with ha-proxy) in front of the app servers - and reconfigure them when the set of app servers is expanded or contracted</li>
685. <li>Configure the proxy for failover with <a href="http://www.linux-ha.org/HeartbeatProgram">heartbeat</a></li>
686. <li>Add new rails apps to the set of app servers</li>
687. <li>Updating/rolling back rails apps</li>
688. </ol>
689. The system is enabled through a combination of the EC2 API (via <a href="http://rubyforge.org/projects/rightaws/">RightAWS</a>) and Chef's REST API as well as using <a href="http://github.com/ezmobius/chef-deploy/tree/master">chef-deploy</a> (think: Capistrano run by a system provisioning agent) to augment Chef's functionality. So far, it seems to be working great!
690. </p>
691. <p>
692. There's a lot of energy in the Chef community (check out <a href="http://code.fotonauts.com/blog/2009/05/introducing-casserole.html">Casserole</a>), combined with monitoring, log management and cloud technologies, I think there's a lot of IT streamlining ahead. Perhaps the old days of labor and communication intensive operations will give way to a new era of autonomic computing. I'll post further about some of the mechanics of working with ruby, rails, chef, EC2, chef-deploy and other tools in the weeks ahead (particularly now that EC2 has native <a href="http://developer.amazonwebservices.com/connect/ann.jspa?annID=446">load balancing, monitoring and auto-scaling</a> capabilities). I'll also talk a bit about this stuff at a <a href="http://en.oreilly.com/velocity2009/public/cfp/66">Velocity BoF</a>. If you're thinking about attending Velocity, O'Reilly is offering <a href="http://en.oreilly.com/velocity2009">30% off to the first 30 people to register today with the code vel09d30</a> <strong>today</strong> (no I'm not getting any kinduva kickback from O'Reilly). And you can catch <a href="http://en.oreilly.com/velocity2009/public/schedule/detail/8324">Infrastructure in the Cloud Era</a> with <em>Adam Jacob (Opscode), Ezra Zygmuntowicz (EngineYard) </em> to learn more about Chef and cloud management.
693. </p>
694. <p>
695. <a href="http://technorati.com/tag/chef" rel="tag">chef</a>  
696. <a href="http://technorati.com/tag/puppet" rel="tag">puppet</a>  
697. <a href="http://technorati.com/tag/cfengine" rel="tag">cfengine</a>  
698. <a href="http://technorati.com/tag/ec2" rel="tag">ec2</a>  
699. <a href="http://technorati.com/tag/aws" rel="tag">aws</a>  
700. <a href="http://technorati.com/tag/ruby" rel="tag">ruby</a>  
701. <a href="http://technorati.com/tag/cloud+computing" rel="tag">cloud computing</a>  
702. <a href="http://technorati.com/tag/velocity" rel="tag">velocity</a>  
703. </p>
704. </description>          </item>
705.    <item>
706.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/cloning_vmware_machines</guid>
707.    <title>Cloning VMware Machines</title>
708.    <dc:creator>spidaman</dc:creator>
709.    <link>http://arachna.com/roller/spidaman/entry/cloning_vmware_machines</link>
710.        <pubDate>Tue, 31 Mar 2009 09:12:22 -0700</pubDate>
711.    <category>Musings</category>
712.            <description>&lt;p&gt;
713. <a href="http://www.vmware.com/" title="VMware">
714. <img src="http://www.arachna.com/images/vmware_black_logo.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="187" height="72" /></a>
715. I bought a copy of <a href="http://www.vmware.com/products/fusion/">VMware Fusion</a> on special from <a href="http://www.smithmicro.com/">Smith Micro</a> (icing on the cake: they had a 40% off special that week) specifically so I could simulate a network of machines on my local MacBook Pro. While I've heard good things about <a href="http://www.virtualbox.org/">Virtual Box</a>, one of the other key capabilities I was looking for from MacIntosh virtualization software was the ability convert an existing Windows installation to a virtual machine. VMware reportedly has the best tools for that kind of thing. I have an aging Dell with an old XP that I'd like to preserve when I finally decide to get rid of the hardware; when it's time to Macify, I'll be good to go.
716. </p>
717. <p>
718. I started building my virtual network very simply, by <a href="http://lists.centos.org/pipermail/centos-virt/2008-February/000203.html">creating a CentOS VM</a>. Once I had my first VM running, I figured I could just grow the network from there; I was expecting to find a "clone" item in the Fusion menus but alas, no joy. So, it's time to hack. Looking around at the artifacts that Fusion created, a bunch of files in a directory named for the VM, I started off by copying the directory, the files it contained that had the virtual machine name as components of the file name and edited the metadata files ({vm name}.vmdk/.vmx/.vmxf). Telling Fusion to launch that machine, it prompted if this was a copy or a moved VM - I told it that it was copied and the launch continued. Both launched VM's could ping each other so voila: my virtual network came into existence.
719. </p>
720. <p>
721. I've since found <a href="http://communities.vmware.com/docs/DOC-5611">another procedure to create "linked clones" in VMware Fusion</a>. It looks like this will be really useful for my next scenario of having two different flavors of VM's running on my virtual network. The setup I want to get to is one where I can have "manager" host (to run provisioning, monitoring and other management applications) and cookie-cutter "worker" hosts (webservers, databases, etc). Ultimately, this setup will help me tool up for cloud platform operations; I have more Evil Plans there.
722. </p>
723. <p>
724. So all of this has me wondering: why doesn't VMware support this natively? Where's that menu option I was looking for? Is there an alternative to this hackery that I just overlooked?
725. </p>
726. <p>
727. <a href="http://technorati.com/tag/vmware" rel="tag">vmware</a>  
728. <a href="http://technorati.com/tag/virtualization" rel="tag">virtualization</a>  
729. <a href="http://technorati.com/tag/centos" rel="tag">centos</a>  
730. <a href="http://technorati.com/tag/vmware+fusion" rel="tag">vmware fusion</a>  
731. <a href="http://technorati.com/tag/vm+cloning" rel="tag">vm cloning</a>  
732.  
733. &lt;/p&gt;</description>          </item>
734.    <item>
735.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/going_to_metallica_s_rock</guid>
736.    <title>Going to Metallica&apos;s Rock and Roll Hall of Fame Induction</title>
737.    <dc:creator>spidaman</dc:creator>
738.    <link>http://arachna.com/roller/spidaman/entry/going_to_metallica_s_rock</link>
739.        <pubDate>Thu, 12 Mar 2009 12:22:51 -0700</pubDate>
740.    <category>Music</category>
741.            <description>&lt;p&gt;
742. <a href="http://www.rockhall.com/" title="Metallica - Rock and Roll Hall of Fame">
743. <img src="http://www.arachna.com/images/metallica_rrhof_invite.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="295" height="137" /></a>
744. Those <a href="http://www.time.com/time/arts/article/0,8599,1877187,00.html">25 things you should know about me</a> memes circulating rarely interest me (honestly, I don't care that you have a collection of rare El Salvadoran currency). However, one thing that my friends know but regular readers may not is that I have a fairly eclectic background. Did you know that I used to hang around the <a href="http://art.sfsu.edu/">art department's hot glass studio in college</a> to blow glass? Did you know that I learned to program in <a href="http://en.wikipedia.org/wiki/Pascal_(programming_language)">Pascal</a> when I was in college and hated it? Yea, yea, I don't care much anymore either. But anyway, back in the 80's I was friends with this Danish dude from LA who shared my interest in the underground heavy metal scene that was burgeoning, particularly in Britain ("New Wave of British Heavy Metal" AKA NWOBHM) and Europe. We used to trade records and demos (the first <em>Def Leppard</em> 3 song EP on 9" vinyl, I was tired of it so I traded him for a bunch of <em>Tygers of Pan Tang</em> and other crap I didn't own already). I think he, like myself, used to pick up copies of Melody Maker and Sounds at the local record store to read about what was going on overseas. Eventually, <a href="http://www.kerrang.com/">Kerrang!</a> came out providing fuller coverage of the metal scene, complete with glossy pictures. But in the meantime back in San Francisco, I helped a friend of mine (Ron Quintana) operate his fanzine <em>Metal Mania</em> (don't be confused, the name was re-appropriated by various larger publishing concerns at different times in the years since but none of them had any relationship to the original gangstas).
745. </p>
746. <p>
747. Back in the day, <a href="http://downwithtyranny.blogspot.com/">Howie Klein</a> was a muckety muck in the music industry, haunting the local clubs like <em>The Old Waldorf</em> and <em>Mabuhay Gardens</em>. Howie hooked us up with a show on <a href="http://kusf.org/">KUSF</a>. I dubbed the show <a href="http://www.rampageradio.org/">Rampage Radio</a>, it ran in the wee hours every Saturday night (right after <a href="http://www.bigrick.fm/">Big Rick Stuart</a> finished up his late night reggae show with those dudes from Green Apple Records on Clement Street). In between hurling insults at "album oriented rock" and big-hair metal bands (posers!), we played a lot of stuff you couldn't hear anywhere else. Among the many obscure noises we aired were demos from East Bay metalheads <a href="http://www.exodusattack.com/">Exodus</a>. Amazingly, Rampage Radio is <em>still on the air</em>. Well, that Danish kid and one of the guys I befriended from Exodus were Lars Ulrich and Kirk Hammett, respectively. In short order, they would be playing together in a band Lars named <a href="http://www.metallica.com/">Metallica</a> (after haggling with Ron about <em>not</em> taking that name for the 'zine).
748. </p>
749. <p>
750. I eventually lost interest in the metal <em>scene</em> (not enough innovation, too much sound-alike derivatives to keep me listening); even though the music from then is still on my playlist, my repertoire has broadened widely (talk to me about gypsy style string jazz, please). I've been peripherally in touch with friends from back then. Over the years, I'd go to a few Metallica shows but the guys are always mobbed at the backstage parties, there's not much of an opportunity to actually talk about anything. Anyway, we have little in common now. I develop software and <a href="http://technorati.com/">crazy assed online services</a>; they tour the world to perform in front of throngs. And I don't drink Jaegermeister anymore. In 2000, I introduced one of the friends I've stayed in touch with, <a href="http://haggisbuffet.blogspot.com/">Brian Lew</a>, who also had a fanzine Back In The Day, to editors at <a href="http://www.salon.com/">salon.com</a> (where I was working at the time). He contributed a great article expressing a sentiment that I shared, <a href="http://archive.salon.com/ent/log/2000/05/09/metallica_fan/index.html" title="Metallica, how could you?">dismay at Metallica's war on Napster</a>. I don't think I've actually talked to Lars in 15 years. After seeing news coverage of him ranting about how people (his most valued asset: his fans) where ripping him off, I'm not sure I wanted to. But I think we're all over that now, let's just play <a href="http://www.rockband.com/news/metallica_lets_fans_dig_deeper">Rock Band</a> and fuhgedaboutit.
751. </p>
752. <p>
753. So here we are decades later and Metallica hasn't just warped the music industry, they <em>are</em> the industry. They're up there with Elvis and the Beatles and all of that (except, barring Cliff Burton, they're not dead). Last week, Brian pings me that Q-Prime (Metallica's management company in New York) is trying to reach me. After a few phone calls, it turns out that Metallica is honoring a handful of us old-schoolers by inviting us to a big shindig in Cleveland for their <a href="http://www.rockhall.com/">Rock and Roll Hall of Fame</a> induction next month. How cool is that?! I'm still kind blown away that this is really happening (am I being punkd??).
754. </p>
755. <p>
756. So, I may be <a href="http://www.arachna.com/roller/spidaman/entry/more_changes_at_technorati_this">leaving Technorati</a> but I'm going to the Rock and Roll Hall of Fame! w00t! That tune keeps humming through my conscience, "...living in sin with a safety pin, Cleveland rocks! Cleveland rocks!" but the way it sounds in my head, it's ganked up, roaring from a massive PA and a wall of <a href="http://www.marshallamps.com/">Marshall</a> stacks. So now you know what my plans will be in a few weeks and now you've learned a dozen or so things about me (if not 25) that you may not have known before.
757. </p>
758. <p>
759. <em>
760. <a href="http://technorati.com/tag/metallica" rel="tag">metallica</a>  
761. <a href="http://technorati.com/tag/rock+and+roll+hall+of+fame" rel="tag">rock and roll hall of fame</a>  
762. <a href="http://technorati.com/tag/metal" rel="tag">metal</a>  
763. <a href="http://technorati.com/tag/kusf" rel="tag">kusf</a>  
764. <a href="http://technorati.com/tag/cleveland" rel="tag">cleveland</a>  
765. <a href="http://technorati.com/tag/nwobhm" rel="tag">nwobhm</a>  
766.  
767. </em>
768. &lt;/p&gt;</description>          </item>
769.    <item>
770.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/more_changes_at_technorati_this</guid>
771.    <title>More Changes At Technorati (this time, it&apos;s personal)</title>
772.    <dc:creator>spidaman</dc:creator>
773.    <link>http://arachna.com/roller/spidaman/entry/more_changes_at_technorati_this</link>
774.        <pubDate>Tue, 10 Mar 2009 14:06:20 -0700</pubDate>
775.    <category>Musings</category>
776.            <description>&lt;p&gt;
777. <a href="http://technorati.com/" title="Technorati">
778. <img src="http://www.arachna.com/images/technorati_logo.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="110" height="15" /></a>
779. My post <a href="http://www.arachna.com/roller/spidaman/entry/new_crawlers_at_technorati" title="New Crawlers At Technorati">last week</a> focused on some of the technology changes that I've been spearheading at <a href="http://technorati.com/">Technorati</a> but this time, I have a personal change to discuss. When I joined Technorati in 2004, the old world of the web was in shambles. The 1990's banner-ads-on-a-CPM-basis businesses were collapsed. The editorial teams using big workflow-oriented content management system (CMS) infrastructure (which I worked on in the 90's) were increasingly eclipsed by the ecosystem of blogs. Web 2.0 wasn't yet the word on everyone's lips. But five years ago, <a href="http://www.sifry.com/alerts">Dave Sifry</a>'s infectious vision for providing "connective tissue" for the blog ecosystem, tapping the attention signals and creating an emergent distributed meta-CMS helped put it there. Being of service to bloggers just sounded too good, so I jumped aboard.
780. </p>
781. <p>
782. Through many iterations of blogospheric expansion, building data flow, search and discovery applications, dealing with data center outages (and migrations) and other adventures, it's been a long strange trip. I've made a lot of fantastic friends, contributed a lot of insight and determination and learned a great deal along the way. I am incredibly proud of what we've built over the last five years. However today it's time for me to move on, my last day at Technorati will be next week.
783. </p>
784. <p>
785. Technorati has a lot of great people, technology and possibilities. The aforementioned crawler rollout provides the technology platform with a better foundation that I'm sure <a href="http://dorion.blogspot.com/">Dorion</a> and the rest of the team will build great things on. The ad platform will create an abundance of valuable opportunities for bloggers and other social media. I know from past experiences what a successful media business looks like and under <a href="http://www.jalichandra.com/" title="Richard Jalichandra">Richard Jalichandra</a>'s leadership, I see all of the right things happening.  The ad platform will leverage Technorati's social media data assets with the publisher and advertiser tools that will make Technorati an ad delivery powerhouse. I'm going to remain a friend of the company's and do what I can to help its continued success, but I will be doing so from elsewhere.
786. </p>
787. <p>
788. I want to take a moment to thank all of my colleagues, past and present, who have worked with me to get Technorati this far. The brainstorms, the hard work, the arguments and the epiphanies have been tremendously valuable to me. <em>Thank You!</em>
789. </p>
790. <p>
791. I'm not sure what's next for me. I feel strongly that the changes afoot in cloud infrastructure, open source data analytics, real time data stream technologies, location based services (specifically, GPS ubiquity) and improved mobile devices are going to build on Web 2.0. These social and technology shifts will provide primordial goo out of which new innovations will spring. And I intend to build some of them, so brace yourself for Web 3.0. It's times like these when the economy is athrash that the best opportunities emerge and running for cover isn't my style. The next few years will see incumbent players in inefficient markets crumble and more powerful paradigms take their place. I'm bringing my hammer.
792. </p>
793. <p>
794. <a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>
795. &lt;/p&gt;</description>          </item>
796.    <item>
797.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/welcome_to_the_technorati_top</guid>
798.    <title>Welcome to the Technorati Top 100, Mr. President</title>
799.    <dc:creator>spidaman</dc:creator>
800.    <link>http://arachna.com/roller/spidaman/entry/welcome_to_the_technorati_top</link>
801.        <pubDate>Wed, 4 Mar 2009 22:59:16 -0800</pubDate>
802.    <category>Musings</category>
803.            <description>&lt;p&gt;
804. <a href="http://technorati.com/blogs/www.whitehouse.gov/blog" title="White House Blog">
805. <img src="http://www.arachna.com/images/white_house_blog_98.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="352" height="347" /></a>
806. Since its inception just 6 weeks ago, the <a href="http://www.whitehouse.gov/blog">White House Blog</a> has had a tremendous center of gravity. I noted the <a href="http://www.arachna.com/roller/spidaman/entry/inbloguration_one_week_later">volume of links</a> coming in to the White House Blog after the first week.  This is an existential moment for the blogosphere because today the White House Blog has 3216 links from 2323 blogs. And so it's official: the White House Blog has reached the <a href="http://technorati.com/pop/blogs">Technorati Top 100</a>.
807. </p>
808. <p>
809. I find myself reflecting on what the top 100 looked like four years ago, after the prior presidential inauguration, and what it looks like today; the blogosphere is a very different place. Further down memory lane, who recalls when <a href="http://web.archive.org/web/20030401084126/http://www.technorati.com/cosmos/top100.html">Dave Winer and Instapundit were among the top blogs</a>? Yep, most of the small publishers have been displaced by those with big businesses behind them. Well, at least <a href="http://www.boingboing.net/">BoingBoing</a> endures but Huffpo and Gizmo better watch out, here comes Prezbo.
810. </p>
811. <p style="clear:both;"><em>
812. <a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>  
813. <a href="http://technorati.com/tag/white+house" rel="tag">white house</a>  
814. <a href="http://technorati.com/tag/inauguration" rel="tag">inauguration</a>  
815. <a href="http://technorati.com/tag/blog" rel="tag">blog</a>  
816. &lt;/em&gt;&lt;/p&gt;</description>          </item>
817.    <item>
818.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/new_crawlers_at_technorati</guid>
819.    <title>New Crawlers At Technorati</title>
820.    <dc:creator>spidaman</dc:creator>
821.    <link>http://arachna.com/roller/spidaman/entry/new_crawlers_at_technorati</link>
822.        <pubDate>Wed, 4 Mar 2009 20:31:16 -0800</pubDate>
823.    <category>Musings</category>
824.            <description>&lt;p&gt;
825. <a href="http://www.flickr.com/photos/kitchenmage/1395622544/" title="Spider Web">
826. <img src="http://www.arachna.com/images/spiderweb.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="128" height="134" /></a>
827.  
828. A lot of changes are afoot at Technorati. Over the last year or so, we've been looking inward at the infrastructure and asking ourselves, "How can we do this better?". The data spigot that Technorati builds on was the first thing to focus on, it's a critical part in one leg of the back-end infrastructure tripod. The tripod consists of data acquisition, search and analytics Technorati; while the ping handling and queuing are relatively simple affairs the crawler is the most sophisticated of the data acquisition subsystems. It's proper functioning is critical to the functioning of the other legs; when it doesn't function well, search and analytics don't either (GIGO="garbage in/garbage out").
829. </p>
830. <p>
831. As <a href="http://dorion.blogspot.com/">Dorion</a> mentioned recently, <a href="http://technorati.com/weblog/2009/02/473.html">we're retiring the old crawler</a>. Why are we giving the old crawler getting an engraved watch and showing it to the door? Well, old age is one reason. The original spider is a technology that dates back to 2003,  the blogosphere has changed a lot since then and we have a much better developed understanding of the requirements. The original spider code has presented a sufficient number of GIGO-related and code maintenance challenges to warrant a complete re-thinking. It contrasts starkly with the replacement.
832. <dl>
833. <dt>Data model</dt>
834. <dd>There are a lot of ways to derive structural information out of the pages and feeds that a blog presents. The old spider used event driven parses, building a complex state as it went with flat data structures (lists and hashes). The new one uses the composed web documents to populate a well-defined object model; all crawls normalize the semi-structured data found on the web to that model.</dd>
835. <dt>Crawl persistence</dt>
836. <dd>The old spider was hard-wired to persist the aforementioned data structure elements to relational databases (sharded MySQL instances) while it was parsing, so that the flow of saving parsed data was closely coupled with parsing events, forsaking transactional integrity and consuming costly resources. The new spider composes and saves its parse result as a big discreet object (not collections of little objects in an RDBMS). This reduced the hardware footprint by an order of magnitude.
837. </dd>
838. <dt>Operational visibility</dt>
839. <dd>Whether a blog's page structure was understood (or not), the feed was  well formed (or not) or any of the many other things that determine the success or quality of a blog's crawl was opaque under the old spider. With the new spider, detailed metadata and metrics are tracked during the crawl cycles. This better enables the team to support bloggers and extend the system's capabilities.</dd>
840. <dt>Unit tests</dt>
841. <dd>Wherever you have complex, critical software you want to have unit tests. The old spider had almost no unit tests and was developed in a way that made testing the things that mattered most exceptionally difficult. The new spider was developed with a test harness upfront, it now has hundreds of tests that validate thousands of aspects of the code. The test are uniformly invoked by the developers and automatically whenever the code is updated (AKA under <a href="http://martinfowler.com/articles/continuousIntegration.html">continuous integration</a>).
842. </dd>
843. </dl>
844. The old spider didn't leverage packages to logically separate the different concerns (fetching, parsing, validation, change determination, etc), the aforementioned flat data structures, mingling of concerns and absence of unit tests made changing it exceedingly difficult. Now, we have a whole that is greater than the sum of the parts; having a well defined data model, sensible persistence, operational visibility and unit tests has added up to an order of magnitude improvement across several dimensions. The real benefits are seen when we've shown that the system is easy to change, I mentioned this several weeks ago when I noted the ease with which we could <a href="http://www.arachna.com/roller/spidaman/entry/inbloguration_one_week_later" title="Inbloguration: One Week Later">adapt custom requirements to crawl the White House blog</a>.
845. </p>
846. <p>
847. Another change that we've made is to the legacy assumption that everything that pings is a blog. That assumption proved to be increasingly untenable as the ping meme spread amongst those who didn't really understand the difference between some random page and a blog, nefarious publishers (spammers) and other perpetrators of <a href="http://en.wikipedia.org/wiki/Sping">spings</a>. Over 90% of the pings hitting Technorati are rejected outright because they've been identified as invalid pings. A large portion of the remainder are later determined to be invalid but we now have a rigorous system in place for filtering out the noise.  We've reduced the spam level considerably (as mentioned in a  <a href="http://www.arachna.com/roller/spidaman/entry/social_media_backlash_against_cheaters" title="Social Media Backlash Against Cheaters and Fleshmongers">prior post</a>). For instance, there's a whole genre of <a href="http://en.wikipedia.org/wiki/Spam_blog">splogs</a> that are pornography focused (hardcore pictures, paid affiliate links, etc) that previously plagued our data; now we've eliminated a lot of that nonsense from the index.
848. </p>
849. <p>
850. Here are a pair of charts showing the daily occurrence of a particular porn term in the index.
851. <div style="float:left; width:48%; height: 220px;">
852. 6 month retrospective as of November 3rd, 2008: <br/>
853. <img src="http://www.arachna.com/images/blowjob_180_day_20081103.png" width="300" height="180" style="bottom:10px; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand; vertical-align:bottom;" border="0" />
854. </div>
855.  
856. <div style="float:right; width:48%; height: 220px;">
857. 6 month retrospective today, 5 months later: <br/>
858. <img src="http://www.arachna.com/images/blowjob_180_day_20090303.png" width="300" height="180" style="bottom:10px; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" />
859. </div>
860. </p>
861. <p style="clear:both;">
862. As you can see, that's an order of magnitude reduction; 90% of the occurrences of that term was spam.
863. </p>
864. <p style="clear:both;">
865. So what's next for the crawler? We've got some stragglers on the old spider, we're going to migrate them over in the next few days. There are still a lot of issues to shake out, as with any new software (for instance, there are still some error recovery scenarios to deal with). But it's getting better all of the time (love that song). We'll be rolling out new tools internally for identifying where improvements are needed, ultimately we'd like to enable bloggers to help themselves to publish, get crawled, be found and recognized more effectively. And there are more changes afoot, stay tuned.
866. </p>
867. <p><em>
868. <a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>  
869. <a href="http://technorati.com/tag/web+crawling" rel="tag">web crawling</a>  
870. <a href="http://technorati.com/tag/software" rel="tag">software</a>  
871. <a href="http://technorati.com/tag/spam" rel="tag">spam</a>  
872. <a href="http://technorati.com/tag/splogs" rel="tag">splogs</a>  
873. &lt;/em&gt;&lt;/p&gt;</description>          </item>
874.    <item>
875.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/code_is_social</guid>
876.    <title>Code Is Social</title>
877.    <dc:creator>spidaman</dc:creator>
878.    <link>http://arachna.com/roller/spidaman/entry/code_is_social</link>
879.        <pubDate>Fri, 27 Feb 2009 18:26:15 -0800</pubDate>
880.    <category>Musings</category>
881.            <description>&lt;p&gt;
882. <a href="http://www.amazon.com/exec/obidos/ASIN/B0019D8HTI/wwwarachnacom-20" title="Social Code">
883. <img src="http://www.arachna.com/images/social_code.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="115" height="115" /></a>
884. Like many software developers, I confess that I have found myself on occasion coming back to old code that I wrote asking, "What was I thinking?" Where it's my code, this <a href="http://www.urbandictionary.com/define.php?term=WWIT">WWIT</a> question doesn't happen very often these days but in general I can't count the number of times I've encountered and had to work on code that was not written to be read. I think sometimes programmers write terse code and regard its brevity as a badge of honor, "If I'm wizardly enough to write this, then only True Wizards will read it." Or maybe it's just laziness or hurriedness, these code mysteries are akin to omitting comments, API documentation and other communication artifacts. When I see non-descriptive variable names, gratuitous indirection, excessive right indenting, monkey patching or unnecessary <a href="http://en.wikipedia.org/wiki/Cyclomatic_complexity">cyclomatic complexity</a>, it's almost anti-social behavior; it's a communication fail more than a functional one. Likewise, gratuitous verbosity stifles communication in the opposite manner; this isn't pre-school - grasp of the ABC's is assumed. So spelling out what code is doing in this belabored fashion is just silly:
885. <pre>
886. # an array to collect permalinks
887. permalinks = []
888. # loop over the feed entries
889. for entry in feed.entries:
890.    permalinks.append(entry.link)
891. </pre>
892. Whereas this is clear
893. <pre>
894. # extract an array of permalinks from the feed entries
895. permalinks = [ entry.link for entry in feed.entries ]
896. </pre>
897. OK, I'm assuming the reader knows what a Python <a href="http://en.wikipedia.org/wiki/List_comprehension">list comprehension</a> does. The first one is using a lot of vertical space to satisfy a very simple intent. I often find the opposite problem, excessive brevity, is authored by those enamored with their language's idioms. Software written with scripting languages often exhibit this; Perl is famous for <a href="http://en.wikipedia.org/wiki/There_is_more_than_one_way_to_do_it">expressiveness</a> (I say this lovingly as a repentant x-Perl Wizard) but even the languages with adherents claiming their tongue is more "readable" have those same users donning wizard hats, trying to be clever. Ever tried to maintain Python code riddled with nested list comprehensions containing lambdas? Ruby, similar idiomatic norms abound, 'nuf said.
898. </p>
899. <p>
900. I've appreciated celebrations of wizardry (see <a href="http://olabini.com/blog/2009/01/a-folding-language/">A folding language</a>) but there's more to wizardry than meta-programming and brevity. Coding like a wizard doesn't mean being so clever that only other wizards can collaborate. In my view, a true wizard has the wisdom to steer clear of verbose indulgences <em>and</em> terse spells; the wizard walks the middle path of clarity so that the code is not dumbed-down but the apprentice will grasp the intent. The wizard's code should read as poetry.
901. </p>
902. <p>
903. When code is unsocial (or anti-social), the quality suffers. Complex software needs a gene pool - lots of eyeballs, lots of variant perspectives and experiences. A small gene pool leads to in-bred ideas. Thus code from a lone wolf (even a kick-ass wolf) will usually be of lower quality than code developed by a plurality (unless its a plurality of novices, then all bets are off).
904. </p>
905. <p>
906. My plea to fellow crafters of bits: please code for clarity. Don't be so brief that your intentions are unclear. And don't be so garrulous that your intention is lost in the verbiage. Again, I'm not claiming innocence of these sins of code. But over the years I've become considerably more <em>aware</em> of the costs and benefits in the choices between brevity and verbosity. Perhaps clarity is in the eye of beholder or perhaps more narrowly, in the eye of the author. But I try to look at my own code objectively and ask, "If I don't see this code for six months and then come back to it to do some maintenance, will today's intent be clear?" I hope the code I write will be approachable by those who come behind me to work on it, especially if it's me lest I ask myself the WWIT question.
907. </p>
908. <p>
909. Sigh, I'm venting because I just got side tracked refactoring some program code (and its single test) that lacked clarity. Thanks for indulging me this far. I'm gonna go listen to some old <a href="http://www.socialdistortion.com/">Social Distortion</a> now, have a great weekend!
910. </p>
911. <p><em>
912. <a href="http://technorati.com/tag/programming" rel="tag">programming</a>  
913. <a href="http://technorati.com/tag/wizardry" rel="tag">wizardry</a>  
914. <a href="http://technorati.com/tag/perl" rel="tag">perl</a>  
915. <a href="http://technorati.com/tag/python" rel="tag">python</a>  
916. <a href="http://technorati.com/tag/ruby" rel="tag">ruby</a>  
917. </em></p>
918.  
919. </description>          </item>
920.    <item>
921.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/inbloguration_one_week_later</guid>
922.    <title>Inbloguration: One Week Later</title>
923.    <dc:creator>spidaman</dc:creator>
924.    <link>http://arachna.com/roller/spidaman/entry/inbloguration_one_week_later</link>
925.        <pubDate>Tue, 27 Jan 2009 12:41:39 -0800</pubDate>
926.    <category>Musings</category>
927.            <description>&lt;p&gt;
928. <a href="http://www.whitehouse.gov/blog" title="The White House Blog">
929. <img src="http://www.arachna.com/images/whitehouse_logo.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="267" height="115" /></a>
930. As mentioned last week on the <a href="http://technorati.com/weblog/2009/01/466.html">Technorati blog</a>, Technorati was crawling the new <a href="http://www.whitehouse.gov/blog" title="The White House Blog">White House blog</a> within a day of its launch. Most of the blogosphere doesn't require individual customization in our crawling framework but in some special cases, it must be done. The White House blog is a prime example of why this is so but I'm pleased to report that Technorati's new crawling technology makes what was impossible with our old crawler easy with the new one.
931. </p>
932. <p>
933. Given the volume of moderation that'd be required, it doesn't surprise me that the posts don't take comments. But there are other more basic blogging practices of concern:
934. <ul>
935. <li>It doesn't have clear feed discovery. There are multiple <code>rel="alternate"</code> elements in the <code>head</code> section of the HTML document.</li>
936. <li>The permalinks are irregular; the first few were under the blog URL, the rest have been under other paths on the whitehouse.gov site.</li>
937. <li>The HTML doesn't have any of the common CSS attributes that indicate structural semantics; if I get to vote on this, I'll opt for hAtom</li>
938. </ul>
939. The absence of full content feeds or full content on the front page isn't a problem for Technorati's new crawler; it can crawl the post pages and extract content to index from those.
940. </p>
941. <p>
942. So how has the blog done in the last week? Well, it emerged in the top 1000 just <em>five days</em> after its inception. Keep an eye on the blog's <a href="http://technorati.com/blogs/www.whitehouse.gov/blog">Technorati blog info page</a> - it's currently ranked 882 (1,876 links from 1440 blogs). Many of the links are to main blog page, citing its existence. But the majority of the links were to the kick-off post (date line reads: "Tuesday, January 20th, 2009 at 12:01 pm") and the inaugural address. Here are the posts and their link counts:
943. <ul>
944. <li><a href="http://www.whitehouse.gov/blog/change_has_come_to_whitehouse-gov/">Change has come to WhiteHouse.gov</a> - 619 links from 552 blogs</li>
945. <li><a href="http://www.whitehouse.gov/blog/inaugural-address/">President Barack Obama's Inaugural Address</a> - 217 links from 194 blogs</li>
946. <li><a href="http://www.whitehouse.gov/blog/a_national_day_of_renewal_and_reconciliation/">A National Day of Renewal and Reconciliation</a> - 146 links from 137 blogs</li>
947. <li><a href="http://www.whitehouse.gov/president-obama-delivers-your-weekly-address/">President Obama delivers Your Weekly Address</a> - 103 links from 97 blogs</li>
948. <li><a href="http://www.whitehouse.gov/statement-released-after-the-president-rescinds/">Statement released after the President rescinds "Mexico City Policy"</a> - 34 links from 33 blogs</li>
949. <li><a href="http://www.whitehouse.gov/executive-orders-to-date/">Executive Orders to date</a> - 27 links from 27 blogs</li>
950. <li><a href="http://www.whitehouse.gov/blog/the_whistle_stop_tour/">The Whistle Stop Tour</a> - 26 links from 24 blogs</li>
951. <li><a href="http://www.whitehouse.gov/blog/dr_king_service/">Honoring Dr. King's Legacy and Serving America</a>  - 22 links from 20 blogs</li>
952. <li><a href="http://www.whitehouse.gov/blog_post/Fromperiltoprogress/">From peril to progress</a> - 17 links from 17 blogs</li>
953. <li><a href="http://www.whitehouse.gov/now-comes-lilly-ledbetter/">Now Comes Lilly Ledbetter</a> - 15 links from 13 blogs</li>
954. <li><a href="http://www.whitehouse.gov/58_years_of_Indian_democracy/">58 years of Indian democracy</a> - 4 links from 4 blogs</li>
955. <li><a href="http://www.whitehouse.gov/the_year_of_the_ox/">The year of the ox</a> - 3 links from 3 blogs</li>
956. <li><a href="http://www.whitehouse.gov/Timothy_Geithner_sworn_in/">Timothy Geithner sworn in as Secretary of the Treasury</a> - 3 links from 3 blogs</li>
957. <li><a href="http://www.whitehouse.gov/blog_post/PressBriefingHighlights/">Press Briefing Highlights</a> - 3 links from 3 blogs</li>
958. <li><a href="http://www.whitehouse.gov/president-obama-on-the-selection-of-kirsten-gillibrand/">President Obama on the Selection of Kirsten Gillibrand</a> - 2 links from 2 blogs</li>
959. </ul>
960. </p>
961. <p>
962. It's really encouraging to see the White House, specifically <a href="http://www.bluestatedigital.com/">Macon Phillips</a>, posting updates about President Obama's policies and political activities. Going forward, I hope to see more consistent publishing practices. For instance, should slug words be separated by underscores or hyphens? Should the proper names in URL slugs be mixed case (e.g. Timothy_Geithner_sworn_in) or down cased (e.g. now-comes-lilly-ledbetter)? It's less important which one but just pick one and stick to it! Also, the White House Blog should use durable URLs for permalinks: there's a posting with the path <code>/blog_post/PressBriefingHighlights/</code>, another with <code>/president-obama-delivers-your-weekly-address</code> - what are they going to use for the next post with highlights from a press briefing or next week's weekly address? If Mr. Phillips needs any further advise on effective blog publishing technology, I'm easy to find and happy to be of service.
963. </p>
964. <p><em>
965. <a href="http://technorati.com/tag/whitehouse" rel="tag">whitehouse</a>  
966. <a href="http://technorati.com/tag/white+house+blog" rel="tag">white house blog</a>  
967. <a href="http://technorati.com/tag/blogging" rel="tag">blogging</a>  
968. <a href="http://technorati.com/tag/blog+technology" rel="tag">blog+technology</a>  
969. <a href="http://technorati.com/tag/obama" rel="tag">obama</a>  
970. <a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>  
971. <a href="http://technorati.com/tag/inbloguration" rel="tag">inbloguration</a>  
972. </em></p>
973.  
974.  
975. </description>          </item>
976.    <item>
977.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/downturn_what_downturn</guid>
978.    <title>Downturn, what downturn?</title>
979.    <dc:creator>spidaman</dc:creator>
980.    <link>http://arachna.com/roller/spidaman/entry/downturn_what_downturn</link>
981.        <pubDate>Thu, 25 Dec 2008 13:20:14 -0800</pubDate>
982.    <category>Musings</category>
983.            <description>&lt;p&gt;
984.  
985. <a href="http://en.wikipedia.org/wiki/Data_center" title="Data centers">
986. <img src="http://www.arachna.com/images/data_center_rack.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="127" height="132" /></a>
987.  
988. Just the other day, <em>Data Center Knowledge</em> asked <a href="http://www.datacenterknowledge.com/archives/2008/12/23/are-colocation-prices-heading-higher/">Are Colocation Prices Heading Higher?</a> My immediate reaction was, that's a stupid question: last time VC funding went into hibernation, data center space was suddenly cheap and abundant. The article suggested that companies operating their own data centers will run to the colos as a cost cutting measure. Maybe, but I'm not so sure. Data center migrations can be expensive, risky operations. Methinks that the F500's inclined to undertake a migration would have done so already. The article cited a report emphasizing a shift from capital expenses to operating expenses.
989. <blockquote cite="http://www.datacenterknowledge.com/archives/2008/12/23/are-colocation-prices-heading-higher/">
990. Tier 1 says demand for data center space grew 14 percent over the past 12 months, while supply grew by just 6 percent, "exacerbating an already lopsided supply/demand curve."
991. </blockquote>
992. However, <a href="http://www.t1r.com/press_release.php?pr=709">Tier 1 attributed the demand</a>, "especially, (to) the primacy of the Internet as a vehicle for service and application delivery." With the litany of Techcrunch <a href="http://search.techcrunch.com/query.php?y=%2Ftc_eng_id%2Fsearch%2Fv1%2Fquery%2Fdeadpool%3Fcategory_id%3DTechCrunch%2520Search%26sort%3Ddate%26client%3Dtechcrunch">deadpool</a>
993. reports, I'm finding it difficult to believe that the data center space supply/demand will continue skewing.
994. </p>
995. <p>
996. Sure, it's not all bad news. Fred Wilson reports that Union Square Ventures will be <a href="http://www.avc.com/a_vc/2008/12/investing-in-th.html">Investing In Thick and Thin</a>. Acknowledging that, "it is easier to invest in thin times. The difficult business climate starts to separate the wheat from the chaff and the strong companies are revealed." Wilson goes on to say
997. <blockquote cite="http://www.avc.com/a_vc/2008/12/investing-in-th.html">
998. I don't feel that its possible, or wise, or prudent to attempt to time these (venture investment) cycles.
999. <br/>
1000. Our approach is to manage a modest amount of capital (in our case less than $300 million across two active funds) and deploy it at roughly $40 million per year, year in and year out no matter what part of the cycle we are in.
1001. <br/>
1002. That way we'll be putting out money at the top of the market but also at the bottom of the market and also on the way up and the way down. The valuations we pay will average themselves out and this averaging allows us to invest in the underlying value creation process and not in the market per se.
1003. </blockquote>
1004. Yes, the economy is gyrating in pain, but the four horsemen aren't galloping nearby. So take a pill, catch your breath and deal with it: the sun will come out, just don't bother trying to time it too carefully.
1005. </p>
1006. <p>
1007. Now, there's no shortage of reasons for gloom and doom: mega-ponzi schemes collapsing, banks and real estate combusting, the big 3 in various states of failure, yet <a href="http://www.autonews.com/article/20081224/ANA05/812249975/1078">BMW North America will raise list prices 0.7%</a>. Before the complete credit breakdown, real estate volume was actually rising in a lot of places (ergo: prices were aligning supply and demand). I was at a William-Sonoma store in Albuquerque the other day, the place was mobbed. My point is that while the economy is retrenching (or the country is <a href="http://www.nytimes.com/2008/12/24/opinion/24friedman.html">rebooting</a>), the detritus will be separated (Wilson's wheat from chaff) and data center space should be cheap and abundant. Everything seems fine to me. At least until the next bubble.
1008. </p>
1009. <p>
1010. For those of you observing that sort of thing, Merry Christmas!
1011. </p>
1012. <p><em>
1013. <a href="http://technorati.com/tag/colocation" rel="tag">colocation</a>  
1014. <a href="http://technorati.com/tag/data+centers" rel="tag">data centers</a>  
1015. <a href="http://technorati.com/tag/economy" rel="tag">economy</a>  
1016. </em></p>
1017.  
1018.  
1019. </description>          </item>
1020.    <item>
1021.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/another_hadoop_backended_database_cloudbase</guid>
1022.    <title>Another Hadoop-backended Database: CloudBase</title>
1023.    <dc:creator>spidaman</dc:creator>
1024.    <link>http://arachna.com/roller/spidaman/entry/another_hadoop_backended_database_cloudbase</link>
1025.        <pubDate>Tue, 23 Dec 2008 16:02:21 -0800</pubDate>
1026.    <category>Java</category>
1027.            <description>&lt;p&gt;
1028.  
1029. <a href="http://hadoop.apache.org/" title="Hadoop">
1030. <img src="http://www.arachna.com/images/hadoop_logo.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="300" height="71" /></a>
1031.  
1032. This post to one of the Hadoop mailing lists caught my eye, <a href="http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200812.mbox/%3Ce75c02ef0812221514y575264a7i63adef15d37ab2a2@mail.gmail.com%3E">Announcing CloudBase-1.1 release</a>. Wait, wasn't Cloudbase the embedded database company that IBM acquired several years back but ended up donating the product to the Apache Software Foundation as <a href="http://db.apache.org/derby/">Derby</a>? No, not that Cloudbase. This is apparently another project that aims to provide data warehousing on top of Hadoop.
1033. </p>
1034. <p>
1035. I've been watching the emergence of <a href="http://hadoop.apache.org/hbase/">HBase</a>, <a href="http://hypertable.org/">Hypertable</a> and most recently <a href="http://mail-archives.apache.org/mod_mbox/incubator-general/200812.mbox/browser">the proposed incubation of Facebook's Cassandra</a> with great interest. The first two are modeled from Google's <a href="http://labs.google.com/papers/bigtable.html">BigTable</a> but all are essentially horizontally scalable column oriented databases. The developers of these systems explicitly steer away having their technologies pegged as relational databases, with the refrain: "We don't do joins." What the <a href="http://cloudbase.sourceforge.net/">CloudBase</a> project aims to do is <em>not</em> model themselves on BigTable but to explicitly support joins between tables built on top of an HDFS cluster. It looks like they've posted extensive documentation and have released a JDBC driver, pretty cool! This is the most interesting database initiative I've seen since <a href="http://www.greenplum.com/resources/mapreduce/">GreenPlum announced their support for mapreduce</a>.
1036. </p>
1037. <p>
1038. Yes, as far as scale-out data analytics, we live in interesting times.
1039. </p>
1040. <p><em>
1041. <a href="http://technorati.com/tag/mapreduce" rel="tag">mapreduce</a>  
1042. <a href="http://technorati.com/tag/hadoop" rel="tag">hadoop</a>  
1043. <a href="http://technorati.com/tag/hbase" rel="tag">hbase</a>  
1044. <a href="http://technorati.com/tag/hypertable" rel="tag">hypertable</a>  
1045. <a href="http://technorati.com/tag/jdbc" rel="tag">jdbc</a>  
1046. <a href="http://technorati.com/tag/cloudbase" rel="tag">cloudbase</a>  
1047. <a href="http://technorati.com/tag/bigtable" rel="tag">bigtable</a>  
1048. <a href="http://technorati.com/tag/derby" rel="tag">derby</a>  
1049. <a href="http://technorati.com/tag/greenplum" rel="tag">greenplum</a>  
1050.  
1051. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1052.    <item>
1053.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/python_svn_bindings_trac_and</guid>
1054.    <title>Python SVN Bindings, Trac and mod_python</title>
1055.    <dc:creator>spidaman</dc:creator>
1056.    <link>http://arachna.com/roller/spidaman/entry/python_svn_bindings_trac_and</link>
1057.        <pubDate>Sun, 21 Dec 2008 14:55:24 -0800</pubDate>
1058.    <category>LAMP</category>
1059.            <description>&lt;p&gt;
1060. <a href="http://trac.edgewall.org/" title="Trac">
1061. <img src="http://www.arachna.com/images/trac_logo.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="214" height="61" /></a>
1062.  
1063. I have some code I want to noodle on outside of work. Since I'm on a holiday break, I'm doing a bit of that (yes, this <em>is</em> what I do for fun, so?). In the past, I had used my own private CVS server for those kinds of things but these days, I could just as well live without CVS. I decided to roll a subversion server into my Apache build (the latest Apache + other modules aren't in the yum repositories for my distro, so I roll my own). While I'm putting a subversion server up, why not <a href="http://trac.edgewall.org/">trac</a>, too? Heh, that's where things got stuck.
1064. </p>
1065. <p>
1066. When I installed the subversion dependencies (specifically, <a href="http://www.webdav.org/neon/">neon</a>), I just used vanilla build params. After installing subversion, I was surprised that Trac couldn't access it. It turns out that the litmus test was this:
1067. <pre>
1068. $ python
1069. Python ...
1070. >>> from svn import core
1071. </pre>
1072. ...it failed miserably. Various recompile efforts seemed to move the problem around. I saw a variety of the symptoms described in the Trac-Subversion integration docs <a href="http://trac.edgewall.org/wiki/TracSubversion#Troubleshooting">troubleshooting</a> section. The missing <code>gss_delete_sec_context</code> symbol error was apparently the telltale critical one, it originated from neon having been compiled without SSL support. The neon compile config that led to success was
1073. <pre>
1074. ./configure --enable-shared --enable-static --with-ssl=openssl
1075. </pre>
1076. Then the real key was to completely start over with the subversion compile, not just the <a href="http://www.swig.org/">swig</a> python bindings.
1077. <pre>
1078. make clean
1079. ./configure \
1080. --with-berkeley-db=/usr/local/BerkeleyDB.4.7 \
1081. --prefix=/usr/local \
1082. --with-apxs=/usr/local/httpd2.2.11/bin/apxs \
1083. --with-apr=/usr/local/apr \
1084. --with-apr-util=/usr/local/apr
1085. make
1086. make swig-py
1087. make check-swig-py
1088. make install
1089. make install-swig-py
1090. ldconfig
1091. </pre>
1092. Only then did the litmus test above pass. One of the things about this setup that is kind of a nuisance is that the python bindings didn't get installed into <code>site-packages</code>, therefore <a href="http://www.modpython.org/">mod_python</a> was quite unhappy. Also, trac seemed to want to put its eggs in the root directory. So the Apache server's <code>envvars</code> script has these variables exported to work around those issues
1093. <pre>
1094. PYTHONPATH=/usr/local/lib/svn-python
1095. LD_LIBRARY_PATH=/usr/local/lib/svn-python/libsvn
1096. PYTHON_EGG_CACHE=/data1/egg_cache
1097. </pre>
1098. The result (including the requisite <code>httpd.conf</code> tweaks) is a working subversion 1.54 and trac 0.11 setup. It was more fiddling for the evening than I'd hoped for and I'm not sure my foibles and remedies were optimal (clearly, I missed an RTFM somewhere) but I hope this resolution helps at least one reader.
1099. </p>
1100. <p>
1101. Happy Hannukah and winter solstice!
1102. </p>
1103. <p><em>
1104. <a href="http://technorati.com/tag/apache" rel="tag">apache</a>  
1105. <a href="http://technorati.com/tag/mod_python" rel="tag">mod_python</a>  
1106. <a href="http://technorati.com/tag/trac" rel="tag">trac</a>  
1107. <a href="http://technorati.com/tag/neon" rel="tag">neon</a>  
1108. <a href="http://technorati.com/tag/subversion" rel="tag">subversion</a>  
1109. <a href="http://technorati.com/tag/swig" rel="tag">swig</a>  
1110. <a href="http://technorati.com/tag/python" rel="tag">python</a>  
1111. </em></p>
1112. </description>          </item>
1113.    <item>
1114.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/world_leader_reflex_tests</guid>
1115.    <title>World Leader Reflex Tests</title>
1116.    <dc:creator>spidaman</dc:creator>
1117.    <link>http://arachna.com/roller/spidaman/entry/world_leader_reflex_tests</link>
1118.        <pubDate>Mon, 15 Dec 2008 21:35:52 -0800</pubDate>
1119.    <category>Musings</category>
1120.            <description>&lt;p&gt;
1121. <img src="http://www.arachna.com/images/bush_vs_shoe.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="124" height="83" />
1122. Will Chuck-The-Shoe-At-The-World-Leader be an Olympic sport in the years ahead? Since finishing with dinner this evening, I've found no less than three flash games and a compendium of animated GIF satires.
1123. <ul>
1124. <li><a href="http://kotaku.com/5110547/shoe-attack-on-president-already-turned-into-a-crappy-flash-game">Shoe Attack On President Already Turned Into A Crappy Flash Game</a> (Kotaku)</li>
1125. <li><a href="http://sockandawe.com/">Sock and Awe</a></li>
1126. <li><a href="http://www.kroma.no/2008/bushgame/">The Bush Game</a></li>
1127. <li><a href="http://www.boingboing.net/2008/12/15/iraq-shoe-tosser-guy.html">Iraq Shoe Tosser Guy: The Animated Gifs</a> (Boing Boing)</li>
1128. </ul>
1129. I bet Jon Stewart is pissed that this had to happen two days before his last show of the year. There's still a month left for more Dubya foibles, cut the vacation short Stewart! In the meantime,  keep it safe kids.
1130. </p>
1131. <p><em>
1132. <a href="http://technorati.com/tag/shoe+toss" rel="tag">shoe toss</a>  
1133. <a href="http://technorati.com/tag/bush" rel="tag">bush</a>  
1134. <a href="http://technorati.com/tag/humor" rel="tag">humor</a>  
1135. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1136.    <item>
1137.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/openedge_vs_net_neutrality_vs</guid>
1138.    <title>OpenEdge vs. Net Neutrality vs. CDN</title>
1139.    <dc:creator>spidaman</dc:creator>
1140.    <link>http://arachna.com/roller/spidaman/entry/openedge_vs_net_neutrality_vs</link>
1141.        <pubDate>Sun, 14 Dec 2008 22:52:25 -0800</pubDate>
1142.    <category>Musings</category>
1143.            <description>&lt;p&gt;
1144. <a href="http://online.wsj.com" title="Wall Street Journal">
1145. <img src="http://www.arachna.com/images/north_american_internet_traffic_chart.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="184" height="276" /></a>
1146. The Wall Street Journal reported today that <a href="http://online.wsj.com/article/SB122929270127905065.html">Google Wants Its Own Fast Track on the Web</a>, describing it as an example of the decline of support for <a href="http://en.wikipedia.org/wiki/Network_neutrality">net neutrality</a> amongst the The Powers That Be (the usual suspects: Google, Yahoo, Microsoft, Amazon). Plenty of deals have been getting struck anyway between TPTB and data carriers (most prominently AT&T + Yahoo DSL) but outright transit preference doesn't seem to be an issue here. What Google appears to be getting into, called <em>OpenEdge</em>, sounds like an arrangement that amounts to co-locating their gear in the major carrier's datacenters. This would move serving capacity closer to the end-users of their services and thereby accelerate the user experience. Since it doesn't concern transit per se, this actually doesn't sound like a net neutrality issue at all, it sounds like another form of datacenter dispersion.
1147. </p>
1148. <p>
1149. So what exactly is the big deal? All of the TPTB and loads of other online services have <a href="http://en.wikipedia.org/wiki/Content_delivery_network">content delivery network</a> (CDN) deals. Yahoo, Amazon, Facebook... they all operate or partner with a CDN in some shape or form (full disclosure: I've been working on a CDN evaluation for <a href="http://technorati.com/">Technorati</a>). With a CDN, publishers pay specifically to have their content cached at points-of-presence (PoP) around the intertubes that, through some DNS and routing magic, enables web content to get to end-users more quickly. The next step beyond a CDN is to put equipment in the carrier's datacenter. Here's what WSJ said
1150. <blockquote cite="http://online.wsj.com/article/SB122929270127905065.html">
1151. Google's proposed arrangement with network providers, internally called OpenEdge, would place Google servers directly within the network of the service providers, according to documents reviewed by the Journal. The setup would accelerate Google's service for users. Google has asked the providers it has approached not to talk about the idea, according to people familiar with the plans.
1152. <br/>
1153. Asked about OpenEdge, Google said only that other companies such as Yahoo and Microsoft could strike similar deals if they desired. But Google's move, if successful, would give it an advantage available to very few.
1154. </blockquote>
1155. It seems perfectly logical, actually.
1156. </p>
1157. <p>
1158. Nonetheless, I am concerned about wavering support for net neutrality. Lawrence Lessig, fresh off of his <a href="http://joi.ito.com/weblog/2008/12/12/big-news-from-l.html">Big News</a> post concerning setting up shop at Harvard Law School, is quoted as saying
1159. <blockquote  cite="http://online.wsj.com/article/SB122929270127905065.html">
1160. There are good reasons to be able to prioritize traffic. If everyone had to pay the same rates for postal service, than you wouldn't be able to differentiate between sending a greeting card to your grandma versus sending an overnight letter to your lawyer.
1161. </blockquote>
1162. But the counter argument says that there's a big difference.  Grandma isn't trying to compete with your attorney (at least, not usually). If the big guys are paying more to be faster, who will be able afford to challenge them? The intertubularly rich will get richer, the poor will be stay poor. The TPTB will ensconce themselves as dynastic media walking on paths paved with gold while all of us commoners walk in the gutter.
1163. </p>
1164. <p>
1165. The dumb pipes should stay dumb. If an internet service wants to operate out of multiple datacenters, lease dedicated pipes to accelerate their inter-datacenter data distribution and peer with the carrier's PoPs proximate to their datacenters, mazel tov. This can be augmented with CDNs. It can even be taken to the next step by directly installing the carrier's datacenters. But at the network exchanges and pipes connecting them, everyone's packets should remain equal.
1166. </p>
1167. <p>
1168. <strong>UPDATE</strong>
1169. <a href="http://gigaom.com/2008/12/14/google-turns-its-back-on-network-neutrality/">GigaOM</a> posted about a clarification from Google which says that the WSJ was "confused". The hubbub in that article really was misplaced, it's a CDN deal.
1170. </p>
1171. <p><em><a href="http://technorati.com/tag/google" rel="tag">google</a>  
1172. <a href="http://technorati.com/tag/cdn" rel="tag">cdn</a>  
1173. <a href="http://technorati.com/tag/content+delivery+network" rel="tag">content delivery network</a>  
1174. <a href="http://technorati.com/tag/net+neutrality" rel="tag">net neutrality</a>  
1175. <a href="http://technorati.com/tag/wall+street+journal" rel="tag">wall street journal</a>  
1176. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1177.    <item>
1178.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/cloud_hype_an_amazon_web</guid>
1179.    <title>Cloud Hype, An Amazon Web Services Post-Mortem</title>
1180.    <dc:creator>spidaman</dc:creator>
1181.    <link>http://arachna.com/roller/spidaman/entry/cloud_hype_an_amazon_web</link>
1182.        <pubDate>Wed, 10 Dec 2008 23:53:19 -0800</pubDate>
1183.    <category>Musings</category>
1184.            <description>&lt;p&gt;
1185. <a href="http://aws.amazon.com/" title="Amazon Web Services">
1186. <img src="http://www.arachna.com/images/aws_logo.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="164" height="60" /></a>
1187. In the last few years, the scope of Amazon Web Services (AWS) has broadened to cover a range of infrastructure capabilities and has emerged as a game changer.   The hype around AWS isn't all wrong, a whole ecosystem of tools and services has developed around AWS that makes the offering compelling. However, the hype isn't all right either.  At Technorati, we used AWS this year to develop and put in production a new crawler and a system that produces the web page screenshot thumbnails now seen on search result pages. But now that that chapter is coming to a close, it's time to retrospect.
1188. </p>
1189. <p>
1190. <img src="http://www.arachna.com/images/variable_capacity.png" style="float:left; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="357" height="106" />
1191. There's a prevailing myth that using the elasticity of EC2 makes it cheaper to operate than fixed assets. The theory is that by shutting down unneeded infrastructure during the lulls, you're saving money. In a purely fixed infrastructure model, Technorati's data aquisition systems must be provisioned for their maximum utilization capacity threshold. When utilization ebbs, a lot of that infrastructure sits relatively idle. That much is true but the reality is that flexible capacity is only saving money relative to the minimum requirements. So the theory only holds if your variability is high compared to your minimum. That is, if the difference between your minimum and maximum capacity is large or you're not operating a 365/7/24 system but episodically using a lot of infrastructure and then shutting it down. Neither is true for us. The normal operating mode of Technorati's data acquisition systems follows the ebb and flow of the blogosphere, which varies a lot but is always on. The sketch to the left shows the minimum capacity and the variable capacity distinguished.
1192. </p>
1193. <p>
1194. In response to some of the fallacies posted on an O'Reilly blog the other day (by George Reese), <a href="http://broadcast.oreilly.com/2008/12/why-i-dont-like-cloud-auto-scaling.html">On Why I Don't Like Auto-Scaling in the Cloud</a>, Don MacAskill from SmugMug wrote a really great post yesterday about his SkyNet system, <a href="http://blogs.smugmug.com/don/2008/12/09/on-why-auto-scaling-in-the-cloud-rocks/">On Why Auto-Scaling in the Cloud Rocks</a>. Don also emphasizes SmugMugs modest requirements for operations staff. In an application with sufficient simplicity and automation around it, it's easy to imagine a 365/7/24 service having meager ops burdens. I think we should surmise that the cost of operating SmugMug with autonomic de/provisioning works because it fits their operating model. I understand Reese's concern, that folks may not do the hard work of really understanding their capacity requirements if they're too coddled by automation. However, that concern comes off as a shill for John Allspaw's capacity planning book (which I'm sure is great, can't wait to read it). Bryan Duxbury from RapLeaf describes their use of AWS and how the numbers work out in his post, <a href="http://blog.rapleaf.com/dev/?p=38">Rent or Own: Amazon EC2 vs. Colocation Comparison for Hadoop Clusters</a>. Since the target is to serve a Hadoop infrastructure,  AWS must get a thumbs down in their case. Hadoop's performance is impaired by poor rack locality and the latencies of Amazon's I/O systems clearly drags it down. If you're going to be running Hadoop on a continuous basis, use your own racks, with your own switches and your own disk spindles.
1195. </p>
1196. <p>
1197. At Technorati, we're migrating the crawl infrastructure from AWS to our colo. While I love the flexibility that AWS provides and it's been great using it as a platform to  ramp up on , the bottom line is that Technorati has a pre-existing investment in machines, racks and colo infrastructure. As much as I'd like our colo infrastructure to operate with lower labor and communication overhead, running on AWS has amounted to additional costs that we must curtail.
1198. </p>
1199. <p>
1200. Cloud computing (or <a href="http://gigaom.com/2008/02/28/how-cloud-utility-computing-are-different/">utility computing</a> or <a href="http://www.arachna.com/roller/spidaman/entry/can_we_just_call_it">flex computing</a> or whatever its called) <em>is</em> a game changer. So when do I recommend you use AWS? Ideally: anytime. If your application is architected to expand and contract its footprint with the demands put upon it, provision your minimum capcacity requirements in your colo and use AWS to "burst" when your load demands it. Another case where using AWS is a big win is for a total green field. If you don't have a colo, are still determining the operating charactersics of your applications and need machines provisioned, AWS is an incredible resource. However, I think the flexibility vs. economy imperatives will always lead you to optimize your costs by provisioning your minimum capacity in infrastructure that you own and operate.
1201. </p>
1202. <p>
1203. There's also another option: instead of buying and operating your own machines and racks, you may be able to optimize costs by renting machines provisioned to your specs in a contract from the services that have established themselves in that market (Rackspace, Server Beach, ServePath, LayeredTech, etc). Ultimately, I'm looking forward to the emergence of a compute market place where the decisions to incur capital expense, rent by the hour or rent under a contract will be easier to traverse.
1204. </p>
1205. <p><em>
1206. <a href="http://technorati.com/tag/amazon+web+services" rel="tag">amazon web services</a>  
1207. <a href="http://technorati.com/tag/aws" rel="tag">aws</a>  
1208. <a href="http://technorati.com/tag/cloud+computing" rel="tag">cloud computing</a>  
1209. <a href="http://technorati.com/tag/rapleaf" rel="tag">rapleaf</a>  
1210. <a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>  
1211. <a href="http://technorati.com/tag/smugmug" rel="tag">smugmug</a>  
1212. <a href="http://technorati.com/tag/hadoop" rel="tag">hadoop</a>  
1213. <a href="http://technorati.com/tag/oreilly" rel="tag">oreilly</a>  
1214. <a href="http://technorati.com/tag/data+centers" rel="tag">data centers</a>  
1215. <a href="http://technorati.com/tag/capacity+planning" rel="tag">capacity planning</a>  
1216. </em></p>
1217. </description>          </item>
1218.    <item>
1219.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/the_solar_decade</guid>
1220.    <title>The Solar Decade</title>
1221.    <dc:creator>spidaman</dc:creator>
1222.    <link>http://arachna.com/roller/spidaman/entry/the_solar_decade</link>
1223.        <pubDate>Tue, 9 Dec 2008 23:55:24 -0800</pubDate>
1224.    <category>Musings</category>
1225.            <description>&lt;SCRIPT TYPE=&quot;text/javascript&quot;&gt;
1226. <!--
1227. function popup(mylink, windowname)
1228. {
1229. if (! window.focus)return true;
1230. var href;
1231. if (typeof(mylink) == 'string')
1232.   href=mylink;
1233. else
1234.   href=mylink.href;
1235. window.open(href, windowname, 'width=720,height=630,scrollbars=yes');
1236. return false;
1237. }
1238. //-->
1239. </SCRIPT>
1240. <p>
1241. <a href="http://www.arachna.com/images/solar_the_graph.gif" onClick="return popup(this, 'The Graph')" title="Fast Company"">
1242. <img src="http://www.arachna.com/images/solar_the_graph_thumb.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="227" height="198" /></a>
1243. Ten years ago, you might have been advised that solar energy, while sounding nice, was a bad investment. The installations were failure prone and not cost effective. I don't know if I bought that then, I know of solar panels in San Francisco installed in the 80's that paid for themselves, just slowly. But what we're seeing isn't your father's solar panel. From <a href="http://blog.makezine.com/archive/2007/06/google_solar_panel_projec.html">Google's solar panels</a> to residential rooftops, it seems pretty clear that the <a href="http://www.smartmoney.com/spending/for-the-home/economics-of-solar-power-are-looking-brighter-21444/">
1244. Economics of Solar Power Are Looking Brighter</a>. Fast Company is running an article <a href="http://www.fastcompany.com/magazine/131/solar-goes-supernova.html">The Solar Industry Gains Ground</a> that sounds a chord that we're hearing a lot of. Solar energy is getting more and more cost effective. What's projected is that the cost of solar power may share up-and-to-the-right properties of <a href="http://en.wikipedia.org/wiki/Moore's_law">Moore's Law</a>. The fabs that make the silicon enabling you to read this may also enable an energy giant leap forward. The Germans have their own "Solar Valley" and their industry projection <a href="http://www.arachna.com/images/solar_the_graph.gif" onClick="return popup(this, 'The Graph')">graph</A> appears pretty Moorish (look at the large yellow area).
1245. </p>
1246. <P>
1247. The big lift off is 10 years away but the investment that has been made in the area and the advances being made seem to put the benefits close at hand. But the big win, when dependence on fossil fuels are on a clear decline, is at leat 10 and 20 years out. But I think it can happen, I think the solar decade is coming. It should be the coming decade. However, it will require an Apollo-mission like focus from the Obama administration to succeed. And I hope we can make it a reality.
1248. </p>
1249. <p><em>
1250. <a href="http://technorati.com/tag/solar+energy" rel="tag">solar energy</a>  
1251. <a href="http://technorati.com/tag/obama" rel="tag">obama</a>  
1252. </em></p>
1253. </description>          </item>
1254.    <item>
1255.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/good_bye_perl</guid>
1256.    <title>Good Bye, Perl</title>
1257.    <dc:creator>spidaman</dc:creator>
1258.    <link>http://arachna.com/roller/spidaman/entry/good_bye_perl</link>
1259.        <pubDate>Mon, 8 Dec 2008 00:01:24 -0800</pubDate>
1260.    <category>LAMP</category>
1261.            <description>&lt;p&gt;
1262. <a href="http://www.perl.org/" title="Perl">
1263. <img src="http://www.arachna.com/images/perl-camel-logo.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="72" height="81" /></a>
1264. The other day, I was patching some Perl code. There I was, in the zone, code streaming off of my finger tips. But wait, I was writing Python in the middle of a Perl subroutine. Um. I found the bare word and missing semi-colon errors invoking <code>perl -cw dingleberry.pl</code> amusing. How did <em>that</em> happen?
1265. </p>
1266. <p>
1267. Truth is, I find myself rarely using Perl anymore. I spent years building applications with Perl. Having made extensive use of mod_perl APIs and various CPAN modules, studied the Talmudic wisdom of Damian Conway, struggled with the double-edged sword of <a href="http://en.wikipedia.org/wiki/There_is_more_than_one_way_to_do_it">TMTOWTDI</a> and rolled my eyes at the Perl haters for their failure to appreciate the strange poetry that is Perl it'd seem like a safe bet that Perl would remain on my top shelf. A lot of the complaints of Perl haters seem superficial ("ewe, all of those punctuation characters", shaddup). Yet, Perl has been long in the tooth, for a long time. I recall 5 years ago thinking that Perl 6 wasn't <em>too</em> far away (after all, O'Reilly published <a href="http://www.amazon.com/Perl-6-Essentials-Allison-Randal/dp/0596004990/">Perl 6 Essentials</a> in June 2003). I'm sorry, my dear Perl friends, insistence that <a href="http://london.pm.org/pipermail/london.pm/Week-of-Mon-20081208/015866.html">Perl is Alive</a> rings hollow, now.
1268. </p>
1269. <p>
1270. I still find it heartening to hear of people doing cool things with Perl. David McLaughlin's uplifting <a href="http://www.dmclaughlin.com/2008/12/07/how-i-learnt-to-love-perl/">How I learnt to love Perl</a>, waxing on about <a href="http://www.iinteractive.com/moose/">Moose</a> and other "modern" Perl frameworks (but come on dude, everyone knows that PHP, sucks, heh). Brad Fitzpatrick released <a href="http://brad.livejournal.com/2388824.html">Perl for Google App Engine</a> today. But Perl, I'm sorry. It's just too little, too late.
1271. </p>
1272. <p>
1273. I'm just weary of the difficulties achieving team adherence to disciplined coding practices (or even appreciate why they're especially necessary in the TMTOWTDI world of Perl). The reliance on Wizardry is high with Perl; the path from novice to master requires grasping a wide range of arcana. Is it too much to ask for less magic in favor of easier developer ramp up? Perl's flexibility and expressiveness, it's high virtues, also comprises the generous reel of rope that programmers routinely hang themselves with. On top of idiomatic obscurities are the traps people fall into with dynamic typing and errors that only make themselves evident at runtime. Good testing practices are usually the anecdote to the woes of dynamic typing and and yet writing a good test harness for a Perl project is often a lot of work compared to the amount of work required to write the application code.
1274. </p>
1275. <p>
1276. I well understand the security that programmers feel using <a href="http://teddziuba.com/2008/12/python-makes-me-nervous.html">static typing</a> but I'm not saying the static typing is the cure to any ills. The compiler is the most basic test that your code can be understood and it gives your IDE a lotta help. That's great but static typing is also an anchor dragging on your time. From what I can tell, Java is the new C/C++ and Jython, JRuby. Groovy and friends (Scala, Clojure, etc) are the ways that people program the JVM with higher productivity. I'm not saying fornever to Perl (that's a long time). But I am saying Hasta La Vista, for now. I've been quite productive lately with Python (I know Perl friends, heresy!) and plan on pushing ahead with that, as well as with Java and other JVM languages. And where necessary, using Thrift to enable the pieces to work together. Python is certainly not perfect, it's quirks are many, too. But I've seen recent success with collaborative software development with Python that would have been difficult to accomplish with Perl. I'm not trying to stoke any language war at all, I'm just reflecting on how I've drifted from Perl. Amongst people I know and things I've read elsewhere, <a href="http://journal.dedasys.com/2008/12/07/python-surpasses-perl">I'm not the only one</a>. Don't fret, Perl, I'm sure I'll see you around.
1277. </p>
1278. <p><em>
1279. <a href="http://technorati.com/tag/perl" rel="tag">perl</a>  
1280. <a href="http://technorati.com/tag/python" rel="tag">python</a>  
1281. <a href="http://technorati.com/tag/java" rel="tag">java</a>  
1282. <a href="http://technorati.com/tag/programming" rel="tag">programming</a>  
1283. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1284.    <item>
1285.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/mysql_vs_postgresql_again</guid>
1286.    <title>MySQL vs. PostgreSQL (again)</title>
1287.    <dc:creator>spidaman</dc:creator>
1288.    <link>http://arachna.com/roller/spidaman/entry/mysql_vs_postgresql_again</link>
1289.        <pubDate>Sat, 6 Dec 2008 12:08:20 -0800</pubDate>
1290.    <category>LAMP</category>
1291.            <description>&lt;p&gt;
1292. <a href="http://www.postgresql.org/" title="PostgreSQL">
1293. <img src="http://www.arachna.com/images/postgresql_logo.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="175" height="68" /></a>
1294.  
1295. <a href="http://www.mysql.com/" title="MySQL">
1296. <img src="http://arachna.com/images/mysql_logo.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="114" height="68" /></a>
1297.  
1298. If <a href="http://feedblog.org/">Kevin Burton</a> wanted to draw attention to <a href="http://feedblog.org/2008/12/04/destroying-mysql/">MySQL's inattention to scalability concerns</a>, it looks like he's <a href="http://www.reddit.com/r/programming/comments/7hiv5/destroying_mysql/">succeeded</a> (126 comments on <a href="http://www.reddit.com/">Reddit</a> in the last day and climbing). I totally understand why he feels that the <a href="http://www.mysql.com/">MySQL</a> folks need to be provoked into action. I'll confess to having serious "MySQL fatigue" after years of struggling with InnoDB quirks (we use MySQL extensively at <a href="http://technorati.com/">Technorati</a>), stupid query plans and difficult to predict performance inflection points (there's a calculus behind table row count, row width, number of indices, update rate and query rate -- but AFAIK nobody has a reliable formula to predict response times against those variables). Frankly, I was really surprised when Sun acquired MySQL (for such a hefty sum, too), I was expecting them to build up a <a href="http://www.postgresql.org/">PostgreSQL</a>-based platform by rolling up acquisitions of <a href="http://www.greenplum.com/">Greenplum</a> and <a href="http://truviso.com/">Truviso</a>.
1299. </p>
1300. <p>
1301. Kevin is totally correct that to find solid innovation with MySQL, don't look to the MySQL corporation. Instead, the consultants and third party shops specializing in MySQL are where the action's at (<a href="http://blog.palominodb.com/">Palomino DB</a>, <a href="http://www.percona.com/">Percona</a>, <a href="https://launchpad.net/drizzle">Drizzle</a>, etc). It's kinda sad, both Sun and MySQL have at various times been home to hot-beds of innovation. Sun has <a href="http://blogs.sun.com/downstream/">great people</a>  groundbreaking with cloud computing, impressive CPU performance per watt improvements and the Java ecosystem. But as far as MySQL goes, look to the outside practitioners.
1302. </p>
1303. <p>
1304. Kevin's post update cites the pluggable back-ends that MySQL supports as a feature but I'm not so sure. I don't have any evidence of this but my intuition is that it's exactly this feature that makes the overall stability and performance such a crap-shoot (or sometimes, just outright crap). I'm working on a personal project that uses <a href="http://postgis.refractions.net/">PostGIS</a> (PostgreSQL + GIS), nothing is live yet so I haven't had to scale it.  But I have a good deal of confidence in the platform.  <a href="https://developer.skype.com/SkypeGarage/DbProjects/SkypePostgresqlWhitepaper">Skype</a> and <a href="http://pandora.com/">Pandora</a> look like good case studies. The PostgreSQL people have been focused on MVCC concurrency, procedure languages, UDFs and data integrity semantics for years. In those realms, the MySQL people are Johnny-come-slowlies (and buggily). On the other hand, if you want the append-only characteristics of logging to a database, MyISAM and merge tables have performance properties that PostgreSQL just can't match.
1305. </p>
1306. <p>
1307. Maybe <a href="http://en.wikipedia.org/wiki/David_Duffield">David Duffield</a> will look beyond <a href="http://workday.com/">enterprise app services</a> and acquire, roll-up and market the PostgreSQL platforms that Sun didn't. Combining big data and event data with Greenplum and Truviso as a way to blow some smoke in Larry Ellison's eye, <em>that</em> would be funny (and smart).
1308. </p>
1309. <p><em>
1310. <a href="http://technorati.com/tag/postgresql" rel="tag">postgresql</a>  
1311. <a href="http://technorati.com/tag/mysql" rel="tag">mysql</a>  
1312. <a href="http://technorati.com/tag/greenplum" rel="tag">greenplum</a>  
1313. <a href="http://technorati.com/tag/truviso" rel="tag">truviso</a>  
1314. <a href="http://technorati.com/tag/sun" rel="tag">sun</a>  
1315. <a href="http://technorati.com/tag/workday" rel="tag">workday</a>  
1316. <a href="http://technorati.com/tag/postgis" rel="tag">postgis</a>  
1317. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1318.    <item>
1319.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/it_s_only_the_biggest</guid>
1320.    <title>It&apos;s Only The Biggest Country In the World</title>
1321.    <dc:creator>spidaman</dc:creator>
1322.    <link>http://arachna.com/roller/spidaman/entry/it_s_only_the_biggest</link>
1323.        <pubDate>Fri, 5 Dec 2008 18:12:19 -0800</pubDate>
1324.    <category>Musings</category>
1325.            <description>&lt;p&gt;
1326. <a href="http://en.wikipedia.org/wiki/China" title="China">
1327. <img src="http://www.arachna.com/images/chinese_flag.gif" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="124" height="83" /></a>
1328. Should the confirmed reports that <a href="http://www.ifgogo.com/251/technorati-is-blocked-in-china/">Technorati is banned in China</a> be worn as a badge honor? I understand the Chinese authorities value stability but these kinds of things, treating billions of people like little children that need to be sheltered, will ultimately destabilize them.
1329. </p>
1330. <P>
1331. We've waited 18 years for <a href="http://www.amazon.com/exec/obidos/ASIN/B001L4MHBE/wwwarachnacom-20">Chinese Democracy</a>, isn't that long enough? (sorry, couldn't resist the joke)
1332. </p>
1333. <p>
1334. Best wishes to the Chinese people. At least most of you.
1335. </p>
1336. <p><em>
1337. <a href="http://technorati.com/tag/chinese+democracy" rel="tag">chinese democracy</a>  
1338. <a href="http://technorati.com/tag/china" rel="tag">china</a>  
1339. <a href="http://technorati.com/tag/censorship" rel="tag">censorship</a>  
1340. <a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>  
1341. <a href="http://technorati.com/tag/guns-n-roses" rel="tag">guns n roses</a>  
1342. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1343.    <item>
1344.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/technorati_releases</guid>
1345.    <title>Technorati Releases Fixes Some UI Peeves</title>
1346.    <dc:creator>spidaman</dc:creator>
1347.    <link>http://arachna.com/roller/spidaman/entry/technorati_releases</link>
1348.        <pubDate>Thu, 4 Dec 2008 18:43:39 -0800</pubDate>
1349.    <category>Musings</category>
1350.            <description>&lt;p&gt;
1351. <a href="http://technorati.com/" title="Technorati">
1352. <img src="http://www.arachna.com/images/technorati_logo.png" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="110" height="15" /></a>
1353. In general, I regard successful user interfaces as the ones that provide the least amount of hunting and astonishment. Noone is delighted when the things they're looking for aren't obvious, the data displayed requires lots of explanation and the paths through an application are click-heavy. In this regard, Technorati was long saddled with a user interface that I regarded as delightless. However, I see that changing now and I'm delighted to see that!
1354. <p>
1355. <a href="http://technorati.com/weblog/2008/12/457.html">Technorati's front end was released today</a> with a handful of significant improvements. One is a long standing peeve of mine: the tag pages we're conflated with keyword search. That meant that if your post was about the president-elect and you tagged it "obama", your expectation that the the posts aggregated at <a href="http://technorati.com/tag/obama">http://technorati.com/tag/obama</a> would also be tagged "obama" would be disappointed; there would also be a bunch of keyword matches mixed in. That came out of last year's attempt to "simplify" the experience by making keyword search and tag browsing the same thing; which was, in all honesty, a George Bush level failure. Sure there are folks who don't know, "What's this 'tag' thing you're talking about?" But for the folks who <em>do</em> know what the difference is between browsing blog posts grouped by tags and keyword search results, the mix wasn't received as a simplification but as a software defect.
1356. <blockquote>
1357. I tagged my post "obama" but all of these other posts aren't tagged "obama", what's going on?
1358. </blockquote>
1359. I'm glad we've gone back to keeping search and tags distinct.
1360. </p>
1361. <p>
1362. The other failed aspect of the prior design was the demotion of the search box. The form input to type in your search was sized down and moved to the right, as if it were a "site search" feature. Yes, we'd like folks to explore our discovery features but the navigation for those features weren't great and the de-emphasis on search was again a source of more puzzlement than anything. The release today puts the search box back where it should be: bigger and right in the middle of the of the top third of the page, yay!
1363. </p>
1364. <p>
1365. Oh, and earlier today Technorati Media released its Engage platform to <a href="https://beta.technoratimedia.com/">beta</a>. This is a major step in opening up the ad market place for the blogosphere.
1366. </p>
1367. <p>
1368. So far, the feedback I've seen on these releases have been thumbs-up. Check 'em out, there's some more goodies in the works but these things only get better with your feedback. And yes, we know there's still more to do, I'm certainly busy with the backend stuff with our cloud platform, ping systems and crawlers (but did you notice the screenshot thumbnails on the search result and tag pages? I need to shake out the latencies producing and refreshing those). Kudos to Dave White, the front end team and the ad platform team for getting these releases out. Onward and upward!
1369. </p>
1370. <p><em><a href="http://technorati.com/tag/technorati" rel="tag">technorati</a>  
1371. <a href="http://technorati.com/tag/folksonomy" rel="tag">folksonomy</a>  
1372. <a href="http://technorati.com/tag/tagging" rel="tag">tagging</a>  
1373. <a href="http://technorati.com/tag/user+interface" rel="tag">user interface</a>  
1374. <a href="http://technorati.com/tag/blogging" rel="tag">blogging</a>  
1375. <a href="http://technorati.com/tag/ad+platform" rel="tag">ad platform</a>  
1376. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1377.    <item>
1378.    <guid isPermaLink="true">http://arachna.com/roller/spidaman/entry/can_we_just_call_it</guid>
1379.    <title>Can We Just Call It &quot;Flex Computing&quot;?</title>
1380.    <dc:creator>spidaman</dc:creator>
1381.    <link>http://arachna.com/roller/spidaman/entry/can_we_just_call_it</link>
1382.        <pubDate>Wed, 3 Dec 2008 22:44:46 -0800</pubDate>
1383.    <category>Musings</category>
1384.            <description>&lt;p&gt;
1385. <a href="http://en.wikipedia.org/wiki/Cloud_computing" title="Cloud?">
1386. <img src="http://www.arachna.com/images/question_cloud.jpg" style="float:right; margin:5px 5px 5px 5px;cursor:pointer; cursor:hand;" border="0" width="80" height="127" /></a>
1387. The moniker "cloud computing" has been overloaded to mean to so many things, it's beginning to mean nothing. When someone refers to it generically, you have to ask them to dismbiguate; which of these are they referring to?
1388. <ol>
1389. <li>IT infrastructure offered as a services</li>
1390. <li>Hosted application functionality</li>
1391. <li>A virtualized server deployment</li>
1392. </ol>
1393. InfoWorld wrote <a href="http://www.infoworld.com/article/08/04/07/15FE-cloud-computing-reality_1.html">What cloud computing really means</a> last spring to help clarify the distinctions but still, I often have to stop folks when the use the C-word just make sure we're talking about the same things.
1394. </p>
1395. <p>
1396. Examples of the first definition are services like <a href="http://aws.amazon.com/">Amazon Web Services</a> (AWS) or <a href="http://gogrid.com/">GoGrid</a>. They provide metered virtual machines, you pay for what you use and have full access (root) on the machines while you use them. Additional goodies such as load balanced clusters, storage facilities and so forth are part of the deal too. Capacity can be scaled up or down on demand and typically, in very rapid fashion. When <a href="http://www.infoworld.com/infoworld/article/08/07/21/30TC-cloud-reviews_8.html">Peter Wayner reviewed these guys last summer in InfoWorld</a>, he was enamored with the GUI front ends. Call me old fashioned (or a dyed in the wool geek) but unless they're really saving me a lot of time, I have an aversion to the slick GUI's. For his part, Wayner complained about the AWS command line utilities. Actually, when I use AWS, I use a GUI for an overview of running instances, it's a Firefox plugin (<a href="http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609">Elasticfox</a>) but what I <em>really</em> like about AWS is programmatic access. Integrating application deployment with command and control functionality is very powerful, my tool of choice is <a href="http://code.google.com/p/boto/">boto</a>, a Python API for AWS.
1397. </p>
1398. <p>
1399. The second definition refers to hosted application functionality, in years gone by they were referred to as <em>Application Service Providers</em> (ASP). The more modern label is <em>Software as a Service</em> (SaaS). However, these services have to provide more than a console for functionality, they have to provide web service API's that enable them to be integrated into other applications. SalesForce.com was an early leader in this space (remember the red cross-outs, "No Software"), their example and the proliferation of RSS is really want inspired the proliferation of <a href="http://www.programmableweb.com/">APIs and mashups</a> we see today.
1400. </p>
1401. <p>
1402. The last definition refers to VMWare, Xen and so forth. By themselves, those aren't really cloud computing in my book. However, you can use them to create your own "private cloud" with tools like <a href="http://www.enomaly.com/">Enomaly</a> and <a href="http://eucalyptus.cs.ucsb.edu/">Eucalyptus</a>. This is an area of great interest to me.
1403. </p>
1404. <p>
1405. In his review, Wayner pointed out how very different all of the services are. I don't know why he included Google's App Engine at all in his write-up. Don't misunderstand, GAE is a great service but it more closely resembles an application container than infrastructure services.
1406. </p>
1407. <p>
1408. I'm imagining IT infrastructure management interfaces coalescing around standards (de-facto ones, not ones fashioned out of IETF meetings). Eucalyptus is a good discussion point. Eucalyptus provides an EC2 "work-alike" interface on top of a Xen virtual server platform. So picture this: if the Rackspaces, ServePaths, Server Beaches and LayeredTechs of this world were to provide a compatible interfaces built on top of Eucalyptus, buying compute power by the hour would become more like buying gasoline. There may be pros and cons to this station or that but fundamentally, if you don't like the pumps at one gas station or the prices are too high, you can go to the gas station across the street. Given compatible interfaces, management of the infrastructure, be it with boto, Elasticfox or using services such as <a href="http://rightscale.com/">RightScale</a> can be as dynamic as the server deployments in those clouds. Such a compute market place would unleash new rounds of innovation as it eases starting up and scaling online services.
1409. </p>
1410. <p>
1411. The Eucalyptus folks will be the first to fess up that their project is more academic in nature that industrial strength. However, it is the harbinger of AWS as a standard. Yes, I'm referring to AWS as a standard because of the level of adoption its enjoyed, the comprehensive set of APIs it provides and the rich ecosystem around it. What I foresee is that the first vendor to embrace and commoditize a standard interface for infrastructure management changes the game. The game becomes one of a meta-cloud because computing capacity will be truly fluid, flexibly shrinking and growing with hosted clouds, private clouds and migrating between clouds.
1412. </p>
1413. <p><em>
1414. <a href="http://technorati.com/tag/cloud+computing" rel="tag">cloud computing</a>  
1415. <a href="http://technorati.com/tag/saas" rel="tag">saas</a>  
1416. <a href="http://technorati.com/tag/virtualization" rel="tag">virtualization</a>  
1417. <a href="http://technorati.com/tag/aws" rel="tag">aws</a>  
1418. <a href="http://technorati.com/tag/xen" rel="tag">xen</a>  
1419. &lt;/em&gt;&lt;/p&gt;</description>          </item>
1420.  </channel>
1421. </rss>

If you would like to create a banner that links to this page (i.e. this validation result), do the following:

1. Download the "valid RSS" banner.

2. Upload the image to your own server. (This step is important. Please do not link directly to the image on this server.)

3. Add this HTML to your page (change the image src attribute if necessary):

   <a href="http://www.feedvalidator.org/check.cgi?url=http%3A//www.arachna.com/roller/rss/spidaman"><img src="valid-rss-rogers.png" alt="[Valid RSS]" title="Validate my RSS feed" /></a>

If you would like to create a text link instead, here is the URL you can use:

http://www.feedvalidator.org/check.cgi?url=http%3A//www.arachna.com/roller/rss/spidaman

Home · About · News · Docs · Terms