Recently in Projects Category

amistad.jpg
Here's a post that will be of little interest to my normal readers but that may be helpful to Googlers. If this helps you, please drop a comment letting me know. I need encouragement to go so far off topic from my normal posts.

Several years ago I bought the history book on which the movie Amistad had been based. The Amistad was a ship carrying slaves to the Americans, and its captives revolted. The movie, which I didn't see, was apparently exciting enough, but the book was tedious. I wanted to never revisit it or anything like it again. But alas, I've encountered what could be called a slave revolt.

MySQL has a strange behavior on slaves with the CHANGE MASTER command that cost me a few hours of sleep. Sometimes when values are set with the command, those values merge into the master.info file. However in other cases after using the command, the values in master.info are lost. A sequence of commands that seemed reasonable to me left me without the proper master bin-log and offset log position, and this caused my slave to get errors like 'Duplicate entry for key 1.'

Here's how I discovered this behavior:

First, I created a dump using the syntax that places within the dump an update statement to set the master's position:

mysqldump --all-databases --master-data=1 --add-locks -u myuser -p > full.db.`date +"%F"`.dmp

Afterward, I can check my dump and find that indeed, it provides the master's bin-log and position:

CHANGE MASTER TO MASTER_LOG_FILE='bin-log.000494', MASTER_LOG_POS=169;

I then bring the dump to my slave server. If I first import the dump and rely on its values to set the master's position, I'll get errors when replication begins. The errors are caused because the replication picks up at the oldest bin-log instead of the right one. The errors, found after running "show slave status\G;" are like this:

Last_Error: Error 'Duplicate entry '3363837' for key 1' on query. Default database: 'myapp'. Query: 'INSERT INTO mytable (
                                    blah,
                                    blah2,
                                    blah3
                                ) VALUES(
                                    '1',
                                    '2009-11-01T00:06:16-05:00',
                                    'stuff'
                                )'

                                
What I really should have done to avoid the errors would have been to run a CHANGE MASTER command that stated everything rather than skipping the details that the dump included.

After looking into this further, I find that as expected, the dump creates a master.info file with the master's proper bin-log and offset, and that master.info doesn't yet have the server connection details. Then after providing just the connection details through CHANGE MASTER, contrary to my expectation it then wipes out the bin-log and offset values rather than properly merging. I can fix this by then providing just the bin-log and offset values, which are properly merged into master.info.

Commands illustrating this are below:

[root@myhost ~]# # import the master's data
[root@myhost ~]# mysql -u root -p{secret} < /tmp/full.db.2009-11-14.dmp
[root@myhost ~]# # see what the dump put into master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this first iteration of the file has no connection info
14
bin-log.000494
169

test

3306
60
0





[root@myhost ~]# # set the partial details as documented
[root@myhost ~]# mysql -u root -p{secret} --execute="CHANGE MASTER TO MASTER_HOST='10.1.1.14', MASTER_PORT=3306, MASTER_USER='repl', MASTER_PASSWORD='supersecret';"
cat /var/lib/mysql/master.info
[root@myhost ~]# # check if that put anything in master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this second iteration dropped the bin-log and log position
14

4
10.1.1.14
repl
supersecret
3306
60
0





[root@myhost ~]# # set the remaining details as though nothing had been in dump
[root@myhost ~]# mysql -u root -p{secret} --execute="CHANGE MASTER TO MASTER_LOG_FILE='bin-log.000494', MASTER_LOG_POS=169;"
[root@myhost ~]# # check if that put anything in master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this third iteration merged in the bin-log and log position
14
bin-log.000494
169
10.1.1.14
repl
supersecret
3306
60
0





[root@myhost ~]# # set everything and see the results:
[root@myhost ~]# mysql -u root -p{secret} --execute="CHANGE MASTER TO MASTER_HOST='10.1.1.14', MASTER_PORT=3306, MASTER_USER='repl', MASTER_PASSWORD='supersecret', MASTER_LOG_FILE='bin-log.000494', MASTER_LOG_POS=169;"
cat /var/lib/mysql/master.info
[root@myhost ~]# cat /var/lib/mysql/master.info # notice this fourth iteration that sets everything looks like the third iteration
14
bin-log.000494
169
10.1.1.14
repl
supersecret
3306
60
0



So in short, don't rely on the dump to set master.info values for you. Just put them all into your mysql prompt similar to this:

mysql> CHANGE MASTER TO MASTER_HOST='10.1.1.14',
    -> MASTER_PORT=3306,
    -> MASTER_USER='repl',
    -> MASTER_PASSWORD='supersecret',
    -> MASTER_LOG_FILE='bin-log.000494',
    -> MASTER_LOG_POS=169;


Enjoy!

Yesterday I returned to Facebook after doing a 10 week experiment in self-imposed exile. I wrote the following note for my friends there, and I post it here in hopes that those outside my Facebook network may find it interesting/rewarding.

---

Hello Friends:

I struggle today to find the proper words to reintroduce and explain myself after a 10 week hiatus from this world where words are usually thrown about so carelessly. It's a new year, and I've been thinking about my Facebook-centric new year's resolutions really since October 22 which was the first day of my experiment. Let me first share my resolutions, then explain the experiment that brought me toward them, then finally describe the discoveries from my experience.

RESOLUTIONS

In the year 2009 I plan to:

* Visit Facebook less compulsively. Instead of logging in many times within an hour, I'll limit myself to twice daily.
* Focus on my objectives within Facebook and consume only the benefits that I actually want. It's important for me to stay in touch with friends. It is not important to entertain myself with comic videos, and I'll abstain from most of your status updates.
* Try to raise the quality of content in your minifeed by posting primarily stimulating information.
* Never be snarky.
* Block most third-party applications than send me invites.

I recommend that each of you also consider some Facebook-centric resolutions for yourself. How much have you thought about maximizing this network's benefits to you while avoiding its distractions and pitfalls? Let me tell you how about my own journey.

MY EXPERIMENT

This past quarter I took a great class at Stanford from a now-favorite professor, Howard Rheingold (http://rheingold.com). Howard famously coined the term "virtual community" and published a book (http://bit.ly/virc) about the concept in 1993. That was the same year the first graphical web browser, Mosaic, was released. The class I took with Howard was called "Virtual Communities and Social Media." After reading several assigned articles about multitasking, I was stumbled into "Is Google Making Us Stupid," published by Nicholas Carr in The Atlantic. Carr explored whether our minds are less able to achieve concerted focus because Google and our hyperlinked world are training our minds to optimize themselves to lightly visit brief snippets content. His article is at http://bit.ly/think.

I've been struggling lately to think clearly, and I've joked that if I had to retake the GRE, there's no way I would earn a score sufficient for Stanford to accept me again. I used to pride myself on my verbal skills, but of late I find my abilities have deteriorated. I've considered several theories. Is my problem that I read less than I used to? Is it that my writing within the workplace needs to be 100% clear and therefore I've stripped out all flourishes? Is it that I'm chronically sleep deprived? Or to paraphrase Carr, has my proclivity for multitasking rewired my brain and "made me stupid?" Facebook seems made for the short attention span, offering bite-sized visits that can be accomplished in under one minute. Maybe I should swear off Facebook for a while, I thought.

What good was Facebook to me anyway? It was impairing my ability to accomplish tasks that required focus, because at any mental pause in the day, I would instinctively click in to see what my friends were up to. This made working from my home office feel more like working from a dorm commons area with friends continually passing by. Further, I wasn't proud of my own contributions to Facebook. In the week leading up to my decision point, I had posted an unflattering picture of John McCain, posted a video of a chimp learning to ride a Segway, and teased a friend about a childhood photo.

So off the grid I went, to see what I could learn.

DISCOVERIES

* I can still read books! Within hours of dropping Facebook, I was enticed into reading a 300+ page book in just a few days. This was exactly the type of activity that Carr's article suggested we have a harder time doing in our over-multitasked digital world. I was delighted to find that yes, I still have it in me to read something more substantial than status updates. Before bed I took from the shelf "Six Degrees" by D.J. Watts, but instead of just scanning its structure as I expected, I ravenously read some eighty pages. Watts brilliantly explores the science of network theory, and he shows it applies to much more than just web-based networks or social networks. Check it out at http://bit.ly/networks.
* I really care about maintaining friendships (strong ties) and connections (loose ties) with you all on Facebook. I was concerned that some of you might send me messages and be put off that I seemed to ignore you. Especially after spending a week studying social capital and taking a few hours in class discussing this topic with the guru Tara Hunt (http://horsepigcow.com), I figured my experiment was costing me somewhat. Ironically, two people who I hadn't been in touch with messaged me on Oct 22, the first day of the experiment.
* Most everything in Facebook is good for at least some people, but that didn't mean that it was going to be good for me. I needed to filter out the content, activities, and habits that didn't lead toward my goals--and as with anything, I needed to know my goals.
* Facebook is an unparalleled way for me to promote a message. My sister Annie Poon created a beautiful animated entry for a Coldplay video competition. I greatly wanted to share it with my friends, and my best distribution channel would have been Facebook. But alas, I was locked out. I'll have it available in the future though.
* Twitter rules! While away from Facebook, Howard asked our Virtual Communities class to use Twitter for a while between ourselves to make sure we understood that medium. I thus set up an account intended for class use only. The problem was that by using Twitter even a little, I began to create some social capital and connections through Twitter with people I care about, and I got hooked. I'll post another note in the future about why I think the interest-based network of Twitter may be a more compelling place for me to spend time than the social-based network of Facebook. You'll be seeing many more of my Facebook status updates originating from Twitter.
* Ten weeks is too long to forgo Facebook. Once I returned, I found a backlog of 30 new friend requests, 18 new inbox messages, and seven other sundry requests. I need to spend some time playing catch up. Most of us can probably handle less frequent visits to Facebook, but ten weeks isn't a maintainable interval.

WRAP-UP

As I said earlier, I recommend that every Facebook user take some time to reflect on whether you are using the technology or the technology is using you to paraphrase hero Michael Wesch's most famous video about Web 2.0 (http://bit.ly/w20). Are you benefiting in excess of the gain Facebook gets from having you view its ads? Do you like the way you use Facebook? I'm certain as I follow my resolutions that I'll be much happier here than I used to be, although I'll be here less frequently.

I'd love to hear your thoughts on this note. Do any of these ideas strike a chord for you? Do you need some encouragement to go cold-turkey from Facebook for a while? Am I late to the realizations that most people already had?

Thanks!

Bill

Installing Movable Type

| | Comments (0)
One part of Oracle's post-acquisition integration is to retire the subsumed company's blogs. Thus, the blog I used to maintain at BEA Systems before we were bought by Oracle is going away. I've initiated the process of getting an Oracle blog, but ... well, why not take this opportunity to break free? I do care to preserve my blog posts, and if they live exclusively within a company, they're subject to certain upheavals now and again. Also, occasionally I'll want to blog about something that won't fit at Oracle, such as ... my notes on installing Movable Type.

I decided to migrate my old BEA posts to http://blog.billbenac.com, and I chose Movable Type as my blog platform. The process went like this:

  • SSH into web server
  • Login to MYSQL to create my database and user.

      drop database thedb;
      create database thedb;
      GRANT USAGE ON thedb.* TO 'theuser'@'localhost';
      GRANT SELECT,INSERT,UPDATE,DELETE,CREATE, INDEX, ALTER
      ON thedb.*
      TO 'theuser'@'localhost'
      IDENTIFIED BY 'thepassword';
  • Download MT: http://www.movabletype.com/download/personal/MTP-4.12-en.tar.gz
  • Extract as a folder under the root of the desired website such as /mt. This gives me blog.billbenac.com/httpdocs/mt/mt.cgi, among other things. This directory will be used for administration.
  • Browse to MT setup page, in my case http://blog.billbenac.com/mt with a login button that brings me to http://blog.billbenac.com/mt/mt-wizard.cgi
  • Use CPAN to download the necessary Perl modules
  • Fill out the wizard screens. When it asks about the configuration file, mt-config.cgi, click the link to view the generated file (Show the mt-config.cgi file generated by the wizard). Copy the contents, and put it on the server manually. Tell the wizard you did so, and continue.
  • After choosing a blog name, edit the URL and publishing path as appropriate. I made the URL and path different from the default. The URL is http://blog.billbenac.com/blog, and the publishing path ends with blog.billbenac.com/httpdocs/blog/. I created the directory manually on the server and set its permissions with: mkdir blog;chmod 777 blog
  • After logging in, the automatic post didn't display. I republished it, and that was that.
  • But man is it ugly. Login and try in the right actions column to refresh blog templates. Refresh the classics. Don't bother backing up templates. Hit continue. That might not have done anything though...
  • In the web admin section, to to Design-> Styles. Pick one. I grabbed Unity Tricolor from the Default Styles category.
  • Finally, I wanted to configure Live Writer to connect to the site so I could migrate my old BEA Dev2Dev posts. It didn't like my password a few times. It turns out Movable Type uses a different password for the API. Thanks to http://www.alexlomas.com/blog/2006/08/live_writer.html for this advice:

      Log into Movable Type.
      Click on your username in the top navigational menu to go to your profile.
      Scroll down to the bottom where you see API Password.
      Input a password of your choice and save. For security reasons, it should be different from your normal author password.
      Use this new password in live writer.
  • At this point I thought I was in business. But when I tried publishing from Live Writer to my site, I got this error:
  • Statement has no result columns to bind (perhaps you need to successfully call execute first) at {my path}/mt/extlib/Data/ObjectDriver/Driver/DBI.pm line 119.
  • One post suggested it could be a database version problem, and indeed, I was an unsupported MySQL from 2003: 3.23.58. I asked my hosting company to upgrade me, and now I'm on 4.1.15.
  • I publish, and? "Server Error Server Occurred. Statement has no result columns to bind (perhaps you need to successfully call execute first) at {my path}/mt/extlib/Data/ObjectDriver/Driver/DBI.pm line 119.
  • When I log into the web interface of my site though, I see that my posts that failed to publish actually did get created in my MT database, but they hadn't made been published entirely. I was able to open them up and successfully republish.
  • When new users registered to comment, the web site said it was sending them confirmation emails, but these didn't arrive. I had to go to System-> Preferences-> General and set the outgoing email address to resolve the problem.

I prefer Drupal!