Project on Hold

Several people have written me asking about the status of this project. Unfortunately, it has stalled, and I've run out of enthusiasm.

I loved Agenda. It was the most amazing program back in the DOS days, but today Agenda just doesn't cut it, even for simple tasks. Its time seems to have passed, maybe our world was simpler then.

I've added extensions that I thought would make it more useful - uploading attachments to items, stuff like that. I've put in a full-fledged note editor (tinyMCE), so I can embed images and links. I've added a search tool so I can run through items and create views 'on the fly'.

But the big problem is that most 'agenda' items arrive by email, facebook, twitter, text message, or shared calendars, and leave the same way. I used Agenda before smartphones and email became common; my phone at my desk, I was attached to it, and Agenda recorded every call, every activity, and every commitment. Today, I would use a CRM or bug-tracking tool for that, with the benefit of integrating with the rest of my team.

I'm back at university I started organizing the hundreds of research papers that I was referencing in my thesis with Agenda, and discovered it was much more effective to use Google's desktop search. I used Tomboy to organize my thesis (a sort of personal wiki), because Agenda couldn't do it the way I wanted. It occurred to me that maybe I could link Tomboy and Agenda functionality, but I couldn't see any extra zip from the Agenda part.

I'm starting to dabble with Twitter, and I can see how an Agenda-like interface to Twitter would be interesting. I'm interested in how teams can coordinate their activities, and perhaps building Agenda into a work environment has some merit. Maybe I'll get back to it next summer, with fresh eyes and new ideas.

I'm feeling like I have reunited with an old girlfriend, and discovered she had not aged well.

The Shallows

This is the first week of classes, and I'm registered in a course called 'Digital Games and Learning'.

I have some trepidation about this course.  One of the books on my summer reading list was Nicholas Carr's 'The Shallows - What the Internet is Doing to Our Brains'.  Strangely, it's a book about why people don't read books anymore.

Marshall McLuhan famously said "The medium is the message", a cryptic way of saying that communication technology doesn't just deliver content but also directly influences how we  think about that content.  Each new technology modifies our brains.

Carr's message is this:  if we had intentionally set out to build the most effective brain-modifying technology possible, it would probably look a lot like the Internet.   The more we use it, the more it changes us.  We're using it a lot.  And it may not be changing us for the better.

Specifically, Carr is talking about 'deep reading'.  Reading has been at the center of mankind's rapid intellectual progress for the last three millennia.   The printed book focused our attention, the written word organized our thinking, the library captured our collective progress.

But books and periodicals are quickly losing ground to newer media.  We now average 8 1/2 hours a day looking at our televisions, computers, and cell phones (often simultaneously), but only 20 minutes reading print publications.

Worse, those 20 average minutes includes seniors who tend to be the heaviest readers.  Young adults between 25 and 34 are only reading print media for 7 minutes a day.  (!)

Maybe that's not a problem.  Reading web pages is still reading.  But it's a jerky, distracted, multi-tasking,  shallow-browsing kind of reading, punctuated with hyperlinks, graphics, and videos.  The average time spent on a web page is only33 seconds - including banners and ads.

Reading an e-book on an iPad sounds similar to reading an old-fashioned book, but the reader is continually interrupted by emails  from friends, advertisers, and spammers, IM pop-ups, news updates, stock market reports, and RSS feeds.  Plus the endless temptations of viral distractions.    It's hard to achieve focus in all that noise.

Carr worries that we may have squandered mankind's greatest treasure, traded it for, well, we're not quite sure what yet.




When Willie Sutton was asked why he robbed banks, he replied, "Because that's where the money is".

So yes, let's explore games as a teaching medium.  Books aren't working anymore, and the Internet isn't going away.  Carr was wrong about the internet being the most effective possible tool - games tap into the most basic wiring of our brains and invoke our deepest primal fears and instincts.

If that isn't enough, we also have the toolbox of operant conditioning - game controllers that serve up electric shocks, and others that read our very thoughts.  We would have to justify to the ethics committee before we could use this stuff on rats.

Maybe we can turn this technology to our use, mindful of McLuhan's warning.  But I have this trepidation...

First Screenshots

First Screenshots

School started today, and I am using my version of Agenda for organizing my courses and interests.

The look-and-feel is still very primitive, especially since I haven't put in much effort with the style sheets yet. 

But the functionality is beginning to emerge, and the underlying code structures are strong enough to keep me moving forward.

Here's the file menu.  Each user has a library of files, usually sorted by last-accessed.  Because they are in a single database, searching them is easy.

Zooming in, below is what an Agenda view looks like.  (This example doesn't show any categories, I'm not happy with that part of the code yet.

Most of the screen widgets are from JQuery.  The menu tool at the top right is based on Jonathon Sharp's  jdMenu ( http://jdsharp.us), it 'floats' in an absolute DIV so that it stays on the page as you scroll through the items.

The tabs at the top are for 'sections'.   This view only has one section so far - 'Tab 1'.  I first tried 'Accordian' tabs, their vertical operations were closer to the original Agenda concept but not quite as functional.

The icons let me attach files to items, edit the details, link the item to my Google calendar (not yet done), and email an item. 

As well, they let me manipulate the order of items.  If anyone remembers 'MaxThink',  I'm hoping to add some of its  hierarchical and bin-sort capabilities.  If you don't know about MaxThink, then you MUST visit  http://www.maxthink.org.

Zooming into an item gives access to the underlying notes and blogs.  I'm using TinyMCE as my note editor, which allows me to embed graphics and hyperlinks into my notes, and also provides spell-checking.  Not pretty yet, but functional.

I've built the horizontal tabs into all the pages.  It's easy to disable later, but it occurs to me that I will soon want to group files, and that the details of an item may become complex enough to justify tabs.

The design is drifting away from the original Agenda.  Two reasons :  my 'university notes' application is pretty specific (and requires richer items than say a contact manager), and the capabilities of a web program are so different from what the original DOS environment offered.

The nicest part about backing Joomla! out of the design is that I can now use AJAX to update pages on the fly.  This potentially solves some of the UI issues , like editing directly on the view without requiring a SUBMIT button.  But no hurry, first I want to finish the functionality.

And that may take some time at the slow pace I am going.  I figure I'm giving this project about 5-6 hours a week.   There's no hurry, I'm up and running, I'll focus first on the features that I need for my own use.   That might mean adding APA-style references before finishing categories.

New Laptop

It's back-to-school time, I've bought a new laptop - a Sony Vaio F-series with a full 1920x1080 resolution.  I read a lot of research papers, which are typically PDF documents formatted into multiple columns.  It was painful on the old laptop - I'd have to scroll up and down each page.   So I would put the screen into 'portrait' mode to view the documents as full pages, but I didn't have quite the resolution to make it comfortable.  With this new machine, reading papers is a joy.

But there's a downside - I'm back to running Windows.   I spent almost a week trying to put my beloved Linux environment on this machine, but gave up defeated.  The problem is the video card - I can't get the NVIDIA drivers available in Linux to work on this machine.  I know that Sony uses NVIDIA hardware under the hood, there's a small sticker advertising 'NVIDIA GEFORCE WITH CUDA on the keyboard.  But I'm not a X-Org guru, and xserver internals defeated me.

It started auspiciously - I repartitioned the disk, loaded an old Ubuntu 'Hardy Heron' CD, and got a basic system up in a very short time (under low resolution, but I didn't know about the problem then).  Then I let the machine upgrade itself version-by-version to the current 'Lucid Lynx', which was time consuming but effortless.

Then I tried to configure the screen - and lost the ability to see what I was doing.  If you mess up the screen, you can't even find an 'undo' button - it's time to power down and roll back, try again.   I must have rebooted about 30 times before I finally gave up.  Finally fell back to the system recovery disk (which took a full day), and put Windows back in.



Not totally Windows, I put in the Ubuntu 'Windows Installer' version - a really neat program that drops Ubuntu as a file under Windows.  It uses the Windows boot-selection instead of GRUB (so the default is to open Windows).  But I'm running Linux without NVIDIA so the resolution is low, Windows is going to be my default environment for a while.  

I've pointed the Apache servers under both Windows and Ubuntu to the same directory so I can test my code under both.  I am going to try to re-install MySQL to share the same database files, not sure if that's possible.



Windows 7 is a huge improvement over its horrible predecessor.  It has copied some nice Linux features like pinning programs to the toolbar.  But it is slow and clunky by comparison with Ubuntu - especially with the mandatory anti-virus programs endlessly chugging away in the background.  I've got an i7 processor that shows 8 CPU's in the task manager, 6GB of Ram, and a 7,500 RPM drive, but it FEELS slower than my old laptop under Linux.

Good news is that a lot of my favorite tools run under Windows.  Firefox, of course, and PHP, Apache, MySQL.  But also Komodo Edit and Inkscape, which are amazing - check them out.  GEdit was my all-purpose quick-and-dirty editor under Linux, Notepad keeps breaking things.  Happily, GNOME provides a Windows build of GEdit.

In fairness, it's nice to know where everything is again.  I spent 20 years developing under Windows, and know it much better than I know Linux.  Nothing important has really changed since NT 4.0, the DOS commands still work.   It's nice that I will be able to use Adobe's FLASH tools, and OFFICE is clearly better than OpenOffice, but the different isn't enough that I would pick Windows if I had the choice. 

The one happy surprise is a program called PDF-XChange Viewer, which replaces Okular as my PDF viewer and annotator.  The free 'base version' is BETTER than Okular (in fairness, Okular was just a student project for a Google Summer-of-Code).




I give the Sony VAIO high marks.  It's lightweight and clean-looking.  It cost much less than I expected - about half of the comparable DELL machine.  I LOVE the big screen - it's amazingly bright and crisp.  Two complaints - the mouse touchpad is not deep enough, so I keep touching it accidentally (I had to turn off the tap features).   And although they have provided every kind of interface imaginable - SATA, HDMI, SD card, ExpressCard, FireWire, optical headphone jack, and their proprietary MagicGate, for some reason there are only two USB ports. 

The typing keyboard isn't centered in front of the screen, since Sony provides a full keyboard with numeric pad.  Seems a bit weird at first.  But with the huge resolution, I rarely run a program in full-screen. 



A number of people have written to me asking for a copy of Web-Agenda when it is ready.  When I ported Web-Agenda onto this machine, I realized that the major installation task facing them will be getting Joomla! running and configured.  There are simple tools for installing PHP, MySQL, and Apache, including WampServer2 that will install all three at one time.  But Joomla! has a sharp learning curve.

So I have walked through the source and ripped out Joomla!, replacing most features with local code.  Still doing some cleanup where I didn't provide my own CSS (so that I inherited the look-and-feel of whatever template was loaded).

I hope I can provide some screenshots with the next blog posting.

Backups

One of the reasons that I wanted Agenda was because I am going back to university in September.  I cannot think of a better tool for organizing research papers and managing my thesis.  That's going to be my first real 'use-case'.

My version of Agenda is now at the point where I can enter items into a file, creating categories as I go.   The basic structure is in place, and the specific fields are extensible.  I plan to keep improving the code over  the next few months, with priority to specific functions as I need them.

But before I trust this program with my precious data, I want a backup utility.  

Backing up my version of Agenda is going to be complex, for a few reasons:

• I'm using a database that contains multiple 'workbooks', instead of the single-workbook text files that the original Agenda used.  But I don't want to back up the whole database (that happens anyhow on a scheduled basis).  I want to take a snapshot of a specific workbook, that I can restore later.  This means extracting specific records from the database. 
  
• I want to be able to restore a copy of a workbook beside the original workbook in the database, so I can't rely on unique keys being unique unless I recreate them.  I need to handle the records intelligently.
  
• My version of Agenda supports 'attachments', such as PDF files.  If I back up a workbook, I expect the attachments to come with the backup.  This means that I need to create a ZIP file or similar, not just a set of SQL statements that I can present as a text file (the usual way of backing up a Joomla database).
  
• The text editor that I am using for Notes allows inserting images.  Most images are 'local' - they are uploaded and then linked from the server rather than from a remote site.  I'd really like the images to be scooped up and added to the backup - if I restore into a different environment or directory structure, then the images may not be there or the links may be broken.  So I want to scan all Notes, grab any inserted images, and stick them into the ZIP file..

The 'broken link' problem is also true for remote links to other web pages - both images and ordinary URL links - but I think they are less likely to cause a problem.  In any case, I can't backup entire web pages that have been pointed to.

Backup and Restore will be a big component of the new Agenda.  It has some design implications to how I support uploading attachments and images, so I'm going to have to make some changes to existing code.

For now, I'm running my Agenda on my laptop.  When I move it to a proper web server, I'll want to backup my local workbooks and restore them on the real server.   And if anyone uses my Agenda on that server, they may eventually want to back up their data and move it to another instance of Agenda that they host themselves.

My Agenda is usable - barely.  I'm keeping some 'ideas' and 'todo' items in it.  But until I have the ability to backup, I won't use it for anything important.

So backup is what I am working on now.

Progress Report

I've been working steadily on the Agenda Rewrite, probably about 15 hours per week.  I'm happy to report that progress is good. 

I've got the basic operations running for adding items and displaying views.   I'm learning  Jquery (http://jquery.com) – what a fabulous tool – as the basis for my user interface.

The strategy of testing before coding has fallen away.  One of the reasons is that I have refactored and refactored – redesigning as I understand each of the design problems better.   I'm in awe of the original designers, and humbled by how hard this program is to understand - even with a working model in front of me.

Instead of test-driven programming, I'm back to my favorite strategy of data-driven programming.  For example, I have a central repository of 'fields' in a CSV file.  Agenda has hundreds of fields that control the behavior of Views, Items, etc.   It controls how the data gets written to databases, what the initial values are, and how HTML forms are presented, validated, and unloaded.  

I've moved as much 'code' to this table as possible.  The code that exercises this data gets thoroughly tested – there is only ONE way to draw a form, even though there are a dozen forms in the system  - and an error in a data table is often easy to find mechanically.



By now, the database objects have been fully refactored three times.  The first cut was to create one for every type of entity – files, views, sections, columns, items, categories, etc.   

I have programmed carefully – there was a database class, and a datatable class that inherited from it.  There is an interface that described the minimal set of operations that these classes would provide.  Each of my entities had a further class which would hold any special SQL statements that I wanted – there was no SQL allowed outside this set of classes.

Although I didn't use an Object Database, my data design centered around a 'cargo' field for every entity containing a serialized array and an object to interpret it.   I get most of the benefits of an OODB, but still use MySQL  

This cargo was the only thing I sent to the datatable classes, they were responsible for reaching in and finding the appropriate key fields that were needed for SQL performance.

I quickly realized that 'file' is different from the rest, if for no more obvious reason than because the rest all had 'fileID' as a key and it didn't.  It struck me that maybe I should have an internal file system for tracking information about files (such as permissions, versions, etc), and maybe that was just a 'system' Agenda file with a special name.

But this fell by the wayside quickly because of the bootstrap problem.  I needed a file system to get some programming done, and I needed a lot of Agenda to be working in order to have a file system.

By this time, I had reconsidered where I wanted to store the database objects.  The 'View' table will be very infrequently updated compared to the 'Item' table, and the 'Assignment' table (linking items to categories) didn't fit into this model well. 

I found my early code would read the file record, and then quickly grab view, section, and column records - they were all required.   It seemed more efficient to park them with the file record even if it got a lot bigger.

So I refactored again – turning the 'File' cargo into a repository for File, View, Section and Column.  When I loaded a file, I got the complete schema for every way that you could present the view.

Now I have a new class layer - ' Instance' – with an interface a set of operations against these entities.  Instance works the same way regardless of whether I'm asking for a View (which is physically in the File table) or an Item (which has its own physical table).

I'm ambivalent about whether to put Categories into this big file object.  There's lots of information in them that is needed to present a view, and many operations need every possible category to be checked.  I've hedged by coding their Instance to put them into the File table, just need to be careful never to short-cut directly to them.



The joy of programming without a schedule is that I can continue to refactor until I get it right.  With each step I take, I learn more about the underlying problems that the original designers had faced.

I have made a few small changes in their design, nothing important.

For example, I never liked 'Sections' in the original Agenda.  If the sections described a set of similar children in the category tree, the view always had to be tinkered with if you added another child (for example adding a new salesman to a view that had a section for each salesman).   But sections could also be completely different from each other.

The need for sections is to allow related stuff to be bundled into a single view.   So that's the way I'm treating it.  Each View will have a set of tabs ('Tabviews') that can be completely different.   Each tab will have either a single section based on a category, or a group of identical sections based on the parent of a set of categories (with an accordion control to view them).

I think this represents what the original programmers were trying to achieve.




I'm starting to think about finding a name for this project.  'Agenda Rewrite' doesn't seem elegant.

If anyone has a name suggestion, I would be happy to hear about it.




Nothing planned for the next three weeks.   My lovely bride is taking me for a bicycle vacation in Italy - I shall be climbing the hills of Emilia-Romagna from  a base at the Belvedere Hotel in Riccione.

http://www.belvederericcione.com/en/about-belvedere-bike/

DOS Lotus Agenda, under Linux

Running Agenda on my DOS netbook isn't convenient, but necessary since both my desktop and laptop machines are running Ubuntu.

But I still need and use Agenda.   It provides both a model for the new application I'm building, and my organizer for ideas, todo lists, etc.

So I looked into the 'Ubuntu Software Center' on my laptop (a friendly wrapper for apt-get, the Linux utility that installs software applications from the internet).  Sure enough, two different DOS emulators waiting to be tried.

A few clicks later, I had a virtual DOS 'C' directory, and copied in the Agenda files from the netbook.  And I'm up and running.